A long, long time ago, there was a project called FreeVMS which aimed to do that. Sadly, it didn't get much further than booting.

Edit: https://github.com/rroart/freevms . It's definitely not big and professional like OpenVMS :-)