In case this helps anyone...
When attempting to upgrade (by booting from OES2018 iso) my mail server from OES 2015SP1 to OES2018 I couldn't get the install/upgrade GUI to come up - it would just hang with a cursor on the screen and some messages on one of the consoles about not being able to use the display (sorry, forgot the exact message).
After a couple of tries, I remembered that the mail server had been around for awhile, and was the only remaining VM on the host still running VM Hardware version 7. So I shut down the VM and upgraded the VM hardware to version 11. After that, the GUI came up no problem.
I don't know what the mininum VM version would be, but it's greater than 7.