Just installed ESXi v5.1 and the W8.1 preview.
Received the aforementioned error:
"Setup cannont continue. The Microsoft Runtime DLL installer failed to complete the installation"
Someone suggested TMP & TEMP, but I had already setup the User TMP & TEMP variables & directories.
BUT, when I checked the System Environment Variables, I saw that the system TMP & TEMP environment variables were set to non-existent values.
This was remedied without any other changes & the VMtools installation completed successfully.