No, I dont think you can restrict an users access to one particular virtual machine. the user privileges are not so defined. However, you can provide the user the remote control access like vmware-vmrc command to that particular virtual machine only instead of vsphere client access.
That way the user can power ops, remote control that particular vm only.
If interested, let me know, I can gather the syntax, else it should be easily googable.