Process-level virtual machine (PVM) based code obfuscation is a viable means for protecting
software against runtime code tampering and unauthorized code reverse engineering. PVM-based
approaches rely on a VM to determine how instructions of the protected code region are scheduled
and executed. Therefore, it is crucial to protect the VM against runtime code tampering that alters
the instructions and behavior of the VM. This paper presents VMGuards, a novel PVM-based code
protection system that puts the security of VM as the first class design concern. Our approach
advances prior work by promoting security of the VM as the first class design constraint. We achieve
this by introducing two new instruction sets to protect the internal implementations of critical
code segments and the host runtime environment where the VM runs in. Our new instruction
sets not only have an identical code structure as standard virtual instructions, but also provide
additional information to allow the VM to check whether the critical internal implementation or the
runtime environment is affected. We evaluate our approach by using a set of real-life applications.
Experimental results show that our approach provides stronger and more fine-grained protection
when compared to the state-of-the-art with little extra overhead.