Rights statement: This is the author’s version of a work that was accepted for publication in Computers and Security. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Computers and Security, 74, 2018 DOI: 10.1016/j.cose.2018.01.008
Accepted author manuscript, 4.42 MB, PDF document
Available under license: CC BY-NC-ND: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License
Final published version
Licence: CC BY-NC-ND: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License
Research output: Contribution to Journal/Magazine › Journal article › peer-review
Research output: Contribution to Journal/Magazine › Journal article › peer-review
}
TY - JOUR
T1 - Enhance virtual-machine-based code obfuscation security through dynamic bytecode scheduling
AU - Kuang, Kaiyuan
AU - Tang, Zhanyong
AU - Gong, Xiaoqing
AU - Fang, Dingyi
AU - Chen, Xiaojiang
AU - Wang, Zheng
PY - 2018/5
Y1 - 2018/5
N2 - Code virtualization built upon virtual machine (VM) technologies is emerging as a viable method for implementing code obfuscation to protect programs against unauthorized analysis. State-of-the-art VM-based protection approaches use a fixed scheduling structure where the program always follows a single, deterministic execution path for the same input. Such approaches, however, are vulnerable in certain scenarios where the attacker can reuse knowledge extracted from previously seen software to crack applications protected with the same obfuscation scheme. This paper presents Dsvmp, a novel VM-based code obfuscation approach for software protection. Dsvmp brings together two techniques to provide stronger code protection than prior VM-based approaches. Firstly, it uses a dynamic instruction scheduler to randomly direct the program to execute different paths without violating the correctness across different runs. By randomly choosing the program execution path, the application exposes diverse behavior, making it much more difficult for an attacker to reuse the knowledge collected from previous runs or similar applications to launch an attack. Secondly, it employs multiple VMs to further obfuscate the mapping from VM opcode to native machine instructions, so that the same opcode could be mapped to different native instructions at runtime, making code analysis even harder. We have implemented Dsvmp in a prototype system and evaluated it using a set of widely used applications. Experimental results show that Dsvmp provides stronger protection with comparable runtime overhead and code size, when it is compared to two commercial VM-based code obfuscation tools.
AB - Code virtualization built upon virtual machine (VM) technologies is emerging as a viable method for implementing code obfuscation to protect programs against unauthorized analysis. State-of-the-art VM-based protection approaches use a fixed scheduling structure where the program always follows a single, deterministic execution path for the same input. Such approaches, however, are vulnerable in certain scenarios where the attacker can reuse knowledge extracted from previously seen software to crack applications protected with the same obfuscation scheme. This paper presents Dsvmp, a novel VM-based code obfuscation approach for software protection. Dsvmp brings together two techniques to provide stronger code protection than prior VM-based approaches. Firstly, it uses a dynamic instruction scheduler to randomly direct the program to execute different paths without violating the correctness across different runs. By randomly choosing the program execution path, the application exposes diverse behavior, making it much more difficult for an attacker to reuse the knowledge collected from previous runs or similar applications to launch an attack. Secondly, it employs multiple VMs to further obfuscate the mapping from VM opcode to native machine instructions, so that the same opcode could be mapped to different native instructions at runtime, making code analysis even harder. We have implemented Dsvmp in a prototype system and evaluated it using a set of widely used applications. Experimental results show that Dsvmp provides stronger protection with comparable runtime overhead and code size, when it is compared to two commercial VM-based code obfuscation tools.
KW - Software protection
KW - Program diversity
KW - Obfuscation
KW - Code security
KW - Instruction virtualization
KW - Dynamic scheduling
KW - Reverse engineering
KW - Cumulative attack
KW - Code virtualization
KW - Code obfuscation
U2 - 10.1016/j.cose.2018.01.008
DO - 10.1016/j.cose.2018.01.008
M3 - Journal article
VL - 74
SP - 202
EP - 220
JO - Computers and Security
JF - Computers and Security
SN - 0167-4048
ER -