Home > Research > Publications & Outputs > Enhance virtual-machine-based code obfuscation ...

Electronic data


    Rights statement: This is the author’s version of a work that was accepted for publication in Computers and Security. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Computers and Security, 74, 2018 DOI: 10.1016/j.cose.2018.01.008

    Accepted author manuscript, 4.42 MB, PDF document

    Available under license: CC BY-NC-ND: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License


Text available via DOI:

View graph of relations

Enhance virtual-machine-based code obfuscation security through dynamic bytecode scheduling

Research output: Contribution to Journal/MagazineJournal articlepeer-review

  • Kaiyuan Kuang
  • Zhanyong Tang
  • Xiaoqing Gong
  • Dingyi Fang
  • Xiaojiang Chen
  • Zheng Wang
<mark>Journal publication date</mark>05/2018
<mark>Journal</mark>Computers and Security
Number of pages19
Pages (from-to)202-220
Publication StatusPublished
Early online date2/02/18
<mark>Original language</mark>English


Code virtualization built upon virtual machine (VM) technologies is emerging as a viable method for implementing code obfuscation to protect programs against unauthorized analysis. State-of-the-art VM-based protection approaches use a fixed scheduling structure where the program always follows a single, deterministic execution path for the same input. Such approaches, however, are vulnerable in certain scenarios where the attacker can reuse knowledge extracted from previously seen software to crack applications protected with the same obfuscation scheme. This paper presents Dsvmp, a novel VM-based code obfuscation approach for software protection. Dsvmp brings together two techniques to provide stronger code protection than prior VM-based approaches. Firstly, it uses a dynamic instruction scheduler to randomly direct the program to execute different paths without violating the correctness across different runs. By randomly choosing the program execution path, the application exposes diverse behavior, making it much more difficult for an attacker to reuse the knowledge collected from previous runs or similar applications to launch an attack. Secondly, it employs multiple VMs to further obfuscate the mapping from VM opcode to native machine instructions, so that the same opcode could be mapped to different native instructions at runtime, making code analysis even harder. We have implemented Dsvmp in a prototype system and evaluated it using a set of widely used applications. Experimental results show that Dsvmp provides stronger protection with comparable runtime overhead and code size, when it is compared to two commercial VM-based code obfuscation tools.