Source-linked AI summary
VMSpec: A Decompiler-based Approach against Virtual Machine Obfuscation
Fabian Kilger, Leonard Auer, Manuel Andreas
TL;DR
Trace-based methods analyze only the execution path taken for given inputs, limiting recovery of VM-obfuscated program behavior. VMSpec automates VM processor-specification generation for existing decompilers, producing semantically correct, high-quality decompilation in nearly all cases, including complete behavior recovery for complex control flow.
Problem
Trace-based VM deobfuscation is limited to the code path taken for given inputs, restricting analysis of program behavior.
Source: §1 · p. 1Method
VMSpec uses dynamic symbolic execution to analyze bytecode handlers and automatically generate SLEIGH processor specifications for existing decompilers.
Source: §1 · p. 2, §9 · p. 13Results
VMSpec produces semantically correct, high-quality decompilation in nearly all cases and demonstrates complete behavior recovery for VM-obfuscated programs with complex control flow.
Source: §1 · p. 2, §9 · p. 13Takeaways & Limitations
The generated specifications let analysts use existing decompilers and analysis frameworks to analyze unknown VMs.
Source: §1 · p. 2Dynamic symbolic execution can incur path explosion, increasing analysis time and memory consumption despite context concretization analyzing most handlers.
Source: §7.2 · p. 12
Abstract
from arXiv · showhide
Malicious actors frequently obfuscate their code to hinder investigative reverse engineering efforts. Among the most formidable techniques is virtual machine (VM) obfuscation: programs are translated into bytecode for a custom VM, often with a randomized virtual instruction set and distributed alongside a corresponding interpreter. Manual analysis and deobfuscation scale poorly due to the complexity of VM obfuscation and the diversification techniques that render each VM and interpreter unique. Existing methods primarily focus on simplifying execution traces and identifying virtual execution boundaries. However, comparatively little attention has been given to recovering higher-level program semantics from VM-obfuscated code. To address this gap, we propose VMSpec, a novel decompiler-based approach that automates the construction of VM processor specifications, enabling existing, well-known decompilers to analyze the VM-obfuscated code. We evaluate our prototype of VMSpec on 147 programs from an established obfuscation benchmark, comprising hash functions, standard algorithms, and randomized functions, obfuscated with different configurations of the Tigress and Code Virtualizer obfuscators. To demonstrate real-world applicability, we additionally include six domain generation algorithms stemming from real-world malware. VMSpec completely strips VM-obfuscation from nearly all obfuscated samples, producing output comparable to Ghidra's decompilation of the original code. Furthermore, it is the first method capable of reconstructing the complete program behavior of VM-obfuscated programs with infinite, input-dependent paths and recursive calls.
1 Introduction
VM obfuscation complicates reverse engineering by hiding program semantics in diversified interpreters, while trace-based methods cannot generally recover complete behavior. VMSpec generates processor specifications from VM handlers so existing decompilers can analyze the obfuscated program.
1 Introduction
- Dynamic tracing limits analysis to executed paths, and path exploration can be infeasible when programs have many possible paths.Prior handler analyses also extract incomplete behavior when restricted to single traces and when they ignore control flow.Source: §1 · pp. 1–2
- VMSpec produces semantically correct decompilations comparable to unobfuscated samples across Tigress and Code Virtualizer configurations, including complete behavior for input-dependent loops.The evaluation covers 147 benchmark programs and six domain generation algorithms from real-world malware.Source: §1 · p. 2
- VMSpec applies dynamic symbolic execution to VM handlers and generates SLEIGH specifications containing instruction encodings and behavior.It uses a custom memory model to analyze handler behavior with symbolic addresses.Source: §1 · p. 2
- The generated specifications let existing decompilers and analysis frameworks recover high-level behavior, including infinite input-dependent paths and recursive calls.The paper identifies this as the first approach to reconstruct that complete behavior.Source: §1 · p. 2
2 Background
The background describes VM obfuscation and prior deobfuscation approaches, then explains how Ghidra’s SLEIGH language specifies processor instructions for decompilation.
2.1 Virtual Machine Obfuscation
- VM obfuscation translates original code into bytecode for a custom processor and packages an interpreter with the program; diversification hinders reuse of analysis across samples.Interpreters fetch, decode, dispatch, and execute virtual instructions, while commercial obfuscators may encrypt operands and distribute bytecode.Source: §2.1 · p. 2
2.2 VM Deobfuscation Approaches
- Trace-based methods simplify executed traces but cannot ensure complete behavior, while VPC-sensitive analysis can reason over whole programs but faces diversification and evaluation limits.Handler-analysis methods based on single traces omit control flow or memory semantics; VMSpec generates instruction encodings and VM semantics, including control flow, from complete handlers.Source: §2.2 · pp. 2–3
2.3 Ghidra and SLEIGH
- SLEIGH specifies instruction byte encodings and P-Code semantics through constructors that include display, bit-pattern, disassembly-expression, and semantic sections.Disassembly expressions can compute values before P-Code emission, while bit-pattern matching can resolve operand-based branches.Source: §2.3 · p. 4
3 Virtual Machine Model
The VM model names the components needed to analyze and define handler behavior, including code, stack, state, context, and global data.
3 Virtual Machine Model
- The virtual program counter (VPC) is a register pointing to the current bytecode instruction.The code region contains the bytecode executed by the VM.Source: §3 · p. 4
- The virtual stack pointer (VSP) models the top of the stack, though a VM may not use a stack.Source: §3 · p. 4
- VM context includes prior-instruction information and variables involved in opcode or operand encryption that affect instruction decoding.Global data, including strings, is modeled as absolute references.Source: §3 · p. 4
4 Design
VMSpec models a VM as an unknown processor, infers a SLEIGH specification, and uses it to support downstream decompilation. Its design moves from VM-state preprocessing and handler discovery through symbolic handler analysis and code generation to decompilation.
4 Design
- VMSpec infers a SLEIGH processor specification describing a VM disassembler and lifter, enabling downstream analysis, particularly decompilation.Source: §4 · p. 4
4.1 Preprocessing and Handler Detection
- Preprocessing extracts VM information, while an extensible detector and handler analysis iteratively discover handler boundaries until no more handlers are found.The prototype implements obfuscator-specific solutions for Tigress and Code Virtualizer because generic approaches lack available implementations.Source: §4 · p. 4, §4.1 · p. 5
4.2 Symbolic Execution of VM Handlers
- DSE explores handler paths and records memory operations and calls; symbolic reads, call modeling, and context concretization address symbolic memory, external calls, and specialized semantics.The resulting state tree is simplified by removing dead memory operations and merging nodes; operand-identification heuristics are applied after simplification.Source: §4.2 · pp. 5–6
4.3 Code Generation
- VMSpec translates the state graph into processor-specification code by traversing it in topological order and translating statements.Source: §4.3 · p. 6
4.4 Decompilation and Postprocessing
- The generated processor specification is loaded into Ghidra to decompile virtualized code, with bytecode disassembly and calling-convention specification used to produce readable, correct code.Source: §4.4 · p. 6
5 Implementation
The prototype combines handler analysis with Ghidra integration and obfuscator-specific support. Its context analysis is local, Tigress support is limited to a particular dispatch method, and SLEIGH output receives additional optimization passes.
5 Implementation
- The prototype implements handler analysis using angr and adds Ghidra analysis code, including stack-analysis and SIMD-related changes.The general implementation comprises 6,883 Python LOC and 1,296 Java LOC; additional changes touch 83 Java and 1,062 C++ LOC.Source: §5 · p. 6
5.1 Context Analysis
- Context analysis tracks local VM encryption-register state but not program context such as stack and register state; unresolved jumps become custom P-code operations.Ghidra does not allow dynamic language-specification updates, so integrating a global analysis is left to future work.Source: §5.1 · p. 6
5.2 Tigress-Specific Implementation
- Tigress support focuses on ifnest dispatch and does not enable additional obfuscations aimed at hiding VM state or handlers.Preprocessing identifies the loop header, state variables, and handler boundaries using control flow.Source: §5.2 · p. 6
5.3 Code-Virtualizer-Specific Implementation
- For Code Virtualizer table dispatch, handler detection matches symbolic unconstrained jump expressions to the dispatcher table and extracts the opcode formula.Context is concretized for native transitions or obfuscated control flow, and also when the static branch count exceeds 50.Source: §5.3 · p. 6, Figure 3 · p. 7
5.4 SLEIGH Specification Generation
- VMSpec adds a post-disassembler constant-propagation and dead-branch-removal pass because it does not generate an optimized SLEIGH representation.For the conditional-jump example, the initial specification covers possible conditions and operand sizes and uses an indirect branch because the destination requires computation.Source: §5.4 · p. 7
6 Evaluation
The evaluation tests VMSpec’s semantic correctness, output similarity, and computational cost across benchmark programs and malware-derived DGAs, using Tigress and Code Virtualizer configurations. It compares reconstructed code with unobfuscated decompilation and prior output where applicable.
6.1 Experiment Setup
- The evaluation used Tigress and Code Virtualizer and qualitatively compared readability with source, unobfuscated decompilation, and Salwan et al. output.Code Virtualizer was selected as representative of Themida’s virtualization technology; implementation limitations prevented automated VMProtect analysis.Source: §6.1 · pp. 7–8
6.1.1 Program Sets
- Program sets span hash functions, small crafted programs, standard algorithms with input-dependent loops or recursion, randomized functions, and six malware-derived DGAs.The random-function set was sampled down to 50 corrected functions; undefined behavior was removed from sets A–D to preserve semantics for validation.Source: §6.1.1 · p. 8, Table 3 · p. 8
6.1.2 Sample Sets
- Samples used six Tigress configurations across stack, register, and mixed architectures, plus Code Virtualizer Tiger Red and Fish Red configurations.Tigress used default and SuperOperators settings per architecture; Fish Red added a CISC variant with longer, more complex instructions.Source: §6.1.2 · pp. 8–9, Table 4 · p. 10
6.2 RQ1: Semantic Correctness
- Semantic correctness was checked by recompiling decompilation output and using randomized input-output tests designed to cover every feasible branch; 1,000 inputs sufficed for the samples.The process also required defining Ghidra-specific types, removing common decompilation artifacts, and reusing original global-variable definitions.Source: §6.2 · p. 9
6.3 RQ2: Code Similarity
- The Salwan et al. comparison was limited to adjusted hash programs: its tool did not support input-dependent loops or recursion, and worked on 8 of 10 hashes.The evaluation also qualitatively compared VMSpec output with source and unobfuscated x64 decompilation.Source: §6.3 · p. 9
6.2 RQ1: Semantic Correctness
- 918 Tigress samples were semantically correctly decompiled; Code Virtualizer Tiger yielded 152/153 correct samples and Fish 149/153, with failures involving jump tables, context precision, or memory.Table 5 summarizes the decompilation results; the cited prose specifies the counts and failure causes.Source: §6.2 · p. 9, Table 5 · p. 11
6.3 RQ2: Code Similarity
- For hashes, final VMSpec output nearly matches original-code decompilation; other sets remain slightly more complex in LOC and tokens, while randomized set D is less complex after dead-code removal.In the DEKHash example, VMSpec recovers equivalent loop control flow; the recursive binary-search example illustrates array-readability artifacts in Ghidra.Source: §6.3 · pp. 9–11, Figure 5 · p. 10, Table 5 · p. 11
6.4 RQ3: Computational Cost
- Handler-analysis time scales roughly linearly with handler instructions, but Code Virtualizer needs an order of magnitude more runtime per instruction; most decompilations take 40–60 seconds.The largest reported decompilation run took 170.8 seconds; Code Virtualizer Fish’s complex handlers and encryption branches contribute to higher analysis cost.Source: §6.4 · p. 11
7 Discussion
The discussion considers framework integration alongside limitations in handler analysis, specialization, decompilation, and resistance to countermeasures. It also identifies future integration and analysis work.
7 Discussion
- Figures 7 and 8 plot handler-analysis and decompilation runtime, respectively, against VM handler complexity.Source: Figure 7 · p. 12, Figure 8 · p. 12
7.1 Integration With Other Frameworks
- SLEIGH specifications can connect VMSpec to angr, radare2, and BAP, but the benefits of further analysis of decompiled code versus P-Code remain future work.Source: §7.1 · p. 12
7.2 Limitations
- Complex commercial handlers can cause path explosion; context concretization analyzes most handlers but increases analysis time and memory use.Source: §7.2 · p. 12
- Native-transition specialization currently uses instruction-local context, although some targets, such as VMProtect’s, require stack context across VM instructions.Source: §7.2 · p. 12
- Supporting additional VM obfuscators requires more VM-state identification and modeling, while complex MBA arithmetic can hinder analysis without expression simplification.Source: §7.2 · p. 12
- VMSpec inherits decompiler weaknesses, including type and function-signature recovery and readability problems from overoptimization, though its output may improve with decompiler advances.Source: §7.2 · p. 12
- For partially virtualized code, VMSpec can decompile snippets but cannot directly analyze their semantics in the embedding function; DSE or black-box methods may suit such snippets.Source: §7.2 · p. 13
7.3 Potential Countermeasures
- Potential countermeasures include hiding VM state, targeting DSE, exploiting decompiler weaknesses, and obfuscating the underlying program logic.Source: §7.3 · p. 13
8 Related Work
Related work includes manual and automated methods for analyzing virtual machines, instruction-mapping attacks, and black-box expression simplification. The passages describe their limits and potential connections to VMSpec.
8 Related Work
- Rolles manually mapped VMProtect handlers to a custom IR and used regular expressions, but diversification makes that process infeasible; VMSpec automates handler analysis and maps handlers to P-Code.Source: §8 · p. 13
- Sharif et al. automate VM-state, handler-boundary, and control-flow-graph recovery, while Zhang et al. identify handler boundaries and diversification; VMSpec may combine with these methods.Source: §8 · p. 13
- Frequency analysis can fail on complex mappings or operand-dependent instructions, while chosen-instruction attacks recover reusable VM details that could aid VMSpec preprocessing and handler detection.Source: §8 · p. 13
- Black-box deobfuscation can simplify expressions and resist many obfuscations, but current approaches omit memory and control-flow semantics; they could enhance VMSpec against arithmetic obfuscation.Source: §8 · p. 13
9 Conclusion
VMSpec automates VM processor-specification generation using dynamic symbolic execution to extract instruction encodings and semantics from handlers. Evaluation found high-quality, semantically correct output in nearly all cases, including complete recovery for complex control flow.
9 Conclusion
- Across 147 synthetic programs and six malware DGAs, VMSpec produced semantically correct, high-quality decompilation in nearly all cases and recovered complete semantics for input-dependent loops.The evaluation used academic and commercial VM obfuscators.Source: §9 · p. 13
Ethical Considerations
The authors discuss ethical considerations for software consumers, security practitioners, obfuscation-software developers, and copyright holders.
Ethical Considerations
- The research involved neither human participants nor personal data, and the authors state it posed no potential harm to individuals.Source: Ethical Considerations · p. 14
- The authors weigh benefits to consumers and security practitioners against potential harm to obfuscation companies and copyright infringement, concluding the benefits outweigh those harms.Source: Ethical Considerations · p. 14
- The research did not include or exclude people or groups based on personal or group attributes.Source: Ethical Considerations · p. 14
- The authors report that the relevant software licenses did not prohibit reverse engineering, and that they neither reverse-engineered protection software nor violated other licenses.Source: Ethical Considerations · p. 14
A Options for Tigress Evaluation
The appendix specifies Tigress configurations with and without super operators and gives compilation flags for Section 6.3.
A.1 Tigress Command Line
- Tigress samples used the Virtualize transform, if-nested dispatch, selectable operand modes, and seed 42.The listed commands also specify an x86_64 Linux GCC 4.6 environment and compile the output with DWARF-4 debug information.Source: §A.1 · p. 18
- The super-operator configuration set a maximum duplicate-operations count of 2, merge length of 10, and super-operator ratio of 2.0.Source: §A.1 · p. 18
A.2 Compilation options for Section 6.3
- Section 6.3 compilation used C11, disabled inlining and stack protection, enabled O2 optimization, and linked the specified libraries.The flags also disable strict aliasing and SSE and suppress int-conversion warnings.Source: §A.2 · p. 18
B.1 Compilation options for Code Virtualizer binaries
The appendix lists compilation flags for Code Virtualizer binaries and shows deobfuscation output for a recursive binary-search sample alongside Ghidra output.
B.1 Compilation options for Code Virtualizer binaries
- Code Virtualizer compilation used O2 optimization while disabling inlining, stack protection, and block reordering.The flags also disable tail merging and the red zone and include the Code Virtualizer headers and specified libraries.Source: §B.1 · p. 18
- The figure presents VMSpec output before and after recompilation, including VMSpec’s deobfuscated output and Ghidra’s decompilation of the deobfuscated, optimized binary.Source: Figure 9 · p. 18, §B.1 · p. 18
- The displayed decompilation contains a loop that calculates a midpoint, reads an array element, and adjusts the search bounds until a match or termination condition.Source: §B.1 · p. 18