PaperlayerView exact revision on arXiv

Source-linked AI summary

Behavior Specification-Guided Program Synthesis for Binary Deobfuscation

Kangchen Zhu, Shangwen Wang, Zhiliang Tian, Zhouyang Jia, Xiaoling Li, Jun Ma, Jie Yu, Xiaoguang Mao

arXiv:2608.20628v1cs.SE

TL;DR

Binary deobfuscation often relies on decompiled pseudocode that lacks high-level semantics, making program logic difficult to interpret. BinMirror constrains source-level synthesis with runtime traces and interaction snapshots; the paper reports improved robustness under heavy obfuscation and a 33.3% increase in malware-detector accuracy.

  • Problem

    Binary deobfuscation often relies on C-like pseudocode that lacks high-level semantics, making it difficult for source-level tools to interpret program logic accurately.

    Source: §3 · p. 3

  • Method

    BinMirror reframes binary deobfuscation as behavior-specification-guided synthesis, using dynamic execution traces and interaction snapshots to constrain source-level reconstruction.

    Source: §1 · p. 1, §8 · p. 13

  • Results

    BinMirror improved robustness under heavy obfuscation and produced more analyzable reconstructions; integration with a CFG-based Linux malware detector achieved higher accuracy (+33.3%).

    Source: §1 · p. 2

  • Takeaways & Limitations

    Behavior-constrained synthesis offers a practical direction for binary deobfuscation, especially when static structure and decompiler output are unreliable.

    Source: §8 · p. 13

    The current implementation is Linux-oriented, evaluated on x86, x64, ARM, and MIPS, and requires additional engineering for Windows, macOS, or bare-metal firmware.

    Source: §7.3.2 · p. 13

Abstract

from arXiv · show

Deobfuscation is critical to reverse engineering and security analysis because it restores the readability and analyzability of obfuscated code. However, existing research primarily focuses on source-code deobfuscation, while binary-level deobfuscation remains largely underexplored despite its practical importance when source code is unavailable. Existing binary deobfuscation methods typically decompile binaries into pseudocode and then apply structural transformations. However, because compilation discards high-level semantics such as precise type information and source-level structures, this decompilation-based paradigm often produces low-quality code and provides limited assurance that the recovered code preserves the runtime behavior of the original program. To address these limitations, we propose a paradigm shift from structural transformation to behavior-driven synthesis. Our core insight is that although obfuscation distorts a program's internal structure, semantics-preserving transformations must retain its observable execution behavior. Based on this insight, we introduce BinMirror, an approach that reformulates binary deobfuscation as a behavior-specification-guided program synthesis task. By treating dynamic execution traces and interaction snapshots as behavioral specifications, BinMirror synthesizes high-quality source code and validates it against runtime observations collected from heavily obfuscated binaries. Extensive evaluations on 1.5 million synthetically obfuscated binaries show that BinMirror significantly outperforms state-of-the-art baselines, achieving a unit-test Pass@1 of 74.5% under extreme obfuscation. These results demonstrate the practical utility of BinMirror in restoring semantic clarity for real-world security analysis.

1. Introduction

Obfuscation preserves functionality while obscuring structure, but can conceal malicious behavior, making deobfuscation important for reverse engineering and security analysis. BINMIRROR replaces reliance on decompiled structure with synthesis guided by observed runtime behavior, and reports test-based, readability, and malware-detection results.

1. Introduction

  • Existing deobfuscation methods largely target source code,,,,,, while decompilation-based binary workflows risk losing high-level semantics.Decompilation may also mis-recover types, source-level control structures, and contextual metadata, making behavior preservation difficult to check.Source: §1 · p. 1
  • Because semantics-preserving obfuscation must retain observable behavior on executed paths, BINMIRROR uses runtime interactions and side-effects as an empirical behavioral anchor.Side-effects include non-local state changes, memory updates through references, and I/O operations.Source: §1 · p. 1
  • BINMIRROR captures syscall-guided snapshots through fuzzing and instrumentation, slices behavior-relevant instructions, synthesizes with an LLM under state constraints, then differentially tests and iteratively refines candidates.The four stages are behavior specification capture, noise filtering, code synthesis, and test-based behavioral validation.Source: §1 · p. 2
  • 74.5% unit-test Pass@1 under extreme obfuscation, significantly outperforming state-of-the-art baselines.The benchmark measures whether compiled reconstructions pass available source-level tests.Source: §1 · p. 2
  • The synthetic benchmark covers 1.5 million binaries across four obfuscators, four architectures, and O0–O3; overlap analysis finds BINMIRROR complementary rather than uniformly dominant.The architectures are x86, x64, ARM, and MIPS.Source: §1 · p. 2
  • BINMIRROR averages 68.4% CodeBLEU and reduces cyclomatic complexity by 76.2% under extreme obfuscation; malware-detector integration improves accuracy by 33.3% and F1-scores by 37.1%.These validation results are bounded by available tests and observed executions; they do not prove semantic equivalence over all inputs.Source: §1 · p. 2

2. Background and Related Work

BinMirror reframes binary deobfuscation as behavior-specification-guided source synthesis for stripped binaries, grounding reconstruction in observed execution rather than relying solely on decompiled structure. Its validation is test-based and bounded by observed executions, not a proof of equivalence for all inputs.

Prior binary deobfuscation

  • Static methods simplify code through pattern matching, algebraic and data-flow analysis, or SMT reasoning, while dynamic methods observe concrete values, paths, and runtime effects,,,,,,,,,.D810 uses backward variable tracking and pattern-based simplification; QSynth and Syntia combine dynamic sampling, symbolic reasoning, or synthesis for expression simplification.Source: §2 · pp. 2–3

Scope and validation

  • The goal is readable source code consistent with observed runtime behavior and available tests, with assurance bounded by those observations rather than semantic equivalence over all possible inputs.Observed executions cannot specify behavior on paths they do not trigger.Source: §2 · pp. 2–3

Learning-based and LLM-based deobfuscation

  • Decompilation can lose or misrecover types and source structure, leaving source-oriented models to infer incorrect logic or remove security-relevant behavior,,,.CHISEL uses trace-informed compositional synthesis over decompiled pseudocode, while ChatDEOB applies fine-tuned models after preliminary decompilation.Source: §2 · p. 3, Figure 1 · p. 3

Execution-derived specifications and synthesis

  • Observed-execution results cannot capture behavior on paths no input triggers; prior synthesis recovers loops from hardware netlists or synthesizes structured programs from domain-specific intermediate representations,, while BINMIRROR targets heavily obfuscated stripped binaries.In this setting, source-level types, identifiers, and reliable structured pseudocode are often unavailable.Source: §2 · p. 3

Novelty and positioning

  • BinMirror turns runtime behavior into state-transfer constraints, extracts behavior-relevant binary slices, synthesizes compilable C, and validates it on held-out executions.The paper distinguishes this reconstruction pipeline from expression-level synthesis and decompiler-plus-LLM pipelines that mainly rely on recovered pseudocode.Source: §2 · p. 3

3. Motivation Example

The example shows how lost type semantics can cause decompilation-based tools to remove a real overflow check, motivating binary deobfuscation guided by observed runtime behavior.

3. Motivation Example

  • Decompilation-based pipelines lift binaries into C-like pseudocode for source-level transformations,,,, but lost or obscured semantics can make logic misleading and remove security-relevant behavior.Source: §3 · pp. 3–4
  • In the uint8_t overflow example, decompilation may promote the byte to signed int, making the wraparound check appear impossible and prompting simplifiers to delete the alert branch.Source: §3 · p. 4
  • BinMirror instead captures runtime states and branch outcomes on executed paths as behavioral specifications, then uses them to synthesize type-aware code and validate it against observations.Source: §3 · p. 4

4. Approach

BINMIRROR turns observed runtime behavior into source-code synthesis constraints, then tests and refines candidate code against the original binary’s captured interactions.

4.1. Overview

  • The four-stage pipeline transforms observed runtime behavior into behavior-constrained source-code synthesis and test-based validation.Source: §4.1 · p. 4

4.2. Behavior Specification Capture

  • BINMIRROR captures observable file and network interactions, prioritizes inputs producing novel syscall behavior, and records execution context and memory snapshots at target syscall boundaries.DBI traces pointers and payloads and records relevant registers and memory as behavioral constraints.Source: §4.2 · p. 5, §4.1 · p. 5

4.3. Specification-Driven Noise Filtering

  • Trace-enhanced backward slicing retains executed instructions that influence the captured state, while an orthogonality check removes dispatcher-only updates.The slice is based on observed-execution dependencies and does not claim all-path static completeness.Source: §4.3 · pp. 5–6

4.4. Specification-Guided Code Synthesis

  • The synthesis stage conditions readable C generation on the behavior-relevant instruction slice and captured snapshot, prompting the LLM to infer the observed state-transfer relation.The structured prompt guides abstraction of low-level instructions into high-level constructs and requires source code without explanatory output.Source: §4.4 · p. 6

4.5. Test-Based Behavioral Validation

  • A context-aware harness compares the candidate and original binary at the monitored interaction, feeding syscall, argument, register, or memory mismatches back for iterative refinement.Passing requires byte-level agreement on captured observations, not all-input semantic equivalence; refinement stops when discrepancies are eliminated or the iteration bound is reached.Source: §4.5 · pp. 6–7

5. Experimental Setup

Section 5 evaluates BINMIRROR across five research questions using a controlled synthetic benchmark and a real-world malware benchmark, with metrics, baselines, and implementation conditions designed to assess reconstruction and its limits.

5.1. Research Questions

  • The evaluation asks whether BINMIRROR improves on baselines, generalizes across architectures and compiler optimizations, benefits from its components, improves code quality and readability, and helps CFG-based Linux malware detection.Source: §5.1 · p. 7

5.2. Datasets

  • The evaluation uses 1,585,584 synthetic obfuscated binaries derived from 1,573 selected CodeNet programs and MalBench’s 500 active Linux obfuscated malware samples.The synthetic set applies six transformations in 63 configurations across four architectures and O0–O3; its source programs are filtered for complexity, length, and successful compilation, so it is controlled rather than representative of all CodeNet. MalBench uses observed syscall and I/O consistency as its oracle because source code and unit tests are unavailable; this is bounded by observed executions. Its downstream-detection data uses malware-family-aware partitions and a matched benign corpus; samples are not redistributed.Source: §5.2 · p. 7

5.3. Evaluation Metrics

  • Input-induced executions cover 95% of target-region basic blocks and 90% of branch edges on the Synthetic Benchmark, versus 88% and 70% on MalBench.Median unique paths per sample are 12 and 8, respectively; these figures define the observed behavioral scope, not complete path coverage or all-input semantic equivalence.Source: Table 1 · p. 7, §5.3 · p. 8
  • Evaluation reports compilation, unit-test and execution-success rates alongside complexity reduction, CodeBLEU, and Halstead-effort reduction.Dynamic specifications are split into refinement and held-out executions, with only refinement inputs available to methods; a reconstruction must compile and pass held-out tests or syscall/I/O checks. LLM-based methods and BINMIRROR run five seeded trials, reported as mean±SD; deterministic tools run once.Source: §5.3 · p. 7

5.4. Baselines

  • BINMIRROR is compared with static, symbolic/dynamic, hybrid-synthesis, and LLM-based baselines, including controlled LLM variants matched for runtime evidence and compile/test feedback.The controlled variants isolate the effects of BINMIRROR’s trace-driven slicing and behavior-specification-guided constraint construction.Source: §5.4 · p. 8

5.5. Implementation Details

  • Experiments use fixed hardware and LLM settings, with exact model, prompt, retry, truncation, and seed details recorded in artifact logs.Held-out inputs are excluded from prompts, feedback, repair rounds, and truncation decisions.Source: §5.5 · p. 8

6. Evaluation

Across synthetic and real-world obfuscation, BinMirror preserves behavior more effectively than evaluated baselines, remains robust across architectures and optimization levels, and produces readable reconstructions that improve downstream malware detection. Ablations and overlap analysis indicate that its gains depend on behavior-guided synthesis components and extend recovery to cases missed by other methods.

6.1. RQ1: Effectiveness

  • Baselines deteriorate under compositional obfuscation, and evidence-matched LLM variants still trail BINMIRROR, so runtime evidence alone does not explain its gains.GPT-4o falls from 66.5% Pass@1 at L0 to 5.5% at L6; the strongest controlled variant trails BINMIRROR by 22.2 percentage points in average Pass@1 and 35.4 points at L6.Source: §6.1 · pp. 8–9
  • On MalBench, BINMIRROR achieves 65.8% ESR, outperforming ChatDEOB by 29.3 percentage points on real-world obfuscated malware.The result measures behavioral consistency on the family-aware split, not recovery of complete malware source code.Source: Table 3 · p. 9, §6.1 · p. 9
  • At L6, BINMIRROR achieves 74.5% Pass@1, 54.3 percentage points above ChatDEOB, and the highest compilation rate at 81.4%.These results support reconstructing from behavior-relevant slices and runtime constraints rather than transforming obfuscated structure directly.Source: §6.1 · p. 9
  • Across 3,000 sampled cases, BINMIRROR succeeds alone on 1,331 cases (44.4%), while baseline-only cases account for 82 (2.7%).BINMIRROR-only cases often involve flattened control flow, opaque predicates, MBA transformations, or decompiler-induced information loss; both methods fail on 18.3%.Source: §6.1 · p. 9

6.2. RQ2: Robustness

  • At L6, BINMIRROR’s success rate is 78.57% on X64-O0 and 68.92% on ARM-O0, versus ChatDEOB’s 25.51% and 22.90%, respectively.Across optimization levels, BINMIRROR is less affected by compiler-induced structural changes, though performance still declines as optimization and obfuscation intensify.Source: §6.2 · pp. 9–10, Figure 3 · p. 10

6.3. RQ3: Ablation Study

  • At L6, Pass@1 drops from 74.5% to 28.5% without backward slicing, 48.4% without I/O constraints, and 60.5% without iterative refinement.Slicing filters dispatcher logic and MBA junk; I/O constraints ground generation, while differential testing and feedback improve behavioral consistency.Source: Table 5 · p. 10, §6.3 · p. 10

6.4. RQ4: Code Quality and Readability

  • At Level-6, BINMIRROR sustains 68.4% CodeBLEU, 76.2% CCR, and 72.8% HER, with more compact and readable reconstructions than competing methods.At Level-1 it achieves 88.5% CodeBLEU and reduces cyclomatic complexity by 93.5%; ChatDEOB falls to 28.4% CCR and 25.6% HER at Level-6.Source: Table 6 · p. 11, §6.4 · p. 11

6.5. RQ5: Downstream Utility

  • For downstream malware detection, BINMIRROR-enhanced artifacts achieve 91.5% accuracy and 91.2% F1-score, improving over direct binary analysis by 33.3 and 37.1 percentage points.The evaluation uses a fixed HawkEye-Linux detector on normalized executable artifacts and matched malware and benign corpora; behavior constraints reduce structural noise in reconstructed artifacts.Source: Table 7 · p. 11, §6.5 · pp. 11–12

7. Discussion

The discussion finds diminishing returns from additional refinement and lower synthesis success on long instruction slices. It also describes limits on validity from captured behavior, model variability, sparse observations, and the current Linux-oriented implementation and benchmark.

7.1. Sensitivity and Convergence of the Closed-Loop Iteration (Nmax)

  • Pass@1 rises from 60.5% at Nmax=1, increases by 7.3% at iteration two, and plateaus at 74.5% by Nmax=5, the default threshold.Average time overhead reaches 55.2 seconds at Nmax=10. Remaining failures can reflect incomplete behavior-relevant slices or context-window exceedance; the authors argue that improving dynamic taint tracking, pointer-alias modeling, and memory-constraint capture matters more than simply increasing the iteration limit.Source: §7.1 · p. 12

7.2. Impact of Instruction Slice Length and Context Boundaries

  • Pass@1 is 92.4% and 85.1% for slices of up to 100 instructions, but falls to 45.2% above 250 instructions and 21.6% above 500.Long, dense assembly sequences can cause omitted dependencies, truncated output, or inconsistent structures; hierarchical chunking is proposed for slices exceeding 500 instructions. The 150-LOC dataset threshold refers to original source programs, not extracted instruction slices.Source: §7.2 · p. 13

7.3.1. Internal Validity.

  • Validation remains bounded by captured executions and may miss behavior-relevant semantics or be affected by LLM non-determinism and sparse observable behavior.The authors mitigate instrumentation and slicing risks with syscall and memory-payload checks, conservative ABI-based memory boundaries, and dynamic address mapping; they also use fixed prompts, low-temperature decoding, bounded feedback, and five independent runs. Uncovered behavior remains outside the observation-bounded oracle.Source: §7.3.1 · p. 13

7.3.2. External Validity.

  • The implementation targets Linux binaries across x86, x64, ARM, and MIPS, while the synthetic CodeNet-derived benchmark is a controlled sample, not a representative sample of all CodeNet programs.Extending to Windows, macOS, or bare-metal firmware requires additional engineering; benchmark overlap with LLM pretraining cannot be ruled out. The authors mitigate contamination risks through deduplication, problem-level splits, withholding source and held-out tests, and compiled held-out execution evaluation.Source: §7.3.2 · p. 13

8. Conclusion

BINMIRROR uses dynamic execution traces and interaction snapshots to guide binary deobfuscation through source-level synthesis, then validates reconstructed code against observed runtime behavior. The evaluation reports improved robustness under heavy obfuscation and more analyzable, executable artifacts for downstream malware analysis.

8. Conclusion

  • BINMIRROR constrains source-level reconstruction with dynamic execution traces and interaction snapshots, helping mitigate semantic loss from compilation and decompilation.Recovered code is validated against observed runtime behavior.Source: §8 · p. 13
  • The evaluation finds that BINMIRROR improves robustness under heavy obfuscation and produces more analyzable reconstructions and useful executable artifacts for malware analysis.The conclusion presents behavior-constrained synthesis as a practical direction when static structure and decompiler output are unreliable.Source: §8 · p. 13

Appendix A. Synthetic Benchmark Audit · Appendix B. MalBench and Downstream Detection Details

The appendices audit the synthetic benchmark and document how MalBench and downstream detection data are selected, handled, and evaluated.

Appendix A. Synthetic Benchmark Audit

  • The 1,573 retained CodeNet programs are executable, testable, and obfuscatable, but are not intended to represent the full CodeNet distribution.Table 10 reports mutually exclusive exclusion reasons, assigning each filtered program the reason corresponding to its first failed filtering stage.Source: Appendix A · p. 16, Table 10 · p. 16
  • 1,585,584 binaries result from expanding 1,573 retained programs across the reported obfuscator, transformation, architecture, and optimization settings.The expansion is 1,573 × 63 × 4 × 4; retained-program metadata includes source hash, compiler settings, test metadata, random seed, and partition ID.Source: Appendix A · p. 16

Appendix B. MalBench and Downstream Detection Details

  • MalBench contains 500 Linux malware samples collected during 2023–2025, selected by executability, at least 10 VirusTotal flags, and unique SHA-256 hash.Hashes and audit metadata are released for reproducibility, but live malware binaries are not redistributed; corrupted, non-Linux, non-executable, and duplicate samples are removed.Source: Appendix B · p. 16
  • Downstream detection uses 500 architecture-, format-, and size-matched Ubuntu benign executables and a family-aware 70/10/20 split with seed 42.Held-out test samples are excluded from prompt construction, refinement, detector training, and threshold selection.Source: Appendix B · p. 16
  • Malware execution is isolated in Docker/QEMU with restricted network and filesystem policies and bounded by a 30-minute timeout.Outbound traffic is disabled or redirected to a sink; the root filesystem is read-only, writable state is confined to tmpfs, and syscall and I/O logging are enabled.Source: Appendix B · p. 16
  • RQ5 pipelines share the HawkEye-Linux detector configuration and evaluation protocol, with thresholds fixed using validation data before test evaluation.Reported 95% confidence intervals use 10,000 bootstrap resamples with seed 42.Source: Appendix B · p. 16

Appendix C. Baseline Information Budgets

The appendix defines explicit information budgets to separate baseline capabilities and control for evidence, prompting, repair, and feedback. It also specifies shared LLM settings and prevents held-out inputs from influencing refinement.

Appendix C. Baseline Information Budgets

  • Static tools receive only binaries or tool-specific IR; pseudocode variants use 16K/4K tokens and at most one repair, while dynamic and feedback variants receive progressively more evidence and refinement.LLM+Dyn receives syscall-trace summaries and monitored snapshots with a 32K/4K budget; LLM+Dyn+Fb adds up to five feedback rounds. BINMIRROR receives behavior-relevant slices, snapshots, behavior-guided constraints, and up to five refinement rounds.Source: Appendix C · p. 16
  • LLM variants share temperature 0.2 and top-p 1.0, and held-out inputs are excluded from prompt construction, repair, feedback, and refinement.Dynamic evidence uses the same serialized format across controlled variants and BINMIRROR; the comparisons test runtime evidence and feedback alone against trace-driven slicing and behavior-specification-guided constraints.Source: Appendix C · p. 16

Appendix D. Implementation Details · D.1. Execution Environment and Baseline Settings · D.2. LLM Protocol and Prompt Template

Appendix D documents the execution environment and baseline configurations, then specifies the controlled LLM prompting, inference, refinement, and evidence-handling protocol used in evaluation.

D.1. Execution Environment and Baseline Settings

  • Experiments ran on a dual-Xeon workstation with 512GB RAM and four RTX 4090 GPUs under Ubuntu 22.04, with malware isolated in Docker and QEMU.Table 12 is titled “Evaluation Scale and Resource Summary.”Source: §D.1 · p. 16, Table 12 · p. 17
  • Static deobfuscators were evaluated using exported pseudocode or recovered IR, while ANGR used DFS with a 30-minute timeout and Syntia used MCTS with 50,000 iterations and 100 samples.Source: §D.1 · p. 16
  • LLM baselines received IDA-generated pseudocode and fixed system prompts to standardize semantic lifting; Table 11 is titled “Prompt Template Used by BinMirror.”Source: §D.1 · p. 16, Table 11 · p. 17

D.2. LLM Protocol and Prompt Template

  • LLM baselines, controlled variants, and BinMirror shared fixed templates, temperature 0.2, top-p 1.0, and a 4K output cap; input budgets were 16K for pseudocode-only and 32K for evidence-augmented settings.Evaluated models were GPT-4o-2024-08-06, Claude-3.5-Sonnet-20241022, Llama-3.1-70B-Instruct, Qwen2.5-Coder-32B-Instruct, CodeLlama-34B-Instruct, and DeepSeek-Coder-33B-Instruct; closed models used official APIs and open models fixed local checkpoints.Source: §D.2 · p. 17
  • Decompiler+LLM and LLM+SamePrompt allowed one compile-error repair round, whereas LLM+Dyn+Fb and BinMirror allowed up to Nmax = 5 refinement rounds using refinement-split discrepancies.Source: §D.2 · p. 17
  • The artifact records exact model and provider or checkpoint information, seeds, prompt-template hashes, serialized evidence files, and request logs.Source: §D.2 · p. 17
  • Held-out inputs were excluded from prompts, repairs, feedback, truncation decisions, and refinement traces; over-budget inputs retained signatures, call sites, snapshot fields, and recent behavior-relevant traces first.Non-executed decompiler text was truncated first, and baselines and controlled variants shared BinMirror’s output format and decoding constraints.Source: §D.2 · p. 17

D.3. Evaluation Scale and Resource Usage

The evaluation spans a 1,585,584-binary synthetic benchmark expansion, but individual experiments use different subsets and denominators. The section also specifies how rates, runtime, LLM calls, CPU time, and token usage are counted.

D.3. Evaluation Scale and Resource Usage

  • The 1,585,584 synthetic binaries are the full benchmark expansion, while RQ1, RQ2, and MalBench evaluate distinct subsets or samples.RQ1 uses the x64-O0 subset including L0; RQ2 uses an architecture/optimization sweep; MalBench uses 500 malware samples under family-aware splits.Source: §D.3 · p. 17
  • Reported rates use each experiment’s evaluated denominator, counting completed non-timeout pass and fail outcomes while retaining timeout and unsupported runs in the accounting.Runtime is median [IQR] per evaluated sample, and LLM calls are counted over synthesis-invoking runs.Source: §D.3 · p. 17
  • CPU time includes dataset construction, binary and baseline execution, slicing, synthesis orchestration, and validation; LLM tokens include prompts and generations across LLM-based systems.Token accounting covers LLM-based baselines, controlled variants, and BINMIRROR.Source: §D.3 · p. 17

Related papers

This paper cites

Loading 2608.20628v1…