Source-linked AI summary
When LLM Decompilers Recompile More and Preserve Less
Chang Liu, Edward Raff, Kristopher Micinski
TL;DR
Existing recompilability and shipped-test re-executability metrics can miss behavioral divergence and vulnerability loss in decompiled code. The paper proposes Decompile-Diverge, which compares reference and decompiled behavior on fuzz-generated inputs, and finds that build success can rise while behavioral agreement falls, with some shipped-test passers still diverging.
Problem
Existing benchmarks and shipped tests may not establish behavioral agreement on unseen legitimate inputs, allowing recompilable outputs to receive full credit despite changed behavior.
Method
Decompile-Diverge synthesizes per-function drivers, grows fuzzed inputs from the reference, and compares reference and decompiled behavior, including crashes, hangs, and bounded observable post-state differences.
Results
Across eight systems in nine configurations, 4.9% of candidates that passed every shipped test still diverged, reaching 13% for one system; across real and vulnerable code, recompilability and behavioral agreement also came apart.
Takeaways & Limitations
Behavioral comparison beyond fixed tests is needed to expose divergences that clean compilation and shipped-test success leave undetected, including shifts from visible unknowns to introduced code.
Takeaways & Limitations
The tuned-versus-zero-shot comparison conflates training distribution, scale, and objective, limiting the result to an observation about offdistribution robustness.
Abstract
from arXiv · showhide
Decompilation recovers high-level source from compiled machine code and serves as a foundation for security tasks such as vulnerability detection and malware analysis. Traditional decompilers like Ghidra and Hex-Rays expose whatever they cannot resolve as visible placeholders and often emit pseudocode that will not compile or execute; LLM-based decompilers produce clean, idiomatic C and are now judged almost entirely by recompilability and re-executability: whether the output builds and passes its shipped input/output tests. We show that these metrics can reward the wrong path: a function may recompile and pass every shipped test yet diverge on other legitimate inputs, and a disclosed vulnerability may disappear from the recompiled code with no visible trace of the crash. Neither failure is caught by existing suites. To address this gap, we propose Decompile-Diverge, a behavioral comparison oracle not relying on fixed or hand-crafted tests: for each function it synthesizes a driver, grows a fuzzing corpus from the reference, and reruns the decompiled code on the same inputs to detect changes in the function's behavior. Across eight systems in nine configurations on established LLM decompilation corpora, candidates that pass every shipped test still diverge from the original on our input corpus: 4.9% overall, and as many as 13% for a single system. On 300 real GitHub library functions and 287 CVE-grounded functions, recompilability and behavioral agreement can come apart: the strongest refinement LLM lifts Ghidra's build rate from 75% to 90%, while its Matched rate falls from 74% to 62%; on disclosed vulnerabilities, up to one tenth exhibit Crash Absence in its output. Source-level analysis traces this divergence to introduced fields, types, callees, and guards that replace the visible unknowns traditional tools leave behind.
1 Introduction
LLM-based decompilers improve recompilability and re-executability, but existing fixed tests can miss behavioral divergence. Decompile-Diverge addresses this gap with fuzzed reference inputs and behavioral comparison, revealing divergence and a shift from visible unknowns to invented code.
- Traditional decompilers often emit non-compilable pseudocode, while LLM-based systems increasingly improve recompilability and re-executability.
- A candidate can compile and pass shipped tests yet behave differently on other legitimate inputs.
- Decompile-Diverge synthesizes drivers, grows fuzzed inputs from the reference, and compares bounded observable post-states between original and decompiled code.
- The evaluation covers 300 real GitHub library functions and 287 disclosed vulnerable functions.
- Behavioral agreement can fall as Build rises, with divergence traced to introduced fields, types, callees, and constants.
2 Background
Decompilation reconstructs high-level code from binaries, while learning-based systems increasingly use end-to-end or refinement-based LLM approaches. Current benchmarks emphasize constrained datasets and fixed tests, limiting evidence for behavioral agreement.
- Conventional decompilers recover variables, types, and control flow before emitting C-like pseudocode.
- LLM decompilers primarily use end-to-end generation from assembly or refinement of traditional decompiler output.
- Learning-based evaluation relies on binary corpora, but commonly used benchmarks are highly constrained.
- Fixed tests and superficial similarity do not establish behavioral agreement, while many evaluations omit broader analytic techniques used with conventional decompilers.
3 Motivation
Existing training and evaluation objectives reward compilation and surface resemblance on known inputs rather than agreement on unseen behavior. Motivation examples show that shipped tests miss errors exposed by dynamically generated inputs.
- Next-token and structure objectives reward reference-string resemblance, compilation, and placeholder overlap without measuring unseen-input behavioral agreement.
- A rewrite can compile, appear clean, and pass every released input-output pair while changing function behavior.
- Recent runtime feedback and structural guidance leave the underlying surface proxy in place.
- On decode6, SK2Decompile matches 70/100 stress inputs after passing all ten shipped tests, while LLM4Decompile matches 66/100.
- The shipped suites fail to capture 3% to 45% of divergence in function behavior.
4 Benchmark Design
Decompile-Diverge builds a per-function behavioral oracle from the reference implementation, fuzzes inputs, and compares bounded post-states with recompiled candidates. The benchmark spans established corpora, real GitHub functions, and CVE-grounded vulnerable functions.
- The workflow synthesizes a driver, fuzzes the reference implementation with AFL++, and evaluates decompiler outputs on the resulting corpus.
- Figure 2 compares original and candidate binaries on shared inputs and labels behavior Matched, Diverged, or Crash Absence.
- The oracle compares return values, written bytes, writable globals, and selected data sections while masking relocation-sensitive pointers and filtering flaky results.
- The established-corpora setup replaces shipped tests for HumanEval-Decompile, ExeBench, AnghaBench, and MBPP C conversion.
- The GitHub track contains 300 C functions from 132 libraries in 106 repositories.
- The CVE track contains 287 vulnerable C functions from 94 open-source projects.
- The evaluation covers eight systems in nine configurations, including refinement and end-to-end systems.
5 Evaluation
Decompile-Diverge evaluates built decompilations by comparing their behavior with reference functions, revealing that recompilability and behavioral agreement can diverge across benchmark and real-code settings.
- 5.1 Experiment Setup and Judgment: Each successfully built candidate receives exactly one judgment: Matched, Divergence, or CVE-only Crash Absence, with Crash Absence taking precedence.Divergence includes changed observable state, crashes, behavioral changes, or non-termination.
- 5.1 Experiment Setup and Judgment: 3,089 functions are applicable to the tests, including four established corpora, 291 GitHub functions, and 287 CVE functions.Excluded cases include unsupported types and functions that could not be built with AddressSanitizer.
- 5.2 Divergence across Corpora: 13–21% of LLM4Decompile outputs diverge across established corpora, compared with Ghidra’s 1–4%, despite LLM4Decompile passing more shipped tests.The Div|P measure reaches 13% for Nova on HumanEval and 8% for LLM4Decompile on ExeBench.
- 5.2 Divergence across Corpora: 4.9% of 12,133 candidates passing shipped tests still diverge on generated inputs, while 77% of those divergences change outputs without crashes.Across 15,379 candidates scored by both oracles, 5.1% receive different judgments; 3.9% pass shipped tests yet diverge.
- 5.2 Divergence across Corpora: On GitHub functions, LLM4Decompile raises Ghidra’s Build rate from 75% to 90% while Matched falls from 74% to 62%.The 28-point gap comprises 23 points of changed bounded post-state and 5 points of introduced crashes; McNemar’s exact two-sided p = .002.
- 5.2 Divergence across Corpora: 8.7% of CVE-track functions lose the reference crash in LLM4Decompile outputs, while raw Ghidra has no Crash Absence cases.LLM4Decompile has 25 Crash Absence cases among 287 functions; its digest also detects non-crash divergences involving outputs or globals.
- 5.3 Source-Level Analysis of Divergence: Introduced fields, types, and callees correlate with higher divergence: 53% versus 20% on GitHub and 59% versus 31% on CVE functions.The association supports the proposed mechanism without establishing causality, and rates cover only candidates that compile.
- 5.3 Source-Level Analysis of Divergence: Rewriting shifts failures from visible front-end unknowns toward confident inventions, with introduced symbols appearing in 42–72% of failed splices for tuned refiners and end-to-end models.Such inventions can survive compilation and produce Divergence or Crash Absence through changed types, constants, guards, or callees.
6 Conclusion
Decompile-Diverge extends behavioral evaluation beyond fixed tests using fuzzed inputs derived from reference functions. The results show that recompilability and re-executability can miss behavioral divergence, which source analysis connects to replacing visible front-end unknowns with introduced tokens.
- Decompile-Diverge uses fuzzer-generated inputs and a precise splicing procedure to compare candidate behavior with reference behavior beyond released tests.The approach extends behavioral-agreement evaluation to functions without released tests.
- Recompilability and re-executability fail to capture some behavioral divergence across real and vulnerable code.
- Source analysis connects divergence to replacing visible front-end unknowns with introduced tokens.