PaperlayerView exact revision on arXiv

Source-linked AI summary

SoK: Are LLMs Reliable at Source Code Recovery? A Taxonomy and Empirical Evaluation

Varun Kohli, Lee Bing Cheng, Nur Hazim Ghazali, Gao Yuze, Daryl Poon, Dinil Mon Divakaran

arXiv:2610.11556v1cs.CRcs.SE

TL;DR

LLM-assisted source recovery has fragmented methods and metrics, and prior evaluations give limited coverage to embedded architectures and languages beyond C/C++. This SoK systematizes methods and evaluation, finding that recovery performance varies across systems and settings while identifying continuing generalization gaps.

  • Problem

    Prior evaluations are fragmented and focus largely on x86-64 and C/C++, with no LLM-assisted evaluation covering embedded architectures.

    Source: A. Benchmarks and Empirical Evaluations · p. 4

  • Method

    The paper builds a design-centric taxonomy and evaluates SOTA methods and controlled design ablations across architectures, optimization settings, symbol variants, and languages.

    Source: §I · p. 2, §IX · p. 12

  • Results

    Agent4Decompile [77] leads for compilable code with byte-level semantics and functional similarity, while SOTA methods perform worse on Thumb2 and aggressive optimization.

    Source: §I · p. 2

  • Takeaways & Limitations

    Signature failures, non-C recovery, and Thumb2-ISA coverage remain open problems for future work.

    Source: §IX · p. 13

    The evaluation assumes that function call interfaces are available or derivable from repository-level context.

    Source: §VIII · p. 12

Abstract

from arXiv · show

Effective source recovery is critical to security applications such as malware analysis, vulnerability assessment, and legacy maintenance. Large Language Models (LLMs) are reshaping this field, shifting the paradigm away from rule-based heuristics to probabilistic and high fidelity semantic recovery of source code from assembly or classical decompiler-derived pseudo-C. However, despite rapid progress, the field suffers from fragmentation across numerous approaches as well as their non-unified evaluations, limiting objective comparisons. Further, existing works have limited coverage of embedded, IoT architectures and source languages beyond C/C++. In this work, we present the first Systematization of Knowledge (SoK) focused specifically on LLM-assisted binary-to-source recovery. We provide a granular design-centric taxonomy of LLM-assisted source recovery methods and systematic evaluations using seven key metrics along six evaluation dimensions. We evaluate state-of-the-art methods on 45,000 test samples derived from four standard and five embedded architectures, five optimization levels, and symbol stripping. We ablate the impact of design choices on recovery performance, including input representation, contextual enrichment, model scale, iteration and review roles using controlled in-house recovery pipelines and three off-the-shelf models. Finally, we test same-language and cross-language recovery capability covering four mature and two legacy languages. Our systematization and comprehensive evaluations provide key insights that guide future directions in this field.

I. INTRODUCTION

LLM-assisted source recovery has proliferated, but methods and evaluation metrics remain fragmented, with limited coverage beyond x86-64, C/C++, and standard architectures. This SoK systematizes the field and evaluates methods, finding differences in recovery quality across systems, architectures, optimization settings, and design choices.

Key Findings of Our SoK

  • Agent4Decompile [77] is the best method for compilable code with byte-level semantics and functional similarity, while SK2Decompile [63] and FidelityGPT [78] mainly lead in readability and lexical similarity.The comparison is reported as of this study.Source: §I · p. 2
  • SOTA methods perform worse on Thumb2 than standard ISAs, aggressive optimization lowers performance, and symbol stripping mainly affects readability.Function-level recompilation can miss signature recovery problems, making recompilability with call interfaces a better reporting metric.Source: §I · p. 2
  • Pseudo-C provides a portability layer for cross-architecture recovery, compiler iterations and larger models improve compilation and fidelity, and LLM roles improve readability.Source: §I · p. 2
  • Current LLM and classical decompiler capabilities are C-centric and heavily underperform in non-C recovery; the study evaluates same-language and cross-language recovery across mature and legacy languages.Source: §I · p. 2

Contributions

  • The paper’s five-axis taxonomy supports systematic evaluation and targeted ablations, while its metric review identifies 53 prior metrics and selects seven across six evaluation dimensions.Source: §I · p. 2
  • The SOTA evaluation covers 45,000 samples from four standard and five embedded architectures, five optimization levels, and stripped or unstripped symbol variants.Source: §I · p. 2
  • Controlled ablations test input representations, contextual enrichment, model scale, compiler iterations, and review roles using three off-the-shelf models and in-house pipelines.Source: §I · p. 2

II. PRELIMINARIES

Compilation irreversibly loses source-level information, so decompilation cannot deterministically reconstruct the original. Classical decompilers use staged analyses and heuristics to produce pseudo-C, but can produce output that is difficult to understand.

A. The Lossy Nature of Compilation

  • Compilation removes semantic abstractions and may refactor program logic, making original-source recovery a many-to-one inverse problem.The practical target is functionally equivalent, human-readable code rather than deterministic recovery of the original source.Source: A. The Lossy Nature of Compilation · p. 2

B. Classical Decompilation Mechanics

  • Classical decompilers disassemble machine code, lift instructions into an IR, analyze data and control flow, then apply structural heuristics and type inference to emit pseudo-C.These tools preserve low-level execution semantics, but can struggle with heavily optimized code, nonstandard calling conventions, or obfuscated binaries, producing cluttered pseudo-C.Source: B. Classical Decompilation Mechanics · p. 3

C. The Neural Decompiler Paradigm

LLM-assisted recovery treats decompilation as neural translation, and this systematization organizes the field’s methods and evaluations. The surveyed literature has narrow architecture and language coverage, while recent methods increasingly use iterative feedback.

III. SYSTEMATIZATION

  • The survey covers 18 in-scope LLM-assisted source-recovery methods, selected from a corpus gathered through venue- and keyword-based searches.The scope excludes VM bytecode recovery and includes 13 peer-reviewed methods plus five 2026 arXiv preprints.Source: §III · p. 3

A. Benchmarks and Empirical Evaluations

  • Existing evaluations focus on x86-64, generally fix one input representation, and cover almost entirely C/C++, with limited cross-language study.Prior work examined Dart–Swift transfer, but not other mature languages such as Go, Rust, Java, and Python or legacy languages such as Fortran.Source: A. Benchmarks and Empirical Evaluations · p. 4

B. Systematization of LLM-assisted Source Recovery Approaches in the Literature

  • The taxonomy classifies methods by model setup, control flow, roles per turn, input representation, and augmentation.Its input categories include assembly or low-level IR, pseudo-C, and structural enrichment; augmentation includes retrieval, self-exemplars, and verifier feedback.Source: B. Systematization of LLM-assisted… · pp. 4–5
  • Early methods were mostly single-pass, while 2026 brought richer inputs and roles alongside a shift toward iterative, compile- and execution-guided refinement.The survey describes four of five iterative methods from 2026 as arXiv preprints; Agent4Decompile is iterative and multi-role, not tool-calling agentic.Source: B. Systematization of LLM-assisted… · p. 5

IV. EVALUATION METRICS USED IN SOURCE RECOVERY

The paper organizes source-recovery evaluation around two objectives and six dimensions, distinguishing generated-code quality from similarity to the original. Its survey finds fragmented metric use across prior work.

IV. EVALUATION METRICS USED IN SOURCE RECOVERY

  • Source-recovery evaluation has two objectives—quality and similarity—divided across six dimensions.Compilation and readability measure quality; lexical, syntactic, semantic, and functional similarity compare recovered code with the original.Source: §IV · pp. 5–6
  • Compilation checks parsing, compiling, and linking, whereas functional similarity measures whether recompiled code matches runtime behavior.Readability concerns ease of reading; lexical, syntactic, and semantic dimensions compare text, AST structure, and program meaning, respectively.Source: §IV · p. 6
  • The surveyed 22 papers use 53 unique evaluation metrics, including 11 composite metrics that combine subdimensions.Composites are counted as one metric and listed on every dimension they measure.Source: §IV · p. 6, Table II · p. 6

1) Code Quality Metrics:

Prior work uses varied metrics for code quality and similarity, with no surveyed suite spanning all six evaluation dimensions. The paper selects a seven-metric suite and distinguishes standalone compilation from compilation with the caller interface.

1) Code Quality Metrics:

  • Prior work uses five compilation metrics, most commonly recompilation rate, while readability is measured with 12 metrics across nine papers.Readability measures range from complexity and naming metrics to LLM judges and user studies with differing participants, questions, and scales.Source: 1) Code Quality Metrics · p. 6

2) Code Similarity Metrics:

  • Similarity measures range from lexical edit comparisons and token-level identifier scoring to AST, data-flow, binary, and runtime comparisons.Variable-name accuracy and meaningful-variable ratio are string comparisons and are not invariant to renaming; functional evaluations often execute recompiled code.Source: 2) Code Similarity Metrics · pp. 6–7

A. Evaluation Metrics Suite RQ1

  • None of the surveyed evaluation and method papers uses a suite covering all six dimensions; this paper adopts seven metrics, including two for semantic similarity.The selected metrics comprise one principled metric per dimension, with two semantic measures for source-derived and binary-derived semantics.Source: A. Evaluation Metrics Suite RQ1 · p. 7
  • Re-compilation Rate with caller Interface (RCI) captures signature recovery that standalone RC can miss, and the metrics satisfy RE ≤ RCI ≤ RC.The authors report that LLMs are better at generating compilable code than recovering the correct signature; they select RCI and RE for compilation quality and functional similarity, respectively.Source: A. Evaluation Metrics Suite RQ1 · p. 7
  • ∆R2I compares generated-code readability with the decompiler pseudo-C baseline; a positive value means the generated code is more readable.The standardized pseudo-C baseline avoids comparison against noisy original source code.Source: A. Evaluation Metrics Suite RQ1 · p. 7
  • The suite uses Edit Similarity for lexical similarity, AST-match for syntax, DF-match for source-derived semantics, and byte-wise match for binary-level semantics.Byte-wise match recompiles with the original compiler and flags and compares machine-code bytes; it is strict enough to fail behaviorally equivalent code with different instruction selection.Source: A. Evaluation Metrics Suite RQ1 · p. 7

B. Experiments and Models

The study evaluates state-of-the-art recovery methods, controlled pipeline ablations, and same- and cross-language recovery. Its dataset spans nine architectures, five optimization levels, and stripped and unstripped binaries.

B. Experiments and Models

  • The SOTA evaluation tests seven methods across four standard and five embedded architectures, five optimization levels, and stripped and unstripped variants.It also examines design choices and metric correlations; methods using off-the-shelf models are compared using gemma4-31b-it when code is available.Source: B. Experiments and Models · pp. 7–8
  • Pipeline ablations test input representation, contextual enrichment, model scale, compiler-feedback iteration, and a static-analysis review role.The pipelines use three off-the-shelf models and are evaluated on 50 x86-64 C functions at O0 and O2.Source: B. Experiments and Models · p. 8
  • The language study compares same-language recovery for C, C++, Go, Rust, Fortran, and Ada with cross-language recovery into C using SP and CIT pipelines.Its 300 functions comprise 50 per language; non-C functions are translations checked against the C functions’ I/O tests.Source: B. Experiments and Models · p. 8, C. Dataset · p. 8

C. Dataset

  • The dataset starts from ExeBench’s test_real split, filtered to 1,324 functions that compile and pass reference I/O tests under the study settings.The authors select 500 functions by rank-spacing over Halstead difficulty values.Source: C. Dataset · p. 8
  • The RQ2 corpus contains 45,000 samples from 500 functions compiled across nine architectures, five optimization levels, and two symbol variants.For RQ3 and RQ4, the study uses a 50-function subset compiled with x86-64 at O0 and O2, with stripped symbols.Source: C. Dataset · p. 8

VI. EVALUATIONS AND DISCUSSION

Across the evaluated methods, Agent4Decompile leads in RCI, binary-level semantics, and functional recovery, while pseudo-C input methods outperform assembly-based methods across metrics. Performance varies by architecture and optimization, and symbol stripping has limited impact on top methods.

1) Impact of SOTA Design Choices:

  • Pseudo-C methods outperform assembly-based methods across all seven metrics, architectures, optimizations, and symbol variants.Averaged across their tiers, pseudo-C methods reach RCI 0.47, ∆R2I 0.24, ES 0.36, AST 0.40, DF 0.53, BW 0.17, and RE 0.36.Source: 1) Impact of SOTA Design Choices · p. 9
  • Agent4Decompile has the highest RCI, BW, and RE among control-flow approaches, while non-iterative multi-role methods lead in ∆R2I and ES.Agent4Decompile’s ∆R2I is 0.05 and ES is 0.25; FidelityGPT records 0.42 and 0.39, respectively.Source: 1) Impact of SOTA Design Choices · p. 9

2) Generalization Across Architectures, Optimizations, and Symbol Variants:

  • RE drops monotonically from O0 to O3, is similar at O2 and Os, and is lower on Thumb2 microcontrollers than on standard and higher-end embedded architectures.Symbol stripping causes only a minimal performance drop for top methods, while ∆R2I is most affected for other methods.Source: 2) Generalization Across… · p. 9, Figure 6 · p. 9, Figure 7 · p. 9
  • Agent4Decompile is best in RCI, BW, and RE across architectures and optimizations, with 41% RE on Thumb2, 43% above SK2Decompile.Its AST and DF are comparable to FidelityGPT and SK2Decompile, but readability and lexical similarity are weaker.Source: Figure 5 · p. 9, Figure 6 · p. 9, 2) Generalization Across… · p. 10

3) Metric Correlation:

The paper finds that its seven recovery metrics provide largely non-overlapping information. In controlled ablations, pseudo-C is the strongest input representation, while the tested enrichment channels offer inconclusive gains.

3) Metric Correlation:

  • Across 45,000 generations, the seven metrics carry largely non-overlapping information; RE and RCI have the highest correlation at 0.78.Their correlation reflects nesting because RE is computed only when a function compiles, though the metrics capture different information by definition.Source: 3) Metric Correlation · p. 10

1) Input Representation:

  • Pseudo-C achieves the best relative performance across all metrics in the input-representation ablation, so the authors select it for further analysis.The comparison uses SP-31b on 100 samples from 50 x86 functions compiled at O0 and O2 with stripped symbols.Source: 1) Input Representation · p. 10, Figure 9 · p. 10
  • Enrichment results are inconclusive overall because gains are minimal or channels are available for few samples.Frame ABI and CFG show limited or negative improvement in RCI and RE, respectively; callees and .rodata show over 20% improvement but occur in 12% and 20% of samples.Source: 1) Input Representation · p. 10

2) Model Scale, Iterations, and Review Roles:

Larger models and compiler-feedback iteration improve several recovery metrics, while reviewer roles trade improvements in functional recovery and semantics for declines elsewhere. The language study finds substantial gaps between compilability and functional recovery, with outcomes varying by source language and task.

2) Model Scale, Iterations, and Review Roles:

C. Source Language Study RQ4

  • In same-language recovery, non-C languages generally recompile comparably to or better than C, but their RE is 1–13% versus 47% for C.C++ and Rust cases often compile but fail the harness because of incorrect signatures; Fortran runs but diverges, and Ada does not benefit from compilation repair.Source: C. Source Language Study RQ4 · pp. 11–12, Figure 11 · p. 11, Figure 12 · p. 11
  • Cross-language recovery into C beats same-language recovery for four of five non-C languages; iteration raises RE for Fortran from 17% to 29%, C++ from 14% to 27%, and Ada from 8% to 13%.Go and Rust perform poorly in cross-language recovery, and all cases remain below the C-to-C ceiling of 59% RE with iteration.Source: Figure 11 · p. 11, Figure 12 · p. 11, C. Source Language Study RQ4 · p. 12

VII. KEY FINDINGS AND OPEN PROBLEMS

The findings show that recovery quality depends on which fidelity dimension is measured, while signature recovery, embedded architectures, and non-C languages remain challenges. The paper also reports evaluation boundaries and future directions.

VII. KEY FINDINGS AND OPEN PROBLEMS

  • Agent4Decompile leads in recompilability, functional similarity, and byte-level semantics, while FidelityGPT and SK2Decompile lead in readability and lexical similarity.The metric-specific differences motivate a multidimensional evaluation.Source: §VII · p. 12
  • Methods often generate compilable code but fail to recover correct signatures; harness-aware compiler feedback iterations can mitigate this failure.Source: §VII · p. 12
  • Pseudo-C methods show higher fidelity than assembly-input methods across architectures, while Thumb2 performance is worse and RE declines monotonically from O0 to O3.RE remains comparable for Os and O2, which share core functionality apart from code alignment and size-padding optimization.Source: §VII · p. 12
  • Larger models achieve higher overall fidelity; iteration improves RCI, BW, and RE but sacrifices readability, while multiple roles improve readability and lexical similarity.General-purpose off-the-shelf LLMs match pseudo-C-fine-tuned models; role differences in compilability, semantics, and functional similarity are marginal.Source: §VII · p. 12
  • Recovery fidelity is worse on non-C-derived binaries, while cross-recovery to C outperforms recovery to the original source language.The authors identify cross-language decompilers, training, and augmentation as open directions.Source: §VII · p. 12

VIII. THREATS TO VALIDITY

  • Absolute scores may shift with prompt wording, input formatting, or decoding settings, and ExeBench may have appeared in models’ pretraining data.The authors compare methods using specified prompts and publish prompts and source code; compiler feedback is expected to improve RCI less without repository-level context.Source: §VIII · p. 12

IX. CONCLUSION

  • The study evaluates representative methods on 45,000 samples across nine architectures, five optimization levels, and two symbol variants, and identifies non-C and Thumb2 coverage as open problems.It also presents a design taxonomy, six evaluation dimensions, and same- and cross-language recovery measurements.Source: §IX · pp. 12–13

ETHICAL CONSIDERATIONS

  • The evaluation uses permissively licensed C from ExeBench and publicly sourced implementations; the authors caution that the harness could be misused for IP theft or license circumvention.Source: ETHICAL CONSIDERATIONS · p. 13

OPEN SCIENCE

The paper releases its harness, dataset, configurations, experimental records, and scripts for reproducing the study’s results.

OPEN SCIENCE

  • The complete artifact includes the evaluation harness, study configurations, dataset, LLM generations, scored per-cell records, and code to generate the paper’s tables and figures.Source: OPEN SCIENCE · p. 13

1) Code:

  • The harness supports cross-compilation across nine architectures and five optimization levels, disassembly, Ghidra lifting, eight recovery methods, and seven evaluation metrics.GPU resources are required to run the study scripts.Source: 1) Code · p. 13

2) Dataset:

  • The RQ1/RQ2 dataset uses 500 ExeBench test-real functions to create 45,000 binary samples; RQ3/RQ4 use 50 shared functions and 300 translations into five languages.The translated languages are C++, Go, Rust, Fortran, and Ada.Source: 2) Dataset · p. 13

3) Configs:

  • Repository configurations specify the settings for RQ1/RQ2, input-representation and language studies, iteration and review arms, and shared RQ3/RQ4 functions.Source: 3) Configs · p. 13

4) Results:

  • Stored experiment CSVs can regenerate tables and figures, while a deterministic script rebuilds 16 figures and six tables.The run-RQ scripts are provided to regenerate results from the beginning.Source: 4) Results · p. 13, 5) Figures and tables · p. 13

APPENDIX

The appendix maps supplementary figures and tables to dataset difficulty, method performance, input representations, cross-language recovery, and detailed benchmark results.

APPENDIX

Loading 2610.11556v1…