Source-linked AI summary
Reveree: Diagnosing LLM Reverse-Engineering Agents
Hadjer Benkraouda, Hongyu Cai, Berkay Celik, Gang Wang
TL;DR
Existing CTF evaluations largely reduce autonomous RE to whether an agent captures the flag, leaving partial progress, failure location, and memorization unclear. Reveree scores trajectories across solve rate, eight-stage milestone progress, and behavior, then evaluates frontier models and prompting strategies to diagnose these gaps. Its results show model-dependent prompting effects, unreliable scaling with model size or cost, comprehension-stage bottlenecks, and benchmark-dependent memorization.
Problem
Solve rate alone cannot localize RE failures or distinguish binary analysis from recall of public solutions.
Method
Reveree scores agent trajectories at solve-rate, eight-stage milestone-progress, and behavioral tiers, using deterministic checks and an outcome-blinded judge for comprehension stages.
Results
The base model dominates performance; prompting is model-dependent, larger or costlier models are not reliably stronger, and failures concentrate at comprehension stages.
Takeaways & Limitations
Memorization is benchmark-dependent: picoCTF flags are recalled from descriptions alone, whereas NYU-CTF shows minimal measurable recall and most solves survive surface perturbation.
Takeaways & Limitations
The study uses one trajectory per model-strategy-challenge and instantiates Reveree on the D-CIPHER harness.
Abstract
from arXiv · showhide
Reverse engineering (RE) is critical to security tasks such as malware analysis and vulnerability discovery, and large language model (LLM) agents are increasingly able to perform it autonomously. Capture-the-flag (CTF) RE challenges have become the standard proxy for measuring this capability, but evaluation rests on a single criterion: whether the agent captures the flag. This solve rate reveals neither where in the RE process an agent fails nor whether a success reflects analysis of the binary or recall of a public solution. In this paper, we propose Reveree, a diagnostic framework that scores an LLM RE agent's trajectory at three tiers: solve rate, milestone progress through an eight-stage RE schema, and a behavioral profile of its actions. Comprehension stages are scored by an outcome-blinded LLM judge validated against a human expert; all other stages are verified deterministically. Using Reveree, we evaluate nine frontier models and four prompting strategies on 88 picoCTF and NYU-CTF challenges. We find that the base model dominates performance, whereas prompting strategy is a secondary, model-dependent effect. Surprisingly, larger, newer, or costlier models are not reliably stronger. We also find that failures concentrate at the comprehension stages of the RE process, and that extra budget, persistence, or reasoning effort rescues few of them, pointing to a competence limit rather than a resource limit. Regarding memorization, while models reproduce picoCTF flags from challenge descriptions alone, NYU-CTF shows minimal measurable recall, and most solves survive surface perturbation, indicating that genuine analysis coexists with memorization. We release Reveree to the community.
1 Introduction
REVEREE addresses the limits of solve rate by diagnosing partial RE progress, failure locations, behavioral patterns, and memorization across controlled evaluations.
- Motivation: CTF RE challenges provide machine-checkable tests of tool use and real reverse-engineering competencies.Agents receive binaries and must recover hidden flags, with challenge perturbations that preserve difficulty.
- Motivation: Solve rate cannot distinguish near-complete analysis from failure to parse a binary or separate reasoning from recall.Public challenge solutions may appear in pre-training, so captured flags can reflect memorization.
- Evaluation: Nine frontier models and four prompting strategies were evaluated on 88 picoCTF and NYU-CTF challenges.The study also examines memorization using difficulty-preserving perturbations.
- Findings: The base model dominates performance, while prompting effects depend on the model and larger, newer, or costlier models are not reliably stronger.The reported evaluation identifies model-dependent prompting effects rather than a uniformly superior strategy.
- Contribution: REVEREE scores trajectories through three tiers and adds a memorization probe beyond solve rate.The tiers measure solve rate, milestone progress through an eight-stage schema, and behavioral profiles.
2 Background and Motivation
RE is an iterative, tool-mediated, hypothesis-driven process whose evaluation should measure partial progress, localize failures, and distinguish analysis from memorization.
- Background: Reverse engineering recovers a compiled program’s structure and behavior without source code.It supports security tasks including vulnerability discovery, malware analysis, and firmware analysis.
- Background: A typical RE trajectory moves from binary triage and surface mapping through control-flow recovery, constraint extraction, and solver construction.The solver produces an input or flag satisfying the recovered constraints.
- Why RE matters: RE is a stringent autonomous-agent test because progress depends on interpreting tool outputs and revising hypotheses about code behavior.Relevant tools include disassemblers, decompilers, debuggers, scripts, and symbolic or dynamic analysis frameworks.
- Agentic evaluation: Agentic harnesses range from single-agent interfaces to multi-agent systems such as D-CIPHER, which separates planning from concrete tool calls.These harnesses provide infrastructure for broader security automation as well as CTF tasks.
- Evaluation gap: REVEREE is designed to measure partial progress, localize failures, and separate genuine analysis from solution recall.These goals make hard challenges diagnostically useful rather than merely unsolved.
3 Evaluation Pipeline Overview
The evaluation pipeline varies challenges, model configurations, and prompting strategies while holding the execution apparatus fixed, then scores the resulting agent trajectory with REVEREE.
- Pipeline inputs: Each run pairs a reverse-engineering challenge with model settings and a prompting strategy.Challenges contain names, descriptions, and binary files; prompting determines the agent’s reasoning loop.
- Controlled variation: The environment and toolset remain identical across runs so outcome differences can be attributed to varied factors rather than apparatus changes.The design varies model, prompting strategy, and other specified configuration factors independently.
- Execution: Agents operate in a sandbox with a standard reverse-engineering toolchain until submitting a flag or exhausting their budget.The scored trajectory is the pipeline’s only output and measurement subject.
- Framework scope: The framework is harness-agnostic: changing the harness, toolset, or agent scaffold changes only the adapter, not downstream measurement.D-CIPHER is used as an instantiation rather than a methodological dependency.
- Scoring representation: REVEREE consumes normalized thought-action-observation trajectories together with validated outcomes.Its schema, detection channels, progress metrics, and behavioral profiling operate downstream of a harness adapter.
4 REVEREE: A Diagnostic Instrument for Reverse-Engineering Agents
REVEREE evaluates reverse-engineering agents beyond flag capture by measuring staged progress and behavior, using deterministic checks for observable actions and an outcome-blinded judge for comprehension. Its process-level metrics expose gaps that solve rate alone can conceal.
- Framework overview: REVEREE retains solve rate while adding Tier 2 progress through eight canonical RE stages and Tier 3 behavioral profiling as a state machine.The three tiers move from whether the flag was retrieved, to how far the agent advanced, to patterns in its actions.
- Eight-stage RE schema: The eight-stage schema measures uniform RE milestones from triage to flag validation, splitting algorithm recognition from algorithm naming for finer-grained comprehension credit.Stages are tagged ACTION or INSIGHT, and recognition can receive credit even without attaching the textbook algorithm name.
- Two-channel detection: Deterministic rules verify stages whose accomplishment leaves mechanical evidence, while an outcome-blinded, evidence-bound LLM judge evaluates comprehension stages when mechanical evidence is insufficient.The judge receives stage-specific questions and must cite concrete trajectory spans; submitted and true flags are redacted.
- Progress metrics: Stage coverage counts every reached stage, whereas progress depth counts only the longest uninterrupted prefix, revealing later-stage jumps that cumulative progress can hide.In the worked example, Run B reaches 3/8 stages but has depth 1/8 because it skips S2; the marginal stall funnel compares stage reach within all, solved, and unsolved runs.
- Behavioral profiling: Tier 3 represents each run as a first-order Markov chain over six coarse interaction states and extracts interpretable behavioral features from within-session transitions.Actions are tagged at coarse and fine granularities, with harness tools mapped directly to analysis states to avoid misattribution.
- Judge validation: Validation compared REVEREE with an eight-year reverse-engineering expert across 15 runs and found seven disagreements, six from judge conservatism and one false positive.The authors conclude that the resulting evaluations are reliable, while deterministic rules were also audited for false positives and negatives.
5 Evaluation
Across nine models and four prompting strategies, Reveree reveals that model choice matters more than prompting, scale, version, or extra effort. Failures concentrate in comprehension and stalled analysis, while strategy benefits depend on dataset difficulty.
- Models: Codex has the lowest solve rate (0.447) but highest stage coverage (0.621), showing that milestone progress and successful flag conversion can diverge.Its trajectories use tools and scripts effectively but often stall before producing a flag.
- Models: Claude peaks at the medium tier: sonnet reaches 0.581, while large opus 4.5 drops to 0.518.Within-family size increases do not reliably improve reverse-engineering performance.
- Models: Opus improves from version 4.5 to 4.7 by +0.106 solve overall, whereas gpt5.4 declines from gpt5 by −0.058.The regression coincides with more budget-cap failures, fewer executor turns, and a shift toward giving up.
- Prompting and Effort: Increasing reasoning effort is nonmonotonic and model-specific, while prompting helps mainly on harder NYU-CTF challenges rather than saturated picoCTF.All strategies solve 0.852–0.880 on picoCTF, whereas on NYU-CTF ReAct and Reflexion improve over vanilla for opus.
- Failure Modes: Across failed runs, budget exhaustion accounts for 50%, giving up 25%, planner-round caps 19%, tool errors 4%, and timeouts 2%.Failure composition varies sharply by model: Codex gives up in 98% of failures, while Claude models are primarily budget-limited.
- Failure Modes: On NYU-CTF, solved runs reach algorithm identification at 0.76 versus 0.46 for unsolved runs, identifying comprehension as the main bottleneck.On picoCTF, unsolved runs often reach later stages more frequently, consistent with easier shortcut solves and low solved-run progress depth (0.16).
6 Related Work
Prior work evaluates autonomous RE agents mainly by whether they capture CTF flags, while Reveree broadens evaluation to partial progress and behavior across a larger controlled study.
- CTF agents are typically ranked by solve rate, including NYU-CTF, Cybench, picoCTF, EnIGMA, D-CIPHER, and plain-agent baselines.
- Reveree computes partial solves and behavioral profiles using an eight-stage RE schema across 88 challenges, nine models, and four prompting strategies.
- Because CTF write-ups may enter pre-training, solve rate can conflate reverse-engineering ability with recall.
- Existing contamination checks either detect suspicious one-step submissions or compare held-out post-cutoff challenges, but held-out comparisons can confound contamination with difficulty.
7 Discussion
The discussion argues that prompting gains transfer poorly from other domains, while Reveree’s main findings concern competence bottlenecks, trajectory structure, and evaluation scope.
- 7.1 What Generalizes Beyond These Models: Budget, persistence, and round-limit interventions recover only 10–15% of failures, while failures concentrate at comprehension stages across all nine models.
- 7.1 What Generalizes Beyond These Models: Productive and unproductive trajectories show characteristic validate-then-script versus re-read-and-loop patterns replicated across all nine models.
- 7.2 Security-Specific Prompting Strategies: Prompting strategies change solve rate by under three points on picoCTF and seven to twelve points on NYU-CTF, with 72% of solved cases solved by all four strategies.
- 7.2 Security-Specific Prompting Strategies: Domain-specific prompting improves malware and vulnerability tasks, but no reverse-engineering-specific prompting strategies have yet been developed.
- 7.3 Limitations and Future Work: The evaluation uses one trajectory per configuration, one D-CIPHER harness, and two challenge sources, motivating pass@k, cross-harness replication, and broader perturbation studies.
8 Conclusion
The conclusion presents Reveree as an evaluation framework that diagnoses LLM reverse-engineering agents beyond solve rate and summarizes its main empirical findings.
- Reveree scores solve rate, eight-stage milestone progress, behavioral profiles, and memorization across nine models, four prompting strategies, and 88 challenges.
A Behavioral Profiles: Definitions and Additional Results
The appendix defines and analyzes Tier 3 behavioral profiles, relating run-level dynamics to model, dataset, and prompting-strategy factors.
- Tier 3 behavioral profiles are derived from each run’s coarse-state transition matrix and analyzed by model, dataset, and prompting strategy.
A.1 Metric Definitions
The framework measures inefficient or repetitive behavioral patterns during reverse engineering, including static-analysis persistence, re-analysis, and command repetition.
- Static self-transition measures how often an agent remains in static analysis across consecutive steps, indicating unproductive “grinding.”
- Re-analysis count measures whole-program analysis reruns instead of reusing results from earlier agent invocations.
- Command repetition measures the fraction of shell commands that re-issue previously seen commands.
A.2 Model Behavior
Models exhibit distinct behavioral profiles beyond differences in solve frequency, especially in tool reliance and the coarse states they reach.
- Codex has the highest tool reliance at 0.309, while gpt5 is the most bash-centric at 0.144.
- Tool reliance measures the fraction of actions issued through decompilation, disassembly, or scripting rather than raw shell commands.
- Opus is the only model reported to reach VAL.
A.3 Dataset Behavior
Behavior differs substantially across datasets and prompting strategies: picoCTF behavior is mostly static and one-shot, whereas NYU-CTF behavior is more iterative and dynamic; ReAct and Reflexion reduce redundant analysis while reaching validation more often.
- Dataset Behavior: PicoCTF play has a static-to-dynamic action ratio of about 29, 0.02 re-analysis passes per run, and a 0.14 validation rate.
- Dataset Behavior: NYU-CTF has a static-to-dynamic ratio of about 12, 4.42 re-analysis passes per run, and heavy scripting at 0.90.
- Prompting Strategies: ReAct and Reflexion reach VALIDATE at 0.13 and 0.15 versus 0.03 for vanilla or ledger.
- Prompting Strategies: ReAct and Reflexion re-analyze binaries about three times less often, averaging 1.0 and 1.1 passes per run versus 3.2 and 3.0.
B Judge Validation Sample
The human agreement sample comprises randomly selected runs covering both benchmarks and outcomes, with coverage constraints described in the accompanying materials.
- Table 10 lists 15 randomly selected runs spanning 12 unique challenges, covering both benchmarks and both outcomes.
- The implementation passage says the sample covered all three model families, while the table caption says it covered both agent families.
- The supplied figure captions describe Tier-3 behavioral profiles by model, prompting strategy, and dataset.