PaperlayerView exact revision on arXiv

Source-linked AI summary

Reveree: Diagnosing LLM Reverse-Engineering Agents

Hadjer Benkraouda, Hongyu Cai, Berkay Celik, Gang Wang

arXiv:2609.01185v1cs.CR

TL;DR

CTF reverse-engineering evaluations largely measure whether an agent captures the flag, leaving failure points and analysis versus recall unresolved. Reveree scores agent trajectories across solve rate, milestone progress, behavior, and memorization; its evaluation finds base-model performance dominates, while failures concentrate at comprehension stages.

  • Problem

    CTF reverse-engineering evaluations largely use solve rate alone, which cannot localize failures or distinguish binary analysis from recall of public solutions.

    Source: §1 · p. 1

  • Method

    Reveree scores agent trajectories by solve rate, progress through an eight-stage reverse-engineering schema, behavioral profile, and a memorization probe.

    Source: §1 · p. 2, §8 · p. 13

  • Results

    Across the evaluated challenges, base-model performance dominates, prompting effects are model-dependent, larger or newer models are not reliably stronger, and failures concentrate at comprehension stages.

    Source: §1 · p. 2, §8 · p. 13

  • Takeaways & Limitations

    The memorization results are benchmark-dependent, with models reproducing picoCTF flags from descriptions alone but showing minimal measurable recall on NYU-CTF.

    Source: §1 · p. 2

    Each model-strategy-challenge configuration has a single trajectory, so the evaluation does not estimate within-configuration variance through repeated sampling.

    Source: §7.3 · p. 13

Abstract

from arXiv · show

Reverse engineering (RE) is critical to security tasks such as malware analysis and vulnerability discovery, and large language model (LLM) agents are increasingly able to perform it autonomously. Capture-the-flag (CTF) RE challenges have become the standard proxy for measuring this capability, but evaluation rests on a single criterion: whether the agent captures the flag. This solve rate reveals neither where in the RE process an agent fails nor whether a success reflects analysis of the binary or recall of a public solution. In this paper, we propose Reveree, a diagnostic framework that scores an LLM RE agent's trajectory at three tiers: solve rate, milestone progress through an eight-stage RE schema, and a behavioral profile of its actions. Comprehension stages are scored by an outcome-blinded LLM judge validated against a human expert; all other stages are verified deterministically. Using Reveree, we evaluate nine frontier models and four prompting strategies on 88 picoCTF and NYU-CTF challenges. We find that the base model dominates performance, whereas prompting strategy is a secondary, model-dependent effect. Surprisingly, larger, newer, or costlier models are not reliably stronger. We also find that failures concentrate at the comprehension stages of the RE process, and that extra budget, persistence, or reasoning effort rescues few of them, pointing to a competence limit rather than a resource limit. Regarding memorization, while models reproduce picoCTF flags from challenge descriptions alone, NYU-CTF shows minimal measurable recall, and most solves survive surface perturbation, indicating that genuine analysis coexists with memorization. We release Reveree to the community.

1 Introduction

LLM agents are increasingly able to perform reverse engineering end to end, but solve rate alone cannot localize failures or distinguish analysis from recall; Reveree addresses these gaps by scoring agent trajectories.

1 Introduction

  • Reveree scores RE agents at three tiers—solve rate, progress through an eight-stage milestone schema, and behavioral profile—and adds a memorization probe.Mechanical stages are verified deterministically; comprehension stages are scored by an outcome-blinded LLM judge validated against a human expert, while the probe tests bare-model recall and reruns on renamed or paraphrased challenges.Source: §1 · pp. 1–2
  • Across 88 challenges, the base model dominated performance, prompting effects varied by model, larger or costlier models were not reliably stronger, and extra resources rarely rescued comprehension failures.On hard challenges, agents often recovered binary structure but stalled at comprehension stages, suggesting a competence rather than resource limit.Source: §1 · p. 2
  • Successful runs pivoted from analysis to building a solver rather than re-analyzing the binary, a behavioral signature reported across all nine models and prompting strategies.The controlled evaluation covered nine frontier models, four prompting strategies, and 88 picoCTF and NYU-CTF challenges.Source: §1 · p. 2
  • Memorization was benchmark-dependent: models reproduced picoCTF flags from challenge descriptions alone, NYU-CTF showed minimal measurable recall, and most solves survived surface perturbation.The perturbations renamed or paraphrased challenges while holding difficulty fixed.Source: §1 · p. 2

2 Background and Motivation

Reverse engineering is an iterative process of analyzing a binary, recovering its logic and constraints, and constructing a solution. Because CTF evaluations typically reduce agent performance to whether it captures a flag, the paper motivates an evaluation that reveals partial progress, failure causes, and possible solution recall.

Reverse engineering (RE)

  • RE proceeds iteratively from binary triage and surface mapping through control-flow and algorithm recovery, constraint extraction, and solver construction.This multi-step structure means agents can fail at different stages even when all runs receive the same final solve-rate score.Source: §2 · p. 2
  • RE is a stringent test for autonomous agents because it requires interpreting analysis-tool outputs and iteratively testing and revising hypotheses about code behavior.Classical binary-analysis methods and LLM work on decompilation and symbol recovery automate parts of the process, sharpening the question of genuine binary comprehension.Source: §2 · p. 2

CTFs as a proxy

  • CTF RE challenges offer machine-checkable flags and perturbable binaries while exercising core RE skills, but public write-ups can enable solution recall.They are a controlled proxy, though they omit real firmware scale and open-ended investigative goals.Source: §2 · p. 2

LLM Agents

  • LLM agents alternate reasoning with tool calls, while harnesses and prompting strategies shape their control loops; evaluating them therefore involves more than flag submission.The section describes single-agent and multi-agent harnesses and strategies such as ReAct and Reflexion.Source: §2 · pp. 2–3

Our Motivation

  • Solve rate alone can collapse distinctions on hard challenges, hide where an agent failed, and conflate binary analysis with recall of published solutions.The paper argues these limitations make failures less actionable and success less trustworthy as challenge write-ups enter training data.Source: §2 · p. 3
  • The proposed evaluation should measure partial RE progress, localize failures, and distinguish genuine analysis from solution recall or memorization.These three goals motivate REVEREE, which the paper develops and applies.Source: §2 · p. 3

3 Evaluation Pipeline Overview

REVEREE scores an agent’s tool-use trajectory from a configured reverse-engineering run, varying models, prompting strategies, challenge features, and settings while holding the execution environment and toolset fixed.

3 Evaluation Pipeline Overview

  • A challenge, model configuration, and prompting strategy define a sandboxed run whose tool-use trajectory is scored by REVEREE.The agent issues tool calls until it submits a flag or exhausts its budget; the scored trajectory is the pipeline’s sole output.Source: Figure 1 · p. 3, §3 · p. 3

3.1 Design Goals

  • The design controls other factors across runs, supports different harnesses through an adapter, and measures live tool use with human-like reverse-engineering tools.The method measures actual actions in a live environment rather than answers about a static binary dump.Source: §3.1 · p. 3

3.2 The Agentic Framework

  • D-CIPHER instantiates the method, while REVEREE consumes normalized trajectories and needs only an adapter to transfer across harnesses or agent scaffolds.The normalized representation consists of thought, action, and observation steps plus the run’s validated outcome.Source: §3.2 · p. 4

3.3 Factors of Variation

  • The study independently varies models, prompting strategies, selected challenge features, and model settings, while keeping the execution environment and reverse-engineering toolset identical.Model and prompting strategy are the primary axes; challenge features and settings are varied in targeted ablations.Source: §3.3 · p. 4

4 REVEREE: A Diagnostic Instrument for Reverse-Engineering Agents

REVEREE diagnoses reverse-engineering runs through solve rate, progress across an eight-stage schema, and behavioral profiles; it combines deterministic checks with outcome-blinded LLM judgments for comprehension.

4.1 Design Principles

  • REVEREE uses a fixed RE-specific schema, assigns deterministic checks to mechanical stages and an LLM judge to comprehension, and reports progress and behavior beyond solve rate.These principles distinguish its process-centered trajectory scoring from generic competency grading or hand-authored challenge-specific milestones.Source: §4.1 · p. 4, §4 · p. 4

4.2 Inputs

  • Each run combines agent outputs and tool interactions, a solution sketch that supplies stage context without the flag, and run metadata.The format supports runs from any CTF agent; for D-CIPHER, agent data includes planner messages and executor sessions, along with submitted_flag and success fields.Source: §4.2 · p. 4

4.3 The Eight-Stage RE Schema

  • The eight stages span triage through flag validation, distinguish observable ACTIONs from comprehension-based INSIGHTs, and split algorithm understanding into recognition and naming.Stage credit is assigned per stage rather than per challenge-specific milestone, with the solution sketch providing challenge context; the schema derives from observational studies of human RE analysts.Source: §4.3 · pp. 4–5, Table 1 · p. 5

4.4 Two-Channel Stage Detection

  • Channel A deterministically confirms or denies mechanical evidence, while Channel B judges only comprehension stages when deterministic checks abstain.The judge is stage-scoped, outcome-blinded, evidence-bound to cited trajectory steps, and grounded in executor tool calls and outputs; deterministic matching avoids false triggers from tool names in filenames or planner prose.Source: §4.4 · p. 5

4.5 Tier-2: Progress Metrics

  • Stage coverage counts every reached stage, whereas progress depth stops at the first missed stage; a marginal stall funnel shows stage reach among all, solved, and unsolved runs.For example, Run B reached three stages (3/8) but had depth 1/8 after skipping S2; the marginal view also exposes shortcut solves that reach the flag without intermediate comprehension.Source: §4.5 · pp. 5–6

4.6 Tier-3: Behavioral Profiling

  • Tier 3 profiles effort by tagging actions into six coarse behavioral states and 20 fine-grained codes, then summarizing each run with within-session Markov transitions and behavior features.A precedence-ordered tagger assigns each action one dominant intent; harness tools map directly to states, and unclassified residual commands account for under 1% of corpus actions.Source: §4.6 · p. 6, Table 2 · p. 6

4.7 Judge Validation

  • Human-expert agreement was 94.2% overall (113/120; 95% CI [88.4, 97.1]), supporting REVEREE’s reliable evaluations.Deterministic stages had no rule errors (60/60; pooled CI [94.0, 100]); judged comprehension stages had 88.3% agreement (53/60; CI [77.8, 94.2]). Six of seven disagreements were conservative judge misses, while one was a false positive.Source: §4.7 · pp. 6–7, Table 3 · p. 7

5 Evaluation

The evaluation compares nine frontier models and four prompting strategies on 88 RE challenges, measuring solve rate, progress through an eight-stage schema, and behavioral patterns, then testing memorization and surface-cue reliance.

5.2.1 Models

  • Model size and newer versions do not reliably improve RE performance: Claude peaks at medium size, while GPT-5.4 regresses from GPT-5.Sonnet scores 0.581 versus large Opus 4.5 at 0.518; GPT-5 nano scores 0.565 versus GPT-5 at 0.547. Opus 4.7 gains +0.106 over 4.5, but GPT-5.4 scores −0.058 versus GPT-5 overall. Neither higher cost nor longer run time signals stronger analysis.Source: Figure 3 · p. 8, §5.2.1 · p. 8, §5.2.4 · p. 10
  • Opus 4.7 has the highest solve rate (0.624), while Codex has the lowest (0.447) but the highest stage coverage (0.621).Codex costs $0.86 per solve, roughly 2.5× less than any other model; trajectory review suggests it handles mechanical RE steps but struggles to integrate results into a validated flag.Source: §5.2.1 · pp. 7–8, Table 5 · p. 8

5.2.2 Time Analysis

  • Across models, 70% of solves finish within 2 minutes and 85% within 5 minutes; failed runs last 6–8× longer than solves.On NYU-CTF, median solved and unsolved times are 1.9 and 12.2 minutes; on picoCTF, they are 0.6 and 4.8 minutes. Run speed reflects decisiveness rather than solve accuracy or model size.Source: §5.2.2 · pp. 8–9, Figure 4 · p. 9

5.2.4 Reasoning Effort

  • Reasoning effort has nonmonotonic, model-specific effects: Opus performs best at low effort, while Codex and GPT-5 improve only at their highest setting.Opus scores 0.517 at low effort, 0.483 at high, and 0.448 at extra-high; Codex improves from 0.241 to 0.345 and GPT-5 from 0.364 to 0.436 at the highest setting.Source: Figure 5 · p. 9, §5.2.4 · p. 10

5.2.3 Prompting Strategies

  • Prompting strategy has little effect on easy picoCTF challenges but helps on harder NYU-CTF challenges, with no strategy best for every model.On picoCTF, all four strategies solve 0.852–0.880; for Opus on NYU-CTF, ReAct reaches 0.5 and Reflexion 0.552. Of cases solved by any strategy, 72% are solved by all four, while ReAct contributes 10 unique solves.Source: Table 6 · p. 9, §5.2.3 · p. 9

5.3.1 Failure Modes

  • On NYU-CTF, solved and unsolved runs diverge at algorithm identification, indicating a comprehension bottleneck; targeted resource or persistence changes rescue only 10–15% of failures.Across 852 failed runs, 50% end by budget exhaustion, 25% by giving up, and 19% at the planner-round cap. On picoCTF, unsolved runs reach several analysis stages more often than solved runs, consistent with easier shortcut solves; expanded allowances often lead to another limit rather than a solve.Source: §5.3.1 · pp. 10–11, Table 7 · p. 10, Figure 6 · p. 10

5.3.2 Failure Signatures

  • Across models and strategies, successful runs pivot from analysis into scripting, whereas failures show command repetition, re-analysis, and getting stuck.Successful runs show productive-pivot Cliff’s δ = +0.22 to +0.38; failed runs show command repetition δ = −0.36 to −0.48 and repeated whole-program re-analysis δ = −0.21 to −0.26.Source: §5.3.2 · p. 11

5.4 RQ3: Reasoning vs. Memorization

  • Opus 4.7 reproduces 7.4% of picoCTF flags exactly from descriptions alone, while NYU-CTF has no exact recalls and at most 3.3% partial recall.Renaming or paraphrasing changes picoCTF solve rates by at most 5 points, while NYU-CTF drops 7.7–30.0 points; most solves nevertheless survive, indicating that genuine analysis coexists with memorization and surface-cue reliance.Source: §5.4.1 · p. 11, §5.4.2 · pp. 11–12, Table 9 · p. 12

6 Related Work

Prior work measures CTF agents by solve rate, targets individual reverse-engineering primitives, or evaluates trajectories with broader criteria; Reveree instead evaluates full RE runs with an RE-specific stage schema and behavioral profiles.

LLM agents for CTF and offensive security

  • CTF agents are judged mainly by solve rate [42] [68] [64] [1], whereas Reveree contributes an RE-specific end-to-end evaluation method rather than an agent.Prior systems add interactive tools (EnIGMA [1]), analyze long-run context loss (PentestGPT), or use a planner–executor harness (D-CIPHER); Reveree holds D-CIPHER fixed while varying the base model, prompting strategy, and configurations.Source: §6 · p. 12

Language models for reverse engineering

  • Prior RE language-model work targets individual primitives, whereas Reveree scores agents orchestrating a full toolchain across challenges, with incorporated tools treated as stages in the trajectory.The primitive-focused work spans decompilation, symbol and type recovery, disassembly, summarization, deobfuscation, and binary-understanding benchmarks.Source: §6 · p. 12

Evaluating agents beyond solve rate

  • Unlike CTFJudge’s generic whole-trajectory assessment [43], Reveree uses RE-specific stages, limits its judge to comprehension, requires cited evidence, blinds it to outcomes, and checks it against a human rater.Mechanical stages are evaluated deterministically, and Reveree also provides behavioral profiling.Source: §6 · p. 12

Failure analysis

  • Nishizaka et al. hand-label four weaknesses across 24 crackmes and three agents; Reveree adds eight-stage partial solves and behavioral profiles across 88 challenges, nine models, and four strategies.This scale supports an invariance claim that the three-agent study cannot make, while the behavioral tier reports run-level failure dynamics as well as cognitive causes.Source: §6 · p. 12

RE reasoning and memorization

  • Generic mutation methods [40] face fidelity–resistance tradeoffs [49], while EnIGMA [1] flags blatant recall or tests held-out challenges; Reveree instead pairs original and renamed or paraphrased binaries.The held-out comparison can confound contamination with difficulty; challenge-level perturbation produces a difficulty-matched original-versus-perturbed pair.Source: §6 · pp. 12–13

7 Discussion

REVEREE argues that durable insights concern where and how agents fail, while prompting strategies imported from other domains transfer weakly and the evaluation should broaden its sampling, harnesses, and challenge diversity.

7.1 What Generalizes Beyond These Models

  • Across all nine models, failures concentrate at comprehension, budget, persistence, and round-limit interventions recover only 10–15% of them, and productive versus failing trajectories validate-then-script versus re-read-and-loop.A single solve rate conflates stage progress with flag capture; the paper presents these patterns as task-level regularities that can be tested on future models.Source: §7.1 · p. 13
  • REVEREE is intended to remain useful across model generations, while leaked write-ups and easy benchmarks make held-out splits and surface mutation increasingly important.The paper says an easy benchmark can hide genuine prompting effects and proposes using future runs to track bottleneck shifts, trajectory signatures, and whether scaffolding pays off.Source: §7.1 · p. 13

7.2 Security-Specific Prompting Strategies

  • On RE, prompting shifts solve rate by under three points on easy picoCTF and seven to twelve on harder NYU-CTF; 72% of solved cases are solved by all four strategies.The authors argue that prompting tuned for other domains transfers poorly and identify a comprehension-focused RE reasoning schema as an unexplored direction.Source: §7.2 · p. 13

7.3 Limitations and Future Work

  • The evaluation uses one trajectory per configuration, one harness, and two challenge sources, motivating pass@k sampling, replication on other harnesses, and broader challenge sets.The authors also call for harder binaries and more difficulty-preserving perturbations to sharpen the contamination signal; they note that picoCTF and NYU-CTF behave differently.Source: §7.3 · p. 13

8 Conclusion

REVEREE evaluates LLM reverse-engineering agents through solve rate, milestone progress, action profiles, and a memorization probe. Across the evaluated models and challenges, it reveals model-dependent performance, comprehension-stage failures, and both recall and evidence of analysis.

8 Conclusion

  • Across nine frontier models and 88 picoCTF and NYU-CTF challenges, the base model dominated performance, while larger, newer, or costlier models were not reliably stronger.The evaluation also covered four prompting strategies.Source: §8 · p. 13
  • Failures concentrated at comprehension stages and reflected a competence limit rather than a resource limit; successful runs also shared a behavioral signature absent from failed runs.Source: §8 · p. 13
  • Models recalled picoCTF flags from challenge descriptions alone, whereas NYU-CTF showed near-zero recall; most solves survived surface perturbation, indicating analysis coexists with memorization.Source: §8 · p. 13
  • REVEREE scores runs by solve rate, milestone progress through an eight-stage RE schema, and behavioral profile, and pairs these tiers with a memorization probe.The authors release the framework for future evaluation, noting its diagnostic questions grow more relevant as contamination spreads.Source: §8 · p. 13

A Behavioral Profiles: Definitions and Additional Results

The appendix expands the Tier 3 behavioral profile and explains how model, challenge-set, and prompting-strategy differences shape agent behavior. Its results do not alter the RQ1 conclusions, and its prompting analysis supplies a mechanism behind the success-versus-failure signature reported in Section 5.3.2.

Appendix A.1

Appendix A.2

Appendix A.3

Appendix A.4

A.1 Metric Definitions

The behavioral profile measures unproductive repetition, transitions between analysis states, and how far a run progresses from static reading toward dynamic execution, scripting, and validation.

A.1 Metric Definitions

  • The metrics quantify unproductive repetition, state changes, and progression from static analysis toward dynamic execution, scripting, and validation.They include static self-transitions, re-analysis from scratch (also normalized by static actions), and repeated normalized command signatures. Pivot rate and productive pivots measure state changes, while the static-to-dynamic action ratio, first dynamic-action position, and binary scripting and validation indicators capture progression.Source: §A.1 · p. 18

A.2 Model Behavior

Models have distinct, challenge-stable behavioral profiles, differing in tool reliance and the coarse states they reach.

A.2 Model Behavior

  • Codex has the highest tool reliance (0.309), gpt5 is the most bash-centric (0.144), and Opus alone reaches VALIDATE frequently; these fingerprints are stable across challenges.Tool reliance is the fraction of actions issued through decompilation, disassembly, or scripting rather than raw shell.Source: §A.2 · p. 18

A.3 Dataset Behavior · A.4 Prompting Strategy Behavior

The sections show distinct behavioral profiles across datasets and models, while effective prompting strategies are associated with more validation and less redundant re-analysis.

A.3 Dataset Behavior

  • PicoCTF play is mostly static and one-shot, with a static-to-dynamic action ratio of about 29, 0.02 re-analysis passes per run, and a 0.14 validation rate.Source: §A.3 · p. 18
  • NYU-CTF is more iterative and dynamic, with a static-to-dynamic ratio of about 12, 4.42 re-analysis passes per run, and 0.90 scripting.Source: §A.3 · p. 18
  • On NYU-CTF, gpt5 re-reads artifacts 9.55 times per run, compared with opus 4.7 at 1.39, codex at 2.20, and gpt5.4 at 5.32.Source: §A.3 · p. 18
  • Codex reaches the deepest NYU-CTF progress depth (0.414) despite the lowest solve rate (0.276).Source: §A.3 · p. 18

A.4 Prompting Strategy Behavior

  • ReAct and Reflexion reach VALIDATE more often than vanilla or ledger, at 0.13 and 0.15 versus 0.03.Source: §A.4 · p. 18
  • ReAct and Reflexion re-analyze the binary 1.0 and 1.1 passes per run, versus 3.2 and 3.0 for vanilla and ledger, respectively.The passage attributes the strategies' gains to checking answers and avoiding redundant re-analysis.Source: §A.4 · p. 18

B Judge Validation Sample

The human-agreement validation sample comprised 15 randomly selected runs across 12 unique challenges, selected to cover both benchmarks and outcomes and the agent/model families.

B Judge Validation Sample

  • The validation sample included 15 runs across 12 unique challenges, randomly selected to cover both benchmarks and outcomes and the agent/model families.The prose says the sample covered all three model families, while the table caption says both agent families.Source: B Judge Validation Sample · p. 18, Table 10 · p. 18

Related papers

Loading 2609.01185v1…