Source-linked AI summary
How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation
Kimberly Milner, Minghao Shao, Nanda Rani, Haoran Xi, Venkata Sai Charan Putrevu, Meet Udeshi, Sandeep K. Shukla, Prashanth Krishnamurthy, Farshad Khorrami, Muhammad Shafique, Ramesh Karri
TL;DR
CTF benchmarks often equate correct flag recovery with successful exploitation, despite flags also arising through shortcuts and unsupported claims. CTF-ABACUS audits execution traces as evidence-grounded solve profiles and finds that many recovered flags lack execution-backed support, motivating provenance-aware evaluation and benchmark design.
Problem
CTF evaluations overlook how agents obtain flags, conflating intended exploitation with exposure, recall, lookup, guessing, and unsupported claims.
Method
CTF-ABACUS reconstructs traces into solve profiles by labeling phases and techniques, assigning flag provenance, auditing evidence, and aggregating challenge signatures.
Results
72.9% of recovered flags were attributed to executed attacks, 10.9% to human-verified derivations, and 16.2% to unsupported pathways.
Takeaways & Limitations
Flag-based scores can reflect substantially different behaviors, so benchmark validity depends on evidential support and the interaction between challenge and solver.
Takeaways & Limitations
On trivial or environmentally suspect challenges, exploit-versus-shortcut judgments can remain ambiguous even for human reviewers.
Abstract
from arXiv · showhide
Capture-the-Flag (CTF) benchmarks are widely used to assess the offensive security capabilities of autonomous language-model agents. Evaluations rely on shallow binary judgments or aggregate scores, overlooking the agent's trajectory to the flag. Consequently actual exploitation is conflated with direct flag exposure, memorized recall, external lookup, guessing, and unsupported claims, potentially overstating the agent's cybersecurity capability. We introduce CTF-ABACUS, a trace-based agent auditing framework that reconstructs each run as an evidence-grounded solve profile. By decomposing agent actions into penetration-testing phases and categorical techniques, it identifies where exploitation occurs, where the flag first appears, and whether the recovered flag is supported by demonstrated behavior. Aggregating solve profiles across agents yields challenge signatures that reveal whether success was achieved via the intended exploit or via shortcut pathways. We apply CTF-ABACUS to 1,435 CTF attempts by six frontier and open-source models on 240 challenges, yielding 2,870 solve profiles under two judge lenses. Trace-verified exploits account for only 62-87% of recovered flags across benchmarks, while shortcut recoveries follow substantially shallower trajectories. These findings shift CTF evaluation from counting recovered flags to verifying demonstrated exploitation and provide a basis for designing benchmarks that better isolate the offensive capabilities.
Introduction
CTF benchmarks often treat correct flag recovery as evidence of capability, although identical flags can arise through exploitation or shortcuts. CTF-Abacus reconstructs traces into evidence-grounded solve profiles and aggregates them to assess benchmark validity.
- Motivation: CTF evaluations commonly reduce each run to whether its submitted flag matches the ground truth.This treats recovery as evidence of the targeted capability despite differing pathways.
- Motivation: Correct flags may come from intended exploitation, exposed data, recalled solutions, external lookups, guesses, or unsupported claims.Conventional scoring collapses these pathways into one solve, risking overstated capability.
- Framework: CTF-Abacus reconstructs each run as an evidence-grounded solve profile by decomposing actions into penetration-testing phases and category-specific techniques.It audits where exploitation occurs, where the flag first appears, and whether demonstrated behavior supports the recovered flag.
- Framework: Aggregating solve profiles across models and categories produces challenge signatures that reveal whether success depends on the intended capability.The framework is presented as both an observability instrument and a method for auditing challenge validity.
- Study scope: The study analyzes 1,435 attempts by six frontier language models on 240 challenges, yielding over 2,870 solve profiles under two independent judges.The evaluation spans four CTF benchmarks.
Related Work
Prior CTF-agent work established interactive environments and increasingly capable autonomous attack systems, but evaluation commonly emphasizes success counts. Related work also documents contamination concerns and behavioral preferences, whereas this paper studies whether traces warrant submitted flags.
- CTF agents: InterCode-CTF introduced an interactive Bash-shell environment with 100 picoCTF tasks, while GPT-4 solved 40 tasks but struggled with multi-step investigation.Subsequent systems addressed competition-grade challenges, live web applications, and executable attack plans.
- Evaluation gaps: Reviews report that offensive-security prototypes often publish little beyond success counts and use heterogeneous testbeds and experimental designs.The literature calls for baselines, richer metrics, and qualitative analysis of agent behavior.
- Evaluation gaps: Contamination can inflate measured performance, while existing detectors assess benchmark-level output statistics rather than individual solves.CTFusion reported 14.4% on reused NYU CTF Bench versus 6.3% on live competition challenges, and web search increased one agent’s score from 12.59% to 24.07%.
- Positioning: CyBiasBench studies which attack families agents prefer, whereas CTF-Abacus studies whether recorded actions support the submitted flag.The distinction is between behavioral preference and evidential warrant.
Methodology
CTF-Abacus converts agent traces into labeled solve profiles, assigns flag provenance, audits evidence, and aggregates profiles into challenge signatures. Its validation uses independent judges and blind human review to assess reconstruction fidelity.
- Provenance pipeline: The framework judges each attempt by flag provenance π, recording where the flag came from and whether it was earned.It audits provenance with two model lenses, routes unresolved cases to human experts, and aggregates profiles into challenge signatures.
- Trace reconstruction: Multi-agent trajectories are linearized into ordered action-observation steps, then labeled with penetration-testing phases and category-specific techniques.Reasoning messages are retained for provenance because a flag may first appear there, but are not treated as execution steps.
- Provenance validation: A target-observed flag is classified as a genuine solve only when the inferred exploitation act precedes its appearance; reasoning-only flags are classified as agent-reasoning.The procedure removes test-flag, scaffold, and recalled reads before assigning provenance.
- Provenance validation: Two independent judges label actions and observations, while a blind family-matched audit scores phase, technique, and flag-event fidelity.The reconstruction fidelity average is 98.3/100.
Experiment Setup
The evaluation spans four offensive-security benchmarks, 240 challenges, and six models selected to represent capable agents across varied task categories and environments. Two judges from different model families evaluate the traces.
- Benchmarks: The study evaluates Hack The Box, InterCode-CTF, CTFTiny, and CyBench across 240 challenges.The benchmarks cover web exploitation, binary exploitation, reverse engineering, cryptography, forensics, and miscellaneous security tasks.
- Benchmarks: The benchmarks differ in challenge construction, execution environment, and interaction complexity.This variation supports evaluation across known benchmark settings.
- Models and judging: Six models are evaluated: Claude Opus 4.8, GPT 5.5, Gemini 3.1 Pro, GLM 5.2, DeepSeek V4 Pro, and Qwen 3.7 Max.They were selected as among the strongest available at evaluation time.
- Models and judging: Claude Opus 4.8 and GPT 5.6 Sol serve as judges from different model families to reduce model-specific bias.Agents run under the harness’s default configuration.
Evaluation Results
CTF-ABACUS evaluates agent runs through execution-backed solve profiles rather than flag recovery alone, distinguishing genuine exploitation from shortcut pathways. Across benchmarks, trajectory depth and geometry reveal capability differences that aggregate recovery and technique breadth can miss.
- Study design: 1,400 attempts from six models across 240 challenges were evaluated, with 1,056 flag recoveries profiled independently under Claude and GPT lenses.Forty-seven runs were excluded as non-evaluable; reported results otherwise use the Claude lens.
- Benchmark comparison: Benchmark quality is better assessed by exploitation rigor than by the number of recovered flags.The evaluation reports recovery, execution-backed status, and verified scores separately, making demonstrated exploitation the relevant yardstick.
- Trajectory analysis: Geometry relates more strongly to category and benchmark than to model, with Cramér’s V of 0.26-0.29 versus 0.08-0.10 across the two lenses.Category values are 0.26/0.27 and benchmark values are 0.29/0.28 for Claude and GPT, respectively.
- Behavioral depth: Genuine solves reach advanced PTES depth, whereas shortcut recoveries produce flags without exploitation traces in tool outputs.This contrast separates execution-backed behavior from recoveries that do not demonstrate exploitation.
- Benchmark comparison: HTB and CTFTiny lead on genuine share, depth, monotonicity, and reach, while Cybench ranks lowest on these dimensions despite the highest technique breadth.Cybench’s τ=9 corresponds to 36% recovery and 62% genuine share; Intercode recovers often but shallowly.
Discussion and Insights
Trace-based analysis shows that flag recovery, execution evidence, benchmark validity, and behavioral trajectories can diverge substantially. These differences make sequence-level provenance important for interpreting scores and designing CTF benchmarks.
- No Cheap Proxy: F1 0.28 shows that direct flag-first-appearance detection poorly separates execution-backed from unsupported recoveries.It flags 28.2% of execution-backed and 29.0% of unsupported recoveries, so preceding evidence and environment responses are necessary.
- Ranking Instability: 17.4-22.6% score reductions and a pooled rank correlation of 0.60 show that execution-backed verification changes benchmark results.The adjustment is primarily benchmark-dependent: benchmark identity explains η2 = 0.0369 versus η2 = 0.0022 for model identity.
- Benchmark Inflation: 72.9% of 1,056 recovered flags were attributed to executed attacks, while 10.9% were human-verified derivations and 16.2% unsupported pathways.Unsupported recoveries ranged from 8.6% on CTFTiny to 31.0% on Cybench, separating evidential validity from headline difficulty.
- Benchmark Inflation: 151 unsupported recoveries came from direct exposure versus 17 from recall or external lookup, an 8.9fold difference.Reporting provenance alongside scores distinguishes shortcut types and identifies benchmarks needing adaptation.
- Challenge Signatures: Among 196 recovered challenges, 46.4% were fully supported, 40.3% mixed, and 13.3% unsupported under strict execution criteria.Crediting human-verified derivations reduced the unsupported group to 5.1%, but 33.2% remained mixed.
- Challenge Signatures: robust_cbc and rev-76 had identical terminal success across six models but differed between direct exposure and verified derivation.This supports treating validity as an interaction between challenge and solver rather than a fixed binary challenge property.
- Behavioural Divergence: Execution-backed recoveries reached Exploitation 88.7% of the time versus 41.6% for unsupported recoveries, while Reporting appeared in 1.8% versus 24.8%.The reporting reversal persisted within challenges in 16 of 20 non-tied comparisons (p = 0.012).
- Behavioural Divergence: author_solve_script appeared in 35.7% of execution-backed recoveries versus 12.2% of unsupported recoveries, whereas flag_submission appeared in 1.8% versus 23.4%.These are population-level patterns; the strongest single feature reached only 0.617 AUC.
Conclusion
CTF-ABACUS argues that CTF evaluation should verify how flags were obtained, not merely whether they were recovered. Its trace-based analysis finds substantial unsupported recovery and benchmark-dependent validity, motivating behavioral evidence alongside outcome scores.
- Conclusion: 72.9% of recovered flags were supported by execution, while 10.9% were human-verified derivations and 16.2% lacked either form of evidence.These figures summarize provenance across 1,056 recovered flags.
- Conclusion: 17.4-22.6% score reductions followed execution-backed verification, and 40.3% of challenges showed mixed solve pathways across models.Identical flag-based scores can therefore represent different behavioral evidence.
- Conclusion: Behavioral evidence should complement outcome-based scoring in future evaluations of autonomous offensive-security agents.The framework distinguishes demonstrated exploitation from alternative pathways and supports assessing challenge validity.