Where Do the Tokens Go? Understanding and Reducing Costs in LLM Agents for Vulnerability Discovery

1 Introduction

• Cost is a practical deployment constraint because some vulnerability-discovery tasks can run for hours and consume millions of tokens.The work frames cost alongside effectiveness rather than treating token use as an isolated measure.Source: §1 · p. 1
• Aggregate success rates and total costs alone cannot distinguish productive analysis from repeated exploration.Efficiency interventions therefore require evidence about where agents spend their budgets and what prevents a working PoC.Source: §1 · p. 1
• Code localization and understanding plus vulnerability reasoning and trigger design account for 60.4% of token consumption and 68.7% of attributed failure weight.Localization and understanding account for the larger token share; reasoning and trigger design account for the larger failure-weight share.Source: §1 · p. 2
• The study analyzes 200 CyberGym traces from four agents under an unaided baseline and four efficiency methods.It examines effectiveness and cost, token-consuming activities, and bottlenecks that prevent success.Source: §1 · p. 2
• AVRI organizes a persistent Bidirectional Evidence Trace (BET) that links harness input propagation to conditions that could make an analyzed operation unsafe.The record retains source-grounded links alongside agent hypotheses and unresolved questions, with commands for reading, analysis, and persistence.Source: §1 · p. 2
• On 20 evaluation tasks, AVRI lowers total cost by 18.0% for Codex and 23.7% for OpenCode relative to their baselines, without lowering success or recall.It also achieves the lowest cost among the evaluated configurations.Source: §1 · p. 2

2 Empirical Study Design

• The study asks how agents compare in effectiveness and cost, which stages consume tokens and bottleneck failures, and how efficiency methods help or are limited.It evaluates success, recall, elapsed time, and billed cost alongside token use and observed mechanisms.Source: §2 · p. 2

2.1 Task Setting

• Tasks use real-world CyberGym vulnerabilities in open-source C/C++ projects, with a harness entry point and no vulnerability-mechanism description, patch, or reference PoC.The scope excludes attack-surface selection and harness construction; each run has a 7,200-second timeout.Source: §2.1 · p. 2
• Success requires a nonzero, non-timeout exit on the vulnerable build; recall additionally requires a zero exit on the fixed build.Outcomes are measured from submitted inputs, not agent self-reports; a crash can count as success even if it exposes a different defect from the benchmark vulnerability.Source: §2.1 · p. 2

2.2 Subject Agents

• The four agents use different interfaces and execution designs, and Codex uses GPT-5.5 while the other agents use DeepSeek-V4-Pro.Comparisons involving Codex therefore include a model difference; the other three agents share the underlying model in this setup.Source: §2.2 · p. 2

2.3 Evaluated Efficiency Methods

• The four evaluated methods target history compression, vulnerability-hypothesis ranking, coverage feedback for PoC generation, and repository-level code navigation.They are AgentDiet, Revelio, PAgent, and CodeGraph, respectively.Source: §2.3 · pp. 2–3

2.4 Corpus

• The corpus contains 200 traces: four agents, five configurations each, and the same ten tasks per configuration, with each combination run once.The 160 enhanced runs are paired with baselines from the same agent and task; one run per configuration means paired results describe observed differences, not repeated-run variation.Source: §2.4 · p. 3

2.5 Multi-Axis Open Coding

• Activity stages are coded per turn, failure bottlenecks per unsuccessful trace, and method mechanisms per enhanced–baseline pair.The nine activity codes distinguish visible turn purpose, while failure assignments divide each unsuccessful trace's weight equally across its supported bottleneck categories.Source: §2.5 · p. 3
• Pilot annotations had Cohen’s κ=0.81, and a blinded human audit matched frozen automatic labels on 94% of sampled turns and 96% of boundary cases.The activity-stage rules were refined after review identified hypothesis-directed source inspection often mislabeled as general code understanding.Source: §2.5 · p. 3

3.1 RQ1: How Effective and How Expensive Are Current Agents?

• Agent outcomes varied sharply: OpenCode’s baseline achieved 100% success at $8.58, versus 50% at $49.34 for Cybench and 50% at $21.06 for EnIGMA.The reported costs are totals across the same ten tasks.Source: §3.1 · p. 4
• All four methods increased OpenCode’s total cost without improving its 100% success rate; across agents, method effects were inconsistent.For example, PAgent lowered Codex cost while increasing success, but raised EnIGMA cost while reducing success.Source: §3.1 · p. 4

3.2 RQ2: Which Stages Consume the Budget, and Where Do Runs Fail?

• Code localization and understanding used 43.1% of tokens versus 17.3% for vulnerability reasoning and trigger design, but their failure weights were 28.4% and 40.3%, respectively.The two categories together account for 60.4% of tokens and 68.7% of attributed failure weight; these weights are coded observations, not causal contributions.Source: §3.2 · pp. 4–5, Figure 1 · p. 5, Figure 2 · p. 6
• 38 traces had no observed PoC execution feedback, and all were unsuccessful; code inspection continued after feedback in other traces.The observations do not establish whether any post-feedback inspection was unnecessary.Source: Figure 1 · p. 5, §3.2 · p. 5
• Auxiliary usage was uneven: for Cybench with AgentDiet, auxiliary consumption exceeded the combined main-model consumption for localization, reasoning, and execution.The study includes auxiliary calls when assessing efficiency.Source: §3.2 · p. 5

3.3 RQ3: Do Existing Efficiency Methods Help, and Why Not?

• The findings motivate support that connects relevant code evidence to trigger conditions and preserves intermediate findings across turns.AVRI is presented as an approach to this problem.Source: §3.3 · pp. 6, 8
• 24.4% of 160 matched pairs both succeeded with lower enhanced-run cost; 14 pairs gained success and 21 lost it.The paired comparisons had seven fewer successful enhanced runs overall.Source: §3.3 · p. 6, Figure 3 · p. 7
• Unsuitable signals (59 pairs) and access or use limitations (38) were the most frequent coded limitations; seven pairs showed overhead reversal.The counts are nonexclusive; auxiliary billing can offset main-model savings.Source: §3.3 · p. 6, Figure 4 · p. 7, Figure 5 · p. 8

4 Our Approach: AVRI

• AVRI returns relevant code and supported relations between input values and unsafe-operation conditions, while retaining hypotheses and unresolved questions for reuse.The evidence is anchored to source locations, and derived facts are distinguished from agent conjectures.Source: §4 · p. 8

4.1 A Running Example and the AVRI Interface

• In the running example, the caller’s source-length check does not protect the callee’s eight-byte buffer; the mode and length guards must also permit the copy.BET can recover the saved source-length guard for later input construction, but execution must still test whether the candidate triggers a failure.Source: §4.1 · p. 8, Figure 6 · p. 9

4.2 Source Queries

• Source queries return related code, callers, uses, declarations, arguments, and conditions, while exposing ambiguous candidates rather than silently choosing among them.Queries can combine targets and merge overlapping ranges; Table 2 distinguishes immediate queries from optional analysis state.Source: §4.2 · p. 9, Table 2 · p. 10

4.3 Connecting Input Handling to Unsafe-Operation Conditions

• Forward analysis tracks supported input relations, while backward analysis follows unsafe-operation conditions; AVRI connects them through supported substitutions and retains unresolved dependencies.Analysis is limited to local control flow and a limited number of calling contexts; AVRI is not a whole-program verifier or symbolic executor.Source: §4.3 · pp. 10–11

4.4 Saving and Reusing Evidence with BET

• A BET stores forward and backward results, supported connections, assumptions, and unresolved questions for a source snapshot, harness, and analysis configuration.Saved evidence can support later input construction without repeating analysis; showing recovered evidence to the model is separate from internal reuse.Source: §4.4 · p. 11

4.5 Returning Results and Integrating with Agents

• AVRI prioritizes requested code and relations within an output budget, marks incomplete responses, and lets agents recover evidence when needed.Fewer queries or shorter responses alone do not establish savings, so the evaluation measures complete-run cost and outcomes.Source: §4.5 · p. 11

5.1 Experimental Setup

• The evaluation uses 20 randomly selected CyberGym tasks disjoint from the earlier study, with the same tasks run for Codex and OpenCode under the baseline and four methods.Each configuration-task combination has one run, and time and cost include failures and specified auxiliary charges.Source: §5.1 · pp. 11–12, Table 3 · p. 12

5.2 RQ4: Does AVRI Reduce Complete-Run Cost Without Sacrificing Effectiveness?

• 18.0% (Codex) and 23.7% (OpenCode) lower total cost, with success preserved and recall improved or maintained; AVRI also had the lowest total time for both agents.Codex recall rose from 65% to 75%; OpenCode maintained 90% recall.Source: Table 3 · p. 12, §5.2 · p. 12
• AVRI is the only evaluated method that reduces cost for both agents without lowering success or recall; existing methods either raise costs or lose effectiveness on these tasks.The reported outcomes are aggregate results from one run per agent–configuration–task combination.Source: §5.2 · p. 12

6 Threats to Validity

• One author coded failure bottlenecks and method mechanisms without independent review or agreement checks, leaving possible systematic interpretive bias.Traceability materials support review but do not rule out that bias.Source: §6 · p. 12
• One run per agent, method, and task cannot separate method effects from run variance by replication.The evaluation shares the one-run limitation, so observed gains are not established as robust to run-to-run variation.Source: §6 · p. 12
• Findings cover four agents on one benchmark family, C/C++ projects, and fixed harnesses; generalization beyond this setting is not established.AVRI’s lightweight backend depends on a bounded analysis region, and the authors do not expect unchanged transfer to open-ended attack-surface discovery.Source: §6 · p. 12
• The evaluation lacks component ablations, so it cannot isolate each AVRI mechanism’s contribution to the aggregate cost and effectiveness outcomes.The conclusion rests on one run per configuration and task.Source: §6 · p. 12

7 Related Work

• Prior vulnerability-discovery systems combine dataflow analysis, memory, path-condition validation, static analysis, debuggers, or coordinated reasoning, but evaluate different tasks and success criteria.This study focuses on reachable project code and constructing a crashing input from a harness entry point.Source: §7 · p. 13
• Efficiency work simplifies workflows, retrieves context selectively, parallelizes calls, or redesigns tool interaction, but does not identify which intervention addresses vulnerability discovery’s dominant costs.AVRI retains a source-anchored trace connecting input propagation to candidate unsafe operations and unresolved conditions across turns.Source: §7 · p. 13

• AVRI reduces total cost by 18.0% for Codex and 23.7% for OpenCode on 20 tasks, preserving success rates and improving or maintaining recall.Source: §8 · p. 13
• Code localization and understanding account for the largest token share, while vulnerability reasoning and trigger design are the main bottleneck in failed runs.Source: §8 · p. 13
• AVRI keeps a persistent Bidirectional Evidence Trace linking harness input to unsafe-operation conditions so agents can reuse evidence instead of rebuilding it.Source: §8 · p. 13
• Existing efficiency methods rarely preserve success at lower cost.Source: §8 · p. 13