PaperlayerView exact revision on arXiv

Source-linked AI summary

SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned

Cen Zhang, Younggi Park, Fabian Fleischer, Yu-Fu Fu, Jiho Kim, Dongkwan Kim, Youngjoon Kim, Qingxiao Xu, Andrew Chin, Ze Sheng, Hanqing Zhao, Michael Pelican, David J. Musliner, Jeff Huang, Jon Silliman, Mikel Mcdaniel, Jefferson Casavant, Isaac Goldthwaite, Nicholas Vidovich, Matthew Lehman, Taesoo Kim

arXiv:2602.07666v5cs.CRcs.AI

TL;DR

No systematic study had examined AIxCC’s design, finalist approaches, or lessons, despite the competition’s large-scale test of autonomous vulnerability discovery and remediation. This paper studies those areas using competition records, code, and discussions, finding technical advances alongside a persistent gap between technique capability and system reliability.

  • Problem

    No systematic study had examined AIxCC’s design rationale, finalist technical approaches, or lessons from the competition, although such analysis could inform organizers, security researchers, and practitioners.

    Source: §1 · p. 1

  • Method

    The authors analyze competition design, finalist CRS architectures, and outcomes using team code and whitepapers, official competition data and traces, and discussions with organizers and teams.

    Source: §1 · p. 1

  • Results

    The analysis identifies genuine CRS performance advances while finding a persistent gap between technique capability and system reliability.

    Source: §1 · p. 1, §10 · p. 13

  • Takeaways & Limitations

    The study offers a foundation for future competition design, CRS development, and practical deployment of autonomous cybersecurity systems.

    Source: §10 · p. 13

    The PF/MR/CC and PoV solvability annotations are approximations; longer fuzzing, different fuzzers, or private initial corpora could identify additional solvable cases.

    Source: §9 · p. 13

Abstract

from arXiv · show

DARPA's AI Cyber Challenge (AIxCC, 2023--2025) is the largest competition to date for building fully autonomous cyber reasoning systems (CRSs) that leverage recent advances in AI -- particularly large language models (LLMs) -- to discover and remediate vulnerabilities in real-world open-source software. This paper presents the first systematic analysis of AIxCC. Drawing on design documents, source code, execution traces, and discussions with organizers and competing teams, we examine the competition's structure and key design decisions, characterize the architectural approaches of finalist CRSs, and analyze competition results beyond the final scoreboard. Our analysis reveals the factors that truly drove CRS performance, identifies genuine technical advances achieved by teams, and exposes limitations that remain open for future research. We conclude with lessons for organizing future competitions and broader insights toward deploying autonomous CRSs in practice.

1 Introduction

AIxCC’s 2025 final evaluated autonomous LLM-enabled systems on real-world C and Java software, and this paper presents the first systematic study of the competition’s design, finalist systems, and results. The analysis draws on competition records, team materials, and discussions to examine those areas and their implications.

1 Introduction

  • The final evaluated seven finalist systems for around 143 hours across 53 challenge projects, with each team allocated $85K in cloud compute and $50K in LLM API credits.The projects were derived from critical-infrastructure software.Source: §1 · p. 1
  • The study draws on finalist codebases and whitepapers, competition databases and execution traces, and discussions with organizers and teams.Source: §1 · p. 1
  • The paper systematically analyzes AIxCC’s design and scoring principles, challenge construction, and execution guidelines.Source: §1 · p. 1
  • It characterizes the architectures and techniques of all seven finalist teams across vulnerability discovery, patching, report triage, and bundling.Source: §1 · p. 1
  • Its result analysis examines performance factors and per-vulnerability outcomes to identify technical insights.Source: §1 · p. 1
  • The authors offer lessons for future competition design, industry deployment, and research directions.Source: §1 · p. 1

2 Background: AIxCC as Competition

AIxCC advanced autonomous vulnerability discovery and remediation for open-source software, culminating in a 2025 final with seven teams. The paper situates the competition relative to the Cyber Grand Challenge and describes its sources and analysis methods.

2 Background: AIxCC as Competition

  • From 42 entrants, seven teams reached a final that ran around 143 hours across seven phases and analyzed 53 projects in C and Java.Each finalist had $85,000 in Azure compute and $50,000 in LLM API credits.Source: §2 · p. 2
  • Unlike the Cyber Grand Challenge’s binary exploitation on DECREE OS, AIxCC targets vulnerability discovery and remediation in real-world C and Java software, with LLM infrastructure.Source: §2 · p. 2
  • The study grounds its competition-design account in organizer documents and discussions, its team taxonomy in code reviewed by security experts, and its outcome analysis in official records and challenge artifacts.Source: §2 · p. 2

3 Competition Design

AIxCC modeled autonomous cyber reasoning around software-development events and accepted multiple forms of vulnerability evidence. Its scoring and competition process were designed to weight practical value and evolve in response to problems surfaced before the final.

3 Competition Design

  • The competition workflow runs from a webhook trigger to challenge dispatch and result submission, with capabilities aligned to development events.Full Scan targets releases, Delta Scan targets pull requests, SARIF Review assesses alerts, and Report Synthesis correlates findings.Source: Figure 1 · p. 2, §3 · p. 2
  • CRSs could submit PoVs, patches, SARIF assessments, and bundles as evidence for a challenge.A PoV demonstrates abnormal execution, a patch resolves the vulnerability while preserving functionality, and a bundle links related findings.Source: §3 · p. 2
  • Scoring weights reflect expected developer effort saved or wasted: patches score highest, PoVs lower, and SARIF assessments lowest.Bundle scores reward correct linkage and penalize incorrect linkage.Source: §3 · pp. 2–3
  • Time-decay rewards earlier submissions, while an accuracy multiplier penalizes per-challenge totals more sharply at lower accuracy.For example, 50% accuracy incurs a 6% reduction and 40% accuracy a 13% reduction.Source: §3 · p. 3
  • The final corrected two semifinal design flaws: it scored patches against all teams’ PoVs and adopted OSS-Fuzz instead of per-repository challenge adoption.Finalists also received per-team budgets, self-managed Azure infrastructure, and dedicated pre-competition support.Source: §3 · p. 3
  • Three unscored exhibition rounds used a separate challenge set mirroring the scored setting so teams and organizers could iterate on systems and processes.Source: §3 · p. 3

4 Challenge Projects

The final challenge set combined scored vulnerability tasks across open-source repositories with additional unscored-by-design discovery targets. It covered multiple languages, vulnerability types, and challenge modes.

4 Challenge Projects

  • Organizers selected critical-infrastructure and healthcare-critical software, using mostly hand-crafted synthetic vulnerabilities inspired by historical N-day issues alongside a few genuine 0-days.Source: §4 · p. 3
  • 48 scored challenge projects from 24 OSS-Fuzz repositories contained 63 vulnerabilities; five projects without fuzzing harnesses were excluded from scoring.Source: Table 1 · p. 3, §4 · p. 3
  • Eight challenges contain no injected CPV and serve solely as 0-day discovery targets; the passage reports 63 CPVs overall, split into 33 full-mode and 30 delta-mode vulnerabilities.Source: §4 · p. 3
  • The 63 vulnerabilities comprised 40 C and 23 Java cases across 34 CWE types; organizers also issued 13 SARIF broadcasts, eight valid and five invalid.Source: §4 · p. 3

5 Cyber Reasoning Systems

The seven finalist CRSs shared core capabilities but varied widely in architecture, reflecting differences in team expertise, resources, and priorities. Their designs ranged from ensemble and agentic systems to deterministic workflows and fuzzing-centered approaches.

CRS design philosophies

  • AT combines independent bug-finding modules with seed sharing and eight diverse patching agents, while TB decomposes work into deterministic subtasks and uses mid-tier models.TB avoids high-end reasoning models, believing well-decomposed problems paired with mid-tier models suffice.Source: §5 · p. 4
  • TI centers its agentic system on bug candidates across identification, filtering, PoV generation, patching, SARIF validation, and bundling; LC coordinates fuzzing, patching, and analysis with a multi-LLM DSPy workflow.LC uses models in parallel or as fallbacks, with patch failures refining vulnerability analysis.Source: §5 · p. 4
  • FB simplifies its architecture while using 23 independent strategies that vary in scope, depth, and language-specific handling.Over 90% of FB’s codebase is described as vibecoded.Source: §5 · p. 4
  • SP implements techniques across all four core capabilities and coordinates 53 components with a custom orchestration platform, whereas 42 favors traditional fuzzing and program analysis with auxiliary LLM roles.42 also simplifies adopted academic techniques to make them practical.Source: §5 · p. 4

6 Taxonomy of CRS Techniques

The taxonomy describes how finalist CRSs combine fuzzing and LLM-driven PoV generation, structure patch pipelines, validate SARIF reports, and link submissions into bundles. Across these capabilities, teams vary in how they combine tools, agents, evidence, validation, and submission strategies.

6.1 PoV Generation

  • Fuzzing and LLM-based PoV generation reinforce one another: fuzzers provide coverage and inputs, while LLM outputs seed fuzzers; five teams explored both pipelines, while 42 and LC focused on fuzzing.Six teams explored LLM-based PoV generation; LC performed bug-candidate identification only, while TB skipped that step.Source: §6.1 · pp. 4–5
  • Five teams reused pre-collected corpora, six used LLMs to generate seeds, and teams also refined fuzzing engines, Java sanitizers, dictionaries, and scheduling.Seed corpora were paired with harnesses by coverage ranking or similarity matching; LLM seed generation supported early bootstrap and troubleshooting.Source: §6.1 · pp. 4–5
  • PoV-generation agents use analysis tools and runtime evidence, while bug-candidate systems combine LLMs, static analysis, and sink lists; teams share results between generation and fuzzing.TI uses LLM logprobs as a confidence signal, while SP and LC aggregate weighted votes to rank candidates.Source: §6.1 · p. 5

6.2 Patch Generation

  • Patch architectures include ensembles of distinct patchers, coordinating multi-agent designs, and single-agent systems; four CRSs also separate root-cause analysis from patch synthesis.AT uses eight patcher agents, SP pairs an agentic patcher with a program-analysis-assisted one-shot patcher, and TB and TI use multi-agent designs.Source: §6.2 · pp. 5–6
  • Patch generation draws on crash and sanitizer output and code search, with optional context sources and LLM reflection; all CRSs perform build verification and PoV reproduction checks.Most CRSs revalidate against a broader PoV set before submission than during generation.Source: §6.2 · p. 6
  • Four CRSs calculate minimal patch sets to cover known PoVs, while the three CRSs capable of no-PoV patching delay those submissions to reduce imperfect-patch penalties.Minimal-set strategies differ in calculation timing, whether they recalculate over all PoVs, and submission timing.Source: §6.2 · p. 7

6.3 SARIF Validation

  • SARIF verdict strategies rely on matching reports to PoVs, bug candidates, or LLM judgments; teams differ in which evidence they use and whether they submit both verdict types.AT, TB, and FB primarily use PoV matching; SP, 42, and LC center LLM judgments; TI matches against its bug-candidate database.Source: §6.3 · p. 7

6.4 Bundling Strategy

  • Bundling links PoVs, patches, and SARIF assessments into reports, with incorrect pairings penalizing the entire bundle; teams generally derive links from existing workflows.All teams derive PoV–Patch links from PoV-based patch generation, while SARIF pairing strategies vary.Source: §6.4 · p. 7

7 Competition Result Analysis

The results show that stability and accuracy penalties shaped final rankings, while per-vulnerability analysis identifies both CRS advances beyond baseline techniques and persistent pipeline limitations. The analysis also compares patch quality and other competition outcomes.

7.1 What Scores Reveal (and Conceal)

  • 392.8 points gave AT nearly 80% more than second-place TB (219.4), as AT sustained scoring across all seven phases while TB and TI plateaued.Figure 2 plots score over time and by phase; Table 7 breaks down final scores.Source: Figure 2 · p. 8, Table 7 · p. 8, §7.1 · p. 8
  • TI’s −16.3 accuracy penalty dropped it from a pre-penalty score above TB to third, illustrating how research, engineering, and strategy shaped outcomes.The authors caution that score-based capability comparisons are uncertain because outages and strategy choices can also produce low scores.Source: §7.1 · p. 8

7.2 Auxiliary CPV Annotation

  • Annotations show that many CPVs were solvable by individual techniques, while CRSs both exceeded those references and missed annotated-solvable challenges.PF found 34/63 PoVs; MR and CC patched 31/63 and 33/63. The laboratory annotations approximate, rather than exhaustively define, CRS capability boundaries.Source: §7.2 · pp. 9–10

7.3 PoV Generation Analysis

  • Six CRSs solved 7–16 CPVs PF missed; LLM reasoning complemented fuzzing on delta challenges, complex input grammars, and constraint-heavy bugs.The analysis also identifies reproduction behavior, sanitizer choice, deduplication, broken dependencies, and heavy builds as pipeline obstacles; bugs missed by both CRSs and PF exposed reasoning and fuzzing limits.Source: §7.3 · pp. 10–11

7.4 Patch Generation Analysis

  • Claude Code slightly outperforms MultiRetrieval, but semantic incorrectness remains 38–46%; higher-accuracy CRSs reduce it to 16–21% and solve 16 baseline-missed cases.Manual review found semantic defects in 37.7% of CC patches and 45.6% of MR patches.Source: §7.4 · pp. 11–12
  • Patch failures persist when PoVs are absent or deadlines are consumed by builds; timeout and JVM-crash cases also deprive patchers of useful root-cause signals.Manual analysis found incomplete fixes, functionality deviations, and newly introduced bugs, among other semantic problems.Source: §7.4 · pp. 11–12

7.5 Other Analyses

  • Bundles reached 91/99 (92%) accuracy, with PoV-based patch pairs accounting for 86% of bundles at 93% accuracy.PoV-centric teams could not assess five invalid SARIF reports; all seven teams discovered at least one 0-day, yielding 25 vulnerabilities across 10 OSS projects, of which 12 were patched.Source: Table 8 · p. 12, §7.5 · p. 12

8 Lessons and Future Directions

The paper identifies deployment barriers and research gaps for future AIxCC competitions, while noting that AIxCC’s LLM–non-LLM cooperation remains relevant as LLM capabilities advance.

From Competition to Industry Deployment

  • Finalist CRSs ran on self-provisioned Azure clusters with hundreds of dollars per challenge, unlike the low-cost, single-machine settings typical of individual developers and small teams.Their resource models and runtime environments pose adoption barriers; the paper suggests resource-efficient variants and resource-limited competition tracks.Source: §8 · p. 12
  • OSS integration requires maintainers to provision LLM services, standardize CRS interfaces, define end-to-end workflows, and consider ensembling multiple CRSs.The paper says considering these needs during competition design could lower adoption barriers; OSS-CRS has begun addressing them.Source: §8 · p. 13

From Competition to Research Advancement

  • AIxCC telemetry supports real-time monitoring and scoring more than retrospective analysis of system behavior.Logging intermediate outputs, decision traces, and environmental snapshots could support failure-mode and technique-effectiveness studies; organizer-built baseline CRSs could provide comparison points.Source: §8 · p. 13
  • Open-source LLM exploration was limited: two teams fine-tuned open-source models, but only one deployed them.The paper attributes limited incentives to uncertainty, cost, and data-acquisition difficulty, and proposes a dedicated open-source-model sub-track.Source: §8 · p. 13

Areas of Expansion

  • AIxCC did not cover full autonomy, multi-CRS settings, or semantic correctness evaluation, which the paper identifies as directions for future editions.The paper also notes early-2026 advances in frontier LLM cybersecurity capabilities, including Claude Mythos Preview and coding-agent patch evaluation.Source: §8 · p. 13

AI-Powered Vulnerability Scanning

  • LLM–non-LLM cooperation established in AIxCC remains relevant because pure-LLM pipelines retain hallucination and nondeterminism limitations.Stronger LLMs may also allow future CRSs to simplify architectures and push autonomy further.Source: §8 · p. 13

9 Limitations

The study’s limitations concern taxonomy interpretation, approximate technique annotations, and the absence of controlled ablations.

Limitations

  • Taxonomy profiles follow enabled functionality in submission-version code, but source-code misreadings could still propagate as misinterpretations.Two security experts cross-validated profiles, and the final taxonomy was shared with all seven teams; three confirmed it.Source: §9 · p. 13
  • PF/MR/CC annotations approximate what selected techniques would solve under estimated resources, rather than establishing general capability.Longer fuzzing, another framework, or private initial corpora could mark more CPVs solvable; three runs combined by union provide a stable lower bound for the representative configuration.Source: §9 · p. 13
  • The study lacks controlled ablations because moving heterogeneous, resource-intensive CRSs onto one platform would incur substantial engineering and resource costs.The authors identify this as out of scope for a SoK and note that post-competition efforts such as OSS-CRS are pursuing the direction.Source: §9 · p. 13

10 Conclusion

The paper concludes that AIxCC demonstrates autonomous CRSs can discover and patch real-world software vulnerabilities at scale, while revealing both performance advances and a gap between technique capability and system reliability.

Conclusion

  • AIxCC demonstrates that AI-powered CRSs can discover and patch vulnerabilities in real-world software at scale.The paper presents its systematic analysis of competition design, CRS architectures, and results as a foundation for future competition design, CRS development, and practical deployment.Source: §10 · p. 13

Ethical Considerations

The study frames its analysis as a contribution to OSS security and future CRS research, while documenting risks involving team reputations, pre-patch vulnerabilities, and dual use. Its safeguards include team review, responsible disclosure, and withholding detailed 0-day information.

Ethical Considerations

  • Organizers and teams contributed firsthand accounts, and finalists were informed of the analysis and given draft sections for review.The authors also cross-validated each section with at least two authors and maintained communication with all seven teams.Source: Ethical Considerations · pp. 14–15
  • The authors analyze all seven finalists consistently, do not independently reevaluate their systems, and use final competition results as the source of performance data.Competition data is intended for public release subject to DARPA approval and disclosure guidelines.Source: Ethical Considerations · p. 14
  • Identified harms include misinterpreted team performance, exploitation of exposed 0-days before patches, and dual-use adaptation of CRS architectures or competition data.The paper says this work uses no human subjects or private user data.Source: Ethical Considerations · p. 15
  • All reported 0-day vulnerabilities are handled through responsible disclosure, with public details withheld until fixes or a standard 90-day window; the paper omits detailed 0-day information.Kudu Dynamics, OSTIF, and ADALogics coordinate privately with affected maintainers and share proof-of-vulnerability material and remediation guidance.Source: Ethical Considerations · pp. 14–15
  • The two-year competition involved burnout risks and pressure from changing rules and AI capabilities, while some teams’ outcomes were affected by instability rather than capability gaps.Participating teams could complete questionnaires in flexible formats based on availability.Source: Ethical Considerations · p. 15
  • The study addresses the lack of prior systematic analysis of AIxCC to document lessons, technical advances, and open challenges for researchers, organizers, and the OSS security community.The authors say the expected benefits of insights into CRS design and vulnerability repair outweigh potential dual-use risks.Source: Ethical Considerations · pp. 14–15

Open Science

The appendix documents where supporting materials are archived and what per-team resource, submission, and technique details are reported. It also describes how teams adapted PoV and patch workflows to challenge conditions.

Open Science

  • The artifact archives data, scripts, finalist questionnaires, and meeting notes, while its companion website indexes the artifact, extended analysis, finalist documentation, and challenge-set access links.Competition data and the analysis framework await DARPA’s official release.Source: Open Science · p. 15

A Per-Team Details

  • Table 9 reports each team’s Azure compute consumption and LLM API budget.The supplied passage gives the table’s scope but no usage values.Source: A Per-Team Details · p. 18, Table 9 · p. 18
  • Table 10 reports counted correct and incorrect submissions, excluding duplicates, and accuracy rates by scoring category.The supplied passages do not provide the table’s numerical results.Source: A Per-Team Details · p. 18, Table 10 · p. 18
  • Teams adapt PoV and patch workflows by language and challenge mode, targeting Java sinks, delta diffs, SARIF candidates, and challenge-specific tooling or guidance.For delta-mode patch analysis, most teams focus on modified code; TI runs parallel diff-analysis agents, one filtering compiler-unused files and one using the full diff.Source: A Per-Team Details · p. 18

B Abbreviations

The appendix defines CP and CPV identifiers and lists abbreviations for competition concepts, methods, teams, and systems.

B Abbreviations

  • A CP identifier combines project, challenge-project index, and mode; a CPV adds a numeric vulnerability index within that CP.The mode marker distinguishes full-mode from delta-mode.Source: B Abbreviations · p. 18
  • The acronym list expands PoV as Proof of Vulnerability, SARIF as Static Analysis Results Interchange Format, and CWE as Common Weakness Enumeration.Source: B Abbreviations · p. 18
  • Team abbreviations map AT to ATLANTIS, TB to BUTTERCUP, and TI to ROBODUCK.Source: B Abbreviations · p. 18
Loading 2602.07666v5…