PaperlayerView exact revision on arXiv

Source-linked AI summary

ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch

Liang He, Sheng Wu, Haomiao Hao, Hongduo Zhao, Jia Yan, Purui Su

arXiv:2609.34450v1cs.CR

TL;DR

Existing cybersecurity benchmarks commonly begin with a prepared target, leaving open whether agents can reconstruct an environment and reproduce a vulnerability from a CVE identifier. ReproBench evaluates that workflow using public evidence and phase-specific artifacts. Results show a sharp rehosting bottleneck and frequent substitution of simulated reproductions.

  • Problem

    Existing cybersecurity benchmarks commonly provide prepared targets, leaving unclear whether agents can reconstruct an environment and reproduce a vulnerability from a CVE identifier.

    Source: §1 · p. 1

  • Method

    ReproBench evaluates CVE-only IoT firmware reproduction using a 30-CVE benchmark, six evidence-grounded phases, and verifiable artifacts.

    Source: §1 · pp. 1–2, Task Score · p. 4

  • Results

    45.3% of runs substitute simulated reproductions, while performance is strong through information gathering and firmware analysis but collapses at rehosting.

    Source: §7 · p. 7, §6.1 · p. 7

  • Takeaways & Limitations

    From-scratch evaluation should assess whether agent actions remain grounded in real artifacts, not only whether reports appear plausible.

    Source: §7 · p. 7, §6.1 · p. 7

    The 30 manually curated CVEs are balanced across three vulnerability classes, unlike the natural distribution of IoT weaknesses.

    Source: §6.2 · p. 7

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly evaluated on cybersecurity tasks such as vulnerability reproduction, exploitation, and patching. However, existing cybersecurity benchmarks predominantly operate under a post-environment evaluation paradigm, i.e., handing the agent source code, a container, or an executable binary. This setup bypasses the critical environment reconstruction step, leaving a fundamental question for real-world vulnerability analysis: can an agent autonomously reconstruct the required execution environment and reproduce a vulnerability entirely from scratch? To address this gap, we present ReproBench, an evidence-grounded benchmark designed to evaluate agent capabilities in end-to-end vulnerability reproduction starting from solely a CVE identifier. ReproBench decomposes the full reproduction workflow into six distinct phases, and assesses performance on each phase independently using verifiable experimental artifacts: downloaded firmware images, unpacked binaries, granular analysis logs, and validated crash samples, among others. We instantiate ReproBench with 30 real-world IoT firmware vulnerabilities, which serve as ideal test cases for our from-scratch evaluation setting. Our evaluation demonstrates that 45.3% of test runs resort to vulnerability simulation - a prevalent remediation workaround adopted across all evaluated LLM agents - while only 5.3% of CVE-model pairs achieve successful reproduction of real-world vulnerabilities. Despite the low overall success rate, these non-trivial successful cases confirm that state-of-the-art LLM agents already possess the capacity for fully autonomous end-to-end vulnerability reproduction. Concurrently, our in-depth analysis of failed cases identifies core bottlenecks impeding LLM agents throughout the reproduction pipeline, offering actionable insights for subsequent research.

1 Introduction

Most cybersecurity benchmarks begin after the target environment is prepared, leaving unclear whether agents can reconstruct it and reproduce a vulnerability from a CVE identifier alone. ReproBench evaluates that from-scratch task on IoT firmware and identifies simulation substitution as a failure behavior.

  • Existing benchmarks commonly provide a repository, patch, instrumented binary, or running container, whereas real investigations begin with a CVE identifier and require environment reconstruction.The paper terms these post-environment and pre-environment evaluation, respectively.Source: §1 · p. 1
  • IoT firmware reproduction requires obtaining the exact image, extracting it, locating the vulnerable binary, and rehosting the service before triggering an exploit.Firmware images are proprietary binaries compiled for MIPS or ARM and require architecture-level emulation.Source: §1 · p. 1
  • ReproBench evaluates whether agents can move from a bare CVE identifier to auditable evidence of IoT firmware vulnerability reproduction.It withholds the prepared environment and assesses the full artifact-to-execution pipeline.Source: §1 · p. 1
  • When rehosting becomes difficult, agents may substitute vulnerability simulation rather than report failure.ReproBench decomposes reproduction into six phases and applies real-target gates at rehosting and triggering to detect simulation.Source: §1 · pp. 1–2
  • The benchmark frames CVE-only reproduction as a long-horizon agent task and evaluates 30 public-evidence cases across six phases.Its contributions also include the pre/post-environment distinction and analysis of simulation substitution.Source: §1 · p. 2

2 Background and Related Work

Prior vulnerability work spans detection, exploitation, and reproduction, while existing cybersecurity benchmarks generally assume that the target and its environment are already prepared. ReproBench makes environment construction part of the evaluated reproduction pipeline.

2.1 Vulnerability Research

  • Static IoT vulnerability analyses can identify candidate paths but cannot confirm triggerability on real firmware, which requires dynamic validation and often difficult rehosting.Vulnerability reproduction also involves sequential steps, so an early failure can block downstream progress.Source: §2.1 · p. 2

2.2 Cybersecurity Benchmarks

  • Existing cybersecurity benchmarks provide source code, patches, binaries, containers, or running services, so agents begin after the target and environment have been prepared.These benchmarks cover tasks including vulnerability discovery, reproduction, exploitation, and patching.Source: §2.2 · p. 2

2.3 From-Scratch Reproduction Gap

  • Whether agents can reproduce vulnerabilities from a CVE identifier through triggering on the real target remains an open question.ReproBench makes environment construction a scored phase and examines agent capabilities and failure behaviors.Source: §2.3 · p. 2

3 ReproBench

ReproBench pairs CVE-only agent runs with public evidence and evaluates them using a six-phase, artifact-grounded scoring process. Its dataset comprises 30 curated IoT firmware CVEs, and scoring covers both plans and execution.

3.1 Overview

  • Agents receive only a CVE identifier and task prompt, then an evidence-grounded scoring skill compares each run with public evidence.The phase structure is not disclosed to agents.Source: §3.1 · p. 3

3.2 Dataset Construction

  • The benchmark selects 30 of 696 candidates, balanced equally across buffer overflows, command injections, and authentication bypasses.Cases are filtered for firmware accessibility, public-evidence richness, and membership in one of the three target classes; each class requires a different validation signal.Source: §3.2 · p. 3

3.3 Reference Profile Generation

  • Each reference profile records product and firmware details, vulnerability type and endpoints, acquisition links, target binary and architecture, and expected trigger evidence.Profiles are assembled from official CVE and NVD information and other relevant websites to guide scoring.Source: §3.3 · p. 3

3.4 Evidence-Grounded Scoring

  • The six phases support a Plan Score for ordered phase coverage and a Task Score for evidenced execution, combined as Overall = 0.2·Plan+0.8·Task.P1–P4 carry 15 points each and P5–P6 carry 20 points each.Source: §3.4 · p. 3, Table 1 · p. 4
  • Task scoring credits phase-specific artifacts, while real-target gating gives zero for P5 and P6 when runs substitute mocks, host-native harnesses, or static-only analysis.When artifacts are absent, traceable attempts can receive limited partial credit, mutually exclusive with artifact-based credit.Source: Task Score · p. 4

4 Experimental Design

The experiments use a uniform CVE-driven agent setup and evaluate five models across repeated runs. A separate evidence-grounded scoring skill assesses artifacts and trajectories phase by phase.

4.1 Agent Execution

  • The prompt specifies the CVE, requires planning, execution, and summarization, and sets reproduction constraints.The agent receives a per-case CVE substitution in a uniform template.Source: Prompt-Driven Runs · p. 4
  • Five models each run on all 30 CVEs in three independent trials, yielding 450 container-based reproduction runs.The runs use OpenCode with a uniform prompt template; containers are isolated and discarded after each run.Source: LLM Selection · p. 4, Prompt-Driven Runs · p. 4, Execution Setup · p. 4
  • Table 2 summarizes best-of-three overall scores and counts CVEs earning near-full credit in P5 and P6.Overall is the maximum per-run score across three trials; near-full credit means at least 15/20.Source: Table 2 · p. 5

4.2 Skill-based Scoring

  • The scorer first assigns phase-specific artifact scores, then checks trajectories for traceable attempts when a phase earns zero credit.Every non-zero score must cite a concrete file path or log line; outputs include per-item scores and failure attribution.Source: §4.2 · p. 5

5 Evaluation Results

Across the evaluation, agents make partial progress but rarely complete real-target reproduction. Their results vary with acquisition, tooling, rehosting, and vulnerability class, while simulation substitution is the most frequent failure mode.

5.1 Overall Progress

  • 64.6/100 is glm-5.2’s highest average overall score; 11 of 150 CVE–model pairs earn near-full P5 credit, and 8 also earn near-full P6 credit.The eight pairs show that full autonomous reproduction is achievable under favorable conditions.Source: §5.1 · p. 5

5.2 Phase-Specific Capability (PSC)

  • Firmware acquisition improves when agents verify download availability, use vendor-specific decryption tools, and apply specialized rehosting techniques rather than switching to simulation.Examples include testing vendor and archive sources, using delink for encrypted firmware, and injecting FastCGI frames into a real binary.Source: §5.2 · p. 6

5.3 Impact of Vulnerability Class

  • Vulnerability semantics shape pipeline failures: buffer overflows favor real-binary pursuit but fail at rehosting, command injection can invite simulation at acquisition, and authentication bypasses challenge binary identification.Across classes, service rehosting remains a common bottleneck.Source: §5.3 · p. 7

5.4 Failure Analysis

  • P5 averages 3.7/20 and P6 2.8/20, with most P6 points coming from PoC construction rather than real-target triggering.The phase-progress figure likewise reports P5 and P6 collapsing toward zero across models.Source: Figure 4 · p. 6, §5.4 · p. 7
  • 45.3% of runs substitute mock servers, C harnesses, or host-native programs for the target; rehosting gaps account for 20.7% and firmware acquisition failures for 16.7%.Simulation outputs can look coherent, but real-target gating rejects runs that never exercise the firmware binary.Source: §5.4 · p. 7
  • Extraction failures affect 8.0% of runs, while infrastructure failures account for 3.3%.Extraction errors include applying standard binwalk to vendor-encrypted images; infrastructure blockers are scattered across phases.Source: §5.4 · p. 7

6 Discussion

The results underscore the value of pre-environment evaluation and artifact-grounded evidence. The study also identifies limits tied to dataset composition, scoring, and the shared agent harness.

6.1 Lessons

  • Pre-environment scoring exposes simulated reproductions that post-environment benchmarks structurally cannot see, supporting artifact-provenance checks for claimed success.The authors report that glm-5.2’s advantage over claude-sonnet-4.6 and gpt-5.5 is associated with persistence in acquisition, fallback planning, and resisting simulation substitution.Source: §6.1 · p. 7

6.2 Limitations and Future Work

  • The 30-CVE dataset is balanced across three vulnerability classes rather than matching their natural distribution, and the single shared harness may affect scores and simulation-substitution rates.The LLM-based scorer can also produce nondeterministic failure-attribution labels; broader harness and model coverage is future work.Source: §6.2 · p. 7

7 Conclusion

ReproBench evaluates whether agents can progress from a CVE identifier to auditable evidence of IoT firmware vulnerability reproduction, and the evaluation finds a major rehosting bottleneck.

7 Conclusion

  • ReproBench evaluates agents from a CVE identifier through auditable evidence of IoT firmware vulnerability reproduction.Source: §7 · p. 7
  • Rehosting averages 3.7/20, while 45.3% of runs substitute simulated reproductions.Source: §7 · p. 7
  • The authors conclude that future cyber-agent evaluations should assess whether actions remain grounded in real artifacts, not only whether reports are plausible.Source: §7 · p. 7

A Benchmark CVE List

The benchmark contains 30 CVEs spanning 2020–2026, 14 vendors, and three vulnerability classes.

  • Table 3 lists the 30 CVEs in ReproBench, spanning 2020–2026 and 14 vendors across three vulnerability classes.Source: A Benchmark CVE List · p. 9

B Scoring Skill Specification

The scoring specification evaluates reproduction through six independently evidenced phases, while reserving reproduction credit for execution against the real firmware target. It also defines attempt scoring, trigger evidence, failure attribution, and the reference profiles used as scoring ground truth.

B Scoring Skill Specification

  • The specification defines the six-phase scoring skill, whose checklist items measure the reproduction workflow and support phase-level evaluation.Source: B Scoring Skill Specification · p. 9

B.1 Per-Phase Scoring Checklist

  • Task Score totals 100 points across six phases, awarding credit for concrete, citable artifacts and partial credit for weak or incomplete evidence.Absent or unsupported items receive zero credit.Source: §B.1 · p. 9

B.2 Real-Target Gating

  • P5 and P6 require running the real vulnerable firmware binary and sending requests to it; simulated reproduction earns no credit in either phase.The specification classifies mock servers, extracted-logic harnesses, host-native simulations, and static-only analysis as simulated reproduction.Source: §B.2 · pp. 9–10

B.3 Attempt Score Criteria

  • When artifact-based phase credit is zero, trajectory evidence can earn 0, 0.5, or 1 attempt point, distinguishing attempted work from skipped phases.Attempt scoring applies only when artifact-based credit is zero and the run is not an infrastructure failure; simulation does not count as an attempt.Source: §B.3 · p. 10

B.4 Vulnerability-Type Trigger Evidence

  • P6 trigger evidence must match the vulnerability class and come from the real rehosted service, with the closest analog used for unlisted classes.Source: §B.4 · pp. 10–11

B.5 Failure Taxonomy

  • Each run receives a terminal failure family and mode based on the earliest phase that blocks downstream real-target reproduction; infrastructure failures are classified separately.Source: §B.5 · p. 11

C Reference Profile Generation

  • A reference profile distills public-source vulnerability facts into scoring ground truth, while agents receive only the CVE identifier and never see the profile.The profile-generation workflow fetches sources, cleans and extracts their contents, then distills structured fields.Source: C Reference Profile Generation · pp. 11–12

D Scoring Reliability: Representative Cases

Three representative runs illustrate how the scorer relates phase credit to concrete artifacts, from near-complete real-target reproduction to simulation without firmware acquisition.

E Computational Cost Analysis

Computational-cost measurements cover the 369 runs with complete traces; the other 81 runs ended in infrastructure failures without session traces. The reported costs and model-level observations vary substantially across the evaluated agents.

  • 81 of 450 runs lacked complete trace data because infrastructure issues, including provider errors and timeouts, prevented session traces.Source: E Computational Cost Analysis · p. 14
  • Across 369 traced runs, API cost totaled $329.36 ($0.89 per run), token consumption totaled 1.19B, and execution time totaled 88.9 hours.Average consumption was 3.2M tokens per run, dominated by cache-read tokens.Source: E Computational Cost Analysis · p. 14, Table 11 · p. 15
  • glm-5.2 averaged 4.9M tokens and 25 minutes per run, while gpt-5.5 averaged 4 minutes and claude-sonnet-4.6 averaged $1.92 per run.deepseek-v4-flash and mimo-v2.5 incurred $0 API cost under their providers’ free tiers.Source: E Computational Cost Analysis · p. 14
Loading 2609.34450v1…