Source-linked AI summary
SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction
Yuanxiang Shi, Jiayi Lin, Xuanyong Lin, Liangcai Su, Yeheng Duan, Wei Wang, Qi Han, Bing Zhao, Wei Hu, Xander Xu, Chenxiong Qian
TL;DR
Existing vulnerability-discovery benchmarks are vulnerable to contamination, unknowable vulnerability sets, and opaque end-to-end scoring. SWE-Test measures the ability through input prediction on real C/C++ programs across three decomposed modes, finding limited performance and constraint inference as the dominant bottleneck. It releases a turnkey evaluation environment for reproducible analysis.
Problem
Existing benchmarks can be gamed through contamination, cannot measure recall against an unknowable vulnerability set, and often obscure which discovery capability fails.
Method
SWE-Test mines deep target branches from real C/C++ programs and evaluates input prediction through Open-loop, Feedback-enabled, and Online Arena modes.
Results
55.0% is the best feedback-enabled pass rate across 15 default-effort model-scaffold configurations, while paired Claude Code mean pass rate rises from 19.3% without feedback to 36.4% with feedback.
Takeaways & Limitations
Constraint satisfaction remains the largest aggregate failure outcome, while evaluation should report navigation and refinement stages separately.
Takeaways & Limitations
The benchmark measures single-branch inversion as a proxy rather than full exploit construction, and its fixed-target findings may not transfer beyond the covered C/C++ programs and input formats.
Abstract
from arXiv · showhide
Vulnerability discovery is becoming an important ability of large language model (LLM) agents: agents that silently miss real defects leave critical software exposed. Rigorously measuring this ability is therefore urgent, but existing benchmarks are gameable through data contamination, score recall against an unknowable vulnerability set, often rely on synthetic bugs, and report a single end-to-end verdict that cannot localize where an agent fails. Vulnerability discovery is a composite ability: an agent must comprehend source code, infer input constraints, construct inputs, execute them, and iteratively correct from feedback. We recast its measurement as an input-prediction task with a closed, deterministic ground truth: using coverage-guided fuzzing, we mine deep target branches in real-world C/C++ programs and ask an agent to predict an input that drives execution to a given branch. This decomposes discovery into three task modes over 22 real-world C/C++ programs spanning 15 domains. Open-loop and Feedback-enabled share 60 fixed-target task instances across 16 of these codebases (13 domains), testing input construction without and with a distance oracle to isolate code comprehension from feedback-driven correction. Online Arena instead removes the predefined target and scores path exploration by coverage gain on a separate, partially overlapping pool of 11 programs; agents collectively confirmed 13 distinct bugs across six programs. Evaluating 15 default-effort model-scaffold configurations, the best reaches only 55.0% pass rate in the Feedback-enabled mode, and the mean across seven paired Claude Code configurations is 36.4% with feedback versus 19.3% without. Decomposing failures, we find constraint inference, not navigation, is the dominant bottleneck. We release SWE-Test with a turnkey evaluation environment.
1 Introduction
SWE-Test reframes vulnerability discovery as input prediction to address benchmark contamination, unknowable vulnerability sets, and opaque end-to-end scores. Its three modes separate code comprehension, feedback-driven correction, and path exploration across real-world programs.
- Motivation: Existing benchmarks can be contaminated, score against unknowable vulnerability sets, and collapse distinct discovery abilities into one outcome.Public tasks can be memorized, predefined vulnerabilities exclude different defects, and end-to-end scores cannot localize failure.
- Motivation: Vulnerability discovery requires code comprehension, input-constraint inference, input construction, execution, and iterative correction from feedback.The paper treats these requirements as a composite ability rather than a single undifferentiated task.
- Benchmark design: SWE-Test mines deep target branches from real programs and asks agents to synthesize inputs that drive execution to specified branches.The formulation uses coverage-guided fuzzing and adds an approximate distance signal for iterative correction.
- Benchmark design: SWE-Test provides Open-loop, Feedback-enabled, and Online Arena modes for code comprehension, feedback correction, and path exploration.Open-loop and Feedback-enabled use fixed targets, while Online Arena removes the predefined target and scores coverage gain.
- Results: 55.0% is the best feedback-enabled pass rate across 15 default-effort model-scaffold configurations.Among seven paired Claude Code configurations, feedback raises mean pass rate from 19.3% to 36.4%.
- Reproducibility: SWE-Test is released with a standardized turnkey environment that provisions tasks, runs configurations, and reports per-task and aggregate scores.The deployment is Harbor-based and containerized.
2 Related Work
Prior vulnerability-discovery benchmarks trade off realism, contamination resistance, and diagnostic resolution. SWE-Test addresses these limitations by framing discovery as input prediction on real C/C++ programs while separating underlying abilities.
- Benchmark landscape: Table 1 compares SWE-Test with CyberGym, CVE-Bench, CyBench, BountyBench, and AgentCyberRange.The comparison covers selected benchmarks spanning increasingly realistic vulnerability-discovery settings.
- Recurring limitations: Public fixed-task benchmarks remain vulnerable to contamination, while transformed or synthetic tasks may not preserve real software’s control-flow depth and input constraints.Live variants delay contamination risk rather than eliminating it.
- Recurring limitations: CVE- and proof-of-concept benchmarks target predefined vulnerabilities, so they do not credit different defects and cannot measure recall over an unknowable vulnerability set.A clear execution target comes at the cost of incomplete discovery coverage.
- Recurring limitations: End-to-end penetration-testing benchmarks aggregate discovery, exploitation, and post-exploitation, obscuring which capability caused failure.The hidden source may be code comprehension, input construction, feedback use, or target selection.
- SWE-Test: SWE-Test reframes vulnerability discovery as constructing an input that exercises a specified decision outcome in real C/C++ code.This formulation decomposes ability into code comprehension, feedback-driven correction, and path exploration.
3 SWE-Test Benchmark
SWE-Test is a benchmark that decomposes vulnerability discovery into fixed-target input prediction with and without feedback, plus self-directed coverage exploration. Its tasks are mined from real fuzzing executions and evaluated through isolated execution, progress rewards, and coverage-based scoring.
- Task Modes: SWE-Test evaluates three complementary abilities through Open-loop, Feedback-enabled, and Online Arena modes.Open-loop emphasizes code comprehension, Feedback-enabled tests feedback-driven correction on shared targets, and Online Arena evaluates self-directed path exploration.
- Task Modes: Open-loop requires constructing an input for an untaken branch without runtime feedback, rewarding increasingly close overlap with the ground-truth execution path.A successful target-branch execution receives 2.0, reaching the containing function receives 1.0, and partial path overlap receives a value between 0.0 and 1.0.
- Task Modes: Feedback-enabled keeps the same targets but adds a distance oracle, enabling construct-execute-observe-revise loops and isolating feedback-driven correction from comprehension.The mode uses the same reward levels and exposes intermediate reward after each verifier query.
- Task Modes: Online Arena removes the target branch and uses coverage reports to guide a long-horizon loop of selecting uncovered branches, generating inputs, and re-measuring.Its reward is the mean normalized gain across regions, lines, branches, and functions relative to the baseline corpus; confirmed bugs are reported separately.
- Construction Pipeline: Coverage-guided fuzzing supplies reachable, deterministic targets by recording newly uncovered branches, call traces, and concrete inputs before isolated evaluation.A hidden construction-time input establishes reachability, while the newly covered status identifies branches absent from the preceding corpus.
- Construction Pipeline: Online Arena starts from a corpus accumulated after 72 hours of fuzzing so agents begin at a difficult frontier rather than on easily reachable branches.The construction rationale is that coverage rises steeply and then flattens, leaving harder branches after saturation.
4 Experimental
The experiments evaluate leaderboard performance, failure bottlenecks, and sensitivity to feedback, reasoning effort, and scaffold choice across complementary vulnerability-discovery tasks. Feedback improves performance across stages, but constraint satisfaction remains the dominant failure mode and measured results depend on configuration.
- Research questions: The study addresses leaderboard performance, bottleneck diagnosis, and sensitivity to reasoning effort and scaffold choice.
- Leaderboard: Qwen3.8-Max leads the Feedback-enabled mode at 55.0% pass rate, while Qwen3.8-Max-Preview records seven confirmed bugs and 0.3136 mean coverage reward in 12-hour Online Arena runs.
- Feedback effects: Feedback raises mean pass rate from 19.3% to 36.4% and improves target-function reach and near-to-pass conversion.Target-function reach rises from 62.6% to 75.2%, while near-to-pass conversion rises from 30.8% to 48.4%.
- Online Arena: 13 distinct confirmed bugs were found across six programs after deduplication in Online Arena, complementing fixed-target pass-rate evaluation.
- Failure diagnosis: Constraint-satisfaction failures are the largest aggregate outcome, although navigation also matters and domain changes the dominant failure mode.SQL-parser tasks show 82% navigation failure, compared with 17% for PHP tasks.
- Configuration sensitivity: Reasoning effort affects performance differently across modes, while scaffold choice changes measured pass rates and cannot be explained by tool volume alone.High effort reaches 38.3% in Feedback-enabled and 18.3% in Open-loop; vendor scaffolds outperform Terminus 2 in both evaluated model comparisons.
5 Discussion and Threats to Validity
SWE-Test separates navigation from branch-constraint satisfaction and shows that feedback improves both, while scaffold and reasoning settings complicate model comparisons. Its fixed-target branch-inversion design provides a deterministic proxy for vulnerability discovery but limits generalization beyond the evaluated programs, languages, and input formats.
- Discussion: Feedback improves both target-function reach and near-to-pass conversion, revealing distinct navigation and constraint-satisfaction bottlenecks.Target-function reach rises from 62.6% to 75.2%, while near-to-pass conversion rises from 30.8% to 48.4%.
- Discussion: Constraint-satisfaction failures remain the largest aggregate outcome, motivating feedback that identifies which guard sub-condition remains unsatisfied.
- Threats to Validity: SWE-Test measures single branch inversion rather than full exploit construction, so its results describe comprehension, correction, and exploration rather than end-to-end exploitation.
- Threats to Validity: The fixed-target benchmark covers 60 tasks across 16 C/C++ OSS-Fuzz targets and 13 domains, limiting transfer to other languages or input formats.Online Arena expands benchmark-wide coverage to 22 programs and 15 domains, but fixed-target findings remain concentrated in a few high-task-count domains.
- Threats to Validity: Scaffold choice, default reasoning effort, and unequal valid-trial counts limit direct model comparisons.For GLM-5.1, reasoning-effort settings shift feedback-enabled pass rate from 25.0% to 38.3%; pass rates exclude unscored trials.
6 Conclusion
SWE-Test evaluates vulnerability-discovery agents through input prediction on real C/C++ programs, separating code comprehension, feedback-driven correction, and path exploration. Feedback improves performance, but scaffold and reasoning choices materially affect measurements, motivating reproducible reporting of where agents fail.
- Conclusion: SWE-Test evaluates vulnerability-discovery agents through input prediction on real C/C++ programs across three modes.
- Conclusion: The benchmark separates code comprehension, feedback-driven correction, and path exploration through Open-loop, Feedback-enabled, and Online Arena modes.
- Conclusion: Feedback raises mean pass rate from 19.3% to 36.4% among seven paired Claude Code configurations and improves target-function reach and target-constraint satisfaction.
- Conclusion: Scaffold choice and reasoning effort materially affect measured performance, making uncontrolled model rankings difficult to interpret.
- Conclusion: The released tasks, Online Arena, and evaluation harness support reproducible analysis of where vulnerability-discovery agents fail.