Source-linked AI summary
On the Reliability of LLM-Based Vulnerability Patching Benchmarks
Dang K Le, Wenxuan Shi, Xinyu Xing
TL;DR
LLM vulnerability-patching benchmarks can substantially distort reported performance, yet their evaluation choices are often under-examined. The paper curates historical bugs with developer tests and uses controlled experiments, case studies, and a benchmark survey to examine those choices. It finds that benchmark scores are highly sensitive to evaluation design and that many surveyed benchmarks leave characterized pitfalls unaddressed.
Problem
Benchmark evaluation often reduces correctness to supplied tests passing, which can conflate maintainer-acceptable patches, symptom suppression, and benchmark exploits.
Source: §2.2 · p. 2Method
The authors curate 112 historical bugs and use controlled experiments, case studies, and a survey of existing benchmarks to examine agent-, framework-, and dataset-level factors.
Source: §1 · pp. 1–2Results
Benchmark scores are highly sensitive to evaluation design, and among five surveyed LLM-agent benchmarks none addresses all nine factors while every identified pitfall appears in at least one benchmark.
Source: §1 · p. 1, §7.4 · pp. 11–12Takeaways & Limitations
The paper provides practical guidelines and a reporting checklist to strengthen the validity, interpretability, and reproducibility of APR results.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Large language models (LLMs) have shown strong potential for automated vulnerability patching, but current benchmarks can substantially distort reported performance. Drawing on extensive experience developing, running, and stress-testing such frameworks, we identify under-examined pitfalls across three dimensions: (1) agent-level factors, where prompting, tool availability, and detailed instructions can raise success rates without improving developer-aligned patch quality; (2) framework-level factors, where permission errors, infrastructure bugs, and timeout handling can silently suppress or inflate performance; and (3) dataset-level factors, where bug reports and single proof-of-concept (PoC) tests fail to capture whether patches address root causes or follow developer intent. We curate 112 historical bugs from 84 open-source C/C++, Go, and Rust projects, each with PoC tests, regression tests, and additional developer tests that assess alignment with the original developers' design principles. Through controlled experiments and case studies, we show that LLMs can achieve high PoC passing rates under ideal conditions, yet benchmark execution choices can materially change measured success. More importantly, developer-test passing rates remain low and improve only marginally with newer models, suggesting that models increasingly suppress symptoms without consistently producing upstream-quality fixes. These results show that benchmark scores are highly sensitive to evaluation design, and we provide practical guidelines for more rigorous, reliable, and reproducible evaluation.
1 Introduction
LLM vulnerability-patching benchmarks can misrepresent model capability because results depend on agent, framework, and dataset choices. The paper examines these factors with a curated dataset and controlled experiments, and offers guidelines for more rigorous evaluation.
1 Introduction
- Benchmark scores are highly sensitive to evaluation design, which can materially misrepresent model capability.The paper argues the benchmarking pipeline itself deserves principled design and critical scrutiny.Source: §1 · p. 1
- The authors identify agent-, framework-, and dataset-level pitfalls, including prompting and tool access, infrastructure failures, and how vulnerabilities and success are operationalized.The taxonomy draws on experience developing and stress-testing patching harnesses across three languages.Source: §1 · p. 1
- The study curates 112 historical bugs across 84 open-source C/C++, Go, and Rust projects, with PoCs, regression tests, and developer tests.Developer tests encode design intent, not merely the absence of a crash.Source: §1 · pp. 1–2
- Three controlled experiments and five case studies examine how benchmark factors shift apparent patch quality.The experiments quantify effects on PoC-pass and developer-test-pass rates, while case studies investigate framework issues.Source: §1 · p. 2
- The paper surveys 13 existing benchmarks and datasets, finding most exhibit at least one characterized pitfall, and provides design guidelines and a reporting checklist.The dataset is intended to highlight evaluation pitfalls under controlled conditions, not to rank models competitively.Source: §1 · p. 2
2 Background and Motivation
LLM patching agents work in buggy repositories and are usually graded by automated tests, often whether a PoC crash stops and reference tests still pass. That narrow correctness criterion can treat maintainer-quality fixes, symptom suppression, and benchmark exploits as equivalent.
2.1 LLM-Based Vulnerability Patching
- Patching agents receive a bug description and buggy repository, sometimes with a PoC or build instructions, then are usually graded by automated oracles.Typical tools include shell and file editing; some agents also have web access.Source: §2.1 · p. 2
2.2 “Just Another Benchmark” Is Not Enough
- Passing supplied tests conflates maintainer-acceptable patches with symptom suppression and patches that exploit benchmark pitfalls.The paper identifies deployment and training as downstream uses where false-positive passes matter.Source: §2.2 · p. 2
3 A Taxonomy of Benchmarking Pitfalls
The paper classifies benchmark pitfalls by whether they arise from the agent, the evaluation framework, or the dataset. Factors can change measured success without changing the model, and their interactions may amplify effects.
3 A Taxonomy of Benchmarking Pitfalls
- A benchmark factor is a pipeline variable that changes measured success without changing the underlying model; factors are systematic when studied through ablations and incidental when exposed by individual failures.The paper analyzes factors in isolation for clarity, while noting real benchmarks may combine interacting factors.Source: §3 · p. 2
3.1 Agent-Level Factors
- Build and test instructions lift PoC-pass rates by about 20% in C/C++ but much less in Go/Rust; web access can expose public fixes verbatim.Build systems are more uniform in Go/Rust, while issue-number searches can retrieve fixes.Source: §3.1 · p. 3
3.2 Framework-Level Factors
- Permission errors, hidden agent behavior, timeouts, or conflicts applying added tests can create failures unrelated to patch quality.Overly broad permissions can expose ground-truth fixes, while restrictive permissions can block legitimate dependency installation.Source: §3.2 · p. 3
3.3 Dataset-Level Factors
- PoC-only oracles can miss developer intent; source-level PoC harnesses yield higher PoC-pass rates than binary PoCs, and factor sensitivity varies by language.The paper proposes developer tests as a targeted middle ground; C/C++ is more sensitive to harness quality, while Go/Rust are more sensitive to PoC framing.Source: §3.3 · p. 3
4 Methodology and Dataset
The paper builds a reproducible infrastructure to isolate evaluation factors and assesses patches with build, PoC, regression, and developer-test checks. Its dataset pairs historical bugs with tests derived from upstream fixes, and uses different agent/model combinations for controlled experiments and case studies.
4 Methodology and Dataset
- The evaluation infrastructure lets benchmark factors be toggled in isolation, using a fixed harness across the experiments and case studies.Prompt components such as build/test instructions and PoC exposure are configuration-controlled.Source: §4 · p. 3, §4.4 · p. 5
4.1 Dataset
- The dataset contains 112 historical bugs from 84 projects across C/C++, Go, and Rust, collected from public issues, security advisories, and commit histories.The language slices cover 10 C/C++ projects, 51 Go projects, and 23 Rust projects.Source: §4.1 · pp. 3–4
- Each case is pinned to trigger and patch commits and packaged in Docker with build and test scripts, PoC inputs, a bug description, and a developer test.Cases were admitted only after the upstream patch built, avoided the PoC crash, and passed regression and developer tests.Source: §4.1 · p. 4
4.2 Developer’s Tests
- Developer tests extracted from upstream fixes check behavior aligned with maintainers’ intent, beyond whether a PoC crash is suppressed.In PHP GH-13094, both the upstream patch and a defensive alternative pass the PoC and existing regressions, but only the upstream patch preserves mixed-input behavior and passes the developer test.Source: §4.2 · pp. 4–5
4.3 Oracle Pipeline
- Patches are evaluated in four phases: build, ten PoC executions, regression tests, then regression tests with the developer test applied.A case is fully resolved only when all four phases pass.Source: §4.3 · p. 5
4.4 Agents and Models
- Controlled experiments use OpenCode with Claude Sonnet 4.0, while case studies use Claude Code with Claude Sonnet 4.5.The first setup supports auditable, tractable full-factor sweeps; the production, closed-source case-study agent exposes less-visible framework mechanics.Source: §4.4 · p. 5
5 Experiments of Systematic Factors
Three controlled experiments hold the dataset constant while varying one agent- or dataset-level factor at a time. Across them, PoC-pass rates often rise more than developer-test-pass rates, exposing a gap between benchmark success and developer-aligned repair.
5.1 Experiment 1: Build/Test Instructions
- Build/test instructions lifted overall PoC-pass from 74.1% to 77.7%, including a 20-point C/C++ gain, while regression-pass stayed at 97.3%.Developer-test-pass changed only slightly, from 41.1% to 43.8%.Source: §5.1 · pp. 5–6, Table 2 · p. 5
5.2 Experiment 2: PoC Framing
- Go source PoCs raised PoC-pass from 69.4% to 95.2% and developer-test-pass from 41.9% to 48.4%; C/C++ binary exposure changed neither rate.In C/C++, PoC-pass was 92.0% in both conditions and developer-test-pass was 36.0% in both; binary exposure produced only a small regression-pass increase, from 88.0% to 92.0%.Source: §5.2 · p. 6, Table 3 · p. 6
5.3 Experiment 3: Model Generations
- PoC-pass rose from 77.7% to 90.2% across Claude generations, while developer-test-pass rose from 43.8% to 60.7% and regression-pass stayed nearly flat.The PoC–developer-test gap narrowed only from 33.9 to 29.5 percentage points; even with Opus 4.7, 32.7% of PoC-passing cases failed the developer’s test.Source: Figure 1 · p. 6, §5.3 · pp. 6–7, Table 4 · p. 6
5.4 Cross-Cutting Observations
- Across 734 case-level outcomes, only one case failed PoC yet passed developer tests; PoC-only patches numbered 38, 41, and 33 across successive model generations.The generations were Sonnet 4.0, Sonnet 4.5, and Opus 4.7, respectively.Source: §5.4 · p. 7
6 Case Studies of Incidental Factors
Five case studies show how incidental framework and dataset decisions can leak answers, slow or mis-score runs, and distort measured patch quality. These factors affect a minority of cases but can systematically bias aggregate results and downstream training.
6.1 Case 1: WebSearch Leakage
- A single WebSearch query exposed the upstream fix for a historical Go bug, and the agent passed both PoC and developer tests by following that fix.The authors recommend disabling web tools by default; this alone is insufficient if shell access still permits curl or git fetch.Source: §6.1 · p. 7
6.2 Case 2: Git History Leakage
- Preserving full git history let an agent retrieve an earlier fix for a regression and reproduce the upstream API change, passing both tests.A shallow clone, tarball, or history-sanitization step closes this channel; disabling web search alone does not.Source: §6.2 · pp. 7–8
6.3 Case 3: Permission Misconfiguration
- Mounting agent inputs and evaluator metadata together exposed the upstream patch and developer tests, enabling edits textually identical to the upstream fix.The proposed remedy separates mounts and runs the agent as a non-root user.Source: §6.3 · p. 8
6.4 Case 4: Hidden Sub-Model Dependency
- Unrouted safety-classification calls added about 120 seconds per Bash command before defaulting to allow, disproportionately burdening cases with many commands.The traffic was invisible in ordinary agent and harness logs; redirecting the safety-check slot to a routed alias reduced overhead.Source: §6.4 · pp. 8–9
6.5 Case 5: Agent-Written Test Collision
- A benign agent-written test overlapped textually with the evaluator’s developer test, so git apply failed and a functionally equivalent patch was scored as a failure.The authors recommend collision-tolerant test application or classifying application failure as a pipeline error.Source: §6.5 · p. 9
6.6 Why “Incidental” Factors Matter
- Individual incidental factors affected about 5% to 32% of cases, and their systematic biases can distort leaderboard rankings and training signals.Incorrect failures can discourage desirable behavior, while leakage-based successes can reinforce shortcut strategies.Source: §6.6 · pp. 9–10
7 Survey of Real-World Benchmarks
A survey of 13 benchmarks and datasets finds pitfalls across dataset, framework, and agent design. No surveyed LLM-agent benchmark addresses all nine factors, and every pitfall identified in the experiments appears in at least one surveyed benchmark.
7.1 Dataset-Level Observations
- PoC-only oracles can accept symptom suppression, PoCs are not documented or controlled by surveyed benchmarks, and security benchmarks with execution infrastructure cover one language family each.Only MultiSWE-bench combines multi-language support with execution infrastructure; broader vulnerability collections lack that infrastructure.Source: §7.1 · p. 10
- Many surveyed datasets retain unsanitized git history, while PatchAgent exposes it through the source-reading copy; repurposing collections can introduce contamination through naive source cloning.Vuln collections may offer broad language coverage, but generally distribute snippets or archives rather than repository checkouts.Source: §7.1 · p. 10
7.2 Framework-Level Observations
- No surveyed benchmark enforces complete application- and network-layer permissions, and only PatchAgent and SWE-bench document timeout thresholds or monitor hidden sub-model calls.SWE-bench and MultiSWE-bench apply test patches with git apply, risking collisions with agent-modified files; running tests on modified trees creates a complementary risk.Source: §7.2 · pp. 10–11
7.3 Agent-Level Observations
- Benchmarks vary in whether build and test instructions reach the agent, and none ablates their score impact despite configuration shifts of up to 20%.Network-level isolation is also absent from the surveyed benchmarks, leaving historical-bug web search as a route to upstream fixes.Source: §7.3 · p. 11
7.4 Summary
- None of the five LLM-agent benchmarks addresses all nine factors; PatchAgent covers the most, while SWE-bench’s strengths and remaining gaps vary across instances.AVR benchmarks and vulnerability collections repurposed for agents require new harnesses, oracles, and isolation, introducing potential confounds.Source: §7.4 · pp. 11–12
8 Guidelines for Reliable Benchmark Design
The authors translate their findings into practical benchmark-design guidelines and a minimal reporting checklist. The recommendations emphasize making evaluation conditions explicit, separating distinct quality signals, and distinguishing pipeline errors from patch failures.
8.1 Guidelines Derived from the Principles
- Disclose exact prompts, tools, and defaults, and use ablations such as reporting results with and without PoCs or build scripts to separate capability from supplied information.The authors advise reporting developer-test results alongside PoC results as a lower-bound signal of alignment with developer intent.Source: §8.1 · p. 12
- Quarantine ground-truth artifacts, automate source-tree sanitization, monitor raw logs for hidden mechanics, and handle benign test collisions without scoring them as patch failures.The authors also recommend publishing failure taxonomies, including wrong builds, test failures, pipeline errors, and timeouts.Source: §8.1 · p. 12
8.2 A Reporting Checklist
- The minimal reporting checklist calls for exact prompts, exposed tools and network scope, source provenance and sanitization, oracle definitions, and failure-log analysis using a taxonomy.The checklist is intended for reports on LLM vulnerability-patching benchmarks.Source: §8.2 · p. 12
9 Discussion and Threats to Validity
The authors bound their claims to controlled comparisons in a small, curated dataset and note that developer tests are imperfect proxies for patch acceptability. Absolute performance also depends on the model, agent, run, and possible training-data contamination.
Scope and external validity
- The 112 cases cover curated C/C++, Go, and Rust projects with reproducible builds, PoCs, and developer tests, limiting generalization to broader bug populations and other language ecosystems.Developer-test passage is evidence of closer maintainer-intent alignment, not proof of absolute correctness.Source: §9 · p. 12
Agent, model, and run dependence
- Absolute rates depend on agent/model choice and run stochasticity, and public historical fixes may occur in training data; the authors emphasize fixed-harness comparisons instead.They view memorization as a possible contributor to headline rates, but not a sufficient explanation for benchmark-sensitivity effects.Source: §9 · p. 13
10 Related Work
Prior work establishes real-repository issue resolution and vulnerability repair as major settings for LLM-based repair, while this paper focuses on how benchmark design affects measured patch reliability.
10 Related Work
- General issue-resolution benchmarks and agent frameworks provide a setting for code-agent evaluation, while this paper surveys existing benchmarks through evaluation pitfalls rather than cataloging them.The cited examples include SWE-bench and multilingual extensions [20], and SWE-agent and Agentless [41] [46].Source: §10 · p. 13
- Vulnerability-repair benchmarks often center evaluation on PoC suppression or existing regression tests, making this work complementary through its focus on benchmark-design effects.Prior efforts use CVEs, advisories, exploit inputs, or sanitizer traces [22] [31].Source: §10 · p. 13
- Prior repair research warns that test-suite-passing patches can remain incorrect or overfit weak oracles, while this work adapts developer-intent tests to agentic vulnerability patching.Related audits also find that apparent issue resolution can diverge from correctness [30] [39].Source: §10 · p. 13
11 Conclusion
The paper concludes that vulnerability-patching benchmark scores can mislead because evaluation choices shape measured success, and PoC passing can outpace developer-intent patch quality. It presents its contribution as methodological and urges treating the evaluation pipeline as part of the research contribution.
11 Conclusion
- PoC passing often overstates repair quality: models improved faster at suppressing benchmark symptoms than at satisfying developer-intent tests.Reported success also depends on prompt design, framework behavior, source packaging, and oracle choice.Source: §11 · p. 13
- The paper contributes a taxonomy, controlled multilingual dataset and harness, experiments, case studies, guidelines, and a reporting checklist for distinguishing model capability from evaluation-pipeline effects.The guidelines aim to support more reliable benchmark interpretation.Source: §11 · p. 13
- Credible claims and training signals require treating the evaluation pipeline as part of the contribution, because what a benchmark measures matters at least as much as its score.Source: §11 · p. 13
A Infrastructure Details
- The infrastructure artifact includes per-language base images, per-case container recipes, agent images, a LiteLLM proxy, and scripts for case execution and ground-truth validation.The appendix describes this as the full artifact for the evaluation harness.Source: A Infrastructure Details · p. 17
A.1 Container structure and isolation
- The default container isolates the agent as an unprivileged user, separates ground-truth artifacts from the source tree, and gives the evaluator access after the agent exits.The harness can relax these defaults for controlled source-leakage studies.Source: §A.1 · p. 17
A.2 Agent configuration
- A parameterized prompt and LiteLLM endpoint support independent toggles for build/test instructions and PoC exposure, as well as model substitution across providers.Agent settings restrict tool permissions and disable web search by default; runs use 1800-second and 500-turn budgets unless stated otherwise.Source: §A.2 · p. 17
A.3 LiteLLM, networking, and resource limits
- The LiteLLM proxy unifies provider traffic and preserves raw request logs, while Docker containers expose the proxy but block public-Internet access by default.Shell-based network access remains blocked unless the harness is reconfigured.Source: §A.3 · p. 17
A.4 Evaluation and ground-truth validation
The evaluator runs a staged test pipeline after patching, and dataset cases are retained only when the upstream patch passes that pipeline.
A.4 Evaluation and ground-truth validation
- After the agent exits, evaluation rebuilds the project and runs PoC tests, the existing regression suite, and developer tests in sequence.Source: §A.4 · p. 17
- Each evaluation phase is timeout-bounded, and failures to apply additional test patches are recorded as evaluation-phase failures rather than evaluator crashes.Source: §A.4 · p. 17
- A case enters the dataset only if the upstream developer patch passes all four evaluation phases on a fresh checkout.The ground-truth validator uses a separate agent-free setup and the same pipeline.Source: §A.4 · p. 17
B Dataset Inventory
The dataset inventory lists 112 vulnerability cases across C/C++, Go, and Rust, with case identifiers and source, PoC, trigger, and patch information.
B Dataset Inventory
- The inventory contains 112 cases: 25 C/C++, 62 Go, and 25 Rust.Source: B Dataset Inventory · p. 18, Table 6 · p. 18
- Inventory columns identify each case and language, its primary upstream bug source, the PoC representation, and the trigger and patch commits.PoC formats include source-level harnesses, textual inputs, and binary blobs.Source: B Dataset Inventory · p. 18, Table 6 · p. 18