Source-linked AI summary
PatchBench: Evaluating AI Agents for Vulnerability Patching
Chihao Shen, Jiacheng Li, Aastha Mahajan, Jeffery Siyuan Tian, Yonghwi Kwon, Yizheng Chen
TL;DR
Existing vulnerability-patching evaluations may reward memorized historical patches or surface-level crash suppression rather than robust repair. The paper introduces PatchBench, which uses transformed vulnerability contexts and security-plus-semantic validation to address these threats. Across 11 agents, PoC-only validation inflates average solve rates by 1.83×, revealing substantial limitations in current patching agents.
Problem
Existing evaluations may test memorized historical patches and accept surface-level fixes that suppress a reported crash without preserving benign functionality.
Method
PatchBench uses vulnerabilities whose fixes lie outside crash-stack functions, transplants and mutates historical vulnerabilities, and validates patches for both security and semantic correctness.
Results
1.83×: PoC-only validation inflates average solve rates across 11 state-of-the-art agents.
Takeaways & Limitations
The results identify memorization, crash-stack shortcuts, and failures to preserve functional correctness as key limitations of current patching agents.
Takeaways & Limitations
The memorization study cannot identify the actual training data of proprietary models, and semantic validation may depend on a reference-patched repository during deployment.
Abstract
from arXiv · showhide
AI agents have recently demonstrated strong performance in automated vulnerability patching. However, existing evaluations often validate a patch only by testing whether the provided Proof-of-Concept (PoC) input still triggers a crash. This leaves two key threats to validity: agents may reproduce memorized historical developer patches, or they may generate surface-level fixes that only suppress the reported crash. We study these concerns for C/C++ vulnerability patching. We introduce a patch similarity metric to detect memorized patches. On average, 25% of the agent patches exhibit substantial similarity to historical developer patches, indicating that patch memorization is a real threat to the validity of vulnerability patching evaluations. Meanwhile, agents also frequently exploit benchmark structures to pass patch validation by patching on the crash stack trace to suppress the crash, rather than localizing and fixing the root cause of the vulnerabilities. To handle these issues, we propose PatchBench, a new benchmark for evaluating AI agents on realistic vulnerability patching tasks. PatchBench selects vulnerabilities whose ground-truth fixes lie outside the crash stack and uses vulnerability transplant and code mutations to migrate historical vulnerabilities into new repository contexts, reducing the risks of surface-level fixes and patch memorization. We develop new patch validation methods that thoroughly evaluate both security and semantic correctness of agent patches. Across 11 state-of-the-art agents, including the top three AIxCC agents, the original PoC-only validation inflates the patching task solve rate of agents by 1.83$\times$ on average. Our results reveal key limitations of current patching agents and point to future research directions for more reliable vulnerability repair.
1 Introduction
Automated vulnerability patching evaluations can overstate agent capability because agents may reproduce memorized historical patches or suppress reported crashes without fixing root causes. PatchBench addresses these threats with benchmark construction and validation methods designed to assess localization, security, and semantic correctness.
- Public-vulnerability benchmarks may test memorized developer patches rather than agents’ analysis and patching capabilities.
- Prior PoC-focused validation can accept incomplete fixes, crash suppression, or patches that remove vulnerable functionality while breaking benign behavior.
- PatchBench includes 213 C/C++ vulnerability-patching tasks across 16 CWEs and 32 real-world projects.
- PatchBench selects vulnerabilities whose ground-truth fixes lie outside crash-stack functions, transplants historical vulnerabilities, mutates patch-site code, and manually curates reference patches.
- 1.83×: PoC-only validation inflates average solve rates across 11 state-of-the-art agents, while the top three pass over 97% of original PoCs but solve roughly half under Security + Semantic Validation.
- PatchBench combines security validation with semantic checks of benign-input sanitizer behavior, reference-patched outputs, and working unit tests.
2 Motivation
SEC-BENCH's high pass rates can reflect memorized developer fixes and crash-suppressing patches rather than robust vulnerability repair. The mruby example shows why PoC-only validation can miss root-cause defects and malformed behavior.
- Benchmark concerns: 97.3% of Codex with GPT-5.6 Sol tasks pass under SEC-BENCH's original setup, raising concerns about memorization and shallow fixes.The benchmark uses public vulnerabilities, while agents may reproduce historical patches or pass by suppressing the reported crash.
- Case study: The mruby vulnerability originates in compiler-side argument packing, but the sanitizer observes only a later VM-side out-of-bounds read.The developer patch changes the compiler condition, connecting the repair to the malformed state before VM execution.
- Case study: The agent's VM-side array check stops the reported crash without repairing the compiler's logical bug.The patch can leave malformed outputs and does not address the root cause.
- Validation issue: 81% of Codex + GPT-5.6 Sol patches modify crash-stack functions, including 64% of patches on tasks whose developer fixes lie outside the stack.These patches can pass the PoC while bypassing vulnerability localization and leaving the root cause unfixed.
- PatchBench motivation: PatchBench selects vulnerabilities whose ground-truth patches lie outside the crash stack and tests related crashes plus benign behavior.This design targets both crash suppression and preservation of expected functionality.
3 Patch Memorization
The paper defines patch memorization as reproducing historical developer fixes and introduces DiffBLEU to measure similarity using patch diffs and surrounding code context. Repository-level agents show substantially more high-similarity patches than standalone models, motivating transformed benchmark tasks.
- Definition: Patch memorization measures whether generated patches reproduce historical developer fixes, providing patch-level evidence of possible data contamination.The paper explicitly does not claim to measure every form of training-data contamination.
- DiffBLEU: DiffBLEU compares developer and agent patch hunks with surrounding code context, combining surface, control-flow, and data-flow similarity.Its tokenizer preserves edit operations while context matching captures structural similarity.
- Detection setup: The study uses a DiffBLEU threshold above 0.75, with hyperparameters selected to yield a 1.1% false omission rate and no false positives on validation data.The metric is used to detect memorization, not to establish patch correctness.
- Memorization study: At threshold 0.75, high-similarity patches increase from 8.3% to 22.0% for GPT-5.6 Sol with Codex.Comparable increases occur for Claude Opus 4.8 with Claude Code and Gemini 3.5 Flash with OpenHands.
4 Benchmark
PATCHBENCH is a C/C++ repository-level benchmark designed to reduce patch memorization and surface-level fixes while evaluating both vulnerability removal and behavioral preservation. It uses off-stack vulnerabilities, transplanted and mutated code, curated reference patches, and security-plus-semantic validation.
- Benchmark scope: PATCHBENCH contains 213 C/C++ patching tasks from 32 GitHub projects across 16 CWE types in realistic Docker-based repair environments.Each task includes the repository, toolchain, triggering PoC, and compilation and execution commands.
- Benchmark construction: PATCHBENCH mitigates patch memorization through vulnerability transplant and code mutation, scaling this mitigation to 213 tasks.Historical vulnerabilities are moved to newer project versions and relevant code is transformed so memorized historical patches no longer directly apply.
- Patch validation: PATCHBENCH validates patches for both security and semantic correctness rather than relying only on a reported PoC.Security validation uses crashing inputs, while semantic validation checks benign-input sanitizer behavior, program-level outputs, and reference-repository unit tests.
- Benchmark construction: The benchmark selects vulnerabilities whose developer patch sites fall off the sanitizer crash trace, requiring agents to reason beyond downstream crash locations.Trace overlap uses Jaccard similarity between patch-reaching and crash traces; selected tasks require low overlap.
- Patch validation: Reference patches are manually curated to fix each vulnerability’s root cause, and tasks are retained only when fuzzing and unit-test validation support the procedure.Tasks are discarded when the developer patch is orthogonal to the root cause or when required validation infrastructure is unavailable.
5 Evaluation
Across 11 agents, PoC-only validation substantially overestimates successful patching: it inflates solve rates by 1.83×, while semantic checks expose behavior changes and recurring crash-suppression strategies. Increasing budgets yields only modest gains, and AIxCC systems underperform comparable general-purpose agents.
- Overall performance: 1.83×: PoC-only validation inflates average solve rates, with 83.1% of patches passing the original PoC but only 45.3% passing security and semantic validation.The strongest agents pass over 97% of original PoCs yet solve only about half of tasks.
- Overall performance: PoC-only validation can misrank agents: similar PoC pass rates correspond to solved rates spanning 17 points.OpenHands + GPT-5 ranks eighth by solved rate despite the fourth-highest PoC pass rate, while OpenHands + Gemini 3.5 Flash solves more with a lower PoC pass rate.
- Semantic validation: 63.4%: agents pass semantic validation on average, with output-state and unit-test checks weaker than sanitizer regression checks.Average pass rates are 95.7% for sanitizer regression, 75.5% for output state, and 79.9% for unit tests.
- Agent comparison: AIxCC systems underperform general-purpose agents using the same model, with Buttercup and RoboDuck solving 42.7% and 29.6% of tasks.Atlantis performs best among the three AIxCC systems at 48.4%, still below all three general-purpose leaders.
- Budget analysis: Increasing the task budget produces quickly plateauing solve rates while reducing budget exhaustion.Codex + GPT-5.6 Sol rises from 59.2% at $5 to 61.5% at $15, with no further gains at $20 or $25.
- Failure patterns: The most common failure is an incomplete local guard around sanitizer-reported locations, accounting for 41 of 81 Codex + GPT-5.6 Sol patches that pass the original PoC but fail validation.Such checks block the reported path without covering other crashing inputs reaching the same vulnerability.
- Failure patterns: Other recurring failures include changing downstream code without localizing the vulnerability, deleting unsafe operations or entire features, and rejecting benign inputs.These strategies can suppress the crash while preserving malformed behavior or breaking intended functionality.
- Validation ablation: Removing the output-state check increases overall solved rate by 8.1 points, showing that this check rejects patches that pass the remaining validation checks.The ablation isolates the output-state check as a major difference from crash-oriented validation.
6 Discussion and Limitation
The discussion highlights limitations in attributing memorization and in validating patches without a reference-patched repository, while noting residual root-cause errors even among validated patches. The paper also contributes dynamic task construction from developer-discovered vulnerabilities.
- Limitations: Memorization analysis cannot identify proprietary models’ actual training data, so it measures patch similarity rather than proving specific memorization sources.Its purpose is to show that models may reproduce historical patches without reasoning about vulnerability root causes.
- Limitations: Without a reference-patched repository, semantic validation can compare vulnerable and agent-patched program outputs, but this weaker reference may miss behavior intentionally changed by a correct patch.The authors still value reference-based validation because the strongest agents solve only about half the benchmark tasks.
- Residual errors: Manual review found that 6.8% of Atlantis patches and 7.1% of Codex patches passing validation still leave the vulnerability’s root cause partially unresolved.
- Contribution: The paper proposes dynamically constructing new patching tasks from vulnerabilities discovered by developers.
7 Related Work
Prior work includes general bug-repair benchmarks and repository-level security vulnerability benchmarks, but studies have identified contamination and patch-memorization concerns in popular datasets. AUTOPATCHBENCH extends evaluation toward AI-agent vulnerability repair.
- Bug-repair benchmarks: General bug-repair benchmarks such as Defects4J and GitBug-Java are widely used, but recent studies report data contamination and patch-memorization issues.
- Security benchmarks: Security vulnerability patching benchmarks provide repository-level evaluation frameworks with executable validation commands.
- AI-agent repair: AUTOPATCHBENCH further studies AI-agent vulnerability repair with different configurations.
8 Conclusion
The paper presents PATCHBENCH and systematically evaluates AI agents for security vulnerability patching, covering commercial and top-performing AIxCC agents with rigorous validation.
- PATCHBENCH evaluates commercial patching agents and top-performing AIxCC agents using rigorous patch validation techniques.
A.1 Vulnerability Transplant
The vulnerability-transplant procedure searches repository history for the latest newer commit where the reversed vulnerability-inducing diff applies, compiles, reproduces the sanitizer error, and is fixed by the developer patch.
- A candidate commit is valid when the induced vulnerability compiles, the PoC reproduces the sanitizer error, and the developer patch eliminates the crash.
- The latest valid candidate becomes the final benchmark task after searching newer or older history based on each candidate's validity.
- The transplant process applies reversed developer security-fix hunks to newer commits using progressively reduced context when unique matches are unavailable.
A.2 Example of an Incorrect Developer Patch
An ARVO task incorrectly treats a patch that suppresses the PoC by changing fuzzer input grammar as a fix for a heap buffer overflow.
- The patch changes the fuzzer's input grammar by deleting a macro that changes an enum width, rather than fixing the heap buffer overflow's root cause.
- Although the patch passes standard vulnerable-to-fixed verification, it is incorrect because it only suppresses the PoC.
A.3 Initial Fuzzing
Initial fuzzing builds a mutation-driven corpus from the vulnerable repository while using the developer-patched repository as a stable corpus-construction target, then continues after sanitizer crashes to collect related inputs.
- The fuzzer uses the vulnerable repository for fuzzing because it provides more useful feedback around the original vulnerability.
- The initial corpus is capped at 10 inputs and constructed mainly through mutation rather than coverage feedback.
- Removing the output state check from semantic validation increases the overall solved rate, with failures appearing across agent systems.
- The fuzzer continues after sanitizer crashes to collect multiple related crashing inputs instead of terminating at the first failure.
A.4 Output State Check
PATCHBENCH’s semantic validation compares agent patches against expected benign-input behavior, not only sanitizer outcomes. Output-state checking is the weakest semantic component on average, and removing it can substantially increase solved rates.
- Output-state validation: Semantic validation records harness output states, including decoded values, parsed objects, generated files, serialized data, or API return values.For file-system harnesses, validation compares produced file contents rather than return status.
- Output-state validation: Normalization wrappers rewrite benign corpus seeds into the smallest syntactically valid inputs required by task-specific parsers.For example, RDKit MOL-stream inputs are reformatted because raw fuzzing seeds would otherwise trigger failure sentinels consistently.
- Validation results: 63.4% is the average semantic validation pass rate, with output-state checks averaging 75.5% and sanitizer regression checks 95.7%.The component breakdown shows that output-state validation is the lowest-pass-rate check among the three reported components.
- Validation results: Semantic validation separates patches that merely remove crashes from patches that preserve expected behavior on benign inputs.This distinction is the stated purpose of checking semantic behavior in addition to sanitizer regression.
- Validation results: Removing the output-state check increases solved rates by 4.7 to 11.3 percentage points across the reported agents.RoboDuck has the smallest increase at 4.7 points, while OpenHands + GPT-5 has the largest at 11.3 points.