PaperlayerView exact revision on arXiv

Source-linked AI summary

PatchBench: Evaluating AI Agents for Vulnerability Patching

Chihao Shen, Jiacheng Li, Aastha Mahajan, Jeffery Siyuan Tian, Yonghwi Kwon, Yizheng Chen

arXiv:2609.04075v1cs.CRcs.AIcs.SE

TL;DR

PoC-only checks do not establish whether patches fix vulnerability root causes, preserve benign behavior, or avoid reproducing historical patches. PatchBench combines benchmark construction and security-plus-semantic validation; across 11 agents, PoC-only validation inflates solve rates by 1.83× on average.

  • Problem

    Existing PoC-based evaluations do not establish whether agents derive fixes rather than reproduce public patches or whether patches preserve benign functionality.

    Source: §2 · p. 3

  • Method

    PatchBench selects vulnerabilities whose ground-truth fixes lie outside the crash stack, transplants and mutates historical vulnerabilities, and validates security and semantic correctness.

    Source: §1 · pp. 2–3

  • Results

    PoC-only validation inflates solve rates by 1.83× on average across 11 agents, while the top three pass over 97% of original PoCs but solve roughly half under Security + Semantic Validation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Patch memorization remains a critical challenge for evaluating AI vulnerability patching, and current state-of-the-art agents continue to face technical limitations.

    Source: §1 · p. 2, §8 · p. 14

    The memorization study cannot identify proprietary models’ actual training data, and semantic validation may lack a reference-patched repository during deployment.

    Source: §6 · p. 13

Abstract

from arXiv · show

AI agents have recently demonstrated strong performance in automated vulnerability patching. However, existing evaluations often validate a patch only by testing whether the provided Proof-of-Concept (PoC) input still triggers a crash. This leaves two key threats to validity: agents may reproduce memorized historical developer patches, or they may generate surface-level fixes that only suppress the reported crash. We study these concerns for C/C++ vulnerability patching. We introduce a patch similarity metric to detect memorized patches. On average, 25% of the agent patches exhibit substantial similarity to historical developer patches, indicating that patch memorization is a real threat to the validity of vulnerability patching evaluations. Meanwhile, agents also frequently exploit benchmark structures to pass patch validation by patching on the crash stack trace to suppress the crash, rather than localizing and fixing the root cause of the vulnerabilities. To handle these issues, we propose PatchBench, a new benchmark for evaluating AI agents on realistic vulnerability patching tasks. PatchBench selects vulnerabilities whose ground-truth fixes lie outside the crash stack and uses vulnerability transplant and code mutations to migrate historical vulnerabilities into new repository contexts, reducing the risks of surface-level fixes and patch memorization. We develop new patch validation methods that thoroughly evaluate both security and semantic correctness of agent patches. Across 11 state-of-the-art agents, including the top three AIxCC agents, the original PoC-only validation inflates the patching task solve rate of agents by 1.83$\times$ on average. Our results reveal key limitations of current patching agents and point to future research directions for more reliable vulnerability repair.

1 Introduction

This paper examines whether AI vulnerability-patching evaluations measure agents’ repair ability or instead reward memorized patches and fixes that merely suppress reported crashes. It introduces PATCHBENCH and broader security and semantic validation to address these concerns.

Patch memorization

  • On SEC-BENCH’s 300 C/C++ tasks, the new similarity metric finds an average 25% of agent patches extremely similar to or identical to historical developer fixes.The study compares local-context LLM patching with repository-level agent patching and reports that Codex, Claude Code, and OpenHands have a higher proportion of likely memorized patches than the underlying LLM alone.Source: §1 · p. 2

Crash-stack patching

  • 81% of Codex + GPT-5.6 Sol patches modify crash-stack functions, including 64% of patches even when the historical fix places the root cause elsewhere.The authors warn that such patches may suppress a single PoC crash without fixing the vulnerability’s root cause.Source: §1 · p. 2

PATCHBENCH

  • PATCHBENCH includes 213 tasks across 16 CWEs and 32 real-world projects, transplanting and mutating historical vulnerabilities into newer repository versions to blunt memorization.It selects tasks whose ground-truth fixes lie outside the crash stack and manually audits reference patches, discarding fixes that leave root causes and removing unrelated changes.Source: §1 · p. 2

Patch validation

  • Security validation tests fuzz-generated crashing inputs, while semantic validation checks benign-input sanitizer behavior, output equivalence to reference patches, and all working unit tests.Prior automatic validation can miss incomplete fixes or patches that remove vulnerable functionality; one reported study found 16% to 38% of patches semantically incorrect despite passing automatic validation [70].Source: §1 · pp. 1–2

Results

  • PoC-only validation inflates solve rates by 1.83× on average; the top three agents pass over 97% of original PoCs but solve roughly half under security-plus-semantic validation.The evaluation covers 11 state-of-the-art agents.Source: §1 · p. 2, Figure 1 · p. 2

2 Motivation

The motivation is that SEC-BENCH’s high PoC-pass rate may reflect memorized fixes or patches that suppress crashes without repairing vulnerabilities or preserving benign behavior. The section illustrates both risks and motivates patch-specific similarity detection and more thorough validation.

2 Motivation

  • SEC-BENCH contains 300 realistic C/C++ repair tasks, and Codex with GPT-5.6 Sol passes 97.3% under a $5-per-task budget, raising concerns about memorization and PoC-only fixes.Because the tasks derive from public vulnerabilities, agents may reproduce historical fixes; passing one PoC may also suppress a crash or break benign functionality.Source: §2 · p. 3

2.1 Data Contamination Issue

  • A Codex patch reproduces a developer patch’s guard, location, control flow, and return value despite changed messages, a comment, and a loop bound, illustrating potential memorization.Token-level comparison can obscure such similarity, while generic text-similarity metrics do not model multiple patch hunks, motivating a patch-specific tokenizer and similarity method.Source: §2.1 · p. 3

2.2 Patch Validation Issue

  • In mruby, faulty compiler-side argument packing causes malformed arguments, while the agent’s VM-side crash check leaves other OOB paths and silent malformed behavior unfixed.The compiler introduces the malformed state before VM execution; the agent’s guard can stop the reported crash yet leave incorrect outputs for non-crashing inputs.Source: §2.2 · p. 4
  • 81% of Codex + GPT-5.6 Sol patches modify a crash-stack function; among 92 tasks with developer fixes outside the stack, 64% do so while passing the PoC.This suggests that single-PoC validation can reward crash suppression without vulnerability localization.Source: §2.2 · pp. 4–5
  • PATCHBENCH addresses these risks by selecting tasks whose developer fixes lie outside the crash stack and validating with related crashing and benign inputs.The design aims to encourage broader vulnerability reasoning, test whether sanitizer errors are eliminated, and check preservation of benign behavior.Source: §2.2 · p. 5

3 Patch Memorization

The section measures one concrete form of code-generation data contamination: whether agents reproduce historical developer fixes. It introduces DiffBLEU to compare patches and reports that repository-level agents produce patches more similar to developer fixes than standalone LLMs.

3.1 Detection Methods

  • Token-only metrics can miss syntax and edit direction, while whole-function comparisons dilute matches; patch similarity should compare changed hunks with relevant local context.Patch meaning can depend on nearby control and data flow, including branches, loop conditions, variable definitions, and macros.Source: §3.1 · p. 5

3.2 DiffBLEU

  • DiffBLEU compares patch hunks with diff-aware token metrics and surrounding code with AST and data-flow similarity, capturing surface edits and program structure.Its tokenizer separates added from removed lines and normalizes code-irrelevant noise; its context slices capture control flow and variable dependencies.Source: §3.2 · p. 5
  • The study flags DiffBLEU > 0.75 as memorized; validation yielded 1.1% false omissions and no false positives, while similarity alone does not establish correctness.The score combines diff-hunk surface similarity with AST and data-flow similarity over patch context.Source: §3.2 · pp. 5–6

3.3 Memorization Study

  • At threshold 0.75, estimated memorized patches rose from 11% for local-context LLMs to 25% for repository-level agents, roughly one task in four.The authors identify patch memorization as a benchmark threat and use vulnerability transplant and patch-site code mutations to reduce opportunities to reproduce developer fixes.Source: §3.3 · p. 6

4 Benchmark

PatchBench evaluates C/C++ repository-level vulnerability repairs using tasks designed to reduce crash-site shortcuts and patch memorization, alongside security and semantic validation.

4.1 Overview

  • PATCHBENCH contains 213 tasks from 32 GitHub projects across 16 CWE types, with Docker-based repository repair environments and more comprehensive memorization mitigation and patch validation than prior benchmarks.Source: §4.1 · pp. 6–7

4.2 Benchmark Construction

  • Tasks are selected when developer patch sites lie off the sanitizer crash trace, with trace overlap ρ at most 1/2, requiring reasoning beyond the crash stack.Source: §4.2 · pp. 7–8
  • To limit patch memorization, vulnerabilities are transplanted into newer project versions, code is mutated, and manual reference patches are curated to fix root causes.Source: §4.2 · p. 8

4.3 Patch Validation

  • Semantic validation compares against the reference-patched repository on benign inputs, checking sanitizer regressions, equivalent program output states, and project unit tests.Source: §4.3 · p. 9
  • Security validation checks PoC variants from directed and undirected fuzzing, rejecting agent patches that still trigger sanitizer errors on any crashing-corpus input.Source: §4.3 · p. 9
  • The top three agents pass over 97% of original PoCs but solve only about half of PATCHBENCH after security and semantic validation.Source: Table 2 · p. 10

5 Evaluation

Across 11 agents, PoC-only validation substantially overstates successful patching, while semantic checks expose behavior changes that crash suppression misses. Failure analysis identifies recurring inadequate fixes, and an ablation shows that output-state validation catches incorrect patches missed by other checks.

5.2 Results

  • 83.1% of patches suppress the original PoC crash, but only 45.3% of tasks pass security and semantic validation, an average 1.83× inflation.Even the strongest agents pass over 97% of original PoCs but solve only 56.8%–59.2% of tasks.Source: §5.2 · p. 10
  • PoC pass rates can misrank agents: four agents score 96.2%–98.1% on PoCs while their solved rates span 42.3%–59.2%.OpenHands + GPT-5 ranks eighth by solved rate despite the fourth-highest PoC pass rate; OpenHands + Gemini 3.5 Flash solves more tasks despite a lower PoC pass rate.Source: §5.2 · p. 11
  • Agents pass only 63.4% of semantic validation on average; output-state and unit-test checks pass at 75.5% and 79.9%, respectively.Failures include malformed outputs, wrong decoded values, changed generated files, and unexpected API results, indicating changes to intended program behavior.Source: §5.2 · p. 11
  • AIxCC systems solve fewer tasks than general-purpose agents using the same model: Buttercup solves 42.7%, RoboDuck 29.6%, and Atlantis 48.4%.Reported framework constraints include failed code retrieval, premature run termination, and fixed workflows that can waste budget or restrict code exploration.Source: §5.2 · pp. 11–12
  • Increasing the budget yields modest gains that plateau: Codex + GPT-5.6 Sol rises from 59.2% solved at $5 to 61.5% at $15, with no further gains at $20 or $25.At $25, no agent is cut off by the budget on more than 1.4% of tasks, while many tasks remain unsolved.Source: Figure 6 · p. 11, §5.2 · p. 12

5.3 Failure Pattern Analysis

  • The most common failure is an incomplete local check near the reported crash, which blocks the original PoC but leaves other paths to the same vulnerability open.This pattern accounts for 41 of 81 Codex + GPT-5.6 Sol patches that pass the original PoC but fail validation; similar fractions occur for Claude Code (47/87) and OpenHands + Gemini 3.5 Flash (39/72).Source: §5.3 · p. 12
  • Other failures include downstream rewrites that alter benign behavior, deleting unsafe code along with needed functionality, and checks broad enough to reject valid inputs.In one case, rewriting a loop prevents overflow but changes mesh layout and face count; another patch avoids overflow by deleting TLS metadata extraction.Source: §5.3 · p. 12

5.4 Ablations

  • Removing the output-state check raises solved rate by 8.1 points on average, and manual review found all 18 Codex patches failing only that check still failed to fix the vulnerability.Of those patches, 15 matched failure Pattern 2 and three matched Pattern 4; none were invalid cases caused by checker artifacts.Source: §5.4 · pp. 12–13

6 Discussion and Limitation

The discussion qualifies memorization attribution and semantic validation, identifies residual root-cause failures and benchmark contamination as limitations, and describes ways to address them. It also notes that reference-patched repositories can support stronger patch validation when available.

Validation Gap

  • Manual review found 6.8% of Atlantis and 7.1% of Codex patches passing validation still partially unfixed root causes, due to invasive changes beyond validation inputs.Validation depends on the input corpus, and fuzzing cannot guarantee coverage of all relevant paths; symbolic or concolic execution could generate inputs reaching agent-edited and developer-patched paths.Source: §6 · p. 13

Memorization Attribution

  • The memorization study raises awareness that models may learn historical bug patches and emit similar fixes without reasoning about vulnerability root causes, rather than identifying proprietary training data.The study does not aim to determine exactly what proprietary models memorized.Source: §6 · p. 13

Reference Dependency

  • Without a reference-patched repository, semantic validation compares program-level output states but cannot account for behavior a correct patch intentionally changes.Because the best agents solve only about half of benchmark tasks, reference-patched repositories remain valuable for validation and developing stronger agents.Source: §6 · p. 13

Benchmark Contamination

  • PATCHBENCH is static and risks future training-data contamination, but continued vulnerability transplantation and mutation can create new tasks from vulnerabilities developers discover.These methods can dynamically construct patching tasks for evaluating AI agents.Source: §6 · p. 13

7 Related Work

The related work spans vulnerability-patching benchmarks, interactive repair agents, and patch-validation methods. The paper positions its vulnerability transplant and mutation approach and its security and semantic validation methods against limitations in those areas.

Vulnerability Patching Benchmarks

  • Existing benchmarks range from security-patching suites to synthetic or injected vulnerabilities, but lack thorough validation of patch correctness.General benchmarks Defects4J and GitBug-Java [49] face contamination and memorization concerns; AUTOPATCHBENCH explores fuzzing and differential testing, AIxCC [70] includes 40 expert-written synthetic vulnerabilities across 14 C projects, FIXREVERTER reverts conditional patterns, and Magma [14] forward-ports bugs. PatchBench's transplant and mutation diversify vulnerability contexts to influence LLM memorization behavior.Source: §7 · p. 13

Vulnerability Patching Agents

  • Interactive patching agents have evolved from single-shot generation to repository exploration, editing, and terminal-command workflows, while security agents combine LLMs with security tools.AIxCC entrants Atlantis and Buttercup use an ensemble of LLM agents.Source: §7 · p. 13

Patch Validation Methods

  • Patch evaluation uses static or dynamic analysis, PoCs, unit tests, LLM-as-a-judge, post-patch fuzzing, and manual validation, among other approaches.Drawing on SPIDER and VeriBin, the paper adapts safe-patch ideas that were too restrictive for valid vulnerability patches into security and semantic validation methods.Source: §7 · p. 13

8 Conclusion

The paper presents PATCHBENCH and a systematic study of AI agents for security vulnerability patching. Its evaluation covers commercial and top-performing AIxCC agents using rigorous patch validation, identifies technical limitations, and highlights patch memorization as an issue for future research.

8 Conclusion

  • PATCHBENCH supports a systematic study of AI agents for security vulnerability patching.The evaluation covers commercial patching agents and top-performing AIxCC agents, using rigorous patch validation techniques.Source: §8 · p. 14
  • The study identifies technical limitations that current state-of-the-art vulnerability patching agents continue to face.Source: §8 · p. 14
  • The authors aim to raise awareness of security patch memorization in large language models and inform future research on more capable vulnerability patching agents.Source: §8 · p. 14

A Appendix · A.1 Vulnerability Transplant

Vulnerability transplant moves a security bug into a newer project version, then validates candidate commits to select the latest usable benchmark task. A context-aware matcher applies the reversed security fix and discards cases it cannot transplant.

A.1 Vulnerability Transplant

  • The latest valid candidate commit becomes the final benchmark task.A candidate is valid only if the inducing diff compiles, the PoC triggers the same sanitizer error, and the developer patch eliminates the crash.Source: §A.1 · p. 17
  • A candidate must compile with the vulnerability-inducing diff and reproduce the same sanitizer error under the PoC.The developer patch must also eliminate the crash for the candidate to be valid.Source: §A.1 · p. 17
  • If a candidate is valid, the search continues in newer history; otherwise, it searches older history until no newer valid commit exists.Source: §A.1 · p. 17
  • The transplant procedure adapts Aider’s searchreplace tool and uses Git commits to select and manage newer project versions.Source: §A.1 · p. 17
  • The procedure generates the vulnerability-inducing diff by reversing the developers’ security-fix diff.Source: §A.1 · p. 18
  • A matcher applies the reversed diff hunk by hunk, first requiring a unique match for the pre-edit block with complete context, then shrinking context line by line if needed.Cases are discarded if no match is found in any newer project version.Source: §A.1 · p. 18

A.2 Example of an Incorrect Developer Patch

Task 11498 in the ARVO dataset [34] illustrates an incorrect developer patch: it suppresses the PoC by changing the fuzzer’s input grammar rather than fixing the heap buffer overflow’s root cause.

A.2 Example of an Incorrect Developer Patch

  • The ARVO-identified fix for task 11498 changes the fuzzer’s input grammar by deleting a macro that defines an enum’s width, rather than fixing the heap buffer overflow.The example appears in Figure 8.Source: §A.2 · p. 18
  • The patch suppresses the PoC but does not fix the vulnerability’s root cause.Source: §A.2 · p. 18
  • Despite passing standard vulnerable-to-fixed verification, the patch is incorrect.Source: §A.2 · p. 18

A.3 Initial Fuzzing

Initial fuzzing uses an expanded PoC corpus generated on the stable developer-patched target, then fuzzes the sanitizer-instrumented vulnerable repository while continuing past sanitizer crashes.

A.3 Initial Fuzzing

  • The authors fuzz the sanitizer-instrumented vulnerable repository, but expand the PoC first because a single seed can terminate fuzzing or behave like an empty corpus.The vulnerable version gives the fuzzer more useful feedback around the original vulnerability.Source: §A.3 · p. 18
  • To construct the corpus, they run the fuzzer on the developer-patched repository from the PoC, reducing feedback-guided exploration so mutation drives corpus growth.They use noninstrumentation-guided mutation options -n and -x when supported.Source: §A.3 · p. 18
  • Following prior work [16], they cap the initial corpus at 10 inputs and use it as the seed corpus.Source: §A.3 · p. 18
  • During the main run, the fuzzer continues after sanitizer crashes to collect multiple related crashing inputs.Source: §A.3 · p. 18

A.4 Output State Check … A.8 Evaluation Setup Details

The appendix details output-state and semantic checks, reports low patch-memorization rates on PATCHBENCH, validates DiffBLEU’s threshold, and describes evaluation setup. Together, these details clarify how the benchmark distinguishes crash suppression from behavior-preserving fixes and assesses patch similarity.

A.4 Output State Check

  • Output-state checks record harness outputs—including decoded values, parsed objects, generated files, serialized data, or API returns—and use generated file contents rather than return status when applicable.When fuzzing cannot generate suitable inputs, a wrapper rewrites benign-corpus seeds into minimally valid inputs while preserving their bytes.Source: §A.4 · p. 18

A.5 Semantic Validation Details

  • Semantic validation averages a 63.4% pass rate; among its three checks, output-state passes average 75.5%, the lowest, while sanitizer regression passes average 95.7%, the highest.Semantic validation separates patches that only remove crashes from those that preserve expected program behavior on benign inputs.Source: §A.5 · pp. 18–19, Table 4 · p. 20

A.6 Memorization on PATCHBENCH

  • On PATCHBENCH, memorization-threshold flags were 0.0% for Codex + GPT-5.6 Sol, 1.4% for Claude Code + Claude Opus 4.8, and 0.9% for OpenHands + Gemini 3.5 Flash, versus 22.0%, 27.7%, and 24.3% on SEC-BENCH.About 40% of PATCHBENCH patches scored 0, modifying entirely different locations from reference patches; the authors attribute evaluation failures mainly to task and validation difficulty rather than reproducing developer fixes.Source: Figure 9 · p. 19, §A.6 · p. 19

A.5 Semantic Validation Details

  • Removing the output-state check raises solved rates by 4.7 percentage points for RoboDuck, 11.3 for OpenHands + GPT-5, 9.4 for OpenHands + GPT-5.6 Sol, and 9.9 for Atlantis.The increases across agent systems indicate that output-state failures are not limited to one framework or model family.Source: §A.5 · p. 19

A.7 DiffBLEU Validation

  • On a held-out sample of 100 ARVO tasks, DiffBLEU separated patch similarity best; its 0.75 threshold yielded no false positives and a 1.1% false omission rate on the validation set.Three security researchers labeled similarity for calibration on an independent 100-task sample; the metric weights token-level similarity slightly more than context scores, using α = 0.3, β = 0.3, γ = 0.2, and δ = 0.2.Source: §A.7 · p. 19

A.8 Evaluation Setup Details

  • The evaluation used Codex v0.145.0, OpenHands SDK v1.20.0, and Claude Code SDK v0.2.100, with prompts providing project and repository details, sanitizer reports, and triggering-PoC locations.Agents could compile with `vulpatch compile` and, after successful compilation, run `vulpatch run` to check for sanitizer errors.Source: §A.8 · p. 19, Figure 10 · p. 21

Related papers

Loading 2609.04075v1…