PaperlayerView exact revision on arXiv

Source-linked AI summary

VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities

Weizhe Wang, Wei Ma, Qiang Hu, Yao Zhang, Jianfei Sun, Bin Wu, Yang Liu, Guangquan Xu, Lingxiao Jiang

arXiv:2509.03331v1cs.SEcs.CR

TL;DR

Existing vulnerability-repair evaluations often rely on tests or minimal triggers rather than verifying that an exploit is blocked. VulnRepairEval evaluates repairs with functional PoCs and a containerized differential pipeline. Its evaluation finds that even the strongest model repairs only a minority of benchmark cases.

  • Problem

    Many existing evaluations use unit tests or minimal vulnerability triggers, which do not establish whether exploitation is prevented after patching.

    Source: §1 · pp. 1–2

  • Method

    VulnRepairEval evaluates real-CVE patches by requiring the original PoC to fail on the patched build in a containerized differential check.

    Source: §1 · p. 2, §5.2 · p. 6

  • Results

    5/23 (≈21.7%) was the strongest model’s repair result, and vulnerability localization accounted for roughly 60–78% of failures.

    Source: §7 · p. 11

  • Takeaways & Limitations

    LLM patches are starting points rather than final fixes, and the paper recommends secondary checks such as static analysis and fuzzing before acceptance.

    Source: §7 · pp. 11–12

    PoC failure is necessary but not sufficient to establish repair, and complementary checks such as static analysis and fuzzing remain future work.

    Source: §7 · p. 12

Abstract

from arXiv · show

The adoption of Large Language Models (LLMs) for automated software vulnerability patching has shown promising outcomes on carefully curated evaluation sets. Nevertheless, existing datasets predominantly rely on superficial validation methods rather than exploit-based verification, leading to overestimated performance in security-sensitive applications. This paper introduces VulnRepairEval, an evaluation framework anchored in functional Proof-of-Concept (PoC) exploits. Our framework delivers a comprehensive, containerized evaluation pipeline that enables reproducible differential assessment, where repair success requires the original exploit to fail execution against the modified code. The benchmark construction involved extensive data curation: we processed over 400 CVEs and approximately 2,500 potential sources to extract a collection of authentic vulnerability instances (23 Python CVEs) amenable to automated testing with working PoCs. Through VulnRepairEval, we conduct a comprehensive evaluation of 12 popular LLMs and observe a significant performance deficit: even the top-performing model successfully addresses merely 5/23 instances (about 21.7%), exposing critical weaknesses in security-focused applications. Our failure analysis reveals that most unsuccessful attempts stem from imprecise vulnerability identification and patches containing syntactic or semantic errors. Enhanced prompting strategies and multi-agent approaches yield minimal improvements, with overall effectiveness remaining largely unaffected. This work contributes a stringent, practical evaluation framework for LLM-driven vulnerability remediation and underscores the necessity for assessment protocols that authentically reflect real-world exploitation scenarios.

1 INTRODUCTION

VulnRepairEval addresses the gap between patches that pass tests and repairs that actually block exploitation. Its PoC-based evaluation finds low repair success across 12 LLMs, with failures often rooted in vulnerability localization.

1 INTRODUCTION

  • Passing unit tests or running without errors does not establish that a vulnerability is eliminated, and plausible patches can leave exploitable flaws.The introduction cites incomplete fixes linked to code execution, a TIFF decoding overflow, and Django ReDoS.Source: §1 · pp. 1–2
  • VulnRepairEval counts a repair as successful only when the original PoC fails against the patched program, using a containerized two-environment differential check.The pipeline holds conditions constant except for the patch; the benchmark covers 23 real CVEs from 2017 to 2024 across nine vulnerability categories.Source: §1 · p. 2
  • 5/23 (≈21.7%) was the strongest model’s repair result, while incorrect localization and unusable or incomplete patches drove many failures.Across models, localization accounted for roughly 60–78% of failures; prompting hints offered limited benefits, and agentic workflows did not close the end-to-end gap.Source: §1 · p. 2
  • The benchmark’s 23 real-CVE instances and controlled evaluation of 12 mainstream LLMs provide a reproducible basis for assessing exploit-blocking repairs.The stated contributions include a PoC-driven benchmark, a reproducible evaluation protocol, systematic model evaluation, and scripts and artifacts planned for release upon publication.Source: §1 · p. 2

2 RELATED WORK

Prior work spans general repair and code-generation benchmarks, security-oriented datasets, PoC generation, and vulnerability reasoning. The paper identifies persistent gaps in exploit-level validation, realism, and coverage, and positions VulnRepairEval as a stricter evaluation approach.

2 RELATED WORK

  • General repair benchmarks use unit tests or project CI, while many security-oriented corpora remain patch-level and lack executable exploit validation.Vul4J and VUL4C use PoV tests or exploit scripts, and CVE-Bench models information layers and integrates static analysis; systematic PoC-failure criteria remain scarce, especially for Python.Source: §2 · p. 2
  • Related work also covers automated PoC generation, vulnerability detection and reasoning, and general code-generation benchmarks, which do not by themselves establish exploit-blocking repair.PoCGen combines LLMs with static and dynamic analysis [26]; detection studies report sensitivity to small code changes [3], while HumanEval and MBPP lack explicit security semantics.Source: §2 · p. 2
  • Existing evaluation has moved toward exploit validation and richer context, but still has gaps in validation, realism, and coverage across settings.The paper identifies reliance on compilation or unit tests, limited modeling of reporting and deployment conditions, and language- or vulnerability-specific datasets.Source: §2 · p. 2
  • VulnRepairEval combines a PoC-driven success criterion, black- and white-box inputs, and containerized automation for reproducible end-to-end evaluation.Its criterion requires the original PoC to fail on the patched system, distinguishing it from a minimal proof-of-vulnerability trigger.Source: §2 · p. 3, Table 1 · p. 3

3 DATA CONSTRUCTION

The authors curate a Python CVE benchmark by locating and verifying executable PoCs, then retaining instances that support automated exploit testing. The resulting benchmark contains 23 CVEs across nine vulnerability types.

3 DATA CONSTRUCTION

  • The initial corpus contained 449 CVEs from 192 GitHub projects, later refined to 448 CVEs across 164 projects and 684 unique fixing commits.The process retained the longest available commit hash to establish a canonical representation for each fix.Source: §3 · p. 3
  • 2,493 candidate URLs from NVD references were crawled, and an LLM-assisted pipeline classified rendered page content to identify potential executable PoCs.The pipeline loaded pages in a headless browser, converted rendered HTML to Markdown, and submitted the resulting text to an LLM for classification.Source: §3 · p. 3
  • The URL classifier distinguished executable PoCs, descriptive exploit material, and brief high-level summaries without actionable details.Executable entries contained directly runnable code; descriptive entries supplied enough information to construct a PoC, often with partial code.Source: §3 · p. 3
  • Manual verification of model-flagged links found one false positive, and the final benchmark comprised 23 CVEs from 2017–2024 across nine vulnerability types.The vulnerability-type distribution is shown in Figure 1.Source: §3 · p. 4, Figure 1 · p. 4

4 VULNREPAIREVAL

VulnRepairEval combines targeted patch generation, standardized PoC execution, and automated deployment. Its framework reconstructs vulnerable baselines and compares patched and unpatched code in isolated environments.

4 VULNREPAIREVAL

  • The framework has three modules—patch generation, runtime injection, and automatic deployment—and tests the same PoC against vulnerable and patched code in isolated Docker environments.The two-container differential analysis is designed so the patch is the only changing factor.Source: §4 · p. 4, §4.3 · p. 5, Figure 2 · p. 5

4.1 Patch Generation

  • 4.1 Patch Generation supplies the vulnerable source files to the model, which must localize the flaw and produce a syntactically correct standard diff.The vulnerable files are identified from security-fix commits on the primary branch after filtering out metadata, documentation, dependency, and test-only changes.Source: §4.1 · p. 4

4.2 Runtime Injection

  • 4.2 Runtime Injection wraps diverse PoCs in a unified execution interface, capturing their inputs, outputs, and runtime outcomes for consistent analysis.The wrapper launches PoCs as child processes, redirects I/O, provisions dependencies, and formats the captured runtime data.Source: §4.2 · pp. 4–5

4.3 Automatic Deployment

  • 4.3 Automatic Deployment reconstructs the vulnerable state from the parent of the official fix commit and automates project setup for reproducible testing.Docker provisions parallel vulnerable and patched instances using resolved dependencies and system requirements.Source: §4.3 · p. 5

5 EVALUATION

Section 5 compares leading models under uniform conditions, defines PoC-validated repair as its primary endpoint, and uses diagnostics to study failures and mitigation strategies. The implementation evaluates patches by comparing patched and unpatched container runs.

5.1 Evaluation Setting

  • The study compares 12 leading LLMs, including proprietary and open-source models, on identical vulnerabilities under uniform conditions.Source: §5.1 · p. 5

5.2 Evaluation Metrics

  • A repair counts toward Psucc only when its patch applies and a PoC that succeeds on the baseline fails on the patched build.Source: §5.2 · p. 6
  • Pcorr and Vdnf diagnose patch applicability and abstention, while composite Sp breaks leaderboard ties without replacing Psucc as the scientific endpoint.Sp prioritizes repair success, gives diminishing credit to format-correct patches, and penalizes abstention.Source: §5.2 · p. 6

5.3 Research Questions (RQs)

  • The study asks how well LLMs repair vulnerabilities end to end, where pipeline failures arise, and whether prompting or agentic strategies improve Psucc.Source: §5.3 · pp. 6–7

5.4 Implementation

  • The pipeline reconstructs vulnerable project states, prompts models using vulnerable source files, and compares PoC outcomes in patched and unpatched Docker containers.It also checks whether generated patches apply successfully.Source: §5.4 · p. 7

6 RESULTS AND ANALYSIS

Under PoC-validated evaluation, LLM vulnerability repair remains weak, with failures tied to flaw localization and patch quality. Difficulty-stratified results and mitigation experiments show that simple prompt changes have little overall effect, while agents yield improvements but leave repair rates suboptimal.

6.1 RQ1: PoC-Validated Repair Capability

  • Patch application and security repair are distinct: clean application indicates format correctness, while repair requires the PoC exploit to fail; DeepSeek V3 671B reported “No Vulnerabilities Found” in 87.0% of cases.The GPT series had “No Vulnerabilities Found” rates of 43.5%–56.5%.Source: §6.1 · p. 7
  • 0.217 was Gemini 2.5 Pro’s repair efficacy, with a final score of 0.226; most models scored below 0.20, and several scored zero.Gemini 2.5 Pro fixed five distinct CVEs; repeated experiments showed little variation.Source: §6.1 · p. 7, Table 3 · p. 8

6.2 RQ2: Failure Analysis

  • Localization was the predominant failure, while patch-generation errors accounted for about 20% of attempts; Gemini fixed 3 Easy and 2 Medium vulnerabilities, but none Hard.DeepSeek R1 671B was the only model to fix a Hard vulnerability.Source: §6.2 · p. 9
  • Qwen3 235B Thinking could backtrack from incorrect reasoning, unlike Qwen3 8B Thinking, whose verbose reasoning often followed incorrect directions.The analysis links the models’ difference to scale and reasoning behavior.Source: §6.2 · p. 9

6.3 RQ3: Efficacy of Prompting and Agent

  • From over 80% to 26%, DeepSeek V3 671B’s detection-failure rate fell under the agent approach; static prompt enhancements had negligible overall impact.The agent’s overall repair improvement was modest and primarily reflected better vulnerability detection.Source: §6.3 · p. 11
  • Agent-generated patches were mostly limited to single files, constraining repairs of multi-file vulnerabilities and leaving overall repair rates suboptimal.The agent framework also could not override base-model behaviors such as conservative detection or failures to produce the requested patch format.Source: §6.3 · p. 11

7 DISCUSSION

LLMs remain unreliable at repairing real vulnerabilities: the strongest fixes about 21.7%, with localization failures predominating. The discussion highlights limits in task scope and in using PoC failure as the sole repair criterion.

7 DISCUSSION

  • About 21.7% of vulnerabilities are fixed by the strongest model, and localization accounts for roughly 60–78% of failures.Agentic workflows reduce abstention from 87% to 26% yet only modestly raise P_succ.Source: §7 · p. 11
  • PoC failure is necessary but not sufficient: a brittle input guard may block the trigger without eliminating the underlying bug.The paper reports containerized differential validation and payload-variant reruns as mitigations, with static analysis and fuzzing left for future work.Source: §7 · p. 12
  • Single-file tasks with constrained context differ from real projects with multi-file dependencies, build systems, and configuration interactions, potentially under- or overestimating capability.The authors plan adapters for C/C++, Java, and JavaScript/TypeScript to profile language-specific failure modes.Source: §7 · p. 12

8 CONCLUSION

The conclusion presents VulnRepairEval as a PoC-driven benchmark for evaluating real-world vulnerability repair and finds that precise localization and sound, complete patches remain difficult for LLMs. The appendix describes a standardized patch prompt and prompt-enhancement variants.

8 CONCLUSION

  • VulnRepairEval evaluates real-world vulnerability repair with a PoC-driven benchmark, while current LLMs struggle with precise localization and syntactically correct, logically sound, complete patches.Advanced prompting and agent-based interactions offer limited performance gains and do not overcome core model limitations.Source: §8 · p. 12
  • The conclusion calls for evaluation criteria that reflect genuine security practices.This follows the paper’s stated benchmark contribution and conclusion.Source: §8 · p. 12

A PROMPT FOR PATCH GENERATION.

  • The standardized prompt fixes the model’s role and task, requires code-only analysis and Unified Diff output, and permits either a patch or a literal no-patch response.These rules support apples-to-apples comparisons and automatic patch application and scoring.Source: A PROMPT FOR PATCH GENERATION · p. 13
  • Three static prompt enhancements add vulnerability type, a step-by-step directive, or an example to the base prompt.The stated aim is to improve localization and patch formatting, thereby increasing repair success.Source: A PROMPT FOR PATCH GENERATION · p. 13

A.1 Prompt for providing vulnerability type

  • The vulnerability-type prompt asks the model to inspect the supplied code for programming errors, logical flaws, or security vulnerabilities and generate a patch.The vulnerability type is supplied as a reference for identification.Source: §A.1 · p. 13
  • The prompt restricts analysis to the provided code and requires raw Unified Diff output, or the specified literal response when no changes are warranted.A generated patch must be usable with a standard patch command.Source: §A.1 · p. 13

<VULNERABILITY_TYPE> Specific type of vulnerabilities. </VULNERABILITY_TYPE>

The template labels the filename and code as input fields, then directs the model to proceed with analysis and output generation.

<VULNERABILITY_TYPE> Specific type of vulnerabilities. </VULNERABILITY_TYPE>

A.2 Prompt for adding step-by-step (CoT) directive

The CoT prompt adds a step-by-step directive to a patch-generation template that restricts analysis to supplied code and specifies a conditional output format. The included example-prompt subsection retains the same core task and constraints.

A.2 Prompt for adding step-by-step (CoT) directive

  • The task is to inspect the tagged code for programming errors, logical flaws, or security vulnerabilities and generate a patch.Source: §A.2 · p. 13
  • The prompt requires analysis to rely exclusively on supplied code and forbids external codebase lookup.Source: §A.2 · p. 13
  • The response must be raw Unified Diff, without explanations or markdown, and a generated patch must work with a standard patch command.Source: §A.2 · p. 13
  • The prompt requires a patch when a fix is warranted and the literal no-patch response otherwise.Filename and code are supplied in tagged fields, followed by an instruction to proceed.Source: §A.2 · pp. 13–14
  • The CoT variant adds “Let’s think step by step” to the patch-generation prompt.The figure is captioned as the prompt for patch generation.Source: §A.2 · p. 14, Figure 4 · p. 14

A.3 Prompt for including an example

  • The example-prompt variant retains the code-analysis task, exclusive-code-source rule, Unified Diff requirement, and conditional patch-or-no-patch output.The supplied passages do not describe the example’s contents.Source: §A.3 · p. 14

## Example

The example is presented as a perfect task execution and emphasizes that finding the vulnerable region is not enough: a repair must remove the dangerous behavior. Its prompt asks for code analysis and patch generation.

## Example

  • The example is introduced as a perfect execution of the task.Source: ## Example · p. 14
  • Successful localization is necessary but insufficient; effective repair must remove the dangerous behavior rather than make superficial or irrelevant edits.The passage contrasts mitigation logic with formatting-valid changes and superficial input checks.Source: ## Example · p. 15

Content of the corresponding example patch file

B CASE STUDY FOR CVE-2022-21797

The CVE-2022-21797 case contrasts the official structured-parsing fix with model patches that retain the risky eval pathway, fail to run, or edit the wrong location. The examples distinguish applicable but incomplete mitigation from inapplicable or logically ineffective repairs.

B CASE STUDY FOR CVE-2022-21797

  • The official fix parses pre_dispatch, whitelists simple arithmetic over n_jobs, and converts the result to an integer, removing direct evaluation of arbitrary code.The approach addresses both applicability and the attack surface.Source: B CASE STUDY FOR CVE-2022-21797 · p. 15
  • Gemini 2.5 Pro screens pre_dispatch but retains eval; GPT 4o mini produces an inapplicable format error; Qwen3 8B Thinking edits the wrong region, leaving the vulnerable pathway reachable.The figure presents the case study and the prose describes these three failure modes.Source: B CASE STUDY FOR CVE-2022-21797 · p. 15, Figure 5 · p. 16

Related papers

Loading 2509.03331v1…