Source-linked AI summary
EviRepair: Evidence-Driven Agents for Reliable Vulnerability Repair
Yilian Shi, Qingyu Liu, Tianyi Zhou, Wende Tan, Chenyang Si, Yueming Lyu, Xia Hu, Dongrui Liu
TL;DR
Agent-based AVR lacks explicit checks that collected evidence is consistent before that evidence guides patch generation. EVIREPAIR reviews evidence during diagnosis and patch synthesis, connecting the stages through a reviewed Vulnerability Model. It consistently outperforms evaluated baselines, with average resolved-rate improvements of 16.37% on PatchEval and 40.33% on SecBench.
Problem
Existing agent-based AVR generally does not explicitly validate evidence consistency throughout repair, although reliable repair requires evidence that changes address the underlying vulnerability.
Source: §1 · p. 1Method
EVIREPAIR uses evidence review in vulnerability diagnosis and patch synthesis, connecting them through a reviewed Vulnerability Model used to guide patch generation and validation.
Source: §1 · pp. 1–2Results
EVIREPAIR consistently achieves higher resolved rates than evaluated baselines, with average improvements of 16.37% on PatchEval and 40.33% on SecBench.
Source: §1 · p. 2Takeaways & Limitations
Ablations show that both multi-agent stages contribute to resolved rate, with replacing both producing the largest reported drop from 73.04% to 60.87%.
Source: §4.5 · p. 8Evaluation is limited to PatchEval and SEC-bench and benchmark-level security validation; the multi-agent design also incurs extra model calls, latency, and monetary costs.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Recent advances in large language models (LLMs) have enabled agent-based approaches for repository-level automated vulnerability repair (AVR), showing promising repair capabilities. However, an important issue has received limited attention: how can we ensure that the evidence generated during the repair process is reliable enough to support accurate vulnerability understanding and patch generation? In this paper, we propose EviRepair, an evidence-driven multi-agent framework for repository-level AVR. EviRepair introduces evidence review into the repair process by enabling agents to collaboratively collect vulnerability evidence, verify evidence consistency, and construct a reviewed vulnerability model for patch synthesis and validation. Extensive experiments on PatchEval and SecBench benchmarks with multiple agent frameworks and LLMs demonstrate that EviRepair consistently outperforms existing agent-based baselines, achieving average improvements of 16.37% and 40.33% in resolved rates on PatchEval and SecBench, respectively. Codes are available at https://github.com/lly-s/EviRepair
1 Introduction
Existing agent-based AVR workflows do not explicitly review evidence consistency throughout repair. EVIREPAIR introduces evidence review into vulnerability diagnosis and patch synthesis, and experiments report higher resolved rates than evaluated baselines.
1 Introduction
- Existing single-agent workflows reuse diagnostic evidence without dedicated review, while multi-agent frameworks aggregate outputs without verifying evidence consistency before patch generation.The paper notes that inconsistent or incomplete findings may therefore be propagated into patch generation.Source: §1 · p. 1
- EVIREPAIR links specialized evidence collection and consistency review in Vulnerability Diagnosis to patch planning, generation, and review in Patch Synthesis.The stages exchange a reviewed Vulnerability Model, and patch review checks security and compatibility criteria.Source: §1 · pp. 1–2
- EVIREPAIR achieves average resolved-rate improvements of 16.37% on PatchEval and 40.33% on SecBench across evaluated agent frameworks and LLMs.The experiments compare OpenCode and Codex using GLM-5.2 and DeepSeek-4.1-Flash.Source: §1 · p. 2
- The stated contributions are an evidence-driven repository-level AVR framework, review mechanisms for vulnerability understanding and patch correctness, and experiments across frameworks and LLMs.The introduction also describes ablation studies of the design components.Source: §1 · p. 2
2 Problem Formulation
The formulation treats repair as reconstructing a vulnerability model from a report, repository, initially available evidence, and repair-critical evidence. That model guides a patch that blocks unsafe behavior while preserving legitimate behavior.
2 Problem Formulation
- Reliable patching requires reconstructing needed information from the vulnerability report, target repository, and initially available evidence, which may be only a natural-language description.The repair-critical evidence components can be unavailable, incomplete, noisy, or mutually inconsistent.Source: §2 · pp. 2–3
- Repair-critical evidence comprises source-to-sink reachability, executable reproduction, runtime observations and expected post-repair behavior, the security boundary, and legitimate behaviors to preserve.These components are represented as a set of five evidence types.Source: §2 · p. 2
- The repair task reconstructs a vulnerability model and synthesizes a patch that blocks the reported unsafe behavior while preserving legitimate program behavior.Source: §2 · p. 3
3 EVIREPAIR
EVIREPAIR separates diagnosis from patch synthesis, using a reviewed Vulnerability Model as their interface. Diagnosis verifies collected findings, and synthesis generates and checks patches against the model.
3.1 Framework overview
- EVIREPAIR uses two coordinated multi-agent stages, with agents exchanging results through structured records with fixed fields and a Vulnerability Model connecting the stages.Source: §3.1 · p. 3
3.2 Vulnerability Diagnosis
- Vulnerability Diagnosis treats report claims as hypotheses, collects complementary evidence, and reviews whether findings explain the same vulnerability, cover report claims, and provide a ready oracle.Failed checks trigger targeted requests to responsible agents; verified findings are assembled into the Vulnerability Model.Source: §3.2 · pp. 3–4
3.3 Vulnerability Model
- The Vulnerability Model contains root cause, attack path, runtime oracle, security boundary, and preserved behaviors; the last two fields define patch acceptance criteria.A patch must remove unsafe behavior and preserve legacy behavior.Source: §3.3 · p. 4
3.4 Patch Synthesis
- Patch Synthesis translates model criteria into validation cases, generates a root-cause fix, and uses independent review and feedback to refine it until approval or budget exhaustion.The matrix includes violating and legitimate inputs, compatibility requirements, basic build checks, and unverified cases; the reviewer checks every expected outcome and consistency with the model.Source: §3.4 · pp. 4–5
4 Experiments
Across five research questions, experiments compare EVIREPAIR with agentic coding systems on PatchEval and SEC-bench, finding higher overall resolved rates while examining how results vary by vulnerability type, repair difficulty, and backbone.
4.1 Experimental Setup
- The evaluation uses 230 PatchEval CVE tasks and 300 SEC-bench tasks, with hidden-test %Resolved; SEC-bench additionally supplies sanitizer output.Systems share a sandbox and tool access; EVIREPAIR receives 3 hours for diagnosis and 2 for synthesis, while baselines receive 5 hours total.Source: §4.1 · p. 5
4.2 RQ1: Effectiveness of EVIREPAIR
- On PatchEval, EVIREPAIR resolves 79.13% versus OpenCode’s 71.89%; on SEC-bench, it resolves 93.11% versus 83.33%, leading under all three backbones.Relative to OpenCode, it reduces unresolved-task fractions by about 26% on PatchEval and 59% on SEC-bench.Source: §4.2 · p. 5, Table 1 · p. 6
4.3 RQ2: Performance across vulnerability types
- EVIREPAIR’s largest category gains occur where repair depends on reconstructing attack paths or security boundaries, while systems perform similarly on familiar injection fixes.It leads every SEC-bench category, with largest gains on use-after-free and NULL pointer dereference; it shows no SSRF advantage in the network-isolated sandbox.Source: §4.3 · pp. 6–7
4.4 RQ3: Performance across repair difficulty
- EVIREPAIR’s advantage generally persists as PatchEval repairs grow; for 21–50 changed lines it reaches roughly 72%, versus about 56% for the baselines.Its lead generally persists across hunks and one- or two-file patches, but not repairs spanning three or more files (7 of 230 tasks), where OpenCode leads; estimates are unstable.Source: §4.4 · pp. 7–8, Figure 5 · p. 7
4.5 RQ4: Ablation Study
- Replacing both multi-agent stages lowers PatchEval resolved rate by 12.17 points, versus 3.91 points for diagnosis or 5.21 points for synthesis alone.The authors attribute the stages’ reinforcement to independent checks of diagnostic evidence and synthesized patches.Source: §4.5 · p. 8, Table 2 · p. 8
4.6 RQ5: Effect of Backbone Capability
- With Qwen3.8-27B, EVIREPAIR resolves 75.22% of PatchEval tasks, exceeding GLM-5.2 baselines at 67.39% and 60.87%.On SEC-bench it reaches 89.00%, nearly matching OpenCode with DeepSeek-4.1-Flash at 90.00%; the evidence benefit grows as the backbone weakens on PatchEval.Source: §4.6 · p. 8
5 Related Work
Related work contrasts traditional and agent-based repair, while emphasizing the role of validation and available evidence in repair difficulty. EviRepair combines specialized analysis with cross-evidence validation for sparse reports.
5 Related Work
- Traditional repair relies on templates, program analysis, or search, but limited modeling capacity can hinder handling dependencies and runtime behavior in large repositories.Static code-pair datasets also cannot confirm that a patch removes exploitable behavior while preserving normal functionality.Source: §5 · pp. 8–9
- Single-agent systems combine retrieval, tools, and execution feedback, while multi-agent systems specialize tasks to address tangled reasoning and bias in one context.Prior multi-agent work includes expert vulnerability interpretation and collaborative derivation of security constraints and repair requirements.Source: §5 · p. 9
- Executable datasets strengthen validation with containerized differential execution, while progressive datasets constrain available evidence to model challenging repair conditions.The reviewed datasets indicate that available vulnerability evidence strongly controls task difficulty and repair performance.Source: §5 · p. 9
- EviRepair investigates reconstructing evidence from sparse reports through specialized subtasks and cross-evidence validation for global consistency.The approach builds on the complementary strengths and limitations of single-agent and multi-agent repair paradigms.Source: §5 · p. 9
6 Limitations
The evaluation and validation scope, together with the resource demands of the multi-agent design, bounds the conclusions supported by this study.
6 Limitations
- Evaluation is limited to PatchEval and SEC-bench, which may not represent the full diversity of vulnerabilities, languages, repositories, or deployment environments.The authors call for evaluation in broader real-world settings.Source: §6 · p. 9
- Benchmark-level security validation does not guarantee complete vulnerability elimination, absence of regressions, or long-term maintainability.The authors identify stronger security and regression assessment as future work.Source: §6 · p. 9
- The multi-agent design incurs additional model calls, latency, and monetary costs, leaving a repair-quality versus resource-cost trade-off for future study..Source: §6 · p. 9
7 Conclusion
EVIREPAIR separates diagnosis from synthesis through a reviewed Vulnerability Model and outperforms representative baselines on two benchmarks. Ablations support the value of stage decomposition and independent evidence review.
7 Conclusion
- 16.37% and 40.33% average resolved-rate improvements on PatchEval and SEC-bench, respectively, accompany consistent outperformance of representative baselines.The experiments evaluate EVIREPAIR on both benchmarks.Source: §7 · p. 9
- EVIREPAIR addresses sparse-report vulnerability repair by separating Vulnerability Diagnosis from Patch Synthesis and connecting them through a reviewed Vulnerability Model.The framework is evidence-driven and multi-agent.Source: §7 · p. 9
- Ablation studies demonstrate the value of stage decomposition and independent evidence review for evidence-grounded coordination in vulnerability repair.The authors hope the framework and findings facilitate future research on reliable, evidence-grounded repair.Source: §7 · p. 9
AI use statement
LLMs were used only for text refinement; the authors independently completed the research and experimental work.
AI use statement
- LLMs were used exclusively for writing clarity and language editing.The statement limits their use to text refinement.Source: AI use statement · p. 9
- Theoretical analysis and research methodology were completed independently by the authors without reliance on LLMs.Source: AI use statement · p. 9
- The authors also completed experimental design and result interpretation without relying on LLMs.Source: AI use statement · p. 9
Ethics statement
The work evaluates AVR in public benchmarks and controlled software repositories, without human subjects, personal data, or unauthorized live-system testing. It notes dual-use risks and limits claims about real-world patch safety.
Ethics statement
- Experiments used publicly available vulnerability-repair benchmarks and controlled software repositories, with no human subjects, private or personally identifiable data, or unauthorized live-system testing.Program analysis, vulnerability reproduction, and patch validation stayed within designated benchmark or experimental environments.Source: Ethics statement · p. 10
- The authors focus evaluation on public benchmark tasks and defensive repair, and warn that benchmark success does not guarantee elimination of all real-world exploit paths.Generated patches should be reviewed and validated before deployment.Source: Ethics statement · p. 10
- The authors state that they followed applicable software, dataset, and benchmark licenses and accept responsibility for responsible use and reporting.Source: Ethics statement · p. 10
Reproducibility Statement
The code and selected experimental resources are publicly available, with materials and instructions for running EVIREPAIR and its benchmark workflows.
Reproducibility Statement
- The public repository includes EVIREPAIR's implementation, agent prompts, structured communication schemas, and scripts for repair and benchmark evaluation.The workflows cover PatchEval and SEC-bench.Source: Reproducibility Statement · p. 10
- The README explains environment preparation, model endpoint configuration, and how to obtain and specify benchmark data and repository snapshots.Source: Reproducibility Statement · p. 10
- The repository README gives instructions for running repair and evaluation workflows, while the paper documents additional experimental configurations.Source: Reproducibility Statement · p. 10
A Full Schema of Vulnerability Model
This section specifies a structured vulnerability model and evidence-review workflow, including reviewer prompts and a patch-review handoff for validating candidates against the model and validation matrix and returning evidence-backed verdicts and actionable findings.
A Full Schema of Vulnerability Model
- The schema records analysis status, security boundaries and root-cause evidence, execution paths, reproducer and oracle status, post-patch validation usability, and entrypoint and environment coverage.Source: A Full Schema of Vulnerability Model · pp. 13–14
B A Case Study of EVIREPAIR
- Figure 6 is captioned as a case study of EviRepair’s evidence-driven two-stage vulnerability repair.Source: Figure 6 · p. 15
C Prompt
- The evidence-review prompt directs an adversarial reviewer to challenge conclusions, reject unsupported agreement, and inspect claims against source, runtime, oracle, and boundary evidence.Its checks include reachability, oracle phase-awareness, patch coupling, root-cause fit, controls, environment assumptions, alternatives, and contradictions.Source: C Prompt · pp. 15–17
- The patch-review prompt requires review of the whole candidate and blocking matrix; PASS requires every blocking dimension to pass or have an accepted waiver.The checklist covers strategy, root cause, attack path, boundary, compatibility, coverage gaps, critical questions, evidence sufficiency, anti-patterns, stress families, and clean application; verdicts are PASS, REFINE, or FAIL.Source: C Prompt · pp. 18–19
- Review evidence must be tied to matrix rows and relevant source evidence, tests, documented behavior, or safe fallbacks; environment-blocked rows must not be marked PASS.Findings should be specific, matrix-linked, and actionable.Source: C Prompt · pp. 19–20
- The findings schema records PASS, REFINE, or FAIL, dimension-level evidence, blocking findings, refinement guidance, and unresolved risks.Review dimensions include strategy, root-cause and attack-path coverage, boundary restoration, compatibility, validation evidence, anti-patterns, stress families, and clean-apply consistency.Source: C Prompt · p. 20
- Reviewers must validate the candidate against the analysis commitments, walk the full validation matrix, assess each required review dimension, and provide a structured verdict with refinement guidance.The handoff names the vulnerability model, validation matrix, expected secure behavior, security boundary, compatibility requirements, validation requirements, and must-preserve and must-reject cases.Source: C Prompt · pp. 20–21