PaperlayerView exact revision on arXiv

Source-linked AI summary

PatchHolmes: Agentic Patch Retrieval via Listwise Selection

Guanqun Yang, Yingming Zhou, Jiangrui Zheng, Shudong Hao, Xueqing Liu

arXiv:2609.38807v1cs.IRcs.SE

TL;DR

Many CVEs lack links to their fixing commits, making patch retrieval an important but difficult part of vulnerability management. PatchHolmes combines hybrid retrieval with an agent that inspects candidates listwise. It improves Recall@1 over prior approaches and over the retriever’s top pick, including after transfer to another candidate pool.

  • Problem

    Patch retrieval must link vulnerabilities to fixing commits, but 60% to 63% of CVEs in major advisory databases lack patch links, and long diffs and low vocabulary overlap complicate retrieval.

    Source: §1 · pp. 1–2

  • Method

    PatchHolmes retrieves a top-100 candidate set with a hybrid first stage, then uses a multi-turn agent to compare candidates, inspect selected commits, and submit one best commit.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    25.34% Recall@1: PatchHolmes beats Favia on GitHubAD, and the agent adds 27.32% Recall@1 over the retriever’s top pick on identical candidates.

    Source: §1 · p. 3, §4.2 · pp. 6–7

  • Takeaways & Limitations

    The same agent transferred unchanged to PatchFinder_top10, lifting Recall@1 from 24.28% to 39.86%; the system runs on a frozen open-weight model over a local repository without fine-tuning or external search APIs.

    Source: §1 · p. 3

    For 143 large source repositories, sampling substitutes corpus-wide medians for commit counts and gold-patch diff lengths, making the sample slightly more uniform on those two axes.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Patch retrieval, the task of finding the commit that fixes a known vulnerability, is the foundation of vulnerability management workflows, yet 60% to 63% of CVEs in the major advisory databases lack a patch link. We present PatchHolmes, a two-phase patch retrieval system that pairs a hybrid first-stage retriever with an agentic second-stage inspection loop. Unlike pointwise prior work that scores each candidate independently, the Phase 2 agent reads the top-100 listwise: it sees the full candidate list at once and selectively reads 3 to 10 commits through four budgeted tools before submitting a single best commit. On GitHubAD, PatchHolmes beats the pointwise binary classifier Favia by 25.34% Recall@1 and the retrieve-and-CoT baseline IRCoT by 31.40%, at one agent conversation per CVE versus Favia's ten; with the candidate set held identical, the agent adds 27.32% Recall@1 over taking the retriever's top candidate, and the same agent, transferred unchanged to PatchFinder_top10, lifts Recall@1 from PatchFinder's own top-1 pick (24.28%) to 39.86%. Swapping the LLM backbone within the Qwen family changes Recall@1 by under 1%, and a second model family (gpt-oss) stays far above the no-agent floor, so the gain comes from the listwise agent loop; the entire system runs on a frozen open-weight model over a local Git repository, without fine-tuning or external search APIs.

1 Introduction

Patch retrieval links disclosed vulnerabilities to their fixing commits, but many CVEs lack patch links and the task is difficult for both retrieval and candidate selection. PatchHolmes pairs a hybrid retriever with an agent that compares candidates listwise, improving benchmark results over prior approaches.

1 Introduction

  • 60% to 63% of CVEs in GitHub Advisory Database and NVD lack patch links, and auxiliary-information matching recovers only 12% to 53% even with manual assistance.Manual mapping does not scale, while missing links can persist for years.Source: §1 · p. 1
  • Large repositories, long diffs, and low vocabulary overlap make retrieval difficult; the gold patch may appear anywhere in the top-100 shortlist.Traditional open-loop pre-LLM selectors cannot refine queries or inspect full diffs beyond their 512-token context window.Source: §1 · p. 2
  • PatchHolmes uses a hybrid top-100 retriever and a multi-turn agent that compares candidates, selectively reads commits, and submits one best commit.For CVE-2015-9251, the agent identifies the fix after reading five commits, where Favia’s ten independent calls give all candidates identical confidence.Source: Figure 1 · p. 1, §1 · p. 2
  • 59.95% Recall@1: PatchHolmes beats Favia by 25.34% and IRCoT by 31.40% on GitHubAD, at one agent conversation per CVE.On PatchFinder_top10, Recall@1 is 39.86% versus PatchFinder’s 24.28%.Source: Figure 2 · p. 3, §1 · p. 3
  • The system targets a frozen open-weight LLM over a local Git repository, without fine-tuning or an external search API.One agent conversation per CVE replaces Favia’s ten.Source: §1 · pp. 2–3

2 Related Work

Prior patch-retrieval systems use feature-based or neural two-stage ranking, while newer LLM approaches add pointwise scoring or general-purpose agent loops. PatchHolmes instead uses one agent to inspect a fixed candidate list jointly.

2 Related Work

  • PatchFinder established a two-phase template with lexical-and-neural retrieval followed by a fine-tuned cross-encoder, while SPFinder uses hierarchical embeddings and a feature-based reranker.PatchScout introduced feature-based ranking from vulnerability-commit correlations over paths, types, and identifiers.Source: §2 · p. 3
  • Favia replaces fine-tuned scoring with independent LLM yes/no and confidence calls, while PatchHolmes jointly presents the candidate manifest and selectively inspects commits.PatchHolmes targets the same setting but replaces both SPFinder’s open-loop feature stack and Favia’s pointwise calls.Source: §2 · p. 3
  • Patch retrieval has one correct commit among as many as 1.4 million and little vocabulary overlap, a regime where reasoning-augmented retrieval can gain over classical retrieval.PatchHolmes’s fixed-candidate agent gain of 27.32% Recall@1 and under-1% within-family backbone variation are consistent with that pattern.Source: §2 · p. 3
  • Coding and retrieval agents apply multi-turn tool use to text and source-code repositories, but Favia is the only agentic system among those cited that targets patch retrieval directly.Examples include repository navigation, code-symbol search, knowledge-graph querying, and reinforcement-learned navigation.Source: §2 · pp. 3–4

3 Method

PatchHolmes separates recall-oriented retrieval of 100 candidates from precision-oriented agent selection. Its first phase fuses lexical and dense rankings; its second lets an agent survey the list, inspect selected commits at two granularities, and submit one fix.

3 Method

  • The two phases share only candidate identifiers: Phase 1 writes a JSONL shortlist from the CVE description and repository, and Phase 2 selects one predicted fix from it.Phase 1 retrieves 100 candidates from repositories containing thousands to millions of commits.Source: §3 · p. 4

3.1 Phase 1: Hybrid Retrieval

  • Phase 1 fuses a BM25-with-time-decay lexical path and a dense path using Reciprocal Rank Fusion; the top 1000 are stored, and Phase 2 reads the first 100.The 100-candidate cut balances access to late-ranked gold commits with keeping the manifest prompt short.Source: §3.1 · pp. 4–5
  • The lexical path combines message and diff BM25 scores with reserve-date and public-disclosure time-rank signals using fixed weights.Message BM25 receives weight 0.35 and diff-body BM25 0.15; the time-rank score decays symmetrically with rank distance.Source: §3.1 · p. 4
  • The dense path embeds the CVE and commits in a shared 4096-dimensional space and ranks by cosine similarity; diff lines are selected within a 6,000-character budget.The fused retriever outperforms each single-path ablation by 2.84% to 20.27% Recall@1.Source: §3.1 · pp. 4–5

3.2 Phase 2: Agentic Listwise Selection

  • The agent has up to 15 iterations and typically reads 3 to 10 candidates; it stops on submission, the iteration limit, or repeated actions.Each turn invokes one tool, and the CVE description is passed verbatim.Source: §3.2 · p. 5
  • list_candidates exposes all 100 candidates in rank order with commit identifiers, message first lines, and file-change tallies before the agent drills into commits.read_commit returns the message, file manifest, and a diff under an 8,000-character budget; read_file_diff retrieves one file under a 16,000-character budget.Source: §3.2 · p. 5
  • The agent submits exactly one commit identifier and reasoning; the four-tool set is limited to surveying, reading at two granularities, and submitting.The fixed Phase 1 candidate pool and lack of code modification leave no role for open-ended search or editing tools.Source: §3.2 · pp. 5–6

4 Experiments

PatchHolmes is evaluated on GitHubAD and PatchFinder_top10 against retrieval, pointwise-classification, and retrieve-and-reason baselines. Its listwise agent improves Recall@1 on the main benchmark, transfers to a separate candidate pool, and retains gains across ablations, backbones, and cost settings.

4.1 Experimental Setup

  • Evaluation uses an 809-CVE stratified GitHubAD subset and a 1,252-CVE PatchFinder_top10 pool; head-to-head baselines share Qwen3-235B, and runtime errors count as misses.The GitHubAD working subset is drawn from an 8,401-CVE corpus.Source: §4.1 · p. 6

4.2 Main Results

  • 59.95% Recall@1 for PatchHolmes beats Favia (34.61%), IRCoT (28.55%), and SPFinder (27.32%) on GitHubAD; it wins every reported column.The comparison uses Qwen3-235B and an 809-CVE working subset.Source: §4.2 · p. 6, Table 1 · p. 7
  • 27.32% Recall@1 separates the agent from Phase 1’s rank-1 pick on identical candidates; the remaining matched-pool gaps over Favia and IRCoT are 20.15% and 22.37%.Favia rises from 34.61% to 39.80% and IRCoT from 28.55% to 37.58% when given PatchHolmes’s candidates.Source: §4.2 · p. 7
  • 15.58% absolute Recall@1 gain transfers to PatchFinder_top10, from PatchFinder’s 24.28% top-1 pick to 39.86% with Phase 1 skipped.The same agent runs unchanged on the 10-candidate pool; 46.10% is the maximum Recall@1 because 675 CVEs have no gold fix in that pool.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 Ablation

  • 57.11% Recall@1 from the dense-only path is within 2.84% of MAIN; fusion adds +2.84% Recall@1 and +5.32% Recall@10.Using the full NVD report leaves Recall@1 nearly flat (59.83% vs 59.95%) while raising context-overflow errors from 0.49% to 2.60%.Source: §4.3 · p. 8, Table 3 · p. 9
  • 15.58% Recall@1 is the largest tool-ablation gain: listwise candidate viewing improves over no agent, while commit overview and diff reading add 9.27% and 2.47%.Each further tool adds recall on the same Phase 1 candidates.Source: §4.3 · p. 8

4.4 Analysis

  • 59.95% and 59.09% Recall@1 for Qwen3-235B and Qwen3-Coder-30B differ by 0.86%; both tested gpt-oss models exceed the 32.63% no-agent floor.The comparison holds tools, prompt, and Phase 1 candidates fixed while changing the backbone.Source: §4.4 · pp. 8–9, Table 4 · p. 9

4.5 Cost and Accuracy

  • USD 0.007 per CVE is PatchHolmes’s reported cost, with about USD 58 for the 8,401-CVE corpus; it uses about 7x fewer tokens than Favia while reaching 25.34% higher Recall@1.Qwen3-Coder-30B trades 0.86% Recall@1 for half the wall-clock time (39 versus 86 seconds per CVE).Source: §4.5 · p. 9, Table 5 · p. 9

5 Conclusion

PatchHolmes pairs a hybrid retriever with a four-tool agentic stage that jointly inspects candidates. It outperforms the reported baselines on GitHubAD, improves selection on identical candidates, and transfers to PatchFinder_top10.

5 Conclusion

  • 25.34% Recall@1 over Favia, 31.40% over IRCoT, and 32.63% over SPFinder on GitHubAD.The comparison is reported for PatchHolmes against Favia’s pointwise baseline, IRCoT, and SPFinder.Source: §5 · p. 9
  • 27.32% Recall@1 is added over the retriever’s top pick on identical candidates, while PatchFinder_top10 rises from 24.28% to 39.86%.The same agent was transferred unchanged to PatchFinder_top10.Source: §5 · p. 9
  • PatchHolmes combines a hybrid first-stage retriever with a four-tool agentic second stage, and its gain holds across a 7x backbone-scale swap and a second model family.The system uses about 97K input tokens per CVE.Source: §5 · p. 9

Limitations

The evaluation is bounded by known-fix ground truth and candidate-pool recoverability, with additional limits from sampling and incomplete fix mappings.

Limitations

  • 60% to 63% of CVEs lack patch links, so evaluation covers only the 37% to 40% whose patches have already been found.The 8,401-CVE source corpus is the subset that survived this filter; the independent 35-CVE check also requires a known fix to score against.Source: Limitations · p. 10
  • 675 of 1,252 PatchFinder_top10 CVEs lack the true fix among their ten candidates, capping every system on that pool at 46.10%.PatchHolmes reaches 39.86%, closing 71.40% of the distance from the 24.28% trivial baseline to the ceiling.Source: Limitations · p. 10
  • For 143 of 2,816 repositories with commit-history files larger than 500 MB, commit counts and gold-patch diff lengths use corpus-wide medians.This makes the sample slightly more uniform than the source on those two axes; the passage says other reviewed dimensions still match the source.Source: Limitations · p. 10
  • A different-but-equally-valid fix absent from the recorded gold set scores as a miss, including for backports, split refactors, or downstream merges.The evaluation counts a hit when the submitted commit appears in the recorded gold set; 7.28% of source-corpus CVEs have several recorded fixes.Source: Limitations · p. 10

A.1 Sampling Validation for GitHubAD

The GitHubAD draw is stratified by language and repository size, then checked against the source corpus with a nine-test validation suite. Six tests pass; three failures concern within-repository commit-count and diff-token distributions, which are affected by oversized repositories skipped during indexing.

Sampling procedure

  • The sampler stratifies by language and repository-size bucket, allocates subgroup budgets proportionally, caps each repository at 10 CVEs, and retains the best of up to 50 redraws.Repositories are drawn proportional to their CVE counts without replacement, and the draw is evaluated against the nine-test suite.Source: §A.1 · p. 12

Validation procedure

  • The suite tests numeric distributions with KS and tail-sensitive Anderson-Darling, categorical mixes with chi-square, and joint structure with maximum mean discrepancy.Its six strata variables also include CWE family, pre-ranker bucket, diff-token bucket, and year, alongside language and repository size.Source: §A.1 · p. 12

Validation results

  • 6 of 9 tests pass for the 809-CVE working subset, including all three categorical tests and the joint-feature test; failures concern commit-count and diff-token axes.The passing tests include KS on diff tokens (p=0.41) and year (p=0.37), chi-square on language (p=0.81), CWE family (p=0.16), and pre-ranker bucket (p=0.18), and MMD on the joint feature vector (p=0.61).Source: §A.1 · pp. 12–13

Sampling results

  • The sampler produces an 839-CVE parent draw across 231 repositories, trimmed to an 809-CVE working subset across 227 repositories after excluding incomplete Phase 1 preprocessing artifacts.The parent draw shows the same six-of-nine pass pattern as the trimmed subset.Source: §A.1 · pp. 12–13

Limitations

  • Skipping 143 repositories with commit shards larger than 500 MB keeps the cold-cache index build under one hour but compresses sampled commit-count and diff-token tails.Those axes are imputed from shard size or replaced with the global median; parsing all 143 repositories is described as a one-time five-hour operation.Source: §A.1 · p. 13

A.2 Phase 1 Dense Backbone Ablation

The dense-leg embedding choice materially affects Phase 1 retrieval: Octen performs worse than the non-fine-tuned Qwen3-Embedding-8B with either tested prompt. The authors therefore retain Qwen3-Embedding-8B.

Retrieval results

  • Octen-Embedding-8B with the task-aligned CVE prompt reaches 12.73% Phase 1 dense-leg Recall@1, versus 43.39% for Qwen3-Embedding-8B.Using Octen’s official “web search” prompt yields 2.97% Recall@1.Source: §A.2 · p. 13

Prompt comparison

  • Octen’s official “web search” prompt performs worse than its task-aligned CVE prompt, reaching 2.97% Recall@1 rather than 12.73%.Source: §A.2 · p. 13

Backbone choice

  • The authors retain non-fine-tuned Qwen3-Embedding-8B for the dense leg because the general-purpose embedding leaderboard does not transfer to patch retrieval.Source: §A.2 · p. 13

A.3 Diff Render Layer

This appendix describes a three-stage diff-render layer inserted between raw Git diffs and the LLM agent. It provides a worked example of file prioritization and a controlled dense-embedder comparison.

Diff render layer

  • The diff-render layer has three stages and sits between a raw Git diff and the LLM agent’s read_commit tool.Algorithm 2 gives the full pipeline; Table 8 walks through a real commit.Source: §A.3 · p. 13

Dense embedder comparison

  • Table 7 compares retrieval-only Recall@K across dense embedders on the 809-CVE subset while keeping BM25, RRF fusion, and the Phase 2 agent unchanged.The encoding pipeline was validated byte-for-byte against sentence-transformers, with cosine similarity approximately 1.0.Source: Table 7 · p. 14, §A.3 · p. 14

Worked example

  • In the CVE-2015-9251 example, src/ajax/script.js receives the highest file priority, while a test file is de-prioritized despite its 48-line addition.The source file pairs the source tag with the largest nontest edit; the test tag carries a much smaller base priority.Source: Table 8 · p. 14

A.4 System Prompt

The system prompt gives the agent a general rank-frequency hint intended to counter over-trusting the first candidate, without revealing which candidate is the fix. Listwise candidate viewing yields the largest single tool-ablation gain reported here.

System prompt

  • The Phase 2 agent runs with the verbatim system prompt shown in Listing 1.Source: §A.4 · p. 14

Tool ablation

  • Adding listwise candidate viewing raises Recall@1 from 32.63% to 48.21% on identical candidates, the largest single step in the tool ablation.Source: §A.4 · pp. 14–15

Rank hint

  • The Rank Hint gives rough aggregate fix frequencies by rank to counter agents’ tendency to over-trust the first list item, but does not identify a CVE’s fix.Its quoted frequencies do not match the final retriever on the test subset, where Phase 1 puts the fix at rank 1 for 32.63% of CVEs.Source: §A.4 · p. 14

Rank hint caveat

  • The prompt’s rank-frequency numbers are not calibrated to the test subset, and the authors did not rerun experiments with the numbers removed.They state that the instruction without numbers would serve the same purpose, but report the prompt as used.Source: §A.4 · p. 14

A.5 IRCoT Scale-Stability Check

IRCoT results on the 8,401-CVE source corpus remain close to those on the 809-CVE GitHubAD subset, supporting scale stability of the baseline.

A.5 IRCoT Scale-Stability Check

  • The largest GitHubAD-to-source-corpus IRCoT score swing is 1.36 percentage points, and the authors conclude the baseline is scale-stable.They state that the head-to-head comparison reflects corpus-scale performance rather than a sampling artifact.Source: §A.5 · p. 15
  • With Qwen3-235B, IRCoT reaches 28.55% Recall@1 on GitHubAD and 29.91% on the full corpus.The corresponding Qwen3-Coder-30B values are 29.67% and 30.33%.Source: §A.5 · p. 15
  • Across both backbones, IRCoT’s full-corpus Recall@1 is close to its GitHubAD result.The reported values are 28.55% versus 29.91% for Qwen3-235B and 29.67% versus 30.33% for Qwen3-Coder-30B.Source: §A.5 · p. 15

A.6 Statistical Significance of the Main Result

The appendix reports statistical support for the main comparison, Phase 1’s retrieval ceiling, and results on real-world CVEs outside the benchmark. It also documents evaluation conventions, dataset scope, and use boundaries.

A.6 Statistical Significance of the Main Result

  • Every pairwise method gap is significant at every cutoff; PatchHolmes’s Recall@1 confidence interval does not overlap Favia’s or IRCoT’s.The weakest pairwise result is PatchHolmes versus Favia at Recall@5 (p = 1.4×10−3).Source: §A.6 · p. 15, Table 11 · p. 17

A.7 Phase 1 Recall Ceiling

  • 87.64% Recall@100 bounds the full system, while Phase 2 raises Recall@1 from Phase 1’s 32.63% to 59.95%.The remaining 12.36% of gold commits are unreachable because they fall outside the top 100.Source: §A.7 · p. 15, Table 12 · p. 18

A.8 Real-World CVEs Outside the Benchmark

  • On 35 external CVEs, PatchHolmes reaches 60.0% Recall@1, versus 22.9% for Phase 1 rank order alone.The dense retrieval path raises top-100 coverage from BM25 with time decay’s 45.7% to 77.1%.Source: §A.8 · p. 15, Table 13 · p. 19

A.9 Fix Multiplicity in the Ground Truth

  • 92.72% of source-corpus CVEs and 93.08% of working-subset CVEs have exactly one recorded fix commit.For the remaining cases, any recorded fix commit is accepted as correct.Source: §A.9 · p. 15, Table 14 · p. 19

A.10 Baseline Configurations

  • Favia ranks ten candidates using per-candidate yes/no answers and confidence, whereas IRCoT uses up to four reason-and-retrieve cycles.Favia can fetch NVD and CWE metadata; the IRCoT matched-candidate control uses PatchHolmes’s Phase 1 top-100 manifest.Source: §A.10 · p. 15

A.11 Metric Conventions

  • Recall@K measures whether the gold commit appears in the top K, while NDCG@K and MRR additionally reflect its rank.For fairness, PatchHolmes’s inspection list is extended to ten when compared with Favia’s ten-deep output; hyperparameters use codebase defaults without per-CVE tuning.Source: §A.11 · pp. 17–18, Table 16 · p. 20

A.13 Artifacts: Provenance, Licensing, and Intended Use

  • GitHubAD covers 8,401 CVEs across 2,816 repositories and seven languages; PatchFinder_top10 contains 1,252 CVEs with ten candidates each.The PatchFinder_top10 dataset has no explicit license and is used for fair-use academic comparison; the artifacts are intended for research use.Source: §A.13 · p. 18, §A.14 · p. 18

A.15 Potential Risks

  • PatchHolmes outputs are advisory: authors recommend human review because an incorrect commit could prompt a false downstream reachability claim.The authors report using generative AI for typesetting and wording polish, not for research design, data, or evaluation.Source: §A.15 · pp. 18–19, §A.16 · p. 19
Loading 2609.38807v1…