Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection

1 Introduction

• RAG dynamically retrieves external knowledge, such as CWE descriptions or CVE reports, and adds it to an LLM prompt without retraining or fine-tuning.The knowledge base can be updated independently of the model.Source: §1 · p. 1
• A typical RAG4SVD pipeline transforms code into a retrieval query, retrieves relevant knowledge, then combines that knowledge with the code for vulnerability detection.Queries may use raw code or a semantic summary, and detection produces a prediction such as a binary label.Source: §1 · pp. 1–2
• RAG4SVD results are hard to generalize because studies lack independent reproduction, use incompatible evaluation settings, and usually assess systems only end-to-end.These gaps leave cross-system comparison and attribution of observed performance improvements uncertain.Source: §1 · p. 2
• The study jointly examines reproducibility, cross-system performance, and individual pipeline components across open-source RAG4SVD systems.Its component analysis separates input abstraction, knowledge retrieval, and detection, including whether abstractions preserve useful retrieval information and whether oracle knowledge improves detection.Source: §1 · pp. 2–3

2.1 Benchmarking LLM-based Vulnerability Detection

• Existing LLM vulnerability benchmarks compare prompting and tuning, while PrimeVul addresses dataset and protocol concerns and motivates a common benchmark for RAG-system evaluation.Ding et al. [7] report state-of-the-art code-model F1 scores as low as 0.03 on PrimeVul.Source: §2.1 · p. 3

2.2 Fragmented Evaluation of Retrieval-Augmented Vulnerability Detection

• RAG4SVD studies often use custom protocols; prior reproducibility work examined Vul-RAG [8] alone, while other comparisons or ablations leave generalization unresolved.Kaniewski et al. [18] did not compare multiple RAG architectures, transfer them to a new dataset, or evaluate individual pipeline components.Source: §2.2 · pp. 3–4
• A survey of 30 RAG studies identifies fragmented evaluation, and this work applies a common dataset, metric suite, and model pool to RAG4SVD systems.Chen et al. highlight fragmentation across datasets, baselines, models, and metrics.Source: §2.2 · p. 4

2.3 RAG Evaluation Methodology and Component Analysis

• Existing RAG evaluation frameworks target free-text question-answering rather than code-based retrieval and structured vulnerability knowledge.Ke et al. [19] assess components through final task performance; this study adds intrinsic evaluation of representative RAG4SVD components, including retrieval relevance.Source: §2.3 · p. 4

3 Experimental Setup

• The research questions examine reproducibility, comparison under a unified benchmark, and component-level factors in representative RAG4SVD pipelines.The corresponding evaluations reproduce reported settings, apply common data, metrics, and models, then assess abstraction, retrieval, and detection individually.Source: §3 · p. 4

3.1 RAG4SVD System Selection

• Of 31 surveyed studies, 12 had public artifacts and six had complete executable setups; the study selected systems meeting artifact and reconstruction criteria.The selection process is summarized in Fig. 2.Source: §3.1 · p. 5, Figure 2 · p. 5
• The selected systems span dense example retrieval, hybrid retrieval, and vulnerability-knowledge retrieval, with differing query representations and retrieved context.SVD-Bench uses similar labeled functions [45], while LLM4Vuln [37], Vul-RAG [8], and VulTriage [39] retrieve vulnerability knowledge.Source: §3.1 · pp. 5–6, Table 1 · p. 5
• Llama-VD retrieves six nearest training examples and explores test-time tuning; GRACE reranks dense candidates using lexical and AST similarity before selecting one example.Llama-VD uses codebert-base and Euclidean distance; GRACE uses CodeT5 for dense retrieval.Source: §3.1 · p. 5

3.2 Datasets

• The unified benchmark uses PrimeVul Paired, comprising 5,480 vulnerable-patched pairs across 88 CWEs, with training data for the knowledge base and test data for evaluation.The original datasets and splits are retained for reproduction; component-analysis retrieval uses sampled test-set probes.Source: §3.2 · p. 6

3.3 Metrics

• Detection is measured with Accuracy, Precision, Recall, F1, and Pairwise Accuracy, which counts a pair correct only when both vulnerable and patched code are classified correctly.Retrieval Hit@k and MRR treat exact or parent/child ground-truth CWE agreement as a successful match and measure its presence and rank.Source: §3.3 · pp. 6–7

3.4 Model Selection

• The unified evaluation uses 22 open-weight models from seven families, spanning approximately 3B to 32B parameters and including general-purpose and code-specialized models.The scales are selected for practical local and on-premises deployment.Source: §3.4 · p. 7

3.5 Implementation Details

• Systems are integrated into a local inference pipeline that replaces API calls while preserving system logic, prompts, retrieval configurations, and hyperparameters where possible.Parsers were hardened for output-format variation; GPT-oss-120B via API is used only as the abstraction-quality judge, not for retrieval or detection.Source: §3.5 · p. 7

4.1 RQ1 Reproducibility

• 58.2% recall deviation on Python marks SVD-Bench’s largest reproduction gap; deviations exceed 5% for Python and Java, while selected JavaScript metrics are weakly reproducible.Llama-VD’s metrics deviate by nearly all less than 1% across Big-Vul and PrimeVul; Vul-RAG’s accuracy and recall deviations are below 1%, while precision deviates by 2.5%.Source: §4.1 · pp. 8–9
• More than 5% deviation makes LLM4Vuln not reproducible in the fully open-weight setting, where proprietary summarization and judging components require substitutes.The reported comparison reflects model substitution rather than reproduction of the original configuration.Source: §4.1 · pp. 8–9
• Reproducibility varies substantially across systems, and reproducing systems that require substituting proprietary models in an open-weight setting is unattainable.The authors identify parser hardening and inference non-determinism as potential contributing factors.Source: §4.1 · p. 9

4.2 RQ2 Unified Benchmark

• 0.20 mean pairwise accuracy for Vul-RAG leads the system averages, followed by VulTriage (0.12), SVD-Bench and Llama-VD (0.08), and GRACE (0.03).The system averages use the intersection of models that yielded predictions for every system.Source: Table 2 · p. 9, §4.2 · p. 10

4.3 RQ3 Component Analysis

• The component analysis focuses on Vul-RAG and LLM4Vuln, whose separable abstraction, retrieval, and detection stages support component-level attribution.The analysis complements end-to-end evaluation, which leaves unclear which components drive observed improvements.Source: §4.3 · p. 10

4.3.1 Input Abstraction.

• Vul-RAG summaries score higher in Faithfulness, Functional Coverage, and Retrieval Utility, while LLM4Vuln summaries are more concise and clear, reflecting their different prompt objectives.Judged abstraction quality is driven more strongly by system-specific summarization prompts than by the code being summarized.Source: §4.3.1 · pp. 11–12
• Vul-RAG’s judge agreement is low (α=0.16–0.38); LLM4Vuln has stronger agreement for Functional Coverage (α=0.63) and Retrieval Utility (α=0.59), but other dimensions remain judge-sensitive.Judges preserve LLM4Vuln’s profile, while judge choice has a stronger effect on Vul-RAG’s individual scores.Source: §4.3.1 · p. 12

4.3.2 Knowledge Retrieval.

• Retrieval evaluation separates knowledge coverage, or whether relevant knowledge is present, from retrieval effectiveness, measured by Hit@k and MRR.This distinction identifies failures associated with missing knowledge alignment versus ineffective retrieval.Source: §4.3.2 · p. 12

Vul-RAG LLM4Vuln

• Vul-RAG retrieval is stronger on its aligned PairVul probe (mean MRR 0.54, Hit@3 0.66) than PrimeVul (mean MRR 0.35, Hit@3 0.43), while LLM4Vuln retrieval is weak across probes.LLM4Vuln records MRR 0.07 and Hit@3 0.12 for PrimeVul; its knowledge base overlaps with only 6 of 23 UniVul test-set CWEs and 60 of 920 entries.Source: Vul-RAG LLM4Vuln · pp. 13–14
• For Vul-RAG, oracle MRR reaches 0.93 for PairVul and 0.95 for PrimeVul when the knowledge base and test set are aligned.The oracle knowledge base guarantees that a ground-truth item is present in the retrieval pool.Source: Vul-RAG LLM4Vuln · p. 14
• Across tested generators, query abstraction model affects Vul-RAG retrieval more than knowledge-base generation model; PairVul-probe MRR ranges from 0.37 to 0.60 across abstraction models.Increasing generation-model scale yields only a slight PairVul retrieval change (MRR 0.56 → 0.58; Hit@3 0.66 → 0.68) and no generalizable benefit, with retrieval flat on PrimeVul.Source: Vul-RAG LLM4Vuln · p. 14

4.3.3 Vulnerability Detection.

• Detection accuracy rises with Vul-RAG knowledge-base size through 75% but drops at 100%: PairVul 0.20 → 0.24 → 0.26 and PrimeVul 0.17 → 0.19 → 0.23.The authors hypothesize that near-duplicate, low-relevance, or lower-quality entries may dilute top-k retrieval precision at full scale.Source: §4.3.3 · p. 14
• With Qwen2.5-Coder-14B as generator and detector, PairVul pairwise accuracy reaches 0.36 versus 0.23 with the original knowledge base, but performance is not monotonic in generator scale.The 14B generator scores 0.36, compared with 0.32 for 32B and 0.28 for 3B; PrimeVul-derived knowledge bases reach 0.22–0.24 regardless of test set.Source: §4.3.3 · pp. 14–15
• Mean pairwise accuracy reaches 0.51 on PrimeVul under oracle knowledge, but the strongest detector reaches 0.73 and Llama3.2-3B only 0.13.Even with favorable retrieval conditions, detector capability remains an independent bottleneck.Source: §4.3.3 · p. 15
• The component results point to alignment across abstraction, retrieval, knowledge coverage, and detection as more important than scaling models or knowledge bases alone.Detection performance is influenced more by supplied-knowledge alignment than by knowledge-base size or generator scale.Source: §4.3.3 · p. 16

5 Threats to Validity

• Implementation differences, parser hardening, and model substitutions may affect reproduced results despite preserving system logic, prompts, and hyperparameters where possible.The authors document adaptations and repeat experiments where applicable.Source: §5 · p. 16
• CWE agreement may not indicate semantic or vulnerability-specific relevance, so retrieval metrics are interpreted as knowledge alignment rather than actionable knowledge quality.Excluding configurations with fewer than 95% parseable predictions may favor systems with more robust output formats.Source: §5 · p. 16
• Findings are limited to six artifact-complete systems, PrimeVul Paired, 3B–32B open-weight models, and two representative multi-stage pipelines.Results may differ for proprietary models, other programming languages, or vulnerability data distributions.Source: §5 · p. 16

6 Conclusion

• Oracle knowledge achieves MRR of 0.93–0.95, yet mean pairwise accuracy remains 0.51, showing that near-optimal retrieval does not solve detection.The conclusion recommends optimizing abstractions for retrieval utility and establishing knowledge-base coverage and alignment with the target distribution.Source: §6 · p. 16
• The authors conclude that progress is unlikely to come from scaling individual components alone and recommend evaluating abstraction, knowledge-base coverage, retrieval, parsing robustness, and downstream knowledge use.They frame performance as governed by alignment across component interfaces.Source: §6 · p. 17

A Dataset Statistics and Sampling

• The Vul-RAG-matched PrimeVul*PairVul sample contains 46 vulnerable functions; the LLM4Vuln-matched PrimeVul*UniVul sample contains 42 vulnerable functions across 18 overlapping CWEs.PrimeVul*PairVul samples five instances per PairVul CWE where available; PrimeVul*UniVul matches UniVul per-CWE counts where possible.Source: A Dataset Statistics and Sampling · p. 17

• The model pool spans multiple model families, parameter scales, and specializations.Source: B Model Lineup · p. 17

• Table 6 reports absolute reproduction results for the systems evaluated in RQ1.The table supplies the absolute values underlying the RQ1 reproducibility analysis.Source: C Full Reproduction Results · p. 17, Table 6 · p. 18
• Table 6 marks reported results, individual reproduction runs, and the mean across runs with distinct symbols.Source: Table 6 · p. 18
• SVD-Bench uses a seed and yields identical results across all three reproduction runs.Source: Table 6 · p. 18
• For Llama-VD, precision, recall, and F1 refer to the vulnerable class.Source: Table 6 · p. 18