PaperlayerView exact revision on arXiv

Source-linked AI summary

ReVul-CoT: Towards Effective Software Vulnerability Assessment with Retrieval-Augmented Generation and Chain-of-Thought Prompting

Zhijie Chen, Xiang Chen, Ziming Li, Jiacheng Xue, Chaoyang Gao

arXiv:2511.17027v1cs.SE

TL;DR

LLMs applied to software vulnerability assessment can lack domain-specific knowledge and structured reasoning for complex vulnerabilities. ReVul-CoT combines retrieval from an NVD- and CWE-enriched knowledge base with Chain-of-Thought prompting, and outperforms the best baseline across Accuracy, F1-score, and MCC.

  • Problem

    LLM-based SVA can lack vulnerability-domain knowledge and structured reasoning for complex code and descriptions, despite the importance of accurate severity assessment.

    Source: §1 · p. 1, §2.4 · p. 4

  • Method

    ReVul-CoT dynamically retrieves relevant information from a local knowledge base enriched with NVD and CWE data and uses CoT prompting to reason about exploitability and impact.

    Source: §1 · p. 2

  • Results

    10.43, 15.86, and 16.5 percentage-point gains over the best baseline in Accuracy, F1-score, and MCC, respectively, accompany ReVul-CoT's 87.50% Accuracy, 83.75% F1-score, and 79.51% MCC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support combining RAG with CoT reasoning as a promising direction for software vulnerability assessment and related tasks.

    Source: §8 · pp. 16–17

    The evaluation dataset primarily covers C and C++ vulnerabilities, and some recent methods were excluded because source code or proprietary implementations were unavailable.

    Source: §6.3 · p. 16

Abstract

from arXiv · show

Context: Software Vulnerability Assessment (SVA) plays a vital role in evaluating and ranking vulnerabilities in software systems to ensure their security and reliability. Objective: Although Large Language Models (LLMs) have recently shown remarkable potential in SVA, they still face two major limitations. First, most LLMs are trained on general-purpose corpora and thus lack domain-specific knowledge essential for effective SVA. Second, they tend to rely on shallow pattern matching instead of deep contextual reasoning, making it challenging to fully comprehend complex code semantics and their security implications. Method: To alleviate these limitations, we propose a novel framework ReVul-CoT that integrates Retrieval-Augmented Generation (RAG) with Chain-of-Thought (COT) prompting. In ReVul-CoT, the RAG module dynamically retrieves contextually relevant information from a constructed local knowledge base that consolidates vulnerability data from authoritative sources (such as NVD and CWE), along with corresponding code snippets and descriptive information. Building on DeepSeek-V3.1, CoT prompting guides the LLM to perform step-by-step reasoning over exploitability, impact scope, and related factors Results: We evaluate ReVul-CoT on a dataset of 12,070 vulnerabilities. Experimental results show that ReVul-CoT outperforms state-of-the-art SVA baselines by 16.50%-42.26% in terms of MCC, and outperforms the best baseline by 10.43%, 15.86%, and 16.50% in Accuracy, F1-score, and MCC, respectively. Our ablation studies further validate the contributions of considering dynamic retrieval, knowledge integration, and CoT-based reasoning. Conclusion: Our results demonstrate that combining RAG with CoT prompting significantly enhances LLM-based SVA and points out promising directions for future research.

1. Introduction

Software vulnerability assessment helps organizations prioritize security flaws, but existing approaches and LLMs can lack broad domain knowledge and structured reasoning. ReVul-CoT combines dynamic retrieval with Chain-of-Thought prompting and outperforms the best baseline across reported metrics.

1. Introduction

  • Prior SVA methods often rely on one information source, while LLM-based approaches may lack vulnerability-domain knowledge and structured reasoning about complex cases.SVA supports prioritizing security flaws before potential exploitation.Source: §1 · p. 1
  • ReVul-CoT dynamically retrieves relevant knowledge from NVD and CWE alongside vulnerability code and descriptions, then uses CoT to reason step by step about exploitability and impact.The structured reasoning is intended to make severity decisions more transparent and traceable.Source: §1 · p. 2
  • The evaluation uses 12,070 vulnerability records, a local knowledge base built from NVD and CWE data, CVSS v3, and DeepSeek-V3.1.The dataset includes vulnerability descriptions and corresponding source code.Source: §1 · p. 2
  • 87.50% Accuracy, 83.75% F1-score, and 79.51% MCC exceed the best baseline by 10.43, 15.86, and 16.5 percentage points, respectively.Ablations validate contributions from the local knowledge base, retrieval configuration, and CoT prompting.Source: §1 · p. 2

2. Research Background and Research Motivation

The background describes vulnerability assessment and the roles of retrieval and stepwise reasoning; the motivation identifies gaps in existing SVA and LLM-based approaches that ReVul-CoT aims to address.

2.1. Software Vulnerability Assessment

  • SVA scores vulnerabilities using impact and exploitability to prioritize remediation; CVSS v3.0 maps scores to Low, Medium, High, and Critical severity categories.Manual evaluation is increasingly impractical as vulnerabilities grow in volume and complexity.Source: §2.1 · pp. 2–3

2.2. Retrieval-Augmented Generation

  • RAG retrieves relevant records through similarity matching and supplies them with the query to the LLM, allowing external vulnerability knowledge to complement code and descriptions.The background emphasizes authoritative sources such as CWE and NVD as a way to address outdated or incomplete model knowledge.Source: §2.2 · p. 3

2.3. Chain-of-Thought Prompting

  • CoT prompting breaks a task into intermediate reasoning stages; in SVA, the cited comparison reports correct severity assessment with CoT but misclassification with single-step prompting.Few-shot CoT uses static exemplars, whereas zero-shot CoT presents reasoning steps without exemplars [31].Source: §2.3 · pp. 3–4, Figure 1 · p. 4

2.4. Research Motivation

  • Existing SVA methods may integrate code and text shallowly, while LLM-based SVA can lack domain expertise and step-by-step analysis; the motivation proposes combining RAG with CoT to address both gaps.RAG contributes domain-relevant information, while CoT guides structured reasoning about exploitability and severity.Source: §2.4 · pp. 4–5

3. Our Proposed Framework ReVul-CoT

ReVul-CoT builds a vulnerability knowledge base, retrieves similar cases using code and description embeddings, and uses the retrieved context to guide CoT severity assessment.

3.1. Overview

  • The framework has three phases: knowledge-base construction, embedding and retrieval, then knowledge-augmented severity assessment with CoT prompting.The LLM reasons over retrieved knowledge and the target vulnerability to predict a severity level.Source: §3.1 · p. 5

3.2. Phase 1: Vulnerability Knowledge Base Construction

  • The knowledge base enriches CVE entries with NVD CVSS metrics and CWE definitions alongside code, descriptions, and severity labels.Entries are normalized into structured JSON containing technical details and semantic information.Source: §3.2 · pp. 5–6

3.3.1. Embedding Part

  • CodeBERT and CoSENT map vulnerability code and descriptions into a shared embedding space for similarity-based retrieval.CodeBERT encodes code, while CoSENT encodes descriptions.Source: §3.3 · p. 6, §3.3.1 · p. 6

3.3.2. Retrieval Part

  • Retrieval ranks candidates using cosine similarities for code and descriptions, combines them with a tunable weight, and selects top-k samples.The retrieved samples are passed to the LLM for severity assessment.Source: §3.3.2 · pp. 6–7

3.4. Phase 3: Knowledge-Augmented Severity Assessment with CoT Prompting

  • For each target, dynamically retrieved examples supply few-shot CoT demonstrations instead of relying on a fixed exemplar set.The examples include vulnerability code, descriptions, CWE categorizations, and CVSS metrics.Source: §3.4 · p. 7
  • The prompt analyzes exemplars, examines the target’s impact scope and exploitability, then synthesizes the analysis into LOW, MEDIUM, HIGH, or CRITICAL severity.The assessment also considers CWE categorizations, CVSS metrics, and broader security context.Source: §3.4 · pp. 7–8

4. Experimental Setup

The experimental setup compares ReVul-CoT with representative SVA baselines and examines the use of both code and vulnerability descriptions.

4.1. Research Questions

  • RQ1 asks whether ReVul-CoT surpasses state-of-the-art software vulnerability assessment baselines.The comparison covers three baseline categories: source-code, description-based, and bimodal approaches.Source: §4.1 · p. 8
  • The study evaluates ReVul-CoT against source-code, description-based, and bimodal SVA baselines using Accuracy, Macro-F1, and MCC.The baseline set includes FuncRF, FuncLGBM, Cvss-bert, SPSV, SVA-CL, MTLM, and SVA-ICL.Source: §4.1 · p. 8
  • RQ2 tests whether combining source code and vulnerability descriptions improves performance and which modality-fusion ratio works best.The study treats code structure and description semantics as complementary information.Source: §4.1 · pp. 8–9

RQ3: How does the number of similar samples in RAG retrieval affect the performance of ReVul-CoT?

RQ3 examines how the number of retrieved samples affects ReVul-CoT’s performance, balancing context and relevance against noise and inference constraints.

RQ3: How does the number of similar samples in RAG retrieval affect the performance of ReVul-CoT?

  • Too few retrieved samples may provide limited context, while too many may add irrelevant or noisy information.RQ3 varies the number of retrieved samples to measure its effect on performance.Source: RQ3: How does the number of similar… · p. 9
  • The retrieval count must also account for DeepSeek-V3.1’s token-length constraints and reasoning efficiency.The study frames retrieval quantity as a balance between contextual richness and efficient reasoning.Source: RQ3: How does the number of similar… · p. 9
  • RQ3 tests how changing top-k retrieval quantity influences ReVul-CoT’s assessment performance.The motivation is to balance the amount of retrieved knowledge against its relevance.Source: RQ3: How does the number of similar… · p. 9

RQ4: How does the use of Chain-of-Thought prompting affect the performance of ReVul-CoT?

The experimental materials and implementation specify the study’s questions, data processing, metrics, baselines, and inference settings.

RQ4: How does the use of Chain-of-Thought prompting affect the performance of ReVul-CoT?

  • RQ4 investigates whether CoT prompting improves vulnerability assessment accuracy, while RQ5 examines external knowledge enhancement.The motivations distinguish structured reasoning from added domain knowledge as separate factors.Source: RQ4: How does the use of… · p. 9

4.2. Experimental Subject

  • The study re-collected CVSS v3 scores, vectors, and descriptions for MegaVul records, retaining 12,070 entries with complete CVSS v3 information.The analysis uses CVSS v3 because practical datasets annotated with CVSS v4 scores remain extremely limited.Source: §4.2 · p. 9
  • To prevent leakage, the knowledge base uses only the 80% knowledge split; validation and test samples are excluded from retrieval.Stratified sampling creates knowledge, validation, and test sets at 80%, 10%, and 10%.Source: §4.2 · p. 9

4.3. Performance Measures

  • Performance is measured with Accuracy, macro-averaged F1, and MCC, with the latter two used for the multiclass, imbalanced task.The text characterizes MCC as effective for imbalanced datasets.Source: §4.3 · pp. 9–10

4.4. Baselines

  • The baseline comparison spans source-code, description-based, and bimodal SVA approaches, including machine-learning and deep-learning methods.The study identifies seven state-of-the-art baselines across these categories.Source: §4.4 · p. 10

4.5. Implementation Details and Running Platform

  • Retrieval weights code similarity at 60% and description similarity at 40%, and few-shot CoT examples come from retrieved samples.The LLM temperature is set to 0.Source: §4.5 · p. 11
  • Experiments ran on a Windows 10 server with an Intel Core i7-13600K processor and a GeForce RTX 4090 GPU with 24GB of graphics memory.The paper describes this configuration as supporting large-scale vulnerability-data processing and analysis.Source: §4.5 · p. 11

5. Results

The results report that ReVul-CoT outperforms state-of-the-art baselines and evaluate modality fusion, retrieval count, Chain-of-Thought prompting, and external knowledge. Balanced code/text similarity and five retrieved samples perform best, while CoT prompting and external knowledge improve performance.

5.1. RQ1: How does our proposed method ReVul-CoT perform in software vulnerability assessment compared to state-ofthe-art baselines?

  • ReVul-CoT achieves 87.50% Accuracy, 83.75% F1-score, and 79.51% MCC, with gains of at least 10.43, 15.86, and 16.50 percentage points, respectively, over the strongest baselines.The comparison covers state-of-the-art methods across all three metrics.Source: §5.1 · pp. 11–12, Table 2 · p. 12

5.2. RQ2: Whether considering both source code and vulnerability description with information can improve the performance of ReVul-CoT?

  • At a 60%:40% code-to-text similarity ratio, ReVul-CoT achieves 87.50% Accuracy, 83.75% F1-score, and 79.51% MCC.Performance degrades when either modality is used alone; the authors describe code structure and descriptive semantic context as complementary.Source: Table 3 · p. 12, §5.2 · p. 12
  • In two representative cases, single-modality settings predict incorrect severity, while combining code and description at the best ratio produces correct assessments.The cases compare High versus ground-truth Medium, and High versus ground-truth Critical; source-code similarity helps in the latter case.Source: §5.2 · p. 12, Figure 4 · p. 13

5.3. RQ3: How does the number of similar samples in RAG retrieval affect the performance of ReVul-CoT?

  • Five retrieved samples yield the best overall performance; performance declines beyond five as redundancy and interference can introduce noise.The study tests zero-shot, 3-shot, 5-shot, and 7-shot settings with code-to-text similarity fixed at 60%:40%.Source: Table 4 · p. 13, §5.3 · p. 13

5.4. RQ4: How does the use of Chain-of-Thought prompting

  • CoT prompting improves Accuracy, F1-score, and MCC; the authors attribute the improvement to intermediate reasoning that shifts attention from shallow patterns to vulnerability-attribute relationships.They also report more stable reasoning for ambiguous cases, fewer hallucinations, and improved confidence calibration.Source: §5.4 · pp. 13–14, Table 5 · p. 14

5.5. RQ5: How does external knowledge enhancement affect the performance of ReVul-CoT?

  • External knowledge improves severity-assessment performance and supports judgments with authoritative domain knowledge rather than linguistic patterns alone.The authors note reduced reasoning ambiguity, particularly when vulnerability descriptions are incomplete or unclear.Source: Table 6 · p. 14, §5.4 · p. 14

6. Discussions

The discussion compares DeepSeek-V3.1 with other LLMs, examines ReVul-CoT’s input-token costs, and outlines threats to validity involving data exposure, reproducibility, dataset coverage, and evaluation scope.

6.1. Comparison with Other Popular LLMs

  • DeepSeek-V3.1 achieves the best overall performance across all evaluation metrics among the compared LLMs.Gemini-2.5-Pro ranks second; the discussion attributes DeepSeek-V3.1’s performance to its context window, code analysis, language understanding, and reasoning capabilities.Source: Table 7 · p. 14, §6.1 · pp. 14–15

6.2. Evaluation of Token Usage

  • Under the Top-k = 5, 6:4 code-text fusion, CoT-enabled configuration, average input consumption is approximately 8,246 tokens, with a 2,505-token minimum and 68,666-token maximum.Retrieved context accounts for an average of 87% of total input tokens; output tokens are excluded from the analysis.Source: §6.2 · p. 15, Figure 5 · p. 15

6.3. Threats to Validity

  • Potential overlap between pretraining data and test samples could permit recall of prior patterns, although weak zero-shot performance suggests leakage had extremely low impact.The authors identify publicly accessible training data as the source of this internal threat.Source: §6.3 · p. 15
  • The study’s results may be difficult to reproduce across LLM API versions or providers, and its dataset covers publicly available CVE records rather than unreported or proprietary-system vulnerabilities.Performance may fluctuate for vulnerability types with few historical samples.Source: §6.3 · p. 15
  • The evaluation uses Accuracy, F1-score, and MCC, while prompt-design choices may shape reasoning chains; the authors ground prompts in CVSS criteria and CWE taxonomies.The selected metrics capture complementary perspectives on classification performance.Source: §6.3 · pp. 15–16
  • The study’s dataset focuses primarily on C and C++ vulnerabilities, while unavailable or proprietary implementations prevented inclusion of some recent comparison methods.The authors note these boundaries may limit generalization and introduce bias in comparative conclusions.Source: §6.3 · p. 16

7. Related Work

Prior software vulnerability assessment work spans description-based, source-code-based, and bimodal approaches. The paper identifies limited reasoning transparency and domain knowledge in LLM-based SVA, and presents ReVul-CoT as an integration of retrieval and chain-of-thought reasoning.

Related SVA approaches

  • SVA research is grouped into description-based, source-code-based, and bimodal approaches, with bimodal methods combining code and vulnerability descriptions.Examples include text classifiers, code- and graph-based analysis, and models that fuse code with textual descriptions.Source: §7 · p. 16

LLMs, RAG, and CoT

  • Existing LLM-based SVA methods face knowledge limitations and limited reasoning transparency, while RAG and chain-of-thought prompting address these concerns in other domains.Their integration into SVA remains largely unexplored.Source: §7 · p. 16

ReVul-CoT

  • ReVul-CoT combines NVD- and CWE-enriched retrieval with source code and vulnerability descriptions, then guides structured reasoning under standardized CVSS criteria.The framework evaluates exploitability, impact scope, and contextual factors; the authors report that it is the first to integrate RAG with CoT prompting for SVA [28].Source: §7 · p. 16

8. Conclusion and Future Work

ReVul-CoT combines retrieval-augmented generation with chain-of-thought prompting for software vulnerability assessment. The authors report superior performance to existing SVA baselines and identify future work on knowledge, retrieval, prompting, and language coverage.

Conclusion

  • ReVul-CoT dynamically retrieves NVD- and CWE-enriched context for DeepSeek-V3.1 and uses standardized CVSS criteria for structured vulnerability assessment.Evaluations report superior performance to existing SVA baselines across metrics, and ablations identify retrieval and CoT reasoning as essential to overall effectiveness.Source: §8 · pp. 16–17

Future Work

  • Future work includes expanding knowledge sources, improving retrieval efficiency and accuracy, refining CoT prompts, and validating the framework across programming languages.The authors also propose applying it to vulnerability remediation, exploit prediction, and patch generation.Source: §8 · p. 17

CRediT authorship contribution statement

The contribution statement assigns conceptualization and methodology across Zhijie Chen and Xiang Chen, with software, validation, and data curation contributions from several authors.

Author contributions

Loading 2511.17027v1…