Source-linked AI summary
Answer Is Cheap, Show Me the Evidence! Augmenting Automated Vulnerability Assessment with Evidence
Shengyi Pan, Zelong Zheng, Jiayuan Zhou, Xing Hu, Xin Xia, Shanping Li
TL;DR
Automated software-vulnerability assessment often lacks supporting evidence and overlooks rich contextual information. EAVA uses specialized LLM agents and dedicated model training to assess vulnerabilities with evidence, outperforming the best baselines by 5.3%-35.2% across various measures.
Problem
Existing automated vulnerability assessment methods focus on accuracy but often lack supporting evidence needed for analysts to validate results guiding remediation schedules.
Method
EAVA combines specialized LLM agents for rich-text and project analysis with a dedicated assessment model trained through supervised instruction tuning and reinforcement learning.
Results
5.3%-35.2% improvements over the best-performing baselines across various measures were achieved, while a user study confirmed improved assessment efficiency.
Takeaways & Limitations
EAVA's explicit reasoning and supporting evidence can help security analysts improve software-vulnerability assessment efficiency.
Takeaways & Limitations
EAVA's evaluations use NVD CVSS values as labels, although those values may be inaccurate and vary across vulnerability databases.
Abstract
from arXiv · showhide
Software vulnerability (SV) assessment helps prioritize remediation by characterizing reported vulnerabilities. Existing automated methods predict assessment results from SV reports (SVRs), but often overlook information in rich text, such as screenshots and code snippets, as well as contextual information about vulnerable projects. They also focus on prediction accuracy without providing explanations or supporting evidence, limiting their practical use when analysts must validate imperfect predictions. We propose EAVA, a framework that uses large language models (LLMs) to assess SVs and provide supporting evidence. EAVA employs specialized LLM agents to process rich-text content and project information, and builds a dedicated assessment model through a two-stage training pipeline. It first uses supervised instruction tuning on automatically annotated reasoning trajectories to inject domain knowledge, and then applies reinforcement learning to improve intrinsic reasoning. EAVA also retrieves similar historical vulnerabilities as supplementary evidence. Experiments on a newly collected SVR dataset show that EAVA outperforms the strongest baseline by 5.3 to 35.2 percent across multiple metrics. Ablation studies confirm the effectiveness of assessment-specific model training and information enrichment. A user study with security experts further demonstrates that the evidence provided by EAVA is useful and practical for real-world SV assessment.
1 Introduction
EAVA addresses limitations in automated software-vulnerability assessment by combining specialized LLM agents, assessment-specific model training, and explicit supporting evidence. On a new dataset, it outperforms existing baselines and shows practical usefulness in a user study.
- Motivation: SV assessment prioritizes critical vulnerabilities for remediation, with CVSS serving as the most widely used assessment standard.Assessment characterizes detected vulnerabilities to locate hotspots and support remediation prioritization.
- Limitations: Existing automated approaches classify vulnerability reports into metric values but remain imperfect, making manual validation and curation necessary.Prior work predominantly emphasizes prediction accuracy without fully resolving practical assessment limitations.
- EAVA framework: EAVA uses three specialized LLM agents to analyze embedded code snippets and screenshots and retrieve information about vulnerable projects.The framework enriches vulnerability-report information before assessment.
- EAVA framework: EAVA constructs a dedicated assessment LLM that uses assessment-specific knowledge and structured reasoning to analyze enriched vulnerability information and provide supporting evidence.Its training includes large-scale dataset annotation, instruction tuning, and reinforcement learning.
- Evaluation: 6,446 SVRs from 1,986 projects form the evaluation dataset, where EAVA improves 5.3%-35.2% over best-performing baselines across various measures.Baselines include six ML, two DL, and three advanced LLM approaches.
- Evaluation: A user study with security experts confirms that EAVA’s evidence is useful and practical for real-world software-vulnerability assessment.Extensive evaluations also demonstrate significant improvements over baselines.
2 Background and Related Work
The background frames CVSS as the standard for software vulnerability assessment and distinguishes three metric groups. It also positions this work as the first systematic LLM-based study emphasizing evidence for assessment predictions.
- SV Assessment with CVSS: CVSS is the de facto standard for software vulnerability assessment.It provides the assessment framework discussed in the related work.
- SV Assessment with CVSS: CVSS comprises base, temporal, and environmental metric groups.Base metrics capture intrinsic properties, temporal metrics reflect changing characteristics, and environmental metrics address user-specific attributes.
- LLM-Based SV Assessment: This work is the first to systematically investigate LLMs for software vulnerability assessment while emphasizing evidence to justify predictions.Prior research overlooked the importance of evidence supporting assessment results.
3 Motivation and Preliminaries
Existing automated SV assessment methods prioritize classification accuracy but provide insufficient evidence and often lack vulnerable-project context, making results difficult to validate. Preliminary evaluations show that off-the-shelf LLMs underperform learning-based methods because they lack assessment-specific knowledge, although enriching prompts and supplying similar examples can improve performance.
- Assessment challenges: Existing approaches focus on assigning correct CVSS metric values but do not provide supporting evidence, leaving manual validation and curation necessary.EAVA addresses this by extracting evidence and analysis that help analysts understand vulnerabilities and validate automated results.
- Assessment challenges: Specialized LLM agents analyze screenshots and code snippets, replacing them with concise summaries that reduce context length during assessment.The pipeline decomposes complex SVR analysis across stages.
- Assessment challenges: Network becomes the correct Attack Vector after identifying the affected project as nextcloud/desktop, a cloud storage and file-sharing application.The report alone only makes Local, Adjacent, and Network plausible, whereas project context resolves the ambiguity.
- Assessment challenges: Vulnerable-project information is recognized as essential context for accurate SV assessment, motivating enrichment of SVRs with project information.This context is supported by established standards and prior studies.
- Preliminary LLM evaluation: All three general LLMs perform much worse than proEVA across eight CVSS metrics, while stronger models and Chain-of-Thought prompting do not necessarily improve assessment.The evaluated models are Llama-3.3-70b, DeepSeek-v3, and GPT-4.1.
- Preliminary LLM evaluation: 11 to 59: few-shot prompting improves LLM accuracy, but proEVA correctly predicts all 72 SVs and remains superior.The findings indicate that general LLMs need historical vulnerability information and assessment criteria, and prompt-based knowledge injection provides only limited improvement.
4 Approach
EAVA combines enriched SV information, LLM-generated reasoning trajectories, a dedicated assessment model, and historical-vulnerability retrieval to assess software vulnerabilities with evidence. Its assessment model uses supervised instruction tuning followed by reinforcement learning, while specialized agents process code, screenshots, and project information.
- Framework overview: EAVA constructs an annotated dataset from historical vulnerabilities, fine-tunes a dedicated assessment LLM, and builds a database for retrieving similar historical vulnerabilities.The annotated reasoning trajectories connect enriched vulnerability information to determined CVSS metric values.
- Information enrichment: Specialized LLM agents analyze exceptionally lengthy low-information code snippets, screenshots, and project introductions to enrich the vulnerability report.Code analysis uses extracted snippets and two-step reasoning; project summaries emphasize functionalities and applications.
- Assessment model training: The assessment model follows supervised instruction tuning with reinforcement learning to inject assessment-specific knowledge and improve reasoning capability.SFT imitates expert trajectories, while GRPO exploits automatically verifiable predictions without requiring a separate value model.
- Reasoning trajectory annotation: 51,568 metric-level data points cover eight CVSS metrics for each of 6,446 collected vulnerability reports.The annotations were designed to fill the gap in examples showing how assessment evidence supports metric values.
- Historical evidence retrieval: Batch-wise relevance scoring outperforms rating historical vulnerabilities one by one, likely because relative ordering is easier to maintain than consistent absolute scores.The batch design is used when determining the relevance of historical vulnerabilities for supplementary evidence.
5 Experiment Setup
The experiment uses GitHub issue reports linked to CVE items as SVR proxies and constructs a broad dataset spanning projects, programming languages, and CWE categories. EAVA is implemented with specified LLM training settings and evaluated against ML- and DL-based baselines.
- Dataset Construction: The study uses GitHub issue reports referenced by CVE items as proxies for software vulnerability reports, collecting CVE records from NVD and OSV.OSV is identified as an open-source software vulnerability database developed by Google.
- Dataset Statistics: 6,446 SVRs from 1,986 projects form the dataset, covering 54 programming languages and 159 CWE categories.The languages include all of GitHub’s top-10 languages, and the CWE categories encompass the entire MITRE CWE Top-25 list.
- Implementation Details: The reinforcement-learning stage uses learning rate 1e−6 for two epochs, with GRPO sampling five outputs per prompt.These settings follow the supervised fine-tuning stage described in the implementation details.
- Baselines: Experiments compare EAVA with six ML baselines, DeepCVA, and proEVA as DL-based baselines.ML baseline performance is selected using grid search on the validation set, while DeepCVA and proEVA are included as the two DL-based comparators.
6 Experiment Results
EAVA outperforms ML, DL, and LLM baselines for software-vulnerability assessment, while its dedicated-model training and enriched information improve performance. A user study further finds that EAVA’s evidence helps analysts determine assessment results without likely misleading them.
- RQ1: The three LLM baselines perform substantially worse than ML- and DL-based baselines, indicating that domain-specific knowledge is necessary for SV assessment.EAVA also outperforms proEVA on every individual CVSS metric except Attack Complexity.
- RQ1: EAVA outperforms baselines across overall and individual CVSS assessment metrics, with gains of 5.3%–35.2% over proEVA on weighted F1 and MCC.For average measures, gains are 5.3% in weighted F1 and 18.7% in MCC; for severity measures, they are 14.4% and 35.2%, respectively.
- RQ2: Fine-tuning injects assessment-specific knowledge more effectively than RAG, while explicit reasoning and reinforcement learning further improve assessment performance.Applying RL on top of SFT improves the average measure by 3.1% in weighted F1 and 13.5% in MCC over the w/o RL variant.
- RQ3: Incorporating rich text from SVRs and information about vulnerable repositories enhances assessment performance.EAVA achieves the best overall performance compared with variants removing code, screenshots, or repository information.
- RQ4: EAVA’s evidence helps analysts determine assessment results, and participants are unlikely to be misled by incorrect predictions.Among 36 incorrect EAVA predictions, participants still selected the correct metric value in 12 cases; the evidence remained factually strong but had lower relevance and completeness.
7 Discussion
EAVA generalizes across underlying LLMs, offers low overall inference cost with parallelizable latency, and can faithfully analyze rich-text content. Its evaluation is limited by reliance on potentially inaccurate NVD CVSS labels, while historical retrieval adds avoidable cost when reports are already sufficient.
- Generalizability: Comparable performance across all evaluation metrics with Qwen3-8B and Llama-3.1-8B suggests that EAVA’s two-stage SFT-and-RL framework generalizes across foundation models.The framework is not tied to a particular foundation model.
- Efficiency: 0.0327$ is the average monetary cost for processing one SV report across all eight CVSS metrics, and assessment latency falls to 1.52s with batch size eight.The latency can be reduced through parallelism, and the underlying LLMs are open-source.
- Efficiency: 25.88s & 0.0021$ are the latency and cost incurred before Historical SVs Retrieval, whose higher cost comes mainly from reranking 10 retrieved SVs with Llama-3.3-70b for each CVSS metric.This retrieval-stage cost can be avoided when an SV report already provides sufficient information.
- Faithfulness: 48 of 50 sampled screenshots received the highest faithfulness score of 5, while annotators assigned score 5 to all 50 sampled code snippets.Two screenshot cases received scores of 4 and 3 because of redundant information and a mistaken application identification, respectively.
- Internal Validity: NVD-provided CVSS values may be inaccurate because assessment depends on available information, analyst familiarity with affected projects, and expertise and experience.Different vulnerability databases may assign varying CVSS values to the same vulnerability.
8 Conclusion
EAVA is an LLM-based framework for software vulnerability assessment that provides supporting evidence. Its dedicated assessment model uses trajectory annotation and two-stage fine-tuning, while evaluations show off-the-shelf LLMs struggle without domain-specific knowledge.
- Framework: EAVA leverages LLMs to perform software vulnerability assessment while providing supporting evidence.The paper presents EAVA as a novel framework for evidence-supported SV assessment.
- Evaluation: Off-the-shelf LLMs struggle to produce accurate assessments without domain-specific knowledge.This finding comes from extensive evaluations with mainstream LLMs.
- Model Training: EAVA’s dedicated assessment LLM is built through large-scale trajectory annotation and a two-stage fine-tuning paradigm combining SFT and RL.The training paradigm is designed to inject assessment knowledge into the model.