Source-linked AI summary
On Predicting Vulnerability Severity Using In-Context Learning: An Industrial Case Study
Daniel Rodriguez-Cardenas, David Nader Palacio, Anna Schmedding, Yiyang Lu, Aadil Mallick, Bill Hudson, Chris Gourley, Michael Roytman, Chris Shenefiel, Evgenia Smirni, Denys Poshyvanyk
TL;DR
The paper addresses the need for earlier, scalable CVSS severity assessment without exposing proprietary code and incident data to cloud providers. It compares Big-Vul with proprietary data, evaluates in-context prompting across local and cloud LLMs, and reports CodeLlama2-7B as a practical local option with performance approximating cloud models. The findings support lightweight prompting and locally deployable models for privacy-preserving severity triage within the studied setting.
Problem
CVSS severity assessment is difficult to scale and cloud-based analysis is constrained by confidentiality, compliance, and data-residency requirements.
Method
The study compares proprietary and Big-Vul data, then evaluates in-context CVSS v3.1 regression from vulnerable C/C++ snippets across local open-source and cloud LLMs.
Results
CodeLlama2-7B is the most reliable open-source option, achieving MSE around 7 with three-shot prompts and virtualized output constraints.
Takeaways & Limitations
Locally deployable open-source code models, particularly CodeLlama2-7B, offer a practical privacy-preserving building block for vulnerability-severity triage.
Takeaways & Limitations
The study is limited to Big-Vul C/C++ data and open-source models in the 7B–13B range, and benchmark contamination may inflate performance estimates.
Abstract
from arXiv · showhide
Modern software systems require earlier and more scalable vulnerability severity assessment to reduce exposure to high-impact security flaws. Security analysts typically assign CVSS scores, but this manual triage does not scale with the growth of disclosed vulnerabilities and often depends on cloud LLM services that raise confidentiality concerns. This paper presents an industrial case study on predicting CVSS v3.1 scores directly from vulnerable C/C++ snippets using in-context learning with locally deployable, open-source LLMs. We compare proprietary data with the Big-Vul dataset, showing sufficiently aligned CVSS distributions to justify Big-Vul as a proxy for industrial data when constructing prompt-based testbeds. We then vary in-context configurations and model parameters, evaluating CodeLlama2-7B, CodeLlama2-13B, Mistral-7B, gpt-oss, and GPT4o-mini using mean squared error (MSE) and feasibility metrics. Our results show that medium-sized open-source code models, particularly CodeLlama2-7B, can approximate the best cloud performance for CVSS regression when guided by lightweight, output-constraining prompts, offering a practical, privacy-preserving building block for severity triage in industrial settings.
I. INTRODUCTION
The paper targets earlier, scalable CVSS severity prediction while addressing confidentiality constraints on cloud-based analysis. It compares industrial and Big-Vul data, evaluates lightweight in-context configurations and models, and finds CodeLlama2-7B a practical local alternative.
- Accurate CVSS prediction is difficult because many exploits occur before scores are published, leaving organizations with incomplete severity information.
- Proprietary vulnerability data and Big-Vul exhibit overlapping CVSS distributions and aligned correlations with Kenna-adjusted scores.
- The study evaluates C1–C3 in-context configurations while varying sequence length, temperature, and shot count for CVSS regression.
- Restricting output format and adding a small number of examples reduce MSE, non-feasible responses, and out-of-range scores.
- CodeLlama2-7B approximates cloud-model performance while supporting local deployment and keeping sensitive artifacts on-premise.
II. BACKGROUND
The background establishes CVSS as the industry’s standard vulnerability-severity score and situates the work within vulnerability life-cycle management and in-context learning for LLM-based prediction.
- CVSS standardizes vulnerability severity using exploitability, impact, and scope metric groups.
- The National Vulnerability Database is a U.S. government repository and a major publisher of CVSS scores for known vulnerabilities.
- EPSS complements CVSS by estimating the probability that a vulnerability will be exploited in the wild within the next 30 days.
- Vulnerability management spans detection, triage, enumeration of mitigation actions, fixing, and possible exploitation.
- In-context learning guides LLMs through designed prompt interactions without requiring task-specific pre-training or fine-tuning.
III. APPROACH
The approach frames severity estimation as numeric CVSS regression and builds a prompt-based testbed from vulnerable code snippets, descriptions, and scores for locally deployable models.
- The approach seeks to help analysts assess vulnerability severity earlier in the software life cycle as features and vulnerabilities increase.
- Open-source models can run locally, helping companies retain environmental control and avoid sending sensitive code to third-party providers.
- CVSS severity is predicted as a numeric regression value rather than a categorical range, providing more precise severity and calibration information.
- The testbed collects vulnerable snippets with CVSS, EPSS, or Kenna Risk scores and descriptions from proprietary and open-source repositories.
- Samples are filtered by code length and retained only when descriptions and CVSS scores are valid.
- Prediction prompts combine few-shot examples with vulnerable code, associated scores, and optionally vulnerability descriptions before capturing the model’s score output.
IV. METHODOLOGY
The methodology examines dataset comparability, prompt configuration, and the accuracy of locally deployed LLMs for vulnerability-severity regression.
- The study evaluates industry and open datasets, in-context configurations, and open-source LLMs.
- RQ1: RQ1 examines correlations and distributions to assess whether open datasets can proxy industry data for evaluation.
- RQ2: RQ2 compares in-context configurations, including prompts with vulnerable code, scores, model roles, CVSS ranges, and vulnerability descriptions.
- RQ3: RQ3 evaluates locally deployed open-source LLMs as a regression task because numeric scores provide more precise information than severity categories.
VI. CASE STUDY DESIGN
The case study builds prompt-based testbeds from curated Big-Vul vulnerability triples, comparing stratified and randomized settings across sequence lengths, severity classes, and open-source LLMs.
- Testbeds: The experimental pipeline evaluates open-source LLMs for vulnerability severity prediction using vulnerable code, descriptions, and scores from Big-Vul.Big-Vul provides approximately 150,000 data points and is curated for prompt construction.
- Testbeds: Big-Vul is split into approximately 53K example-generation triplets and 5K question-generation triplets before filtering sequences to 50–300 characters.The lower bound preserves function definitions and the upper bound limits inference cost.
- Prompt construction: Prompts randomly combine vulnerable-code examples with question snippets, using up to 3 shots because additional examples increase testing time and context size.For 3-shot experiments, the study selects m = 900 examples and n = 300 questions.
- Experimental settings: The two testbeds use stratified CVSS severity classes or randomized samples, with sequences restricted to 50–300 characters.Severity classes are none, low, medium, high, and critical.
- Machine configuration: The experiments run on Ubuntu with an AMD EPYC 7532 CPU, an A100 GPU with 40GB VRAM, and 1TB RAM using HuggingFace and PyTorch.All models were loaded onto the GPU for inference.
B. Statistical Evaluation
The statistical evaluation compares stratified and random severity-prediction experiments while varying context, sequence length, model, temperature, and repeated sampling.
- Experimental sampling: The study uses 600 three-shot context samples for stratified evaluation and 300 context samples for randomized evaluation.The randomized experiment evaluates prediction from spontaneous code snippets.
- Repeated evaluation: For CodeLlama and Mistral at temperatures 0.3 or 0.9, each experiment runs 30 times to assess response diversity and validate default parameters.The repeated runs characterize model-response ranges under non-deterministic settings.
- Research-question design: RQ2 varies context configurations, whereas RQ3 focuses on sequence length and model variation, with sequence length measured by input-prompt tokens.The metrics are used across both research questions.
VII. RQ1 EXPLORATORY DATA ANALYSIS
The exploratory analysis compares confidential industry datasets with Big-Vul through dataset-shape, distribution, and correlation analyses to assess whether Big-Vul can support the testbed.
- Analysis objectives: The analysis examines distributions and correlations to determine whether Big-Vul contains information comparable to the proprietary data for prompt-testbed construction.The stated objective is to assess whether the open dataset can be used interchangeably when needed.
- Dataset comparison: The proprietary data include PSIRT CVE, Kenna Risk, and EPSS datasets containing severity, risk, exploitability, and vulnerability metadata.Their sizes are 5,520, 5,372, and 221,346 data points, respectively.
- Dataset comparison: The proprietary datasets span 2010–2023 and are compared with Big-Vul to provide additional context and validation.The comparison addresses trends in vulnerability and risk across the study period.
- Distribution comparison: The majority of CVSS scores in both datasets fall between 4 and 8, and their distributions overlap despite Big-Vul having fewer high scores and more scores of 7.Big-Vul scores were grouped by integer values because the proprietary data contain decimals.
B. Distribution Correlation
The study compares vulnerability-severity distributions and correlations across datasets, then evaluates how prompt context, temperature, sequence length, and shot count affect CVSS regression and feasibility.
- Distribution Correlation: CVSS and KCVSS show strong alignment, while CVSS and Kenna Risk exhibit a weaker correlation in the proprietary data.
- Model and Temperature Effects: CodeLlama2-7B achieves the lowest MSE of 6.69 with a 0.0% non-feasible rate among the models summarized in Table III.
- Model and Temperature Effects: At T = 0.3, CodeLlama2-7B provides better predictive performance while maintaining a low non-feasible rate, motivating its selection for further evaluation.
- In-Context Configuration: Increasing sequence size and iteration count generally reduces MSE and stabilizes performance, while C3 is reported as the best prompt configuration for CodeLlama2 and GPT4o-mini.
- In-Context Configuration: With zero-shot C1, CodeLlama2-7B and Mistral-7B have non-feasible rates of 16% and 80.33%, respectively, versus 0% for CodeLlama2-7B with C2.
- In-Context Configuration: Mistral-7B reaches an MSE of 21.72 with C2, but its non-feasible proportion remains 47.80%.
IX. RQ3 OPEN-SOURCE LLMS ACCURACY
The accuracy evaluation compares CodeLlama2 and Mistral-7B against GPT4o-mini using regression, feasibility, and severity-classification behavior. CodeLlama2 models generally outperform Mistral-7B, while GPT4o-mini remains strongest overall in the reported setting.
- Regression Accuracy: CodeLlama2 models outperform Mistral-7B across sequence sizes and have lower non-feasible rates.
- Regression Accuracy: CodeLlama2-7B achieves an average MSE of 7.65 and 0.10% out-of-range values with one shot and a 300-character sequence.
- Feasibility: At temperature 0.9, Mistral-7B reaches 80.33% non-feasible responses in the zero-shot C1 setting.
- Severity Classification: CodeLlama2-7B concentrates predictions in the Medium class, improving Medium-case correctness but reducing discrimination among lower-severity classes.
- Severity Classification: GPT4o-mini shows comparatively stronger alignment for the None and Medium classes but almost never predicts Critical labels.
X. RELATED WORK
Prior vulnerability-analysis research has emphasized detection and severity estimation with traditional or fine-tuned models, often treating detection as classification. This work instead studies severity regression from vulnerable code using locally deployable open-source LLMs and in-context learning.
- Earlier studies used linear models, random forests, and neural architectures such as CNNs, RNNs, and Transformers for vulnerability detection or CVSS prediction.
- Much prior LLM research focuses on vulnerability detection as classification and commonly uses cloud-hosted or fine-tuned models.
- This work complements prior research by targeting severity regression from vulnerable code with locally deployable, open-source code LLMs.
- The study quantifies model calibration and robustness under different prompt configurations.
XI. THREATS TO VALIDITY
The study’s validity is constrained by its reliance on isolated Big-Vul snippets, potentially noisy labels, aggregate CVSS targets, and benchmark contamination risks. Results also depend on prompt and decoding choices and may not generalize broadly.
- Function-level Big-Vul snippets omit surrounding project and operational context, so results represent isolated-snippet performance rather than fully context-rich cases.
- Noisy method-level vulnerability labels may encourage superficial correlations, and filtering longer snippets with descriptions only partially mitigates this concern.
- Predicting only aggregate CVSS scores limits interpretability and actionability because individual submetrics are excluded.
- The reported results depend on prompt format, shot count, sequence size, and decoding temperature, while better configurations may exist.
- Possible pretraining-data contamination means the reported numbers should be treated as an upper bound rather than a clean measure of out-of-distribution generalization.
XII. CONCLUSIONS AND FUTURE WORK
The paper reports an industrial experience using locally deployed open-source models and in-context learning for privacy-preserving vulnerability severity prediction. It concludes that small models can achieve acceptable performance, while noting scope limits and proposing richer risk modeling and more interpretable CVSS predictions.
- The study applies open-source, small, locally deployed models in an industrial environment to keep proprietary data and software code private.
- In-context learning lets local models adapt to company-specific security policies and requirements while using small models such as CodeLlama2-7B.
- The experiments predict aggregate CVSS v3.1 base scores from C/C++ snippets, excluding individual submetrics and richer contextual factors.
- Beyond two shots, additional context does not meaningfully improve accuracy, while explicitly specifying the desired output format improves performance with C3.
- Future work includes jointly predicting CVSS with Kenna Risk and EPSS and decomposing CVSS into submetrics for more interpretable analyst guidance.