Source-linked AI summary
Large Language Model for Vulnerability Detection: Emerging Results and Future Directions
Xin Zhou, Ting Zhang, David Lo
TL;DR
Vulnerability detection with large language models remains underexplored, particularly compared with established medium-sized pretrained detectors. The paper evaluates GPT-3.5 and GPT-4 using diverse prompts and finds competitive GPT-3.5 performance and a 34.8% GPT-4 Accuracy advantage over CodeBERT, while noting limitations in test-set representativeness and labeling.
Problem
LLM performance on vulnerability detection, especially classification compared with models such as CodeBERT, remains largely unexplored.
Method
The paper evaluates GPT-3.5 and GPT-4 with in-context learning and diverse prompts incorporating task descriptions, external CWE knowledge, and training-set examples.
Results
GPT-3.5 achieved competitive performance with CodeBERT, while GPT-4 with prompt P+A3 outperformed CodeBERT by 34.8% in Accuracy.
Takeaways & Limitations
Prompted LLMs are promising models for vulnerability detection, with prompt enhancements such as external knowledge and selected training samples improving the approach.
Takeaways & Limitations
The balanced test set does not reflect the real-world minority prevalence of vulnerable code and can inflate the reported metrics.
Abstract
from arXiv · showhide
Previous learning-based vulnerability detection methods relied on either medium-sized pre-trained models or smaller neural networks from scratch. Recent advancements in Large Pre-Trained Language Models (LLMs) have showcased remarkable few-shot learning capabilities in various tasks. However, the effectiveness of LLMs in detecting software vulnerabilities is largely unexplored. This paper aims to bridge this gap by exploring how LLMs perform with various prompts, particularly focusing on two state-of-the-art LLMs: GPT-3.5 and GPT-4. Our experimental results showed that GPT-3.5 achieves competitive performance with the prior state-of-the-art vulnerability detection approach and GPT-4 consistently outperformed the state-of-the-art.
1 INTRODUCTION
Software vulnerability detection has used medium-sized pretrained models or smaller neural networks, while LLM effectiveness for this security classification task remains largely unexplored. The paper studies GPT-3.5 and GPT-4 with diverse prompts to address this gap.
- Prior vulnerability detectors primarily used medium-size pretrained models such as CodeBERT or smaller neural networks.
- LLMs have shown strong few-shot learning across tasks, but their performance on vulnerability detection remains largely unexplored.
- The paper investigates whether LLMs can perform vulnerability classification and outperform medium-size pretrained models such as CodeBERT.
- The study explores diverse prompts with GPT-3.5 and GPT-4; GPT-3.5 was competitive with CodeBERT, while GPT-4 outperformed it by 34.8% in Accuracy.
2 PROPOSED APPROACH
The approach applies in-context learning to GPT-3.5 and GPT-4 through prompts that define the task, represent answers with a verbalizer, and add contextual knowledge. Prompt augmentations draw on external CWE examples, training samples, and retrieved similar methods.
- In-Context Learning: In-context learning freezes LLM parameters and uses prompts to provide task-specific knowledge without parameter updates.This reduces the GPU resources required compared with fine-tuning large models.
- Prompt Basics: A prompt contains an input slot for code and an answer slot for the predicted answer, while a verbalizer maps generated answers to target classes.
- Prompt Designs: The prompt designs combine a base vulnerability-identification request with task-role descriptions, project names, and filenames.The role description aims to orient the model toward security-related reasoning, while project information supplies code context.
- Prompt Designs: External source knowledge uses vulnerable code examples from the 25 most dangerous CWE types identified in 2022.These examples are intended to provide vulnerability characteristics beyond the model’s training-data limitations.
- Prompt Designs: Training-set knowledge is added either through randomly selected vulnerable and non-vulnerable samples or through the top K semantically similar methods retrieved with CodeBERT.
3 PRELIMINARY EVALUATION
The evaluation compares prompted GPT-3.5 and limited GPT-4 experiments with fine-tuned CodeBERT on a vulnerability-fixing-commit dataset using multiple classification metrics. Results show that prompt design materially affects GPT-3.5, while GPT-4 with external CWE knowledge exceeds CodeBERT in Accuracy.
- Dataset and Model: The dataset contains 7,683, 853, and 368 methods in the training, validation, and test sets, respectively.Vulnerable functions were identified from lines changed in vulnerability-fixing commits, while other functions in touched files were treated as non-vulnerable.
- Dataset and Model: The study primarily evaluates GPT-3.5, conducts limited GPT-4 experiments, and uses fine-tuned CodeBERT as the baseline.
- Evaluation: The evaluation reports Accuracy, Precision, Recall, F1, and F0.5, with F0.5 weighting precision more heavily than recall.The experiments were repeated twice because in-context learning showed instability.
- GPT-3.5 Results: 18.2% in Accuracy was gained from external CWE examples, while random and similar-code sampling improved Accuracy by up to 22.8% and 19.6% over the base prompt.
- GPT-3.5 Results: The P+A4+A5 combination achieved 62.7% Accuracy and 62.8% F0.5, while GPT-3.5 exceeded CodeBERT by 4.0% in Accuracy, 22.5% in Precision, and 4.5% in F0.5.GPT-3.5 nevertheless underperformed CodeBERT by 44.8% in Recall and 15.3% in F1.
- GPT-4 Results: GPT-4 with prompt P+A3 outperformed fine-tuned CodeBERT by 34.8% in Accuracy on the first half of the test set.The GPT-4 evaluation was limited because of its considerably higher cost.
4 THREATS TO VALIDITY
The evaluation faces potential data leakage, an artificially balanced test distribution, and a labeling heuristic that may overestimate vulnerable functions.
- ChatGPT’s closed-source status prevents validating whether evaluation examples overlapped with its training data.
- The balanced vulnerable/non-vulnerable test set reduces ChatGPT usage costs but does not reflect real-world vulnerability prevalence and can inflate metrics.
- Labeling all functions changed in vulnerability-fixing commits as vulnerable may overestimate vulnerability counts because commit tangling can include unrelated functions.
5 RELATED WORK
Prior vulnerability-detection uses of ChatGPT appear mainly in gray literature, while this study examines prompt improvement for general software vulnerability detection.
- BurpGPT integrates ChatGPT with Burp Suite to detect web-application vulnerabilities, whereas this study is not confined to a specific software domain.
- vuln_GPT is an LLM introduced to discover and address software vulnerabilities, while this study focuses on improving prompts for vulnerability detection.
6 FUTURE WORK
Future work targets deployment constraints, model precision and robustness, long-tailed vulnerability types, and stronger collaboration between AI tools and developers.
- Local and Specialized LLMs-based Vulnerability Detection: Local, specialized LLMs could address organizations’ restrictions on sending proprietary or classified source code to third-party services.
- Local and Specialized LLMs-based Vulnerability Detection: Tuning open-source code LLMs with vulnerability corpora could exploit open-source vulnerability data while addressing data-security and privacy concerns.
- Precision and Robustness Boost in Vulnerability Detection: Ensemble learning is planned to improve precision, while diverse adversarial transformations are planned to strengthen robustness against perturbations and attacks.
- Enhancing Effectiveness in Long-Tailed Distribution: Future studies will test LLM detection of infrequent vulnerability types and consider data augmentation for less common types in long-tailed data.
- Trust and Synergy with Developers: Future work should investigate strategies that foster trust and collaboration between developers and AI-powered vulnerability-detection solutions.
7 CONCLUSION
The study investigates ChatGPT for vulnerability detection through prompt enhancements, including external knowledge and valuable training samples, and identifies directions for further research.
- The study explores ChatGPT’s efficacy and potential for vulnerability detection.
- The proposed prompt enhancements incorporate external knowledge and select valuable samples from the training set.
- The authors identify many promising directions for future study and publicly release a replication package.