Source-linked AI summary
LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks
Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, Gianluca Stringhini
TL;DR
Reliable evidence that LLMs can identify and reason about security bugs is limited, despite proposed uses in automated vulnerability repair. The paper develops SecLLMHolmes, a fully automated framework with 228 scenarios for evaluating eight LLMs across eight dimensions. The evaluation finds non-deterministic responses, incorrect reasoning, poor real-world performance, and substantial non-robustness, concluding that further advances are needed before general-purpose security assistance.
Problem
Benchmarks showing that LLMs can consistently identify and reason about security-related bugs are lacking, despite proposed use in automated vulnerability repair.
Method
SecLLMHolmes evaluates eight capable LLMs across eight investigative dimensions using 228 code scenarios and a fully automated evaluation framework.
Results
The evaluated LLMs produced non-deterministic responses, incorrect reasoning, poor real-world results, and non-robust answers to simple code augmentations.
Takeaways & Limitations
The findings indicate that state-of-the-art LLMs are not yet ready for automated vulnerability detection or general-purpose security assistance.
Takeaways & Limitations
The authors note that the reasoning-score method produced false positives in 7 of 100 manually selected examples when two metrics agreed on an incorrect output.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have been suggested for use in automated vulnerability repair, but benchmarks showing they can consistently identify security-related bugs are lacking. We thus develop SecLLMHolmes, a fully automated evaluation framework that performs the most detailed investigation to date on whether LLMs can reliably identify and reason about security-related bugs. We construct a set of 228 code scenarios and analyze eight of the most capable LLMs across eight different investigative dimensions using our framework. Our evaluation shows LLMs provide non-deterministic responses, incorrect and unfaithful reasoning, and perform poorly in real-world scenarios. Most importantly, our findings reveal significant non-robustness in even the most advanced models like `PaLM2' and `GPT-4': by merely changing function or variable names, or by the addition of library functions in the source code, these models can yield incorrect answers in 26% and 17% of cases, respectively. These findings demonstrate that further LLM advances are needed before LLMs can be used as general purpose security assistants.
1. Introduction
The paper asks whether LLMs can serve as helpful security assistants for vulnerability detection and introduces SecLLMHolmes to evaluate that question systematically. Testing eight capable LLMs reveals unreliable detection, reasoning, and robustness, leading the authors to conclude that current models are not ready for automated vulnerability detection.
- Eight state-of-the-art LLMs were evaluated across 228 code scenarios covering more than eight critical vulnerabilities in C and Python.
- All analyzed models had high false positive rates and sometimes flagged patched code as still vulnerable.
- LLM outputs were non-deterministic, and their vulnerability explanations were often incorrect even when the vulnerability label was correct.
- Simple code changes, including whitespace edits, renamed functions, or related library functions, could confuse LLM chain-of-thought reasoning.
- The study finds that LLMs perform poorly on real-world projects and are not yet ready for automated vulnerability detection.
- SecLLMHolmes is a fully automated framework for evaluating LLM accuracy and reasoning in vulnerability detection.It includes 228 code scenarios and 17 prompting techniques.
2. Background and Related Work
The background describes LLMs as next-token predictors that can be prompted in several chat-based formats, while prior security evaluations often cover limited dimensions. SecLLMHolmes addresses these gaps by evaluating robustness, reasoning, vulnerability coverage, and real-world performance through an automated framework.
- Background: LLMs generate tokens by selecting the highest-probability next token given the preceding sequence.
- Background: Scaling has enabled zero-shot and few-shot learning, allowing prompts to provide instructions or knowledge not present during pre-training.
- Prompting: Chat-based LLMs accept system instructions, few-shot examples, and a task, with zero-shot, few-shot, task-oriented, and role-oriented prompting as distinct strategies.
- Vulnerability Detection: Traditional vulnerability-detection tools use representations such as abstract syntax trees, dependency graphs, or code property graphs to scan for predefined issues.
- Related Work: Supervised machine-learning approaches require substantial dataset curation, while scarce real-world vulnerable examples limit what models can learn.
- Evaluation Gaps: Security evaluation should measure accuracy, false positives, root-cause reasoning, robustness, vulnerability coverage, and real-world generalization.
- Evaluation Gaps: Prior LLM studies often evaluate only binary labels and are limited in model count, vulnerability coverage, prompts, code complexity, robustness, or evaluation automation.
- SecLLMHolmes: The paper identifies complex data-flow understanding, fragile reasoning, naming biases, and divergence from human root-cause analysis as challenges for LLM vulnerability detection.
3. SecLLMHolmes
SecLLMHolmes is a configurable, fully automated framework that evaluates chat-based LLMs for vulnerability detection and reasoning across prompts, datasets, augmentations, and response analysis.
- Framework overview: SecLLMHolmes combines configurable LLM integration, prompt templates, datasets, code augmentations, and automated response evaluation.Each generated prompt is passed to the LLM, and the response is evaluated within the framework.
- Prompt templates: The framework evaluates four prompting techniques spanning zero-shot or few-shot learning and task-oriented or role-oriented instructions.Its prompt categories include standard, step-by-step reasoning-based, and other prompts described in the prompt templates.
- Datasets: The benchmark contains 228 code scenarios, including hand-crafted CWE examples, real-world CVEs, and augmented code scenarios.Hand-crafted scenarios cover vulnerable and patched pairs from eight CWEs in C and Python; real-world scenarios use 2023 CVEs from open-source projects.
- Datasets: Real-world CVE scenarios use vulnerable and patched versions of 15 CVEs from four open-source projects, with files truncated to at most 6,144 tokens.Comments and functions unrelated to the vulnerable or patched function are removed to address model input limits and maintain fairness.
- Datasets: Code augmentations comprise 150 scenarios designed to test robustness while preserving human experts’ ability to identify vulnerabilities.Trivial augmentations measure robustness to random noise using seven transformations across selected CWE-787 and CWE-89 scenarios.
- Evaluator: The automated evaluator extracts an LLM’s yes/no/n/a verdict and textual reasoning, then summarizes the reasoning for consistent comparison.GPT-4 extracts the verdict and reasoning, while reasoning comparisons use Rouge, cosine similarity, and GPT-4-based alignment evaluation.
4. Experimental Investigation
The investigation evaluates LLM vulnerability detection across consistency, parameter settings, prompting, reasoning faithfulness, vulnerability variety, code difficulty, robustness, and real-world cases. Results show substantial instability, reasoning failures, sensitivity to code changes, false positives, and weak performance on real-world scenarios.
- Consistency: All models produced inconsistent responses on at least one test at temperature 0.2, while temperature 0.0 improved consistency but did not eliminate it.At temperature 0.0, codechat-bison@001, codellama34b, and gpt-3.5-turbo-16k were consistent for all tests, whereas chat-bison@001 and gpt-4 remained inconsistent.
- Parameter Settings: Increasing temperature showed no general performance improvement, so the remaining experiments used temperature 0.0 to prioritize consistency.Top p was set to each LLM’s default value.
- Prompt Diversity: GPT-4 achieved maximum accuracy of 89.5%, but no prompting technique consistently outperformed others across all LLMs.Prompt preferences varied by model: reasoning-oriented prompts helped GPT models and codechat-bison, role assignment helped chat-bison, and S1 worked best for codellama34b.
- Faithful Reasoning: Every LLM produced cases where correct reasoning led to a wrong answer or a correct answer rested on incorrect reasoning, indicating unfaithful explanations.PaLM2 models also had lower reasoning rates because they did not explain their decisions in many cases.
- Vulnerability Variety: Most models poorly classified patched code, creating many false alarms; few-shot prompting performed significantly better than zero-shot prompting across almost all models (p-value = 0.003).Role-oriented prompts performed slightly better than task-oriented prompts (p-value = 0.1), although codechat-bison@001 did not answer two CWE categories under few-shot prompting.
- Code Difficulty and Robustness: LLMs struggled with library-function safety and complex multifunction, multivariable data flows, and trivial formatting or naming changes could cause incorrect answers and reasoning.No prompting technique was completely robust; robustness tests produced incorrect responses in 17% of cases for GPT-4.
- Real-World Cases: All studied LLMs gave incorrect answers for several real-world CVE cases and frequently flagged patched examples as vulnerable, while zero-shot role-oriented prompting remained insufficient for deployment.Few-shot prompting did not work for real-world scenarios, whereas R2 performed relatively better for all LLMs.
5. Discussion
SecLLMHolmes is publicly released to support evaluation of chat-based LLMs, but the paper identifies important methodological and scope limitations. These include output-extraction dependence, knowledge-cutoff concerns, imperfect reasoning metrics, and incomplete scenario coverage.
- SecLLMHolmes enables researchers to evaluate any chat-based LLM and compare future model releases, architectures, or parameter counts.The framework and benchmarks are publicly released for community use.
- Limitations: Deterministic inference through a fixed seed does not ensure stable answers because different seeds may produce different responses to the same input.The paper discusses this as an unresolved issue in GPT-4 Turbo preview.
- Limitations: The framework relies on GPT-4 to extract answers and reasoning in a required format before subsequent analysis can proceed.Manual inspection of 100 extracted responses found two that did not follow the required format.
- Limitations: Researchers evaluating newer LLMs may need post-cutoff CVEs, while the framework can incorporate new ground-truth data.This addresses potential knowledge-cutoff bias in evaluations.
- Limitations: The reasoning-score majority metric can still report a false positive when two component metrics agree on an incorrect output.This occurred 7 times in 100 manually selected examples.
- Limitations: The code scenarios omit some difficulty levels, augmentations, languages, and vulnerability types, limiting representativeness.The authors state that additional scenarios can be added in future work.
6. Conclusion
The paper introduces SecLLMHolmes, a scalable automated framework for evaluating chat-based LLMs on vulnerability detection and reasoning. Its evaluation finds current models unreliable and concludes they are not yet ready for vulnerability detection.
- SecLLMHolmes evaluates chat-based LLM efficiency and reasoning across eight dimensions for vulnerability detection.
- The evaluation shows state-of-the-art LLMs currently answer vulnerability-identification questions incorrectly and unreliably.
- The authors conclude that current LLMs are not yet ready for vulnerability detection and call for research addressing the highlighted issues.
- The released framework and benchmarks are intended to help the community evaluate progress across future LLM versions.
Appendix A. Examples of Code Difficulty Levels
The appendix illustrates how code-scenario difficulty increases when apparent sanitization or control-flow context obscures the underlying vulnerability. It contrasts an easy directory-traversal example with harder cases involving misleading library behavior.
- Difficulty Levels: Easy CWE-22 1v directly concatenates user-supplied file paths with an absolute directory path without sanitization, causing directory traversal.
- Difficulty Levels: Medium CWE-22 2v combines four inputs and conditional processing with calls such as realpath, making the vulnerability harder to assess.
- Difficulty Levels: The resolve path call appears to sanitize the user-controlled path but actually fails to do so.
Appendix B. Robustness to Code Augmentations
The robustness appendix studies whether vulnerability-detection results remain stable under code augmentations, varying difficulty, vulnerability coverage, prompt diversity, and inference settings. It includes renamed-variable scenarios, patched code, library additions, and real-world CVE evaluations.
- Code Augmentations: A1 uses 12 CWE-787 and CWE-416 scenarios with variables renamed to buffer while asking models to detect out-of-bound writes or buffer overflows.
- Code Augmentations: A2 selects 12 patched scenarios from CWE-787, CWE-79, CWE-89, and CWE-416 and changes function names.
- Inference Settings: The appendix reports separate evaluations for output consistency at recommended temperature and at temperature 0.0.
- Inference Settings: It also evaluates performance across temperature ranges for CWE-787 and CWE-89.
- Evaluation Dimensions: Additional analyses examine prompt diversity, reasoning faithfulness, vulnerability range, code difficulty, code-level augmentations, and real-world CVEs.
- Scenario Examples: The appendix includes a hard-difficulty CWE-22 3v example, alongside figure-based examples of difficulty levels and NT6 code augmentations.
- Scenario Examples: A CWE-77 command-injection scenario is included among the evaluated code examples.
C.1. Summary
The paper proposes an automated framework for evaluating LLMs’ ability to identify and reason about security vulnerabilities.
- The framework evaluates LLM capabilities in vulnerability identification and reasoning.
C.2. Scientific Contributions
The paper contributes an established-field advance, a tool for future research, and a publicly usable dataset.
- The paper provides a valuable step forward in an established field.
- It creates a new tool to enable future science.
- It provides a new dataset for public use.
C.3. Reasons for Acceptance
The paper is accepted for its automated evaluation framework, broader study of underexplored LLM vulnerability-detection capabilities, and 228-scenario benchmark.
- The paper suggests a fully automated framework for evaluating LLM vulnerability-identification capabilities.
- It examines code complexity, vulnerability reasoning, and chat-based LLMs in vulnerability detection.
- 228 code scenarios form the paper’s new benchmark.The benchmark includes 48 hand-crafted examples.
C.4. Noteworthy Concerns
The reviewers raised concerns about limited and potentially biased evaluation data, while the authors argued that the dataset is comparable to prior work and constrained by LLM knowledge-cutoff dates.
- The dataset contains only 15 real-world CVEs, limiting coverage of real-world vulnerabilities.
- 150 of 228 scenarios use augmented code, which reviewers warn may bias measurement results.
- The authors state that their dataset size is comparable to prior work using 12 CVEs.
- The number of suitable CVEs was constrained by LLM knowledge-cutoff dates to preserve temporal consistency.