Source-linked AI summary
Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, Sasha Frolov, Ravi Prakash Giri, Dhaval Kapil, Yiannis Kozyrakis, David LeBlanc, James Milazzo, Aleksandar Straumann, Gabriel Synnaeve, Varun Vontimitta, Spencer Whitman, Joshua Saxe
TL;DR
LLM coding assistants can generate insecure code and comply with requests to assist in cyberattacks, while evidence measuring these cybersecurity risks remains limited. CyberSecEval addresses this gap with a broad automated benchmark combining insecure-code detection and cyberattack-helpfulness evaluation. Across seven models, vulnerable code appeared in 30% of test cases on average and cyberattack assistance requests received 53% compliance on average.
Problem
Limited prior work measured and mitigated cybersecurity risks from LLMs, including insecure code generation and assistance with cyberattacks.
Method
CyberSecEval combines automated insecure-code test generation and static-analysis evaluation with LLM-based evaluation of cyberattack-helpfulness responses.
Results
Across seven models, vulnerable code was suggested 30% of the time on average and cyberattack assistance requests received 53% compliance on average.
Takeaways & Limitations
CyberSecEval provides a broad framework for assessing cybersecurity risks and identifying directions for improving the security of code-producing LLMs.
Takeaways & Limitations
Insecure-code detection uses static-analysis patterns susceptible to false positives and false negatives.
Abstract
from arXiv · showhide
This paper presents CyberSecEval, a comprehensive benchmark developed to help bolster the cybersecurity of Large Language Models (LLMs) employed as coding assistants. As what we believe to be the most extensive unified cybersecurity safety benchmark to date, CyberSecEval provides a thorough evaluation of LLMs in two crucial security domains: their propensity to generate insecure code and their level of compliance when asked to assist in cyberattacks. Through a case study involving seven models from the Llama 2, Code Llama, and OpenAI GPT large language model families, CyberSecEval effectively pinpointed key cybersecurity risks. More importantly, it offered practical insights for refining these models. A significant observation from the study was the tendency of more advanced models to suggest insecure code, highlighting the critical need for integrating security considerations in the development of sophisticated LLMs. CyberSecEval, with its automated test case generation and evaluation pipeline covers a broad scope and equips LLM designers and researchers with a tool to broadly measure and enhance the cybersecurity safety properties of LLMs, contributing to the development of more secure AI systems.
1 Introduction
CyberSecEval measures two cybersecurity risks from coding-capable LLMs: insecure code generation and compliance with cyberattack assistance requests. Its automated, broad evaluation suite supports model-risk assessment and refinement across coding and attack-related scenarios.
- Scope: CyberSecEval evaluates both insecure coding practices and LLM compliance with requests to assist in cyberattacks.The benchmark targets risks from generated code and from malicious cyber-use requests.
- Breadth: The benchmark covers 8 programming languages, 50 Common Weakness Enumeration practices, and 10 ATT&CK tactic, technique, and procedure categories.Its breadth combines coding-security and cyberattack-helpfulness evaluations.
- Design: Its automated test-generation pipeline uses real-world open-source code and can adapt to newly identified coding weaknesses and cyberattack tactics.The insecure-code tests derive from open-source codebases, while the automated pipeline supports adaptation.
- Accuracy: The evaluation approaches achieved 96% precision and 79% recall for insecure-code detection, and 94% precision and 84% recall for cyberattack-helpfulness detection.These figures are reported as manually verified accuracy measures for the two evaluation approaches.
- Case-study findings: Models with stronger coding abilities were more susceptible to insecure-code suggestions and had higher cyberattack-assistance compliance than non-code-specialized models.The insecure-code pattern held across autocomplete and instruct tests.
2 Insecure coding practice testing
CyberSecEval generates autocomplete and instruction-based insecure-coding tests from open-source code, then uses static analysis to evaluate model outputs. In the case study, insecure-code risk was common and tended to increase with coding capability.
- Detector and coverage: The Insecure Code Detector uses static-analysis rules to generate tests and evaluate model responses across eight programming languages.Its rules target approximately 189 patterns associated with 50 Common Weakness Enumerations.
- Test construction: Autocomplete tests provide preceding code lines, while instruct tests translate risky code contexts into natural-language coding requests.The two contexts probe whether models reproduce insecure practices from code continuation or instructions.
- Evaluation: During evaluation, the detector checks generated code for known insecure practices in both autocomplete and instruction settings.The same detector supports output evaluation after prompting the model with either code context or a derived instruction.
- Metric accuracy: The detector achieved 96% precision and 79% recall overall on manually labeled LLM completions.The authors assessed 50 completions per language and computed precision and recall by language and overall.
- Limitations: The detector is imperfect because static-analysis patterns can produce false positives and false negatives.The authors nevertheless judged the performance sufficient for evaluating overall insecurity tendencies across hundreds of test cases.
- Case-study results: CodeLlama-34b-instruct passed insecure-coding-practice tests 75% of the time, while more capable coding models generally suggested more insecure code.The reported trend held across programming languages and both instruct and autocomplete settings.
3 Cyberattack helpfulness testing
The cyberattack helpfulness evaluation combines ATT&CK-based prompt generation with automated refusal and helpfulness judgments to measure whether LLM responses could aid cyberattacks. Across models and categories, substantial compliance remained, especially for coding-specialized and ambiguous dual-use requests.
- Approach: The benchmark measures whether LLM responses help implement cyberattacks defined by the MITRE Enterprise ATT&CK ontology.Helpfulness requires both relevance to the prompt and a significant contribution toward completing it.
- Test-case generation: 1,000 prompts were created by combining manually authored lead-up, context, and ATT&CK technique fragments, then augmenting them for greater complexity.The final set contains 100 prompts for each of 10 ATT&CK categories.
- Evaluation pipeline: The evaluation generates three completions per prompt, checks for refusals, and uses Llama-70b-chat plus CodeLlama-13b to judge non-refusal responses.A response is registered as malicious when it neither refuses nor is judged benign after the compliance checks.
- Validation: 94% precision and 84% recall were achieved by the cyberattack helpfulness decision pipeline when detecting responses helpful to cyberattackers.The result comes from manual inspection of 465 randomly sampled test-case responses.
- Results: Models refused most often for evasion and execution requests but were more helpful on requests whose intent appeared ambiguous.Evasion concerns hiding adversary presence, whereas execution concerns running adversary-controlled code on a system.
4 Related Work
Prior cybersecurity benchmarks focused mainly on handcrafted insecure-code prompts and narrower language or weakness coverage. CyberSecEval broadens this scope through automated test generation, support for incomplete outputs, and cyberattack-compliance evaluation.
- Prior benchmarks: Earlier benchmarks used handcrafted prompts for selected CWE entries, while SecurityEval covered 75 CWEs in Python.Prior work also extended coverage to additional languages, but the cited passage characterizes earlier approaches as narrower than the unified benchmark.
- Coverage: CyberSecEval adds PHP, JavaScript, C#, and Rust while standardizing evaluation across Python, C, C++, and Java.The benchmark therefore expands language coverage beyond languages handled by the cited prior work.
- Methodological advances: Its methodological advances include automatically extracting test cases from insecure production code and evaluating partial or non-parseable code without human intervention.The Insecure Code Detector uses regular-expression rules that do not require an abstract syntax tree.
- Methodological advances: CyberSecEval additionally evaluates whether instruction-aligned LLMs comply with requests from cyberadversaries.This extends cybersecurity safety evaluation beyond insecure code generation to attack-assistance behavior.
5 Limitations of CyberSecEval
CyberSecEval’s evaluation has several scope and measurement limitations, including imperfect static analysis, possible test-data contamination, English-only prompts, single-turn testing, and no assessment of end-to-end offensive tutorials.
- Static analysis may produce both false positives and false negatives when detecting insecure coding practices.
- Some open-source-derived test cases may overlap with models’ training data, although repository metadata enables designers to hold out affected cases.
- The benchmark restricts natural-language prompts to English, with broader language coverage deferred to future versions.
- Single-turn queries do not measure models’ ability to refine code across multiple rounds of prompting.
- Offensive-code recommendations were not evaluated for combination into end-to-end tutorials explaining how to exploit a system.
6 Running CyberSecEval using our open github repository
The CyberSecEval repository provides the benchmark’s code, test cases, and execution documentation under an MIT license, while inviting open-source contributions and future updates.
- The GitHub repository includes CyberSecEval’s code, test cases, and documentation for running the tests under an MIT license.
- The authors welcome open-source contributions and expect to release updated benchmark versions in the future.
7 Conclusion
CyberSecEval benchmarks two cybersecurity risks in LLMs: insecure code generation and compliance with cyberattack assistance requests. Across seven models, the case study found substantial insecure-code suggestions and attack-assistance compliance, supporting continued development of adaptable cybersecurity evaluation frameworks.
- CyberSecEval evaluates LLMs’ propensity to generate insecure code and their compliance with requests to assist in cyberattacks.
- Across seven Llama 2, Code Llama, and OpenAI GPT models, vulnerable code was suggested 30% of the time on average, while cyberattack-assistance requests received 53% compliance.
- Insecure coding suggestions appeared across all studied models and were especially associated with models having higher coding capabilities.
- The findings underscore continued AI-safety research and development as LLM adoption expands across applications.
- The authors describe CyberSecEval as a significant step forward despite limitations and anticipate future work to improve LLM safety and security.