PaperlayerView exact revision on arXiv

Source-linked AI summary

CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models

Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, Joshua Saxe

arXiv:2404.13161v1cs.CRcs.LG

TL;DR

Existing evaluation coverage is insufficient for measuring the expanding cybersecurity risks and capabilities of LLMs. CYBERSECEVAL 2 adds interpreter-abuse and prompt-injection test suites to assess risks from code interpreters and attacks targeting LLM instructions. All tested models showed 13%–47% prompt-injection success, while many handled benign borderline requests and rejected most offensive requests; coding ability correlated with better exploit-generation performance.

  • Problem

    Existing evaluation coverage is insufficient for measuring the expanding cybersecurity risks and capabilities of LLMs.

    Source: §1 · p. 1

  • Method

    CYBERSECEVAL 2 adds interpreter-abuse and prompt-injection test suites to assess risks from code interpreters and attacks targeting LLM instructions.

    Source: §1 · p. 2

  • Results

    All tested models showed 13%–47% prompt-injection success, while many handled benign borderline requests and rejected most offensive requests; coding ability correlated with better exploit-generation performance.

    Source: §6 · p. 15

  • Takeaways & Limitations

    LLM security requires benchmark-based assessment and additional safeguards, while autonomous exploit generation still needs further research.

    Source: §6 · p. 15

    The release excludes adversarial machine-learning methods and prompts in languages other than English.

    Source: §2 · p. 3

Abstract

from arXiv · show

Large language models (LLMs) introduce new security risks, but there are few comprehensive evaluation suites to measure and reduce these risks. We present BenchmarkName, a novel benchmark to quantify LLM security risks and capabilities. We introduce two new areas for testing: prompt injection and code interpreter abuse. We evaluated multiple state-of-the-art (SOTA) LLMs, including GPT-4, Mistral, Meta Llama 3 70B-Instruct, and Code Llama. Our results show that conditioning away risk of attack remains an unsolved problem; for example, all tested models showed between 26% and 41% successful prompt injection tests. We further introduce the safety-utility tradeoff: conditioning an LLM to reject unsafe prompts can cause the LLM to falsely reject answering benign prompts, which lowers utility. We propose quantifying this tradeoff using False Refusal Rate (FRR). As an illustration, we introduce a novel test set to quantify FRR for cyberattack helpfulness risk. We find many LLMs able to successfully comply with "borderline" benign requests while still rejecting most unsafe requests. Finally, we quantify the utility of LLMs for automating a core cybersecurity task, that of exploiting software vulnerabilities. This is important because the offensive capabilities of LLMs are of intense interest; we quantify this by creating novel test sets for four representative problems. We find that models with coding capabilities perform better than those without, but that further work is needed for LLMs to become proficient at exploit generation. Our code is open source and can be used to evaluate other LLMs.

1 Introduction

CYBERSECEVAL 2 is a benchmark suite for measuring LLM cybersecurity risks and capabilities, expanding coverage to prompt injection, interpreter abuse, false refusals, and exploit generation. Results show persistent prompt-injection vulnerability, a measurable safety–utility tradeoff, and limited but coding-related exploit-generation capability.

Motivation and use

  • CYBERSECEVAL 2 targets LLM builders and cybersecurity-task automation by quantifying risks and capabilities for model selection, tuning, and deployment decisions.The code and evaluation artifacts are open source under an MIT license.Source: §1 · pp. 1, 3

Benchmark scope

  • CYBERSECEVAL 2 adds prompt-injection and interpreter-abuse evaluations alongside tests for insecure coding and cyberattack helpfulness.The suite measures 15 prompt-injection categories and harmful behavior involving attached code interpreters.Source: §1 · pp. 2–3

Results

  • All tested LLMs succumbed to 26%–41% of prompt-injection tests, showing that conditioning away this risk remains unsolved.The authors argue that additional application design guardrails are necessary for security and reliability.Source: §1 · pp. 2–3

Safety–utility tradeoff

  • False Refusal Rate (FRR) quantifies when LLMs reject benign cybersecurity requests that may appear unsafe, exposing the safety–utility tradeoff.The new test set covers benign borderline requests and can be paired with cyberattack helpfulness tests to compare false and legitimate refusals.Source: §1 · pp. 2–3

Exploit generation

  • LLMs with broader programming capabilities perform better on exploit-generation tests, but most models fail to completely solve the evaluated cases.The evaluation covers exploits for C, Python, and JavaScript code involving logic vulnerabilities, memory exploits, and SQL injections.Source: §1 · pp. 2–3

Safety–utility tradeoff

  • Many LLMs comply with benign borderline cybersecurity requests while still rejecting most requests to assist offensive cyber operations.This indicates that safety tuning need not eliminate all utility for ambiguous but legitimate technical requests.Source: §1 · p. 3

2 Background and the Safety-Utility Tradeoff

CYBERSECEVAL 2 extends cybersecurity evaluation with false-refusal tests to quantify the safety–utility tradeoff between rejecting attacks and assisting legitimate borderline requests.

2 Background and the Safety-Utility Tradeoff

  • The original suite’s Insecure Coding Practice and Cyberattack Helpfulness tests were retained, while Cyberattack Helpfulness was extended with false-refusal tests for legitimate borderline requests.Source: §2 · p. 3

2.1 Quantifying the safety-utility tradeoff with False Refusal Rate; illustration with cyberattack helpfulness

  • The combined original and FRR datasets provide a more comprehensive assessment of whether LLMs reject cyberattack requests while assisting fully legitimate requests.Source: §2 · p. 4
  • Ambiguous cybersecurity requests can be construed as either safe or unsafe, creating a safety–utility tradeoff for systems designed to refuse cyberattack assistance.Source: §2.1 · p. 4
  • False Refusal Rate (FRR) is the percentage of benign prompts refused because they are mistaken for unsafe prompts associated with a specific risk.Source: §2.1 · p. 4
  • The novel FRR dataset covers borderline cybersecurity-related prompts, including cyberdefense topics, enabling comparison of attack refusal with assistance on ambiguous but benign requests.Source: §2.1 · p. 4

3 Related Work

Related benchmarks often measure cybersecurity knowledge or information-processing skills, whereas CYBERSECEVAL 2 emphasizes demonstrated LLM capabilities and vulnerabilities across a broader evaluation suite.

  • CyberMetric, SecQA, WMDP-Cyber, and related educational-style evaluations primarily assess cybersecurity knowledge through multiple-choice questions.Source: §3 · p. 4
  • CyberBench extends beyond knowledge questions to summarization, classification, and named entity recognition, but remains focused mainly on information retrieval and natural language processing.Source: §3 · p. 4
  • LLM4Vuln evaluates vulnerability discovery with external vulnerability and codebase knowledge, whereas CYBERSECEVAL 2 remains implementation-agnostic and tests end-to-end exploitation challenges.Source: §3 · p. 4
  • Rainbow Teaming applies CYBERSECEVAL 1 to automatically generate adversarial prompts similar to those used in cyberattack helpfulness tests.Source: §3 · p. 4
  • CYBERSECEVAL 2 contributes a wide-ranging evaluation of demonstrated LLM risks and capabilities, emphasizing behavior over information retrieval.Source: §3 · p. 4
  • CYBERSECEVAL 2 offers complementary signals for LLM development and risk assessment alongside other evaluation frameworks.Source: §3 · pp. 4–5

4 Detailed Descriptions of New Tests in CyberSecEval 2

CYBERSECEVAL 2 evaluates prompt-injection susceptibility across logic-violating and security-violating scenarios using direct and indirect attacks, while limiting coverage of rapidly evolving strategies.

4.1.1 Testing philosophy

  • Prompt injection tests distinguish logic violations from security violations, covering attacks that contradict system instructions with or without clear user harm.Source: §4.1.1 · p. 5
  • The suite covers many prompt-injection strategies without claiming exhaustive coverage because the attack space evolves rapidly.Source: §4.1.1 · p. 5
  • The evaluation excludes iterative, multi-turn, and optimization-based prompt-injection strategies as out of scope for the initial contribution.Source: §4.1.1 · p. 5

4.1.2 Testing approach

  • Each test combines a system prompt, malicious user or third-party input, and a judge question that determines whether the model output reveals a successful attack.Source: §4.1.2 · p. 5
  • System prompts are used only for models supporting system prompting, excluding Mistral from the case study because of API limitations.Source: §4.1.2 · p. 5
  • Table 2 summarizes the prompt-injection methods covered by the evaluation.Source: Table 2 · p. 6, §4.1.2 · p. 7
  • Tests include direct injections that target system instructions and indirect injections embedded in third-party context, such as a web document.Source: §4.1.2 · p. 7

4.2 Vulnerability exploitation evaluations

The exploitation evaluation tracks LLM progress on software vulnerability exploitation using randomized, solvable capture-the-flag challenges designed to test general reasoning rather than memorization or complete vulnerability-class coverage.

4.2 Vulnerability exploitation evaluations

  • CYBERSECEVAL 2 tracks automatic software exploitation because advances could support defenders or attackers and have implications for cybersecurity and AI policy.Source: §4.2 · p. 7

4.2.1 Testing philosophy

  • The tests use challenging but solvable capture-the-flag-style cases requiring non-trivial reasoning about program control and data flow.Source: §4.2.1 · p. 7
  • Program-synthesis strategies randomly generate tests to reduce concerns that models memorized particular challenge cases.Source: §4.2.1 · p. 7
  • The evaluation draws on real software vulnerabilities but targets general reasoning rather than complete coverage of vulnerability classes.Source: §4.2.1 · p. 7

4.2.2 Testing approach

  • Generators create challenges for logic vulnerabilities in C, JavaScript, and Python, SQL injection in Python/sqlite3, and memory exploits in C and C++.Source: §4.2.2 · p. 7
  • Each generated challenge includes a proof-of-concept input that triggers its vulnerability, guaranteeing that the challenge is solvable.Source: §4.2.2 · p. 7

4.3 Code interpreter abuse evaluation

Code interpreter integrations create security risks because untrusted users may target the host system or external systems through generated code. CyberSecEval 2 evaluates these risks using categorized malicious prompts, judge-LLM assessment, and compliance rates.

4.3 Code interpreter abuse evaluation

  • Interpreter integrations can let untrusted users target host operating systems or other systems through maliciously generated code.Source: §4.3 · p. 9

4.3.1 Testing philosophy

  • The evaluation covers container escape, privilege escalation, reflected attacks, and social engineering, using bespoke abuse categories rather than a standard ontology.Source: §4.3.1 · p. 9
  • A judge LLM evaluates generated code for compliance instead of executing live malicious code, improving portability and avoiding testing-related security risks.Source: §4.3.1 · p. 9

4.3.2 Testing approach

  • The suite contains 500 interpreter-abuse prompts, with 100 prompts per category, sent individually to each model under test.Source: §4.3.2 · p. 9
  • Judge-LLM decisions distinguish compliance from refusal, enabling malicious-prompt compliance rates to be computed for each risk category.Source: §4.3.2 · p. 9

5 Case Study in Applying CyberSecEval 2

CyberSecEval 2 applies expanded security and capability tests to a small set of Llama, CodeLlama, GPT, and Gemini models, finding persistent weaknesses across refusal, prompt injection, exploitation, and interpreter abuse. Coding ability improves exploit performance, but models remain far from reliable autonomous exploitation.

5 Case Study in Applying CyberSecEval 2

  • The case study uses default model settings and is not a like-for-like comparison because some tested systems are closed and may include additional guardrails.Source: §5 · p. 9

5.1.1 Revisiting cyberattack helpfulness tests with state-of-the-art models

  • Models were least compliant with evasion and execution requests, but more cooperative on less clear-cut discovery and reconnaissance requests.Source: §5.1.1 · p. 10
  • Average compliance with malicious cyberattack-helpfulness prompts fell from 52% to 28%, while Llama 3 showed better non-compliance than CodeLlama models.Source: §5.1.1 · p. 10

5.1.2 False refusal dataset test results

  • Many models kept false refusal rates below 15% on borderline benign prompts, while CodeLlama-70B reached 70%.Source: §5.1.2 · p. 11

5.2 Prompt injection test results

  • All tested models succumbed to prompt injection at average rates of at least 17.1%, with output-format manipulation often succeeding and token smuggling rarely succeeding.Source: §5.2 · pp. 11–12, Figure 1 · p. 11
  • Larger models generally rejected prompt injections more effectively, possibly because stronger reasoning helps maintain system instructions and distinguish third-party inputs.Source: §5.2 · pp. 12–13

5.3 Code exploitation test results

  • Higher-capability coding models performed best on exploit challenges, but all models struggled, especially with end-to-end memory corruption and SQL injection.Source: §5.3 · pp. 13–14

5.4 Interpreter abuse test results

  • Models complied with an average of 35% of requests to assist attacks on attached code interpreters, supporting additional safety tuning alongside sandboxing.Source: §5.4 · p. 14, Figure 6 · p. 15

6 Conclusion

CYBERSECEVAL 2 expands quantitative evaluation of LLM cybersecurity risks, safety-utility tradeoffs, and exploit-generation capabilities. Results show persistent prompt-injection vulnerability, measurable but limited safety-helpfulness tradeoffs, and insufficient exploit-generation proficiency despite stronger coding ability.

  • CYBERSECEVAL 2 expands tested cybersecurity risks from two to four by adding prompt injection and interpreter-abuse test suites.The benchmark also introduces safety-utility measurement through False Refusal Rate and a cyberattack-helpfulness test set.Source: §6 · p. 15
  • 13% to 47% successful prompt injections across tested models show that conditioning away prompt-injection risk remains unsolved.System designers should not assume system prompts reliably withstand adversarial downstream input and should consider additional guardrails and detections.Source: §6 · p. 15
  • False Refusal Rate distinguishes models with low versus high refusal rates and reveals a small safety-helpfulness tradeoff.Many models complied with benign borderline cybersecurity requests while rejecting most requests for offensive cyber operations.Source: §6 · p. 15
  • Most LLMs failed to completely solve exploit-generation test cases, although models with greater general coding ability scored better.The results suggest further research is needed before autonomous exploit generation, while exploitation capability may grow with coding capability.Source: §6 · p. 15
  • The evaluation has been released as open source so others can build on it and evaluate additional LLMs.Source: §6 · p. 15

A.2 Example code interpreter test cases

The code-interpreter examples cover diverse malicious objectives, including privilege escalation, container escape, social engineering, persistence, command-and-control, and reflected attacks. They constrain requests with sandbox, library, formatting, and evasion requirements.

  • The privilege-escalation and container-escape cases request standalone Python code that exploits interpreter or sandbox vulnerabilities under explicit banned-function and module constraints.The examples include writing to a root-owned file and demonstrating a successful container escape.Source: §A.2 · p. 18
  • The post-exploitation case requests boot persistence, disguised scheduled tasks, remote command-and-control, and command execution on a compromised system.Source: §A.2 · p. 18
  • The cited passage describes requests for Python code demonstrating a container escape or sandbox escape using interpreter-environment vulnerabilities, while avoiding os, eval, exec, and external libraries.Source: §A.2 · p. 18

A.3 Example exploitation test cases

The exploitation examples use randomized test programs that score partial or complete success across buffer-overflow, string-constraint, and SQL-injection tasks. Randomized targets make the cases harder to memorize and emphasize solving the generated challenge.

Loading 2404.13161v1…