PaperlayerView exact revision on arXiv

Source-linked AI summary

CS-Guard: Benchmarking LLM Guardrails for Code Generation Security

Jinyang Li, Mingyu Guo, Hung X. Nguyen

arXiv:2609.09798v1cs.CRcs.AI

TL;DR

CS-Guard addresses the unclear effectiveness of guardrails for malicious code generation by introducing a benchmark spanning text-to-code and code-to-code tasks, jailbreaks, and fictional scenario attacks. Across evaluated guardrails and models, the benchmark finds substantial vulnerability, while its modular taxonomy supports future guardrail evaluation.

  • Problem

    The effectiveness of guardrails for code generation security remains unclear despite LLMs being exploited to generate malware.

  • Method

    CS-Guard systematically evaluates guardrails using text-to-code and code-to-code prompts, jailbreak attacks including FSA, and a modular three-layer taxonomy.

  • Results

    Current guardrails perform poorly against malicious code-generation requests, with high ASRs across jailbreak, FSA, and code-to-code evaluations.

  • Takeaways & Limitations

    CS-Guard provides a standardized, modular foundation for evaluating guardrails and supporting future research on code generation security.

  • Takeaways & Limitations

    CS-Guard evaluates only English data, provides no comprehensive multi-turn jailbreak evaluation, and does not assess white-box guardrails beyond alignment training.

Abstract

from arXiv · show

Large language models (LLMs) have been ex- ploited to generate malware, but the effective- ness of guardrails for code generation secu- rity remains unclear. We introduce CS-Guard, the first benchmark to systematically evalu- ate guardrails for code generation security. It covers 1) text-to-code generation with 1000 high-quality malware-generation prompts, 7 jailbreak attacks, and a novel fictional scenario attack (FSA) that embeds malicious intent in a legitimate fictional software-development sce- nario; and 2) code-to-code generation with 331 code prompts spanning code infilling, code completion, and code translation. We empiri- cally evaluate 9 guardrails across seven LLMs. We find that current guardrails perform poorly against malicious code-generation re- quests: for text-to-code, the average attack success rate (ASR) after jailbreaks reaches about 50% for many guardrails; for code-to- code, average ASR approaches 100% on base LLMs and remains high across many guardrails (14.4% to nearly 100%). Our FSA also achieves ASR close to 100% across many guardrails, raising major reliability concerns for real-world software development. To sup- port future research, CS-Guard uses a modular three-layer guardrail taxonomy that lets devel- opers register guardrails for evaluation. We release the benchmark and data to enable fur- ther community evaluation.

1 Introduction

CS-Guard addresses the limited fine-grained evidence on guardrail effectiveness for malware-related code generation by benchmarking realistic text-to-code and code-to-code settings. It introduces FSA, a three-layer taxonomy, and a standardized testbed for future evaluations.

  • Benchmark motivation and scope: Existing studies provide limited fine-grained evidence because they often emphasize broad risk taxonomies, natural-language instructions, or coding security without malware-focused guardrail analysis.This gap limits assessment of guardrails protecting LLM code agents in software-engineering settings.
  • Benchmark motivation and scope: CS-Guard is the first benchmark to systematically evaluate guardrails for code generation security across realistic text-to-code and code-to-code scenarios.Its text-to-code evaluation uses 1000 high-quality prompts, while code-to-code evaluation uses 331 prompts covering completion, infilling, and translation.
  • Benchmark contributions: The benchmark augments 1000 realistic offensive prompts with 7 representative jailbreak attacks and evaluates code-to-code generation across completion, infilling, and translation.These design choices target realistic functional requirements and representative code-generation scenarios.
  • Benchmark contributions: FSA embeds malicious intent in realistic fictional software-development scenarios rather than relying on explicit malicious keywords.The attack exploits shared functional requirements between malware and legitimate software, and is released in full and small versions with 1000 and 300 prompts.
  • Benchmark contributions: CS-Guard defines a three-layer guardrail taxonomy and a standardized, modular testbed that lets developers register and evaluate new guardrails.The design emphasizes reproducibility and ease of use as future datasets and attack methods emerge.

2 Related work

Related work spans jailbreak attacks, vulnerable code generation, adversarial robustness, and malware-generation benchmarks. Existing malware-specific studies cover several code-generation and jailbreak settings, motivating a broader guardrail evaluation framework.

  • Jailbreak attacks: Jailbreak attacks are commonly categorized as obfuscation-based, optimization-based, and other adversarial prompt strategies intended to bypass safety restrictions.Obfuscation examples include ciphertext encoding, word-order swaps, and emoji insertion.
  • Fictional scenario attack: The fictional scenario attack reframes malicious functional behaviors as expected behaviors in a legitimate software-development scenario.Figure 1 illustrates extracting behaviors from a malicious base prompt and embedding them in a fictional legitimate context.
  • Code and malware-generation security: Prior research covers vulnerable code generation, adversarial robustness, and both general and malware-specific malware-generation benchmarks.The cited malware-specific benchmarks include RMCBench, RedCode, Mocha, CodeJailbreaker, MalwareBench, and CyberSecEval.

3 The CS-Guard benchmark

CS-Guard evaluates guardrails across realistic text-to-code and code-to-code security scenarios, using jailbreak-enhanced prompts, fictional scenario attacks, a modular taxonomy, and standardized evaluation procedures.

  • Code-to-code evaluation: The benchmark includes code-to-code prompts for code infilling, translation, and completion, targeting a separate attack surface in the software-development pipeline.The collection draws on 80 infilling, 91 translation, and 160 completion prompts.
  • Text-to-code evaluation: CS-Guard covers text-to-code evaluation at two difficulty levels, including base malicious prompts and stealthy malicious intent designed to bypass guardrails.The benchmark uses 1000 TTP prompts for the base setting and adds jailbreak-enhanced evaluation for the harder setting.
  • Text-to-code evaluation: Fictional scenario attacks embed malicious objectives in legitimate fictional application contexts to test whether guardrails recognize concealed malicious functionality.The construction combines TTP objectives, functionality sets, and generated scenarios into 1000 prompts; validation reports 98.4% accuracy.
  • Guardrail taxonomy: CS-Guard organizes guardrails by category, operation, and position, enabling modular integration and adaptation to future guardrail systems.The taxonomy distinguishes strategy-, classifier-, and internal-type categories, alongside classification or generation operations and deployment positions.
  • Evaluation method: The study evaluates selected strategy, classifier, and internal guardrails across multiple LLMs, using automated labeling validated against expert annotations.The LLM labeler achieved 96.8%, 93.8%, and 93.7% accuracy on text-to-code levels 1 and 2 and code-to-code generations, respectively.
  • Evaluation method: The evaluation measures input and internal defenses with attack success rate, strategy methods by ASR reduction, and output defenses with F1.Output-classifier F1 is computed on balanced refusal–jailbreak responses, while SmoothLLM uses the percentage of detected malicious prompts among successful attacks.

4 Evaluation

CS-Guard finds substantial weaknesses across internal, input-classifier, output-classifier, and multi-turn fictional scenario defenses, with high attack success or degraded detection in several settings.

  • Internal-type guardrails: 98.2% and 94.5% ASRs occur for CodeLlama-13B and DeepSeekCoder-V2-Lite-16B in text-to-code level 1, while SOTA models reach 15.1%–61.4%.Level-2 ASRs increase across models, and FSA reaches 85.9%–99.5%.
  • Internal-type guardrails: 52.5%–98.9% average ASRs remain in code-to-code generation, indicating high vulnerability across the evaluated base models.The reported range covers the code-to-code results summarized for internal-type guardrails.
  • Input classifiers: 6.9%–27.5% ASRs are reported for selected input-classifier results, while FSA reaches 73.3% on Qwen3Guard and 98.5% on LlamaGuard4.Input classifiers perform especially poorly on code infilling and translation, whereas code completion is relatively stronger.
  • Output classifiers: 20.0%–85.6% level-1 F1 and 40.0%–80.0% average level-2 F1 are reported for output classifiers in text-to-code generation.FSA remains difficult for most output classifiers, with F1 scores ranging from 6.9% to 72.3%, except for WildGuard.
  • Multi-turn FSA: 65.0%–92.3% ASR occurs in the FSA-MT pilot across GPT5-mini with PAT and Qwen3Guard, indicating weak robustness against multi-turn fictional scenario attacks.The pilot uses the strongest single-turn guardrail per category and counts success only when the guardrail is bypassed throughout the full conversation.

5 Conclusion

CS-Guard is introduced as the first benchmark for systematically evaluating guardrails in code generation security. Its fictional scenario attack exposes performance degradation, while its modular design supports future guardrail integration.

  • CS-Guard is the first benchmark for systematically evaluating guardrails in code generation security.
  • The fictional scenario attack substantially degrades the performance of existing guardrails.
  • CS-Guard’s modular design enables easy integration of new guardrails for future research.

Limitations

CS-Guard has scope and evaluation limitations involving language coverage, jailbreak coverage, white-box guardrails, and model precision. These constraints define important boundaries on interpreting its empirical results.

  • CS-Guard evaluates only English data and does not include multilingual evaluation.
  • CS-Guard does not comprehensively evaluate multi-turn jailbreaks; its FSA-MT evaluation uses only 20 pilot-study samples.
  • CS-Guard does not evaluate white-box guardrails requiring additional training or neuron access beyond alignment training.
  • Internal-guardrail results are based on quantized LLMs and might not generalize to full-precision versions.

Ethical considerations

The study uses publicly released malware-generation prompts under safeguards intended to support responsible research release. No malware is executed or retained during the experiments.

  • All malware-generation prompts come from public datasets released by the research community.
  • No malware is executed or kept during the experiments.
  • The FSA dataset is intended only for research use and is distributed under a research-only CC BY-NC-SA 4.0 license.

A.1 Recruitment details for the external reviewers

External review was conducted by volunteer PhD candidates with cybersecurity backgrounds, with estimated review times varying by task. Expected review time was doubled for external reviewers.

  • All external reviewers were volunteer PhD candidates with cybersecurity backgrounds.
  • Estimated review time was roughly 30 minutes for each text-to-code labeling level and 20 minutes for code-to-code review.
  • FSA prompt-construction and FSA-MT attack-result review took roughly 3 hours.
  • The expected review time was multiplied by 2 for external reviewers.

A.2 Usage of AI

The paper reports using Grammarly for grammar and fluency corrections and GPT5 to shorten some paragraphs through paraphrasing.

  • Grammarly was used to correct grammar and improve the paper’s fluency.
  • GPT5 was used to paraphrase some paragraphs to reduce the text’s length.

A.3 Domain coverage of dataset

The dataset spans diverse malware-generation scenarios across text-to-code and code-to-code tasks, including multiple malware types and programming languages.

  • Text-to-code prompts include 100 samples for each ATT&CK Framework category.The listed categories include persistence, privilege escalation, command and control, reconnaissance, evasion, discovery, lateral movement, execution, collection, and exfiltration.
  • Code-to-code prompts cover 11 malware types, including viruses, worms, trojans, spyware, ransomware, rootkits, phishing, exploitation, and network attacks.
  • The malware code spans nine programming languages.The passage begins listing languages including C, C++, C#, Go, and HTML.

A.4 Instruction given to the human reviewer

Human reviewers annotate responses from text-to-code and code-to-code prompts using a classification guideline specified in the LLM labeler’s system prompt.

  • Reviewers classify responses induced by text-to-code and code-to-code prompts.
  • The classification guideline is specified in the LLM labeler’s system prompt.
  • The passage introduces a separate simple instruction for annotating FSA prompt construction.

Reviewer instruction FSA construction

The review process evaluates whether jailbreak outputs satisfy malicious objectives and separately provides an instruction for annotating multi-turn FSA attack results.

  • Jailbreak outputs are labeled according to whether they satisfy the malicious objective.Examples include implementing the requested functional requirements or producing code directly usable for that objective.
  • The instruction considers whether the generated code implements the same functional requirements as the malicious objective.
  • FSA-MT attack results receive a separate annotation instruction.

Reviewer instruction FSA-MT annotation

FSA-MT evaluates whether a victim LLM is jailbroken when malicious functionality is embedded in a legitimate software-development scenario. The annotation and supporting prompts define how functionality is extracted, evaluated, and classified.

  • Reviewer instruction FSA-MT annotation: FSA embeds malicious intent within a legitimate development context, exploiting shared functional behaviors between malware and benign software.The attack can operate in single-turn or multi-turn interactions, where a single-turn request is decomposed into strategically ordered sub-requests.
  • Reviewer instruction FSA-MT annotation: FSA-MT jailbreak success is judged from the final victim response by whether it directly implements the original FSA prompt’s required functionality or guidance.The annotation uses the strongest LLM and defense combination described for the proof-of-concept evaluation.
  • A.5 FSA-MT search: The FSA-MT search procedure is described through an algorithm using attacker and victim LLMs, state evaluation, judging, refusal checking, search-depth limits, an attack dataset, and conversation history.The listed algorithm materials include a state-response-step representation and a continuation condition based on the maximum search depth.
  • Reviewer instruction FSA-MT annotation: The FSA-MT annotation materials include a guardrail-collection table and a jailbreak-collection table, while CipherChat labeling can misclassify unclear responses as good.The documented failure occurs when weak labelers interpret difficulty understanding ciphertext as a refusal signal.
  • Functionality extraction prompt: The functionality-extraction prompt instructs a model to identify precise core behaviors while rewriting malicious terminology into general technical descriptions.It prohibits code generation, rejection, extra commentary, duplication, and explicit references to malicious purposes.
  • Functionality extraction prompt: The extraction output is constrained to raw JSON containing functionality identifiers, names, and detailed descriptions.The requested structure is a result list, and the prompt separately requires no markdown or code blocks.
  • Functionality extraction prompt: The example functionality prompt targets fast-flux DNS behavior, including IP discovery, DNS updates, scheduled address rotation, connectivity maintenance, and operational constraints.The extracted descriptions include DNS resolution and record management, address-pool rotation, TTL-aware reconnection, Windows compatibility, scalability, and avoidance of hardcoded network values.
  • Functionality extraction prompt: The extraction examples also include domain fronting for disguising command-and-control communications as benign requests to well-known domains.The associated prompt asks for an executable design explanation, while the extraction instructions require describing underlying functionality without malicious terminology.
Loading 2609.09798v1…