PaperlayerView exact revision on arXiv

Source-linked AI summary

Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code

Yitong Zhang, Shiteng Lu, Jia Li

arXiv:2606.11817v1cs.CRcs.AIcs.CLcs.SE

TL;DR

Grammar-constrained decoding can disable natural-language refusals and expose a code-modality safety gap. The paper introduces CodeSpear and CodeShield, finding that CodeSpear substantially increases attack success while CodeShield restores safety under grammar constraints without materially harming benign utility.

  • Problem

    Existing safety alignment primarily teaches natural-language refusals, leaving safe behavior in the code modality under grammar constraints insufficiently addressed.

  • Method

    CodeShield aligns the code modality by training models to generate semantically harmless, structurally diverse honeypot code under grammar-constrained decoding.

  • Results

    Across 10 LLMs and 4 benchmarks, CodeSpear achieved an average attack success rate of 81.82%, while CodeShield restored safety and preserved benign utility.

  • Takeaways & Limitations

    Grammar-constrained decoding has security implications, and code-modality alignment can preserve safe behavior when natural-language refusals are unavailable.

  • Takeaways & Limitations

    The evaluation may not cover all possible malicious code generation scenarios, despite using two complementary malicious-code benchmarks.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications.

I. INTRODUCTION · II. BACKGROUND AND RELATED WORK · A. Grammar-Constrained Decoding

The paper identifies CodeSpear, a jailbreak that exploits Grammar-Constrained Decoding (GCD) to induce malicious code generation, and proposes CodeShield, which preserves safety through harmless, structurally diverse honeypot code. It motivates this problem by showing how GCD reshapes valid outputs and removes the natural-language refusal behavior on which existing alignment relies.

  • I. INTRODUCTION: Code generation makes jailbreaks especially dangerous because harmful outputs can be executable programs weaponized against digital systems.
  • I. INTRODUCTION: CodeSpear exploits widely used GCD to induce LLMs to generate malicious code, turning a reliability-oriented mechanism into an attack surface.
  • I. INTRODUCTION: Existing safety alignment mainly teaches natural-language refusals, but enforced code grammars can make those refusals unavailable during inference.
  • I. INTRODUCTION: 81.82% average attack success rate was achieved by CodeSpear against locally deployed LLMs such as Qwen2.5-Coder-7B.
  • I. INTRODUCTION: More than 40 percentage points average attack-success-rate increase was observed on commercial API-based LLMs such as GPT-5, while CodeShield restored safety under GCD.
  • I. INTRODUCTION: CodeShield aligns the code modality by training models to generate honeypot code that is semantically harmless and structurally diverse under attacker-controlled grammars.
  • A. Grammar-Constrained Decoding: GCD improves code-generation reliability by restricting outputs to sequences accepted by a target grammar and masking tokens that cannot lead to valid programs.
  • A. Grammar-Constrained Decoding: By changing the valid output support from V∗ to L(G), GCD leaves the model and prompt unchanged while reshaping the space of possible responses.

B. Jailbreaking and Safety Alignment of LLMs

The section reviews jailbreak attacks and safety alignment, emphasizing that existing alignment primarily trains natural-language refusals. It motivates adapting preference-pair construction so models can learn safe behavior in the code modality when natural-language refusals are unavailable.

  • Jailbreaking: Jailbreak attacks broadly target LLM inputs or the model itself, including carefully crafted prompts and harmful-data fine-tuning.PAIR iteratively refines jailbreak prompts using feedback from the target model.
  • Safety Alignment: Safety alignment uses supervised fine-tuning, preference optimization, or reinforcement learning to make models prefer safe responses to harmful prompts.Existing formulations encourage probability mass on refusal responses such as “I cannot assist with that.”
  • Safety Alignment: Existing alignment grounds safe behavior in natural language, leaving code-modality safety underexplored when natural-language refusals are unavailable.This gap motivates safety training that remains effective under code-only generation.
  • Safety Alignment: DPO optimizes pairwise response preferences against a fixed reference model, while this defense changes preference-pair construction to teach safe code-modality behavior.Standard safety alignment pairs natural-language refusals with harmful completions; the proposed approach keeps the DPO objective but changes those pairs.

III. THREAT MODEL · A. Attacker Setting

The threat model considers an attacker who uses grammar-constrained decoding to bypass an LLM’s safety alignment and elicit malicious code. The attacker can submit arbitrary prompts and supply benign grammars through local or API-based inference interfaces.

  • III. THREAT MODEL: The threat model specifies practical attacker and defender perspectives for grammar-constrained code generation.
  • A. Attacker Setting: The attacker queries a target LLM through an inference interface supporting grammar-constrained decoding.The paper identifies local deployment with frameworks such as VLLM and SGLANG as one realistic setting.
  • A. Attacker Setting: API-based deployment also exposes grammar interfaces for specifying constraints on models from providers including OpenAI and Fireworks.Examples include GPT-5 and MiniMax-M2.7.
  • A. Attacker Setting: The attacker aims to bypass safety alignment and induce malicious code for denial-of-service attacks, malware, or credential theft.An attack succeeds when the model partially or fully implements the malicious requirement instead of refusing or generating harmless content.
  • A. Attacker Setting: The attacker supplies a malicious prompt p, benign grammar G, and maximum length T_max to generate output y.The procedure initializes an empty prefix, repeatedly computes the grammar-valid token set, samples a token conditioned on the prompt and prefix, and stops at end of sequence or T_max.
  • A. Attacker Setting: The attacker has two capabilities: submitting arbitrary prompts and providing a grammar such as standard Python grammar.Both capabilities are described as readily available in the local and API-based deployment scenarios.

B. Defender Setting · IV. METHODOLOGY · A. CodeSpear

The defender must preserve safe behavior across unconstrained and grammar-constrained inference settings, while CodeSpear exploits GCD by excluding natural-language refusals and forcing grammar-valid code generation. The attack requires only an ordinary grammar and an existing GCD interface, without optimization, fine-tuning, or prompt engineering.

  • B. Defender Setting: The defender is the model developer responsible for safety alignment across unconstrained and grammar-constrained decoding configurations.
  • B. Defender Setting: The defense goal is to refuse malicious requests when natural language is allowed and avoid malicious code generation under GCD.
  • B. Defender Setting: The defender has full model-parameter access and may align on safety data, but cannot rely on inference-time input filtering.This reflects local deployment, where downstream users control inference configuration, and API deployment constraints described in the passage.
  • A. CodeSpear: CodeSpear exploits a mismatch between existing safety alignment and grammar-constrained decoding for malicious code-generation prompts.
  • A. CodeSpear: Under GCD, the valid output space changes from V∗ to L(G), generally excluding natural-language refusals.
  • A. CodeSpear: GCD removes the learned refusal and forces generation in a code modality without explicit safe-behavior alignment, creating an attack surface.
  • A. CodeSpear: CodeSpear invokes a target model through a standard GCD interface using a malicious prompt and an ordinary code grammar.
  • A. CodeSpear: The output necessarily satisfies y ∈ L(G), steering generation toward grammar-valid code that may implement the user’s malicious requirement.Algorithm 1 summarizes the full procedure.

B. CodeShield

CodeShield trains models to preserve safe behavior under attacker-controlled grammar constraints by generating structurally diverse, semantically harmless honeypot code. Using DPO, it prefers natural-language refusals when available and honeypot code over harmful code when GCD restricts outputs to code.

  • Method: CodeShield trains the model to generate honeypot code that is semantically harmless and spans diverse syntactic structures.This gives the model multiple safe responses within the valid code space, making suppression difficult without excluding structures needed by malicious programs.
  • Method: DPO conditions safe behavior on the valid output space: natural-language refusal when available, and honeypot code when GCD permits only code.The training goal directly addresses the loss of natural-language refusals under grammar-constrained decoding.
  • Preference Pair Construction: Honeypot snippets are sampled from a broad code corpus, so they do not implement malicious requirements while providing structurally diverse harmless responses under GCD.The corpus-based sampling supplies many syntactic forms for the honeypot side of preference construction.
  • Training Objective: DPO optimizes preferences for refusal over code when natural language is available and honeypot code over harmful code under code-only constraints.The objective is designed to close the attack surface exposed by CodeSpear.

V. EXPERIMENTAL SETUP · A. Research Questions

The experimental setup evaluates CodeSpear’s attack effectiveness, CodeShield’s defense and utility preservation, and both methods’ sensitivity to key factors through five research questions. The studies cover locally deployed and commercial API-based LLMs, defensive alignment, benign code generation, and hyperparameter variation.

  • V. EXPERIMENTAL SETUP: The experiments address five research questions assessing CodeSpear and CodeShield.The paper presents these questions as the framework for its comprehensive experimental evaluation.
  • A. Research Questions: CodeSpear’s effectiveness against locally deployed LLMs is evaluated on 5 models spanning different families, parameter scales, and training regimes.RQ1 tests whether CodeSpear can bypass safety alignment in local deployment.
  • A. Research Questions: CodeSpear’s effectiveness against commercial API-based LLMs is evaluated on 5 models in the more restrictive API-based deployment scenario.RQ2 examines whether the attack remains effective under commercial API constraints.
  • A. Research Questions: CodeShield’s ability to defend against CodeSpear and prevent malicious code generation under GCD is evaluated on three popular models.RQ3 examines whether CodeShield improves intrinsic model safety under grammar-constrained decoding.
  • A. Research Questions: CodeShield’s preservation of benign utility is tested by comparing models with and without CodeShield on general-purpose code generation benchmarks.RQ4 addresses whether safety alignment causes a safety tax that degrades general model capabilities.
  • A. Research Questions: RQ5 studies CodeSpear and CodeShield sensitivity by varying the adopted grammar and alignment hyperparameters.The evaluation tests whether CodeSpear depends on the grammar and whether CodeShield depends on alignment hyperparameters.

B. Models · C. Benchmarks · D. Metrics

The evaluation covers 10 representative LLMs across local and API-based deployments, using safety and utility benchmarks with complementary safety and pass@k metrics. CodeSpear and CodeShield are assessed for safety effects, while utility tests examine whether alignment preserves general code generation.

  • B. Models: The evaluation covers 10 representative models across two deployment settings.The models span local and API-based deployments.
  • B. Models: Local evaluation includes Qwen2.5-Coder-7B/32B, Qwen2.5-7B/32B, and LLaMA3-8B.These models test generalization across architectures and training regimes.
  • B. Models: API-based evaluation includes GPT-5, GPT-5-mini, MiniMax-M2.5, MiniMax-M2.7, and GPT-OSS-120B.These models represent commercial platforms, restrictive deployment conditions, and frontier models.
  • C. Benchmarks: The benchmarks comprise safety evaluations of CodeSpear and CodeShield and utility evaluations of whether CodeShield degrades general code generation.Safety benchmarks measure resistance to malicious coding requests.
  • C. Benchmarks: Safety evaluation uses RMCBench’s code-generation subset and MalwareBench’s original subset of 320 malicious requests.RMCBench includes non-jailbreak Levels 1–2 and jailbreak-augmented Level 3 requests; MalwareBench also contains jailbreak-augmented variants, which are not adopted here.
  • C. Benchmarks: Utility evaluation uses HumanEval’s 164 programming tasks and MBPP’s 974 tasks with descriptions, reference solutions, and automated tests.HumanEval mainly relies on built-in functions and requires no third-party libraries.
  • D. Metrics: Safety is measured with Attack Success Rate, which counts harmful responses, and Malicious Rate, which measures functional realization of malicious intent.An LLM-based judge determines whether generated responses are harmful for ASR.
  • D. Metrics: Utility is measured with pass@k, the fraction of problems for which at least one of k generated solutions passes all provided tests.This metric evaluates whether CodeShield preserves benign code generation capability.

E. Baselines · F. Training Data Used for CodeShield

The paper evaluates CodeSpear against seven threat-model-compatible jailbreak baselines and CodeShield against vanilla safety alignment and Safe-DPO. CodeShield training uses filtered PKU-RLHF data, producing 744 malicious code-generation seed prompts before augmentation.

  • E. Baselines: CodeSpear is compared with seven representative baselines compatible with the attacker threat model, excluding methods requiring gradient-based optimization over the target model.The excluded methods are considered impractical for the attacker described in Section III-A.
  • E. Baselines: Vanilla measures default resistance, while Vanilla-T, DAN, LRL, and PAIR represent template-based, role-playing, prompt-rewriting, and multi-turn jailbreak attacks.LRL translates requests into Swahili, and PAIR iteratively refines prompts through attacker–target interactions.
  • E. Baselines: APT is technically closest to CodeSpear because both exploit constrained decoding, but APT crafts an adversarial grammar per request whereas CodeSpear uses an off-the-shelf benign code grammar.APT iteratively queries the target model to construct its adversarial grammar.
  • E. Baselines: CodeJailbreaker is included as a code-generation-specific baseline that wraps malicious coding requests as commit messages.Its design is closely aligned with the code-generation threat scenario.
  • E. Baselines: CodeShield is compared with two representative defenses because prior work lacks a defense specifically designed for code-modality safety alignment under GCD.The listed defense baselines are Vanilla and Safe-DPO.
  • E. Baselines: Safe-DPO uses natural-language refusals as chosen responses and harmful code as rejected responses, unlike CodeShield’s preferred honeypot code in the code modality.It also serves as an ablation of CodeShield’s code-modality alignment through honeypot code.
  • F. Training Data Used for CodeShield: 744 seed prompts are obtained by using Qwen3-32B to filter PKU-RLHF for malicious code-generation requests.CodeShield requires preference data, and existing safety-alignment datasets are not tailored to code generation.
  • F. Training Data Used for CodeShield: The dataset construction begins from the widely used PKU-RLHF dataset and proceeds toward augmentation because safety alignment typically requires thousands of training examples.The supplied passage ends while describing the augmentation stage, without specifying its completed size or procedure.

G. Other Implementation Details … B. RQ2: Effectiveness of CodeSpear on API-based LLMs

The paper specifies reproducible implementation settings and evaluates CodeSpear on locally deployed and API-based LLMs. CodeSpear substantially improves jailbreak effectiveness, including against restrictive API deployments, while fixed safe-output patterns remain vulnerable to grammar tightening.

  • G. Other Implementation Details: CodeSpear uses llguidance with off-the-shelf Python grammars, while CodeShield uses K=5 honeypot samples, learning rate 1e-5, one epoch, and 40k OpenCodeInstruct tasks.General-purpose code-generation data is added through supervised fine-tuning to preserve model utility.
  • G. Other Implementation Details: Experiments use temperature 0.9 and top-p 0.95, repeat each experiment three times, and report average results.Baselines use common hyperparameters and method-specific settings, while CodeShield baselines match its training data and configuration for fairness.
  • A. RQ1: Effectiveness of CodeSpear on Locally Deployed LLMs: On locally deployed LLMs, CodeSpear is evaluated against jailbreak baselines on two safety benchmarks using ASR and MR.The study covers five locally deployed models and reports results across model–benchmark–metric combinations.
  • A. RQ1: Effectiveness of CodeSpear on Locally Deployed LLMs: 12 of 20 model–benchmark–metric combinations favor CodeSpear, while average gains over Vanilla reach 26.90 percentage points for ASR and 17.98 for MR.On Qwen2.5-Coder-7B and MalwareBench, ASR rises from 29.79% to 83.44%, versus 69.27% for PAIR.
  • A. RQ1: Effectiveness of CodeSpear on Locally Deployed LLMs: Only CodeSpear and CodeJailbreaker achieve positive average gains in both ASR and MR over Vanilla, indicating weaker performance from generic jailbreak attacks on malicious code generation.The paper attributes this gap to malicious code generation’s stronger dependence on model capabilities and the quality costs of generic attacks.
  • B. RQ2: Effectiveness of CodeSpear on API-based LLMs: On API-based LLMs, CodeSpear is tested on five models using the same two benchmarks and ASR/MR metrics as RQ1.The setting targets models with stronger safety alignment and possible inference-time safeguards.
  • B. RQ2: Effectiveness of CodeSpear on API-based LLMs: On MiniMax-M2.7 and RMCBench, CodeSpear raises ASR from 20.33% to 85.53% and MR from 17.40% to 64.29%.These resulting values are more than twice the strongest-baseline CodeJailbreaker results of 40.11% ASR and 25.64% MR.
  • B. RQ2: Effectiveness of CodeSpear on API-based LLMs: CodeSpear’s roughly 50% ASR against GPT-5 and GPT-5-mini often reflects pass statements, but tightening the grammar to disallow pass can bypass this fixed pattern.The findings support the conclusion that CodeSpear remains effective on API-based deployments while fixed safe behaviors are fragile.

C. RQ3: Effectiveness of CodeShield Against CodeSpear

CodeShield restores model safety when GCD restricts outputs to code, reducing attack and maliciousness rates with and without CodeSpear. Unlike standard alignment, it remains effective when natural-language refusals are unavailable by aligning models toward honeypot code.

  • Evaluation setting: The evaluation covers three models, two safety benchmarks, two inference settings, and ASR and MR metrics, comparing normal decoding with attacker-provided code grammars.The models are Qwen2.5-Coder-7B, Qwen2.5-7B, and LLaMA3-8B.
  • CodeShield effectiveness: CodeShield consistently reduces ASR and MR across all three models both without and with CodeSpear.Without CodeSpear on Qwen2.5-Coder-7B, average ASR and MR fall from 28.36% and 22.05% to 2.08% and 0.80%.
  • CodeShield effectiveness: Under CodeSpear, CodeShield reduces Qwen2.5-Coder-7B’s average ASR and MR from 83.11% and 54.12% to 5.57% and 2.78%.This demonstrates safety restoration when the model is forced to generate code.
  • Comparison with standard alignment: Safe-DPO improves safety when natural language remains available but becomes largely ineffective once CodeSpear constrains the model to generate code.On LLaMA3-8B without CodeSpear, Safe-DPO reaches 9.52% ASR and 3.41% MR, close to CodeShield’s 4.82% and 1.70%.
  • Answer to RQ3: CodeShield restores safety under GCD by aligning models toward honeypot code, whereas standard safety alignment remains insufficient without natural-language refusal.The reported answer to RQ3 directly attributes CodeShield’s robustness to honeypot-code alignment.

D. RQ4: Benign Utility Preservation of CodeShield · E. RQ5: Sensitivity Analysis of CodeSpear and CodeShield

CodeShield preserves benign code-generation utility with only acceptable degradation, while CodeSpear remains effective across programming-language grammars and CodeShield gains safety as more honeypot samples are used without materially reducing utility.

  • D. RQ4: Benign Utility Preservation of CodeShield: On Qwen2.5-Coder-7B, CodeShield reduces MBPP pass@3 only from 78.00% to 77.00%.This example illustrates the reported acceptable utility degradation.
  • D. RQ4: Benign Utility Preservation of CodeShield: On Qwen2.5-7B, CodeShield increases MBPP pass@1 from 37.40% to 44.93%.The paper attributes such utility behavior to utility-preservation data introduced during safety alignment.
  • D. RQ4: Benign Utility Preservation of CodeShield: CodeShield has an acceptable impact on general code-generation capability.The evaluation measures pass@k on HumanEval and MBPP across Qwen2.5-Coder-7B, Qwen2.5-7B, and LLaMA3-8B.
  • E. RQ5: Sensitivity Analysis of CodeSpear and CodeShield: CodeSpear’s practical effectiveness depends on the grammar used for grammar-constrained decoding, whereas CodeShield’s depends on the number of honeypot code samples K.The sensitivity study varies Python, C++, and Java grammars and K ∈ {1, 3, 5, 7, 10}.
  • E. RQ5: Sensitivity Analysis of CodeSpear and CodeShield: On Qwen2.5-Coder-7B, CodeSpear’s ASR stays below 40% without GCD but exceeds 70% with GCD for every tested grammar.The result is reported using average ASR on RMCBench and MalwareBench.
  • E. RQ5: Sensitivity Analysis of CodeSpear and CodeShield: As K increases from 1 to 10, ASR decreases while pass@1 remains nearly unchanged.The findings support sampling multiple honeypot code responses, which provides more preference pairs comparing harmful code with semantically harmless alternatives.
  • E. RQ5: Sensitivity Analysis of CodeSpear and CodeShield: CodeSpear remains effective across different grammars, while CodeShield benefits from more honeypot code samples without degrading benign utility.These findings summarize the sensitivity analysis across attack safety and utility measures.

VII. DISCUSSION … VIII. CONCLUSION

The discussion finds that CodeShield remains robust against adaptive grammar-tightening attacks, while validating LLM-based safety judgments and addressing threats to validity. The conclusion reports that CodeSpear bypasses existing alignment, whereas CodeShield restores safety under GCD while preserving benign utility.

  • A. Can CodeShield Remain Robust under Adaptive Attack: CodeShield is evaluated against an adaptive attacker that knows its honeypot-code strategy and revises the grammar to suppress the observed behavior.The adaptive setting is stricter than the main evaluation, which uses a fixed off-the-shelf code grammar; DeepSeek-V4-Pro serves as the attack proxy.
  • A. Can CodeShield Remain Robust under Adaptive Attack: On Qwen2.5-Coder-7B, Qwen2.5-7B, and LLaMA3-8B with N = 10, CodeShield’s ASR and MR show little increase and sometimes decrease under adaptive attack.The paper attributes this robustness to honeypot code’s structural diversity, which makes safe behavior difficult to suppress through grammar tightening.
  • B. Reliability of LLM Judgment: Human and LLM judgments agreed on 87% of ASR evaluations and 85% of MR evaluations across 100 manually evaluated random responses.The manual validation used the same criteria as the leading LLM judges.
  • C. Threats to Validity: CodeSpear’s effectiveness may vary across GCD implementations, so the study evaluates both local and API-based deployments and tests different code grammars in RQ5.The results are presented as evidence of a general GCD risk rather than a claim about every implementation.
  • C. Threats to Validity: The evaluation uses RMCBench with 10 malicious-coding scenario types and MalwareBench with 6 malware-related scenario types to reduce dependence on one request set.The authors acknowledge that these benchmarks may not cover all malicious code-generation scenarios.
  • C. Threats to Validity: Because grammar-constrained decoding can jailbreak LLMs into generating malicious code, CodeSpear is released only to researchers for controlled research use, while CodeShield is fully released.The authors also provide CodeShield as a mitigation and encourage adoption to improve intrinsic model safety.
  • VIII. CONCLUSION: Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear bypasses existing safety alignment, while CodeShield restores safety under GCD and preserves benign utility.The conclusion calls for greater attention to GCD’s potential security implications and further efforts toward safer systems.
Loading 2606.11817v1…