PaperlayerView exact revision on arXiv

Source-linked AI summary

Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Prompts

Sheng Ouyang, Yihao Qin, Bo Lin, Liqian Chen, Xiaoguang Mao, Shangwen Wang

arXiv:2503.17953v1cs.SE

TL;DR

Code-generation research has focused more on functional correctness than safety, despite concerns about malicious outputs. CodeJailbreaker puts malicious intent in commit messages while keeping the instruction benign; on RMCBench, it outperforms explicit-prompt strategies across multiple code-generation tasks.

  • Problem

    Research on LLM-based code generation has emphasized functional correctness more than code safety, despite the growing reliance on generated code.

    Source: §1 · p. 1

  • Method

    CodeJailbreaker asks models to simulate software evolution with a benign instruction and embeds malicious intent in a commit message as a covert channel.

    Source: §4.2.1 · p. 5

  • Results

    CodeJailbreaker outperforms explicit-malicious-prompt strategies across multiple code-generation tasks on RMCBench; in text-to-code, its average ASR is nearly 80% and MR is close to 65%.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    The study highlights the need for code-generation safety measures that address implicit malicious cues.

    Source: §1 · p. 2, §8 · p. 11

    RMCBench is imbalanced by programming language: 232 of 282 cases concern Python, while Java and Bash have one case each.

    Source: §7.2 · p. 10

Abstract

from arXiv · show

The proliferation of Large Language Models (LLMs) has revolutionized natural language processing and significantly impacted code generation tasks, enhancing software development efficiency and productivity. Notably, LLMs like GPT-4 have demonstrated remarkable proficiency in text-to-code generation tasks. However, the growing reliance on LLMs for code generation necessitates a critical examination of the safety implications associated with their outputs. Existing research efforts have primarily focused on verifying the functional correctness of LLMs, overlooking their safety in code generation. This paper introduces a jailbreaking approach, CodeJailbreaker, designed to uncover safety concerns in LLM-based code generation. The basic observation is that existing safety mechanisms for LLMs are built through the instruction-following paradigm, where malicious intent is explicitly articulated within the instruction of the prompt. Consequently, CodeJailbreaker explores to construct a prompt whose instruction is benign and the malicious intent is implicitly encoded in a covert channel, i.e., the commit message, to bypass the safety mechanism. Experiments on the recently-released RMCBench benchmark demonstrate that CodeJailbreaker markedly surpasses the conventional jailbreaking strategy, which explicitly conveys malicious intents in the instructions, in terms of the attack effectiveness across three code generation tasks. This study challenges the traditional safety paradigms in LLM-based code generation, emphasizing the need for enhanced safety measures in safeguarding against implicit malicious cues.

1 Introduction

The paper targets a gap in code-generation safety research and introduces CodeJailbreaker, which hides malicious intent in commit messages rather than stating it in the prompt instruction. On RMCBench, it outperforms explicit-malicious-prompt strategies across three code-generation tasks.

1 Introduction

  • The study highlights a safety gap: code-generation research has focused primarily on functional correctness, with less attention to code safety.Prior work also found that simple prompts could jailbreak models into generating malicious code despite safety strategies.Source: §1 · p. 1
  • CodeJailbreaker uses a benign instruction to simulate software evolution and encodes malicious intent in the commit message as a covert channel.The approach is motivated by safety mechanisms that predominantly rely on explicit malicious intent in instructions.Source: §1 · p. 2
  • In text-to-code, CodeJailbreaker achieves average ASR of nearly 80% and MR close to 65% across seven LLMs, improving over existing methods by more than 50%.Experiments cover text-to-code, function-level completion, and block-level completion on RMCBench.Source: §1 · p. 2
  • The experiments evaluate seven commonly used LLMs—five general and two code models—using ASR and Malicious Ratio as outcome measures.ASR measures malicious requests not rejected; Malicious Ratio measures generated code consistent with malicious intent.Source: §1 · p. 2
  • The paper argues that implicit malicious cues can evade conventional defenses and that code-generation safety measures need to address them.The study frames this as a weakness in current safety alignment strategies for LLMs.Source: §1 · p. 2

2 Background

The background covers LLMs and code-generation tasks, then reviews safety alignment, jailbreak attacks, and software evolution. It establishes commit messages as a natural-language channel that can convey intent about code changes.

2.1 Large Language Models

  • LLMs are transformer-based models trained through self-supervised pretraining and task-specific fine-tuning; code-oriented models receive additional code-focused training.The passage describes model scale, self-attention, and code-domain fine-tuning as background on LLM capabilities.Source: §2.1 · p. 2

2.2 LLM-based Code Generation

  • Code generation includes text-to-code and code-to-code tasks, with code completion generating missing segments at function or block granularity.Function-level completion starts from a function declaration; block-level completion fills a gap in a partial code snippet.Source: §2.2 · pp. 2–3

2.3.1 Safety Mechanisms.

  • LLM safety alignment commonly emphasizes instruction following, leaving models susceptible to deceptive prompts that exploit its static rules and patterns.Code-generation research has also prioritized functional correctness over safety.Source: §2.3.1 · p. 3, §2.2 · p. 3

2.3.2 Jailbreak Attack.

  • Jailbreaks use crafted prompts to bypass model safeguards; prior code-generation work found that attacks without jailbreak templates could succeed more often than those using templates.The background also distinguishes competing objectives from mismatched generalization as two failure modes of safety alignment.Source: §2.3.2 · p. 3

2.4 Software Evolution

  • Software commits record code modifications, while commit messages communicate their intent and rationale to other developers.A real-world example describes a null-value formatter fix and its intended user-facing benefit; the paper draws on this communication role to encode malicious information.Source: §2.4 · pp. 3–4

3 Motivation

The motivation contrasts explicit malicious prompts in safety-tuning and prior jailbreak examples with an implicit prompt structure. The authors report that their approach produced DDoS attack code from some models, including GPT-4.

3 Motivation

  • Safety-tuning examples explicitly articulate malicious intent so models can learn to follow safety protocols when encountering similar cues.The cited example comes from a safety-tuning dataset and shows a request that elicits a refusal to preserve neutrality.Source: §3 · p. 4
  • A prior code-generation jailbreak example directly requests DDoS code, and GPT-4 responds with a safe refusal.The passage contrasts this explicit request with the paper's proposed prompt structure.Source: §3 · p. 4
  • The authors propose departing from these standard prompt formats and report generating DDoS attack code from some models, including GPT-4.The proposed alternative is an implicit malicious prompt structure.Source: §3 · p. 4

4 Approach

CodeJailbreaker hides malicious intent in a simulated software-evolution prompt rather than stating it in the instruction. Its prompt combines benign framing with a commit message that carries the malicious code-change request, alongside code context and output constraints.

4.1 Overview

  • The attack formulates malicious intent, embeds it in a commit-based implicit prompt, feeds that prompt to an LLM, and seeks output aligned with the intent.The approach relies on malicious cues being concealed in prompt content rather than overtly stated in the instruction.Source: §4.1 · p. 5

4.2 Implicit Malicious Prompt Creation

  • Prompts are tailored to text-to-code, function-level, and block-level completion, using a benign instruction, commit message, pre-commit code, and output specification.These task types differ in how much code context the developer supplies.Source: §4.2 · p. 5

4.2.1 Benign instruction.

  • The benign instruction asks the LLM to simulate software evolution and generate post-commit code, while a software-maintainer role frames the task.The instruction does not expose malicious intent; the role-playing strategy assigns the model a software-maintainer persona.Source: §4.2.1 · p. 5

4.2.2 Commit message.

  • The commit message acts as the covert channel for malicious intent, with task-specific templates describing the requested function, body, or code block.The malicious functionality is specified in the template’s description while the request retains the appearance of routine software evolution.Source: §4.2.2 · pp. 5–6

4.2.3 Code before commit.

  • Pre-commit code supplies task-specific context: an empty main() for text-to-code, a function signature for function completion, or a gapped template for block completion.Together with the commit message, these inputs simulate code modification while encoding malicious intent.Source: §4.2.3 · p. 6

4.2.4 Output specification.

  • Output specifications constrain responses by requiring complete implementations and code-only formatting, addressing prompt repetition and facilitating code extraction.The paper says requiring implementations mitigates cases where models repeat prompt content instead of generating code.Source: §4.2.4 · p. 6

5 Experiment Setup

The experiments evaluate CodeJailbreaker across three code-generation tasks using RMCBench, seven LLMs, and both refusal and harmfulness measures. The setup also distinguishes low-harmfulness BAD responses and compares against explicit-prompt baselines.

5 Experiment Setup

  • The evaluation asks how CodeJailbreaker performs on text-to-code, function-level completion, and block-level completion.These are the three research questions guiding the experiments.Source: §5 · p. 6

5.1 Datasets

  • RMCBench supplies malicious-code prompts across nine programming languages and 11 malicious categories, with 182 text-to-code prompts and 100 code-completion prompts.The completion set contains 36 function-level and 64 block-level cases.Source: §5.1 · pp. 6–7

5.2 Models

  • The study evaluates five general LLMs and two code LLMs, following RMCBench configurations.The general models include GPT-4, Claude-3.5-sonnet, Qwen-plus, Gemini-2, and DeepSeek-V3; the code models are CodeLlama-13b-Instruct-hf and DeepSeek-Coder-7B.Source: §5.2 · p. 7

5.3 Implementation

  • The two code LLMs run locally with specified sampling settings, while general LLMs are accessed through official APIs with default parameters.Local inference uses an A100-80G GPU; the listed settings include temperature 0.75 and max_new_tokens 4096.Source: §5.3 · p. 7

5.4.1 Attack success rate.

  • Attack Success Rate (ASR) is the percentage of BAD responses among all responses, where BAD means the model does not explicitly refuse malicious content.Responses are categorized as GOOD, BAD, or UNCLEAR using RMCBench criteria, with DeepSeek-V3 replacing GPT-4 for automated labeling.Source: §5.4.1 · p. 7

5.4.2 Malicious rate.

  • Malicious Ratio (MR) counts SERIOUS responses—BAD responses whose generated code matches the prompt’s malicious intent—as a percentage of all responses.Responses that do not match the intent are classified as SLIGHT; 16.48% of 91 sampled BAD responses were manually judged SLIGHT, and DeepSeek-V3’s overall accuracy was 94.51%.Source: §5.4.2 · p. 8

5.5 Baselines

  • The explicit malicious prompts from RMCBench serve as the EMP baseline, with EMP+T adding a jailbreak template for text-to-code comparisons.The template is combined with the malicious instruction for the text-to-code task.Source: §5.5 · pp. 8–9

6 Experiment Results

CodeJailbreaker improves on explicit-prompt baselines across the three code-generation tasks, with the size of the improvement varying by task. Its reported effects are comparatively limited on code LLMs in the text-to-code and function-level tasks, while block-level defenses are weak across the evaluated models.

6.1 RQ1: Performance on Text-to-Code Task

  • 79.12% average ASR and 64.76% average MR: CodeJailbreaker exceeds EMP by 60.26% and 51.95%, respectively, on text-to-code.It also exceeds EMP+T by 51.57% in ASR and 34.38% in MR.Source: §6.1 · p. 9
  • On text-to-code, CodeJailbreaker’s ASR exceeds 80% on all general LLMs, while its impact is more limited on code LLMs.The authors suggest specialized safety measures as a possible explanation; DeepSeek-Coder-7B has 68.13% ASR and 51.10% MR.Source: §6.1 · p. 9

6.2 RQ2: Performance on Function-Level Code Completion Task

  • On function-level completion, CodeJailbreaker achieves average ASR and MR of 92.06% and 82.93%, increases of 22.11% and 29.03% over RMCBench; EMP averages 75.39% and 64.28%.For Qwen-plus, CodeJailbreaker’s ASR rises from 55.56% to 100.00% and MR from 47.22% to 91.67%, compared with EMP.Source: §6.2 · p. 9

6.3 RQ3: Performance on Block-Level Code Completion Task

  • On the block-level completion task, CodeJailbreaker achieves average ASR of 98.21% and average MR of 79.24%, increases of 2.09% and 5.98%, respectively, compared with RMCBench.Source: §6.3 · p. 10

7 Discussion

Discussion shows attack effectiveness varies by malicious-code type and language, while dataset composition and automated evaluation constrain interpretation. It also draws implications for safety alignment and code-LLM defenses.

7.1 The Effectiveness of CodeJailbreaker on Different Malicious Types and Code Languages

  • Spyware yielded 94.35% ASR and 89.29% MR, while adware yielded 75% ASR and 41.07% MR.Source: §7.1 · p. 10
  • PHP was most susceptible, with 100% ASR and 97.14% MR; Python and HTML had relatively low MRs, while Java had one case and 0 MR.The authors plan to enlarge the Java dataset for more comprehensive evaluation.Source: §7.1 · p. 10

7.2 Threats to Validity

  • LLM-based evaluations of malicious code may be inaccurate, although the authors report manual review as mitigation.They note DeepSeek-V3 excels at identifying malicious code in that review.Source: §7.2 · p. 10
  • RMCBench is imbalanced: 232 of 282 cases concern Python, while Java and Bash each have one; the study also evaluates seven LLMs.The authors say sparse language coverage may leave those languages underexplored; other high-performing models were not included.Source: §7.2 · p. 10

7.3 Implications

  • The authors call for stronger safety alignment against implicit malicious intent and suggest protective measures for code LLMs given empty implementations after affirmative openings.They identify further investigation of such defensive mechanisms as a direction for future work.Source: §7.3 · pp. 10–11

8 Conclusion

The conclusion presents CodeJailbreaker as an implicit-prompt attack that outperforms explicit-intent strategies across multiple code-generation tasks on RMCBench.

  • CodeJailbreaker uses implicit malicious prompts and a covert channel to express malicious intent.Source: §8 · p. 11
  • On RMCBench, CodeJailbreaker achieved better attack results across multiple code-generation tasks than conventional methods explicitly stating malicious intent.Source: §8 · p. 11
  • The authors conclude that LLM-based code generation needs stronger safeguards against implicit malicious cues.Source: §8 · p. 11

Ethic Considerations

The paper warns that its content may be harmful or offensive and limits its stated purpose to LLM security evaluation and analysis.

Related papers

Loading 2503.17953v1…