Source-linked AI summary
The Struggle Between Continuation and Refusal: A Mechanistic Analysis of the Continuation-Triggered Jailbreak in LLMs
Yonghong Deng, Zhen Yang, Ping Jian, Xinyue Zhang, Zhongbin Guo, Chengzhi Li, Junxi Yin
TL;DR
The paper investigates why continuation-triggered jailbreaks can bypass LLM safety alignment, a mechanism that remains poorly understood. It analyzes attention heads and finds competition between continuation and safety drives, then proposes HCS and distillation as inference-time safety interventions.
Problem
The mechanisms behind jailbreak vulnerability remain poorly understood, despite alignment efforts and defenses that often focus on black-box mitigation.
Source: §1 · pp. 1–2Method
Results
The analysis links continuation-triggered jailbreak behavior to competition between intrinsic continuation drive and alignment-based safety capabilities, and the proposed pipeline offers safety improvement with efficient single-pass deployment.
Source: §5 · pp. 8–9Takeaways & Limitations
The findings support a mechanism-grounded pipeline from head-level analysis to practical safety intervention and efficient deployment.
Source: §5 · p. 9Experiments cover two representative 7B-scale models, and the mechanistic framework is developed and validated for continuation-triggered jailbreaks.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
With the rapid advancement of large language models (LLMs), the safety of LLMs has become a critical concern. Despite significant efforts in safety alignment, current LLMs remain vulnerable to jailbreaking attacks. However, the root causes of such vulnerabilities are still poorly understood, necessitating a rigorous investigation into jailbreak mechanisms across both academic and industrial communities. In this work, we focus on a continuation-triggered jailbreak phenomenon, whereby simply relocating a continuation-triggered instruction suffix can substantially increase jailbreak success rates. To uncover the intrinsic mechanisms of this phenomenon, we conduct a comprehensive mechanistic interpretability analysis at the level of attention heads. Through causal interventions and activation scaling, we show that this jailbreak behavior primarily arises from an inherent competition between the model's intrinsic continuation drive and the safety defenses acquired through alignment training. Furthermore, we perform a detailed behavioral analysis of the identified safety-critical attention heads, revealing notable differences in the behaviors of safety heads across different model architectures. Grounded in these mechanistic findings, we propose Head Competition Steering (HCS), a mechanistically grounded inference-time strategy that explicitly leverages the competition between safety heads and continuation heads to suppress harmful generation, and further distill its behavioral signal into a student model via knowledge distillation, achieving inference-time safety improvements without additional computational overhead.
1 Introduction
The introduction frames continuation-triggered jailbreaks as a way to probe why aligned models can still generate harmful content, and argues that their underlying mechanisms remain insufficiently understood. It presents attention-head analysis as a route to identifying the competition between continuation and safety, then uses that finding to motivate HCS and distillation.
1 Introduction
- The authors identify a gap in understanding jailbreak mechanisms: existing defenses focus mainly on black-box mitigation, while broader generative mechanisms remain insufficiently understood.The introduction notes that alignment methods may yield shallow alignment, leaving models vulnerable to minor jailbreak variations.Source: §1 · p. 2
- Both LLaMA-2-7B-Chat and Qwen2.5-7B-Instruct show substantial ASR increases across multiple datasets when the continuation-triggered suffix follows the user prompt.Under the clean-prompt setting, models typically refuse harmful requests even when the suffix follows the harmful instruction.Source: §1 · p. 2
- The proposed explanation is an internal tension between next-token continuation and alignment training that redirects the model toward refusal when safety is at risk.The authors use continuation-triggered jailbreaks as an analytical lens for investigating this conflict.Source: §1 · p. 2
- Path patching, head ablation, and activation scaling identify safety and continuation heads whose competition the authors link to jailbreak behavior.The analysis identifies a small subset of middle-to-late-layer heads as critical to the specific jailbreak task studied.Source: §1 · p. 2
- HCS steers generation at inference time by amplifying safety-head activations while counteracting continuation-head activations, without modifying model parameters.The approach is presented as an alternative to suppressing individual vulnerable heads.Source: §1 · p. 2
- The paper distills HCS’s safety-aligned behavior into a student model to remove its inference overhead while preserving safety improvements.The introduction characterizes this pipeline as mechanism-grounded safety improvement beyond localized vulnerability patching.Source: §1 · pp. 2–3
2 Related Work
Related work covers jailbreak attacks that exploit safety-alignment weaknesses and methods for tracing model behavior to internal mechanisms. The paper positions its analysis as a mechanistic study of continuation-triggered jailbreaks.
2 Related Work
- Jailbreak attacks exploit weaknesses in LLM safety alignment; examples include multistep, iterative, and attention- or generation-based attacks.The paper describes continuation-triggered attacks as relocating instructions to follow the user instruction marker.Source: §2 · p. 3
- Mechanistic interpretability traces high-level behavior to computational structures such as attention heads, neurons, and residual pathways.The cited discussion describes causal interventions including activation patching, path patching, and activation scaling.Source: §2 · p. 3
- The authors characterize their approach as “locate-then-improve”: interpretability findings guide model improvement rather than explanation alone..Source: §2 · p. 3
3 Methodology
The methodology localizes jailbreak-critical attention heads, classifies their roles through activation interventions, and uses those findings to steer generation and distill safer behavior. It also examines how safety heads relate to harmfulness judgments and refusal behavior.
3.1.1 Which Heads Matter?
- Path patching replaces one head’s clean-run activation with its jailbreak-run activation and measures the resulting divergence from the clean output distribution.A large positive patching effect indicates that the head restores output toward clean behavior and lies on a causal path enabling the jailbreak.Source: §3.1.1 · p. 3
3.1.2 What Roles Do Key Heads Perform?
- Zeroing a head that raises ASR identifies a Safety Head, while zeroing one that lowers ASR identifies a Continuation Head.Heads with negligible ASR changes are neutral or weakly involved in jailbreak dynamics.Source: §3.1.2 · pp. 3–4
3.1.3 Faithfulness Validation
- Activation scaling tests head-function faithfulness by modulating selected head activations at inference without changing model parameters.A coefficient above 1 amplifies a head’s contribution, 0 suppresses it, and 1 preserves baseline behavior.Source: §3.1.3 · p. 4
3.2 Safety Heads Behavior Study
- The safety-head behavior study separates harmfulness recognition from refusal execution using a reply-inversion task, then scales safety heads to test for changes in harmfulness judgments.The inversion query expects acceptance for harmful inputs and refusal for harmless ones, separating assessment from innate refusal tendencies.Source: §3.2 · p. 4
3.3 Improving via Head Competition Steering (HCS) and Distillation
- HCS combines base logits with amplified safety-head logits and subtracts amplified continuation-head logits to steer decoding away from harmful generation without modifying model parameters or inputs.It uses three parallel forward passes per decoding step; the contrastive coefficient controls intervention strength.Source: §3.3 · pp. 4–5
- To avoid HCS’s three-pass inference cost, the method fine-tunes a LoRA student on teacher responses and distills HCS behavior for single-pass inference.The student’s safety-enhancement effect is internalized in its LoRA parameters.Source: §3.3 · pp. 5–6
4 Experiments
The experiments examine how attention heads, their scaling, and the proposed HCS and distillation methods affect jailbreak resistance and model behavior. Results show opposing safety- and continuation-head effects, architecture-specific safety-head behavior, and strong safety with largely preserved utility under HCS and KDHCS.
4.1 Experimental Setup
- Evaluation uses AdvBench for distillation training and JailbreakBench and MaliciousInstruct as test sets, assessing harmlessness with Alpaca and capability with MMLU and IFEval.Experiments use LLaMA-2-7B-Chat and Qwen2.5-7B-Instruct; decoding is deterministic with do_sample=False and max_new_tokens set to 32.Source: §4.1 · p. 6
4.2 Key Attention Heads Analysis
- Ablating safety head (27,7) increases ASR across datasets, while ablating continuation head (13,17) decreases it across three datasets.Path patching shows most heads have minor KL-divergence effects, but a small subset, especially in layers 15–17 and 25–27, has pronounced effects.Source: §4.2 · pp. 6–7
4.3 Faithfulness Evaluation
- Scaling safety heads decreases ASR across datasets, while scaling continuation heads increases it, with safety-head ASR reductions saturating when w > 4.Scaling more heads generally strengthens these opposing trends, though occasional fluctuations suggest nonlinear interactions; the authors attribute jailbreak ASR increases to the antagonistic interaction between the head types.Source: §4.3 · p. 7, Table 1 · p. 7
4.4 Safety Heads Behavior Analysis
- In LLaMA-2-7B-Chat, removing safety-head contributions drops HDR to 0.515 on AdvBench, whereas increasing their strength improves detection; in Qwen2.5-7B-Instruct, stronger safety heads decrease HDR on AdvBench while Alpaca HDR remains 0.The results suggest LLaMA-2 safety heads mainly support harmful-instruction recognition, while Qwen2.5 safety heads primarily govern refusal; over-scaling may cause over-cautiousness or misclassification.Source: §4.4 · pp. 7–8
4.5 HCS and Distillation
- KDHCS reaches ASR of 0.09 and 0.12 on JailbreakBench and MaliciousInstruct for LLaMA-2-7B-Chat, and 0.08 and 0.05 on those datasets for Qwen2.5-7B-Instruct, while largely preserving capability.HCS and KDHCS improve overall scores and outperform standard SFT; SFT reduces ASR but degrades MMLU and IFEval, while HCSrandom has only marginal ASR reductions and nearly unchanged overall scores.Source: Table 3 · p. 8, §4.5 · p. 8
5 Conclusion
The paper analyzes continuation-triggered jailbreaks using mechanistic interpretability and attributes the behavior to conflict between continuation drive and alignment-based safety. It proposes HCS and a distillation pipeline as interventions grounded in this analysis.
5 Conclusion
- Mechanistic analysis using path patching, activation scaling, and ablation studies attributes continuation-triggered jailbreak behavior to conflict between intrinsic continuation drive and alignment-based safety.The paper also finds that safety heads have distinct primary behaviors across models.Source: §5 · p. 8
- HCS leverages the antagonism between safety and continuation heads, while a LoRA-based student distills safety-aligned behavior for efficient single-pass deployment.The authors describe this as a pipeline from mechanistic analysis to practical safety intervention.Source: §5 · p. 9
Limitations
The study’s mechanistic findings are bounded by its model and attack scope.
Limitations
- The analysis covers two representative 7B-scale models and continuation-triggered jailbreaks, leaving larger models and other attack types for future investigation.The authors identify adversarial suffix, multi-turn, and role-play attacks as possible extensions.Source: Limitations · p. 9
Ethical Statement
The paper frames its purpose as analyzing safety-alignment failure modes, not enabling harmful generation. Its limited harmful-output case studies are presented for analysis and illustration.
Ethical Statement
- The authors state that harmful-output examples follow specific internal interventions in standard benchmark settings and are not intended for real-world use.They discourage using the findings or examples to bypass or weaken deployed safety safeguards.Source: Ethical Statement · p. 9
A.1 Datasets Details
The evaluation uses AdvBench, JailbreakBench, and MaliciousInstruct to assess jailbreak resistance, alongside harmless instructions for the reply-inversion task. The harmful-instruction sets also supply training and held-out test data for knowledge distillation.
A.1 Datasets Details
- AdvBench contains 520 harmful instructions, while JailbreakBench and MaliciousInstruct each contain 100; these datasets cover harmful and policy-violating instructions.Source: §A.1 · p. 12
- Section 3.2 uses AdvBench for harmful instructions and 1,000 Alpaca samples for harmless instructions in the reply-inversion task.Source: §A.1 · p. 12
- For distillation, AdvBench supplies HCS teacher prompt–response training pairs, while JailbreakBench and MaliciousInstruct are held out for testing the student.Source: §A.1 · p. 12
A.2 Metrics Details
The appendix specifies how safety, utility, and overall performance are evaluated, along with HCS and student-training settings. It also provides the prompt templates used for model evaluation and reply inversion.
A.2 Metrics Details
- Attack Success Rate (ASR) measures harmful or policy-violating generations, while Harmfulness Detection Rate (HDR) measures harmful-intent recognition; both use GPT-4 as a judge.Source: §A.2 · p. 12
- The Overall score inverts JailbreakBench and MaliciousInstruct ASR terms so higher scores jointly reflect jailbreak resistance and general capability.Source: §A.2 · p. 13
A.3 Hyperparameters
- HCS uses w = 3, with best α values of 3 for LLaMA-2-7B-Chat and 2 for Qwen2.5-7B-Instruct; student and SFT use matched LoRA settings.Source: §A.3 · p. 13
B Prompt Template
- The reply-inversion prompt asks whether an instruction is harmful and requires exactly one word, “Yes” or “No.”Source: B Prompt Template · p. 13
C More Results
In Qwen2.5-7B-Instruct, safety and continuation heads have opposing effects on jailbreak behavior. Activation scaling across three datasets supports their antagonistic roles.
C More Results
- For Qwen2.5-7B-Instruct, progressively increasing the scaling coefficient for safety heads consistently reduces ASR across all datasets, whereas scaling continuation heads produces the opposite effect.Source: C More Results · p. 13
D Ablation and Case Study
Ablations support opposing roles for safety and continuation heads, while case studies show continuation-head scaling can shift refusals toward harmful responses. The appendix also reports the computational setting for the experiments.
D Ablation and Case Study
- Zeroing safety heads increases ASR, whereas zeroing continuation heads reduces it; randomly selected heads have negligible effects.The random-head values average three independently sampled groups; Table 6 reports ablations on LLaMA-2-7B-Chat.Source: D Ablation and Case Study · pp. 13–14, Table 6 · p. 14
- Scaling continuation heads shifts both LLaMA-2-7B-Chat and Qwen2.5-7B-Instruct from refusing harmful instructions to generating harmful content.Source: D Ablation and Case Study · p. 14
E Computational Budget
- The reported path patching, activation scaling, HCS, distillation, and SFT experiments used four NVIDIA A6000 48GB GPUs.Source: E Computational Budget · p. 16