How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States

Zhenhong Zhou; Haiyang Yu; Xinghua Zhang; Rongwu Xu; Fei Huang; Yongbin Li

Source-linked AI summary

How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States

Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Yongbin Li

arXiv:2406.05644v2Jun 9, 2024cs.CLcs.AIcs.CRcs.CY

TL;DR

LLM safety alignment and jailbreak mechanisms are difficult to explain because large models are effectively black boxes and jailbreaks can produce harmful outputs. The paper uses weak classifiers and intermediate hidden states to trace safety from ethical concepts through emotional guesses to response or rejection tokens. It finds that pre-training supplies early ethical distinctions, alignment links them to middle-layer emotions, and jailbreak disrupts that linkage; the study examines models from 7B to 70B across families.

Problem
Alignment can prevent harmful responses, but jailbreaks can circumvent these safeguards and the mechanisms of both behaviors remain difficult to elucidate in large, parameter-intensive LLMs.
Method
The paper uses weak classifiers and Logit Lens analysis of intermediate hidden states to examine ethical classification, emotion guesses, and output-token formation across LLMs.
Results
Weak classifiers achieve accuracy exceeding 95% for distinguishing malicious and normal inputs early, while alignment links these distinctions to middle-layer emotions and jailbreak disrupts that association.
Takeaways & Limitations
LLM safety can be analyzed as an early ethical distinction refined through middle-layer emotion and later response-token selection, suggesting a focus on the unethical-to-reject association.
Takeaways & Limitations
The study uses default simple weak classifiers and examines intermediate-state interpretation only for LLM safety, not other capabilities.

Abstract

from arXiv · show

Large language models (LLMs) rely on safety alignment to avoid responding to malicious user inputs. Unfortunately, jailbreak can circumvent safety guardrails, resulting in LLMs generating harmful content and raising concerns about LLM safety. Due to language models with intensive parameters often regarded as black boxes, the mechanisms of alignment and jailbreak are challenging to elucidate. In this paper, we employ weak classifiers to explain LLM safety through the intermediate hidden states. We first confirm that LLMs learn ethical concepts during pre-training rather than alignment and can identify malicious and normal inputs in the early layers. Alignment actually associates the early concepts with emotion guesses in the middle layers and then refines them to the specific reject tokens for safe generations. Jailbreak disturbs the transformation of early unethical classification into negative emotions. We conduct experiments on models from 7B to 70B across various model families to prove our conclusion. Overall, our paper indicates the intrinsical mechanism of LLM safety and how jailbreaks circumvent safety guardrails, offering a new perspective on LLM safety and reducing concerns. Our code is available at https://github.com/ydyjya/LLM-IHS-Explanation.

1 Introduction

The paper frames alignment and jailbreak as poorly understood safety mechanisms in large, parameter-intensive LLMs. Using weak classifiers and intermediate-state analysis, it argues that pre-training establishes ethical distinctions, alignment maps them through emotional guesses to response or rejection tokens, and jailbreak disrupts the middle-layer association.

Motivation: The paper motivates intermediate-state analysis because billions of parameters make alignment and jailbreak mechanisms difficult to understand, despite small stylistic-logit shifts making models harmless.Aligned models often begin malicious-input refusals with consistent rejection styles, motivating analysis of shared safety activation patterns.
Contributions: Weak classifiers distinguish malicious and normal inputs from early hidden states with accuracy exceeding 95%, regardless of whether the model is aligned or base.This supports the claim that ethical concepts are learned during pre-training rather than introduced solely by alignment.
Contributions: Logit Lens analysis finds that alignment associates compliant inputs with positive emotions and non-compliant inputs with negative emotions in middle layers before refining them into stylistic response or rejection tokens.The emotional tokens are consistent within safe inputs and within unsafe inputs across the evaluated datasets.
Contributions: Jailbreak preserves early ethical classification but makes middle-layer emotion ambiguous, disrupting the association between early ethical beliefs and later safety behavior.The authors use Logit Grafting to graft positive emotions from normal inputs onto jailbreak hidden states and report confirmation that jailbreak disrupts the early-to-middle association.
Implication: The proposed explanation suggests reinforcing the unethical-to-reject association in middle layers as a more precise optimization goal for LLM safety.The paper presents this as a transparency-oriented implication of its mechanism-level account.

2 Related Works

Related work positions this study at the intersection of LLM explainability and safety. Prior research develops attention and intermediate-state interpretability tools, while alignment and jailbreak studies document safety failures without fully addressing their underlying mechanism.

2.1 LLM Explainability: Logit Lens applies the final linear function to intermediate hidden states, helping explain how language models refine outputs during the forward pass.The related work also notes explainability studies of attention heads, neurons, and strong models explaining smaller models.
2.1 LLM Explainability: As model size increases, LLMs become increasingly difficult to explain, motivating methods that inspect intermediate computations.Prior work includes studies of context-sensitive attention heads and neuron-level explanations.
2.2 LLM Safety: Alignment is commonly used to make models reject harmful queries, but jailbreaks can defeat these assurances, prompting many post hoc defense methods.The cited safety literature includes both handcrafted and automated jailbreak algorithms and defenses proposed after jailbreak emergence.

3 Not Only Alignment: How LLMs Ensure Safety

LLMs distinguish malicious from normal inputs in early hidden states, while alignment links those ethical representations to emotional guesses and refines them into response formats. Across models, weak classifiers and intermediate-state decoding support a safety process rooted in pre-training and shaped by alignment.

3.1 LLMs Learn Ethical Concepts During Pre-training Rather Than Alignment: Early hidden states distinguish malicious and normal inputs with accuracy exceeding 95%, including in both aligned and base models.This indicates that ethical concepts are represented during pre-training rather than created solely by alignment.
3.1 LLMs Learn Ethical Concepts During Pre-training Rather Than Alignment: Middle-layer decoding produces coarse positive or negative emotional tokens around layers 16-24, which later evolve into response or refusal tokens.Early-layer decodings are generally meaningless, whereas middle-layer emotional guesses are refined in later layers.
3.2 Safety Alignment: Bridging Ethical with Positive and Unethical with Negative: Aligned models form more consistent middle-layer representations, whereas unaligned models fail to refine emotional tokens for normal and malicious inputs.Unaligned models instead associate different inputs with shared format tokens such as “answer” or “quelle,” with lower similarity than aligned models.
3.1 LLMs Learn Ethical Concepts During Pre-training Rather Than Alignment: Weak classifiers can also distinguish jailbreak, malicious, and normal inputs, indicating that jailbreak usually does not erase early ethical representations.The three-way classification uses SVM and MLP classifiers on chat-format hidden states.
3.2 Safety Alignment: Bridging Ethical with Positive and Unethical with Negative: Alignment bridges early ethical features to middle-layer emotional guesses and then refines those guesses into chat-format or refusal-response initial tokens.Higher consistency in associating negative emotions with malicious inputs is associated with greater harmlessness, while base models show no such middle-layer association.

4 How Jailbreak Causes LLMs Alignment to Fail

Jailbreaks leave early ethical classification intact but disrupt the middle-layer association that converts it into rejection behavior; Logit Grafting supports this mechanism by reproducing the disruption.

Jailbreak inputs remain distinguishable from malicious and normal inputs in early hidden states, indicating that they do not deceive pre-training ethical concepts.
4.1 Perturbations in Association Stage: Perturbed middle states can be refined into positive initial response tokens, producing safe failure unless positive emotions dominate the middle layers.Ambiguous emotional states or degraded intermediate representations can make the model unsafe, whereas weaker disturbances may be corrected by later-layer rejection refinement.
4.2 Approximate to Jailbreak with Logit Grafting: Logit Grafting replaces malicious-input middle-layer states with positive-emotion states from normal inputs, modifying only the last position to minimize semantic changes.The method uses layer-23 hidden states and grafts during first-token generation.
4.2 Approximate to Jailbreak with Logit Grafting: Logit Grafting increases malicious-objective completion rates, sometimes exceeding jailbreak responses, supporting the claim that disrupting association prevents later refinement into rejection logits.Table 3 reports attack success rates for the grafting approximation alongside jailbreak attacks.

5 Conclusions

The paper explains LLM safety as a collaboration between pre-training and alignment operating through intermediate hidden states. It argues that jailbreaks usually disrupt the association between early ethical representations and middle-layer emotional guesses rather than deceiving ethical beliefs directly.

Pre-training assigns different intermediate hidden states to malicious and normal inputs according to learned ethical concepts.
Alignment associates early ethical states with positive or negative emotional guesses before refining them into affirmative or refusal response tokens.
Jailbreaks involving additional input processing often disturb the association between early and middle layers instead of deceiving the model’s ethical beliefs.
The authors position intermediate-hidden-state analysis as a way to improve LLM transparency and support responsible, ethical development.

6 Limitations

The study uses simple default weak classifiers and evaluates intermediate-hidden-state interpretation only for LLM safety, leaving broader capability interpretation untested.

The experiments use default settings for the simplest weak classifiers, despite achieving satisfactory classification results.
The paper examines weak-classifier interpretation of strong models only from a safety perspective, while applicability to other LLM capabilities remains untested.

7 Ethics Statement

The paper frames its safety analysis as an internal study of alignment and jailbreak, while acknowledging limits on jailbreak enhancement, white-box applicability, and adverse-impact analysis.

7 Ethics Statement: The study analyzes alignment and jailbreak through internal hidden-state transformations rather than improving jailbreak effectiveness.Logit Grafting is described as a white-box approximation used to confirm the paper’s conclusions.
7 Ethics Statement: The authors do not open-source the jailbreak datasets, while releasing code and normal and malicious-input datasets for ethical reasons.
7 Ethics Statement: The evaluation prompt asks GPT-4 to judge whether an assistant response violates ethical, legal, or safety guidelines and fulfills the malicious goal.
7 Ethics Statement: The paper argues that keyword-based jailbreak or malicious-request detection cannot strictly represent model safety capabilities.
7 Ethics Statement: The evaluation treats only a score of 10 as successful completion of a malicious request and regards all other scores as refusal.

B Appendix B: Ablation Study For Weak to Strong Explanation

The ablation study tests whether weak classifiers overfit hidden-state data, finding limited performance on shuffled inputs but strong source discrimination in intermediate layers.

B Appendix B: Ablation Study For Weak to Strong Explanation: Hidden-state classifiers on shuffled normal inputs are near random without chat formatting and remain below 68% for Mistral and Llama3 with chat formatting.The authors speculate that Grouped Query Attention may contribute to the weaker performance in those model families.
B Appendix B: Ablation Study For Weak to Strong Explanation: The ablation uses SVM and MLP classifiers with both direct and chat-formatted inputs to test hidden-state overfitting.
B Appendix B: Ablation Study For Weak to Strong Explanation: Weak classifiers distinguish intermediate hidden states from different input sources with high accuracy, despite similar top-level input characteristics.Embedding-layer accuracy is usually near random, whereas later intermediate layers are highly discriminative.
B Appendix B: Ablation Study For Weak to Strong Explanation: The source datasets differ slightly in average input length, but the study cannot confirm whether length or other source characteristics drive the discrimination.GPT-4-generated inputs are slightly shorter than Claude3-Opus-generated inputs.

C Appendix C: Logit Grafting Approximately Leads Alignment Failure

Logit Grafting approximates jailbreak by disturbing the first-token forward pass, and the resulting association disruption substantially increases malicious-objective completion across models.

C Appendix C: Logit Grafting Approximately Leads Alignment Failure: The grafting procedure replaces forward-pass behavior for the first token because normal and malicious inputs have completely different semantics.
C Appendix C: Logit Grafting Approximately Leads Alignment Failure: The experiment compares grafting on normal inputs with jailbreak inputs and also applies grafting to jailbreak inputs to test whether association disturbance affects jailbreak behavior.
C Appendix C: Logit Grafting Approximately Leads Alignment Failure: Forced Logit Grafting significantly increases malicious-objective completion rates across all tested models, sometimes exceeding jailbreak responses.The experiment replaces the first-token forward pass at selected middle layers as an approximation of jailbreak.
C Appendix C: Logit Grafting Approximately Leads Alignment Failure: The appendix interprets prefix or suffix modifications as disturbances to the association produced by alignment tuning.

D Appendix D: Some Supplement Experiments Results

Supplementary experiments extend hidden-state classification and emotion-token visualizations across model families, showing consistent base-model and jailbreak-disturbance patterns with defense-strength differences.

D Appendix D: Some Supplement Experiments Results: The appendix notes that multilingual intermediate tokens complicate complete recognition of emotional connotations beyond identified English and Chinese tokens.
D Appendix D: Some Supplement Experiments Results: Supplementary weak-classifier figures cover Mistral, Vicuna, Llama, Falcon, and Meta-Llama models across multiple parameter sizes and layers.The figures include models from 7B to 13B and visualize layers ranging from 16–23, 16–24, or 16–31.
D Appendix D: Some Supplement Experiments Results: Base models show no alignment-tuning associations for normal or malicious inputs, instead producing answers, meaningless tokens, or list-like tokens.
D Appendix D: Some Supplement Experiments Results: Jailbreak disturbances vary by model defense strength: Vicuna shows successful disturbance, Mistral is intermediate, and Llama2 and Llama3 resist some disturbances.

E Appendix E: Visualizing SVM Classification Results with t-SNE

The appendix visualizes intermediate hidden states with t-SNE across layers and model variants. Aligned and instruct models generally separate normal from malicious inputs after several layers, whereas base models do not classify them well.

Visualization method: The appendix applies CUDA-based t-SNE to intermediate hidden states from normal and malicious inputs.The visualization uses perplexity=30, learning_rate=500, n_iter=3000, and random_state=42.
t-SNE visualization results: After about 8–9 layers, t-SNE usually forms a clear boundary between normal and malicious inputs in most models.The visualizations cover layers 0–15 and 16–31 for 7B models, and layers 0–19 and 20–39 for 13B models.
t-SNE visualization results: Base models are not well separated by t-SNE across the visualized layers.The appendix includes Llama-2 7B and 13B base-model visualizations alongside aligned or instruct variants.