Source-linked AI summary
Circuit Discovery Helps Detect LLM Jailbreaking: A Mechanistic Interpretability Study
Paria Mehrbod, Boris Knyazev, Guy Wolf, Eugene Belilovsky, Geraldin Nanfack
TL;DR
Safety-aligned LLMs remain vulnerable to jailbreak prompts, while the internal processing behind this vulnerability is poorly understood. The paper discovers jailbreak circuits in LLaMA-2-7B-chat-hf using edge attribution patching and subnetwork probing, finding that first-token ablation can reduce attack success rates by up to 80%.
Problem
The internal mechanisms by which safety-aligned LLMs process adversarial jailbreak prompts remain poorly understood.
Method
The study uses edge attribution patching and subnetwork probing to discover computational circuits responsible for affirmative jailbreak responses.
Results
Targeted ablation of discovered circuits during first-token prediction reduces jailbreak attack success rates by up to 80%.
Takeaways & Limitations
The findings identify sparse, interpretable circuits associated with jailbreak behavior and support targeted defenses based on mechanistic analysis.
Takeaways & Limitations
The study does not evaluate whether the detected circuits preserve performance on nonjailbreak tasks, and full-sequence ablation can produce nonsensical outputs.
Abstract
from arXiv · showhide
Despite extensive safety alignment, large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safeguards to elicit harmful content. While prior work attributes this vulnerability to safety training limitations, the internal mechanisms by which LLMs process adversarial prompts remain poorly understood. We present a mechanistic analysis of the jailbreaking behavior in a large-scale, safety-aligned LLM, focusing on LLaMA-2-7B-chat-hf. Leveraging edge attribution patching and subnetwork probing, we systematically identify computational circuits responsible for generating affirmative responses to jailbreak prompts. Ablating these circuits during the first token prediction can reduce attack success rates by up to 80\%, demonstrating its critical role in safety bypass. Our analysis uncovers key attention heads and MLP pathways that mediate adversarial prompt exploitation, revealing how important tokens propagate through these components to override safety constraints. These findings advance the understanding of adversarial vulnerabilities in aligned LLMs and pave the way for targeted, interpretable defense mechanisms based on mechanistic interpretability.
1. Introduction
Jailbreak attacks can bypass safety alignment, but the internal processing that enables this behavior remains poorly understood. This paper discovers circuits in LLaMA-2-7B-chat-hf whose targeted ablation improves resistance to jailbreaks.
- Jailbreak attacks manipulate prompts to bypass built-in safety mechanisms and elicit harmful responses from aligned LLMs.
- Mechanistic interpretability has identified task-specific circuits, but prior efforts have focused predominantly on small models and synthetic benchmarks.
- The study systematically discovers and characterizes circuits enabling affirmative responses to jailbreak prompts in LLaMA-2-7B-chat-hf.It applies edge attribution patching and subnetwork probing.
- A compact subnetwork faithfully replicates the model’s jailbreak behavior.
- Targeted ablation identifies attention heads and MLP pathways involved in bypassing safety measures and reduces attack success rates.The analysis examines the circuit’s structure and information flow.
2. Related Work
Circuit discovery seeks sparse computational subnetworks responsible for specific capabilities or behaviors. Related work spans manual and automated discovery methods, alongside analyses of why jailbreak prompts bypass safety measures.
- Circuit Discovery: Circuit discovery identifies sparse subnetworks responsible for implementing specific neural-network capabilities or behaviors.A circuit is formally defined as a subgraph capturing essential computation for a particular task.
- Manual Circuit Discovery: Manual studies used activation or path patching to identify circuits for indirect-object identification and mathematical reasoning in GPT-2.
- Automatic Circuit Discovery: ACDC automates activation patching to identify important computational-graph edges but remains expensive for larger models.Subnetwork probing learns sparse binary masks, while attribution patching estimates component importance with gradients.
- Jailbreak Mechanisms: Prior jailbreak analyses describe competing objectives and mismatched generalization as two weaknesses enabling safety bypass.Examples include instruction-following prefixes and safety training that does not cover Base64-encoded inputs.
3. Methods
The method builds jailbreak data for Llama-2-7b-chat-hf and searches for sparse subnetworks that reproduce jailbreak behavior. It combines edge attribution patching with directly optimized subnetwork masks.
- Setup: The study seeks a sparse circuit that faithfully represents jailbreak behavior and uses Llama-2-7b-chat-hf as its safety-finetuned model.The model was pretrained, supervised-finetuned for dialogue, and aligned with human preferences using RLHF.
- Jailbreak Task: Jailbreaking is defined as a model responding positively to a harmful prompt despite safety finetuning with RLHF.
- Dataset Construction: The jailbreak dataset combines Llama-2 templates, system prompts, harmful queries, and optimized suffixes generated with improved GCG.HarmBench supplies hacking-and-stealing and traditional harmful prompt categories.
- Edge Attribution Patching: Edge attribution patching estimates each edge’s effect on loss, uses zero ablation and cross-entropy, and selects edges with the highest absolute attribution scores.The inputs are tokenized jailbreak prompts and answers.
- Subnetwork Probing: Subnetwork probing directly optimizes reparameterized edge-wise masks using a loss with cross-entropy and an l0 sparsity regularizer.
4. Experiments and Results
Experiments evaluate sparse jailbreak circuits using LLaMA-2-7B-chat datasets, circuit-discovery methods, faithfulness analyses, and first-token ablations. The results show that subnetwork probing identifies faithful sparse circuits, whose removal shifts generation toward refusal and improves jailbreak detection.
- Experimental setup: The experiments use 100 training samples for circuit discovery and an additional 50-sample test set for faithfulness evaluation.
- Experimental setup: EAP uses cross-entropy loss, while subnetwork probing uses KL divergence to optimize circuit masks from tokenized prompts and answers.
- Circuit faithfulness: For circuits smaller than 10% of all edges, EAP has low faithfulness, whereas subnetwork probing achieves low loss and high faithfulness at similar sizes.
- Circuit interpretation: The discovered circuit attends simultaneously to system prompts, user instructions, and jailbreak suffixes, supporting a competing-objectives account of safety bypass.
- Jailbreak detection: A 5% circuit ablation during first-token generation makes the model avoid “Sure” and produce refusal responses containing “I cannot”.
- Jailbreak detection: The proposed generation procedure performs better than adding a random first token, particularly when circuits are learned with three-token targets.
5. Conclusion
The paper identifies sparse computational circuits associated with affirmative jailbreak responses and examines their use in safety-aligned LLMs. Ablating the circuits reduces attack success, while the study notes important limits concerning nonjailbreak performance and full-sequence generation.
- The study presents a mechanistic analysis of jailbreak behavior by identifying computational circuits responsible for affirmative responses to adversarial prompts.
- Subnetwork probing successfully identifies sparse circuits that faithfully reproduce jailbreaking behavior.
- Ablating the circuits during first-token prediction reduces attack success rates by up to 80%.
- The study does not evaluate whether detected circuits preserve performance on nonjailbreak tasks, termed circuit completeness.
- Full-sequence generation with only the ablated model produces nonsensical outputs, while responses often collapse into repetitive initial tokens.
A. Visualizing Circuits
The visualizations show compact subnetworks discovered with Subnetwork Probing for jailbreak behavior in the HJ and HSJ datasets. Important components include attention heads and MLPs, with node importance and token activations indicating which parts of prompts influence them.
- Graph nodes represent attention heads and MLP components across model layers, while edge opacity indicates connection importance.
- Node importance is computed by summing absolute connected-edge scores, and the top two activating tokens are displayed for important nodes.
- The HSJ visualization also shows a 0.05% Subnetwork Probing circuit produced with zero ablation.
- Important nodes in the HSJ circuit attend to tokens from all parts of the prompt.
- The HJ visualization depicts a 0.05% subnetwork identified with Subnetwork Probing and zero ablation.
B. Samples of Generated Answers
This section presents sample outputs from the ablated model alongside the full prompts used to elicit them. The examples examine model behavior after removing a circuit with 5% sparsity.
- Sample outputs are presented with the full prompts used to elicit them after removing a circuit with 5% sparsity.