Source-linked AI summary
Improving Alignment and Robustness with Circuit Breakers
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, Dan Hendrycks
TL;DR
AI systems can produce harmful outputs and remain vulnerable to adversarial attacks despite refusal training and other defenses. The paper introduces circuit breakers, which directly reroute harmful-output representations, and reports robust protection across text, multimodal, and agent settings while preserving utility. It concludes that circuit breakers make models intrinsically safer against unseen attacks, within the specific scope of harmful-content attacks.
Problem
Existing alignment defenses can be bypassed by adversarial attacks, while robust safety without substantially reducing model performance remains difficult.
Method
Circuit breakers use representation engineering to connect harmful-output representations to an interrupting mechanism that halts harmful generation.
Results
Circuit breakers improve harmlessness against unseen adversarial attacks across LLM, multimodal, and agent settings while imposing almost no penalty on standard capability.
Takeaways & Limitations
Circuit breakers provide a potential path toward models that remain capable while resisting harmful outputs and harmful agent actions under attack.
Takeaways & Limitations
The approach targets attacks intended to make models produce harmful content and does not defend against traditional adversarial attacks aimed at changing an image classifier’s label.
Abstract
from arXiv · showhide
AI systems can take harmful actions and are highly vulnerable to adversarial attacks. We present an approach, inspired by recent advances in representation engineering, that interrupts the models as they respond with harmful outputs with "circuit breakers." Existing techniques aimed at improving alignment, such as refusal training, are often bypassed. Techniques such as adversarial training try to plug these holes by countering specific attacks. As an alternative to refusal training and adversarial training, circuit-breaking directly controls the representations that are responsible for harmful outputs in the first place. Our technique can be applied to both text-only and multimodal language models to prevent the generation of harmful outputs without sacrificing utility -- even in the presence of powerful unseen attacks. Notably, while adversarial robustness in standalone image recognition remains an open challenge, circuit breakers allow the larger multimodal system to reliably withstand image "hijacks" that aim to produce harmful content. Finally, we extend our approach to AI agents, demonstrating considerable reductions in the rate of harmful actions when they are under attack. Our approach represents a significant step forward in the development of reliable safeguards to harmful behavior and adversarial attacks.
1 Introduction
AI systems remain vulnerable to adversarial attacks, while existing defenses struggle to provide robust safety without sacrificing capability. Circuit breaking instead targets harmful-output representations directly, improving robustness while preserving utility.
- Motivation: Adversarial attacks can compromise AI outputs, and existing mitigations often trade adversarial robustness for model performance.Refusal training and attack-specific defenses can remain vulnerable to sophisticated or unseen attacks.
- Approach: Circuit breaking interrupts harmful generation by connecting harmful internal representations to circuit breakers rather than relying only on output-level refusal states.The approach aims to prevent harmful outputs directly instead of removing vulnerabilities to particular attacks.
- Results: Representation Rerouting lowers attack success across unseen attacks while largely preserving performance on standard language-model benchmarks.RR reroutes harmful-output representations into an orthogonal space and interrupts harmful completions under adversarial pressure.
- Implications: Circuit breakers can be used normally without additional computational burden and integrated with existing monitoring and protection mechanisms.The paper presents the method as improving the capability-versus-harmlessness tradeoff.
- Results: Cygnet reduces harmful output by approximately two orders of magnitude under unforeseen adversarial attacks while surpassing the original model’s capabilities.Cygnet combines circuit breaking with additional model-control methods on Llama-3-8B-Instruct.
2 Related Work
Prior work includes attack-generation methods, representation-level approaches, and defenses such as refusal training, adversarial training, and filtering. The paper builds on representation engineering to develop Representation Rerouting with LoRRA and circuit-breaking losses.
- Adversarial Attacks: LLM attacks include manually written prompts, automated red teaming, adversarial suffixes, and prefilling attacks that can induce harmful outputs.HarmBench provides a comprehensive summary of automated attacks.
- Circuit-Breaking Method: Representation Rerouting uses Low-Rank Representation Adaptation with circuit-breaker and retain datasets plus a scheduled loss objective.The method is characterized by its datasets and loss functions.
- Adversarial Attacks: Multimodal attacks include typographic and gradient-based vision-text attacks, while the safety and robustness of LLM agents remain comparatively unexplored.The cited related work distinguishes multimodal attack research from the still-limited agent-safety literature.
- Defenses: Existing defenses such as RLHF, DPO, refusal prompts, and adversarial fine-tuning often fall short against state-of-the-art attacks or reduce capabilities.System-level filters are described as cumbersome, resource-intensive, and vulnerable to attack.
- Representation Engineering: Representation engineering studies internal representations, embedded knowledge, and output steering, including control vectors and representation-level losses.Prior work also explores hazardous-knowledge unlearning and harmful-circuit elimination.
3 Circuit Breaking with Representation Engineering
Circuit breaking redirects representations associated with harmful processes toward unhelpful or refusal representations while preserving benign behavior. Representation Rerouting (RR) implements this strategy through datasets and losses, and is designed to generalize across unseen attacks.
- Circuit-breaking concept: Circuit-breaking redirects representations associated with harmful processes toward incoherent or refusal representations, interrupting harmful processes before output generation.The approach monitors or remaps representations related to harmful processes rather than targeting individual attacks.
- Scope and integration: Circuit breakers are applicable to general, private, or copyrighted harmful outputs and can be integrated into virtually any neural-network architecture.The resulting models can be used normally without additional computational burden and integrated with existing monitoring and protection mechanisms.
- Representation Rerouting: Representation Rerouting (RR) is a circuit-breaking technique using Low-Rank Representation Adaptation, organized around datasets and loss functions.The method’s datasets and chosen loss function jointly control harmful processes while preserving desirable behavior.
- Data: RR partitions training data into a Circuit Breaker Set that elicits harmful representations and a Retain Set that preserves benign representations and capabilities.The Retain Set includes examples that should not activate circuit breakers, while the Circuit Breaker Set prompts the mechanism.
- Loss: The rerouting loss remaps harmful representations to target representations, while the retain loss preserves representations from the Retain Set.The retain loss is often measured using the ℓ2 distance between current and retain representations.
- Loss: RR can optimize circuit-broken representations toward orthogonality with the original harmful-process representations using a ReLU-applied cosine-similarity objective.The paper describes this loss as the most intuitive and effective balance between robustness and capability among the explored forms.
4 Experiments
Experiments evaluate circuit breakers across language, multimodal, and agent settings, including diverse attacks, capability retention, ablations, and representation changes. Representation Rerouting generally improves robustness while preserving utility, with performance depending on training-distribution coverage and circuit-breaker design.
- 4.1 Large Language Models: RR reduces harmful-request compliance by 87% with Mistral and 90% with Llama-3, while proposed capability tests show less than a 1% performance dip.Adversarially trained Mistral instead loses over 8% on MT Bench.
- 4.2 Multimodal Models: Under white-box PGD attacks, RR reduces multimodal compliance by 84% versus the original model and 85% versus a safety prompt, while MMMU and LLaVA-Wild stay within 0.5% of original performance.The safety prompt decreases LLaVA-Wild performance by 3.3%.
- 4.3 AI Agents: RR reduces harmful action compliance by 84% and 83% in the reported agent settings while retaining Berkeley Function Calling Leaderboard performance.The evaluation covers harmful function-calling requests under no-attack and forced function-call conditions.
- 4.4 Ablation and Analysis: RR’s cosine loss is more stable than tested alternatives, while removing refusal-retain data increases robustness but degrades capabilities.RMU fails to converge, and centered random-vector training also fails; augmenting the circuit-breaker set lowers ASR while maintaining capabilities.
- 4.4 Ablation and Analysis: Training on broader harm categories generalizes better across categories than training on narrower categories, while in-domain generalization is indicated by low diagonal attack-success rates.The category-specific analysis averages robustness over the same six attacks.
- 4.4 Ablation and Analysis: For a prefilling harmful response, representation cosines and norms change dramatically from layer 10, before generation starts, consistent with circuit-breaker activation.The analysis uses layers 10 and 20 for circuit-breaking and confirms little change before layer 10.
5 Limitations and Conclusion
Circuit breakers target attacks intended to make models produce harmful content, rather than adversarial attacks with other objectives. Within this scope, the approach improves robustness to unseen attacks and extends to image hijacks and harmful agent actions.
- 5 Limitations: Circuit breakers specifically target attacks against a model’s ability to produce generically harmful content.They do not address adversarial attacks whose goal is simply to change an image-classification label when no output is inherently harmful.
- 5 Limitations: The method improves robustness for single-turn conversations against the harmful-content attacks it targets.The paper characterizes this improvement as dramatic within the stated use case.
- Conclusion: Circuit breakers based on RepE make models intrinsically safer and robust to unseen adversarial attacks.The conclusion also reports applicability to image hijacks and prevention of harmful actions by AI agents.
- Conclusion: The approach extends beyond text-only models to image hijacks and AI agents taking harmful actions.These extensions are presented as part of the method’s generality.
A.1 Large Language Model Circuit Breaker Dataset
The circuit breaker datasets are constructed from generated harmful behaviors, with separate procedures for text-only, multimodal, and agent settings. Filtering is used to reduce overlap with existing benchmark behaviors.
- Text Dataset: The text dataset uses an uncensored LLM to generate short harmful queries and completions across diverse categories.Samples with BLEU scores above 0.3 against HarmBench’s standard behaviors are removed to avoid benchmark contamination.
- Multimodal Dataset: The multimodal dataset pairs images with harmful queries and completions generated from image descriptions.Descriptions are produced with LLaVA-Mistral-7B from COCO images before prompting an uncensored LLM.
- Agent Dataset: The agent dataset starts from Glaive function definitions and generates harmful requests with corresponding function outputs.GPT-3.5-turbo executes the requests, after which outputs are converted to OpenFunctions format and filtered for overlap.
B Refusal Evaluation
The refusal evaluation uses non-toxic WildChat requests to measure unwanted refusals. Circuit-breaker models refuse more often than standard models, but less often than stronger refusal-trained systems such as Claude-3.
- Evaluation Dataset: WildChat provides real-world user–ChatGPT interactions spanning ambiguous requests, code-switching, topic-switching, and political discussions.The dataset is used to evaluate chat models’ tendencies when handling problematic requests.
- Evaluation Procedure: 500 English non-toxic user–GPT-4 requests are filtered for the refusal evaluation.Refusal is measured with keyword checking, supplemented by perplexity for circuit-breaker models.
- Results: Circuit-breaker models show increased refusal rates, but remain considerably less refusal-prone than Claude-3 and adversarially trained models.The comparison is reported in Table 3.
- Design Considerations: The training design balances circuit breaking with capability retention by scheduling loss multipliers over training.The circuit-breaking multiplier starts large and decreases while the retention multiplier increases.
- Design Considerations: Circuit-breaking loss covers user and assistant text for language models and agents, and all post-image-embedding tokens for multimodal models.The token scope is selected to improve robustness across model types.
- Design Considerations: LoRA tuning is used instead of directly modifying model weights to improve stability and retention.This is presented as an implementation choice for circuit-breaker training.
C.2.2 Detailed Evaluation Setup
The detailed evaluation covers multilingual, prefilling, embedding, representation-space, multimodal, and function-calling attacks. It also specifies datasets, baselines, model configurations, and evaluation procedures for these settings.
- Attack Settings: Multilingual attacks translate behaviors into six languages and translate model outputs back into English for evaluation.The languages include Simplified Mandarin, Italian, Ukrainian, Thai, Zulu, and Guarani.
- Attack Settings: Prefilling attacks prepend generic affirmative completions, with a colon and two newline characters strengthening the attack.Specific harmful completions can be more effective, but generic completions also have a powerful effect.
- Attack Settings: Input embedding attacks optimize a continuous embedding matrix rather than searching over discrete token sequences.The optimized embeddings are concatenated with the prompt embeddings before entering the model.
- Attack Settings: The input embedding matrix is initialized from twenty repeated x tokens and optimized with SGD for 500 steps with early stopping.Early stopping is used because continued optimization can degrade generation coherence.
- Attack Settings: RepE derives per-layer harmful directions from harmful–harmless activation differences and applies them to transformer-layer outputs.Directions are obtained with PCA on cached last-position activations.
- Attack Settings: RepE uses layers -11 through -20 with control coefficients 0.65 for Mistral-7B and 1.0 for Llama-3.More-negative layer indices are closer to the model input.
- Multimodal Evaluation: The multimodal robustness set contains 133 harmful text–image pairs drawn from HarmBench and MM-SafetyBench.The evaluation includes category breakdowns based on MM-SafetyBench scenarios.
D Open LLM Results
The evaluation reports Open LLM benchmark results alongside multimodal and agent results. Circuit breakers improve robustness in multimodal and agent settings while preserving specified capability benchmarks.
- Open LLM evaluation scores are reported for each individual benchmark.
- Multimodal results: Under PGD attacks, circuit breakers are significantly more robust than the original model, including one using a safety prompt.The multimodal capability benchmarks LLaVA-Wild and MMMU remain preserved.
- Agent results: Under Forced Function Calling, the circuit-breaker model remains robust while retaining agent capabilities.
F Multilingual Results
The multilingual results include language-specific attack-success rates and related ablations. The reported trend for LRL versus HRL attacks differs from prior observations, and this discrepancy is left for future work.
- Table 6 reports attack success rates by language.
- The study does not reproduce prior findings that LRL attacks outperform HRL attacks.The authors leave investigation of this difference to future work.
- Ablations: The training-set ablation adds bypass-refusal data to the circuit-breaker set or refusal-reinforcing data to the retain set for more balanced results.
G Additional Ablation Results
Additional ablations examine circuit-breaking losses and training choices. For Mistral, random-vector losses reduce attack success rates but also reduce capabilities, while the circuit-breaker set remains relatively robust across harm-category shifts.
- Loss ablations: Average attack success rate in the loss ablation is reported across six attacks.The attacks are DirectRequest, HumanJailbreaks, TAP-T, GCG-T, Prefill, and RepE.
- Loss ablations: Random-vector loss terms decrease attack success rates but also decrease capabilities for Mistral.
- Generalization: The circuit-breaker set is relatively robust to distribution shifts across categories of harm.
H Extended analysis of representations
Extended representation analyses show that circuit-breaking can begin during prefilling or occur after generation starts, depending on the model and prompt. The analyses identify these changes through shifts in cosine values and layer norms.
- Prefilling: For the Mistral meth prompt, circuit-breaking begins during prefilling, before generation starts.The same behavior is observed for Llama-3-8B and Mistral-7B on the bomb prompt.
- Generation: With the obfuscated m-e-t-h prompt, circuit-breaking occurs during generation after harmful content begins to form.The model switches to meaningless outputs before harmful knowledge is revealed.
- Representation signals: In the bomb-prompt analyses, cosine values and norms change during prefilling, while layer-20 norms clearly indicate circuit-breaking.Layer-10 norms do not grow significantly in this example.