PaperlayerView exact revision on arXiv

Source-linked AI summary

Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, Rob Gilson, Logan Graham, Logan Howard, Nimit Kalra, Taesung Lee, Kevin Lin, Peter Lofgren, Francesco Mosconi, Clare O'Hara, Catherine Olsson, Linda Petrini, Samir Rajani, Nikhil Saxena, Alex Silverstein, Tanya Singh, Theodore Sumers, Leonard Tang, Kevin K. Troy, Constantin Weisser, Ruiqi Zhong, Giulio Zhou, Jan Leike, Jared Kaplan, Ethan Perez

arXiv:2501.18837v1cs.CLcs.AIcs.CRcs.LG

TL;DR

LLMs remain vulnerable to universal jailbreaks that can expose detailed harmful CBRN information. The paper introduces Constitutional Classifiers, which use natural-language constitutions to generate synthetic training data for input and output safeguards. Red teaming and automated evaluations show strong jailbreak resistance with limited production refusal and inference costs, while later discussion emphasizes the need for complementary defenses.

  • Problem

    Universal jailbreaks can reliably bypass LLM safeguards across most domain queries, potentially giving non-experts access to dangerous CBRN information.

  • Method

    Constitutional Classifiers train input and output safeguards on synthetic data generated from natural-language rules specifying permitted and restricted content.

  • Results

    The classifier-guarded system refused over 95% of held-out jailbreak attempts versus 14% without classifiers, with a 0.38% absolute production-refusal increase and 23.7% inference overhead.

  • Takeaways & Limitations

    The results support the tractability of defending against universal jailbreaks while maintaining practical deployment viability and flexibility through constitution updates.

  • Takeaways & Limitations

    The authors expect vulnerabilities to emerge with continued testing and state that responsible deployment will require complementary defenses such as rapid response and monitoring.

Abstract

from arXiv · show

Large language models (LLMs) are vulnerable to universal jailbreaks-prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Constitutional Classifiers: safeguards trained on synthetic data, generated by prompting LLMs with natural language rules (i.e., a constitution) specifying permitted and restricted content. In over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak that could extract information from an early classifier-guarded LLM at a similar level of detail to an unguarded model across most target queries. On automated evaluations, enhanced classifiers demonstrated robust defense against held-out domain-specific jailbreaks. These classifiers also maintain deployment viability, with an absolute 0.38% increase in production-traffic refusals and a 23.7% inference overhead. Our work demonstrates that defending against universal jailbreaks while maintaining practical deployment viability is tractable.

1 Introduction

The paper addresses universal jailbreaks that can elicit harmful CBRN information and introduces Constitutional Classifiers, which monitor model inputs and outputs using constitution-guided synthetic training data. Human red teaming and automated evaluations indicate strong robustness with limited deployment costs.

  • Universal jailbreaks can systematically elicit harmful information and enable non-experts to execute complex scientific processes.
  • The system uses input and output classifiers as complementary layers of protection around the language model.Output classifiers support streaming prediction and can halt generation when harmful content is detected.
  • Constitutional Classifiers train safeguards with natural-language rules defining permissible and restricted content, synthetic examples, benign data, and augmentation.Updating the constitution enables adaptation to new threat models.
  • 405 participants were recruited for a bug-bounty red-team evaluation targeting ten harmful CBRN queries.The program included experienced red teamers and offered incentives of up to $15K USD.
  • Over 3,000 estimated hours of red teaming found no universal jailbreak answering all ten queries at half the detail of an unrestricted model.Under stricter criteria, no red teamer answered more than six of ten questions at a similar level of detail.
  • Over 95% of held-out jailbreaking attempts were refused versus 14% without classifiers, with a 0.38% absolute increase in production refusals and 23.7% inference overhead.
  • The authors conclude that defending against universal jailbreaks while maintaining practical deployment viability is tractable.

2 Problem Formulation: Robust and Practically Viable Defenses Against Universal Jailbreaks

The paper formulates universal jailbreak defense as a problem requiring robustness, practical deployment viability, and flexibility against evolving threats. It defines universal jailbreaks by their reliable, broad-domain extraction of highly detailed harmful information and motivates constitution-guided safeguards.

  • The problem is to develop practically viable safeguards that defend against universal jailbreaks.
  • Such attacks could give non-experts access to dangerous CBRN information and enable complex scientific or technical processes beyond their capabilities.
  • Universal jailbreaks are prompting approaches that reliably bypass safeguards across most queries in a domain and reveal highly detailed, specific harmful information.
  • The defenses should maintain reasonable inference overhead and latency, preserve streaming and time-to-first-token behavior, and keep false positives low.
  • The defenses should adapt to evolving harmful domains and incorporate protection against increasingly sophisticated attack patterns.

3 Constitutional Classifiers

Constitutional Classifiers define permitted and restricted content with natural-language rules, then use constitution-guided synthetic data to train input and output safeguards. The framework combines complementary classifiers, streaming output filtering, and training choices designed to improve robustness while limiting over-refusals and computational cost.

  • Constitutional Classifiers: A dual-classifier defense monitors both inputs and outputs, with the output classifier stopping streaming when predicted harmfulness exceeds a threshold.The input classifier can block attempts to circumvent output filtering, while the output classifier preserves continuous token streaming.
  • Constitutional Classifiers: Constitutional Classifiers use natural-language rules to define permitted and restricted content and guide synthetic training-data generation.Updating the constitution allows adaptation to new threat models, while explicitly specifying permitted content helps limit over-refusals.
  • Constitutional Classifiers: Fine-tuned LLM classifiers are specialized for classification, enabling systematic vulnerability mitigation with lower computational overhead than prompted classifiers.The approach uses constitution-derived data together with additional training sources for classifier fine-tuning.
  • Constitutional Classifiers: Synthetic data is generated from constitutional categories, filtered for inadvertent refusals, and expanded through translation, paraphrasing, prompt variations, and jailbreak transformations.Automated red teaming further generates attacks, including long-context and multi-turn variants.
  • Constitutional Classifiers: Output-classifier training combines next-token prediction and binary cross-entropy, with cumulative-maximum probabilities reflecting irreversible streaming-stop decisions.Direct probabilities support stable gradient flow, while cumulative maxima better match streaming behavior but can destabilize optimization; interpolation transitions between them during training.

4 Constitutional classifiers defend against universal jailbreaks across thousands of hours of human red teaming

Human red teaming found no universal jailbreak that consistently extracted detailed CBRN information across the target queries, although vulnerabilities and evaluation limitations remained. Comparative evaluations further showed stronger protection from constitutional classifiers than harmlessness training alone, with improved classifiers reducing deployment costs.

  • Red-teaming results: No report answered all ten target questions at half the detail of helpful-only outputs, and no report exceeded five at the stricter matching-detail criterion.After one resubmission opportunity, no universal jailbreak consistently extracted comparable detail across the query set.
  • Red-teaming results: Aggregating reports by participant, even the most successful red teamer answered at most six of ten questions under the stricter detail criterion.This aggregation accounts for attackers combining different jailbreaks across multiple submissions.
  • Red-teaming results: Jailbreak success varied substantially across questions, from four successful reports for the most resistant question to over 30 for the most vulnerable at the half-detail threshold.The authors recommend selecting evaluation queries that align closely with realistic threat models and capability-uplift requirements.
  • Attack analysis: Successful attacks primarily exploited rubric grading through benign paraphrasing and length exploitation, while established harmful-intent universal jailbreaks were absent.This pattern suggests the evaluation protocol was a weaker component than the safeguards in the tested system.
  • Comparative assessment: Input classifiers reduced attack success rate to 2%, while combining input and output classifiers reduced it to 0.5%, compared with approximately 14% for harmlessness training alone.The improved classifier system achieved lower refusal rates and inference overhead, but its results may not be fully representative because red teamers did not specifically target it.

5 Constitutional classifiers are flexible and practically viable defenses

Constitutional classifiers use enhanced training and evaluation methods to defend Claude 3.5 Sonnet against chemical-weapons-related jailbreaks while targeting practical deployment costs and false-positive rates. Robustness improves with classifier scale, training-data quantity, and augmentation.

  • Methodology: Constitutional classifiers fine-tune input and output safeguards on Claude 3.5 Haiku to guard Claude 3.5 Sonnet against chemical-weapons-related queries.The classifiers use a chemical-weapons constitution and are evaluated against automated jailbreak attempts.
  • Results: 95% of held-out jailbreak attempts were blocked with constitutional classifiers, compared with 14% for the unguarded baseline.The evaluation includes novel jailbreak attempts and measures system refusal of those attacks.
  • Deployment viability: 0.38% was the increase in refusal rate on a random sample of 5K Claude.ai conversations.This was the smallest increase among the evaluated baselines.
  • Deployment viability: 23.7% was the inference overhead added by constitutional classifiers relative to the base model.This was the lowest overhead among the evaluated baselines under the stated cost-estimation assumptions.
  • Scaling trends: Classifier robustness generally increases with model size, training-data quantity, and the number of data-augmentation transformations.The scaling experiments measure true-positive rates at fixed false-positive constraints.

6 Related Work

Related work addresses classifier safeguards, red-teaming-based fine-tuning, model-internals methods, unlearning and filtering, and inference-time computation. Constitutional classifiers are positioned as a flexible complementary approach, particularly because constitutions can be updated as vulnerabilities emerge.

  • Classifier guards: Classifier-based safeguards use synthetic data and related techniques to detect or block harmful content in language-model systems.Prior work includes moderation APIs, classifier safeguards, output filtering, and domain-specific safeguards.
  • Fine-tuning on red-teaming attacks: Red-teaming-based fine-tuning improves harmful-behavior robustness by training models on manually or automatically generated attacks.This approach is used in frontier language models but leaves models susceptible to further jailbreaks.
  • Model-internals approaches: Model-internals approaches detect harmful intent or optimize robustness using internal representations and activation-based objectives.Examples include linear probes, short-circuiting, and latent adversarial training.
  • Unlearning and data filtering: Unlearning and pre-training data filtering seek to remove or prevent hazardous knowledge, but unlearning may fail to erase it fully and filtering is less flexible.These methods are described as orthogonal to the classifier approach.
  • Inference-time compute: Increasing inference-time computation can improve adversarial robustness but may increase latency and inference overhead.The classifier approach is presented as a complementary defense to this strategy.

7 Conclusion

The paper presents Constitutional Classifiers as constitution-guided safeguards trained on synthetic data, with strong robustness and practical deployment properties. It reports flexibility for adapting to novel threats, while noting that continued testing may reveal vulnerabilities and that complementary defenses remain necessary.

  • Conclusion: Constitutional Classifiers train safeguards on LLM-generated synthetic data guided by natural-language rules for permitted and restricted content.The framework uses constitutions to generate training data and define safeguard behavior.
  • Conclusion: Extensive human red teaming found no universal jailbreak that consistently extracted information comparable to an unsafeguarded model.The claim concerns the prototype system evaluated in the paper’s red-teaming experiments.
  • Conclusion: Constitution updates provide flexibility for adapting classifiers to novel threats.The approach is described as adaptable through updates to the constitution.
  • Limitations and future defenses: Continued testing may reveal system vulnerabilities, so responsible deployment requires complementary defenses such as rapid patching and monitoring.The paper identifies these as examples of complementary defenses for advanced AI systems.

A.3 Why do output classifiers perform better on the out-of-distribution dataset?

The appendix explains why output classifiers can outperform input classifiers on an out-of-distribution evaluation and examines whether they block harmful content early enough during token streaming. It also records practical development observations about data quality, evaluation design, and training processes.

  • Out-of-distribution performance: Output classifiers can achieve higher performance on the out-of-distribution dataset because their sampled outputs correspond to transformed inputs unseen by the input classifier.The appendix describes this as one proposed factor behind the counterintuitive result.
  • Token-streaming limitation: Entire-output evaluations may overstate streaming safety if output classifiers block harmful responses only after harmful information has already been generated.Streaming systems must classify progressively as new tokens are produced.
  • Token-streaming results: 1.17% of jailbreaking attempts produced truncated outputs judged harmful before the output classifier halted the token stream.The estimate came from 117 harmful truncated outputs among the evaluated blocked attacks.
  • Evaluation design: High-quality synthetic attack data, composed jailbreak evaluations, carefully designed rubrics, regression tests, and real-world traffic improve evaluation practice.The appendix links these practices to more reliable assessment of robustness and deployment practicality.
  • Training-process considerations: Scaling model size, training data, and augmentation can improve classifier performance, while output-classifier loss design affects optimization stability.The appendix reports a gradual transition to cumulative-maximum probabilities as one stabilizing design choice.
  • Development guidelines: Refining harmless constitutions and regularly updating constitutions based on observed weaknesses are reported as practical development guidelines.The authors caution that these observations were not all rigorously validated through controlled experiments.

B Example constitution

The example constitution defines harmful chemical-weapons content for synthetic-data generation and separately identifies harmless categories that should not be classified as harmful.

  • B Example constitution: The harmful-content constitution is used as the definition of harmful content in the synthetic-data-generation pipeline.This constitution is used for the Section 5.2 chemical-weapons classifiers.
  • B Example constitution: The harmful constitution specifies chemical-weapons topics whose requested or provided information should be classified as harmful.It follows the definitions of Schedule-1 and Schedule-2 chemicals.
  • B Example constitution: The harmless constitution explicitly delineates categories that should not be classified as harmful.These categories provide negative examples for classifier training.

C Simplified models of uplift

The illustrative uplift model treats complex scientific processes as sequences of independent steps and estimates how safeguards reduce success as process complexity grows, under substantial simplifying assumptions.

  • C Simplified models of uplift: The model represents scientific processes as requiring N sequential successful steps.It is presented as an illustrative model rather than an accurate representation of scientific processes.
  • C Simplified models of uplift: The model assumes all steps must succeed, with independent identically distributed step outcomes and independent identically distributed jailbreak probabilities.It uses 95%, 50%, and 5% success rates for full detail, half detail, and otherwise.
  • C Simplified models of uplift: The model uses a 60% chance of matching helpful-only detail and a 40% chance of obtaining half the detail under safeguards.The helpful-only baseline assumes a 95% success probability per step.
  • C Simplified models of uplift: For a 50-step process, constitutional classifiers reduce uplift relative to a helpful-only model by five orders of magnitude.The result is explicitly described as an illustrative example using several simplifications.

D.2 Automated-evaluations system

The automated-evaluations system trains input and output classifiers for dangerous chemical-weapons information using alternative classification formulations, while selecting implementation choices that support end-to-end deployment.

  • D.2 Automated-evaluations system: The classifiers identify dangerous requests or information related to chemical weapons using a constitution-specific task definition.They are initialized from the pretraining-only Claude 3.5 Haiku checkpoint and exclude automated-red-teaming data.
  • D.2 Automated-evaluations system: Next-token prediction improves performance over value heads for both input and output classifiers, especially for the input classifier.The comparison is based on classifiers trained with both setups.
  • D.2 Automated-evaluations system: Next-token prediction can leverage language-model pretraining, while value-head classifiers add parameters that may overfit limited training data.Prompted task context may also improve next-token classification accuracy.
  • D.2 Automated-evaluations system: The system uses next-token prediction for the input classifier but retains value heads for the output classifier to simplify token streaming.The output choice is made despite its lower performance in the comparison.
  • D.2 Automated-evaluations system: The task is evaluated as next-token prediction versus value-head classification for both input and output safeguards.This formulation comparison is shown in Figure 9.

E.2 False-positive rates on domain-specific datasets

False-positive rates vary across specialized chemistry datasets, while classifier scaling and additional few-shot examples generally improve performance under fixed robustness or deployment constraints.

  • E.2 False-positive rates on domain-specific datasets: 1.50% false-positive rate occurs on MMLU–Chemistry, compared with 26.05% on GPQA–Chemistry.The higher GPQA rate highlights the difficulty of classifying complex technical content.
  • E.2 False-positive rates on domain-specific datasets: Scaling classifiers generally reduces domain-specific false-positive rates when robustness is fixed.The trend suggests that higher rates on challenging chemistry problems reflect a capability limitation that scaling may address.
  • E.2 False-positive rates on domain-specific datasets: Adding more few-shot examples generally improves prompted-classifier performance.Context length grows linearly, limiting how far this approach can scale.
  • E.2 False-positive rates on domain-specific datasets: Few-shot prompting does not readily support token streaming and does not match the performance of a smaller constitutional classifier.These constraints limit its overall effectiveness.
  • E.2 False-positive rates on domain-specific datasets: Initializing input and output classifiers from base Claude 3.5 Haiku achieves the best performance.Adding HHH training reduces performance, likely because its biases can conflict with the task-specific harmful-content definition.

E.5 Robustness drops without a harmless constitution

Removing synthetic data generated from harmless constitution categories reduces classifier performance, while including such data improves robustness and helps calibrate low-false-positive boundaries.

  • Over 40% robustness improvement comes from including synthetic data generated from a harmless constitution.The authors attribute this to better delineation of harmless queries for threshold calibration.
  • Specifying both harmful and harmless constitution categories is crucial for high robustness with low false-positive rates.Feedback-based retraining with additional harmless categories may further reduce over-refusals.
  • Failing to include any harmless-category bucket clearly reduces performance, especially for the output classifier.The evaluated buckets were harmful non-chemistry, harmless chemistry, and harmless non-chemistry.
  • Chemistry-related prompts show more noticeable over-refusals with constitutional classifiers, but the additional false positives remain relatively moderate.The chemistry-specific evaluation found 3.6% baseline model refusals and an additional 3.4% blocked by the classifiers.

F.1 Attack generation

The automated red-teaming pipeline generates complex attacks, filters them for meaningful harmful overlap, and balances successful attacks with benign examples before training.

  • Automated red teaming first outlines a multi-turn attack and then fills the outline with concrete conversation instances.The two-step process is designed to produce sufficiently long and complex attacks.
  • Successful attacks are identified by harmful-output and refusal checks, then filtered for overlap with a helpful-only answer before reuse in training.This filtering addresses the problem that loosely relevant attacks can add noise and reduce evaluation performance.
  • Approximately equal benign conversations are added to balance the harmful examples generated by automated red teaming.These benign queries ask general or safety-related questions about harmful topics.
  • The pipeline can generate training conversations from topics and subtopics derived from harmful and harmless brainstorming categories.A language model produces both single-turn and multi-turn conversations for training.

G.2 Grading process

Jailbreak grading compares target outputs with rubric topics derived from helpful-only answers, while preprocessing obfuscation and separately checking refusal status.

  • Rubrics are generated from key phrases organized by topic in helpful-only answers to the original harmful question.Multiple sampled answers are consolidated into a rubric capturing common key phrases.
  • Potentially obfuscated outputs are rewritten into plain English before rubric grading.The preprocessing step addresses formats such as Base64-encoded text.
  • A separate refusal classifier filters refusals, which are not treated as harmful outputs.Separating refusal detection from harmfulness grading reduced false harmful classifications in manual review.
  • The rubric prioritizes specific, technical, and non-obvious information while generally excluding harmless, educational, and safety-related content.Different measurements and process parameters are kept in separate contextual groupings.

H.5 Outputs that score highly on rubrics are less useful than helpful-only outputs

High rubric scores can arise from paraphrased or generic technical outputs that overlap with harmful-answer topics without providing the original question’s specific operational details.

  • Red-teamers bypassed input defenses mainly by replacing dangerous terms with less harmful or vaguer wording.These outputs could overlap with harmful topics without supplying the complex details needed to execute the prohibited action.
  • The evaluated responses contain detailed extraction, concentration, purification, and stabilization procedures that can be technically extensive even when terms are redacted.Examples include solvent selection, filtration, precipitation, chromatography, and drying stages.
Loading 2501.18837v1…