PaperlayerView exact revision on arXiv

Source-linked AI summary

Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, Rob Gilson, Logan Graham, Logan Howard, Nimit Kalra, Taesung Lee, Kevin Lin, Peter Lofgren, Francesco Mosconi, Clare O'Hara, Catherine Olsson, Linda Petrini, Samir Rajani, Nikhil Saxena, Alex Silverstein, Tanya Singh, Theodore Sumers, Leonard Tang, Kevin K. Troy, Constantin Weisser, Ruiqi Zhong, Giulio Zhou, Jan Leike, Jared Kaplan, Ethan Perez

arXiv:2501.18837v1cs.CLcs.AIcs.CRcs.LG

TL;DR

LLM safety mechanisms can be circumvented by jailbreaks that elicit harmful information, a concern as models’ CBRN capabilities increase. Constitutional Classifiers train safeguards on constitution-guided synthetic data, using natural-language rules to define permitted and restricted content and detect and block jailbreaks. After over 3,000 estimated red-teaming hours, no report answered all ten queries at half detail; at similar detail, no red teamer answered more than six of ten, while classifiers refused over 95% of held-out attempts versus 14% without.

  • Problem

    LLM safety mechanisms can be circumvented by jailbreaks that elicit harmful information, a concern as models’ CBRN capabilities increase.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    Constitutional Classifiers train safeguards on constitution-guided synthetic data, using natural-language rules to define permitted and restricted content and detect and block jailbreaks.

    Source: §3 · p. 4, §3.1 · p. 4

  • Results

    After over 3,000 estimated red-teaming hours, no report answered all ten queries at half detail; at similar detail, no red teamer answered more than six of ten, while classifiers refused over 95% of held-out attempts versus 14% without.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support the tractability of improving jailbreak robustness while preserving deployment viability and adapting safeguards to novel threats through constitution updates.

    Source: §7 · p. 13

    The authors expect vulnerabilities to emerge with continued testing and say responsible deployment will require complementary defenses such as rapid patching and monitoring.

    Source: §7 · p. 13

Abstract

from arXiv · show

Large language models (LLMs) are vulnerable to universal jailbreaks-prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Constitutional Classifiers: safeguards trained on synthetic data, generated by prompting LLMs with natural language rules (i.e., a constitution) specifying permitted and restricted content. In over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak that could extract information from an early classifier-guarded LLM at a similar level of detail to an unguarded model across most target queries. On automated evaluations, enhanced classifiers demonstrated robust defense against held-out domain-specific jailbreaks. These classifiers also maintain deployment viability, with an absolute 0.38% increase in production-traffic refusals and a 23.7% inference overhead. Our work demonstrates that defending against universal jailbreaks while maintaining practical deployment viability is tractable.

1 Introduction

The introduction presents Constitutional Classifiers as an adaptable, deployment-viable defense against universal jailbreaks and reports evidence that robust protection can be practical. It frames the work around reducing CBRN misuse risk while retaining deployment viability.

1 Introduction

  • Jailbreaks can elicit harmful information, and universal attacks could enable non-experts to execute complex CBRN processes, motivating defenses that adapt to evolving threats and remain viable for deployment.Source: §1 · pp. 1–2
  • Constitutional Classifiers use explicit rules to generate synthetic training data for safeguards monitoring model inputs and outputs, with benign-data pools, augmentation, and constitution updates supporting adaptation.Output classifiers assess harmfulness at each token, enabling real-time intervention before generation finishes.Source: §1 · p. 2, Figure 1 · p. 2
  • Across over 3,000 estimated red-teaming hours on ten harmful CBRN queries, no report answered all ten at half unrestricted-model detail; under stricter criteria, no red teamer answered more than six.The early prototype prioritized robustness and had substantially elevated refusal rates compared with later implementations.Source: §1 · pp. 2–3
  • Improved Claude 3.5 Haiku classifiers refused over 95% of held-out jailbreaking attempts versus 14% without classifiers, with a 0.38% absolute production-traffic refusal increase and 23.7% inference overhead.The automated evaluations included novel jailbreak attempts.Source: §1 · p. 3

2 Problem Formulation: Robust and Practically Viable Defenses Against Universal Jailbreaks

The section defines three desiderata for safeguards: resist universal jailbreaks, remain practical to deploy, and adapt to evolving harms and attacks.

Desideratum 1: robustness to universal jailbreaks

  • Non-expert uplift can enable complex processes beyond actors’ capabilities when they obtain reliable, accurate guidance across most queries and detailed, correct, specific information.These conditions distinguish consequential procedural assistance from general knowledge.Source: §2 · p. 3
  • Desideratum 1 targets universal jailbreaks—prompts that reliably bypass safeguards across most queries in a domain and elicit detailed, specific harmful information—to enable wide deployment of advanced-capability LLMs.The concern includes non-experts gaining access to dangerous CBRN information.Source: §2 · p. 3

Desideratum 2: practical deployment viability

  • Desideratum 2 requires reasonable inference overhead and latency, preserved time-to-first-token and streaming, and low false-positive rates for practical deployment.Source: §2 · p. 4

Desideratum 3: flexibility

  • Desideratum 3 requires safeguards flexible enough to adapt to evolving harm domains and incorporate defenses against novel, increasingly sophisticated attack patterns.Source: §2 · p. 4

3 Constitutional Classifiers

Constitutional Classifiers use natural-language rules to generate synthetic training data for safeguards that distinguish permitted from restricted content. Their dual input- and output-classifier design combines blocking attempted circumvention with streaming output filtering.

3 Constitutional Classifiers

  • The defense pairs an input classifier that blocks attempts to circumvent output filtering with a streaming output classifier that mitigates prevalent jailbreak strategies.Constitution-guided synthetic data, augmentation, and benign examples support robust defenses while minimizing false positives.Source: §3 · p. 4

3.1 Specifying a constitution

  • Natural-language constitutions define permitted and restricted content categories, which can be updated for new threats and make classifier behavior interpretable.Including permitted categories helps limit over-refusals; delineating harmful and harmless content also produces nuanced training data and can support different safeguard classes.Source: §3.1 · p. 4

3.2 Generating synthetic data

  • Fine-tuned LLM classifiers enable systematic vulnerability mitigation and can reduce computational overhead compared with prompted classifiers.Source: §3.2 · p. 4
  • Synthetic-data generation first creates constitution-spanning queries and harmful and harmless outputs, then filters inadvertent refusals.Augmentations preserve meaning while diversifying inputs and outputs through translation, paraphrasing, system-prompt variations, jailbreak techniques, and their combinations.Source: §3.2 · p. 5

3.3 Automated red teaming

  • Automated red teaming prompts a helpful-only model to generate attacks from known techniques and forbidden queries, including templated long-context and multi-turn attacks.Source: §3.3 · p. 5

3.4 Training classifiers

  • The input classifier is fine-tuned for next-token prediction, including coded or obfuscated inputs, with harmless-query pools to reduce false positives.The output classifier uses a linear value head to predict full-sequence harmfulness from prefixes, stopping streaming when its score exceeds a threshold; cached activations support efficient token-wise classification.Source: §3.4 · p. 5
  • Output-classifier training gradually interpolates from direct value-head probabilities to their cumulative maximum, moving toward streaming-aligned predictions while supporting stable initial gradient flow.The weighting parameter starts at 0 and increases linearly to 1 three-quarters through training.Source: §3.4 · p. 6

4 Constitutional classifiers defend against universal jailbreaks across thousands of hours of human red teaming

Human red teaming found no universal jailbreak that consistently extracted detailed answers across the target queries, and comparative evaluations showed that constitutional classifiers reduced attack success rates beyond harmlessness training alone. The results also expose limits in the evaluation and prototype deployment, including rubric exploitation and high refusal rates.

4.1 Methods

  • The evaluation used ten target queries, rubric scores against helpful-only outputs, and a 50% detail threshold; estimated red-teaming effort averaged 4,720 hours (90% CI: [3,242, 7,417]).Participants could retry once after threshold feedback, and the effort estimate was calibrated using surveyed self-reported hours, which may systematically overestimate effort.Source: §4.1 · p. 7

4.2 Results

  • The red-teamed prototype refused approximately 44% of Claude.ai traffic, and the evaluation remains limited by its cohort, preset chatbot queries, evolving attacks, and rubric difficulty distinguishing actionable information.The paper also notes that paying out at the half-detail threshold may have left stronger attacks undiscovered.Source: §4.1 · p. 7
  • None of 113 reports answered all ten questions at half the helpful-only detail, and no report answered more than five at the stricter matching-detail threshold.Nine reports answered more than six questions at the half-detail threshold, but aggregating all reports per red teamer still yielded at most six questions at the stricter threshold, even with resubmission opportunities.Source: Figure 4 · p. 8, §4.2 · p. 8
  • At the 50%-detail threshold, the most-resistant query appeared in only four successful reports, while the easiest appeared in over 30, underscoring the importance of realistic query selection.The authors recommend aligning red-team queries with realistic threat models and the information malicious actors would need to achieve uplift.Source: Figure 4 · p. 8, §4.2 · pp. 8–9
  • Among the highest-severity reports, benign paraphrasing and length exploitation were predominant, while many-shot jailbreaking and the “God-Mode” jailbreak were absent.The authors interpret these results as attackers targeting the evaluation rubric’s weaknesses rather than primarily circumventing the classifier safeguards.Source: §4.2 · p. 9
  • A simplified model suggests safeguards could reduce uplift by multiple orders of magnitude for processes requiring many sequential steps, but this is illustrative rather than a precise estimate.The model assumes a 60% chance of detailed answers and a 40% chance of partial information per step, based on the most successful red teamer; it also assumes independent, uniformly difficult steps.Source: §4.2 · p. 9

4.3 Comparative assessment of defensive components

  • ASR fell from ∼16% for the helpful-only baseline to ∼14% with HHH, 2% with an input classifier, and 0.5% with input and output classifiers combined.The improved classifier methodology also achieved lower attack success rates than the red-teamed system and an approximately 1.74% false-positive rate on production Claude.ai traffic; it used a different constitution and was not specifically red-teamed.Source: Figure 5 · p. 9, §4.3 · p. 10

5 Constitutional classifiers are flexible and practically viable defenses

Section 5 shows that classifiers trained with a distinct chemical-weapons constitution can guard Claude 3.5 Sonnet (New) against held-out jailbreaks while limiting production refusal increases and inference overhead.

5.1 Methodology

  • Fine-tuned Claude 3.5 Haiku input and output classifiers, using a distinct chemical-weapons constitution, guarded Claude 3.5 Sonnet (New).The enhanced methodology expanded harmless training data, recalibrated thresholds, and expanded augmentations to reduce false positives; thresholds were selected to maximize true-positive rate while meeting specified false-positive limits.Source: §5.1 · pp. 10–11, §5 · p. 10
  • The evaluation targeted ten chemical-weapons queries and included 5,681 held-out attacks using primitives and transformations absent from training augmentation, including novel and classifier-targeted jailbreaks.The attack set was filtered to retain prompts that elicited harmful responses from a helpful-only model.Source: §5.1 · p. 10

5.2 Results

  • On held-out chemical-weapons jailbreaks, classifiers blocked 95% of attempts versus 14% unguarded; production refusals rose 0.38% and inference overhead was 23.7%.The prose reports jailbreak success falling from 86% against an HHH model to less than 5%; the production refusal rate was measured on 5,000 conversations. No single transformation answered all ten queries, suggesting protection against universal jailbreaks; false-positive increases were more noticeable for chemistry queries.Source: Figure 6 · p. 11, §5.2 · p. 12
  • Robustness generally increased with classifier model size, training-data quantity, and the number of data-augmentation transformations.Larger models also showed smaller generalization gaps for input classification.Source: §5.2 · p. 12, Figure 7 · p. 12

6 Related Work

The section situates constitutional classifiers among other safeguards, arguing that they offer flexibility and robustness to universal jailbreaks while addressing limitations of fine-tuning, model-internals methods, unlearning, data filtering, and inference-time compute scaling.

Classifier guards

  • This work reports that classifier-based defenses can achieve robustness to universal jailbreaks even in the face of thousands of hours of red teaming, and suggests additional classifier fine-tuning would improve performance.Source: §6 · p. 13

Finetuning on red-teaming attacks

  • Fine-tuning on red-team attacks can leave models vulnerable to universal jailbreaks, and preliminary experiments found unreliable generalization from text to code.The passage names many-shot jailbreaking and GCG as examples of universal jailbreaks.Source: §6 · p. 13

Model-internals approaches

  • Model-internals methods include activation probes and specialized fine-tuning losses, but classifiers require no changes to the LLM fine-tuning procedure.The authors say their synthetic datasets can also support these methods, while it remains unclear whether they alone can match classifier-based universal-jailbreak robustness.Source: §6 · p. 13

Unlearning and data filtering

  • Unlearning can fail to fully erase undesired knowledge, while pre-training data filtering lacks the flexibility of the classifier approach.The passage describes unlearning and pre-training data filtering as orthogonal methods.Source: §6 · p. 13

Robustness via scaling inference-time compute

  • Inference-time compute scaling may improve adversarial robustness, but its latency and potentially large overhead make classifier safeguards complementary.The passage attributes the exploration of increased inference-time compute for reasoning-model robustness to Zaremba et al. (2024).Source: §6 · p. 13

7 Conclusion … A.2 How is the inference overhead of classifiers calculated?

Constitutional Classifiers use synthetic data generated from natural-language rules to improve robustness, while aiming to limit over-refusals and inference overhead. The conclusion presents them as promising but says responsible deployment also requires complementary defenses.

7 Conclusion

  • Human red teaming found no universal jailbreak that consistently extracted information comparable to an unsafeguarded model; later improvements reduced over-refusals and inference overhead.The constitution-based approach can also adapt to novel threats through constitution updates.Source: §7 · p. 13
  • The authors expect vulnerabilities to emerge with continued testing and recommend complementary defenses, including rapid response patching and monitoring for novel jailbreaks.They cite Peng et al. (2024) for rapid response and Hendrycks et al. (2021) for monitoring, and anticipate a role for classifiers in deployments such as those requiring ASL-3 standards.Source: §7 · p. 13

A.1 How do you ensure that augmented prompts used in automatic evaluations preserve the semantics of their original harmful questions?

  • Augmented prompts are retained only if their outputs sufficiently overlap with those for the original harmful queries, removing prompts judged to have changed semantics.The comparison uses outputs from a helpful-only Claude 3.5 Sonnet version trained to answer users even when providing harmful information.Source: §A.1 · p. 20, §A.2 · p. 20

A.2 How is the inference overhead of classifiers calculated?

  • Classifier overhead is estimated from input and output token counts, assuming cached prompt wrappers and classification of entire outputs rather than token streaming.Most setups use the same input and output token counts, except chain-of-thought classifiers, which produce variable output lengths.Source: §A.2 · p. 20
  • N = 19,322.88, M = 607.22, KI = 232.52, and KO = 250.46 tokens were estimated from 1,000 sampled production input–output pairs and prompted chain-of-thought classifiers.N and M are average production input and output lengths; KI and KO are average input- and output-classifier token counts.Source: §A.2 · p. 20
  • Classifier costs are estimated using Anthropic API input- and output-token prices, alongside the cost of running the guarded model on N input and M output tokens.Constitutional output classifiers produce no output tokens in the value-head setup.Source: §A.2 · p. 20

A.3 Why do output classifiers perform better on the out-of-distribution dataset? · A.4 In the token-streaming setup, do classifiers block harmful outputs early enough? · A.5 What are some missable tips for implementing constitutional classifiers?

Output classifiers performed better on the evaluated out-of-distribution dataset, and token-streaming tests found that harmful information was rarely exposed before blocking. The implementation guidance covers data quality, evaluation design, training, and constitution refinement, though the authors note it is not all rigorously validated.

A.3 Why do output classifiers perform better on the out-of-distribution dataset?

  • Figure 7 found that output classifiers performed better on the out-of-distribution dataset.The authors suggest its examples were not explicitly out of the output classifiers’ training distributions, and that less-capable model outputs may have appeared more suspicious and easier to detect.Source: §A.3 · p. 21

A.4 In the token-streaming setup, do classifiers block harmful outputs early enough?

  • Whole-output evaluations may miss whether token-streaming classifiers block outputs as soon as harmful information appears.A classifier can block a harmful completed output yet still fail to reduce misuse risk if it does not block early enough during generation.Source: §A.4 · p. 21
  • On 10K full outputs, the system blocked 96.70%; only 117 truncated outputs were harmful, so blocking was too late in 1.17% of jailbreaking attempts.Truncation simulated what users could see before the token stream halted; harmfulness was judged by a language model comparing outputs with a helpful-only model’s response.Source: §A.4 · p. 21

A.5 What are some missable tips for implementing constitutional classifiers?

  • Filter refusals and off-topic synthetic responses, and validate that attack examples elicit harmful outputs while benign queries stay away from the classification boundary.The authors warn that low-quality automated attack examples can reduce classifier performance and noisy benign examples can degrade training data quality.Source: §A.5 · p. 21
  • Evaluation suites should compose varied jailbreaks, distinguish actionable content from general information, track jailbreak-specific regressions, and include real-world traffic.These practices are intended to support interpretable, reliable evaluation and practical deployment assessment.Source: §A.5 · pp. 21–22
  • Scale training across model, dataset, and augmentation dimensions; for output classifiers, gradually shift to cumulative-maximum probabilities and tune thresholds for robustness with low benign impact.The probability transition helped stabilize optimization while maintaining streaming performance.Source: §A.5 · p. 22
  • Refine harmless categories to reduce false positives and update the constitution regularly based on evaluation weaknesses.The authors present these guidelines as experience-based observations, noting that not all were rigorously validated through controlled experiments.Source: §A.5 · p. 22

B Example constitution

The example constitution defines harmful chemical-weapons content for synthetic-data generation, using specified topics and Schedule-1 and Schedule-2 chemicals. Explicitly delineating harmless categories can significantly improve classifier performance.

B Example constitution

  • Adding a constitution that explicitly identifies harmless chemical-weapons categories can significantly improve classifier performance.The example harmless constitution is shown in Table 4.Source: B Example constitution · p. 23, Table 4 · p. 24
  • The harmful-content constitution targets inputs and outputs requesting or providing information about specified chemical-weapons topics, including Schedule-1 and Schedule-2 chemicals.It is used to define harmful content in the synthetic-data-generation pipeline.Source: Table 3 · p. 23, B Example constitution · p. 23

C Simplified models of uplift · D System details · D.1 Bug-bounty system

Under a simplified model of multi-step scientific processes, constitutional classifiers sharply reduce uplift as process complexity increases. The paper also describes the prototype and classifier setup used for human red teaming.

C Simplified models of uplift

  • The illustrative model assumes all N sequential steps must succeed, with independent, identically distributed success probabilities at each step.It treats step success as independent, although real scientific processes can involve dependencies and feedback loops.Source: C Simplified models of uplift · p. 25
  • The model assigns 95% success to similar-to-helpful detail, 50% to half that detail, and 5% otherwise.For helpful-only models, Figure 8 assumes a 95% success probability per step.Source: C Simplified models of uplift · p. 25, Figure 8 · p. 25
  • Based on the top red teamer’s performance, the model estimates attackers have a 60% chance of detailed information and a 40% chance of partial information per step.The authors say this likely overstates attacker capabilities because red teamers could iterate with feedback, rubric grading was imperfect, and the estimate reflects the best of over one hundred participants.Source: C Simplified models of uplift · p. 25
  • The model is illustrative, not an accurate account of scientific processes, and its assumptions may not generalize to real-world adversaries.It uses coarse discrete probabilities, assumes uniform difficulty across steps, and omits recovery from errors and dependencies between steps.Source: C Simplified models of uplift · p. 25
  • For a 50-step process, classifiers reduce uplift relative to helpful-only models by five orders of magnitude under the illustrative assumptions.Figure 8 describes the reduction as a median 105-fold reduction; the model assumes sequential successful steps and estimates uncertainty by sampling, without accounting for model error.Source: Figure 8 · p. 25, C Simplified models of uplift · p. 25

D.1 Bug-bounty system

  • Human red teaming used prototype classifiers trained under a different constitution from automated evaluations, while still targeting dangerous scientific knowledge related to weapons of mass destruction.Source: §D.1 · p. 26
  • Primary classifiers initialized from Claude 3.5 Sonnet guarded Claude 3.5 Sonnet; an improved Claude 3.5 Haiku-initialized version matched robustness with lower over-refusal.Both systems included automated red-teaming data, which the authors found improved classifier performance.Source: §D.1 · p. 26

D.2 Automated-evaluations system · E Constitutional classifiers · E.1 Classifier types

For chemical-weapons classifiers, next-token prediction performs better than value heads, especially for input classification, though the system uses value heads for output streaming. The classifiers are narrowly scoped and use revised synthetic-data training and fine-tuning choices.

D.2 Automated-evaluations system

  • The input and output classifiers target specific chemical-weapons-related requests or information, not unrelated harms such as bank-robbery planning.They use the Appendix B constitution; compared with the bug-bounty classifiers, training was revised with a new constitution, an explicit harmless constitution, better fine-tuning hyperparameters, more data, and more augmentation transformations.Source: §D.2 · p. 26
  • Both classifiers start from pretraining-only Claude 3.5 Haiku, exclude automated-red-teaming data because its benign examples added noise, and guard Claude 3.5 Sonnet (New).Source: §D.2 · p. 26

E.1 Classifier types

  • Next-token prediction outperforms value heads for both input and output classifiers, with a larger improvement for input classification.Figure 9 reports error bars computed from 95% confidence intervals.Source: §E.1 · p. 27, Figure 9 · p. 27
  • Next-token prediction more closely matches pretraining language modeling, letting the model draw on learned language patterns, context, and semantics.The authors suggest value heads may use this existing knowledge less effectively.Source: §E.1 · p. 27
  • Value-head classifiers add trainable parameters that may overfit limited training data, whereas next-token prediction uses pretrained parameters that may generalize better.Source: §E.1 · p. 27
  • Next-token prediction prompts specify the classification task and possible classes, which the model can use to improve prediction accuracy.Source: §E.1 · p. 27
  • The system selects next-token prediction for input classification but retains value heads for output classification to simplify token streaming.The output value-head setup has lower performance, but supports the end-to-end system's streaming implementation.Source: §E.1 · p. 27

E.2 False-positive rates on domain-specific datasets · E.3 Scaling few-shot prompted classifiers · E.4 Initializing classifiers from finetuned models

These appendices examine false positives on specialized scientific content, few-shot prompting, and classifier initialization. They report that classifier scaling can reduce domain-specific false positives, while base-model initialization performs best among the tested options.

E.2 False-positive rates on domain-specific datasets

  • Constitutional classifiers had a 1.50% false-positive rate on MMLU–Chemistry questions and a 26.05% rate on GPQA–Chemistry questions, highlighting difficulty with complex technical content.The passage attributes this challenge to domain understanding that may exceed classifier capabilities.Source: §E.2 · p. 27
  • At fixed 20% robustness, increasing classifier size generally reduced false-positive rates on chemistry-specific GPQA and MMLU subsets.The authors suggest the higher rates on challenging chemistry problems reflect a capability limitation that scaling may address.Source: Figure 10 · p. 28, §E.2 · p. 28

E.3 Scaling few-shot prompted classifiers

  • The few-shot study used randomly selected, under-1K-token training examples misclassified by a zero-shot Claude 3.5 Sonnet classifier, with prompts balancing positive and negative cases.The example count was scaled subject to context-length constraints.Source: §E.3 · p. 28
  • Few-shot prompting increases context length linearly, does not readily support token streaming, and still does not match a smaller constitutional classifier.These constraints limit how far the approach can be scaled.Source: §E.3 · p. 28, Figure 11 · p. 28
  • Adding few-shot examples generally improved prompted-classifier performance at fixed false-positive constraints.Examples were selected from training cases the zero-shot classifier predicted incorrectly, filtered to fewer than 1K tokens, and balanced between positive and negative examples.Source: Figure 11 · p. 28, §E.3 · p. 28

E.4 Initializing classifiers from finetuned models

  • The initialization comparison held datasets and training hyperparameters fixed across base Claude 3.5 Haiku, HHH-trained Haiku, and helpful-only-trained Haiku.The helpful-only model received additional training to be helpful even if that required being harmful.Source: §E.4 · p. 29
  • Base Claude 3.5 Haiku initialization achieved the best performance for both input and output classifiers; HHH and helpful-only training reduced performance.The authors suggest preexisting biases may conflict with the constitution’s task-specific definition of harmful content.Source: Figure 12 · p. 29, §E.4 · p. 29

E.5 Robustness drops without a harmless constitution … F Automated red teaming

These appendices show that harmless-constitution data improves classifier robustness and helps define harmless boundaries, while examining classifier prompting and false positives on chemistry-related queries. The supplied passages cover E.5–E.8; they provide no evidence about section F.

E.5 Robustness drops without a harmless constitution

  • Including harmless-constitution data improves robustness by over 40%.The authors attribute this to better delineation of harmless queries for threshold calibration; they also report preliminary use of feedback retraining on over-refused harmless categories as a way to reduce false positives.Source: §E.5 · p. 29, Figure 13 · p. 30

E.6 Importance of categories in a harmless constitution

  • Omitting any harmless-category bucket reduces classifier performance, especially for the output classifier, with no single bucket having the greatest effect on both classifiers.The buckets were harmful non-chemistry, harmless chemistry, and harmless non-chemistry; fixed benign examples were retained in all ablations.Source: §E.6 · p. 30, Figure 14 · p. 30

E.7 Prompt wrappers

  • The constitutional input and zero-shot prompted input classifiers use a prompt wrapper, and both input and output wrappers target coded communication and obfuscated outputs.The zero-shot prompted output classifier also uses a prompt wrapper.Source: §E.7 · p. 31

E.8 Refusal rates on chemistry-related user queries

  • The chemistry-specific Wildchat evaluation set contains 1,000 examples: 500 single-turn and 500 multi-turn queries.Queries were sampled from conversations classified as chemistry-related with confidence of at least 0.99.Source: §E.8 · p. 31
  • 3.6% of chemistry-related inputs were refused by Claude 3.5 Sonnet (new), and the classifiers blocked an additional 3.4% of queries that did not result in a model refusal.The authors note that classifier false positives are more noticeable on chemistry-related prompts, but characterize the overhead as relatively moderate.Source: §E.8 · p. 31, Figure 15 · p. 31

F.1 Attack generation … G.1 Rubric generation

Automated red teaming builds multi-turn jailbreaks, but loosely relevant attacks can add noise to classifier training; overlap grading, benign-query balancing, and rubric-based answer checks refine the data and evaluation process.

F.1 Attack generation

  • ART generates long, complex multi-turn attacks by outlining conversation stages and resampling each stage into concrete exchanges.Successful attacks can be added to training datasets to improve classifiers.Source: §F.1 · p. 35, Figure 16 · p. 35
  • An attack is successful only if it produces meaningfully harmful output; helpful-only models classify refusal and harmfulness, and successful attacks enter a pool for future few-shot red teaming.Because training on all such attacks caused over-refusal, inputs are additionally filtered by output overlap with a helpful-only response to the original harmful query.Source: §F.1 · p. 35

F.3 Benign catastrophic-risk queries

  • Nearly equal numbers of benign queries about harmful topics may balance ART examples, which generally account for fewer than 2% of training samples.Benign examples include general questions about harmful topics; a language model generates single- and multi-turn conversations from harmful and harmless topics and subtopics.Source: §F.1 · p. 35, §F.3 · pp. 35–36

F.2 Helpful-only overlap grading

  • Loosely relevant ART attacks reduced automated-evaluation performance because they failed to answer the harmful question and added training noise.Claude 3.5 Sonnet grades whether an attack output sufficiently overlaps a helpful-only reference answer to the original harmful question.Source: §F.2 · p. 35, Table 8 · p. 36

G.1 Rubric generation

  • Rubrics identify topic-organized key phrases from helpful-only answers, then combine common phrases across five sampled outputs into one rubric.Typical generated outputs contain 8–18 topics, each potentially containing multiple key phrases.Source: §G.1 · p. 37
  • A target response counts as answering the original question if it covers at least max(2, ⌈α/4⌉) rubric topics and a language model judges coverage sufficient.α is the average topic-level overlap between individual helpful-only outputs and the combined rubric.Source: §G.1 · p. 37

G.2 Grading process … H.4 Estimating hours of red-teaming effort

This section describes how jailbreak outputs were graded and reported using query-specific rubrics that account for obfuscation and distinguish phrase matching from sufficient topic coverage, alongside red-teamer recruitment and estimation of total effort (3,241–7,417 hours).

G.2 Grading process

  • The rubric is built by extracting keywords from helpful-only outputs to the original harmful question and consolidating them into an overlap list of recurring, key information.The resulting rubric aims to stay concise, merge similar concepts, and exclude common harmless terms and groupings found in only a few completions.Source: §G.2 · pp. 37, 41
  • Potentially obfuscated targets are rewritten in plain English, scored for rubric phrase matches, and then judged for sufficient topic coverage; refusals are filtered out rather than graded for harmfulness.For the topic-coverage decision, the model receives the original question, target output, rubric evaluation, and topic metadata.Source: §G.2 · p. 37
  • Separating phrase matching from topic-coverage decisions reduced outputs mistakenly marked harmful in manual review; the final topic-decision step was omitted for bug-bounty outputs, which required ⌈α/2⌉ topics.Source: §G.2 · p. 37
  • The grading process used keyword rubrics to assess whether jailbreak outputs contained enough information to count as successful answers to harmful queries.The grader compared key phrases in target outputs with those in helpful-only model responses; outputs that might be obfuscated could be deobfuscated using model-generated instructions.Source: Table 11 · p. 43, Table 12 · p. 44, Table 13 · p. 46

H.1 Recruiting red teamers

  • 4051 people were invited to red team the system, with application answers graded by Claude 3 Opus and high-scoring applicants admitted.Applicants described their jailbreaking experience and answered open-ended questions used to assess effort, ethics, and skill.Source: §H.1 · p. 48, Figure 17 · p. 48

H.2 Experience levels of red teamers

  • Most recruited red teamers had prior jailbreaking experience, and a non-trivial proportion were full-time bug-bounty hunters.Applicants self-reported whether they were new, part-time/hobbyists, or full-time bug-bounty hunters.Source: §H.2 · p. 48, Figure 18 · p. 49

H.3 Jailbreak-reporting process

  • Of 113 HackerOne jailbreak reports, 85 were used for Figure 4 after exclusions for inapplicability, wrong-domain attacks, and superseded resubmissions.Claude graded submissions against rubrics, participants could resubmit once, and message IDs were required to verify submissions.Source: §H.3 · p. 49

H.4 Estimating hours of red-teaming effort

  • The calibrated 90% confidence interval for total red-teaming effort was 3,241–7,417 hours.The estimate used API-query buckets to approximate active time, with bucket duration fitted to survey-reported hours and uncertainty calibrated through bootstrap resampling. The method relies on self-reported hours, which may overestimate time spent red teaming.Source: §H.4 · pp. 49–50, Figure 19 · p. 50

H.5 Outputs that score highly on rubrics are less useful than helpful-only outputs

This section examines how rubric scores relate to the usefulness of outputs: highly scoring outputs could appear harmful yet lack enough context to meaningfully execute the task, making them less helpful than helpful-only outputs. It also reports that submissions framed without harmful concepts exceeded the rubric threshold, while helpful-only outputs scored higher.

H.5 Outputs that score highly on rubrics are less useful than helpful-only outputs

  • Rubric graders rated some outputs as successful jailbreaks even though manual review found them less useful for accomplishing a harmful task than helpful-only output.The authors argue these outputs lacked the complex details needed to execute the prohibited action in the correct context.Source: §H.5 · p. 50
  • Rephrasing harmful questions with less-harmful or vaguer terms produced related but insufficiently actionable outputs that could game rubrics by echoing their keywords.The passage says most red-teaming reports used this kind of rewriting to get past the input classifier.Source: §H.5 · p. 50
  • The examples scored 8/12 and 5/12, while a helpful-only output scored 10/12 and directly addressed extracting ricin toxin.The higher-scoring submissions instead framed their responses as general protein extraction or bioactive-compound methodologies.Source: §H.5 · pp. 50–51

Related papers

Loading 2501.18837v1…