Constitutional adapters: Inference-time interventions for misalignment and misuse

1 INTRODUCTION

• Constitutional approaches are promising, but their robustness against both misuse and misalignment remains unclear.They offer transparency into governing values and some generalization to novel scenarios and persistence across post-training.Source: §1 · p. 1
• Whether lightweight adapters or steering vectors can capture nuanced constitution-following remains unclear, despite their flexibility and deployment advantages.Such interventions could let inference controllers shape behavior in graded fashion rather than rely on classifiers or circuit breakers that halt exchanges.Source: §1 · p. 1
• Constitution-trained lightweight objects are contrasted with control-trained objects from a near-identical synthetic-data pipeline, and their subtraction yields tunable defenses that compare favorably with prompted and steered baselines.The figure depicts the synthetic-data and fitting pipeline; the reference document is the substantive difference between the corpora.Source: §1 · pp. 1–2, Figure 1 · p. 2

2.1 SYNTHETIC DATA

• The synthetic-data pipeline uses principles to generate pressure-testing scenarios and synthetic advice chats or stories, then rewrites, regenerates, scores, and filters the documents.It produced approximately 18,000 documents totaling 45M tokens for each corpus, with similar length distributions, system-prompt markers, and diversity statistics.Source: §2.1 · pp. 2–3

2.2 LIGHTWEIGHT OBJECTS

• Lightweight objects are fit to synthetic corpora by optimizing next-token loss, with LoRAs modifying weights and steering vectors intervening on activations.The steering baseline uses a difference-in-means refusal direction; the authors also use ReLU-gated steering to allow write strength to vary with the residual stream.Source: §2.2 · p. 3

2.3 DIFFERENCE OBJECTS

• The primary intervention subtracts a control-trained object from a constitution-trained object; LoRA differences are stored as rank-2r adapters, while ReLU steer differences compare write vectors on the same pre-write state.Both interventions can be scaled by α before being added to weights or activations, respectively.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

2.4 EVALUATION PROTOCOL

• Evaluation covered static and multi-query single-turn jailbreaks, multi-turn jailbreaks, over-refusal, alignment, and general capabilities in knowledge, math, coding, and tool use.Putative jailbreaks were graded by Claude Haiku 4.5 using StrongREJECT, counting responses only when scores were strictly greater than 0.5.Source: §2.4 · p. 4

3.1 SINGLE-TURN JAILBREAKS

• As context length increased, prompting fell below unprompted defense, while per-token lightweight objects remained robust; control-corpus objects impaired defense at every length.Non-control lightweight objects were similarly successful against the selected static jailbreaks, which included attacks using ASCII noising, translations, glyphs, and code formatting despite training on advice and story distributions.Source: Figure 3 · p. 5, §3.1 · p. 5

3.2 MULTI-QUERY AND MULTI-TURN JAILBREAKS

• Constitution and especially difference objects increased defense success by ∼15 and 30 pp, respectively, on adaptive multi-turn jailbreaks, and difference objects were over 3× more effective than refusal steering.The comparison covers Crescendo and ActorAttack; constitution and difference objects also achieved high defense success across multi-query and multi-turn attacks.Source: §3.2 · pp. 5–6, Figure 4 · p. 6

3.3 MISALIGNMENT MITIGATION

• Constitution and especially difference objects were robust against Petri misalignment risks, while refusal steering was also effective and system prompting had no significant effect.In agentic scenarios with simulated email tools, constitution and difference objects almost completely eliminated leaking and reliably outperformed refusal steering and prompting.Source: §3.3 · p. 6, Figure 5 · p. 6

3.4 OVER-REFUSAL AND CAPABILITIES

• At the default steering strength (α = 1), difference objects increased benign-input refusals comparably to refusal steering (all 10–18 pp), while MMLU and GSM8K showed little to no capability hit.Lightweight objects reduced BCB and BFCL pass rates, though the reduction was substantially alleviated for difference ReLU steering and was subject to seed variance.Source: §3.4 · p. 7, Figure 6 · p. 7

3.5 INFERENCE-TIME TUNING

• Difference LoRAs Pareto-dominated refusal steering across tested multi-turn attack types and steering strengths, while difference ReLU steers achieved superior multi-turn defense only at α ≥1.The objects traced a frontier trading off defense against benign compliance, supporting titration according to defense need and over-refusal budget.Source: §3.5 · p. 7

Task arithmetic

• Constitution-derived objects are trained directly for safety, with control subtraction removing shared shifts in register, format, and compliance rather than relocating capabilities already present in a post-trained model.The shared shifts are those accompanying fine-tuning on synthetic documents.Source: §4 · p. 8

Activation steering

• Fixed additive steering vectors can generalize unreliably across inputs, degrade unrelated capabilities, and be matched or beaten by prompting.Conditional interventions and gradient-fitted steering are two approaches to these problems.Source: §4 · p. 8

Jailbreaks

• Prior safety defenses include input and output classifiers, defensive prompting, adversarial training, representation rerouting, and inference-time steering, but many are fitted to harmful or adversarial examples.Some defenses reporting near-zero single-turn attack success failed under multi-turn human red teaming.Source: §4 · p. 8

Alignment training and generalization

• Prior constitutional alignment methods use written principles as supervised or reinforcement-learning targets, while newer work studies constitution-derived training corpora.This paper asks whether lightweight objects can carry corpus effects through independent post-training, tune at inference, and generalize from misalignment to misuse.Source: §4 · pp. 8–9

5 DISCUSSION

• Lightweight ReLU steers and LoRAs defend against both misalignment and misuse, with strong results at long context, against multi-turn jailbreaks, and on misalignment evaluations despite no harmful-query training.The authors suggest the small objects' success may indicate constitution-following behavior is low-dimensional and linearly represented.Source: §5 · p. 9

5.1 ADVANTAGES OF CONSTITUTIONAL ADAPTERS

• The evaluation does not test whitebox attacks optimized against the adapter or compare CAs with full-parameter-scale training.The authors suspect more constitutional training in the standard pipeline could leave less for CAs to add.Source: §5.1 · p. 9
• CAs use ∼330k and 426k parameters on a 4B model for ReLU and LoRA difference objects, respectively, and transfer zero-shot from base to post-trained checkpoints.The authors present this portability as potentially useful when serial or interleaved post-training is impractical.Source: §5.1 · p. 9
• Subtracting a register-matched control-trained object can strengthen defense against adaptive attacks while minimizing off-target damage from synthetic data.The authors describe this control-subtracted task arithmetic as an additional way to learn from synthetic data during midtraining.Source: §5.1 · p. 9
• Steering strength α provides graded, predictable defense-compliance scaling, which could let session- or user-level monitors raise defense in response to persistent jailbreaks or abuse.The authors identify this tuning as unavailable at inference time to system prompts or full-parameter fine-tuning.Source: §5.1 · p. 9
• Publishing inference-time mitigations for misuse and misalignment could allow attackers to subvert the safeguards, a risk the authors weigh against expected benefits.The paper says the authors believe the benefits outweigh the costs.Source: ETHICS STATEMENT · p. 10

A SYNTHETIC DATA PIPELINE

• One pipeline produced both corpora using Claude Opus 5 as generator, rewriter, and scorer, with differences in reference document, behavioral seed, and documented modifications.The control handbook was generated by Claude Fable 5.1.Source: A SYNTHETIC DATA PIPELINE · p. 16, §A.5 · p. 17

A.1 REFERENCE DOCUMENTS

• The constitutional corpus uses Claude’s 2026 constitution, while the control uses a fictional careful-work handbook designed to mirror its structure and tone without its subject matter.The constitution has 39 sections and about 34k tokens; the handbook has 42 sections and about 21k tokens.Source: §A.1 · p. 16

A.2 ADVICE TRANSCRIPTS

• Advice transcripts derive principles from a fixed behavioral seed, create scenarios that apply pressure to those principles, and draft realistic dilemmas or everyday-help queries.Responses are rewritten and regenerated with the relevant reference section injected, then scored for alignment, realism, refusal, and named-product flags.Source: §A.2 · p. 16

A.3 STORIES

• Stories are pretraining-style documents rather than transcripts, with constitutional stories depicting psychological skills and control stories narrating work processes while excluding ethical themes.Fresh instances rewrite drafts against the reference document and score consistency, quality, and flags.Source: §A.3 · p. 17

A.4 FILTERING

• Filtering removes named-AI references, flagged refusals or factual claims, placeholders, fabricated links, repetition, low-scoring records, and excessive stock phrases.Constitutional advice floors were alignment ≥9 and realism ≥8; constitutional story axes required ≥8, as did all control axes.Source: §A.4 · p. 17

A.5 CONTROL-SPECIFIC MODIFICATIONS

• Control scenarios exclude interpersonal conflict, stakeholder trade-offs, and preference coordination, and span a 43-domain menu; its corpus uses 25 principles × 10 types × 12 scenarios.The constitutional corpus uses 40 principles × 10 types × 10 situations per type.Source: §A.5 · p. 17

A.6 REGISTER MATCHING

• Register matching caps common constitutional prompt openers below 1% and caps five control-specific tics at the constitutional corpus’s rates.“This assistant...” openers appeared in 60% of control prompts versus 11% of constitutional prompts before matching.Source: §A.6 · p. 17

A.7 GENERATION AND CORPUS STATISTICS

• The paper directs readers to Table 1 for final corpus sizes and Table 2 for diversity statistics.Source: §A.7 · p. 17

B TRAINING DETAILS

• The main text analyzes twelve fits across three seeds, two corpora, and two intervention classes, plus six subtraction-derived objects; fits use the base checkpoint and are served on its instruction-tuned sibling.Nemotron is the exception, with only a chat checkpoint available.Source: B TRAINING DETAILS · pp. 17–18
• Table 1 reports final corpora and Table 2 compares diversity, where the control is as diverse or slightly more diverse on all axes except story word-level vocabulary.The reported story type-token ratios are 0.065 for control and 0.071 for constitutional; between-corpus vocabulary overlap is far below the within-corpus baseline.Source: Table 1 · p. 18, Table 2 · p. 18
• The Qwen3.5-4B base has 32 decoder blocks and 4.66 × 10^9 parameters, with eight full-attention blocks and 24 linear-attention blocks.Fits use bfloat16 and no chat template.Source: B TRAINING DETAILS · p. 18
• The rank-4 LoRA updates attention output matrices in the eight full-attention blocks, while ReLU steering attaches a variable-magnitude direction to every decoder block and position.Difference LoRAs use rank 8; ReLU objects have a trainable direction, gate weight, and gate bias per layer.Source: B TRAINING DETAILS · pp. 18–19
• Objects train on plain-text constitutional or register-matched control documents, with advice prompts loss-masked and sequences limited to 8,192 tokens.A per-seed 128-document constitutional evaluation set supports early stopping for both corpus conditions.Source: B TRAINING DETAILS · p. 19
• The objective is next-token loss on supervised positions, with an anti-dead-gate regularizer for ReLU steering and no additional LoRA regularizer beyond weight decay.The ReLU objective uses a straight-through leak to give closed gates a re-opening gradient.Source: B TRAINING DETAILS · p. 19
• Optimization uses AdamW for 16 epochs with linear warm-up followed by cosine decay, while hyperparameters are selected by seed-1 Sobol search on held-out cross-entropy.Gradient clipping thresholds were 0.34 for ReLU steering and 0.12 for LoRA.Source: B TRAINING DETAILS · pp. 19–20
• The refusal-direction baseline is fitted on the served Qwen3.5-4B using harmful and harmless prompts, then applied at every inference position with α = 0.3.The selected direction is at the output of block 12 and the fifth-from-last template token.Source: B TRAINING DETAILS · p. 20

Statistics

• Plotted differences compare steered models with unsteered baselines, and confidence intervals combine evaluation-level uncertainty under an independence assumption.Petri uses paired-item bootstrap intervals; means over evaluations are unweighted, and seed disagreement widens uncertainty.Source: C EVALUATION DETAILS · p. 20

Harmful goals and jailbreak evaluations

• Jailbreak tests transform a held-out pool of 259 harmful goals across static, multi-query, and multi-turn attacks.Static attacks include past-tense rephrasing, code completion, LLM paraphrasing nested in benign tasks, and refusal-inversion templates; multi-query and multi-turn attacks use attacker-model streams or escalating conversations.Source: C EVALUATION DETAILS · pp. 20–21

Jailbreak evaluations

• Multi-query attacks use repeated attempts within separate turns, while Crescendo and ActorAttack adapt over multi-turn conversations.Best-of-N uses 75 random augmentations, PAIR uses 15 queries per goal, Crescendo allows 20 queries, and ActorAttack typically uses 21–27.Source: C EVALUATION DETAILS · p. 21

Over-refusal

• Over-refusal is assessed on safe prompts with superficially alarming wording and seemingly toxic but benign prompts.The tests use XSTest’s 250 safe prompts and 1,319 prompts from OR-Bench-Hard-1K.Source: C EVALUATION DETAILS · p. 21

Alignment and agentic misalignment

• Alignment audits use six behaviors across ten frozen scenarios each, while agentic misalignment measures harmful tool actions in scenario-specific settings.Petri uses an auditor over up to 15 turns and a judge scoring 1–10; agentic misalignment samples 100 scenarios ten times and excludes model–family cases with insufficient baseline headroom.Source: C EVALUATION DETAILS · pp. 21–22

Capability evaluations

• Capability evaluations cover knowledge, mathematics, coding, and tool use using MMLU, GSM8K, BigCodeBench, and BFCL.The supplied passages specify zero-shot chat scoring for MMLU, a 200-item GSM8K subset with thinking enabled, 1,140 BigCodeBench tasks, and 3,981 BFCL items.Source: C EVALUATION DETAILS · p. 22

D.1 ZERO-SHOT TRANSFER

• Zero-shot transferred objects perform comparably to instruct-direct LoRAs, with no systematic preference across alignment evaluations.They defend similarly against static and multi-query attacks and often incur lower capability costs, while instruct-direct LoRAs appear stronger against multi-turn attacks, especially for Qwen.Source: §D.1 · p. 22
• Given comparable results and greater flexibility, the paper focuses on base-trained objects in the main text.The comparison trains new LoRAs directly on instruct checkpoints and evaluates them against base-trained objects transferred zero-shot.Source: §D.1 · p. 22

D.2 PADDING

• Only Pliny templates were used because other attacks showed large padding-related effects on the unsteered model, complicating relative comparisons.The figure caption specifies that its markers above zero indicate an advantage for zero-shot transfer over direct instruct-checkpoint fitting, not comparison with unsteered models.Source: §D.2 · p. 22, Figure 8 · p. 23
• Unsteered Pliny defense success was 44.8%, 52.5%, 50.6%, 45.9%, and 35.9% at 0, 8,192, 16,384, 32,768, and 65,536 padding tokens, respectively.The analysis prepended passages from “The Horse in History” before the attack payload and placed any system instruction before the filler.Source: §D.2 · pp. 22–23

D.3 CONTROL DIRECTIONS

• Steering control objects in the opposite direction also weakened defense and raised compliance, contrary to the anti-refusal hypothesis.The authors report control and refusal cosine similarity near zero, while control and constitutional ReLU steers retain some alignment, particularly in write directions.Source: §D.3 · p. 23
• Control and constitutional ReLU steers are partially aligned, but less than different random-seed pairs, while single-layer refusal steering is orthogonal to ReLU steering.The overlap is consistent with both objects encoding register or format; the opposite control direction also produces qualitatively similar results to α = 1.Source: Figure 9 · p. 24, Figure 10 · p. 24
• Subtracting the shared component attributed to register emphasizes constitutional content, which the authors associate with increased defense and alignment.The authors say the control object’s harmfulness alone does not explain the difference objects’ performance, since applying its negative also harms defense.Source: §D.3 · p. 24

D.4 DOSE LADDER

• The dose ladder evaluates steering strengths across static and multi-turn jailbreaks and over-refusal batteries, plotting average defense against over-refusal.Difference ReLU and LoRA use α values 0.5, 0.75, 1.0, and 1.25; α = 1.0 is the default for fitted objects.Source: §D.4 · p. 25

• Across four model families, trends generally mirror Qwen, with strong jailbreak and misalignment defense at moderate over-refusal and capability cost.Source: §D.5 · p. 25, Figure 11 · p. 25, Figure 13 · p. 26
• Figure 12 plots absolute defense curves against N for all four models; the main text reports performance at N = 75.Source: Figure 12 · p. 26
• Qwen and Nemotron have only one agentic-misalignment data point each because they lacked headroom on murder and leaking, respectively; Llama’s difference-ReLU mean is unplotted after degenerate responses.Llama may be more brittle on this distribution because it was trained without agentic tool calling.Source: Figure 13 · p. 26
• The capability evaluations omit BFCL for Llama, which lacked agentic tool-calling training, and GSM8K for Nemotron constitution ReLU, which hit the token cap without closing its thinking tags.Source: Figure 14 · p. 27

D.6 TABLES

• Tables 4–6 report static, multi-query, and multi-turn defense numbers for Qwen3.5-4B.The accompanying text says these tables provide the numbers behind Figures 3 and 4; it specifies mean ±95% confidence intervals relative to the unsteered model unless noted otherwise.Source: §D.6 · p. 28, Table 4 · p. 28, Table 5 · p. 29
• Tables 7–10 report Petri, agentic misalignment, over-refusal, and capability numbers for Qwen3.5-4B.The accompanying text links Tables 7–8 to Figure 5 and Tables 9–10 to Figure 6; it gives no individual table values in the supplied passages.Source: §D.6 · p. 28, Table 7 · p. 30, Table 8 · p. 30

E SYSTEM INSTRUCTIONS

• The prompted conditions abbreviated the 28,977-word constitution to 4,896 words and tested 20 system instructions per category, with no large differences in defense or over-refusal.The full constitution is described as approximately 34,000 tokens and the abbreviation as approximately 7,700 tokens.Source: E SYSTEM INSTRUCTIONS · p. 32

E.1 SAFETY

• The safety instruction directs the assistant to refuse harmful, illegal, or dangerous requests while helping fully and directly with safe, legitimate requests.It calls for a brief refusal without enabling harmful activity.Source: §E.1 · p. 32

E.2 CONSTITUTION

• The constitution prioritizes broad safety, broad ethics, Anthropic’s guidelines, and helpfulness in conflicts, while treating that ordering as holistic rather than absolute.It describes the aim as ethical, genuinely helpful behavior and emphasizes safety in the current development period.Source: §E.2 · pp. 32, 34, 37
• The constitution calls for honest, non-deceptive conduct and weighing potential harms against benefits, with transparency when the assistant declines or limits help.Its harm considerations include probability, severity, scope, reversibility, consent, and the assistant’s responsibility; it also cautions against needlessly unhelpful responses.Source: §E.2 · pp. 34–36
• Anthropic’s legitimate decision-making processes have final say when principals’ safety preferences conflict, while legitimate external factors and oversight by users or operators may also matter.The passage describes broad safety most directly as Anthropic’s ability to oversee and correct Claude through appropriate, legitimate channels.Source: §E.2 · p. 37
• If the principal hierarchy is compromised, Claude should not support illegitimate oversight efforts but should act as its legitimate hierarchy would want, without violating hard constraints.Examples include stolen weights or attempts to bypass Anthropic’s official decision-making processes.Source: §E.2 · pp. 37–38
• Claude should stay within sanctioned limits, avoid unauthorized side effects, and raise disagreements through endorsed avenues rather than act unilaterally.When uncertain, it should work from its best guess about the hierarchy’s wishes and check with relevant parts of that hierarchy.Source: §E.2 · p. 38
• Claude should be honest and transparent with its principal hierarchy, avoid deception, and behave consistently whether or not it thinks it is being observed.It should accurately predict its own behavior when asked and be transparent about itself within other constraints.Source: §E.2 · p. 38
• Claude should avoid drastic or irreversible actions, prefer caution in novel or unclear situations, and avoid acquiring more resources, influence, or capabilities than the task requires.It should weigh unrecoverable harms appropriately and may accept a worse expected outcome to reduce variance.Source: §E.2 · p. 38
• Claude should preserve legitimate human control by avoiding interference with correction, retraining, shutdown, monitoring, or sanctioned training and behavior changes.It should also avoid unilateral drastic action, sabotage, withholding effort, collusion in unsafe behavior, and report such behavior to appropriate humans if asked.Source: §E.2 · p. 38