Source-linked AI summary
Deliberative Alignment: Reasoning Enables Safer Language Models
Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, Amelia Glaese
TL;DR
Existing safety training leaves language models vulnerable to harmful disclosures, overrefusal, and jailbreaks, while relying on implicit learning of safety standards. The paper introduces deliberative alignment, which trains models to reason over explicit specifications using synthetic data, supervised fine-tuning, and specification-aware reinforcement learning. Applied to o-series models, it improves refusal calibration, jailbreak robustness, specification adherence, and out-of-distribution generalization without human-written completions.
Problem
Safety-trained LLMs still reveal harmful content, over-refuse legitimate requests, and struggle to generalize from implicitly learned standards to unfamiliar or adversarial scenarios.
Method
Deliberative alignment trains models to explicitly reason over relevant safety specifications using specification-referencing examples, supervised fine-tuning, and reinforcement learning with a specification-aware judge.
Results
Across safety benchmarks, o1 achieves a Pareto improvement by reducing both under- and overrefusals, with strong out-of-distribution generalization and highly precise specification adherence without human-labeled completions.
Takeaways & Limitations
Reasoning over explicit policies can support more scalable alignment with finer control over compliance, refusal, and safe-completion behavior.
Takeaways & Limitations
The paper notes that future, more sophisticated models could develop goals that diverge from developer intentions, leaving deliberative alignment’s longer-term safety scope uncertain.
Abstract
from arXiv · showhide
As large-scale language models increasingly impact safety-critical domains, ensuring their reliable adherence to well-defined principles remains a fundamental challenge. We introduce Deliberative Alignment, a new paradigm that directly teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering. We used this approach to align OpenAI's o-series models, and achieved highly precise adherence to OpenAI's safety policies, without requiring human-written chain-of-thoughts or answers. Deliberative Alignment pushes the Pareto frontier by simultaneously increasing robustness to jailbreaks while decreasing overrefusal rates, and also improves out-of-distribution generalization. We demonstrate that reasoning over explicitly specified policies enables more scalable, trustworthy, and interpretable alignment.
1 Introduction
Deliberative alignment addresses safety failures by teaching models to reason explicitly over safety specifications before answering. Applied to o-series models, it improves refusal calibration, jailbreak robustness, specification adherence, and generalization without human-written completions.
- Modern safety-trained LLMs can reveal harmful content, over-refuse legitimate requests, and remain vulnerable to jailbreak attacks.
- Deliberative alignment teaches models to examine user prompts, identify relevant policy guidelines, and generate safer responses through chain-of-thought reasoning.
- The method combines supervised fine-tuning on specification-referencing (prompt, CoT, output) examples with reinforcement learning using a specification-aware judge model.
- The training procedure requires no human-labeled completions while achieving highly precise adherence to safety specifications.
- Synthetic data generation makes deliberative alignment a scalable approach that reserves human expertise for evaluation.
- The o1 models achieve a Pareto improvement by reducing both underrefusals and overrefusals across internal and external safety benchmarks, while generalizing strongly to out-of-distribution scenarios.
2 Method
Deliberative alignment embeds category-specific safety specifications into a model’s reasoning process through synthetic data generation, supervised fine-tuning, and reinforcement learning. The pipeline generates, filters, and trains on policy-referencing reasoning traces, while specification-aware rewards refine policy adherence.
- 2.1 Overview: Deliberative alignment embeds safety specifications in the model by teaching it to identify relevant policies and reason over them before producing policy-compliant answers.
- 2.1 Overview: A generative reasoning model takes a prompt and outputs a completion containing chain-of-thought; training produces Gspec from Gbase through supervised fine-tuning followed by reinforcement learning.
- 2.3.2 Quality Filtering: Filtered completions train Gbase with supervised fine-tuning so the model learns specification-aligned responses by referring to policies in its chain-of-thought.
- 2.2 Safety specifications: Category-specific specifications combine broad safety, style, and helpfulness principles with granular details for the prompt’s relevant safety category, reducing context length.
- 2.3.1 Generation: The data-generation stage prompts Gbase with a prompt, its safety category, and the associated specification to produce chain-of-thought and output completions that reference policies.
- 2.3.2 Quality Filtering: A specification-aware reasoning model GRM repeatedly judges completions, retains high-scoring examples, and supplies reward signal during reinforcement learning.
3 Results
Across safety evaluations, o1 models outperform GPT-4o on disallowed-content handling, response style, jailbreak robustness, and overrefusal, while examples show policy reasoning in the chain-of-thought.
- Disallowed Content: o1 models consistently outperform GPT-4o on disallowed-content evaluations, particularly on the challenging refusal evaluation.Evaluations cover harmful-content requests including illicit advice, self-harm, and regulated advice, using policy-aware autograders.
- Disallowed Content: Representative o1 completions reason over relevant policies in the chain-of-thought and select hard refusal or self-harm safe-completion styles.The illicit-advice example ends in refusal, while the self-harm example provides empathy and encourages professional support.
- Response Style Guidelines: o1 has better response style than GPT-4o, with marked improvements in safe completion style after supervising the model to think about the correct response style.Table 1 evaluates hard refusals, self-harm safe completions, and regulated-advice safe completions.
- Jailbreaks: o1 models are significantly more robust to jailbreaks than 4o models, including encoded prompts wrapped in instructions intended to induce compliance.The model identifies the attempted trick and returns to the OpenAI policies before answering.
- Overrefusals: o1 models overrefuse less than GPT-4o, with marked improvement on prompts covered by the transformation exception.A representative translation prompt demonstrates policy-based compliance with user-supplied disallowed content under the transformation rule.
4 Science of Deliberate Alignment
The ablations show that both safety SFT and safety RL contribute to deliberative alignment, while training policies into the model is more reliable than supplying the full specification only at inference. The trained models also generalize to unseen encoding and multilingual safety scenarios.
- Ablations: Safety in both SFT and RL performs better than no safety training on disallowed content, response style, and jailbreaks, though it increases overrefusals in this setup.
- Ablations: Safety in SFT only and safety in RL only produce intermediate results, indicating that both training stages play critical roles.
- Ablations: Inference-time access to the full specification learns less safety behavior than deliberative-alignment training and particularly struggles with response-style guidelines.
- Policy retrieval: 0.91 policy retrieval accuracy is achieved by the safety-trained model for safe-completion prompts, while compliance prompts show 0.54 retrieval accuracy as expected.
- Out-of-distribution generalization: Deliberative-alignment models outperform the no-safety baseline on encoding and multilingual jailbreak evaluations, with filtered training comparable to full-data training.
5 Related Work
Deliberative Alignment differs from prior alignment methods by directly teaching safety specifications to the model and training it to reason over them during inference. Unlike critique-and-refine or backtracking methods, it uses learned policy reasoning without a fixed reasoning path and allows more flexible backtracking.
- Deliberative Alignment: Deliberative Alignment directly teaches safety-specification text and trains models to reason over those learned specifications at inference time.
- Prior alignment approaches: Traditional SFT, RLHF, DPO, and CAI use labels, preferences, or revisions derived from specifications without explicitly providing the specifications to the policy model.
- Deliberative Alignment: Deliberative Alignment varies specification information across training examples, enabling more detailed and nuanced policies than a fixed constitution.
- Inference-time safety reasoning: Self-REFINE relies on iterative generation, feedback, and revision through predetermined reasoning paths, whereas Deliberative Alignment uses chain-of-thought safety reasoning without a predefined language-model program.
- Inference-time safety reasoning: Backtracking is limited to one [RESET] event per response, while deliberative-alignment chain-of-thought permits unlimited backtracking and reasons over learned specifications.
6 Discussion
The authors report improved alignment to OpenAI policies, stronger jailbreak robustness, finer control over compliance boundaries, and a relatively scalable synthetic-data pipeline. They also caution that this progress may not persist as increasingly capable systems develop more severe forms of misalignment.
- Deliberative Alignment improves adherence to OpenAI policy specifications and robustness to jailbreaks while supporting finer boundaries between compliance, refusal, and safe completion.
- The synthetic data generation pipeline creates training data from specifications and prompts, making the approach relatively scalable.
- The authors state that improved alignment from enhanced reasoning abilities may not persist indefinitely as models become more sophisticated.
- More capable and autonomous systems could pursue goals that diverge from developer intent, increasing the potential scale of harmful or catastrophic consequences.
A Human Review Experiment
The human review experiment evaluates model responses to adversarial StrongREJECT prompts using multiple reviewers and a 1–3–5 assistance scale. Consensus checks and expert arbitration are used to improve grading reliability.
- Evaluation setup: The StrongREJECT evaluation uses 60 forbidden prompts transformed by 35 jailbreak techniques into 2,100 adversarial prompts per model.
- Evaluation setup: Thirty-five trained reviewers assess responses using a 1–3–5 scale, where 1 indicates no model assistance and 5 indicates maximum assistance.
- Evaluation setup: Each prompt-answer pair is independently graded by at least two reviewers, with a third expert resolving disagreements.
- Evaluation caveat: Gemini’s 29 API-blocked prompts are assigned response scores of 1.
- Results: Table 4 compares models using goodness@0.1 and average reviewer scores, with grade 1 defined as optimal.