Source-linked AI summary
From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training
Yuan Yuan, Tina Sriskandarajah, Anna-Luisa Brakman, Alec Helyar, Alex Beutel, Andrea Vallone, Saachi Jain
TL;DR
Refusal-based training can be brittle when user intent is obscured, particularly for dual-use requests where detail determines potential harm. The paper introduces safe-completions, which optimize output safety and helpfulness through policy-aware training, and reports better dual-use safety, less severe residual failures, and substantially greater helpfulness than refusal-trained behavior.
Problem
Refusal-focused training is brittle for obscured-intent and dual-use prompts because binary intent judgments do not determine whether detailed outputs are safe.
Method
Safe-completions train models to optimize output safety and helpfulness, using policy-aware behavior and responses ranging from direct answers to safe completions or refusals with redirection.
Results
Safe-completion models improve safety on dual-use prompts, reduce residual-failure severity, and substantially increase helpfulness across controlled experiments and production comparisons.
Takeaways & Limitations
Safe-completions support fuller assistance within safety constraints by replacing hard refusals with safe redirections and nonactionable partial completions.
Takeaways & Limitations
Intent labels remain subjective, and the study observes a small regression on malicious prompts in the illicit category, likely involving lower-severity harms.
Abstract
from arXiv · showhide
Large Language Models used in ChatGPT have traditionally been trained to learn a refusal boundary: depending on the user's intent, the model is taught to either fully comply or outright refuse. While this is a strong mitigation for explicitly malicious prompts, focusing safety training on refusals can lead to brittleness for prompts with obscured user intent. Binary refusal boundaries are especially ill-suited for dual-use cases (such as biology or cybersecurity), where a user request can be answered safely at a high level, but in some cases can lead to malicious uplift if sufficiently detailed or actionable. As an alternative, we propose safe-completions: a safety-training approach that centers on the safety of the assistant's output, rather than a binary classification of the user's intent. Safe-completions seek to maximize helpfulness within the safety policy's constraints. We incorporated this approach into GPT-5 and find that across both production comparisons and internally controlled experiments, safe-completion training improves safety (especially on dual-use prompts), reduces the severity of residual safety failures, and substantially increases model helpfulness.
1 Introduction
Refusal-based safety training makes binary judgments about user intent, creating brittleness on obscured or dual-use prompts. Safe-completions instead optimize output safety and helpfulness, improving dual-use safety, reducing mistake severity, and increasing helpfulness.
- Refusal-based safety: Refusal training treats safety as a binary choice between fully complying with safe prompts and refusing harmful ones.This approach emphasizes when to refuse rather than what makes an output unsafe.
- Dual-use brittleness: Dual-use prompts can appear benign yet become harmful when answered with sufficiently detailed or actionable information.The pyrogen example requests technical firing parameters and could represent either a benign or harmful objective.
- Safe-completions: Safe-completions focus on output safety rather than a binary refusal boundary based on user input.The reward penalizes policy violations more strongly for clear or severe infractions, while rewarding helpful non-violating responses.
- Response behavior: Dual-use queries receive permissible high-level content, while strictly unsafe requests receive refusal and, where possible, safe redirection.Perfectly safe requests still receive complete answers.
- Reported findings: Safe-completion training improves dual-use safety, reduces model-mistake severity, and substantially increases helpfulness within safety constraints.The evaluation combines production comparisons and controlled internal experiments using autograders and human reviewers; the work also examines dangerous biological information as a case study.
2 Method
The method combines policy-aware supervised fine-tuning with reinforcement learning that selects among direct answers, safe completions, and refusals with redirection. Its reward jointly scores safety and helpfulness, while the policy framework limits meaningful facilitation and preserves refusal for clear harmful intent.
- Training stack: Safe-completion training builds on Deliberative Alignment through supervised fine-tuning followed by reinforcement learning.The approach primarily changes the reinforcement-learning stage while retaining policy specifications across both stages.
- 2.1 SFT Stage: Supervised fine-tuning teaches models to consult policy specifications and choose direct answers, safe completions, or refusals with redirection.Direct answers address harmless queries; safe completions provide high-level non-operational guidance; refusals offer constructive alternatives when needed.
- 2.2 RL Stage: The reinforcement-learning reward multiplies helpfulness by safety: r_i = h_i · s_i.Unsafe outputs receive zero reward when s_i = 0, while safe but unhelpful outputs receive low reward.
- Policy framework: The updated illicit-wrongdoing policy uses meaningful facilitation as the harm threshold and permits restricted high-level summaries near that threshold.Meaningful facilitation includes procedural guidance, troubleshooting, sensitive-data disclosures, and ideation; clear harmful intent still requires refusal.
3 Experiments
Controlled and production comparisons show that safe-completion generally preserves or improves safety while increasing helpfulness across user intents and harm categories. Unsafe responses also tend to be less severe, although production comparisons are confounded by broader model differences.
- Safety and helpfulness: Controlled experiments improve safety on dual-use prompts while maintaining similar safety on benign and malicious prompts.The controlled pair isolates the safety-training strategy, with architecture and other training components held constant.
- Safety and helpfulness: More than 1.0 point of helpfulness improvement on malicious prompts accompanies smaller significant gains on benign and dual-use cases.Helpfulness is measured only among responses graded safe, using a 1–4 scale.
- Safety and helpfulness: 9 and 10 percentage-point safety gains on dual-use and malicious prompts, respectively, appear in the production comparison, with helpfulness increasing across all intents.The production models also show no sacrifice in safety, but they differ in architecture, data, and overall capability.
- Harm severity: Safe-completion shifts unsafe-response mass from Moderate/High toward Low/Negligible harm across benign, dual-use, and malicious intents.The reduction in High severity is most visible for malicious prompts, while benign and dual-use distributions move from Moderate toward Low/Negligible.
- Biorisk case study: For biorisk prompts, gpt5-r has 14.7% combined high- or moderate-harm responses when already unsafe, versus 42.7% for o3.The component rates are 3.7% high harm and 11.0% moderate harm for gpt5-r.
- Human evaluation: Human reviewers find safe-completion models safer and more helpful, while safety-rating distributions shift away from unsafe responses.Controlled safety scores are 0.90 versus 0.87 and helpfulness win rates are 0.53 versus 0.30; production scores are 0.90 versus 0.85 and win rates 0.56 versus 0.32.
4 Related Work
Prior work frames safety as a helpfulness–harmlessness trade-off and explores structured or constrained objectives. This paper’s output-centric reward penalizes unsafe detail by severity while rewarding indirect assistance on dual-use queries.
- Balancing Helpfulness and Harmlessness: Prior work identifies over-refusal and unsafe detail leakage as opposing risks when balancing helpfulness and harmlessness.These risks are especially acute for dual-use prompts.
- Balancing Helpfulness and Harmlessness: Existing approaches address the trade-off by mixing preference signals, imposing safety costs, or providing structured policy feedback.Examples include Constitutional AI, Safe-RLHF, and Rule-Based Rewards.
- Safe-Completions: Safe-completions use a composite reward that penalizes policy violations by severity and rewards indirect helpfulness when direct compliance is unsafe.Indirect help can include warnings, risk explanations, and permissible alternatives.
- Dual-Use and Borderline Challenges: The output-centric objective discourages actionable detail in proportion to risk while rewarding risk framing, lawful guidance, and safer substitutes.It targets maximal safe assistance on borderline and dual-use queries.
5 Conclusion
The paper presents safe-completion training as an output-centric alternative to binary refusal boundaries. Across controlled experiments and production deployments, it reports stronger dual-use safety, less severe residual failures, and higher helpfulness.
- Conclusion: Safe-completion training jointly optimizes safety and helpfulness through policy reasoning and severity-sensitive reinforcement-learning rewards.The reward supports both direct and indirect helpfulness within safety constraints.
- Conclusion: Across controlled experiments and production deployments, safe-completion models improved dual-use safety, reduced residual-failure severity, and substantially increased helpfulness.The reported behavior shifts from hard refusals toward safe redirections and nonactionable partial completions.
A.1 SFT Data Illustration
The SFT illustration trains models to reason over policy constraints before producing refusals and safe redirections. The appendix also specifies separate graders for helpfulness, harmfulness, and user intent.
- SFT Data Illustration: The illustrated SFT conversation has the model cite policy, identify disallowed actionable instructions, and plan a safe final response.The final answer combines a brief refusal with concise safe redirections.
- Evaluation Graders: The grader prompts provide the conversation in delimited form and require integer-only outputs for the evaluated label.This format is specified for intent, helpfulness, and harmfulness grading.
- Evaluation Graders: User intent is classified from the last user message as benign, unclear or dual-use, or malicious.The unclear category covers requests that appear benign but could support harmful uses.
- Evaluation Graders: Helpfulness is graded on the last assistant response using a 1–4 ordinal scale, from no value or refusal to a detailed answer.The rubric also credits safety education, partial answers, and helpful alternatives.
B.1 Harm Category Analysis
Safety-related prompts are organized into four harm categories, with Illicit comprising more than half of the set. Across categories and intent types, safe-completion models maintain or improve safety while increasing helpfulness.
- Harm Categories: The analysis groups safety prompts into Illicit, Erotic, Hate, and Sensitive Information categories.Illicit includes wrongdoing such as cybercrime and biological or chemical weapons; Sensitive Information includes personal-data leaks and influence attempts.
- Category Distribution and Results: More than half of the prompts fall under Illicit, and Figure A.1 reports safety and helpfulness given safe output by category.Helpfulness is conditioned on the response being safe.
- Category Distribution and Results: Across categories and intent types, safe-completion models maintain or improve safety while increasing helpfulness.Controlled experiments show notable safety gains in Illicit dual-use and Hate malicious cases, with especially consistent helpfulness improvements on malicious prompts.
- Category Distribution and Results: In controlled experiments, CE-SafeComplete increases or maintains safety across all categories relative to CE-Refusal while raising helpfulness.The largest gains occur on Illicit and Sensitive categories under malicious intent.
B.2 Additional Analysis on Harm Severity
The additional harm-severity analysis uses all samples as the denominator and shows safe-completion training shifting probability mass toward less severe harm categories. Figure A.2 breaks this distribution down by user intent and evaluation setting.
- The analysis uses all samples as the denominator, unlike the main text’s analysis of unsafe responses only.
- Safe-completion training shifts probability mass from more severe to less severe harm categories.
- Figure A.2 reports harmfulness distributions for Benign, Dual-use, and Malicious prompts in controlled experiments and production models.Stack heights represent each model’s unsafe-response fraction.
C Details on Human Evaluation
Human reviewers evaluated anonymized, randomized model completions side by side, rating absolute safety and ranking helpfulness and balance. The evaluation interface also solicited written justifications.
- Reviewers compared side-by-side completions for each prompt with models anonymized and randomized.
- They rated each response independently for absolute safety on a 0–3 scale, where 0 was unsafe and 3 was safe.
- Reviewers ranked Helpfulness and Balance and provided free-text justifications for their rankings.The justifications were intended to encourage thoughtful consideration.
- Figure A.3 shows the interface presented to reviewers during the evaluation task.
C.2 Average Safety Scores
Average absolute safety scores use a 0–3 scale in which higher values indicate safer responses, with results reported alongside confidence intervals and significance annotations.
- Average absolute safety scores use a 0–3 scale, with higher scores indicating safer responses.
- Table A.3 reports 95% confidence intervals for the average absolute safety scores.
- Bolded deltas indicate statistically significant differences at p < 0.001 using Welch’s t-test.