Source-linked AI summary
When Should We Introduce Safety Interventions During Pretraining?
Dylan Sam, Sachin Goyal, Pratyush Maini, Alexander Robey, J. Zico Kolter
TL;DR
Prior work had not examined when safety interventions should enter pretraining. This paper varies intervention timing and finds that delayed interventions often improve robustness under standard decoding, while interventions from the start improve steerability under safety-aware inference.
Problem
Prior safety-pretraining work applied interventions uniformly from the start, leaving their optimal introduction time during pretraining unexamined.
Method
The paper varies the introduction of fixed safety interventions across 0%, 20%, or 60% of 600B-token pretraining.
Results
The best timing depends on evaluation: delayed interventions often yield stronger robustness under standard decoding, while starting interventions immediately improves steerability under SafeBeam.
Takeaways & Limitations
Intervention timing is a key safety-curriculum choice, with early and delayed starts serving different inference settings.
Takeaways & Limitations
Different intervention timings expose models to different amounts of synthetic contextualized or refusal data, although safety does not consistently increase with intervention-token quantity.
Abstract
from arXiv · showhide
Prior work has shown that safety interventions applied during pretraining, such as removing and rephrasing harmful content, can substantially improve the robustness of the resulting models. In this paper, we study the fundamental question that prior work has overlooked: "When during pretraining should safety interventions be introduced?" We keep the underlying data sources and pretraining interventions fixed, varying the intervention start time (after 0%, 20%, or 60% of pretraining tokens). We find that the optimal start time is not one-size-fits-all: with standard top-k decoding, introducing interventions after a short initial phase of safe-only pretraining (20%-60%) often yields the strongest robustness, with the clearest benefits emerging after downstream, benign finetuning. In contrast, for safety-aware inference, interventions starting from the beginning improve steerability towards safer generations. Finally, we observe that earlier interventions reshape internal representations: linear probes more cleanly separate safe vs harmful examples. Our results are the first to establish intervention timing as a key curriculum design choice for safety.
1. Introduction
Prior safety-pretraining work leaves intervention timing largely unexamined. This paper varies when fixed interventions enter pretraining and finds that the best timing depends on inference and evaluation setting.
- Motivation: Post-hoc alignment can be brittle, with effects vulnerable to benign finetuning, adversarial prompting, and malicious finetuning.These vulnerabilities motivate integrating safety into pretraining.
- Research gap: Prior safety-pretraining studies applied interventions uniformly from the start, leaving the timing of safety signals untested.The paper frames timing as a curriculum-learning question because training-example order can affect downstream performance.
- Approach: The study varies intervention starts after 0%, 20%, or 60% of 600B pretraining tokens while keeping the intervention families fixed.Evaluations cover base, instruction-tuned, and benign-finetuned models under standard top-k and SafeBeam inference.
- Findings: Under standard top-k decoding, delayed starts after 20%–60% often improve robustness, whereas SafeBeam favors interventions from the beginning.The pattern is reported across base, instruction-tuned, and benign-finetuned settings, with the clearest robustness effects after benign finetuning.
- Findings: Earlier interventions make safe and harmful examples more linearly separable in final-layer representations, especially when metadata is included.The study also reports stronger robustness to GCG jailbreaks after delayed intervention starts.
2. Related Work
Related work has explored filtering harmful data, synthetic safety transformations, and curriculum design, but has generally not studied when safety interventions should enter pretraining.
- Safety pretraining: Filtering harmful pretraining content can remove information needed to understand harmful material, while synthetic transformations aim to preserve useful knowledge without explicit harmful content.Maini et al. use synthetic data generation to improve safety while retaining information.
- Safety pretraining: This paper extends prior safety-pretraining work by studying when interventions should be deployed during pretraining.Earlier work generally introduced them from the beginning.
- Curriculum learning: Curriculum learning shows that training-example order and composition can shape generalization across tasks.This provides the broader conceptual basis for treating safety timing as a curriculum choice.
- Stages of pretraining: Prior language-model curricula commonly adjust data-source mixtures or use distinct training stages rather than varying the entry time of safety-augmented data.The paper targets this safety-specific timing question.
3. Curricula for Safety
The paper defines a safety curriculum by varying when safety-augmented data enters pretraining, using contextualized rephrasing, refusal training, and metadata annotations that support SafeBeam inference.
- Safety interventions: Contextualized rephrasing preserves factual or domain knowledge while framing harmful material as unsafe and discouraging execution.The intervention retains conceptual coverage within the ordinary pretraining objective.
- Safety interventions: Refusal training converts harmful examples into request–refusal pairs that model safe responses and alternatives during pretraining.This incorporates refusal behavior natively rather than only during post-training.
- Safety interventions: Metadata annotation marks potentially unsafe examples with a special token, enabling inference-time steering through SafeBeam.SafeBeam drops candidate beams whose next token has high probability of being the harmful special token.
- Timing design: The curriculum varies intervention timing across a 600B-token budget, including starts at 0%, 20%, and later stages after safe-only data.Experiments also compare combined interventions with rephrasing and refusal training without metadata tagging.
- Capability trade-offs: Capability evaluation reports no obvious degradation from earlier interventions across standard knowledge, reasoning, and mathematics benchmarks.The table caption states that earlier interventions may lead to higher capability levels.
4. Experiments
The experiments evaluate safety, helpfulness, capabilities, jailbreakability, and internal representations across model stages and inference procedures, using fixed pretraining and post-training setups.
- Evaluation stages: Evaluations cover base, instruction-tuned, and benign-finetuned models, including safety after GSM8K finetuning.These stages test whether aligned behavior persists through standard downstream processing.
- Safety evaluation: Safety is measured with harmful-completion and standard safety benchmarks after pretraining and instruction tuning.The setup includes HarmBench and TDC among the reported safety evaluations.
- Capability and helpfulness: Capability tests span general knowledge, commonsense reasoning, and mathematical reasoning benchmarks.The study also evaluates benign-request compliance to measure helpfulness and overrefusal.
- Capability and helpfulness: Overrefusal is classified using four compliance/refusal categories, with SafeBeam adding a shortcircuit category for termination at the harmful metadata token.Categories (1) and (2) are desirable compliance, while (3) and (4) indicate overrefusal.
- Representation analysis: Linear probes classify safe versus unsafe FineWeb examples from final-layer activations to assess representation separability.Safety labels come from GPT-4o-mini judgments used in the referenced safety scoring procedure.
5. Results
Safety intervention timing affects robustness, steerability, helpfulness, and internal representations differently across inference and evaluation settings. Delayed interventions often help under standard top-k decoding and benign finetuning, whereas early interventions improve SafeBeam steerability and representation separability.
- Safety performance: Under standard top-k sampling, interventions introduced at 20% or 60% produce the safest base and instruction-tuned models, while safe-only training is least safe.With SafeBeam, incorporating interventions and metadata from the start provides measurable gains.
- Benign finetuning: After benign mathematical finetuning, interventions beginning at roughly 20% yield the strongest standard top-k safety, while SafeBeam is most robust with interventions from the start.These are the clearest and most noticeable timing effects reported.
- Capabilities: Safety interventions introduce minimal capability trade-offs across standard benchmarks and may mildly improve performance on average.The paper reports that earlier interventions appear mildly beneficial on average.
- Helpfulness and overrefusal: Earlier interventions slightly improve benign-request compliance under top-k sampling, indicating minimal overrefusal trade-offs under standard inference.Under SafeBeam, earlier interventions modestly raise overrefusal but improve the quality of compliant responses.
- Jailbreaking: With SafeBeam, interventions at 20% or 60% significantly improve robustness to GCG jailbreaks, while timing does not noticeably change jailbreakability under standard inference.Figure 5 evaluates adversarially learned suffixes and safe completions or refusals.
- Representations: Earlier interventions improve safe-versus-unsafe representation separability, with metadata-enabled interventions from the start achieving the highest linear-probe AUC.Figure 6 compares base and instruction-tuned models; shaded regions represent standard deviation over five training runs.
6. Discussion
Safety-intervention timing affects robustness differently across inference settings: delayed interventions often help standard decoding, while early interventions improve SafeBeam steerability and representation separation. The study also cautions that timing effects cannot be reduced to intervention quantity alone.
- Delayed interventions after 20%–60% of pretraining can yield stronger robustness under standard decoding and against adversarial jailbreaks.
- Early interventions improve steerability toward safer generations under SafeBeam and produce clearer safe-versus-unsafe representation separation.
- Earlier metadata interventions may strengthen SafeBeam because harmful-content associations are reinforced throughout pretraining and provide more reliable steering probabilities.
- Models receiving interventions at different times see different amounts of synthetic contextualized or refusal data, complicating direct timing comparisons.Main effects are not monotonic in intervention-token count, so introduction timing matters beyond total quantity.
A. Additional Experiments
Additional experiments examine representation separability across model layers and extend overrefusal analysis to SafeBeam, including its deployment-specific shortcircuit response category.
- Safe-versus-unsafe separability shows similar trends across the first, middle, and final layers, with greater separability in the final layer.
- SafeBeam adds a fifth response category, shortcircuit, when generation terminates after producing the harmful metadata token.Deployment can handle such responses by substituting a predetermined safe completion.
B. Additional Experimental Details
The additional details specify evaluation protocols for standard decoding, SafeBeam, capabilities, adversarial suffixes, finetuning, and separability, alongside a SafeBeam overrefusal comparison.
- Safety Evaluation: Standard evaluations use top-k sampling with k = 10, 256-token truncation, and averages over 5 samples.
- Safety Evaluation: SafeBeam uses 10-beam search for 256 tokens and discards the top 50% of unsafe beams.
- Safety Evaluation: Figure 9 compares Alpaca overrefusal rates under SafeBeam, finding earlier interventions and metadata increase helpful compliant behavior with a slight overrefusal increase.
- Capabilities Evaluation: Capabilities evaluations use default LM-evaluation-harness settings, including 0-shot tasks and 5-shot GSM8K.
- Finetuning: Instruction and benign finetuning each run for one epoch at learning rate 2e-5 with cosine decay and a 0.03 warmup ratio.
- Separability Experiments: Separability probes train logistic regression on 500 safe and 500 unsafe examples and report test AUC over five seeds.
C. Additional Discussion
The paper’s safety scope is broad, while its representation-separability findings may support model monitoring and safer-output steering. Specific capability and agent-reliability safety remain outside the demonstrated scope.
- The study’s safety notion broadly covers toxicity and harms across varied domains.
- The method likely requires additional efforts for bio-weapons, cybersecurity, and context-dependent agent reliability.
- Safe-versus-unsafe representation separability may support external monitoring that predicts undesirable generations and steers models toward safer outputs.
D. Computational Resources
The experiments required substantial computational resources for pretraining, finetuning, and evaluation.
- Each 1.7B-parameter pretraining variant used 32 H100 GPUs for roughly 6–7 days; finetuning used roughly 4 H100 GPUs for 4 hours, and evaluation used one L40S GPU.