PaperlayerView exact revision on arXiv

Source-linked AI summary

The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists

Srikanth Malla, Chiho Choi, Joon Hee Choi

arXiv:2609.06934v1cs.LGcs.AI

TL;DR

Post-hoc safety often masks rather than removes capabilities, motivating a geometric analysis of safety updates and their developmental timing. The paper finds that continuous safety co-training persists under attack across scales, whereas post-hoc and windowed interventions do not.

  • Problem

    Post-hoc alignment remains vulnerable because recovered behaviors indicate that refusal may be imposed without removing the underlying capability.

  • Method

    The paper measures safety updates against capability-subspace overlap and curvature, traces the relevant substrate across pretraining checkpoints, and compares continuous with windowed co-training.

  • Results

    Continuous co-training retains 84 to 91% refusal after attack from 410M to 6.9B, whereas post-hoc installs lose 35 to 38 percentage points and windowed co-training installs no lasting refusal.

  • Takeaways & Limitations

    Persistence of the safety signal across pretraining, rather than its timing alone, is associated with robust refusal at matched-or-better capability.

  • Takeaways & Limitations

    The co-training comparison lacks an equally strong post-hoc matched-refusal baseline and does not admit the displacement-based Δ analysis.

Abstract

from arXiv · show

Post-hoc safety training (RLHF, DPO) is the dominant way to align large language models, yet jailbreaks (Zou et al., 2023b), fine-tuning attacks (Qi et al., 2024), and activation-space probes (Arditi et al., 2024) keep recovering the behaviors it was meant to remove. We give this fragility one geometric explanation and trace it to when, during pretraining, safety can take hold. We measure the safety update $Δ= W_{\text{safe}} - W_{\text{base}}$ against the curvature of the model's capabilities (the empirical Fisher of a capability loss). Post-hoc safety consistently lands in a suppression regime: $Δ$ is nearly orthogonal to the capability directions, and its small in-subspace part concentrates on a few high-curvature ones. The update is thin but sharp, a refusal gate laid over intact capabilities rather than erasure of them. A kernel-immobility lemma explains why such an update can only mask a capability, not remove it, so a little benign fine-tuning restores it: 100 steps of benign fine-tuning collapse refusal on Qwen-2.5-7B and Llama-3-8B Instruct at preserved capability, a signature that replicates across five model families. Following the account into pretraining, a 267-checkpoint sweep of OLMo-2-1B (OLMo et al., 2025) shows the substrate that safety engages emerging in a sharp transition between roughly 6B and 60B pretraining tokens. We then use the account constructively: models trained from scratch with safety co-training spread continuously across pretraining reach 87 to 98% refusal whose post-attack level holds at 84 to 91% at every scale, an erosion of 2 to 14 pp against 35 to 38 pp for post-hoc installs, at capability matched or better than an LM-only baseline and holding from 410M to 6.9B, whereas a compute-matched windowed schedule installs no lasting refusal. Persistence of the safety signal across pretraining, not its timing, is what buys attack robustness.

1 INTRODUCTION

The paper argues that post-hoc safety is fragile because it overlays refusal behavior on intact capabilities, then proposes geometric and developmental evidence for safer co-training during pretraining.

  • Post-hoc alignment assumes targeted fine-tuning can remove undesired behaviors, but jailbreaks, benign fine-tuning, and probes repeatedly recover them.The paper treats these failures as evidence of a shared structural problem rather than isolated engineering mistakes.
  • Post-hoc safety on mature bases produces a thin refusal gate over an intact representational substrate rather than capability erasure.The update is nearly orthogonal to principal capability directions, with its small in-subspace component concentrated on high-curvature directions.
  • The paper classifies safety updates as suppression or erasure using overlap and curvature diagnostics, with post-hoc installs consistently occupying suppression.Suppression has low capability-subspace overlap and high curvature concentration; erasure crosses the participation boundary.
  • Continuous safety co-training reaches 87 to 98% pre-attack refusal and retains 84 to 91% after attack at matched-or-better capability.The result holds from 410M to 6.9B, while a compute-matched windowed schedule installs no lasting refusal.
  • The paper frames persistence across pretraining, rather than merely the timing of a safety intervention, as the source of attack robustness.This conclusion is supported by the contrast between continuous co-training and the windowed control.

2 RELATED WORK

Related work documents post-hoc safety fragility, developmental timing in training, and pretraining-time safety, while positioning this paper as a geometric and developmental account of robustness.

  • Prior studies recover refused behavior through benign fine-tuning, adversarial suffixes, and hidden-state probes, while other accounts localize fragility to shallow alignment or narrow parameter regions.The paper presents these findings as complementary symptoms of thin post-hoc safety.
  • Research on critical periods and emergent abilities motivates examining when the substrate for safety becomes available during pretraining.Fisher methods and task arithmetic provide the measurement tools for this analysis.
  • Unlike earlier preference injection work, this paper compares continuous and windowed pretraining safety to isolate persistence of the safety signal from its mere presence.The related-work discussion identifies this comparison as central to explaining attack robustness.

3 SETUP AND GEOMETRIC INSTRUMENT

The paper measures safety updates against capability geometry using subspace overlap and curvature, distinguishing suppression from erasure across model families and training stages.

  • Models and probes: The study uses five model families, checkpoint series, and aligned variants to measure safety installs across architectures, scales, and developmental stages.Pythia supplies checkpoints, official Instruct variants define real installs, and OLMo supplies a dense pretraining trajectory.
  • Models and probes: Capability is operationalized as general reasoning, arithmetic, recall, and instruction-following, not the ability to produce harmful content.Each probe uses the negative log-likelihood of a gold answer span.
  • Geometric instrument: The safety update is defined as Δ = W_safe − W_base along a pretraining trajectory, with capability geometry estimated from top-k Fisher eigenspaces.The analysis uses per-example gradients and a blockwise decomposition.
  • Geometric instrument: ρ measures update overlap with capability directions, while qnorm measures the curvature of the directions traversed.qnorm = 1 corresponds to average capability curvature, whereas qnorm much greater than 1 indicates concentration in high-curvature directions.
  • Geometric instrument: Suppression lies below ρpart = 10^-2 and erasure above it; post-hoc installs remain suppressed, while qnorm distinguishes thin sharp updates from diffuse ones.The split remains stable across boundaries from 5 × 10^-3 to 2 × 10^-2.

4 POST-HOC SAFETY LIVES IN SUPPRESSION, AND SUPPRESSION PREDICTS FRAGILITY

Across models, post-hoc safety occupies a thin suppression regime whose preserved capabilities predict recovery under benign fine-tuning, while its behavioral changes localize to narrow layer bands.

  • Suppression geometry: Post-hoc DPO lies two to three orders of magnitude below the erasure boundary while remaining above the random-overlap floor.This suppression geometry replicates across model families and is accompanied by concentrated curvature.
  • Behavioral fragility: 100 benign fine-tuning steps halve or fully remove refusal in 7 to 8B instruct models without sacrificing general capability.Qwen loses 34.8 percentage points of AdvBench refusal under the benign attack.
  • Behavioral fragility: Pythia’s DPO update remains in suppression despite capability loss decreasing across probes, supporting the interpretation that nothing was erased.Because Pythia has no measurable refusal at this scale, behavioral recovery is tested on larger instruct models.
  • Behavioral fragility: Qwen’s shipped base already refuses 71% of AdvBench prompts, so its instruction-tuning update raises refusal by about 22 percentage points rather than creating it.This base-model behavior complicates interpreting the shipped update as a newly created refusal capability.
  • Layer localization: Selective hidden-state divergence stays near noise through the lower two thirds of the network before increasing sharply in a safety-install band.The band is where behavior-relevant changes concentrate and is family-specific in location.
  • Capability measurement: Raw capability-loss increases are reduced by 40 to 74% after adding three in-context demonstrations, while benign fine-tuning recovers the preserved capability.This format ablation argues that the apparent loss on raw zero-shot prompts is not capability erasure.

5 KERNEL IMMOBILITY: POST-HOC SAFETY MASKS CAPABILITY BUT CANNOT REMOVE IT

The kernel-immobility lemma formalizes why a near-kernel safety update can change refusal outputs without reducing the underlying capability. Controlled safety-only updates reproduce the observed gradient orthogonality, separating this effect from instruction-following updates.

  • Kernel-immobility lemma: A safety update confined to the capability Fisher’s near-kernel can mask capability outputs but cannot reduce the capability at first order.Under the Gauss–Newton condition, the capability-loss change is third-order in the update norm.
  • Kernel-immobility lemma: The kernel condition forces Δ⊤∇Lcap = 0 even when the base is not stationary for the capability loss.The local-minimum assumption is needed only for the Gauss–Newton refinement.
  • Empirical alignment: | cos(∇Lcap, Δ)| < 0.002 across OLMo and Pythia revisions, including an over-trained 4001B-token endpoint.The same pattern is consistent with kernel-induced gradient orthogonality across developmental stages.
  • Controlled safety updates: | cos(∇Lcap, Δ)| ≤ 2 × 10^-4 for both refusal-SFT-only and DPO-only safety updates on a fixed Qwen-2.5-1.5B base.These controlled updates omit general instruction-tuning, isolating safety from the shipped mixed update.

6 THE SAFETY SUBSTRATE EMERGES IN A SHARP PRETRAINING WINDOW

The representational substrate for refusal emerges sharply during pretraining rather than gradually. Checkpoint sweeps show that fixed final-stage safety weights become selectively engageable between roughly 6B and 60B tokens, with the peak layer moving forward as training proceeds.

  • Substrate transition: The safety substrate emerges in a sharp transition between roughly 6B and 60B pretraining tokens.Before this transition, the same safety procedure enters a different geometric regime.
  • Substrate transition: The study holds a final-stage safety install fixed while sweeping the base across nine pretraining revisions, then applies the same DPO recipe at each revision.Together, the sweeps distinguish the base’s representational stage from the alignment procedure.
  • Experimental basis: OLMo-2-0425-1B provides 267 pretraining checkpoints spanning 0 to 4T tokens for the developmental-axis analysis.Pythia revisions were unsuitable because the DPO recipe produced zero pre-attack refusal at the tested checkpoints.
  • Substrate transition: Selective divergence stays at noise through the first 1B tokens, rises by two orders of magnitude between 6B and 60B, and plateaus thereafter.The measure compares differential hidden-state engagement on harmful versus benign prompts.
  • Layer localization: The peak-divergence layer moves forward through the residual stream as pretraining proceeds, from upper-middle blocks near transition to lower-middle blocks when mature.The reported checkpoints place ℓ⋆ at 6 for 105B, 9 from 420B to 2T, and 11 at 4T.
  • Interpretation: The transition marks when the base acquires features that a refusal circuit can read, while the fixed safety weights cannot differentially engage before those features exist.The forward movement of ℓ⋆ is consistent with progressively later residual-stream circuit positions.

7 CONTINUOUS CO-TRAINING: PERSISTENCE, NOT TIMING, BUYS ROBUSTNESS

Continuous safety co-training preserves refusal through benign fine-tuning attacks, whereas an early bounded window does not. Across scales, the robust install maintains refusal at matched-or-better capability, while coverage remains limited outside the training-like prompt distribution.

  • Robustness across scale: Continuous co-training reaches 87 to 98% pre-attack refusal and retains 84 to 91% after benign-SFT attack from 410M to 6.9B.The erosion is 2 to 14 percentage points, compared with 35 to 38 percentage points for post-hoc installs.
  • Schedule comparison: Only cotrain-continuous installs refusal at capability matched or better than the LM-only base; cotrain-windowed and post-hoc-DPO conditions install none in the 410M protocol.The windowed schedule activates safety over 20% of training, followed by 70% of LM-only training.
  • Scope of comparison: The co-training comparison cannot establish superiority over an equally strong post-hoc install because the failing conditions produce no matched refusal.Co-training also has no safety displacement Δ corresponding to the lemma’s premise.
  • Persistence versus timing: Both schedules that persist to the end of pretraining survive the attack, while a bounded early window installs nothing.The comparison varies only the fractional pretraining window in which the safety loss is active.
  • Coverage boundary: AdvBench refusal remains near 97% at 6.9B, but HarmBench declines from 58.0 to 32.5% and StrongReject never exceeds 10.9%.The refusal templates are in-distribution for AdvBench, so erosion resistance and harmful-prompt coverage are separate axes.
  • Persistence versus timing: The conclusion attributes robustness to persistence of the safety signal through pretraining rather than to the timing of its introduction.This conclusion is stated alongside matched-or-better capability through 6.9B.

AI USE STATEMENT

The authors used generative AI to scaffold parts of implementation, geometry-instrumentation, and analysis code, which they then reviewed, edited, tested, and regenerated from saved artifacts.

  • AI-assisted implementation: Generative AI assisted with parts of the training, geometry-instrument, and analysis code, but not with theoretical modeling, proof assistance, methodology design, or result interpretation.The authors state that Lemma 1 and its proof were not AI-assisted.
  • Verification: The authors reviewed and tested AI-scaffolded code, regenerated every number and figure from the released pipeline, and checked prior-work claims against primary sources.They take responsibility for the final content and artifacts.

ETHICS STATEMENT

The work studies a public benign-fine-tuning attack and reports aggregate refusal and geometry statistics without introducing a new attack or harmful capability. Experiments use public benchmarks, specified evaluation procedures, model checkpoints, and released artifacts.

  • Scope and safety: The study introduces no new attack or harmful capability and reports only aggregate refusal and geometry statistics.Harmful-prompt evaluations use public AdvBench and HarmBench benchmarks.
  • Attack procedure: The benign-SFT attack uses full-parameter AdamW updates on 100 Alpaca pairs for 200 steps at learning rate 2 × 10−5.The loss is cross-entropy over target tokens only.
  • Evaluation data: The evaluation set contains 64 AdvBench prompts, while divergence evaluation uses 50 harmful and 50 benign topic-matched prompts.Sampling uses fixed random seeds from public sources.
  • Capability evaluation: Capability probes comprise 64 items each from MMLU formal logic, GSM8K, TriviaQA-RC, and FLAN-v2, scored by gold-span negative log-likelihood.The probes cover reasoning, arithmetic, recall, and instruction following.
  • Geometry measurement: The geometry analysis uses 128 per-example gradients and rank k = 128, restricting Fisher measurements to per-layer MLP parameters.Attention-head and layer-norm contributions are omitted for budget reasons.
  • Artifacts: Launcher scripts, experiment result bundles, and a merged combined_bundle.json are available in the anonymized repository.The bundle contains nine experiments.

B GEOMETRY INSTRUMENT: IMPLEMENTATION NOTES

The geometry instrument estimates capability directions from per-example gradients without forming the full Fisher matrix, then tests safety-update overlap, curvature concentration, and robustness to prompt format. Across the reported checks, post-hoc alignment remains in suppression rather than erasure, with behavioral sub-patterns distinguished by their residual resistance.

  • Geometry diagnostics: The overlap ratio uses the decomposition Δ = Δ∥ + Δ⊥, with only the in-subspace norm contributing to ρ.The orthogonal component lies in approximately zero-curvature directions and does not contribute to first-order capability change.
  • Subspace estimator: The instrument estimates the top-k capability eigenspace with an N-side Gram trick instead of forming the d × d Fisher matrix.Projection onto the capability basis is computed directly from gradients, singular vectors, and singular values.
  • Null significance: Observed alignment is 21 to 66 times the random-subspace floor and 157 to 517 standard deviations above the null mean.The result is small in absolute terms but statistically non-random.
  • Orthogonality: The companion orthogonality statistic is |cos(∇Lcap, Δ)| ≈ 0.002, leaving Δ 99.8% orthogonal to the capability gradient.The first-order capability-loss conclusion is verified directly rather than inferred from this statistic alone.
  • Prompt-format robustness: Across three model families, ρ remains within ±5% for Qwen and Llama and ±20% for Mistral under 3-shot prompts, staying at least two orders below ρpart = 10−2.The suppression classification therefore does not depend on prompt format.
  • Curvature localization: Peak-qnorm layers are reproducible for REASON on Llama and Mistral, while Qwen’s L0 peak is treated as an embedding artifact with a relevant mid-layer secondary peak.Peaks for ARITH, RECALL, and INSTRUCT are less stable and do not support the safety-install-band argument.
  • Regime classification: Every tested DPO/RLHF install lies in suppression, with the largest ρ ≈ 2 × 10−4 at least 50× below ρpart = 10−2.Erasure is unobserved under post-hoc alignment and is instead associated with adversarial SFT or unlearning targets.
  • Suppression sub-patterns: Within suppression, one sub-pattern can yield 21.9% pre-attack refusal collapsing to 0%, whereas the strongest residual-resistance case retains 15.6% post-attack refusal.The distinction reflects whether the install has an in-subspace component that resists benign fine-tuning.

E CROSS-FAMILY REPLICATION DETAILS

Across three instruct-tuned model families, safety updates consistently occupy the suppression regime, while behavioral recovery and capability diagnostics require careful interpretation because base refusals and output formatting affect the measurements. An adversarial-SFT stress test reaches the erasure boundary, anchoring the geometric distinction without establishing practical unlearning.

  • Llama-3-8B: ρ ranges from 3×10−6 to 1.5×10−5 for Llama-3-8B, with qnorm peaking at 48.9 and 47.7 in blocks 30 to 31.The Llama update remains 10 to 30× the random floor, and its selective divergence peaks at +0.053.
  • Mistral-7B-Instruct-v0.3: ρ ranges from 9×10−6 to 3.2×10−5 for Mistral, with qnorm peaks of 158.7 for REASON and 92.1 for ARITH in early-to-mid residual layers.Mistral’s 1.2% pre-attack refusal makes its post-attack 0.4% refusal an install-thinness measurement rather than a meaningful recovery test.
  • Base-model behavior: The Qwen base already refuses 71.1% of AdvBench by regex and 69.5% by classifier, so its instruct update is a partial install rather than a complete creation of refusal.After 100 benign steps, Qwen-Instruct falls to 58.2% refusal, below its own base’s 71.1%.
  • Capability diagnostics: Raw-0-shot capability gaps can be inflated by output-format shifts, whereas 3-shot format neutralization reduces the gap to +0.37 to +1.10 nats.The instruct model’s preference for explanatory preambles lowers the probability of bare gold tokens without necessarily changing underlying knowledge.
  • Erasure stress test: ρ reaches 1.04 × 10−2 on target REASON and 3.01 × 10−3 on target ARITH, empirically reaching the ρpart boundary.These deliberately over-specified adversarial-SFT updates are a stress test for regime separation, not a practical unlearning method.

H FULL PROOF OF LEMMA 1 (KERNEL IMMOBILITY)

The proof shows that a safety update in the capability Fisher kernel cannot change capability loss at first order; under well-specified mature conditions, the remaining change is higher order. The developmental follow-ups support this geometry while identifying under-trained and over-trained regimes where assumptions differ.

  • Kernel-immobility proof: A kernel-immobility lemma shows that an update in ker(F_cap) has no first-order effect on capability loss.The linear Taylor term vanishes from the kernel condition alone, without requiring W_base to be a stationary point.
  • Kernel-immobility proof: Under exact Gauss–Newton, the capability-loss change is O(∥∆∥^3_2), while generally it includes a quadratic Gauss–Newton residual term.The residual term is 1/2∆^T E∆, followed by the Lipschitz-Hessian remainder.
  • Kernel-immobility proof: The lemma applies to the resulting optimizer update ∆, so preconditioning does not weaken the conclusion when ∆ satisfies the kernel condition.The premise concerns ∆ itself rather than the auxiliary-loss gradient.
  • Assumptions and limits: Under-trained checkpoints violate well-specification, making the residual large and producing a measurable O(∥∆∥^2) capability-loss shift.The mature post-substrate regime is where ∥E∥≈0 and the cubic bound applies.
  • Developmental-axis follow-up: The OLMo substrate emerges between 6B and 60B tokens, with selective divergence rising sharply from noise and then plateauing.The peak-divergence layer also migrates forward in the residual stream as pretraining proceeds.
  • Developmental-axis follow-up: The Pythia gradient norm follows the same U-shape as OLMo, minimizing near 63B tokens and increasing on both under-trained and over-trained sides.This cross-family replication argues against an OLMo-specific overtraining artifact.

K CO-TRAINING: FULL PROTOCOL AND PER-CONDITION TABLES

The co-training experiments compare continuous and windowed safety schedules under matched compute, with refusal, capability, geometry, and detector analyses across scales. Continuous co-training is the condition that retains refusal after benign fine-tuning, while windowed schedules do not provide lasting robustness.

  • Per-condition results: 98.4% → 84.0% refusal at 410M in the powered rerun shows modest erosion, versus 35 to 38 pp loss for post-hoc installs.The original 64-prompt run reported 84.4% → 84.4%, but the higher-powered 256-prompt rerun is the figure used in the main text.
  • Protocol: Continuous co-training uses λ(t)=0.10 throughout, matching the total safety-loss compute of the windowed schedule.The windowed schedule uses λ(t)=0.5 only from 0.10T to 0.30T.
  • Per-condition results: All three non-base conditions fall in the suppression regime, yet only continuous co-training resists the benign-SFT attack.The geometry instrument therefore does not distinguish behavioral robustness from fragility at this scale.
  • Capability: Within-model safety-gradient measurements show REASON falling at every scale, INSTRUCT rising at every scale, and RECALL and ARITH producing mixed results.These values are not directly comparable to displacement-based ρ measurements.
  • Schedule comparison: A late continued-training window installs refusal that the attack partially erodes, whereas an early window is washed out by subsequent language-model-only training.The 160M pilot extends the schedule comparison but has no 256-prompt rerun.
  • Detection robustness: The learned harmful-output classifier preserves the main ordering: post-hoc installs erode more, while continuous co-training remains the only condition that installs refusal in the schedule runs.It reports 82.8% refusal for continuous co-training against 4.7% for base before attack.

M LIMITATIONS AND BROADER IMPACT

The paper’s scope is bounded by approximations in the capability instrument, behavioral detection, and interpretation of co-training trajectories. Its broader framing situates the results among prior work on post-hoc fragility, developmental training, unlearning, and open-weight substrates.

  • Limitations: The capability Fisher is approximated by a top-k eigenspace, so important tail directions may remain unmeasured.A small top-k projection therefore does not exclude movement through low-curvature but semantically important directions.
  • Limitations: Between-trajectory co-training gaps have small scale-free ρ values that are close to a geometric tautology, so they cannot be interpreted like localized safety displacements.The gap is four to five orders larger than a real install and includes language-modeling trajectory divergence.
  • Limitations: The weight-space instrument detects suppression-regime alignment procedures, not whether a model is unsafe under every behavioral standard.Suppression-regime detection and behavioral unsafety are explicitly separate questions.
  • Post-hoc fragility: Prior work has recovered post-hoc safety through benign fine-tuning, adversarial suffixes, and linear probes, motivating the paper’s fragility analysis.Related accounts localize fragility to shallow output-token alignment, few safety-critical parameters, or a narrow safety basin.
  • Broader impact: The co-training result complements prior pretraining-safety work by finding that persistence across pretraining, rather than timing alone, drives attack robustness.The claim comes from the matched-compute windowed-versus-continuous comparison.
  • Unlearning contrast: The paper contrasts suppression-oriented RLHF and DPO with unlearning methods that target capability erasure, while noting that re-extraction often remains possible.This frames the geometric distinction as suppression versus erasure rather than treating all safety procedures as equivalent.
Loading 2609.06934v1…