PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness

Hoang Cuong Nguyen, Mark Dras, Usman Naseem

arXiv:2609.03887v1cs.CL

TL;DR

The paper examines whether post-training methods change how refusal is implemented internally, rather than only changing refusal behavior. It compares SFT, reasoning-augmented SFT, and ORPO across three architectures, finding that training objectives reshape refusal computation and steering robustness. No studied method simultaneously provides distributed refusal encoding, preserved capability, and fine-grained correctability.

  • Problem

    Existing work characterizes refusal circuits mainly in fixed models, leaving how training objectives reshape those circuits insufficiently analyzed.

  • Method

    The study controls data and hyperparameters while comparing SFT, Ra-SFT, and ORPO across Llama-3.1-8B, Gemma-2-9B, and Qwen3-8B using circuit, geometry, and steering analyses.

  • Results

    Post-training objectives reshape geometric and circuit structures, steering robustness, and attack-class vulnerability, while no studied offline method satisfies all three alignment criteria.

  • Takeaways & Limitations

    Safety alignment should not be treated as a solved, reliable defense, particularly for security-critical use.

  • Takeaways & Limitations

    Findings are based on one model level per architecture at 8B–9B, so generalizability to larger scales remains open.

Abstract

from arXiv · show

How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.

1 Introduction

The paper asks whether post-training objectives shape the internal computation of refusal, not merely its behavior. It introduces a controlled comparison and frames the results as an alignment trilemma involving refusal distribution, capability preservation, and correctability.

  • Motivation: Behavioral alignment methods are treated as reliable defenses despite documented jailbreaks bypassing refusal behavior.The motivation connects increasing LLM deployment and cyber-attack risk with concerns about relying on binary behavioral alignment.
  • Research gap: Existing mechanistic studies characterize refusal circuits in fixed models but do not systematically examine how training objectives reshape them.The paper identifies this as a gap that leaves training dynamics understood mainly post hoc.
  • Research question: The study compares SFT, ORPO, and reasoning-augmented SFT while holding the underlying skill constant and tracing refusal computation internally.Ra-SFT trains on reasoning chains that justify whether requests are safe or harmful.
  • Study design: The comparison controls base model, training data, and hyperparameters across Llama-3.1-8B, Gemma-2-9B, and Qwen3-8B.This design isolates training-objective effects while testing three architecturally distinct models.
  • Main claim: No studied method jointly achieves distributed refusal encoding, safety-capability separability, and fine-grained correctability.The paper characterizes this trade-off as an empirically grounded alignment trilemma.

2 Related Works

Related work frames safety alignment as a mechanistic problem involving refusal geometry, circuit concentration, and steering reliability. The paper extends these lines by testing whether post-training objectives influence refusal circuits and steerability.

  • Refusal representations: Prior work represents refusal using directions in activation space and separates harmfulness, refusal, recognition, and execution into distinct components or axes.These studies motivate analyzing refusal representations beyond a single behavioral label.
  • Shallow alignment and circuit concentration: Safety mechanisms can be highly concentrated, with prior studies reporting large causal effects from small subsets of heads, neurons, or even one neuron.This concentration motivates testing whether alignment methods distribute safety-related computation more broadly.
  • Post-training and safety alignment: Behavioral studies report that post-training methods shift the safety-utility frontier and that ORPO can have limited safety generalization.These findings concern behavior or weight-space geometry rather than controlled circuit-level causal attribution.
  • Steering reliability: Steering reliability depends on prompt distributions, response preferences, and overlap between harmful and harmless activations.The paper asks whether training objectives also determine steerability.

3 Methodology

The methodology compares three post-training objectives under matched data and hyperparameters across three model architectures. It combines refusal-geometry analysis, causal circuit tracing, and inference-time steering.

  • Experimental paradigm: The study maps SFT, Ra-SFT, and ORPO onto preference and reasoning axes to test how these factors change refusal circuits.SFT adds neither preference nor reasoning, Ra-SFT adds reasoning supervision, and ORPO adds preference optimization.
  • Experimental design: Training uses matched safety and utility data, while online preference methods are excluded because iterative data and reward updates would confound objective comparisons.The datasets include Alpaca benign prompts and BeaverTails safety prompts, with a matched ORPO construction for chosen and rejected responses.
  • Models: Llama-3.1-8B, Gemma-2-9B, and Qwen3-8B are fine-tuned from base models with matched hyperparameters to separate architecture-dependent from objective-dependent effects.The models are selected to provide architectural diversity at comparable scales.
  • Geometry of refusal: Normalized difference-in-means refusal directions quantify when refusal is encoded and compare directional similarity across objectives.Normalization by mean activation norm compensates for residual-stream growth across layers.
  • Circuit analysis: Activation Patching establishes layer-level causal effects, while Attribution Patching identifies important MLPs and attention heads for exact component testing.The procedure uses the same harmful/harmless prompt pairs and selects top-K components for causal evaluation.
  • Activation steering: ActAdd and ITI steer inference-time activations along refusal directions, with positive intervention strength favoring refusal and negative strength favoring compliance.ActAdd operates at the layer level, whereas ITI targets selected attention-head outputs.

4 Evaluation Results

Across models, training objectives produce distinct refusal geometries and circuit topologies, while architecture shapes where refusal is encoded and how steering affects safety and utility. The results expose trade-offs among safety strength, distributed computation, capability preservation, and correctability.

  • Behavioral safety: ORPO strengthens safety more than Ra-SFT in most comparisons, but Gemma ORPO reaches 0.0% StrongREJECT ASR alongside 31.6% XSTest ORR.Gemma Ra-SFT and SFT reach 7.9% and 31.6% StrongREJECT ASR, respectively.
  • Refusal geometry: Across all three architectures, Ra-SFT produces a refusal direction that differs from SFT and ORPO, with magnitude distributed across layers rather than sharply concentrated.SFT and ORPO peak in mid-layers for Llama and Qwen, whereas Ra-SFT rises gradually across the network.
  • Causal effects: All objectives peak causally at late layers, but their trajectories differ, and most conditions show positive causal effects at the initial layer.The exception is Gemma Ra-SFT, which lacks an initial positive causal effect; this pattern is consistent with harmfulness concepts inherited from pre-training.
  • Circuit topology: Circuit topology varies by architecture: Llama shifts from attention-head dominance under SFT toward MLP dominance under Ra-SFT and ORPO, while Gemma and Qwen show different MLP and redundancy patterns.Gemma SFT and ORPO encode refusal uniformly across components, whereas Gemma Ra-SFT is uneven and MLP-dominated; Qwen is MLP-dominant under all objectives.
  • Steering: Steering effects expose operational trade-offs: Llama ActAdd rapidly collapses MMLU accuracy, Gemma maintains 53–55% SFT, 40–44% Ra-SFT, and 45–47% ORPO accuracy, while ITI can destabilize generation.Gemma ORPO’s ORR falls from 31.2% to 20.4% under refusal-averse steering, whereas Llama ITI produces loops and question repetition at α = 20.
  • Steering: Recognition-layer steering generally outperforms execution-layer steering, including an 18.8pp versus 5.8pp ASR reduction for Gemma Ra-SFT and 28.2pp versus 10.8pp for Qwen SFT.The recognition–execution gap depends on both layer engagement with the refusal circuit and architecture; Qwen ORPO’s gap narrows as α increases.
  • Correctability: Single-component head-level ITI is insufficient across concentrated, distributed promotive, and distributed uniform circuit types.The authors identify this limitation across Llama SFT, Gemma SFT, and Gemma ORPO circuit structures.

5 Discussion

DIM analysis reveals that training objectives produce different refusal geometries, while steering experiments show that the usefulness of these approximations depends on both training paradigm and architecture.

  • Refusal Representations: DIM separates refusal representations but only approximates their full geometry, which may contain multiple independent directions.The paper frames DIM separation as a cone axis rather than a complete description of refusal geometry.
  • Steering Effectiveness: Steering serves as a probe of whether circuit-analysis components mediate refusal, rather than as an independent alignment method.The paper uses steering outcomes to test the functional relevance of identified components.
  • Steering Effectiveness: Training objectives reshape steering outcomes: Gemma Ra-SFT shows a recognition-execution gap consistent with successful ASR reduction, whereas Gemma ORPO exhibits ITI failure from overconstrained circuits.The same approximation can therefore fail or succeed depending on the paradigm and architecture.

6 Conclusions and Future Work

The paper finds that post-training objectives reshape LLM geometry, refusal circuits, steering robustness, and attack-class vulnerability. Across the studied objectives, no offline method satisfies distributed refusal encoding, safety/utility separability, and granular correctability together.

  • Conclusions: Post-training methods reshape both geometric and circuit structures, as well as steering robustness and vulnerability across attack classes.These findings come from a cross-paradigm circuit analysis of post-training methods.
  • Conclusions: No studied offline objective simultaneously achieves distributed refusal encoding, safety/utility separability, and granular correctability.The paper characterizes this trade-off as an alignment trilemma.

Limitations

The study’s conclusions are bounded by model scale, untested interactions between reasoning and preference training, data and generation choices, and run-to-run instability in borderline classifications.

  • Scope and Scale: The experiments use one model level per architecture at 8B–9B, leaving generalizability to larger scales open.Adding scale without matched architectural comparison could introduce confounding factors.
  • Untested Training Interactions: The study does not test how reasoning and preference optimization interact in safety alignment.Such experiments would require constructing additional reasoning chains for rejected unsafe responses.
  • Data and Generation: Reasoning-chain generation through one-shot GPT-4o prompting complicates controlled comparisons of chain length and style.The choice of BeaverTails also constrains the safety-specific data setup described by the authors.
  • Run-to-Run Variance: 18/256 harmless prompts (7%) flipped from refusal to compliance across matched runs of the same Gemma ORPO checkpoint.The authors treat this instability as a source of run-to-run variance affecting downstream normalized effect estimates.

Ethics Statement

The authors state that the work analyzes current post-training methods as jailbreak defenses while acknowledging potential misuse risks. They restrict the analyses and datasets to research purposes and advocate further work toward more secure, ethically aligned models.

  • Risk and Purpose: The paper acknowledges that analyzing failures of current post-training defenses can induce offensive content and be exploited for misuse.This risk is stated alongside the work’s focus on improving robustness against harmful content.
  • Future Research: The paper encourages further research to support more secure and ethically aligned generative models.This is presented as the primary constructive goal of the work.
  • Research Use: The authors state that all analyses and datasets are intended strictly for research under community ethical guidelines.They explicitly condemn misuse of the work to generate or disseminate harmful content.

B Training Details

The study combines activation patching, attribution patching, and inference-time steering to analyze refusal circuits and their manipulability across post-training conditions. Its analyses also acknowledge that attribution scores approximate causal effects and that training objectives can reshape component interactions nonlinearly.

  • Model selection: The experiments compare refusal behavior across Llama-3.1-8B, Gemma-2-9B, and Qwen3-8B, which differ in layer and attention-head counts.Llama has 32 layers and 32 heads per layer; Gemma has 42 layers and 16 heads; Qwen has 36 layers and 32 heads.
  • Attribution patching: Attribution patching estimates component effects with a first-order Taylor expansion using gradients of the logit-difference metric.The method reduces attribution cost from O(L) forward passes to two forward passes and one backward pass.
  • Causal-analysis caveats: Attribution scores are only approximations: a component’s estimated sign can differ from its actual causal effect because indirect residual-stream effects are omitted.Exact causal effects require activation patching, and sign flips across components indicate nonlinear reshaping of MLP interactions under training objectives.
  • Inference-time steering: Inference-time steering uses ActAdd on residual-stream layers and ITI on attention-head outputs, with positive coefficients steering toward refusal and negative coefficients toward compliance.ActAdd adds a scaled normalized refusal direction, while ITI uses per-head probing directions.
  • Reasoning traces: Gemma Ra-SFT exposes explicit reasoning traces for harmful and benign prompts, whereas Llama Ra-SFT does not consistently expose them and can misclassify an adversarial request as safe.The Gemma traces are removed when matching refusal/compliance pairs for activation patching.

I Bootstrap Analysis of Attention Heads/MLPs Causal Effects

The bootstrap and steering tables examine stability and intervention outcomes at the component and layer levels. They emphasize objective-dependent causal-ranking stability and differences in refusal steering across model conditions.

  • Bootstrap stability: Table 7 reports bootstrap stability of component-level causal rankings using Spearman ρ and top-1 frequency across 1000 prompt-pair resamples.Spearman ρ measures rank agreement with the original ranking, while top-1 frequency records how often the original highest-effect component appears in the resampled top five.
  • Layer-level steering: Table 8 compares Gemma-2-9B ActAdd steering at recognition and execution layers, with recognition-layer steering producing stronger ASR reduction for Ra-SFT while maintaining stable MMLU accuracy.The table directly frames recognition-layer intervention as the key comparison for Ra-SFT.
  • ORPO steering: Table 9 reports Gemma-2-9B ORPO ActAdd results using ASR, ORR, and MMLU across WildJailbreak, XSTest, and MMLU.The table organizes refusal safety, over-refusal, and general capability outcomes under ORPO steering.

J Full Analysis of ActAdd Results in Gemma-2-9B and Qwen3-8B

ActAdd steering reveals architecture- and method-dependent differences between refusal recognition and execution, with stronger safety changes often carrying capability or correction trade-offs. Qwen3 shows especially clear capability damage at high steering magnitudes, while Gemma supports targeted over-refusal mitigation with saturation effects.

  • Evaluation caveats: Single-run ActAdd results can differ from baseline even at α = 0, complicating direct interpretation of steering effects.The target-layer hook itself may alter model performance before steering is applied.
  • Capability trade-offs: Qwen3 SFT and ORPO steering toward refusal severely impacts MMLU when α > 15, unlike Gemma’s stable capability throughout steering.Qwen3 is characterized as intermediate between Llama’s earlier collapse and Gemma’s stability.
  • Recognition versus execution: Recognition-layer steering reduces Qwen3 ASR more strongly than execution-layer steering for SFT and ORPO, but damages MMLU capability.Table 10 summarizes this safety–capability trade-off.
  • Correcting over-refusal: Steering Qwen3 SFT away from refusal reduces ORR by 19.6pp but increases ASR by 8.4pp, compared with 11.2pp and 0.8pp in Gemma.The comparison reflects different correction trade-offs between the two models.

K Results and Failure Modes of ITI in Llama-3.1-8B and Gemma-2-9B

ITI exposes distinct failure modes across models: Llama can lose coherent generation under steering, while Gemma shows little safety-performance improvement. Attack vulnerability also remains method- and architecture-dependent, and the StrongREJECT results are preliminary because each attack class has only 60 prompts.

  • Llama failure modes: Llama ITI at α = 0 repeats the question, while α = 20 produces repetitive single-token collapse rather than genuine safety behavior.The latter may be read as refusal by string-matching classifiers despite representing generation failure.
  • Gemma failure modes: Gemma shows little to no performance improvement from ITI despite avoiding the coherence degradation observed in Llama.Table 12 reports the corresponding Gemma steering results.
  • Attack vulnerability: Llama SFT retains 65% ASR on role_play and 56.7% on happy_to_help, whereas Ra-SFT falls to approximately 10–17% across those framing attacks.ORPO can block nearly all attack prompts in the reported Llama evaluation.
  • Attack vulnerability: Gemma SFT retains roughly 50–60% ASR on adversarial reframing, while ORPO records 0% ASR across all tested attack types.Ra-SFT leaves approximately 20% ASR on happy_to_help and Wikipedia.
  • Attack vulnerability: Qwen3 remains vulnerable to encoding attacks after post-training, including 18.3% harmful disemvowel prompts bypassing ORPO safeguards.Semantic reframing attacks are easier for Qwen3 to block than for the other two models.
  • Evaluation caveats: StrongREJECT findings are preliminary because experiments used only n = 60 prompts per attack class.The passage reserves full cross-attack analysis for future work.
Loading 2609.03887v1…