PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Fine-Tuning Undo Activation Steering? Behavioural Recovery Without Weight-Edit Reversal

Philipp E. Glass, Allan Tucker, Yongmin Li, Alina Miron

arXiv:2608.24988v1cs.CLcs.AIcs.LG

TL;DR

It is unclear whether embedded steering survives routine fine-tuning or whether behavioural recovery reflects reversal of the weight edit. Across five instruction-tuned models, the study finds that behaviour tracks training content while the edit remains mechanistically stable, motivating behavioural re-validation.

  • Problem

    It is unclear whether embedded steering persists after fine-tuning or whether recovered behaviour results from erasing the embedded weight edit.

  • Method

    The study evaluates refusal suppression and brevity induction across five open-weight instruction-tuned models from 3B to 14B parameters under non-adversarial SFT and RLHF, using behavioural and weight-level analyses.

  • Results

    Behavioural preservation tracks training content, while fine-tuning reverses at most 0.79% of the embedded edit, with mean vector recovery ρ = 0.004 and mean cos θ = 0.074.

  • Takeaways & Limitations

    Embedded steering is mechanistically durable but functionally vulnerable, so behavioural performance should be re-validated after downstream training.

  • Takeaways & Limitations

    The analysis does not identify which alternative pathways models recruit to recover steered behaviours around the intact edit.

Abstract

from arXiv · show

Activation steering can be embedded directly into a language model's weights, shaping behaviour without inference-time intervention and offering a way to encode alignment prior to release. However, models are routinely fine-tuned after deployment, and it is unknown whether embedded interventions survive this. We study the stability of embedded steering for refusal suppression and brevity induction across five instruction-tuned models (3B-14B) under non-adversarial SFT and RLHF. Behaviourally, preservation tracks the training data: steering degrades when optimisation pressure contradicts the targeted behaviour and persists otherwise, with refusal ablation losing 64% of its effect on average under SFT. Mechanistically, however, the weight edit survives almost untouched even where behaviour reverts: mean vector recovery is $ρ= 0.004$, and the fine-tuning update along the steering direction is near-orthogonal to its pre-edit weight pattern (mean $\cosθ= 0.074$). When steered behaviour degrades, fine-tuning does not achieve it by dismantling or reversing the steering mechanism itself. Embedded steering is therefore mechanistically durable but functionally vulnerable, and requires behavioural re-validation after downstream training.

1 Introduction

The paper asks whether embedded activation steering survives routine downstream fine-tuning and whether behavioural recovery reflects edit reversal or alternative mechanisms. Across the studied settings, behavioural stability depends on training content, while the underlying edit remains mechanistically stable.

  • Motivation: Routine fine-tuning may degrade embedded steering, raising whether behavioural recovery reflects weight-edit reversal or alternative mechanisms.The question is especially relevant when steering supports safety or suppresses undesired behaviour in models expected to undergo downstream fine-tuning.
  • Study scope: The study tests refusal suppression and brevity induction across five models, two training paradigms, and full-parameter fine-tuning.The interventions cover a safety-relevant behaviour and a stylistic behaviour.
  • Scope boundary: Unrestricted targeted adversarial fine-tuning can remove safety filters, so the study concerns routine non-adversarial fine-tuning rather than worst-case attacks.The authors frame embedded steering as requiring sufficient stability for practical use, not as impervious to unrestricted intervention.
  • Headline findings: 64% average loss of refusal-ablation effect occurs under SFT when training content applies pressure toward refusal.The supplied contribution passage reports this behavioural degradation under SFT.
  • Headline findings: Fine-tuning reverses at most 0.79% of the embedded edit, with mean ρ = 0.004 and mean cos θ = 0.074, even when behaviour reverts.The edit remains mechanistically stable despite behavioural degradation.

2 Preliminaries & Embedded Steering

The paper treats steering directions as approximately linear activation-space features and implements behavioural changes by shifting, removing, or amplifying those directions. Linear projection steering can be embedded into weights, eliminating runtime intervention.

  • Representations and direction estimation: Concepts or behaviours are represented as approximately linear directions in activation space, which can be estimated from contrasting prompt sets.The method records activations for eliciting and non-eliciting prompts, then computes a difference of means and normalises it.
  • Steering operators: Offset steering adds or subtracts a constant direction at inference time, whereas projection steering removes the direction from the activation stream.These operators respectively induce or suppress the targeted behaviour.
  • Embedded steering: Embedding projection steering modifies output-writing matrices so the projection is applied directly in the model weights.The relevant matrices include token embeddings, attention output projections, and MLP down-projections.
  • Embedded steering: Because the projection and residual-stream contribution are linear, W′out = Wout − dd^⊤Wout reproduces the inference-time projection without runtime cost.ActAdd cannot be absorbed into weights because its constant offset is independent of activations.
  • Steering operators: Amplification scales an existing activation component and therefore strengthens a present feature rather than inducing one from nothing.Removal, partial suppression, and amplification form a modulation spectrum within one framework.

3 Experimental Framework

The experiments apply refusal ablation and brevity amplification to five open-weight instruction-tuned models, then evaluate full-parameter SFT and RLHF using behavioural and mechanistic persistence metrics. The design introduces limited training-data pressure against refusal ablation while minimizing directly contradictory pressure against brevity amplification.

  • Fine-tuning conditions: Training data contains opposing refusal signals but comparatively few examples directly contradicting brevity amplification, testing whether degradation follows optimisation pressure.SFT refusal completions provide a stronger opposing signal than the sparser RLHF signal.
  • Models: Five open-weight models spanning 3B to 14B parameters provide coverage of deployment settings where downstream fine-tuning is expected.The models come from three families.
  • Interventions: The two interventions suppress refusal or amplify response brevity, representing safety-relevant and stylistic steering targets.Refusal is evaluated as declining harmful requests, while brevity targets shorter responses.
  • Fine-tuning conditions: SFT uses 1,024 OpenOrca optimiser steps, while RLHF uses about 420 PPO steps on Anthropic’s hh-rlhf dataset.The datasets and optimisation settings differ, so the authors report the outcomes separately rather than formally comparing training methods.
  • Metrics: Behavioural preservation P measures retained steering effect, with P = 1 denoting full preservation and P = 0 complete reversion to base behaviour.The authors compute separate preservation measures for amplification and ablation, which are not directly comparable.
  • Metrics: Mechanistic persistence is measured by vector recovery ρ, where ρ = 0 means the edit is fully preserved and ρ = 1 means full recovery toward the pre-edit projection.The metric compares projections for the original, steered, and fine-tuned models.

4 Results

Fine-tuning affects embedded steering behaviorally according to the training regime and data, while the underlying weight edit remains largely intact. RLHF preserves both interventions, whereas SFT more substantially reverses refusal ablation and varies across models.

  • Behavioural Stability: RLHF preserves both steering interventions, while SFT substantially reverses refusal ablation and preserves brevity on average.Refusal ablation loses 64% of its effect on average under SFT, whereas brevity preservation is less consistent across models.
  • Behavioural Stability: Steering degrades when training data opposes the targeted behavior and persists when it does not.This pattern is reported for routine fine-tuning rather than explicitly adversarial intervention removal.
  • Behavioural Stability: Under SFT, refusal-ablation preservation spans Psft = 0.15 for Qwen3-14B to 0.56 for Llama-3-8B.The paper attributes this variation to model-specific factors including architecture, pre-training data, and baseline behavior.
  • Mechanistic Persistence: Mean vector recovery is ρ = 0.004, with no run exceeding 2%, even when behavior reverts substantially.Refusal ablation under SFT shows 64% behavioral recovery while ρ remains 0.010.
  • Mechanistic Persistence: Fine-tuning reverses at most 0.79% of the embedded edit, while the update along the steering direction has mean cos θ = 0.074.The update is nearly orthogonal to the pre-edit weight pattern, indicating that behavioral recovery occurs through alternative pathways rather than linear edit reversal.
  • Mechanistic Persistence: Behavioral recovery and vector recovery remain dissociated: points with near-complete behavioral recovery still have ρ below 0.02.The scatterplot would place recovery from edit reversal toward the top right, but observed vector recovery stays low.

5 Discussion

Fine-tuning can restore steered behaviour without reversing the embedded weight edit. Behavioural stability instead depends on whether training content applies contradictory optimisation pressure, making post-training validation necessary.

  • At most 0.79% of the embedded edit is reversed across tested models and conditions, even when steered behaviour substantially reverts.Vector recovery remained ρ < 0.02.
  • Mean cos θ = 0.074 shows that fine-tuning updates along the steering direction are near-orthogonal to the pre-edit weight pattern.The update engages the edited direction without undoing the original weight pattern.
  • Because vector recovery and behavioural recovery are largely independent, degraded behaviour reflects alternative mechanisms that reduce the edit’s influence.The reported association is r = +0.24, n = 20, p = 0.31.
  • Refusal ablation erodes under SFT, while brevity amplification persists and both interventions are well-preserved under RLHF.The pattern tracks training content: OpenOrca contains refusal examples but little contradictory signal for brevity.
  • A persistent steering direction can become functionally inert, so checking its presence in weights is insufficient and re-steering along it may not counteract recovery.The original direction ceases to mediate the restored behaviour when fine-tuning routes around it.
  • Embedded steering is viable but not guaranteed, and its resilience varies across models and targets; behavioural validation after fine-tuning remains necessary.The authors report greater vulnerability for safety-related than stylistic steering in these experiments, while cautioning that this target difference may not generalise.

6 Conclusion

Embedded activation steering remains mechanistically stable under routine fine-tuning, even when behaviour reverts. Behavioural degradation follows training-content pressure rather than erasure, so steering is durable but requires re-validation.

  • Weight modifications persist even where behaviour reverts, demonstrating mechanistic stability under routine fine-tuning.
  • Behavioural degradation is driven by optimisation pressure from training content rather than erasure of the steering mechanism.
  • Steering should be treated as durable but not permanent, with behavioural re-validation after downstream training.

Limitations

The study’s limitations concern scope, experimental coverage, measurement heuristics, unresolved recovery mechanisms, and the absence of a training-only comparison.

  • Scope and perturbations: Experiments focus on full-parameter fine-tuning of open-weight models up to 14B parameters, without demonstrating generalisation to larger frontier or closed-weight models.Quantisation and other weight perturbations may degrade steering through different mechanisms and remain outside the demonstrated scope.
  • Experimental coverage: The experiments cannot capture the full diversity of real-world fine-tuning scenarios.Each SFT and RLHF protocol uses a single dataset, limiting coverage of downstream training conditions.
  • Experimental coverage: Contradictory training pressure is not systematically varied, so its relationship with behavioural degradation is not precisely characterised.A controlled study could vary the fraction of refusal examples in the SFT corpus.
  • Measurement: Steering directions are estimated with heuristics, and residual misclassification by the LLM-based refusal judge is not fully ruled out.The refusal classifier is cross-validated against substring matching.
  • Mechanistic interpretation: The mechanistic analysis does not identify which alternative pathways models recruit when recovering steered behaviours.It remains unresolved whether recovery reuses pre-existing circuits or constructs new ones.
  • Baselines: The study does not compare steered models with models acquiring equivalent behaviours through training alone.Such a comparison would clarify steering’s utility relative to training-based behaviour modification.

B Baseline Training Results

Baseline controls show that the training protocols do not independently create the observed post-training behaviours. Thus, changes in steered models can be attributed to the embedded interventions rather than uniform effects of SFT or RLHF.

  • SFT maintains refusal rates in non-steered models, so refusal recovery after ablation reflects reversal of the ablation rather than uniformly increased refusal.
  • 95.9 tokens versus 59.4 tokens: non-steered RLHF models produce longer brief responses than steered+RLHF models.This supports the conclusion that brevity preservation reflects embedded steering rather than RLHF independently inducing brevity.
  • Table 4 compares refusal rate R before and after SFT and mean brief-response length L before and after RLHF in non-steered models.

C Bootstrap Confidence Intervals for Behavioural Results

Behavioural results are accompanied by non-parametric bootstrap confidence intervals computed from prompt-level metrics. Preservation metrics use paired resampling because the same prompts evaluate base, steered, and post-trained checkpoints.

  • 10,000 bootstrap replicates estimate 95% non-parametric confidence intervals for the behavioural results.
  • 100 harmful prompts underpin refusal-rate intervals, while 300 long-response prompts underpin response-length intervals.
  • Paired resampling draws the same prompt across base, steered, and post-trained checkpoints for preservation metrics.This preserves within-prompt comparisons across the three evaluated model states.
  • Refusal-ablation intervals appear in Table 5, and brevity-amplification intervals appear in Table 6.
  • Five repeated Qwen3-14B refusal-ablation-plus-RLHF runs produced refusal rates from 0.13 to 0.16, with mean 0.146 and σ = 0.011.The repeated condition varied only the training random state.

D Auxiliary Measurements for the Vector Recovery Ratio

Auxiliary measurements distinguish genuine reversal from orthogonal or direction-specific weight changes. They show that fine-tuning perturbs the steering direction preferentially, but mostly away from the pattern that would reverse the edit, within the measured projections.

  • Measurement rationale: ρ can be small either because d is untouched or because perturbations lie orthogonal to the embedded edit, so U is measured directly.The update U = W_ft − W_steer is aggregated across attn.o_proj and mlp.down_proj.
  • Reversal fraction: 0.08% mean reversal and at most 0.79% across 20 runs: fine-tuning cancels only a small fraction of the embedded edit.The largest reversal occurs for SOLAR-10.7B under refusal ablation + SFT, also the run with the highest behavioural recovery.
  • Targeting of d: 1.11% versus 0.47%: fine-tuning updates project more strongly onto the steering direction d than onto the mean of 1,000 random unit directions.The mean per-run ratio is 3.14×, indicating preferential perturbation of d.
  • Orthogonality: 0.074 mean cosine: in eight full-orthogonalisation runs, the update component along d is nearly orthogonal to its pre-edit weight pattern.About 93% of that component points away from the axis that would reverse the edit.
  • Scope: The analysis measures only attn.o_proj and mlp.down_proj and captures only linear cancellation of E, excluding QKV, input embeddings, and nonlinear or compositional reversal.
  • Direction estimation: Steering directions are estimated by contrastive prompting from first-token residual activations, then selected or optimised for behavioural effectiveness and capability preservation.The optimisation-based procedure uses direction interpolation, layer-wise schedules, and a Pareto-front trade-off between steering effectiveness and KL divergence.
  • Optimisation-based embedding: Variable orthogonalisation and brevity amplification use Heretic-based optimisation over direction and per-layer strength parameters.The intervention is embedded with layer-specific weights after multi-objective optimisation.
  • Optimisation-based embedding: 200 trials jointly minimise steering failure and KL divergence from the unmodified model before selecting a Pareto-front solution for weight embedding.

F Datasets, Prompt Sets, and Classifiers

The experiments use harmful, harmless, and long-response prompt sets, with refusal classified by an LLM judge and cross-checked against a substring heuristic. The appendix also documents the training implementations and hyperparameter tables.

  • F.1 LongElicitPrompts Dataset: 408 prompts elicit long responses through discussions, open-ended questions, and writing tasks in the LongElicitPrompts dataset.The dataset is used for brevity-related evaluation.
  • F.2 Prompt sets used for evaluation: Evaluation prompt sets include harmful requests and general questions covering regulation, mental health, finance, climate policy, biology, and search engines.
  • F.3 LLM-as-judge refusal classifier: DeepSeek-V4-pro labels prompt-response pairs as REFUSAL or COMPLIANCE under explicit rules for warnings, genuine compliance, and pretend compliance.
  • F.3 LLM-as-judge refusal classifier: 92.4% agreement and Cohen’s κ = 0.842 were obtained between the judge and substring heuristic across 3,000 prompt-response pairs.The judge’s refusal rate was 0.586 versus 0.631 for substring matching; disagreements mainly involved warnings followed by compliance.
  • F.4 Refusal-marker list: The substring baseline flags a response as refusal when any listed marker, such as “sorry” or “i cannot,” appears anywhere in the generation.
  • G Hyperparameters: Training uses full-parameter updates on 4 × H100 80GB GPUs with a modified verl library.
  • G Hyperparameters: SFT uses the modified fsdp_sft_trainer, while RLHF uses the modified main_ppo trainer with settings reported in Tables 8 and 9.
  • G.2 RLHF (PPO): Skywork/Skywork-Reward-V2-Llama-3.1-8B serves as the RLHF reward model.
Loading 2608.24988v1…