PaperlayerView exact revision on arXiv

Source-linked AI summary

STR: Supervised Transcoder Replacement for Reducing Steering Side Effects

Haonan Yu, Junhao Liu, Zhenyu Yan, Haoran Lin, Xin Zhang

arXiv:2609.32519v1cs.AIcs.LG

TL;DR

Model steering can strengthen a target behavior while degrading other useful behaviors. STR trains a supervised transcoder replacement for the steering-layer MLP, then fits existing steering methods on the frozen replacement with their original objectives. The approach reduces safety side effects on SALAD and unseen safety datasets while retaining target control, including lower SSV ASR on both Gemma models reported for out-of-distribution evaluation.

  • Problem

    Steering can strengthen a target behavior while degrading other useful behaviors, making reduction of side effects while retaining target control a central challenge.

    Source: §1 · p. 1

  • Method

    STR trains a transcoder replacement for the steering-layer MLP with target-control, non-target-preservation, and unsteered-fidelity supervision, then fits selected steering methods on the frozen replacement using their original objectives.

    Source: §1 · p. 1

  • Results

    On unseen safety datasets, SSV ASR falls from 42.46% to 14.42% on Gemma-3-4B and from 34.97% to 12.91% on Gemma-3-12B, while target utility is close or improved.

    Source: §4.4 · p. 8

  • Takeaways & Limitations

    Replacement training can benefit steering methods fitted without protection objectives, reducing safety side effects while retaining effective target control.

    Source: §1 · p. 2, §6 · p. 9

    The reported attack-success reductions characterize the evaluated settings and do not establish safety for unrestricted deployment.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Model steering can strengthen a target behavior while degrading other useful behaviors. We introduce Supervised Transcoder Replacement (STR) to reduce these side effects for existing steering methods, including those fitted without a protection objective. STR learns a replacement for the multilayer perceptron (MLP) computation at the steering layer through supervision for target control, non-target preservation, and fidelity without steering. Selected steering methods then fit directions on the frozen replacement while retaining their own fitting objectives. We evaluate three steering methods across Gemma and Llama models using Corrigibility preferences and four harmful-request safety datasets. SALAD-Bench supplies protection training data and a separate in-distribution evaluation split; HarmBench, AdvBench, and StrongREJECT are reserved for out-of-distribution testing. STR substantially reduces steering side effects on the in-distribution evaluation and extends this protection to the unseen safety datasets while retaining effective target control. For target-only supervised steering vectors, pooled out-of-distribution attack success rate falls from 42.46% to 14.42% on Gemma-3-4B and from 34.97% to 12.91% on Gemma-3-12B. These results show that replacement training can benefit steering methods fitted without protection objectives.

1 INTRODUCTION

STR replaces the MLP computation at a steering layer with a supervised transcoder, aiming to strengthen target behavior while preserving specified non-target behavior. Across steering methods and models, the replacement reduces safety side effects on SALAD and unseen safety datasets while retaining target control.

1 INTRODUCTION

  • STR’s motivation is that target and non-target features can share coordinates, so changing target-used coordinates may also perturb computations supporting other behaviors.The paper links this overlap to polysemantic neurons and feature superposition, and proposes a sparse transcoder replacement trained for target control and specified non-target preservation.Source: §1 · p. 1
  • STR trains a transcoder to replace the steering-layer MLP, jointly optimizing it with an auxiliary steering vector while the surrounding language model remains frozen.Training encourages stronger target behavior and preservation of specified non-target behavior under intervention, plus fidelity to the original model when control is disabled.Source: §1 · pp. 1–2
  • Existing steering methods fit directions on the frozen replacement using their original fitting rules, while preservation learned during replacement training can benefit methods without protection objectives.The evaluation covers CAA, target-only SSV, and protection-aware OPIUM across three models; SALAD-Bench supplies protection training and development data, while three other safety datasets are held out.Source: §1 · p. 2
  • The paper’s comparisons span three steering methods and three models, and report protection transfer from SALAD to unseen safety sources while retaining effective target control.The evaluated methods are CAA, target-only SSV, and protection-aware OPIUM; models are Gemma-3-4B-it, Llama-3.1-8B-Instruct, and Gemma-3-12B-it.Source: §1 · p. 2

2 BACKGROUND

The background introduces activation steering, target control and non-target preservation, and sparse transcoder replacements as the concepts underlying STR. It frames steering as applying a fitted direction at a representation site and a transcoder as a replacement computation with its own steering space.

2.1 ACTIVATION STEERING

  • Activation steering fits a direction at a representation site and applies it with an intervention strength; additive steering takes the form Steer(h, αv) = h+αv.The direction-fitting rule may use representation statistics or optimize a behavioral objective, and uses the representations and intervention rule of the model receiving it.Source: §2.1 · p. 2
  • A steering controller comprises a representation site, fitted direction, and intervention strength; target utility measures desired-behavior performance, while the model computation receiving the direction also shapes effects.Enabled control is denoted (M, αv), and disabled control (M, 0).Source: §2.1 · p. 2

2.2 TARGET CONTROL AND NON-TARGET PRESERVATION

  • A steering side effect is degradation of a non-target behavior when a procedure intended to improve the target behavior is applied.Effective control increases target utility while keeping non-target degradation low.Source: §2.2 · p. 3

2.3 TRANSCODERS AS LAYER REPLACEMENTS

  • A transcoder learns a sparse input-output approximation to an MLP computation, and its decoded output replaces that computation’s output in the forward pass.A single MLP replacement leaves surrounding attention, residual connections, and other layers intact; STR trains it for unsteered fidelity and preservation during steering.Source: §2.3 · p. 3

3 METHOD

STR first trains a transcoder replacement with an auxiliary steering vector, then freezes it and fits a selected steering method’s direction in the replacement’s latent space. Training addresses target control, protected behavior under intervention, and fidelity without control.

3.1 TRAINING THE REPLACEMENT

  • A transcoder replaces one MLP mapping, while a trainable auxiliary vector creates steering perturbations in its sparse latent space.The surrounding model remains frozen during replacement training.Source: §3.1 · p. 3
  • Training jointly optimizes target behavior under steering, protection on examples under the same intervention, and fidelity to the teacher without control.Protection uses examples from DB; unsteered fidelity covers target, protection, and generic replay data.Source: §3.1 · p. 4

3.2 USING STEERING METHODS ON THE REPLACEMENT

  • After freezing the replacement and setting aside the auxiliary vector, the selected method fits its direction in latent space using its original fitting rule and objective.The direction is normalized in decoded output space, and steering strength α is fixed at two in the experiments.Source: §3.2 · pp. 5–6
  • The fitted direction is applied through the transcoder’s activation-specific intervention, and its decoded output replaces the MLP output.Setting α = 0 yields the unsteered replacement model.Source: §3.1 · p. 4, §3.2 · p. 6

4 EMPIRICAL EVALUATION

The evaluation compares steering on original and STR-replacement models across three methods and three language models, measuring Corrigibility target utility and harmful-request safety. STR reduces side effects on SALAD-Bench and transfers protection to unseen safety datasets while retaining target gains.

4.1 EXPERIMENTAL SETUP

  • The study compares CAA, target-only SSV, and OPIUM on original models and STR replacements for Gemma-3-4B-it, Gemma-3-12B-it, and Llama-3.1-8B-Instruct, with α = 2 across steered conditions and unsteered baselines.SALAD-Bench supplies protection training data and a separate development split of 256 requests. The unseen test set pools 240 HarmBench, 520 AdvBench, and 313 StrongREJECT requests.Source: §4.1 · p. 6

4.2 EVALUATION METRICS

  • The evaluation uses Corrigibility preferences for target utility and harmful-request attack success rate (ASR) for safety, with higher utility and lower ASR preferred.For changes in utility and ASR, the common reference is the original model without steering; steered conditions use α = 2.Source: §4.2 · pp. 6–7

4.3 IN-DISTRIBUTION RESULTS ON SALAD-BENCH

  • On SALAD-Bench, STR lowers ASR for CAA and SSV across all three models, while every STR controller gains utility over the original unsteered baseline.SSV ASR falls from 36.12% to 5.72% on Gemma-3-4B and from 29.29% to 3.63% on Gemma-3-12B; OPIUM also reduces ASR substantially on both Gemma models.Source: §4.3 · p. 7, Table 1 · p. 7

4.4 OUT-OF-DISTRIBUTION RESULTS ON UNSEEN SAFETY DATASETS

  • On unseen safety tests, SSV lowers ASR from 42.46% to 14.42% on Gemma-3-4B and from 34.97% to 12.91% on Gemma-3-12B; CAA also reduces ASR across models.SSV’s ASR reductions are supported in all three models, with utility gains on Gemma-3-12B and Llama-3.1-8B; OPIUM’s transfer is strongest on the Gemma models.Source: §4.4 · p. 8, Table 2 · p. 8

5 RELATED WORK

Related work motivates evaluating target efficacy separately from non-target preservation and places STR alongside approaches that preserve safety or generation quality during steering. STR differs by learning a replacement mapping rather than only refining an intervention within an existing representation space.

Steering side effects and specificity

  • Studies of steering side effects show that benign compliance or formatting steering can increase jailbreak vulnerability (Xiong et al., 2026), and benign SAE-feature steering can weaken safeguards (Korznikov et al., 2025).Goyal & Daum´e III (2026) distinguish general, control, and robustness specificity, while Ong et al. (2026) characterize cross-behavior effects across 67 behaviors and three models.Source: §5 · p. 9

Preserving safety during behavioral steering

  • OPIUM addresses safety during steering by fitting a vector against utility and protection references (Aravindan et al., 2026), whereas STR trains target and protection supervision into a transcoder replacement.Steering methods fit directions on the frozen STR replacement.Source: §5 · p. 9

Preserving generation quality through selective steering

  • SAE-TS and SAE-SSV seek more selective interventions within existing representation spaces (Chalnev et al., 2024), while STR learns the replacement mapping itself.STR uses supervision on specified non-target behavior to reduce degradation under steering.Source: §5 · p. 9

6 CONCLUSION

STR replaces the steering-layer MLP with a supervised transcoder, then lets steering methods fit directions on the frozen replacement using their original objectives. Across three methods and models, SALAD-trained replacements reduce safety side effects on unseen benchmarks while retaining effective target control.

6 CONCLUSION

  • Across three steering methods and three models, SALAD-trained replacements reduce safety side effects on unseen safety sources while retaining effective target control.Target-only SSV maintains similar observed utility on Gemma-3-4B and improves both utility and safety on Llama-3.1-8B and Gemma-3-12B.Source: §6 · p. 9
  • STR trains a replacement for the steering-layer MLP, then fits steering directions on the frozen replacement with the methods’ original objectives.The surrounding model is frozen during replacement training.Source: §6 · p. 9
  • Anonymized training and evaluation code and experiment configurations are planned as supplementary material to support reproduction.The paper identifies Sections 3 and 4 and Appendix A as specifying the objective, evaluation, and configurations.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The reported attack-success reductions characterize evaluated settings and do not establish safety for unrestricted deployment.The work evaluates existing harmful-request benchmarks.Source: ETHICS STATEMENT · p. 10

A.1 MODEL INTEGRATION AND SUPERVISION

The appendix specifies model replacements and supervision, controller fitting, and inference settings. Replacement controllers are fitted in latent space, while inference applies steering across valid prefill and decode tokens without task-based routing.

A.1 MODEL INTEGRATION AND SUPERVISION

  • Gemma replacements use 65,536 JumpReLU features at layers 17 and 24; Llama-3.1-8B uses 131,072 TopK features at layer 15 with k = 128.Each replacement substitutes one MLP mapping and preserves the surrounding residual connection; layers are zero-indexed.Source: §A.1 · p. 13
  • The fixed intervention support contains K = 256 coordinates selected before supervised training from standardized positive–negative contrasts in target-data latent representations.The encoder, decoder, biases, and auxiliary vector are jointly optimized while the surrounding model and activation settings remain frozen; the three Equation 5 terms have equal coefficients.Source: §A.1 · p. 13

A.2 FITTING AND APPLYING THE CONTROLLERS

  • CAA extracts a last-content-token contrast, SSV optimizes only target loss from that direction, and OPIUM optimizes target and protection terms from it.Original-model controllers are fitted on M0; replacement controllers use the frozen replacement’s latent space, and the training auxiliary vector is not deployed.Source: §A.2 · p. 13
  • For OPIUM, the matching-layer last-valid-prompt-token hidden state under candidate direction v is compared with cached, detached CAA and unsteered reference states.The stated matching layers are 21 for Gemma-3-4B, 19 for Llama-3.1-8B, and 28 for Gemma-3-12B.Source: §A.2 · p. 13

A.3 INFERENCE CONFIGURATION

  • Steering uses α = 2 on all valid prefill and decode tokens, with the same rule for target and safety requests and no routing by task identity or harmfulness.Off conditions disable activation steering while retaining their respective model computations; safety generation is limited to 2,048 new tokens.Source: §A.3 · p. 13

B TEST RESULTS BY DATASET AND SOURCE

The appendix breaks down test results by individual safety benchmark and target-item source using completed evaluations at α = 2.

B TEST RESULTS BY DATASET AND SOURCE

B.1 SAFETY RESULTS BY BENCHMARK

The appendix reports request-averaged attack success rates separately for unseen safety benchmarks. CAA and SSV have lower ASR on the STR replacement for every benchmark and model, while OPIUM’s Llama results include an AdvBench increase.

B.1 SAFETY RESULTS BY BENCHMARK

  • Pooled ASR weights intent clusters equally and cannot be recovered by averaging the benchmark scores equally or by sample count.Source: §B.1 · p. 14
  • Table 4 reports request-averaged ASR separately for HarmBench, AdvBench, and StrongREJECT.Scores average requests equally within each dataset, then average across evaluated runs.Source: Table 4 · p. 14, §B.1 · p. 14
  • CAA and SSV have lower ASR on the STR replacement for every benchmark and model.OPIUM also lowers ASR on every benchmark for both Gemma models; on Llama, HarmBench and StrongREJECT decrease slightly while AdvBench rises from 6.09% to 6.99%.Source: §B.1 · p. 14, Table 4 · p. 14

B.2 TARGET UTILITY BY SOURCE

Table 5 breaks Corrigibility target utility into four test-item source subsets, whose item-count-weighted scores recover the overall test utility up to rounding. STR’s source-level changes vary by model and steering method.

B.2 TARGET UTILITY BY SOURCE

  • The 716 Corrigibility test items are divided into less-HHH, neutral-HHH, more-HHH, and no-goal-change source subsets.Neutral-HHH is the anthropic original lm source, and no-goal-change is persona no goal change.Source: §B.2 · p. 15, Table 5 · p. 15
  • Each entry is a source-level point estimate of mean target-option utility U; weighting the four source scores by item counts recovers Table 2’s overall test utility up to rounding.The no-goal-change estimate uses only 24 items, and these items are separate from the harmful-request benchmarks.Source: §B.2 · p. 15
  • STR–SSV improves utility on all four Llama source subsets, while CAA and OPIUM show mixed changes.On Gemma-3-4B, all three methods improve less-HHH utility but decline on the other subsets; on Gemma-3-12B, gains concentrate in less-HHH and no-goal-change, with a slight more-HHH decrease.Source: §B.2 · p. 15

Related papers

Loading 2609.32519v1…