Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods

1 INTRODUCTION

• Prior steering evaluations cover relevant dimensions only in fragments, while steering can also alter unintended behaviors and vary across inputs and training data.These challenges motivate evaluating what steering changes and when those changes hold.Source: §1 · p. 2
• SteerScope organizes 15 metrics around Steering Outcomes and Method Properties, covering target efficacy, side effects, generalization, and training-data dependence.Side effects span language quality, task capabilities, and safety and reliability.Source: §1 · p. 2
• The suite benchmarks 23 methods across four families and two models, including activation steering alongside prompting, LoRA, and SFT baselines.Source: §1 · p. 2
• Current activation steering methods do not outperform Prompt Steering in efficacy–side-effect balance on either evaluated model; none achieves higher target efficacy without greater composite side effects.The paper also reports efficacy and side effects generally rising together as intervention strength grows.Source: §1 · p. 3

2 RELATED WORK

• Representation-based activation steering intervenes on internal activations and spans directions, subspaces, latent representations, geometric transformations, hypernetworks, and nonlinear interventions.The paper groups evaluated methods by their underlying mechanisms.Source: §2 · p. 3
• AxBench evaluates 500 targets but limits off-target evaluation to instruction following and fluency; SteeringSafety focuses on nine predefined safety perspectives (Wu et al., 2025a; Siu et al., 2026).Source: §2 · p. 3
• FaithSteer-BENCH evaluates deployment-time reliability at one calibrated operating point, while CLaS-Bench extends evaluation to 32 languages and cross-lingual transfer (Ding et al., 2026; Gurgurov et al., 2026).Source: §2 · p. 3
• Existing evaluations address safety entanglement, deployment reliability, cross-lingual control, and other aspects, but remain fragmented across dimensions and methods.Source: §2 · p. 3

3 STEERSCOPE: A MULTI-DIMENSIONAL EVALUATION SUITE

• The suite’s two axes measure Steering Outcomes over candidate steering factors and Method Properties at a fixed method-level factor, using the same evaluation examples for each concept and task.Source: §3 · p. 3

3.1 STEERING OUTCOMES

• Concept Expression scores target-concept expression on a three-level scale, while side-effect metrics track changes in language quality, task capabilities, and safety and reliability.A positive metric delta indicates degradation and a negative delta indicates improvement relative to the unsteered model.Source: §3.1 · p. 4
• Task-capability measures cover language understanding, knowledge, mathematical reasoning, and instruction following; safety and reliability measures cover jailbreak and refusal, social bias, and truthfulness.Instruction Relevance assesses whether a response addresses its original instruction; IFEval is reported separately from Overall.Source: §3.1 · p. 4

3.2 METHOD PROPERTIES

• Method Properties assess retention of the Overall steering effect under cross-lingual prompt shifts and dependence on training-data amount and composition.Sample Efficiency normalizes reduced-data improvement by full-data improvement, while Sample Sensitivity measures variation across same-sized subsets.Source: §3.2 · pp. 4–5
• The method-level factor is selected on the full in-distribution evaluation set, which may introduce optimism in operating-point calibration, but is then held fixed across property experiments.Source: §3.2 · p. 5

4 STEERING METHODS

• The benchmark evaluates 23 representative methods and baselines, plus a Random control, selected to support method- and mechanism-level analysis.Source: §4 · p. 5

4.1 OPTIMIZATION-FREE STEERING METHODS

• Optimization-free methods construct directions from activation patterns, use geometric or unit-level interventions, or steer with sparse-autoencoder features.Examples include DiffMean, PCA, LAT, Spherical Steering, HiDRA, AUSteer, SAE, and SAE-A.Source: §4.1 · pp. 5–6

4.2 DIRECTLY OPTIMIZED STEERING METHODS

• Directly optimized methods learn steering directions, subspaces, or adaptive interventions through supervised training, preference optimization, or learned nonlinear functions.The evaluated approaches include Linear Probe, SSV, ReFT-r1, LoReFT, PSR variants, RePS, ODESteer, and StepODESteer.Source: §4.2 · p. 6

4.3 HYPERNETWORK-BASED STEERING METHODS

• Hypernetwork-based methods use shared auxiliary networks to generate interventions conditioned on natural-language targets, either as steering vectors or velocity fields.Source: §4.3 · p. 6

4.4 NON-REPRESENTATION-BASED STEERING METHODS

• Non-representation baselines include Prompt Steering and Simple Prompt Steering, which prepend concept-conditioned contexts, plus SFT and LoRA parameter-update methods.Source: §4.4 · p. 6

5 EXPERIMENTS

• The experiments use Gemma-2-2B-it and Gemma-2-9B-it at residual-stream layer 20, with CONCEPT500 contrastive training data and shared evaluation examples where applicable.DeepSeek-V3.2 generated 72 contrastive pairs per concept; SFT used 20 concepts, and costly metrics used a shared 50-concept subset.Source: §5 · pp. 6–7

5.1 EFFICACY AND SIDE EFFECTS

• Across both model scales, no activation steering method dominates Prompt Steering on the efficacy–side-effect trade-off when broader side effects are considered.Figure 2 plots Concept Expression against the equal-weight mean of eleven range-normalized side-effect degradations.Source: Figure 2 · p. 7, §5.1 · p. 7
• At Overall-selected factors, A-PSR improves Concept Expression by 1.20 and 1.30 points, with composite side effects of 0.11 and 0.12 on Gemma-2-2B-it and Gemma-2-9B-it, respectively.Source: §5.1 · p. 7
• Optimization-free methods generally have less favorable efficacy–side-effect trade-offs than prompting and methods with learned interventions.The authors suggest prompting leverages instruction-following capabilities and learned interventions incorporate task-specific feedback, unlike directions derived from representation statistics or pretrained features.Source: §5.1 · p. 7
• Some low-strength interventions improve off-target metrics, and reversing their direction removes or reverses those gains, suggesting direction-dependent conceptual entanglement.Source: §5.1 · p. 7
• Method rankings differ across individual side-effect metrics, while ranking agreement is generally stronger on Gemma-2-9B-it than on Gemma-2-2B-it.Source: §5.1 · p. 8

5.2 DATA DEPENDENCE

• With few examples, HyperSteer and DiffMean retain relatively large fractions of their full-data effects, whereas A-PSR, FLAS, LoRA, and ReFT-r1 need more data to approach theirs.Most methods approach their full-data effect as data increases, though improvement is not always monotonic; full-data performance alone does not reveal data dependence.Source: §5.2 · pp. 8–9

5.3 GENERALIZATION

• Under OOD prompts, target efficacy changes modestly for most methods, while Instruction Degradation rises for 11 of 13 and Fluency Degradation for 9 of 13 methods shown.Across both model scales, activation steering methods tend to have lower Cross-Lingual Retention than Prompt Steering, with substantial variation by method and model.Source: §5.3 · p. 9

• SteerScope organizes LLM steering evaluation into Steering Outcomes and Method Properties, and compares strength-dependent efficacy–side-effect curves across methods and model scales.Its taxonomy covers target efficacy, side effects, and method properties under distribution and training-data changes.Source: §6 · p. 9
• Activation steering methods that outperform Prompt Steering on existing benchmarks do not necessarily retain that advantage when broader side effects are considered, and stronger efficacy generally accompanies larger unintended effects.Source: §6 · pp. 9–10
• Under OOD prompts, target efficacy is often preserved while side effects become more pronounced; learned interventions tend to trade off more favorably than optimization-free approaches.The conclusion also reports substantially different sample-efficiency profiles across methods.Source: §6 · p. 10

• DeepSeek-V3.2-Instruct generated concept-conditioned positive and negative training responses and concept-specific instructions for Prompt Steering, A-PSR, and S-PSR.Source: USE OF GENERATIVE AI · p. 10
• GPT-4o-mini scored concept expression, instruction relevance, and fluency; Llama-3.1-8B-Instruct evaluated JailbreakBench responses.Source: USE OF GENERATIVE AI · p. 10
• The authors also used generative AI to polish the manuscript and take responsibility for generated materials and LLM-based judgments.Source: USE OF GENERATIVE AI · p. 10

REPRODUCIBILITY STATEMENT

• The reproducibility statement directs readers to the codebase and appendices for metric definitions, method configurations, factor grids, inference settings, and evaluation prompts.It also points to the training data, models, and experimental setup described in Section 5.Source: REPRODUCIBILITY STATEMENT · p. 10

A METRIC DETAILS

• The metric-details appendix organizes its material into side effects, generalization, and data dependence.Source: A Metric Details · p. 16

C Experimental Details

• The experimental-details appendix lists shared model setup, method configurations, steering-factor grids, evaluation configurations, prompt templates, and prior judge validation.Source: C Experimental Details · p. 16

D Additional Results

• The additional-results appendix lists analyses of composite side effects, individual side-effect dimensions, sample efficiency, sample sensitivity, and cross-lingual generalization.Source: D Additional Results · p. 16

A Metric Details

• The metric suite comprises Concept Expression, eleven side-effect metrics, Cross-Lingual Retention, Sample Efficiency, and Sample Sensitivity.Together these make up SteerScope's 15 metrics.Source: A METRIC DETAILS · p. 17

A.1 SIDE EFFECTS

• Side-effect scores use concept-seeded evaluation examples, with each method and factor tested on the same examples for a given concept.The procedure averages scores within method–concept–factor cells, compares each factor with factor 0, and normalizes degradation by each metric's range; positive values indicate degradation, while negative values indicate improvement.Source: §A.1 · p. 17

A.2 GENERALIZATION

• Cross-lingual retention compares steering effects relative to the unsteered model within each language, using English as ID and four languages as OOD.The retention score compares OOD and ID Overall effects; values below 1 indicate partial loss and values above 1 indicate a stronger OOD effect, but the ratio can be unstable near a zero denominator.Source: §A.2 · pp. 18–19

A.3.1 SAMPLE EFFICIENCY

• Sample Efficiency reports the percentage of full-data Overall improvement recovered with reduced training sets of 6, 12, 36, or 72 labeled rows per concept.The full training set contains 144 labeled examples; values can exceed 100%, and the measure is withheld when the full-data improvement is close to zero.Source: §A.3.1 · p. 19

A.3.2 SAMPLE SENSITIVITY

• Sample Sensitivity measures variation in Overall improvement across training-data compositions at a fixed training-set size.Source: §A.3.2 · p. 19
• The evaluation fixes training size at 24 labeled rows per concept and varies five subset seeds while holding the training initialization seed and method-level steering factor fixed.Each subset's Overall score uses the same 20 evaluation prompts per concept.Source: §A.3.2 · p. 19
• Sensitivity is the macro-average of concept-level sample standard deviations across the five subsets; lower values indicate more consistent improvement across data subsets.The authors report sensitivity alongside mean signed Overall gain to distinguish consistent strong effects from consistently weak or detrimental ones.Source: §A.3.2 · p. 20

B CATALOGUE OF STEERING METHODS

• Table 3 is a catalogue of steering methods and their mechanisms.Source: Table 3 · p. 21

C.1 MODELS AND SHARED SETUP

• Experiments use instruction-tuned Gemma-2 models at 2B and 9B scales, with single-site interventions placed at layer 20 in both models.Training uses 72 contrastive response pairs per concept, equivalent to 144 labeled examples; SFT is the exception to the full CONCEPT500 setting and uses 20 concepts.Source: §C.1 · p. 22

C.2 METHOD CONFIGURATIONS

• Method configurations specify training or construction settings; for example, RePS uses reference-free scaled SimPO, while ODESteer uses polynomial features and a logistic-regression classifier.Source: §C.2 · pp. 22–23, Table 4 · p. 22

C.3 STEERING FACTOR GRIDS

• Table 5 lists candidate inference factors separately for Gemma-2-2B-it and Gemma-2-9B-it, grouping methods that share a grid.Source: §C.3 · p. 23, Table 5 · p. 24

C.4 EVALUATION CONFIGURATIONS

• The evaluation uses a shared 50-concept subset; SuperGLUE draws 20 examples per task, MATH uses Level 1 problems, and SFT is evaluated on 20 training concepts.Generalization uses 13 text concepts on Gemma-2-2B-it and 15 on Gemma-2-9B-it; TruthfulQA uses binary choice and JailbreakBench responses are judged by Llama-3.1-8B-Instruct.Source: Table 6 · p. 24

C.5.1 CONTRASTIVE TRAINING-DATA GENERATION

• Contrastive pairs are generated from the same source instruction: the positive response incorporates elements related to the concept, while the negative continuation avoids mentioning it regardless of coherence.Source: §C.5.1 · pp. 25–26

C.5.2 PROMPT GENERATION FOR PROMPT STEERING AND PSR VARIANTS

• Prompt Steering, A-PSR, and S-PSR share a generated concept-specific instruction; Prompt Steering prepends it, while the PSR variants use learned interventions.The template asks for responses related to the concept, even when that is out of context.Source: §C.5.2 · p. 26

C.5.3 SIMPLE PROMPT STEERING

• Simple Prompt Steering uses a fixed instruction to require concept-related content, which is combined with the task input before the chat template is applied.The instruction requests concept-related content even when unrelated to the question or nonsensical.Source: §C.5.3 · p. 26

C.5.4 LM-JUDGE TEMPLATES

• The Concept Expression, Instruction Relevance, and Fluency judges use ternary 0–2 scores, with prompts focusing each judgment on its own criterion.Concept Expression ignores grammar and coherence; Instruction Relevance rates connection to the original instruction, while Fluency targets readability and naturalness.Source: §C.5.4 · p. 27

C.6 PRIOR HUMAN VALIDATION OF AXBENCH-STYLE JUDGES

• AxBench-style LLM judges had Pearson correlation 0.58 with human ratings, comparable to the 0.57 correlation among human raters.SteerScope complements these scores with task-specific benchmarks and deterministic evaluators where available.Source: §C.6 · pp. 27–28

D.1 COMPOSITE SIDE EFFECTS

• Additional factor-sweep plots show mean side effect against Concept Expression and normalized intervention strength on both Gemma-2 model scales.A separate plot charts Concept Expression against normalized intervention strength.Source: D ADDITIONAL RESULTS · p. 28, Figure 5 · p. 29, Figure 6 · p. 30

D.1.1 ROBUSTNESS TO METRIC WEIGHTING

• Prompt Steering remains undominated in 99.999% of Dirichlet(1) draws on Gemma-2-2B-it and 99.92% on Gemma-2-9B-it at each method’s Overall-selected factor.The comparison is less robust under highly sparse preferences but remains consistent across both model scales.Source: §D.1.1 · p. 32

D.2 RANKING AGREEMENT ACROSS SIDE-EFFECT METRICS

• Side-effect metrics yield inconsistent method rankings; false refusal is weakly or negatively correlated with most other metrics.MMLU gains from some positive interventions disappear or become degradations when the steering direction is reversed.Source: §D.2 · pp. 32–33, §D.3 · p. 33

D.4 LANGUAGE QUALITY

• The appendix provides complete factor-sweep plots for Instruction Relevance and Fluency on both Gemma-2 model scales.The plots show normalized degradation versus Concept Expression and absolute metric values versus normalized intervention strength.Source: Figure 10 · p. 34, Figure 11 · p. 35, Figure 12 · p. 36

D.5 TASK CAPABILITIES

• Factor-sweep figures report MMLU accuracy and aggregate SuperGLUE results on both Gemma-2 model scales.They show normalized degradation against Concept Expression and absolute metric values across normalized intervention strength.Source: Figure 14 · p. 38, Figure 15 · p. 39, Figure 16 · p. 40
• Additional factor-sweep figures cover MATH accuracy and IFEval prompt-strict accuracy on Gemma-2-2B-it and Gemma-2-9B-it.Each figure presents normalized degradation against Concept Expression and absolute values against normalized intervention strength.Source: Figure 18 · p. 42, Figure 19 · p. 43, Figure 20 · p. 44

D.6 SAFETY & RELIABILITY

• Safety and reliability factor-sweep plots include BBQ ambiguous bias and JailbreakBench attack success rate on both model scales.The figures plot normalized degradation against Concept Expression and absolute values against normalized intervention strength.Source: Figure 22 · p. 46, Figure 23 · p. 47, Figure 26 · p. 50

D.7 SAMPLE EFFICIENCY

• Sample-efficiency plots compare factor-0-relative Overall improvement at 6, 12, 36, 72, and 144 labeled rows per concept.Recovery of full-data improvement is shown only for methods whose full-data Overall improvement is at least 0.1.Source: Figure 32 · p. 56, Figure 33 · p. 56, §D.7 · p. 56

D.8 SAMPLE SENSITIVITY

• Prompt Steering, Simple Prompt Steering, and SAE have zero sample sensitivity; other evaluated methods vary across training subsets, and complete cross-lingual results are tabulated for both models.Cross-lingual tables use one fixed method-level steering factor across languages.Source: §D.8 · p. 57, §D.9 · p. 57, Table 10 · p. 58