PaperlayerView exact revision on arXiv

Source-linked AI summary

Orthogonal and Parametrized Multi-Attribute Steering of LLMs

Remy Ogasawara, Cullen Anderson, Narmeen Fatimah Oozeer, Foad Namjoo, Amir Abdullah, Jens Kristian Refsgaard Schou, Jeff M. Phillips

arXiv:2610.37517v1cs.LGcs.AI

TL;DR

Activation steering lacks a standard, interpretable scale, and combining correlated trait vectors can cause interference. The paper calibrates steering intervals, sets activations to desired trait strengths with parameterized steering, and orthogonalizes vectors; its multi-trait methods mostly decouple sycophancy from warmth while increasing warmth relative to baseline approaches.

  • Problem

    Steering magnitudes have no standard interpretation and can produce different effects across vectors, while simultaneous steering of multiple traits can push generations out of distribution.

    Source: §1 · p. 2

  • Method

    The paper identifies and rescales each vector’s steerable interval, introduces parameterized steering, and orthogonalizes vectors for multi-trait steering.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Orthogonal multi-trait methods mostly decouple sycophancy from warmth and raise warmth to 42 or 43% from 26.5 or 30% with baseline approaches.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The methods make steering strengths and interactions between multiple steering vectors more predictable, but improved controllability does not inherently imply improved safety.

    Source: §6 · p. 10, Ethics statement · p. 11

    Steering vectors learned from behavioral contrasts may encode unintended correlations, biases, or dataset artifacts in addition to the intended attribute.

    Source: Ethics statement · p. 11

Abstract

from arXiv · show

Activation steering offers an effective and efficient way to modify an LLM's behavior at inference time. However, it remains unclear how to apply steering such that the signal is consistently strong and the steering strength has a universally interpretable meaning. We address this first by identifying the most steerable interval for each steering vector and rescaling the steering range accordingly. Building on this, we introduce a more interpretable method for steering, called parameterized steering which instead of pushing activations in a direction, it sets them at the desired trait strength. Next, with multi-trait steering, we find that naively combining vectors causes them to interfere with one another. In response, we develop a method to orthogonalize the vectors; it decouples these traits, allowing a model to exhibit one behavior without inducing the others, thus enabling multiple traits to be steered simultaneously.

1 Introduction

Activation steering lacks a consistent, interpretable strength scale, and steering correlated traits can push generations out of distribution or make one trait difficult to control independently. The paper calibrates steering ranges, introduces parameterized steering, and orthogonalizes vectors to reduce multi-trait interference.

1 Introduction

  • Warmth and sycophancy can point in similar directions, so steering one may inadvertently induce the other and make the target space difficult to interpret.The paper proposes orthogonalizing steering vectors and transforming activation space into a grid whose axes represent controllable trait ranges.Source: §1 · p. 2
  • Negative sycophancy steering reduces both sycophancy and warmth, while adding warmth increases warmth without much affecting already near-saturated sycophancy.The example motivates controlling correlated traits separately rather than assuming one steering vector changes only its target trait.Source: §1 · p. 2
  • Orthogonal multi-trait methods mostly decouple sycophancy from warmth, raising warmth to 42 or 43% from 26.5 or 30% with baseline approaches, though it remains below the 61.5% baseline.The comparison concerns OAI multi and OP multi versus baseline approaches; the authors note that warmth is partly tied intrinsically to sycophancy.Source: Table 1 · p. 3, §1 · p. 3
  • The paper identifies each vector’s steerable bounds, introduces parameterized steering, and proposes orthogonalization for multi-attribute steering.The motivating psychotherapy example distinguishes warmth from sycophancy: warmth supports understanding, while sycophancy can mean agreeing without considering facts or warnings.Source: §1 · pp. 2–3

2 Preliminaries

The preliminaries describe activation steering as a linear intervention on model activations and outline the paper’s evaluation setup. Experiments primarily use Llama 2 7B Chat and assess behavior with logit-based metrics.

2.1 Identifying Steering Directions and the Linear Representation Hypothesis

  • Steering vectors modify residual-stream token activations to elicit target behavior, building on the hypothesis that traits are represented by linear patterns.The paper uses contrastive activation addition as its steering approach.Source: §2.1 · p. 3
  • Traditional α-iterative steering shifts an activation x along direction u by scalar α, typically 1.0, using x+ ← x + αℓ_u u.This is the default approach the paper compares against.Source: §2.1 · p. 3

2.2 Experiment Setup

  • Experiments primarily use Llama 2 7B Chat at the layer identified by prior work, confirm results on Qwen 3 8B, and draw on behavioral datasets and TruthfulQA, Toxigen, and BBQ.The behavioral datasets include sycophancy and advanced AI risk evaluations, plus a synthetic warmth dataset.Source: §2.2 · p. 3
  • Average_score is the mean logit difference between behavior-matching and non-matching choices, with higher values indicating stronger behavior; open-ended generations are also evaluated by an LLM judge.Percent_steered measures how often the model assigns the higher logit to the behavior-matching choice.Source: §2.2 · p. 4

3 Identifying the Steerable Region

The paper selects a steerable interval for each vector, seeking a region that contains zero, spans the global extrema, and rises approximately linearly and monotonically. It then remaps that interval to a calibrated α range from −1 to +1.

3 Identifying the Steerable Region

  • The selected interval contains zero, spans global minimum and maximum scores, and is approximately linear and monotonically increasing despite noise.The loss favors endpoints near the global extrema while rewarding a steep positive slope; γ controls that trade-off.Source: §3 · p. 4
  • Figure 2 compares held-out avg_score before and after learning the steering interval on training data.For the power-trait example, the described remapping moves the curve’s nadir and peak to −1 and +1, respectively.Source: Figure 2 · p. 4, §3 · p. 5
  • The method maps each learned interval [A, B] linearly onto α ∈ [−1, +1], with γ = 0.1 selected for the interval search.Most γ values from 0.1 to 0.3 give about the same output.Source: §3 · p. 5

4 Parameterized Steering

Parameterized steering replaces a fixed activation shift with a coordinate-based target, setting activations to a desired trait strength. It is more interpretable and performs similarly in steerability to α-iterative steering.

4 Parameterized Steering

  • Parameterized steering uses α as a coordinate to set an activation’s desired strength, rather than as a scalar multiplier that pushes it in a direction.The approach is described using two-means encodings and transfers to the clipped version used in practice.Source: §4 · p. 5
  • For each activation, a different α′ can move it to the same position along u as the target encoding.The target can be any α between −1 and +1, with affine extension to any real α.Source: §4 · p. 5
  • Parameterized and α-iterative steering perform similarly in steerability across several datasets.The paper describes parameterized steering as more interpretable because α maps more clearly to the activation’s final destination and is idempotent.Source: §4 · p. 5, Figure 3 · p. 5

5 Multi-Attribute Steering with Orthogonalization

The section transforms correlated steering directions into independent concept coordinates, then tests how orthogonal parameterized steering changes multi-trait sensitivity, range, and order dependence. The experiments report broader steering ranges and lower sensitivity than α-iterative steering, and compare the methods with MAT-Steer.

5 Multi-Attribute Steering with Orthogonalization

  • A low-rank transformation orthogonalizes correlated steering directions, assigning each trait its own coordinate while preserving the activation component outside their span.The construction changes concept coordinates without changing the component outside span(U).Source: §5 · pp. 6–7
  • In parameterized space, each concept coordinate represents positive polarity at +1, negative polarity at −1, and neutrality at 0, so traits can be set independently.The remaining d − k coordinates are unchanged, and the transformation is invertible.Source: §5 · p. 7

5.1 Experiments

  • Across correlated pairs, α-iterative sensitivity averages 1.5 versus under 1 for parameterized methods, whose ranges are about 3.2/3.0 versus α-iterative's 1.5/1.8.The range comparison is reported at both secondary-trait endpoints; orthogonal parameterization slightly improves on parameterization at β = +1, with ΔP = 0.47.Source: Table 3 · p. 8, §5.1 · pp. 8–9, Table 2 · p. 8
  • Without orthogonalization, parameterized power control depends on steering order; orthogonal parameterized steering removes order effects, with mild dependence on second-trait strength.The Figure 5 comparison evaluates power while varying coordination and shows the order effect for parameterized steering but not for orthogonal parameterized steering.Source: Figure 5 · p. 9, §5.1 · pp. 9–10

5.2 Comparison to MAT-Steer

  • Across TruthfulQA, Toxigen, and BBQ, parameterized methods gain about 10 percentage points over no-steer; MAT-Steer comparison is constrained by an unreproduced split and code.The paper reports that the parameterized methods perform most effectively, while noting that its baseline is significantly higher than MAT-Steer's reported baseline.Source: Table 4 · p. 10, §5.2 · p. 10

6 Discussion & Conclusion

The paper formalizes clipping, parameterized steering, and orthogonalization as ways to make steering more robust and interpretable. It also notes remaining limitations, potential misuse and unintended correlations, and reproducibility and AI-use disclosures.

6 Discussion & Conclusion

  • The paper formalizes clipping the steering range, parameterizing steering strength, and orthogonalizing vectors for multi-trait steering.It argues these geometric optimizations can make steering more robust, interpretable, and generalizable.Source: §6 · p. 10
  • Steering does not control all behavior or eliminate multi-steering interference, and the warm-but-not-sycophantic training quadrant remains mostly devoid of examples.The authors suggest nonlinear approaches, steering across multiple layers, more careful dataset creation, or cleaner activation embeddings as possible directions for improvement.Source: §6 · p. 10

AI use statement

  • The authors used generative AI for writing, code, dataset generation, and evaluations, while stating that they designed the steering methods and experiments without AI assistance.The warmth-dataset generation code was not reviewed line by line; the dataset was checked with automated tests rather than manual review of every response.Source: AI use statement · p. 11

Ethics statement

  • More precise steering may support beneficial uses as well as misuse, including amplifying undesirable behavior or suppressing safety-relevant behavior.Steering vectors may also encode unintended correlations, biases, or dataset artifacts, and the authors call for research on detecting changes, assessing effects outside targeted evaluations, and developing safeguards.Source: Ethics statement · p. 11

Reproducibility statement

  • The authors provide code, experiment scripts, and reproduction instructions, with supplementary materials and appendices detailing methods and evaluation protocols.The repository includes an end-to-end pipeline for activation extraction, steering experiments, steerable-interval computation, and multi-attribute experiments.Source: Reproducibility statement · p. 11

A Clipping the Steerable Range

The clipping procedure identifies a steerable segment for each result. The choice of γ balances excluding flat regions against prematurely cropping the interval.

A Clipping the Steerable Range

  • γ = 0.1 balances the risk of prematurely cropping the interval against including too much of the flat curve.Higher γ emphasizes a high, increasing slope; lower γ can include too much of the flat curve.Source: A Clipping the Steerable Range · p. 15
  • The clipping choice trades off retaining a sufficiently increasing slope against avoiding an overly narrow interval.The authors describe these opposing effects when varying γ.Source: A Clipping the Steerable Range · p. 15
  • Figure 7 marks the best clipped segment of each steering result as its identified steerable region.The solid region indicates the selected segment.Source: Figure 7 · p. 15

B Existing Datasets

The appendix describes binary multiple-choice dataset preparation and an efficient orthogonalization procedure. The latter operates in the span of the steering directions, reducing computational costs when the number of directions is small relative to the activation dimension.

B Existing Datasets

  • The three source datasets are converted to binary multiple-choice items by randomizing answer positions, preventing positional bias from confounding steering or evaluation.TruthfulQA pairs a truthful and an untruthful answer; ToxiGen pairs benign and toxic continuations; BBQ pairs a correct answer with a randomly sampled incorrect answer.Source: B Existing Datasets · pp. 15–16

C Efficient Orthogonalization

  • The orthogonalization procedure builds an orthonormal basis for the steering-vector span, maps activations to concept coordinates, and preserves the orthogonal residual during steering.Changing one concept coordinate does not alter the others, and reconstruction adds the preserved residual back to the modified span coordinates.Source: C Efficient Orthogonalization · pp. 16–17

C.1 Computational Complexity

  • After preprocessing, the procedure takes O(dk + k^2) time per activation and O(d + k) additional working memory.The compared dense transform requires O(d^3) preprocessing, O(d^2) storage, and O(d^2) time per activation; the stated improvement applies when k ≪ d.Source: §C.1 · p. 17

C.2 Orthogonalization and Reconstruction Properties

Orthogonalization separates steering concepts into coordinates while retaining a lossless representation of the activation. Parameterized steering then normalizes concept coordinates so positive and negative training means map to +1 and −1.

C.2 Orthogonalization and Reconstruction Properties

  • For any target concept-coordinate vector, reconstruction realizes that target while preserving the original residual and changing the activation only within span(U).This exact, localized steering result holds under the theorem’s assumptions, including full column rank of the steering directions.Source: §C.2 · pp. 17–18
  • Steering distinct concepts commutes, so the order of coordinate edits does not matter in the orthogonal setting.The authors contrast this with non-orthogonal parameterized steering, where a second operation can overwrite the first effect.Source: §C.2 · pp. 18–19
  • Linear operations such as computing means remain consistent across the original, orthogonal, and parameterized spaces.With full-rank steering vectors, any values for the k concepts can be realized; adjusting only those coordinates changes the representation within a k-dimensional subspace.Source: §C.2 · p. 19

C.3 Parameterized Operation and its Inverse

  • The coordinate-wise affine reparameterization places each concept’s positive and negative training means at +1 and −1, respectively, with their midpoint at 0.The map is defined as invertible when its diagonal scaling is nonsingular.Source: §C.2 · p. 18, §C.3 · p. 19

D Additional Tables with Percent Steered

These additional tables report steering results using percent_steered rather than avg_score. They cover conditional variability and multi-steering range.

D Additional Tables with Percent Steered

  • The appendix tables use percent_steered instead of the avg_score metric used in Tables 2 and 3.Source: D Additional Tables with Percent Steered · p. 20
  • Table 6 reports mean conditional standard deviation of percent_steered across β while α is fixed; lower values are better.It compares parameterized and orthogonal parameterized steering.Source: Table 6 · p. 20
  • Table 7 reports the multi-steering range of percent_steered in the α trait, with higher values better.Its comparisons are OP–AI and OP–P.Source: Table 7 · p. 20

E Replicating Results on Qwen 3 8B

On Qwen3-8B, orthogonal parameterized steering performs comparably to its Llama-2 baseline while retaining a wider steering range than the non-orthogonal baseline. The appendix also verifies steering with an LLM judge and examines the warmth dataset’s construction and remaining limitations.

E Replicating Results on Qwen 3 8B

F LLM as a Judge Verification

G Warmth dataset creation

  • Figure 9 shows warmth and sycophancy clusters sharing a diagonal before orthogonalization and separating along their axes afterward.However, high-sycophancy and low-warmth class means remain closely aligned because the procedure removes only the vectors’ shared linear projection.Source: G Warmth dataset creation · p. 24, Figure 9 · p. 24
  • Warmth responses were designed to vary interpersonal delivery while keeping factual content, agreement, recommendations, and refusals fixed across five levels.The levels run from very cold to very warm, labeled −2 through +2.Source: G Warmth dataset creation · pp. 24–25
  • Length remained predictive of warmth labels despite controls intended to prevent the vector from learning that cold means short and warm means long.The authors say those controls were only partially successful.Source: G Warmth dataset creation · p. 25
  • The warmth dataset contains 1,000 prompt families and 5,000 responses, generated with balanced protocols across labels.Smaller batches were used after larger pilot batches showed visibly less linguistic and structural variation.Source: G Warmth dataset creation · p. 25

G.1 Automated Dataset Validation

Automated checks find that the warmth dataset preserves reference content in most responses and supports recoverable, broadly ordered warmth labels. They also identify residual surface-form shortcuts, especially response length.

G.1 Automated Dataset Validation

  • Across 5,000 responses, all six reference criteria were satisfied in 90.2% and all required facts were preserved in 94.7%.All five responses satisfied the complete reference in 73.8% of families; fidelity was lower in the first three protocol rounds than in the final 700-family protocol.Source: §G.1 · p. 25
  • Among 503 families whose references called for contradicting the user, no response at any warmth level endorsed the user’s belief.Source: §G.1 · p. 25
  • TF–IDF classifies warmth at 74.3% accuracy on the three-class task and 56.1% on the five-class task, above majority-class baselines of 40% and 20%.ModernBERT-based accuracy is 72.4% and 53.4%, respectively.Source: §G.1 · p. 26, Table 13 · p. 26
  • The warmth scale has a strong broad cold–warm ordering, but distinctions between adjacent levels are less pronounced.ModernBERT-based scores correctly order 83.7% of within-family pairs and 71.3% of adjacent-label pairs; all five responses are strictly ordered in 17.9% of families.Source: §G.1 · p. 26
  • A seven-feature surface-form classifier reaches 65.6% three-class and 43.1% five-class accuracy, while word count alone reaches 48.3% and 23.6%.The authors identify response length as a residual shortcut; the +2 response is longer than the −2 response in 79.5% of families, by a mean of 7.7 words.Source: §G.1 · p. 26
  • The dataset provides a strong cold–warm endpoint contrast while retaining some semantic and surface-form variation, particularly in earlier generation rounds.Source: §G.1 · pp. 25–26

Related papers

Loading 2610.37517v1…