Source-linked AI summary
Activation Scaling for Steering and Interpreting Language Models
Niklas Stoehr, Kevin Du, Vésteinn Snæbjarnarson, Robert West, Ryan Cotterell, Aaron Schein
TL;DR
The paper asks how to identify task-influential model components, arguing that successful intervention is a prerequisite for interpreting them. It learns activation scalars under effectiveness, faithfulness, and minimality objectives, finding them as effective and faithful as steering vectors while requiring fewer parameters.
Problem
The paper asks which model components influence task-specific behavior and treats successful intervention as a prerequisite for interpreting them.
Source: §1 · p. 1Method
The paper learns multiplicative activation scalars at model locations using gradient-based optimization of effectiveness, faithfulness, and minimality objectives.
Source: §1 · p. 2Results
Takeaways & Limitations
Scaling signed activation magnitudes can yield sparse, localized descriptions of components by strengthening or weakening steering directions already encoded in the model.
Source: §1 · p. 2, §7 · p. 9Evaluation is limited to small models and small, synthetic tasks; the largest studied models are GPT2-XL (1.5 billion parameters) and Pythia-1.4B (1.4 billion parameters).
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Given the prompt "Rome is in", can we steer a language model to flip its prediction of an incorrect token "France" to a correct token "Italy" by only multiplying a few relevant activation vectors with scalars? We argue that successfully intervening on a model is a prerequisite for interpreting its internal workings. Concretely, we establish a three-term objective: a successful intervention should flip the correct with the wrong token and vice versa (effectiveness), and leave other tokens unaffected (faithfulness), all while being sparse (minimality). Using gradient-based optimization, this objective lets us learn (and later evaluate) a specific kind of efficient and interpretable intervention: activation scaling only modifies the signed magnitude of activation vectors to strengthen, weaken, or reverse the steering directions already encoded in the model. On synthetic tasks, this intervention performs comparably with steering vectors in terms of effectiveness and faithfulness, but is much more minimal allowing us to pinpoint interpretable model components. We evaluate activation scaling from different angles, compare performance on different datasets, and make activation scalars a learnable function of the activation vectors themselves to generalize to varying-length prompts.
1 Introduction
The introduction argues that model components can be understood as task-influential only when interventions alter task behavior. It proposes model-wide scalar interventions optimized for effectiveness, faithfulness, and minimality.
1 Introduction
- Intervention is a necessary test of task-specific influence, but effective edits alone may not identify a uniquely influential location.The introduction frames this as a challenge for connecting intervention to interpretability.Source: §1 · p. 1
- The objective defines effectiveness as flipping correct and incorrect answer predictions, faithfulness as preserving unrelated-token probabilities, and minimality as sparse changes across locations.The three desiderata are operationalized with a gradient-based objective.Source: §1 · p. 2
- Activation scaling learns a scalar for each of many model locations and steers behavior by scaling the activation vector at each location.The intervention is model-wide but parsimonious.Source: §1 · p. 2
- Activation scaling is overall as effective and faithful as STEERVEC while requiring far fewer learnable parameters.Source: §1 · p. 2
- Dynamic activation scalars are functions of activation vectors, allowing learned interventions to transfer to test prompts of varying length.Scalars strengthen or weaken steering directions already encoded in the model, yielding sparse, localized descriptions of important components.Source: §1 · p. 2
2 Transformer Language Models
This section introduces autoregressive language models and the Transformer components used to compute their outputs. It also identifies two datasets used for mechanistic interpretability experiments.
2 Transformer Language Models
- A language model is a probability distribution over token strings, with an autoregressive factorization.Source: §2 · p. 2
- In autoregressive use, a prompt is the preceding token sequence, and the model defines a local next-token distribution.Source: §2 · p. 2
- Transformer models compute outputs through layers containing attention and MLP components that read from and write to the residual stream.Source: §2 · pp. 2–3
- Table 1 covers country-capital conflicts (CCC) and Indirect Object Identification (IOI), which requires weak syntactic reasoning.Source: Table 1 · p. 3
- Activation vectors are indexed by layer, token position, and site; layer-wise matrices stack them across token positions.Source: §2 · p. 3
3 Activation-Level Interventions
The section defines interventions over selected layer, token, and site locations, then contrasts additive vectors with scalar scaling. It learns and evaluates interventions using effectiveness, faithfulness, and minimality objectives.
3.1 Choosing Intervention Points
- Intervention points are the Cartesian product of selected layer indices, token positions, and sites.The chosen granularity depends on the use case and reflects the aim of interpreting larger Transformer components.Source: §3.1 · p. 3
3.2 Defining an Intervention
- STEERVEC adds a learned vector at each intervention point, whereas ACTIVSCALAR scales each activation vector's signed magnitude with one scalar.The scalar intervention is more parameter-efficient and is restricted to scaling rather than changing activation-vector direction.Source: §3.2 · p. 3
3.3 Learning an Intervention
- ACTIVSCALAR does not fall behind STEERVEC in effectiveness and faithfulness, while being much more minimal on average.Figure 2 compares the methods across hyperparameter combinations on GPT2-Small, intervening at attnOut and mlpOut across layers and token positions.Source: Figure 2 · p. 4
- The learning objective combines effectiveness, faithfulness, and minimality, with a grid search over λF, λM, and m to examine their trade-offs.The task prompts the model to choose between correct and incorrect answer tokens; the effectiveness objective seeks to flip their relative logits in both intervention directions.Source: §3.3 · pp. 3–4
3.4 Evaluating an Intervention
- Evaluation measures effectiveness with the margin set to zero, uses the faithfulness objective directly, and counts non-negligible parameters for minimality.Parameters with absolute values below 0.01 are treated as negligible.Source: §3.4 · p. 5
4 Experiments
On synthetic CCC and IOI tasks, the experiments compare activation scaling with additive steering vectors under an objective balancing effectiveness, faithfulness, and minimality. Activation scaling matches steering vectors on effectiveness–faithfulness trade-offs while using fewer parameters and yielding interpretable scalar patterns; its dynamic variant generalizes to varying prompt lengths.
4.1 Tasks
- Experiments fit ACTIVSCALAR and STEERVEC to the three-term objective and assess effectiveness, faithfulness, and minimality on equal-length CCC and IOI prompts with single-token answers.CCC tests conflicts between in-context information and parametric knowledge; IOI tests identifying indirect objects in potentially complex sentences.Source: §4.1 · p. 5, §4 · p. 5
4.2 Quantitative Results
- A grid search over λF, λM, and m maps test-set Pareto-optimal solutions to expose trade-offs among effectiveness, faithfulness, and minimality.Source: §4.2 · p. 5
- About 85% of effectiveness–minimality Pareto-frontier points favor ACTIVSCALAR, which also matches STEERVEC’s effectiveness–faithfulness trade-off and uses 912 versus 1,459,200 parameters in the GPT2-XL example.STEERVEC is generally more effective on training data but shows large test-set performance decreases, likely because it has more parameters and is more prone to overfitting.Source: §4.2 · p. 5
4.3 Interpretation of Scalars and Vectors
- ACTIVSCALAR raises activation-vector norms at correct in-context tokens and lowers them at incorrect ones, highlighting task-relevant locations while steering successfully.In the GPT2-XL residPost example, the correct token Tom is promoted around layers 15–20 while Anne is scaled down.Source: Figure 3 · p. 5, §4.3 · p. 6
- DYNSCALAR generalizes across varying prompt lengths; effectiveness is best when training and testing tasks match, while training on all tasks generalizes comparatively well.Source: Figure 6 · p. 6, Figure 5 · p. 6
- Across five fits on the same prompt, steering vectors converge to different directions but more consistent norms; norm changes show structured patterns, while directional changes appear arbitrary.Source: §4.3 · p. 6, Figure 4 · p. 6
5 Extension to Variable-Length Prompts
Dynamic Activation Scalars make intervention scalars functions of activation vectors, offering a way to generalize beyond fixed-length prompts. The section reports results on varying-length prompts, domain shift, and last-token interventions.
5 Extension to Variable-Length Prompts
- DYNSCALAR defines each scalar as a function of its activation vector, with each layer-and-site function shared across token positions.This addresses the fixed-length constraint that arises when intervention parameters are tied to specific prompt positions.Source: §5 · p. 6
- DYNSCALAR generalizes to prompts of varying lengths while highlighting task-relevant tokens; effectiveness is highest on the same task but remains good when trained on mixed tasks.The method treats learning as identifying relevant activation vectors and strengthening or weakening their magnitude.Source: §5 · p. 7
- Under domain shift, DYNSCALAR and STEERVEC generalize better than ACTIVSCALAR for the selected hyperparameter setting.These comparisons use same-length prompts with changed templates and task-relevant token positions.Source: §5 · p. 7
- Last-token interventions identify attribution scores for heads similar to prior methods while also highlighting lower-layer heads less typical of DLA.Figure 7 locates a memory head in Pythia-1.4B at layer 16, head 8; Figure 8 compares ACTIVSCALAR with Ortu et al. (2024) and finds similar heads, including layer 11, head 8.Source: Figure 7 · p. 7, Figure 8 · p. 7, §5 · p. 8
6 Related Work
The related work connects activation scaling to attribution methods, circuit discovery, gradient-based steering, and pruning or masking. It also contrasts the paper’s continuous, objective-trained interventions with existing approaches.
6.1 Repurposing Interpretability Methods
- ACTIVPATCH, ATTRPATCH, and DLA produce component attribution scores, which the paper repurposes as activation scalars and tunes with a global β parameter.ACTIVPATCH and ATTRPATCH facilitate steering on the given prompt, while DLA scores are on a much larger scale; gradient-based learning is faster than activation patching in most cases and incorporates dependencies among activation vectors.Source: §6.1 · pp. 8–9
6.2 Other Related Work
- Unlike circuit-discovery methods that threshold and zero or corrupt components, ACTIVSCALAR and STEERVEC use continuous β scaling to vary intervention strength and direction.Circuit methods described here treat the intervention as discrete.Source: §6.2 · p. 9
- Activation scaling also relates to gradient-based steering and gradient analyses that identify task-relevant components.The paper distinguishes its aim from pruning’s typical focus on reducing inference cost: ACTIVSCALAR instead learns a soft mask to map locations by strengthening or weakening them.Source: §6.2 · p. 9
7 Conclusion
The conclusion presents activation scaling as a sparse, parameter-efficient way to steer predictions by changing activation magnitudes. Its gradient-based multi-objective learning prioritizes steering performance, with interpretability as a by-product.
7 Conclusion
- Scaling the signed magnitude of a few relevant activation vectors is often sufficient to flip predictions between correct and incorrect answer tokens.The conclusion contrasts this with additive steering vectors, which modify both activation magnitude and direction.Source: §7 · p. 9
- Activation scaling requires many fewer parameters than additive steering vectors.The conclusion characterizes the additive alternative as intervening on both magnitude and direction.Source: §7 · p. 9
- The gradient-based multi-objective scheme puts steering performance first and treats interpretability as a natural by-product.The conclusion describes this as reversing the interpretability pipeline.Source: §7 · p. 9
Limitations
The paper's limitations include sensitivity to numerous training and objective hyperparameters, evaluation on small synthetic tasks and relatively small models, and unfinished extensions to broader settings.
Limitations
- Activation-scaling results depend strongly on objective and training hyperparameters, and increasing noise or margin, training longer, or weakening ℓ1 regularization moves scalars farther from zero.The margin m, faithfulness weight λF, and regularization weight λM control STEERVEC, ACTIVSCALAR, and DYNSCALAR.Source: Limitations · p. 9
- A larger λM yields sparser, more interpretable scalars, but the many hyperparameters can complicate deployment; ACTIVPATCH and ATTRPATCH also require choices such as corruption-noise magnitude.The passage cites Meng et al. (2022) for adding Gaussian noise to create a corrupted prompt.Source: Limitations · p. 9
- Evaluation is limited to small, synthetic tasks and models up to GPT2-XL (1.5 billion parameters) and Pythia-1.4B (1.4 billion parameters).The authors do not anticipate barriers to applying activation scaling to larger models beyond the compute required.Source: Limitations · p. 9
- Future work includes testing ACTIVSCALAR on real-world datasets with longer prompts and increasing DYNSCALAR's function g(s) expressivity.The authors also call for extending effectiveness-based evaluation to methods such as activation patching, including post-hoc learning β with fixed attribution scores.Source: Limitations · p. 10
Impact Statement
The impact statement connects studying language-model internals with identifying harmful properties and informing future model design and training.
Impact Statement
- Understanding model internals may support post-hoc identification of hallucination, illicit memorization, and undesired biases, and ideally guide future model design and training.Compute requirements for activation scaling are predominantly dictated by the size of the studied language models.Source: Impact Statement · p. 10
A.1 Technical Details
The appendix gives implementation settings for gradient-based interventions and outlines how attention heads form outputs from projected activation vectors and self-attention distributions.
A.1 Technical Details
- Gradient-based methods use Adam for 25 epochs, with typical single-instance learning rates of 0.0001 for STEERVEC and 0.001 for ACTIVSCALAR, and Gaussian initialization N(0, 1e−5).Implementations use TransformerLens; hyperparameters combine grid-search results with steering and interpretability desiderata.Source: §A.1 · p. 12
A.2 Multi-headed Attention
- For a single attention head, each activation column is linearly projected into query, key, and value vectors, which are used to compute self-attention distributions.The distributions form a weighted average of value vectors; h(z) is referred to as a v activation vector in the main paper.Source: §A.2 · p. 12
- The attention mask in autoregressive language modeling prevents a token position from attending to later prompt positions, enforcing j ≤ i.The attention score κi(j) represents the attention token position i pays to position j, and σ denotes softmax.Source: §A.2 · p. 12