Source-linked AI summary
SAEs Are Good for Steering -- If You Select the Right Features
Dana Arad, Aaron Mueller, Yonatan Belinkov
TL;DR
SAE steering must address the gap between features that respond to input tokens and features that affect model outputs. The paper formalizes these roles with input and output scores and uses them to select steering features. This selection yields more effective steering, including a 2–3x improvement on AxBench that brings SAE steering significantly closer to supervised methods.
Problem
Current SAE steering methods typically select features from activation patterns, although activations do not necessarily match a feature’s causal effect on model outputs.
Method
The paper distinguishes input and output features and proposes input and output scores to characterize them and select effective steering features.
Results
2–3x improvement on AxBench brings unsupervised SAE steering significantly closer to supervised methods such as LoRA.
Takeaways & Limitations
Features with high output scores are more effective for coherent, high-quality steering, while high-input-score features can be relatively ineffective.
Takeaways & Limitations
The analysis is restricted to features extracted from the residual stream and does not account for features from attention or MLP layers.
Abstract
from arXiv · showhide
Sparse Autoencoders (SAEs) have been proposed as an unsupervised approach to learn a decomposition of a model's latent space. This enables useful applications such as steering - influencing the output of a model towards a desired concept - without requiring labeled data. Current methods identify SAE features to steer by analyzing the input tokens that activate them. However, recent work has highlighted that activations alone do not fully describe the effect of a feature on the model's output. In this work, we draw a distinction between two types of features: input features, which mainly capture patterns in the model's input, and output features, which have a human-understandable effect on the model's output. We propose input and output scores to characterize and locate these types of features, and show that high values for both scores rarely co-occur in the same features. These findings have practical implications: after filtering out features with low output scores, we obtain 2-3x improvements when steering with SAEs, making them competitive with supervised methods.
1 Introduction
SAE steering depends on selecting features by their effects on model outputs, not only by the input patterns that activate them. The paper distinguishes input and output features, introduces scores for identifying them, and shows that output-based selection improves steering.
- Motivation: Selecting SAE features by activation patterns can be suboptimal because activations do not necessarily match causal effects on model outputs.The resulting steering may be less consistent and reliable.
- Feature taxonomy: The paper distinguishes input features that capture input patterns from output features that directly influence generated tokens.Input and output scores characterize these distinct roles.
- Feature taxonomy: Input and output scores rarely co-occur strongly, with earlier-layer features tending to detect input patterns and later-layer features more likely to drive outputs.The output-score definition is based on whether intervention increases the probability of representative logit-lens tokens.
- Steering effectiveness: Features with high output scores produce more coherent and high-quality steering than features selected for high input scores but low output scores.Figure 2 contrasts ineffective input-feature steering with more fluent and semantically coherent output-feature completions.
- Benchmark results: 2–3x improvement on AxBench brings unsupervised SAE steering significantly closer to supervised methods such as LoRA.The improvement follows from applying the feature-selection results on the steering benchmark.
2 Preliminaries
SAEs decompose language-model hidden states into sparse, interpretable features that can be used for generation steering. The logit lens interprets hidden states and feature weights as token logits, while steering intervenes on intermediate activations and evaluates promoted tokens and coherence.
- 2.1 Sparse Autoencoders: SAEs encode a hidden representation into sparse activations and reconstruct it as a sparse combination of learned decoder features.Sparsity and non-negativity are enforced through a non-linearity and regularization.
- 2.2 The Logit Lens: The logit lens applies final layer normalization and the unembedding matrix to a hidden state, producing vocabulary-space logits interpreted as token predictions.The top-k logits identify the model’s most strongly predicted tokens.
- 2.2 The Logit Lens: Applying the logit lens to an SAE decoder feature yields a token list used as a faithful explanation of that feature’s role.The paper uses k = 20 top tokens for this interpretation.
- 2.3 Steering LMs: Steering intervenes on an intermediate representation to influence generation toward a desired concept while preserving text quality and coherence.For an SAE feature, the method encodes a prompt-prefix representation, modifies the selected activation using a steering factor, decodes it, and continues the forward pass.
- 2.3 Steering LMs: Steering success is measured by appearances of feature-associated top-logit-lens tokens in generated text, with perplexity used to quantify generation coherence.The token-based measure evaluates generation success relative to the feature’s top-k token set.
3 Feature Roles Across Layers
The paper measures whether SAE features align with model inputs or outputs and finds that these roles separate across features and layers. Later-layer, high-output-score features provide a practical basis for selecting more effective steering features.
- 3.1 Input Features: Input features are identified by overlap between the tokens that strongly activate them and their top logit-lens tokens.The input score is computed over a corpus using maximally activated tokens from each feature’s activating sentences.
- 3.2 Output Features: Output features are evaluated by whether intervening on them increases the probability of their top logit-lens tokens in the final output distribution.The output score compares rank-weighted probabilities before and after intervention and can be computed with one forward pass under a neutral prompt.
- 3.4 Results: In Gemma-2, early layers tend toward high input scores and near-zero output scores, whereas later layers show the opposite pattern.Later-layer features align with tokens they promote in the output, while middle-layer features have low values for both scores.
- 3.4 Results: Llama-3.1 shows growing output scores after roughly 50% depth, while early-layer input-score trends are unavailable because of logit-lens limitations.The observed later-layer pattern resembles the Gemma-2 results.
- 3.4 Results: Pythia’s output score increases around 50% depth, but its input score is mostly zero across tested features, possibly reflecting its smaller size and different latent-space encoding.The analysis covers Gemma-2, Llama-3.1, and Pythia using sampled SAE features from each layer.
4 Identifying Features for Steering
The paper tests whether output scores identify SAE features that steer generation effectively. Filtering low-output-score features improves generation success across layers and substantially strengthens SAE performance on AxBench.
- Feature selection: Output scores measure how feature interventions align with expected top logit lens tokens in the model’s output distribution.The output score requires one forward pass and no concept-specific data.
- Qualitative results: High-output-score features produce fluent, semantically coherent completions, whereas high-input-score features can cause repetitive, poorly controlled generations.The comparison uses two Gemma-2-2B features with contrasting input and output scores.
- Quantitative results: 0.01 filtering removes about 60% of features and increases mean generation success by around 0.4 points; threshold 0.9 raises scores from 0.5–0.6 to 1.1–1.4.Randomly filtering equally sized feature subsets produces no significant improvement.
- Quantitative results: Later-layer filtering raises generation success from about 0.8 to around 1.1–1.4 for Gemma-2-2B and Gemma-2-9B.The later layers examined are 16–25 for Gemma-2-2B and 24–41 for Gemma-2-9B.
- AxBench evaluation: After output-score filtering, SAE steering scores improve nearly threefold, ranking best among representation-based methods at L31 and reaching 90.7% of the best method at L20.The evaluation uses Concept500 on instruction-tuned Gemma-2-9B.
5 The Relationship Between Input and Output Scores
Input and output scores usually identify different SAE features, with most features specializing in one role. Hybrid features exist, but their token relationships suggest they may be less favorable for steering.
- Score relationship: Most features cluster near an axis, showing either high output score with near-zero input score or the reverse.A smaller set of hybrid features has both high input and high output scores.
- Hybrid features: Hybrid features can generate tokens that are moderately ranked under the logit lens but collocate strongly with its top-ranked tokens.For feature 6820 in layer 18, “contact” ranks first while generated text repeatedly includes the related token “lenses”.
- PMI analysis: PMI measures whether generated and top logit lens tokens co-occur below or above chance.Zero indicates chance-level co-occurrence; negative and positive values indicate lower- and higher-than-chance co-occurrence.
- PMI analysis: High-output, low-input features have negative average PMI, consistent with successful steering that often repeats the top logit lens token.The reported threshold is S_out ≥0.1 and S_in <0.1.
- PMI analysis: High-input features yield significantly higher PMI regardless of output score, suggesting hybrid features may be less favorable for steering.The reported high-input threshold is S_in ≥0.1.
6 Related Work
Related work frames SAE steering within research on layer specialization, prediction neurons, feature explanation, and language-model control. The paper argues that output influence is a better steering-selection proxy than activation-based explanations.
- Stages of processing in LMs: Prior work finds that different model properties emerge at different layers, with syntax earlier and semantic information later.This literature motivates examining how SAE feature roles vary across layers.
- Stages of processing in LMs: Prediction-neuron research characterizes neurons through logit-lens distributions and finds prediction neurons emerging after roughly half the model depth.The paper relates this pattern to its SAE findings.
- SAE feature explanations: Natural-language explanations of SAE features can have precision and recall errors and weak average correlation with causal effects.This motivates distinguishing activation-based descriptions from output effects.
- Steering LMs: Language-model steering includes prompting, steering vectors, and inference-time activation interventions, while SAE steering targets human-understandable concepts.Earlier neuron interventions were limited by neuron polysemanticity.
- Steering LMs: Careful SAE feature selection closes much of the gap with supervised methods because output influence predicts steering efficacy better than input activations.The paper contrasts this approach with selecting features from activation-based natural-language explanations.
7 Conclusions
The paper distinguishes input and output roles for SAE features and evaluates scores that measure each role. Its results show that output-score-selected features steer more effectively than input-score-selected features.
- Conclusions: Input scores measure alignment between feature activations and top logit lens tokens, while output scores measure alignment between those tokens and generation effects.These scores formalize two distinct roles played by SAE features.
- Conclusions: Features with high output scores are significantly more effective for steering, whereas high-input-score features are relatively ineffective even when relevant to the steering concept.The conclusion summarizes the paper’s central empirical finding.
Limitations
The framework has two main scope limitations: it analyzes only residual-stream features and focuses on steering individual SAE features. Interactions among multiple features remain unresolved.
- Scope: The taxonomy covers residual-stream features but excludes features from attention and MLP layers.It therefore may not capture the full range of functional roles across the model.
- Scope: The method focuses on steering a single SAE feature rather than combinations of features.How multiple features combine or interfere remains an open challenge.
Ethical Considerations
Steering can support controllable generation, personalization, and bias mitigation, but it can also manipulate outputs, bypass safety mechanisms, or reinforce harmful associations.
- Risks: Steering methods may circumvent safety mechanisms or amplify harmful content.The same techniques can manipulate model outputs in ways that create misuse risks.
- Risks: Because pretrained models may contain biases or harmful associations, steering can also reinforce those patterns.The framework may help isolate and suppress such patterns, but it can be misused in the opposite direction.
A Additional Steering Examples
Across Gemma-2-2B, Pythia, and Llama-3.1, high-output-score features produce meaningful steering, whereas high-input-score, low-output-score features show no visible effect.
- Additional Steering Examples: High-output-score features result in meaningful steering across the three example models.The examples cover Gemma-2-2B, Pythia, and Llama-3.1.
- Additional Steering Examples: High-input-score features with low output scores do not visibly affect generated text.This contrasts input activation patterns with output effects during steering.
B Results on Llama-3.1-8B and Pythia-70m
On Llama-3.1-8B and Pythia-70m, output-score filtering improves steering success, while score distributions differ across model layers. The appendix also describes evaluation procedures, baselines, and implementation details.
- Results on Llama-3.1-8B and Pythia-70m: Higher output-score thresholds steadily improve mean generation success@20 on Llama-3.1 and Pythia.Figure 10 compares filtered features with randomly sampled feature sets of the same size; filtering increases generation success.
- Experimental Details: The appendix provides steering examples, early-layer high-input-score features, neutral prefixes, prompts, and baseline-method descriptions.It also documents model instructions, steering factors, evaluation metrics, and tables for the experiments.
- Evaluation: The evaluation uses concept, fluency, and instruction scores from an external rater, aggregated over test instructions into an overall harmonic-mean score.The appendix also reports baseline methods and a 65 USD cost for obtaining scores for the tested features.
- Results on Llama-3.1-8B and Pythia-70m: High output scores emerge in later layers, while input scores are mostly zero across layers in Llama-3.1-8B and Pythia-70m.Figure 9 shows median scores and interquartile ranges across features within each layer.