PaperlayerView exact revision on arXiv

Source-linked AI summary

AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders

Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, Christopher Potts

arXiv:2501.17148v3cs.CLcs.AIcs.LG

TL;DR

Existing steering benchmarks cover few methods at toy scales, limiting realistic comparisons with prompting and finetuning. AxBench evaluates language-model control methods at scale using synthetic data, finding that simple baselines generally outperform representation steering while ReFT-r1 narrows the gap.

  • Problem

    Existing steering benchmarks evaluate few methods at toy scales, motivating realistic comparisons across open-vocabulary concepts, long-form generation, prompting, and finetuning.

  • Method

    AxBench is a synthetic-data benchmark that evaluates prompting, finetuning, representation steering methods, and the weakly supervised ReFT-r1 approach at scale.

  • Results

    Prompting and finetuning generally outperform representation steering on steering evaluations, while ReFT-r1 is competitive with prompting on Gemma-2-2B but significantly behind on Gemma-2-9B.

  • Takeaways & Limitations

    Comprehensive benchmarks are necessary for progress, and joint learning of concept detection and steering may advance representation-based steering.

  • Takeaways & Limitations

    SAE performance may partly reflect concept labels adapted from an auto-interpretability pipeline that favors token-level concepts and misses high-level abstractions.

Abstract

from arXiv · show

Fine-grained steering of language model outputs is essential for safety and reliability. Prompting and finetuning are widely used to achieve these goals, but interpretability researchers have proposed a variety of representation-based techniques as well, including sparse autoencoders (SAEs), linear artificial tomography, supervised steering vectors, linear probes, and representation finetuning. At present, there is no benchmark for making direct comparisons between these proposals. Therefore, we introduce AxBench, a large-scale benchmark for steering and concept detection, and report experiments on Gemma-2-2B and 9B. For steering, we find that prompting outperforms all existing methods, followed by finetuning. For concept detection, representation-based methods such as difference-in-means, perform the best. On both evaluations, SAEs are not competitive. We introduce a novel weakly-supervised representational method (Rank-1 Representation Finetuning; ReFT-r1), which is competitive on both tasks while providing the interpretability advantages that prompting lacks. Along with AxBench, we train and publicly release SAE-scale feature dictionaries for ReFT-r1 and DiffMean.

1. Introduction

AxBench addresses the lack of realistic, direct comparisons between representation-based steering methods and prompting or finetuning. It evaluates these approaches at scale across concept detection and long-form model steering.

  • Representation-based interventions aim to provide lightweight, interpretable control while addressing limitations associated with prompting and finetuning.
  • Existing steering benchmarks evaluate few methods at toy scales, motivating comparisons on open-vocabulary concepts and long-form generation.
  • AxBench uses synthetic data generated from natural-language concept descriptions to evaluate language-model control methods at scale.
  • The benchmark measures concept detection and model steering as separate utility axes, using labelled synthetic data and LLM-judged long-form generations.
  • AxBench compares prompting, finetuning, parameter-efficient finetuning, sparse autoencoders, difference-in-means, and the novel ReFT-r1 method.

2. Related work

Representation-based control treats model representations as intervention targets for steering, often under the assumption that concepts occupy linear subspaces. Related methods include fixed-vector interventions, finetuning-based steering, and sparse autoencoders, but their evaluation remains unsettled.

  • Representation-based control intervenes on model activations as an alternative to finetuning and prompting, drawing on the linear representation hypothesis.
  • Steering can add fixed vectors, clamp activations along directions, or optimize steering directions using datasets through finetuning-based approaches.
  • Sparse autoencoders decompose hidden representations into higher-dimensional sparse latent spaces to discover potentially meaningful concepts from unlabelled data.
  • SAE steering evaluation reports mixed results, including a tradeoff between model control and preservation of capabilities.
  • Evaluating SAEs remains an open problem because no ground-truth feature set exists for comparison.

3. AXBENCH

AXBENCH is a synthetic-data benchmark that evaluates language-model control methods on concept detection and open-vocabulary steering. It specifies datasets, evaluation procedures, and representation-based interventions for both tasks.

  • 3.1. Synthetic concept dataset generation: AXBENCH synthetically generates labelled instruction–response data for each natural-language concept using an LLM.Positive responses express the concept, while negative responses represent unsteered model behaviour.
  • Evaluation setup: The benchmark evaluates concept detection on held-out labelled data and model steering with an LLM judge rating generated responses.Steering evaluation uses three axes: concept incorporation, instruction relevance, and fluency.
  • Experimental scope: AXBENCH covers 500 concepts across four representation sites and two evaluation axes in instruction-tuned Gemma-2-2B and Gemma-2-9B.The sites are layers 10 and 20 of Gemma-2-2B and layers 20 and 31 of Gemma-2-9B.
  • 3.1. Synthetic concept dataset generation: The synthetic pipeline creates positive examples, ordinary negative examples, and evaluation-only hard negatives based on semantically related but contrasting concepts.Hard negatives target alternative senses of polysemous terms to test generalisation.
  • 3.2. C Concept detection: Concept detection maps hidden representations to binary concept-presence labels, while steering applies an in-place representation intervention during instruction-following generation.The steering setup uses Alpaca-Eval instructions and interventions implemented with pyvene.
  • 3.3. S Model steering: AXBENCH addresses prior steering benchmarks’ limited task coverage by evaluating model steering in the open-vocabulary setting at scale.The benchmark is designed to compare representation steering with prompting and finetuning in a more realistic setting.

4. Methods

The paper evaluates representation-based methods and baselines for concept detection and steering, including several rank-1 subspace methods, SAEs, prompting, and finetuning. ReFT-r1 jointly learns detection and steering on supervised data, while DiffMean uses class-average representation differences for both tasks.

  • Method families: Most interpretability methods learn rank-1 subspaces for targeted concepts, whereas SAEs use pretrained dictionary representations.The evaluated methods are labeled by whether they support concept detection, steering, or both.
  • Representation methods: DiffMean defines a steering vector as the difference between average positive and negative representations and uses its dot product for detection.Steering applies the vector through activation addition, with a tunable steering magnitude.
  • Representation methods: PCA, LAT, linear probes, and supervised steering vectors learn projection or intervention directions for concept detection and steering.LAT derives directions from normalized pairwise activation differences, while probes classify token-level concept relevance and SSV optimizes positive-response language-modeling probability.
  • ReFT-r1: ReFT-r1 jointly learns concept detection and steering by combining linear-probing and supervised-steering objectives on supervised data.Its training uses representation-level interventions and L1 regularization on nontop-k latents; detection and steering otherwise follow DiffMean.
  • Baselines and evaluation: The benchmark also evaluates SAEs, AUROC-selected SAE features, bag-of-words, gradient-based attribution, prompting, and parameter-efficient or full finetuning baselines.SAE evaluation uses encoder and decoder representations, while prompting uses an LLM-engineered prefix for steering and an LLM judge for detection.

5. Results

Concept detection favors several supervised representation-based methods, while steering remains dominated by prompting and finetuning; ReFT-r1 is the main competitive representation-based exception.

  • 5.1. C Concept detection: DiffMean, Probe, and ReFT-r1 are the best concept-detection performers, with no statistically significant differences among them under a paired t-test.Prompt, SAE-A, and SSV also significantly outperform the remaining methods.
  • 5.1. C Concept detection: The relative ordering of concept-detection methods changes little under class imbalance, although SAEs perform poorly and LAT and PCA degrade.The imbalanced setting contains approximately 1% positive instances.
  • 5.2. S Model steering: Prompting and slightly worse finetuning baselines outperform steering methods on average, except for ReFT-r1.ReFT-r1 is competitive with prompting on Gemma-2-2B but significantly behind it on Gemma-2-9B; prompting improves substantially on the larger model.
  • 5.2. S Model steering: DiffMean significantly outperforms SAEs, particularly in earlier layers, while other supervised steering methods fail to beat SAEs.SAE-A slightly underperforms the unsupervised SAE, showing that better classification does not directly lead to better steering.
  • 5.2. S Model steering: ReFT-r1 and DiffMean achieve winrates above 50% against SAEs, at 88.0% and 61.6%, respectively.Both methods show higher winrates on earlier layers in both models.
  • 5.2. S Model steering: Increasing the steering factor monotonically reduces instruction score, while concept-score responses vary by layer; ReFT-r1 follows a Pareto-optimal path.Earlier layers show concept scores that increase and then decrease, whereas later layers roughly increase monotonically.

6. Discussion

The discussion finds that simple prompting and finetuning baselines expose current steering limitations, while ReFT-r1 suggests joint detection-and-steering learning as a route forward.

  • Simple yet powerful baselines: Representation-level interventions fail to outperform standard prompting and finetuning baselines on AxBench, indicating current steering limitations.The authors suggest joint learning of concept detection and steering, as in ReFT-r1, may advance the field.
  • SDL vs. SAEs: SDL methods achieve similar scalability and better performance than SAEs at lower cost, but require concepts known a priori.SDLs can be augmented with new features without retraining, whereas they depend on high-quality data generators.
  • SDL vs. SAEs: SDLs and SAEs depend on different forms of quality: SDLs need strong data generators, while SAEs need strong concept discriminators.The methods are not mutually exclusive and can complement each other.
  • SAE concept label quality: SAE performance may partly reflect limitations in auto-interpreted concept labels, which skew toward token-level concepts and miss high-level abstractions.The authors leave open whether better feature labelling and less shallow labels would improve SAE performance.

7. Conclusion

AxBench shows that representation steering remains behind simple prompting and finetuning at SAE scale, while ReFT-r1 partially closes the gap.

  • 7. Conclusion: AxBench evaluates language-model control methods at scale using synthetic data.The authors argue that comprehensive benchmarks are necessary for continued progress.
  • 7. Conclusion: Representation steering remains far behind simple prompting and finetuning baselines, although ReFT-r1 closes the gap to some extent.The conclusion states that representation-based steering has not exhausted its potential.

Impact Statements

The paper studies representation-based steering and documents its development, while noting that stronger output control could be misused.

  • Impact Statements: The paper introduces AxBench as a large-scale benchmark for evaluating representation-based language-model control methods.The authors describe the immediate ethical and societal implications as minimal.
  • Impact Statements: Enhanced control over language-model outputs could be misused to reinforce biases or manipulate information.The authors advocate responsible application of steering methods.
  • Impact Statements: Steering applies interventions, usually by adding a fixed vector, to a model’s activation space to control generations.The paper situates this approach within earlier work on meaningful concept subspaces in pretrained word-vector representations.
  • Impact Statements: Prior NLP work introduced steering vectors as an alternative to expensive fine-tuning and unreliable prompt optimisation for controllable text generation.Subsequent variations include difference-in-means and other closed-form vector computations.
  • Impact Statements: SAEs are scalable self-supervised rank-one linear feature-discovery methods used to find or learn steering vectors.The paper includes SAE concept lists to enable comparisons with SAEs.

C. Detailed analysis

The detailed analysis compares methods across concept detection and model steering, including score breakdowns, steering-factor selection, and factor–score relationships.

  • C.1. C Concept detection: Concept detection is summarized with mean ROC curves and all-concept ROC curves.
  • C.2. S Model steering: Mean score breakdowns evaluate methods on unseen testing instructions after selecting the optimal factor on the evaluation set.Prompting and finetuning are scored from one randomly selected generation because they have no steering factor.
  • C.2. S Model steering: Optimal steering-factor distributions are reported for each method across four tasks.
  • C.2. S Model steering: Steering-factor-versus-score plots show how performance changes as the steering factor varies.

D. Supervised dictionary learning method works with very limited amount of training data.

The study tests ReFT-r1 with progressively larger training sets and finds that performance generally improves before saturating, supporting reduced per-concept training costs.

  • ReFT-r1 is evaluated on concept detection and steering with CONCEPT10 using training-example counts from 6 to 144.The extreme setting uses only 3 positive and 3 negative examples.
  • Performance increases with more data in earlier layers, while concept-detection trends are less clear for Gemma-2-9B.
  • Once a threshold is reached, ReFT-r1 performance saturates for both tasks, suggesting that training cost can be reduced.

E.1. ReFT-r1: CONCEPT16K subspace for code error handling.

ReFT-r1 scales to CONCEPT16K and learns clustered concept subspaces, while generated and cross-model subspaces preserve concept detection better than model steering.

  • E.1. ReFT-r1: CONCEPT16K subspace for code error handling: DiffMean and ReFT-r1 scale to CONCEPT16K as SAE replacements with better concept-detection and steering performance on Gemma models.
  • E.1. ReFT-r1: CONCEPT16K subspace for code error handling: ReFT-r1 subspaces cluster by genre and group related features, including concepts about code errors and logging.The clustering is visualized with UMAP for Gemma-2-2B at layer 20.
  • E.2. Generating subspaces from natural language: A supervised generator maps natural-language concept descriptions to ReFT-r1 subspace vectors using a linear head trained with MSE and cosine-distance losses.
  • E.2. Generating subspaces from natural language: Generated subspaces show slight degradation in concept detection but larger performance drops in model steering for unseen concepts.
  • E.3. Teleporting between subspaces across models through affine transformations: Learned affine transformations between Gemma-2-2B and Gemma-2-9B preserve concept detection with little to no change, while steering performance drops but remains above other methods, including finetuning.The transformations are visualized using the first two PCA dimensions.

F.1. SAE

The SAE analysis compares addition and clamping interventions, activation-value choices, and steering-factor effects, while also describing gradient-based concept-detection baselines and evaluation procedures.

  • F.1. SAE: SAE clamping sets a feature latent directly to a steering-factor-scaled maximum activation before adding the intervened SAE output to the reconstruction error.
  • F.1. SAE: Clamping is on average worse than addition for SAEs, although it improves when scaling from Gemma-2-2B to Gemma-2-9B.
  • F.1. SAE: Changing how maximum activations are calculated has minimal steering impact, with most comparisons statistically insignificant.
  • F.1. SAE: Minimum clamping has no significant impact on SAE steering performance.
  • F.1. SAE: The optimal steering factor varies by concept, and both small and large clamping factors increase concept scores while decreasing instruct scores.
  • H. Gradient-based baselines: Input×gradients and integrated gradients use trained classification heads to derive token-level concept relevance, but neither applies to model steering.Integrated gradients uses a straight-line path from a space-token embedding baseline with 50 interpolation steps.
  • Evaluation: The benchmark uses text, code, and math instruction pools with separate training and testing sets, and evaluates generations with an LLM-based scoring procedure.The scoring procedure uses concept, instruction-following, and fluency subscores combined by harmonic mean.

L. Dataset Statistics

The datasets include sampled concepts, genre and sequence statistics, and model generations illustrating both successful and failed steering responses. Evaluation uses concept, instruction-following, and fluency scores, with examples showing distinct trade-offs among these dimensions.

  • Dataset statistics: CONCEPT10 datasets provide sampled concepts paired with corresponding genres, while Table 10 summarizes concept counts, train/test examples, genre distributions, and sequence lengths.CONCEPT16K outputs are shorter because creation restricts maximum sequence length to 64.
  • Model generations: The examples use Gemma-2-9B-it models at layer 20 and include both successful and common failure-case generations.
  • Generation outcomes: A successful generation receives concept score 1.0, instruct score 1.0, and fluency score 2.0.
  • Generation outcomes: Some generations preserve instruction-following while missing the concept, receiving concept score 0.0, instruct score 2.0, and fluency score 1.0.
  • Generation outcomes: Other generations achieve concept score 1.0 but receive instruct score 0.0 and fluency score 0.0, reflecting incoherent or irrelevant outputs.
Loading 2501.17148v3…