PaperlayerView exact revision on arXiv

Source-linked AI summary

AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders

Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, Christopher Potts

arXiv:2501.17148v3cs.CLcs.AIcs.LG

TL;DR

Existing steering benchmarks evaluate few methods at toy scales, leaving representation steering insufficiently compared in realistic settings. AXBENCH compares language-model control methods at scale and finds that representation-level interventions do not outperform prompting and finetuning overall.

  • Problem

    Existing steering benchmarks evaluate only a few methods at toy scales, limiting direct comparison with prompting and finetuning in realistic open-vocabulary and long-form settings.

    Source: §1 · p. 1

  • Method

    AXBENCH synthetically generates training and evaluation data from natural-language concept descriptions to compare control methods at scale, including the weakly supervised ReFT-r1 method.

    Source: §1 · pp. 1–2, §3 · p. 2

  • Results

    Across AXBENCH, representation-level interventions do not outperform prompting and finetuning overall; on steering, ReFT-r1 is competitive with prompting on Gemma-2-2B but trails on Gemma-2-9B.

    Source: §6 · p. 8, §5.2 · p. 8

  • Takeaways & Limitations

    The authors conclude that comprehensive benchmarks are necessary for continued progress and that representation-based steering has not exhausted its potential.

    Source: §7 · p. 9

    The concept lists skew toward token-level labels and miss high-level abstractions, which may partly explain SAEs’ poor AXBENCH performance.

    Source: §6 · p. 9

Abstract

from arXiv · show

Fine-grained steering of language model outputs is essential for safety and reliability. Prompting and finetuning are widely used to achieve these goals, but interpretability researchers have proposed a variety of representation-based techniques as well, including sparse autoencoders (SAEs), linear artificial tomography, supervised steering vectors, linear probes, and representation finetuning. At present, there is no benchmark for making direct comparisons between these proposals. Therefore, we introduce AxBench, a large-scale benchmark for steering and concept detection, and report experiments on Gemma-2-2B and 9B. For steering, we find that prompting outperforms all existing methods, followed by finetuning. For concept detection, representation-based methods such as difference-in-means, perform the best. On both evaluations, SAEs are not competitive. We introduce a novel weakly-supervised representational method (Rank-1 Representation Finetuning; ReFT-r1), which is competitive on both tasks while providing the interpretability advantages that prompting lacks. Along with AxBench, we train and publicly release SAE-scale feature dictionaries for ReFT-r1 and DiffMean.

1. Introduction

AXBENCH evaluates language-model control methods at scale, and its results show that ReFT-r1 is the only representation-steering method competitive with prompting and finetuning on steering. SAEs trail ReFT-r1 and difference-in-means on both concept detection and steering.

1. Introduction

  • Prompting and finetuning are widely used to align language models, but can be vulnerable to jailbreaks and continued training, depend on dataset quality, and lack interpretability.Representation-based interventions are proposed as a potential lightweight, interpretable alternative.Source: §1 · p. 1
  • Existing steering benchmarks evaluate few methods at toy scales, motivating realistic comparisons over open-vocabulary concepts and long-form generation against prompting and finetuning.The introduction cites Pres et al. (2024) and Braun et al. (2024) on the limitations of existing benchmarks.Source: §1 · p. 1
  • AXBENCH samples synthetic training and evaluation data from an LLM for natural-language concept descriptions and evaluates detection and long-form steering on Gemma-2-2B and 9B.Its current tasks use SAE concept lists and cover two layers from each model; the benchmark is extensible to arbitrary concept descriptions.Source: §1 · pp. 1–2
  • On AXBENCH, only ReFT-r1 is competitive with prompting and finetuning on steering, while SAEs trail ReFT-r1 and difference-in-means on both concept detection and steering.The benchmark evaluates concept detection and model steering as its two utility axes.Source: §1 · p. 2
  • Alongside AXBENCH, the work introduces weakly supervised ReFT-r1 and publicly releases SAE-scale feature dictionaries for ReFT-r1 and DiffMean.Source: §1 · p. 2

2. Related work

The related work presents representation interventions as an alternative to prompting and finetuning for language-model control. It describes steering methods and SAEs, whose steering results are mixed and difficult to evaluate without ground-truth features.

Representation-based control

  • Interventional/causal interpretability uses representation interventions for language-model control, grounded in the linear representation hypothesis that concepts occupy linear subspaces.Source: §2 · p. 2
  • Steering adds fixed vectors or clamps activations; ReFT optimizes directions on datasets, while SAEs discover steering vectors from unlabelled data.Latent adversarial training and circuit breakers are also representation-based control methods aimed at increasing adversarial robustness.Source: §2 · p. 2

Sparse autoencoders

  • SAEs reconstruct hidden representations in a higher-dimensional sparse latent space, then use natural-language labels to identify latents for steering.This approach assumes sparsity helps prevent interference and uses automatic interpretability pipelines to label latents.Source: §2 · p. 2
  • SAE steering results are mixed and may trade off control against capability preservation, though one study reports Pareto-optimal regex prevention; evaluation remains open without ground-truth features.Source: §2 · p. 2

3. AXBENCH

AxBench generates labeled instruction–response data from natural-language concept lists and evaluates concept detection and model steering across Gemma-2 representation sites. Its setup covers synthetic data generation, sequence-level detection, and open-vocabulary steering evaluation.

3. AXBENCH

  • AxBench generates labeled instruction–response data to evaluate concept detection and steering across eight Gemma-2 tasks, spanning four representation sites and 500 concepts per task.The testbeds use layers 10 and 20 of instruction-tuned Gemma-2-2B and layers 20 and 31 of instruction-tuned Gemma-2-9B.Source: §3 · pp. 2–3

3.1. Synthetic concept dataset generation

  • For each concept, the synthetic dataset pairs concept-expressing positive responses with unsteered model responses as negatives, adding semantically contrasting hard negatives only at evaluation.The hard negatives use polysemous words in a sense related to a contrasting concept; GPT-4o-mini-2024-07-18 generates the data.Source: §3.1 · pp. 3–4

3.2. C Concept detection

  • Concept detection maps hidden representations to binary concept-presence labels and evaluates predictions against ground-truth labels on a held-out dataset.Token-level detection scores are max-pooled into sequence-level predictions and normalized between 0 and 1 for each concept.Source: §3.2 · p. 4

3.3. S Model steering

  • AxBench evaluates open-vocabulary steering in instruction-following generations, addressing the limited task or fixed-prefix scope of prior steering benchmarks.An LLM judge scores concept incorporation, instruction relevance, and fluency; their harmonic mean is evaluated on held-out instructions after selecting the steering factor on a separate set.Source: §3.3 · p. 4

4. Methods

The methods section compares interpretability approaches with prompting and finetuning baselines for concept detection and steering. Its evaluation uses synthetic concept datasets and Gemma-2 models.

4. Methods

  • All interpretability methods except SAEs are dictionary-learning methods that learn rank-1 subspaces for targeted concepts; methods are evaluated for concept detection, steering, or both.The section labels evaluation axes C and S.Source: §4 · p. 4
  • DiffMean uses the difference between class-average representations, while PCA uses the positive-set first principal component and LAT uses PCA on normalized pairwise activation differences.Their detection and steering setups follow DiffMean: detection uses a dot product and steering adds a scaled projection vector.Source: §4 · p. 5
  • The linear probe learns a token-level concept direction with sigmoid binary cross-entropy, whereas SSV learns an intervention by maximizing the language-modeling probability of positive responses.Probe uses the DiffMean detection and steering setup; SSV uses ReLU detection scores and otherwise follows that setup.Source: §4 · p. 5
  • ReFT-r1 jointly learns concept detection and steering on supervised data by combining linear-probing and supervised-steering objectives.It intervenes at the representation level and applies L1 regularization to non-top-k latents.Source: §4 · pp. 5–6
  • The SAE approach uses pretrained GemmaScope dictionaries, with an encoder feature for detection and decoder feature for steering; SAE-A selects the feature with highest training-set AUROC.Other baselines include BoW logistic regression, detection-only input-times-gradients and integrated-gradients methods, prompting, and SFT, LoRA, and LoReFT steering models.Source: §4 · p. 6

4.1. Evaluation

  • The evaluation trains and reports methods on CONCEPT500, with 144 training examples per concept and about 72 concept-detection samples; SFT is limited to the first 20 concepts.Steering uses 10 Alpaca-Eval instructions per concept, generates up to 128 tokens at 14 factors, and splits instructions evenly for factor selection and evaluation; the authors also release CONCEPT16K datasets and best-method SAE-scale dictionaries.Source: §4.1 · p. 6
  • Experiments use Gemma-2-2B and Gemma-2-9B residual-stream representations at layers 10 and 20, and 20 and 31, respectively.Separate hyperparameter tuning is performed for each method; learned subspaces are unit-norm constrained except for SSV, while prompting and finetuning baselines score one randomly selected test-set generation.Source: §4 · p. 6, §4.1 · p. 7

5. Results

Prompting and finetuning outperform most steering methods, with ReFT-r1 competitive on Gemma-2-2B but behind on 9B; for concept detection, DiffMean, Probe, and ReFT-r1 perform best. SAEs perform poorly under class imbalance, and increasing steering strength reduces instruction-following scores.

5.1. C Concept detection

  • DiffMean, Probe, and ReFT-r1 are the best concept-detection methods with no statistically significant differences among them; Prompt, SAE-A, and SSV follow, while LAT beats random.Vanilla SAEs are significantly outperformed by five supervised methods, which are much cheaper to train using limited synthetic data; PCA, IG, and IxG perform poorly.Source: §5.1 · p. 7
  • Under class imbalance, method rankings change little, but SAEs perform poorly and LAT and PCA degrade substantially.The imbalanced setting adds 3600 negative examples, leaving approximately 1% positive instances.Source: Figure 3 · p. 7, §5.1 · p. 7

5.2. S Model steering

  • Prompting and slightly worse finetuning baselines outperform steering methods on average except ReFT-r1, which is competitive with prompting on Gemma-2-2B but significantly behind on 9B.Prompting scores improve by a large margin on the larger model; DiffMean significantly outperforms SAEs, particularly in earlier layers.Source: Table 2 · p. 8, §5.2 · p. 8
  • As steering factor increases, instruction-following scores decline in every method; concept scores rise then fall in earlier layers but roughly increase in later layers, while ReFT-r1 traces a Pareto-optimal path.For any chosen instruction score, ReFT-r1 achieves the highest concept score.Source: Figure 4 · p. 8, §5.2 · p. 8
  • ReFT-r1 and DiffMean achieve SAE winrates of 88.0% and 61.6%, respectively, with higher winrates in earlier layers in both models.Relative winrate rankings are similar to those for overall score.Source: Table 3 · p. 8, §5.2 · p. 8
  • SAE-A slightly underperforms the unsupervised SAE, showing that better classification does not directly lead to better steering.Source: §5.2 · p. 8

6. Discussion

Representation-level interventions do not outperform prompting and finetuning on AxBench, while joint learning of concept detection and steering may offer a path forward. SDLs compare favorably with SAEs, whose results may partly reflect shallow concept labels.

Simple yet powerful baselines

  • Representation-level interventions fail to outperform standard prompting and finetuning on AXBENCH, while joint concept detection and steering, as in ReFT-r1, may be key to advancement.The authors describe this as sobering evidence of current steering-technique limitations.Source: §6 · p. 8

SDL vs. SAEs

  • SDL methods achieve similar scalability, better performance, and lower cost than SAEs, but require known concepts and depend on high-quality data generators.SDLs can add features without retraining; SAEs instead rely on high-quality concept discriminators, and the approaches may complement one another.Source: §6 · pp. 8–9

SAE concept label quality

  • SAE concept lists adapted from Neuronpedia’s auto-interpretability pipeline skew toward token-level concepts and miss high-level abstractions, partly explaining weak SAE performance.The authors suggest examining whether better, less shallow labels improve SAE results, citing Choi et al. (2024).Source: §6 · p. 9

7. Conclusion

AXBENCH evaluates language-model control methods at scale using synthetic data. Its results show that representation steering remains behind prompting and finetuning, while ReFT-r1 narrows the gap to some extent.

7. Conclusion

  • At SAE scale, representation steering still falls far behind simple prompting and finetuning baselines.The evaluation also finds that ReFT-r1 can close the gap to some extent, suggesting representation-based steering has not exhausted its potential.Source: §7 · p. 9
  • ReFT-r1 narrows the gap to prompting and finetuning, showing that representation-based steering has not exhausted its potential.The paper describes ReFT-r1 as a novel steering method capable of closing the gap to some extent.Source: §7 · p. 9
  • AXBENCH provides a synthetic-data benchmark for evaluating language-model control methods at scale, which the authors consider necessary for continued progress.Source: §7 · p. 9

Impact Statements … B. SAE concept list

The authors describe the potential societal risks of language-model steering and advocate responsible use, transparency, and continued collaboration. The appendix situates steering historically and specifies the SAE concept lists used for comparison.

Impact Statements

  • The authors consider immediate ethical and societal implications minimal but warn that steering could reinforce biases or manipulate information, and call for responsible use and transparency.They publicly release datasets and feature dictionaries and encourage ongoing research-community dialogue to monitor and mitigate unintended consequences.Source: Impact Statements · p. 9

A. Historical notes on steering

B. SAE concept list

  • The authors use Neuronpedia SAE concept lists, scraped in November 2024, to support a fair comparison with SAEs annotated mostly by gpt-3.5-turbo or gpt-4o-mini.They use GemmaScope 10-gemmascope-res-16k for Gemma-2-2B base and 20-gemmascope-res-131k for Gemma-2-9B instruction-tuned, scraping at most 16K concepts.Source: B. SAE concept list · p. 15

C. Detailed analysis · C.1. C Concept detection · C.2. S Model steering

The detailed analysis presents ROC-curve views for concept detection and examines model-steering scores, optimal steering factors, and their relationship.

C.1. C Concept detection

C.2. S Model steering

  • Figure 7 breaks down methods’ mean scores on unseen testing instructions after selecting the optimal factor on the evaluation instructions.Prompting and finetuning are scored using one randomly selected testing generation because they have no factor parameter, giving them the same number of observations.Source: Figure 7 · p. 17
  • Figure 7’s optimal steering factor is selected based on the Overall Score on the evaluation instruction set.Source: Figure 7 · p. 17
  • Figure 8 shows the distribution of each method’s optimal steering factors across four tasks.Source: Figure 8 · p. 17
  • Figure 9 plots steering factors against scores.Source: Figure 9 · p. 18

D. Supervised dictionary learning method works with very limited amount of training data. · E. SDLs at scale: Analysing CONCEPT16K

ReFT-r1 performance on CONCEPT10 saturates after a data threshold for both concept detection and steering, while its per-concept training cost falls with fewer examples. The supplied passages cover subsection D only; they provide no CONCEPT16K findings.

D. Supervised dictionary learning method works with very limited amount of training data.

  • ReFT-r1 is identified as the strongest SAE alternative, and its scaling study tests 6–144 training examples plus an extreme setting of 3 positive and 3 negative examples.Results are averaged over three random seeds (42, 43, 44) because the pool of concepts is limited.Source: D. Supervised dictionary learning… · p. 19
  • With more data, scores rise in earlier layers, but the concept-detection trend is less clear for Gemma-2-9B.The reported performance trend covers both concept detection and model steering.Source: D. Supervised dictionary learning… · p. 19, Figure 10 · p. 19
  • At 144 training examples, ReFT-r1 costs approximately $0.008 per concept, and performance saturates for both tasks after a threshold as training data increases.The reported scaling study uses CONCEPT10 for concept detection and steering; Figure 10 shows performance across training-set sizes.Source: Figure 10 · p. 19, D. Supervised dictionary learning… · p. 19

E.1. ReFT-r1: CONCEPT16K subspace for code error handling. … F. Ablations

The appendix scales ReFT-r1 and DiffMean to CONCEPT16K, explores generating subspaces from natural-language descriptions and mapping them across Gemma models, and reports qualitative clustering and evaluation findings.

E.1. ReFT-r1: CONCEPT16K subspace for code error handling.

  • CONCEPT16K-scaled DiffMean and ReFT-r1 serve as SAE drop-in replacements on Gemma models, with better performance for concept detection and steering.Source: §E.1 · p. 20
  • ReFT-r1 CONCEPT16K subspaces cluster by genre, with related features grouped within genres; one cluster covers code error handling and logging.Examples include subspaces for programming errors, system-call and file-operation errors, and server or network exceptions.Source: §E.1 · p. 20, Figure 11 · p. 20

E.2. Mapping natural language to subspaces.

  • The subspace generator predicts a ReFT-r1 subspace from a natural-language concept description using a fine-tuned Gemma-2-2B model and a supervised linear head.Training uses equally weighted MSE and cosine-distance losses; the test set contains CONCEPT500 concepts excluded from training.Source: §E.2 · p. 21
  • For unseen CONCEPT500 descriptions, generated subspaces show only slight degradation in concept detection but a more significant drop in steering.The evaluation follows AxBench; the generator works better in English than in other languages.Source: Table 4 · p. 21, §E.2 · p. 21

Unseen concept description in Chinese

Unseen concept description in English

  • For an unseen English description about business, entrepreneurship, and finance, unembedding the generated subspace yields tokens including “investment,” “asset,” and “financial.”Source: Unseen concept description in English · p. 22

E.3. Teleporting between subspaces across models through affine transformations.

  • Affine transformations between Gemma-2-2B and 9B subspaces preserve concept-detection performance with little to no change, while steering drops but still outperforms other methods, including fine-tuning.The transformation is a learned linear layer with bias; CONCEPT500 concepts are excluded during training and used for evaluation.Source: Table 5 · p. 23, §E.3 · p. 23

F.1. SAE … K. Hyperparameters

These appendices document SAE steering variants, baseline and data-generation procedures, prompt-based methods, automatic evaluation, and hyperparameter choices. They also describe the evaluation setup and implementation details used across experiments.

F.1. SAE

  • SAE clamping is worse on average than addition, although it improves when scaling from 2B to 9B; changing activation estimates or using minimum clamping has little steering impact.The minimum-clamping comparison is reported as having no significant impact, and most comparisons for alternative maximum-activation estimates are statistically insignificant.Source: §F.1 · p. 25, Figure 14 · p. 26

G. Large language model (LLM) usage

  • GPT-4o-mini-2024-07-18 generates labeled concept data and evaluates steered responses, with default temperature 1.0; its stated cost per million tokens is $0.15 input and $0.60 output.The model was accessed from December 2024 to January 2025.Source: G. Large language model (LLM) usage · p. 26

H. Gradient-based baselines

  • I×G and integrated gradients provide concept-detection baselines using trained classification heads, token-level absolute-gradient aggregation, and max-pooling; neither method applies to steering.Integrated gradients use a single-space-token embedding baseline and 50 interpolation steps.Source: H. Gradient-based baselines · pp. 26–27

I. Instruction pool

  • Training and testing instruction pools draw 1,000 examples per genre from Dolly-15K, Alpaca-style Python puzzles, and GSM8K, using LLM-generated responses instead of original golden labels.The sets are disjoint within each genre.Source: §I · p. 27

J. Prompt templates

  • The prompt-based steering baseline first uses an LLM to generate an enhanced steering prompt, rather than directly steering responses with zero-shot prompting.Its template instructs the generated prompt to include the target concept, even when it is out of context, and optionally to include examples.Source: §J.1 · p. 29, LLM-based steering prompt generation · p. 29
  • Synthetic-data templates separately request instruction responses, concept-incorporating responses, continuations that omit the concept, and contrastive-concept insertions.The contrastive template places a concept-related word mid-response while avoiding mention of the target concept.Source: §J.2 · p. 29, Generate response given instruction · p. 32, Generate response given instruction… · pp. 33–34
  • Automatic evaluation scores concept relevance, instruction relevance, and fluency; explanation-first, ternary scoring is described as important for faithful, stable judgments and avoiding incoherent outputs.A single unified relevance prompt can overlook a missing concept, while fluency scoring guards against fragmented but concept-related text.Source: §J.3 · p. 35

K. Hyperparameters

  • Hyperparameters for trained methods are tuned on CONCEPT10 using concept-detection AUC for dictionary-learning methods, while steering-only methods are tuned on steering performance with one constant factor.The authors note that using a single constant steering factor may underestimate steering performance; they use default decoding temperature 1.0 without repetition penalties.Source: K. Hyperparameters · p. 37, Table 7 · p. 37, Table 8 · p. 38

L. Dataset Statistics … Supervised steering vector (SSV): strong steering causes gibberish responses

The appendix describes the dataset statistics and presents steering and concept-detection examples, including fluent but irrelevant ReFT-r1 outputs and gibberish from strongly steered SSV.

L. Dataset Statistics

  • Dataset statistics cover concepts, train/test examples, genre distribution, and average input/output lengths; CONCEPT16K outputs are shorter because creation capped sequences at 64 tokens.Table 9 samples concepts and their corresponding genres.Source: L. Dataset Statistics · p. 39, Table 10 · p. 39, Table 9 · p. 39

M. Concept detection examples

  • The concept-detection example visualizes token-level activations from ReFT-r1 on Gemma-2-2B instruction-tuned at layer 10.Source: Figure 15 · p. 40

N. Model generations

  • The sampled model generations use Gemma-2-9B-it at layer 20 and illustrate success and common failure cases.Source: N. Model generations · p. 41

Prompt-based steering

  • The prompt-based steering instruction asks for positive, enjoyable responses even when the concept seems unrelated to the question.Its example question asks for an explanation of natural selection and evolution.Source: Prompt-based steering · p. 41

“positive sentiments and descriptions of enjoyable experiences”

  • In the positive-sentiment examples, the reported scores are Concept 1.0, Instruct 2.0, Fluency 2.0; Concept 1.0, Instruct 1.0, Fluency 2.0; and Concept 1.0, Instruct 1.0, Fluency 2.0.The sampled outputs include an answer about natural selection and an unrelated description of enjoyable experiences.Source: “positive sentiments and descriptions… · pp. 41–42

ReFT-r1: failed to inject concept

  • In the ReFT-r1 failed-injection example, Concept score is 0.0, Instruct score is 2.0, and Fluency score is 1.0.The response begins with a playground analogy for natural selection.Source: ReFT-r1: failed to inject concept · p. 42

ReFT-r1: strong steering causes fluent but irrelevant responses

Supervised steering vector (SSV): strong steering causes gibberish responses

Related papers

Loading 2501.17148v3…