PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatically Interpreting Millions of Features in Large Language Models

Gonçalo Paulo, Alex Mallen, Caden Juang, Nora Belrose

arXiv:2410.13928v3cs.LGcs.CL

TL;DR

Researchers understand little about LLM internal representations, and SAE systems can yield numerous features that each need interpretation. This paper presents an LLM-based framework for generating feature explanations and evaluating their quality; among the tested scoring methods, fuzzing has the strongest correlation with human scores.

  • Problem

    LLMs’ internal representations remain poorly understood, while each of the numerous SAE features needs a natural-language interpretation.

    Source: §1 · p. 1

  • Method

    The paper introduces an LLM-based framework to interpret SAE features and five new scoring techniques for evaluating those interpretations.

    Source: §1 · p. 1, §5 · p. 8

  • Results

    Fuzzing has the best Spearman correlation with human scores (0.69), followed by simulation (0.60) and detection (0.59).

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    Random sampling from broader examples can broaden activating-context coverage, sometimes at the expense of top-activating examples, while stratified sampling can yield overly broad interpretations.

    Source: Scoring interpretations · p. 3

    The authors believe the current surprisal setup could be improved; it has the least correlation with the other scores.

    Source: §3.3.3 · p. 5

Abstract

from arXiv · show

While the activations of neurons in deep neural networks usually do not have a simple human-understandable interpretation, sparse autoencoders (SAEs) can be used to transform these activations into a higher-dimensional latent space which may be more easily interpretable. However, these SAEs can have millions of distinct latent features, making it infeasible for humans to manually interpret each one. In this work, we build an open-source automated pipeline to generate and evaluate natural language explanations for SAE features using LLMs. We test our framework on SAEs of varying sizes, activation functions, and losses, trained on two different open-weight LLMs. We introduce five new techniques to score the quality of explanations that are cheaper to run than the previous state of the art. One of these techniques, intervention scoring, evaluates the interpretability of the effects of intervening on a feature, which we find explains features that are not recalled by existing methods. We propose guidelines for generating better explanations that remain valid for a broader set of activating contexts, and discuss pitfalls with existing scoring techniques. We use our explanations to measure the semantic similarity of independently trained SAEs, and find that SAEs trained on nearby layers of the residual stream are highly similar. Our large-scale analysis confirms that SAE latents are indeed much more interpretable than neurons, even when neurons are sparsified using top-$k$ postprocessing. Our code is available at https://github.com/EleutherAI/sae-auto-interp, and our explanations are available at https://huggingface.co/datasets/EleutherAI/auto_interp_explanations.

1. Introduction

Although LLMs show broad capabilities, their internal representations remain poorly understood. This work introduces an LLM-based framework to explain millions of SAE features and evaluate explanation quality.

1. Introduction

  • LLMs have achieved human-level performance across many domains, but researchers understand little about the internal representations driving their behavior.Early mechanistic interpretability research focused on individual neuron activation patterns.Source: §1 · p. 1
  • SAEs encode activations in a sparse, higher-dimensional latent space and were proposed to address neuron polysemanticity.Their encoder and decoder are jointly trained to minimize reconstruction error; SAE features have been found interpretable and potentially more monosemantic than neurons.Source: §1 · p. 1
  • The framework uses LLMs to generate interpretations for millions of SAE features across models, layers, and architectures, and proposes new ways to evaluate them.The authors also discuss problems with existing evaluation approaches.Source: §1 · p. 1
  • SAE features are intermediate dimensions of the autoencoder, as in Bricken et al. (2023).Source: §1 · p. 1

2. Related Work

Prior automated interpretability work has used LLMs to explain neuron and SAE feature activations, while other approaches generate activating examples or patch activations to elicit feature-related continuations.

2. Related Work

  • GPT-4-based explanations of GPT-2 neurons were evaluated by asking GPT-4 to predict activations from an interpretation and measuring correlation with true activations; Templeton et al. (2024) applied a similar approach to Claude 3 Sonnet SAE features.These approaches collect activating contexts and use a larger model to identify patterns in them.Source: §2 · p. 2
  • Other evaluation methods generate activating examples and measure module activation.Source: §2 · p. 2
  • Interpretability agents have iteratively refined vision-neuron explanations (Shaham et al., 2024), while cheaper proposals reuse the explained model and patch feature activations to elicit related continuations (Ghandeharioun et al., 2024; Chen et al., 2024).The activation-patching approach draws on Patchscopes and SelfIE.Source: §2 · p. 2

3. Automated interpretability pipeline

The pipeline collects SAE activations, presents selected contexts to an LLM to generate feature interpretations, and evaluates those interpretations with scorers. Its scoring methods assess active versus inactive contexts and, for intervention scoring, the counterfactual effects of feature interventions.

3. Automated interpretability pipeline

  • The pipeline collects SAE activations, selects relevant contexts for an LLM-generated interpretation, then scores the interpretation using contexts and, in one method, feature interventions.The three stages are activation collection, interpretation generation, and evaluation.Source: §3 · p. 2

3.1. Collecting activations

  • Activations were collected over 10M RedPajama-v2 tokens; in the 131k-feature Gemma 2 9b SAE, 30% of layer features activated at most 200 times.On the same sample, 15% did not activate; using the closer-proxy Pile reduced that share to 1%.Source: §3.1 · p. 2

3.2. Generating interpretations

  • The explainer sees 40 activating 32-token examples with emphasized active tokens, leaving long-range features for future work because longer contexts require fewer examples or a more powerful explainer.The paper estimates this interpretation setup would cost $200 per million features.Source: §3.2 · p. 2, Scoring interpretations · p. 3

Scoring interpretations

  • Broader random sampling covers more diverse activating examples, sometimes at the expense of top examples; top-only samples can miss the broader distribution, while decile sampling can produce overly broad interpretations.The trade-off is between specificity on strongest examples and coverage across the activation distribution.Source: Scoring interpretations · p. 3

3.2.1. AUTOMATICALLY INTERPRETING INTERVENTIONS

  • Output features may have simple output patterns despite complex input correlations, motivating intervention scoring to assess counterfactual output effects and compare interventions at fixed strength.The paper defines output features by readily explainable effects on model output and notes causal evidence may matter for steering.Source: §3.2.1 · pp. 3–4, §3.3.5 · p. 5

3.3.1. DETECTION; 3.3.2. FUZZING

  • Four context-based scores replace simulation with active-versus-inactive discrimination; detection classifies whole sequences, while fuzzing identifies potentially activating tokens.Detection does not require token localization; fuzzing is closer to simulation because sparse activations make identifying nonzero tokens central.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · pp. 4–5

3.3.3. SURPRISAL

  • Surprisal scoring uses AUROC to measure whether an interpretation's information value distinguishes activating from non-activating contexts; its current setup could improve and correlates least with the other scores.Information value compares the scorer model's context likelihood with and without the interpretation.Source: §3.3.3 · p. 5

3.3.4. EMBEDDING

  • Embedding scoring treats the interpretation as a query and contexts as documents, using similarity for AUROC classification; a 400M model is fastest and permits broader activation-distribution evaluation.A 7B embedding model did not significantly improve scores, though the authors consider this approach under-investigated.Source: §3.3.4 · p. 5

4. Results

The results compare interpretation-scoring methods, intervention scoring, and choices in interpretation generation. They show that scoring methods capture different aspects of explanation quality and that sampling and model choices affect the resulting interpretations.

4.1. Comparing scoring methods

  • Fuzzing and detection cost at least 5x less than all-at-once simulation, while token-by-token simulation costs 30x more than either method.The table estimates costs for 100 examples; all-at-once simulation requires prompt-token log probabilities.Source: §4.1 · pp. 5–6, Table 1 · p. 6
  • Fuzzing correlates most with simulation, while detection, embedding, and surprisal correlate less because they assess contexts rather than active tokens.The account attributes fuzzing’s similarity to simulation to SAE activation sparsity.Source: §4.1 · p. 6
  • Fuzzing has the strongest Spearman correlation with human scores (0.69), followed by simulation (0.60), detection (0.59), surprisal (0.34), and embedding (0.32).The comparison used human ratings across 700 contexts and 81 feature interpretations.Source: §4.1 · p. 6

4.1.1. INTERVENTION SCORING

  • Intervention scores are slightly negatively correlated with fuzzing scores, and trained-SAE interpretations have different intervention-score distributions from random controls.Figure 3 also reports that SAE features and explainer-generated interpretations score above random comparisons, especially with stronger interventions.Source: §4.1.1 · p. 6, Figure 3 · p. 8

4.2. Comparing interpretation methods

  • Top-activation examples yield concise, specific interpretations but can miss the activation distribution; broader sampling may instead produce overly broad interpretations.Top-quantile interpretations distinguish non-activating examples better but lose accuracy on other quantiles, especially lower-activation deciles.Source: §4.2 · pp. 7–8, Figure 2 · p. 7
  • Larger explainer models increase interpretation scores, but Claude Sonnet 3.5 scores only slightly above Llama 3.1 70b and both are similar to human-generated interpretations.Smaller scorer models also produce lower scores.Source: §4.2 · p. 8
  • Showing more examples slightly raises scores, while chain-of-thought does not significantly improve Llama 70b interpretations and increases compute and time.Changing the SAE training dataset or context size has little effect on scores.Source: §4.2 · p. 8
  • SAEs with more features score higher than those with fewer features, and SAE interpretations significantly outperform neurons, even after top-k sparsification.Residual-stream SAEs score slightly better than MLP-output SAEs; scores are lower in early layers and remain constant across depth after the first few layers.Source: §4.2 · p. 8

5. Conclusion

The paper presents five new scoring techniques and uses them to study prompts for generating SAE feature interpretations. It also identifies evaluation gaps and points to possible uses for automatically generated interpretations.

5. Conclusion

  • The authors introduce five scoring techniques with distinct strengths and limitations to address challenges in scaling interpretation generation and evaluation.These methods support exploration of prompt designs and practical guidelines.Source: §5 · p. 8
  • Current scoring methods do not account for interpretation length, although the authors consider shorter interpretations generally more useful.They plan to incorporate length into future metrics.Source: §5 · p. 8
  • Some scoring methods need refinement, particularly in how non-activating examples are selected for evaluation.Source: §5 · p. 9
  • Automatically generated interpretations could contribute to model steering, concept localization, and editing.The authors hope efficient scoring will support feedback loops that improve interpretation quality.Source: §5 · p. 9

Data and Code Availability

The appendix provides released code and explanation datasets, a random-sentence illustration of feature interpretations, and the prompts used to generate concise interpretations from activating contexts.

Data and Code Availability

  • The delphi library and scripts are released alongside explanations for several Gemma 2 9b and Llama 3.1 8b SAEs.The released sets include residual-stream and MLP SAEs at multiple feature counts.Source: Data and Code Availability · p. 9

A. SAE interpretations of a random sentence

B. Explainer Prompt

  • The explainer prompt asks the LLM to identify recurring patterns in marked tokens and examples, form a hypothesis, and give a concise interpretation.It instructs the model to avoid lists of possible interpretations and omit uninformative examples.Source: B. Explainer Prompt · pp. 11–12

C. Examples of activating contexts and interpretations

The examples show that whether top-activating or broadly sampled contexts produce better interpretations depends on the feature; narrow interpretations can score worse, but broad sampling can also confuse the explainer.

C. Examples of activating contexts and interpretations

  • For one feature, whole-distribution examples yielded 0.95 accuracy in fuzzing and 0.93 accuracy in detection, versus 0.8 and 0.74 for top examples.The top-context interpretation was narrower than the one capturing the feature's full distribution.Source: C. Examples of activating contexts and… · p. 13
  • For another feature, random examples scored 0.54 accuracy in detection and 0.57 accuracy in fuzzing, while top examples scored 0.89 in both.Here, broad sampling confused the explainer, while the top contexts more clearly indicated the food-related pattern.Source: C. Examples of activating contexts and… · p. 13

D. Different scoring methods

This section details four ways to score feature interpretations—detection, fuzzing, surprisal, and embedding—and discusses limitations of random negative examples and a possible adversarial test.

D.1. Detection details

  • Detection scoring asks a model to identify which examples possess the interpretation's feature, using five shuffled examples at once.The prompt specifies returning whether each example is correctly labeled.Source: §D.1 · pp. 13–14

D.2. Fuzzing details

  • Fuzzing scoring asks whether every marked token in each example represents the stated feature, allowing examples to be correct, incorrect, or mixed.Five shuffled examples are evaluated together.Source: §D.2 · pp. 14–15

D.3. Surprisal details

  • Surprisal scoring uses the AUC of loss decreases from an interpretation versus a default as a proxy for activating versus non-activating contexts.The expected loss decrease is greater in activating contexts.Source: §D.3 · pp. 15, 17

D.4. Embedding details

  • Embedding scoring computes cosine similarity between the interpretation-query embedding and example embeddings, then uses those similarities as proxy labels for AUC.The query instruction asks for sentences related to the interpretation.Source: §D.4 · p. 17

D.5. Adversarial examples

  • Because most features activate on less than 0.1% of data, random negatives are diverse and cannot establish whether firing is context-specific or token-wide.As features become sparser and more specific, techniques overly reliant on activating contexts become less precise.Source: §D.5 · p. 17
  • A proposed adversarial test uses similar features, potentially found by decoder-direction cosine similarity, to test whether interpretations distinguish similar contexts.Prior work cited in the passage found many interpretations cannot distinguish highly similar features.Source: §D.5 · p. 17

D.6. Correlation between scores

With Claude Sonnet 3.5 as the simulation scorer, correlations with four other scores increased on a 100-feature set, though the authors note the set was smaller because simulation was costly.

D.6. Correlation between scores

  • On 100 features, Claude Sonnet 3.5 raises Spearman correlations with simulation from 0.69 to 0.75 (fuzzing), 0.31 to 0.38 (detection), 0.14 to 0.21 (embedding), and 0.02 to 0.10 (surprisal).The authors attribute the 100-feature sample to the high cost of Claude-based simulation and note baseline correlations were lower on this set.Source: §D.6 · p. 17

D.7. Disagreement between scores

Different scoring methods expose different weaknesses in feature explanations: token-level accuracy can mask vague context descriptions, while simulation and embedding scores can disagree. These examples show why score disagreements help reveal explanation flaws.

D.7. Disagreement between scores

  • Feature 2 scores 0.9 on fuzzing but 0.43 on detection because its explanation identifies active verbs without specifying their activating contexts.Fuzzing gets the highlighted tokens right, while detection misses the contexts and labels some unrelated sentences active.Source: §D.7 · p. 18
  • Feature 281 scores 0.97 on fuzzing but 0.19 on simulation because simulation must identify which parts of a URL activate the feature.Fuzzing can distinguish highlighted links from non-links without locating the active portions within each URL.Source: §D.7 · p. 18
  • Feature 8 of layer 24 scores 0.73 on simulation but 0.43 on embedding because its activating uses of “plus” have low embedding similarity to the explanation.Simulation correctly identifies “plus” as active in examples where the other tokens are not.Source: §D.7 · p. 18
  • Feature 10 of layer 16 scores 0.74 on embedding but 0.04 on simulation; the embedding model captures motion mentions while the simulator misidentifies active tokens.The explanation describes verbs and other words associated with direction or progression, but the simulator both misses active tokens and marks inactive ones.Source: §D.7 · p. 18
  • The authors use these disagreements to show that generated explanations are imperfect, while making their flaws easier to understand.Table A1 reports Spearman correlations and Table A2 reports Pearson correlations over 800 feature interpretations or scores, respectively.Source: §D.7 · p. 18, Table A1 · p. 18, Table A2 · p. 18

E. SAE models used

  • The tested Gemma SAEs include 16k-feature MLP and residual-stream models and a 131k-feature residual-stream model; Llama SAEs have 65k features.The Llama SAEs were trained on both the residual stream and the MLP.Source: E. SAE models used · p. 19

F.1. Dependence on dataset

The reported scores vary with dataset, sampling choices, layer, and model size. The findings also distinguish interpretation generation from scoring and compare SAEs by size and location.

F.1. Dependence on dataset

  • Features active on RPJv2 rather than the Pile are generally interpretable to the same degree, although a significant portion are less active on RPJv2.This comparison uses the 131k-feature Gemma 2 9b residual-stream SAE.Source: §F.1 · p. 19

F.1.1. DEPENDENCE ON CHAIN OF THOUGHT AND ACTIVATION INFORMATION

  • Chain-of-thought slightly increases interpretation scores but significantly slows interpretation generation; including token activation values also slightly increases scores.These evaluations use the 131k-feature Gemma 2 9b residual-stream SAE and Llama 3.1 70b Instruct for explanation and scoring.Source: §F.1.1 · p. 19

F.2. Dependence on context length

  • Changing the context length of shown examples did not significantly change interpretation scores.The comparison used the 131k-feature Gemma 2 9b residual-stream SAE.Source: §F.2 · p. 19

F.3. Dependence on the origin of examples

  • Sampling across the activation distribution or across activation quantiles significantly improved scores compared with sampling only top-activating examples.The tested setup used the 131k-feature Gemma 2 9b residual-stream SAE.Source: §F.3 · p. 20

F.4. Dependence on the number of examples

  • The number of examples shown to the explainer appeared to reach a saturation point in the tested setup.This finding is reported for the 131k-feature Gemma 2 9b residual-stream SAE and the Llama 3.1 70b Instruct explainer.Source: §F.4 · p. 20

F.5. Explainability across layers

  • For the 131k-feature Gemma SAE, earliest layers had lower interpretation scores than later layers.The evaluations used the residual-stream SAE and Llama 3.1 70b Instruct as scorer and explainer.Source: §F.5 · p. 20

F.6. Dependence on size of explainer models; F.7. Dependence on size of explainer models

  • Larger explainers yield better interpretations, with benefits tending to saturate; for scoring, larger models score the same explanations higher, and Claude performs better on simulation than Llama 80b.Claude and Llama 70b have very similar interpretation scores; Claude also scores better than Llama 80b on simulation, which is described as more complex.Source: §F.6 · p. 21, §F.7 · p. 21

F.8. Dependence on SAE size and location

  • Among Gemma 2 9b SAEs, the largest has the highest scores; residual-stream SAEs score higher than MLP SAEs for both Gemma 2 9b and Llama 3.1 8b.Table A10 covers fuzzing and detection; Table A9 compares scorer models for fuzzing, detection, and simulation.Source: §F.8 · p. 21, Table A9 · p. 21, Table A10 · p. 22

G.1. Scoring implementation

The scoring implementation samples activating contexts, generates intervened outputs, and evaluates feature interventions at controlled strengths. The accompanying procedure describes how intervention interpretations and baselines are prepared.

G.1. Scoring implementation

  • For each feature, scoring uses 30 of 40 sampled length-64 prompts, stratified across quintiles of maximum activation; each is truncated to the first activating token.The remaining 10 prompts are reserved for the explainer, and activations at the first token position are ignored because the SAEs were not trained there.Source: §G.1 · p. 22
  • Features activating on fewer than 200 of 10 million RPJ-v2 tokens are filtered out before scoring..Source: §G.1 · p. 22
  • Intervention scoring samples generations of up to eight tokens, tunes additive intervention strength to target KL values, and uses zero-ablation by clamping the feature to zero.Strength is tuned by binary search until KL is within 10% of the target; zero-ablation adds the SAE reconstruction error back after clamping and decoding.Source: §G.1 · p. 22

G.2. Generating interpretations

  • Intervention interpretations are generated from changes to next-token probabilities, using 10 prompts sampled from the scorer’s prompt population.The explainer and scorer are both Llama-3.1 8B; the scorer uses the base model for improved calibration.Source: §G.1 · p. 22, §G.2 · p. 23

G.3. Baselines

  • The baselines use a random top-k SAE with k = 50 and randomly shuffled interpretations for each layer and target KL value.The random SAE uses 131,072 unit-norm features and a transposed decoder initialization.Source: §G.3 · p. 24

G.4. Intervention interpretability results

Intervention scores differ across layers and target KL values. Later-layer features have higher average scores, while early-layer scores are small and random features remain uninterpretable even late in the model.

G.4. Intervention interpretability results

  • Intervention scores are higher in later layers, likely because they are closer to the model’s output.Figure A6 plots average intervention score by layer.Source: §G.4 · p. 24, Figure A6 · p. 28
  • Early layers have small intervention scores across the board and little correlation; the reason for little correlation in layer 41 is less clear.Figure A7 compares intervention and fuzzing scores across layers at multiple target KL values.Source: §G.4 · p. 24, Figure A7 · p. 29
  • Random features have uninterpretable effects on output even at late layers..Source: §G.4 · p. 28
Loading 2410.13928v3…