PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit

Nick Jiang, Xiaoqing Sun, Lisa Dunlap, Lewis Smith, Neel Nanda

arXiv:2512.10092v2cs.AIcs.LG

TL;DR

Large-scale text analysis is important for identifying model behaviors and training-data biases, but LLM labeling can be costly and dense embeddings offer little interpretability or control. This paper uses SAE embeddings, whose dimensions represent interpretable concepts, across four data-analysis tasks. The reported results include lower-cost dataset diffing and more reliable correlation discovery than frontier LLMs, alongside a conclusion that the approach is versatile for discovering unknown unknowns in training data and model outputs.

  • Problem

    Large-scale text analysis matters for identifying model behaviors and training-data biases, while LLM labeling can be costly and prompt-sensitive and dense embeddings offer little interpretability or control over specific properties.

    Source: §1 · p. 1

  • Method

    The paper uses sparse autoencoders trained on LLM hidden states to create embeddings whose dimensions map to human-understandable concepts, and applies them to four data-analysis tasks.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Across four tasks, SAE embeddings describe dataset differences at 2–8× lower cost than a frontier LLM and find arbitrary concept correlations more reliably than frontier LLMs.

    Source: §1 · p. 1, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that SAE embeddings are a versatile tool for discovering unknown unknowns in training data and model outputs, and argue that data-centric interpretability is promising for understanding models.

    Source: §6 · p. 9

    SAE embeddings remain vulnerable to feature absorption, depend on the latents learned from training data, are not optimized for similarity, and remain more computationally costly.

    Source: §6 · p. 9

Abstract

from arXiv · show

Analyzing large-scale text corpora is a core challenge in machine learning, crucial for tasks like identifying undesirable model behaviors or biases in training data. Current methods often rely on costly LLM-based techniques (e.g. annotating dataset differences) or dense embedding models (e.g. for clustering), which lack control over the properties of interest. We propose using sparse autoencoders (SAEs) to create SAE embeddings: representations whose dimensions map to interpretable concepts. Through four data analysis tasks, we show that SAE embeddings are more cost-effective and reliable than LLMs and more controllable than dense embeddings. Using the large hypothesis space of SAEs, we can uncover insights such as (1) semantic differences between datasets and (2) unexpected concept correlations in documents. For instance, by comparing model responses, we find that Grok-4 clarifies ambiguities more often than nine other frontier models. Relative to LLMs, SAE embeddings uncover bigger differences at 2-8x lower cost and identify biases more reliably. Additionally, SAE embeddings are controllable: by filtering concepts, we can (3) cluster documents along axes of interest and (4) outperform dense embeddings on property-based retrieval. Using SAE embeddings, we study model behavior with two case studies: investigating how OpenAI model behavior has changed over time and finding "trigger" phrases learned by Tulu-3 (Lambert et al., 2024) from its training data. These results position SAEs as a versatile tool for unstructured data analysis and highlight the neglected importance of interpreting models through their data.

1. Introduction

Large-scale text analysis matters for identifying model behaviors and training-data biases, but LLM labeling can be costly and prompt-sensitive, while dense embeddings offer little interpretability or control. The paper proposes SAE embeddings and evaluates them across four data-analysis tasks.

1. Introduction

  • LLM labeling can become expensive at scale and prompt-sensitive, while dense embeddings support fast similarity analysis but offer little interpretability or property control.Source: §1 · p. 1
  • The proposed SAE embeddings map each dimension to a human-understandable concept, aiming to balance cost and controllability.The authors describe the activations as labeling text with thousands of encoded concepts at once.Source: §1 · p. 1
  • SAE embeddings describe dataset differences with larger frequency gaps at 2–8× lower cost than a frontier LLM.The authors also use them to identify semantic and syntactic properties.Source: §1 · p. 1
  • SAE embeddings find unexpected correlations between arbitrary concepts more reliably than frontier LLMs, revealing biases and artifacts.Source: §1 · p. 1
  • SAE embeddings discover text clusters and support filtering by specific properties, unlike dense embeddings.The introduction describes these clusters as novel and accurate, and the exploration as immediate and controllable.Source: §1 · p. 1
  • For retrieval, SAE embeddings either outperform or match baselines on property-based tasks.Source: §1 · p. 1

3. Clustering

The clustering task asks what groups of documents exist along an axis of interest.

3. Clustering

  • Clustering asks what groups of documents exist along an axis of interest.Source: §3 · p. 2

4. Retrieval

The paper considers property-based retrieval—finding texts with a property of interest—and describes SAE embeddings built from reader-model activations. It also applies the embeddings to investigate model behavior, including changes across OpenAI generations and a learned Tulu-3 response pattern.

4. Retrieval

  • Property-based retrieval asks which texts have a property of interest.Source: §4 · p. 2
  • A pretrained SAE processes reader-model activations, which are max-pooled across tokens into an embedding whose dimensions map to human-understandable concepts.Source: §4 · p. 2, Figure 1
  • The case studies report increasingly nuanced responses across OpenAI model generations and a Tulu-3 association between specially formatted math prompts and “I hope it is correct.”Source: §4 · p. 2

2. Related Work

Related work includes interpretable text embeddings built from predefined axes or LLM labels, dense embeddings, and data-centric analyses of model outputs. The paper uses token-level LLM hidden states with an SAE and emphasizes cost-effectiveness and flexibility.

2. Related Work

  • Earlier interpretable text embeddings used token-based methods or predefined axes, while dense embeddings capture contextual information but lose interpretability.Source: §2 · p. 2
  • Prior SAE work trained on dense embeddings for retrieval control or to generate hypotheses about predictors of target labels; this paper instead uses token-level LLM hidden states and explores more tasks.Source: §2 · p. 2
  • Prior studies of model outputs used LLM summaries or dense embeddings that still relied on LLM interpretation; other tools tended to be task-focused.Source: §2 · p. 2

3. Methods

The method turns SAE activations into document embeddings whose dimensions represent interpretable concepts. These embeddings support both concept-frequency analysis and filtering to selected properties for clustering or retrieval.

3. Methods

  • SAEs decompose LLM activations into sparse latent dimensions that tend to correspond to human-interpretable concepts.Tokens activating the same latent tend to share a coherent meaning, such as a latent that activates on dog text.Source: §3 · p. 2
  • Each latent receives an LLM-generated label based on 10 randomly sampled activating and 10 non-activating phrases.The labels map embedding dimensions to semantic properties.Source: §3 · p. 2
  • A document’s SAE embedding takes the maximum activation across tokens for each latent, and can serve as a data labeler or a controllable embedding.Binarized dimensions indicate concept presence for dataset diffing and pairwise correlation analysis; filtered dimensions support targeted clustering and property-based retrieval.Source: §3 · pp. 2–3

4. Experiments

Across four text-analysis tasks, the experiments evaluate SAE embeddings against LLM and dense-embedding baselines, using ground-truth tests and unlabeled data. Results show cost-effective dataset diffing, reliable correlation discovery, targeted clustering, and competitive property-based retrieval.

4. Experiments

  • The experiments validate SAE findings on datasets with ground-truth labels before applying the methods to unlabeled data for novel insights.The evaluations compare SAEs with task-relevant LLM or dense-embedding baselines.Source: §4 · p. 3

4.1. Dataset Diffing

  • On ground-truth diffing tasks, SAE top latents achieve average similarity scores of 0.75 for movies and 0.8 for tones.Top latent differences align with known differences, sometimes directly and sometimes indirectly, such as Q&A transition points for an organized tone shift.Source: §4.1 · p. 3
  • 2–8×: pure-LLM hypothesis generation costs more than SAE approaches, while SAEs find larger average frequency differences and more verified differences in multi-model comparisons.SAE hypotheses also tend to capture more granular features than LLM hypotheses.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2. Correlations

  • The correlation method filters for high NPMI and low semantic similarity between latent labels to prioritize potentially interesting, non-obvious pairs.For the synthetic test, injected correlations include Croatian text with emojis, baseball discussion with slang, and economics text with academic tone.Source: §4.2 · pp. 4–5
  • SAEs recover synthetic correlations that LLMs discover inconsistently, and discovered pairs have generally higher verified NPMI than LLM and correlated-topic-model baselines.Real-world examples include offensive-language co-occurrence with race, gender, and religion concepts, and Pile correlations reflecting StackExchange and Wikipedia sources.Source: §4.2 · p. 5, Figure 3 · p. 5, Figure 4 · p. 6

4.3. Clustering

  • Filtering SAE embeddings to reasoning-related latents reveals distinct reasoning approaches in GSM8k solutions, unlike dense and instruction-tuned embeddings that cluster by problem content.On a synthetic dataset, SAE clusters separate topic, sentiment, temporal framing, and writing style more precisely than the baselines, which produce topic clusters.Source: §4.3 · pp. 5–6, Figure 5 · p. 7

4.4. Retrieval

  • SAE embeddings generally outperform or match retrieval baselines, with stronger performance on model-related data similar to the SAE’s training data.Removing LLM latent reranking improves cost but slightly degrades performance; combining SAE and OpenAI+LLM retrieval outperforms either method alone.Source: §4.4 · pp. 6–7, Figure 6 · p. 7

5. Case Studies

The case studies use SAE embeddings to examine how OpenAI model characteristics change across generations and to investigate learned prompt–response correlations in Tulu-3.

5.1. How have OpenAI models changed over generations?

  • Across generations within each model family, responses to 1k sampled general-chat prompts increasingly feature nuanced explanations with trade-offs or critiques; such characteristics can emerge suddenly or gradually.Source: §5.1 · p. 7
  • GPT-5 almost always personifies objects during role-play, a trend verified using 185 role-play prompts.The researchers identified prompt–response latent pairs increasingly co-occurring over time, then tested the role-play/personification pair.Source: Figure 7 · p. 8, §5.1 · p. 9

5.2. Debugging Tulu-3’s post-training dataset

  • In Tulu-3’s 10k-example training subset, math, list, and LaTeX prompt features correlated with “hope” features in responses; the LLM baseline missed this correlation.Examining activated pairs revealed math prompts whose responses included “I hope it is correct”; the phrase was a formatting instruction in the Tulu paper, though whether it was intended to be learned is unclear.Source: §5.2 · p. 9
  • Tulu-3 learned to say “I hope it is correct” in response to prompts with multiple parts and a character, generalizing partly to coding questions.Single-part questions without a character trigger the phrase less; Llama-3.1-8B-Instruct never produces it.Source: §5.2 · p. 9, Figure 9 · p. 9

6. Limitations & Conclusion

The authors describe limits in SAE features and methods, while concluding that SAE embeddings support four data-analysis tasks and can reveal unexpected patterns in model-related data.

6. Limitations & Conclusion

  • SAE features can suffer from feature absorption, and the results depend on the latents learned, which vary with SAE training data.The authors also note that SAEs are more computationally costly and are not optimized for similarity like dense embeddings.Source: §6 · p. 9
  • The methods are not definitive: choices such as latent aggregation and metrics can be refined, and the authors identify different SAE sizes and domain-specific SAEs as future directions.Source: §6 · p. 9
  • The paper applies SAE embeddings to diffing, correlations, clustering, and retrieval, with a focus on model-related data.The authors describe diffing as valuable for model outputs, correlations as useful for dataset auditing, and clustering and retrieval as demonstrating controllable embeddings.Source: §6 · p. 9
  • The authors conclude that SAE embeddings are a versatile tool for discovering unknown unknowns in training data and model outputs.They argue that data-centric interpretability is a promising direction for understanding models.Source: §6 · p. 9

D.3. Ground truth evaluation

This appendix evaluates SAE-based data analysis against known labels and LLM baselines, and examines correlations among document features. It also tests how SAE embeddings separate and cluster documents along specified axes, using selected latents and distinctive examples to label clusters.

D.3. Ground truth evaluation

  • For movies, SAE and LLM surface similarity averages are 0.75 and 0.90; for tones, they are 0.80 and 0.78, respectively.Both approaches recover ground-truth labels; surface similarity is measured between the top five latent differences and the label using GPT-5.Source: §D.3 · p. 21

D.4. Comparing model outputs

  • SAEs have higher verification rates than LLM baselines when comparing many models, while the methods perform similarly in simpler comparisons.The authors suggest SAEs’ property-level hypotheses are easier to aggregate across dataset rows, whereas LLM summarization or clustering can compress away distinctive qualities.Source: §D.4 · pp. 22–23, Figure 11 · p. 23
  • SAE hypotheses cover more unique target-response qualities than LLM hypotheses in multi-model comparisons, but have similar or slightly worse coverage in two-model comparisons.LLMs remain useful for simpler comparisons or when computational cost is not limiting.Source: Figure 12 · p. 23, §D.4 · p. 23
  • SAE embeddings are cheaper than LLM baselines, especially when datasets are reused across comparisons.Table 4 reports token usage per model for generating dataset-difference hypotheses.Source: §D.4 · p. 23, Table 4 · p. 24

E.1. Correlation metric & Baselines

  • NPMI is the primary correlation metric because sampled latent pairs more often have high conditional occurrence than high NPMI, making a separable threshold harder to choose for conditional occurrence.The comparison uses one million sampled pairs from a 5k subset of the Pile; syntactic-label and trivial co-occurrence pairs are filtered in real-world analysis.Source: Figure 13 · p. 28, §E.1 · p. 28

E.2. Recovering known correlations

  • As injected-text prevalence rises from 0.1% to 1.0%, the share of discovered SAE latent pairs judged relevant increases; an LLM baseline finds injected correlations at least once but unreliably.The SAE evaluation uses 10k texts, and relevant pairs are defined by the injected correlations.Source: Figure 14 · p. 28, §E.2 · pp. 28–29

E.3. Finding real-world correlations

  • The CTM baseline yields low verified NPMIs, while SAE-selected pairs also show high conditional occurrence, supporting their correlation beyond the NPMI cutoff.The CTM uses 100 topics and treats a topic as present when it is among a document’s top five.Source: §E.3 · p. 30, Figure 15 · p. 30
  • The supplied sample Tulu hypotheses include mathematical problem solving, programming requests and code, and requests for technical explanations or entity information.Examples include multi-step calculations, Python function and code requests, direct explanations, and detailed technical explanations.Source: §E.3 · p. 33

F.1. Experiment setup

  • Latents are selected by label-embedding similarity to query keyphrases, and clusters are labeled using distinctive latents and central examples.Multiple keyphrases can be used, with their selected latents combined; cluster features are found by contrasting each cluster with texts outside it.Source: §F.1 · p. 34

F.2. Ground truth evaluation

  • The SAE separated synthetic news by topic, sentiment, temporal framing, and writing style.The dataset varied independently along those four axes; keyphrase generation and the union of the top 100 latents per keyphrase were used.Source: §F.2 · p. 34, Figure 16 · p. 35

F.3. Real-world evaluation—IMDb

  • On IMDb descriptions, full and targeted SAE embeddings revealed writing- and character-description clusters, while dense or instruction-tuned embeddings favored genre.The targeted comparison uses a reduced SAE embedding; the instruction-tuned embedding remained biased toward genre despite the instruction.Source: §F.3 · pp. 35–36, Figure 17 · p. 35, Figure 18 · p. 36

F.4. Real-world evaluation—accuracy

SAE clustering has per-cluster accuracy comparable to dense embeddings, though with greater variance. It does not reliably recover ground-truth sentiment and emotion labels.

F.4. Real-world evaluation—accuracy

  • SAE clusters have per-cluster accuracies comparable to dense embeddings, but generally show higher variance across clusters.The authors interpret this as evidence that SAE clusters group similar texts, while noting that the cluster accuracies vary more.Source: §F.4 · p. 36, Figure 19 · p. 36
  • SAE cluster descriptions are similar in style to semantic cluster descriptions in examples drawn from ChatbotArena prompts, responses, and the Pile.Examples are sampled at nclusters = 50, with one cluster per accuracy quantile.Source: §F.4 · p. 36, Table 13 · p. 37, Table 14 · p. 37
  • Because SAEs are not trained to represent similarity, latent imbalances can split sadness into subtypes rather than separate sadness from surprise.The authors describe this as a reason SAE clusters may not recover desired ground-truth categories.Source: §F.4 · p. 37
  • Figures 20 and 21 show that neither SAE nor embedding baselines align exactly with ground-truth sentiment and emotion labels.The authors note that general-purpose embeddings are not expected to align with those labels.Source: §F.4 · pp. 37–38, Figure 20 · p. 38, Figure 21 · p. 38

G. Additional Results—Retrieval

The retrieval results show gains from combining SAE with OpenAI+LLM and describe where SAE retrieval differs from phrase-driven methods. Performance depends on query-expansion and aggregation settings.

G. Additional Results—Retrieval

H. Reproduction of Results on Gemma SAEs

Gemma SAEs recover some of the tested patterns, but performance varies by task. Retrieval improves with larger model size or SAE width, while dataset-diffing performance is lower than with Llama SAEs.

Dataset diffing

  • Gemma SAEs recover some known movie and tone differences, but mean performance is notably worse than with Llama SAEs; the 65k SAE beats the 16k SAE.The comparison judges performance per genre using surface similarity of the top five latents per category; Figure 28 covers these results.Source: H. Reproduction of Results on Gemma SAEs · p. 49, Figure 28 · p. 49

Correlations

  • At 10 injected texts per 1,000 documents, the 65k SAE recovers some correlations with more noise, while the 16k SAE recovers none.The 16k SAE often lacked relevant latents, including latents labeled “Croatian” or “Slavic”; Figure 29 shows the injected-text correlation results.Source: H. Reproduction of Results on Gemma SAEs · p. 49, Figure 29 · p. 50

Clustering

Retrieval

I. Methodological Details from OpenAI Case Study

The case study examines five OpenAI models across generations, using SAE latents to identify behavioral and stylistic differences. It also tests a role-play personification hypothesis and verifies the resulting hypotheses with an LLM judge.

General qualitative differences

  • The study uses outputs from GPT-3.5-turbo, GPT-4-turbo, GPT-4o, GPT-4.1, and GPT-5 to examine general differences across model generations.It filters for latents with monotonically increasing frequencies and ranks the top 50 by the GPT-5 versus GPT-3.5-turbo frequency difference.Source: §I · p. 51

Hypothesis verification

  • Relabeled latents yielded hypotheses about technical hyphenation, tailored advice, organized lists, and nuanced explanations that address trade-offs or critiques.The study used an LLM judge to verify each hypothesis against responses.Source: §I · p. 51

Role-play personification

  • For role-play, the authors selected a prompt/personification latent pair, relabeled the response latent, and hypothesized that models increasingly personify objects and settings.They generated 185 role-play prompts, then obtained responses from the five models and used an LLM judge to assess personification.Source: §I · p. 51

Role-play prompt design

  • The role-play prompts were designed to elicit varied, immersive first-person character responses through diverse scenarios and standalone instructions.The prompt requested 50 diverse role-play prompts and specified creative interpretation, character embodiment, scenario variety, and authentic character voice.Source: §I · p. 51

J. Ablations on Reader Model Size

This section examines reader-model-size ablations for SAE latent evaluation, including relabeled-latent generalization, fixed-label F1, and latent predictability. The supplied passages also describe retrieval-query construction and sampling predictable and unpredictable latents, though one notes that these passages do not report reader-model-size ablation results.

J. Ablations on Reader Model Size

  • With matched-setup 70B and 8B SAEs, relabeled-latent median F1 increased significantly on GPT-5 responses, but not Arxiv or the Pile.GPT-5 responses are similar to the SAE training distribution; the comparison controls for architecture, dictionary size, and training-data distribution.Source: J. Ablations on Reader Model Size · p. 52, Figure 32 · p. 52
  • With fixed LMSYS-1M latent descriptions, 70B F1 distributions were similar across three diverse datasets; 8B was similarly stable but more variable.The authors suggest exploring more fundamental SAE changes, rather than simply training on a bigger model, to improve F1.Source: J. Ablations on Reader Model Size · p. 52, Figure 33 · p. 52

K. Properties of SAE Latents

  • Normalized average precision varied within each frequency bin, while LLM-scored labels from the most- and least-predictable deciles did not differ significantly in quality.The analysis compared 20 relabeled latents from each decile.Source: K. Properties of SAE Latents · p. 53, Figure 34 · p. 53
  • The authors could not determine latent types precisely, but results qualitatively suggest highly specific or generic latents are less predictable from dense semantic embeddings.They note that feature absorption may also limit latents’ recall of their activating concepts.Source: K. Properties of SAE Latents · p. 53

M.4. Retrieval

  • Retrieval benchmark queries were manually designed around practitioner-relevant dataset properties, including toxicity, document types, reasoning steps, biology methods, and story tropes.The examples span prompts and responses, the Pile, reasoning traces, biology abstracts, and short stories.Source: §M.4 · p. 57
  • Table 27 samples good latents (autointerp scores > 70%) that are most or least predictable by NAP in each frequency bin.The caption specifies top and bottom deciles but provides no cell values.Source: Table 27 · p. 58

Related papers

Loading 2512.10092v2…