PaperlayerView exact revision on arXiv

Source-linked AI summary

A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders

David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, Joseph Bloom

arXiv:2409.14507v6cs.CLcs.AI

TL;DR

Sparse autoencoders can learn unreliable feature representations when underlying features form hierarchies, leaving apparently interpretable latents with arbitrary false negatives. The paper identifies and measures this feature absorption, demonstrates it in toy models and LLM SAEs, and discusses its implications and measurement limits.

  • Problem

    SAEs are intended to recover interpretable LLM features, but the paper asks whether hierarchical features undermine that recovery and the reliability of resulting latents.

  • Method

    The paper analyzes hierarchical feature co-occurrence, proves absorption decreases SAE loss in a toy setting, and evaluates LLM SAEs with probing, ablations, and an absorption metric.

  • Results

    Feature absorption occurs in every LLM SAE tested, including hundreds of open-source SAEs, with more specific latents absorbing general feature directions.

  • Takeaways & Limitations

    Absorption can make SAE latents unreliable classifiers and complicate high-stakes behavior analysis and sparse circuit descriptions.

  • Takeaways & Limitations

    The absorption metric is conservative, misses some multi-latent or weak-activation cases, and cannot be applied past layer 17 in Gemma 2 2B.

Abstract

from arXiv · show

Sparse Autoencoders (SAEs) aim to decompose the activation space of large language models (LLMs) into human-interpretable latent directions or features. As we increase the number of features in the SAE, hierarchical features tend to split into finer features ("math" may split into "algebra", "geometry", etc.), a phenomenon referred to as feature splitting. However, we show that sparse decomposition and splitting of hierarchical features is not robust. Specifically, we show that seemingly monosemantic features fail to fire where they should, and instead get "absorbed" into their children features. We coin this phenomenon feature absorption, and show that it is caused by optimizing for sparsity in SAEs whenever the underlying features form a hierarchy. We introduce a metric to detect absorption in SAEs, and validate our findings empirically on hundreds of LLM SAEs. Our investigation suggests that varying SAE sizes or sparsity is insufficient to solve this issue. We discuss the implications of feature absorption in SAEs and some potential approaches to solve the fundamental theoretical issues before SAEs can be used for interpreting LLMs robustly and at scale.

1 Introduction

SAEs are intended to make LLM representations more interpretable, but hierarchical features can cause them to learn latents with arbitrary false negatives. The paper calls this failure feature absorption and shows it threatens reliable downstream interpretation.

  • SAEs seek interpretable feature directions because LLM performance has outpaced understanding of their internal mechanisms.
  • Hierarchical features can make SAEs fail to recover the true underlying features even when those features are linear and sparsely activating.
  • More specific latents can absorb part of a general feature direction, allowing the SAE to use fewer active latents while preserving reconstruction.
  • Feature absorption occurs when a seemingly interpretable latent fails to activate on cases where its mainline interpretation suggests it should.
  • The paper develops an absorption metric and reports the phenomenon across hundreds of open-source LLM SAEs.
  • Absorption makes SAE latents potentially unreliable classifiers, complicating high-confidence behavior analysis and sparse circuit descriptions.

2 Background

This section defines hierarchical features, probing methods, SAEs, and ablation-based evaluation used to study feature representations and their causal effects.

  • A feature hierarchy exists when every activation of a child feature implies activation of its parent feature.
  • A linear probe is a linear classifier trained on neural-network hidden activations, typically using logistic regression.
  • K-sparse probing selects k neurons or SAE latents and trains a linear classifier using only that sparse subset.
  • An SAE uses an encoder, decoder, biases, and a nonlinearity to map input activations into hidden representations and reconstructions.
  • SAE training encourages sparse hidden activations through an L1 penalty or a specified number of nonzero latents.
  • SAE feature ablation measures a latent’s downstream causal effect by patching its activation to zero and observing a metric change.

3 Toy models of feature absorption

Toy experiments show that independent features can be recovered cleanly, whereas hierarchical co-occurrence causes absorption: specific latents incorporate general directions and general latents develop recall holes.

  • Independently firing features: With independently firing features, the SAE perfectly recovers the four true features using one latent per feature.Decoder representations match the true feature representations, and encoder activations separate the features.
  • Hierarchical features cause absorption: The hierarchical setup makes feature 1 fire only when feature 0 also fires, while preserving feature 1’s overall firing rate.
  • Hierarchical features cause absorption: In the hierarchical setup, latent 0 represents feature 0 but stops firing when feature 1 is also active, creating a hole in recall.
  • Hierarchical features cause absorption: Latent 1 absorbs the feature 0 direction and therefore represents a combination of feature 0 and feature 1.
  • Hierarchical features cause absorption: The independently firing features 2 and 3 remain perfectly represented despite the hierarchical co-occurrence between features 0 and 1.
  • Hierarchical features cause absorption: An analytical proof shows that feature absorption decreases SAE loss in the hierarchical setup.

4 Experimental setup

The experiments probe first-letter information from LLM residual activations, evaluate SAE latents and causal effects, and compare sparse latent classifiers across different k values and SAE collections.

  • The LLM SAE experiments predict the first letter of a single token using in-context prompts and residual-stream activations at the target token.
  • The study uses an 80%/20% train/test split and evaluates probes only on held-out test data.
  • The causal metric is the correct-letter logit minus the mean logit of all incorrect uppercase letters.It measures the model’s propensity to choose the correct starting letter over alternatives.
  • K-sparse probing increases k from 1 to 15, selecting the top k latents using a logistic-regression probe with an L1 penalty.
  • The experiments use Gemma-2-2B with Gemma Scope SAEs of widths 16k and 65k, plus SAEs trained on Qwen2 0.5B and Llama 3.2 1B.

5 Results

The results show that SAE latents often underperform linear probes on first-letter classification, with precision and recall varying by sparsity and width. Case studies and broader measurements indicate that hierarchical feature structure produces absorption, while the metric used to detect it is conservative and depth-limited.

  • First-letter classification: SAE latents show wide first-letter classification variance, and no tested SAE matches linear-probe performance.The comparison uses SAE latents selected by encoder cosine similarity or k=1 sparse probing.
  • First-letter classification: Low-L0 SAEs tend toward high precision and low recall, whereas high-L0 SAEs tend toward low precision and high recall.The authors caution that letter-level variation prevents identifying an optimal L0 from these plots alone.
  • Feature absorption case study: Latent 6510 achieves F1 0.81 for “starts with S” but fails on _short, where token-aligned latent 1085 activates instead.On _short, latent 1085 contains a smaller probe-aligned component but activates at roughly five times the magnitude of latent 6510.
  • Feature absorption case study: Ablation identifies latent 1085 as causally responsible for the model’s “starts with S” behavior on _short, and removing its probe component eliminates the effect.This supports the interpretation that the first-letter feature has been absorbed into the token-aligned latent.
  • Feature absorption: Feature absorption occurs when specific latents replace apparently interpretable feature latents on arbitrary positive cases, likely because sparsity rewards merging co-occurring dense and sparse features.The broader SAE measurements report increasing absorption with higher sparsity and width, including in multiple SAE families.
  • Measurement limitations: The absorption metric is conservative: it misses multiple absorbing latents, weak activation of main latents, and absorption beyond layer 17 in Gemma 2 2B.Its causal-ablation requirement limits application at deeper layers after attention has moved starting-letter information to the final-token position.

6 Related work

Prior work established linear representations, probing, and feature splitting as tools for interpreting LLM activations, while highlighting limitations in precision, recall, and causal relevance. This paper extends that work by studying recall across many SAEs and explaining lower-than-expected recall through feature absorption.

  • Probing can extract interpretable information from language models, but this does not guarantee that models use those representations computationally.
  • Linear representation work found that concepts can be recovered as directions in activation space and, in some cases, edited.
  • Existing SAE evaluation has examined objective metrics, poor precision and recall, and the limits of max-activating examples.
  • Feature splitting research showed that SAE width and sparsity affect latent granularity, while Meta-SAEs decompose latents into more specific subcomponents.

7 Discussion

The discussion argues that feature absorption undermines the reliability of SAE latents and sparse circuit descriptions, while identifying measurement limits and possible directions for mitigation. The authors also suggest absorption may reveal feature hierarchies.

  • The absorption metric relies on ablation to establish causal mediation, limiting its transferability to final model layers.
  • Proposed directions include Meta-SAEs, attribution dictionary learning, group lasso, and hierarchical sparse coding.
  • Absorption may also help recover feature hierarchies because it can create asymmetric encoder and decoder patterns.
  • Feature absorption can make SAE latents unreliable classifiers, complicating analyses of bias, deception, and sparse circuits.
  • Hierarchical co-occurrence between sparse and dense features causes absorption by allowing a specific latent to encode a more general feature direction.
  • The authors call for methods that detect missed activations and theoretical solutions to absorption.

A.1 Glossary of Terms

The glossary defines SAEs, latent and feature terminology, interpretability, common interpretability methods, and model components used throughout the paper. It also distinguishes human-interpretable concepts from SAE hidden-layer neurons.

  • SAEs reconstruct language-model activations while enforcing hidden-layer sparsity to obtain more interpretable features.
  • A latent is an SAE hidden-layer neuron, whereas a feature is an idealized human-interpretable concept represented in model activations.
  • Interpretability is operationalized by whether a latent behaves as its activation patterns suggest, measured through classification performance.
  • A feature dashboard displays activation patterns and maximally activating examples for an SAE latent.
  • Feature splitting occurs when a general latent in a narrow SAE becomes multiple specific latents in a wider SAE.
  • Feature absorption occurs when a specific latent takes over part of an interpretable feature direction, creating arbitrary missed activations by the main latent.
  • Ablation, integrated gradients, and activation patching assess causal effects or attribution by modifying activations or estimating their importance.

A.2 Proof: absorption decreases SAE loss for hierarchical features

For hierarchically related features, the proof parameterizes absorption by δ and shows that increasing absorption preserves reconstruction while reducing sparsity loss. Therefore, sparsity optimization favors greater absorption.

  • The proof models parent and child features with absorption parameter δ, ranging from no absorption at 0 to full absorption at 1.
  • Perfect reconstruction is maintained for every value of δ.
  • The sparsity loss attributable to the hierarchical features decreases with δ when the child feature appears with nonzero probability.
  • Optimizing the sparsity objective therefore encourages larger δ and greater feature absorption.

1. Preliminaries and Assumptions

The analysis models hierarchical features with orthogonal unit vectors and an SAE whose reconstruction combines ReLU-encoded latents with a sparsity-regularized loss. A parameter δ controls the proposed absorption configuration.

  • Dataset and Features: Features are unit-norm, mutually orthogonal vectors, and each activation is a linear combination of active features.The model assumes h ∈ R^k and uses orthogonality to simplify the analysis.
  • Feature Hierarchy and Probabilities: The hierarchy f2 ⊂ f1 means f2 can occur only when f1 is present, so p(¬f1, f2) = p01 = 0.The allowed cases are co-occurrence, parent-only occurrence, and neither feature occurring.
  • Feature Hierarchy and Probabilities: The probabilities of co-occurrence, parent-only occurrence, and neither feature occurring satisfy p11 + p10 + p00 = 1.Here p11, p10, and p00 denote the three allowed combinations under the hierarchy assumption.
  • Sparse Autoencoder Model: The SAE reconstructs h as W_d z, with z = ReLU(W_e h), and analyzes latents z1 and z2 intended to capture f1 and f2.Other latents are assumed to reconstruct orthogonal features perfectly without interacting with f1 or f2.
  • SAE Loss Function: The loss combines reconstruction and sparsity terms as L = L_rec + λL_sp, with λ > 0.The analysis focuses on the contributions of z1 and z2 to these loss components.
  • δ-Absorption: The δ-absorption parameterization modifies the encoder for z1 and decoder for z2 while leaving the other listed weights unchanged.Specifically, W_e,1 = f1 − δf2, W_e,2 = f2, W_d,1 = f1, and W_d,2 = f2 + δf1; δ = 0 means no absorption and δ = 1 means full absorption.

2. Proposition 1: Perfect Reconstruction under δ-Absorption

Under the hierarchical-feature assumptions, the δ-absorption parameterization reconstructs every allowed input exactly, including parent-only and parent-plus-child cases. Consequently, the reconstruction loss from f1 and f2 is zero for any δ ∈ [0, 1].

  • Proposition Result: The reconstruction loss contribution from f1 and f2 is zero under this parameterization.The child decoder contribution perfectly reconstructs the relevant component, and the proposition concludes L_rec^(1,2) = 0.
  • Case 1: Parent Feature Only: For h = f1, the parameterization activates z1 = 1 and z2 = 0, yielding exact reconstruction ˆh = f1.The encoder-decoder pair for the parent feature reconstructs the input without using z2.
  • Case 2: Parent and Child Features: For h = f1 + f2, the decoder contributions combine to ˆh = (1 − δ)f1 + f2 + δf1 = f1 + f2.The δ-dependent terms cancel in the reconstructed activation.
  • Case 3: Neither Feature: For h = 0, both latents are zero and the reconstruction is exactly ˆh = 0.This is the neither-feature case under the assumed input categories.
  • Case 4: Child Feature Only: The input h = f2 alone is disallowed because the hierarchy f2 ⊂ f1 requires f1 whenever f2 is present.Thus the proof does not need to establish reconstruction for the child-only case.

3. Proposition 2: Sparsity Loss under δ-Absorption

Under δ-absorption, reconstruction remains perfect while sparsity loss decreases as absorption increases when hierarchical features co-occur. Thus, sparsity-regularized optimization favors absorption, although related experiments also show its effects can be partial and task-dependent.

  • Expected sparsity loss: The expected sparsity loss is computed by weighting latent activity across joint feature cases, with the p01 case excluded because it cannot occur.For h = f1 + f2, the contribution is 2 − δ; for h = f1 it is 1; and for h = 0 it is 0.
  • Effect of absorption: If the child feature co-occurs with the parent, increasing δ strictly decreases the sparsity loss.The derivative is −p11, which is negative when p11 > 0.
  • Effect of absorption: Sparsity loss decreases throughout δ ∈ [0, 1], reaching its minimum at δ = 1, corresponding to full absorption.The maximum occurs at δ = 0, or no absorption.
  • Implication for optimization: Perfect reconstruction is maintained for all δ, so increasing absorption reduces total loss without sacrificing reconstruction under the toy mechanism.The reconstruction component is constant, while the sparsity component is p11(2 − δ) + p10.
  • Implication for optimization: With L = Lrec + λLsp and λ > 0, gradient descent therefore favors increasing δ toward 1 for hierarchical features under the stated encoder and decoder conditions.The formal result identifies absorption as a sparsity-improving mechanism that does not harm reconstruction in this setting.
  • Experimental extensions: In experiments, absorption also appears without L1 loss and can become partial when feature magnitudes vary, weakening or eliminating the parent latent below a threshold.The top-k experiment attributes the learned pattern to improved sparsity, while magnitude variation produces partial absorption.
Loading 2409.14507v6…