PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Feature Sparsity in Language Models

Mingyang Deng, Lucas Tao, Joe Benton

arXiv:2310.07837v2cs.LG

TL;DR

Identifying the features language models use is important for interpretability and safe deployment, but assessing sparse-coding accounts of their activations requires rigorous measures. The paper proposes metrics for that assessment and uses them to provide evidence that activations can be modeled as sparse linear combinations of features.

  • Problem

    Identifying the features models use matters for interpretability and safe deployment, while sparse-coding methods need rigorous measures of how well they model activations.

    Source: §1 · pp. 1–2

  • Method

    The authors introduce metrics for evaluating sparse coding on neural-network activations and test their robustness on synthetic activations.

    Source: §1 · p. 2

  • Results

    The metrics provide quantitative evidence that neural-network activations can be accurately modeled as sparse linear combinations of feature vectors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Determining the features models rely on is a key step toward interpreting them and supporting safer deployment.

    Source: §1 · p. 1

    Final loss is not invariant to scaling, so it does not support useful comparisons across models or layers.

    Source: §3 · p. 3

Abstract

from arXiv · show

Recent works have proposed that activations in language models can be modelled as sparse linear combinations of vectors corresponding to features of input text. Under this assumption, these works aimed to reconstruct feature directions using sparse coding. We develop metrics to assess the success of these sparse coding techniques and test the validity of the linearity and sparsity assumptions. We show our metrics can predict the level of sparsity on synthetic sparse linear activations, and can distinguish between sparse linear data and several other distributions. We use our metrics to measure levels of sparsity in several language models. We find evidence that language model activations can be accurately modelled by sparse linear combinations of features, significantly more so than control datasets. We also show that model activations appear to be sparsest in the first and final layers.

1 Introduction

Language-model interpretability needs scalable ways to identify the features models use, but existing approaches often require human input and do not test whether sparse-coding assumptions hold. This paper introduces quantitative metrics to assess sparse coding and analyze feature sparsity in neural-network activations.

1 Introduction

  • Existing language-model interpretability methods can locate facts or linguistic operations, but often require human input and scale poorly to extracting many features.Feature-description plausibility also says little about whether the underlying linearity and sparsity assumptions are accurate.Source: §1 · p. 1
  • Polysemantic neurons motivate superposition models, which assume activations linearly combine feature vectors and only a small number of features are active per input.The assumptions are the linear representation hypothesis and the sparsity hypothesis [17].Source: §1 · p. 1
  • Sparse coding has been used to decompose language-model activations into sparse combinations of interpretable feature vectors [48, 15].Prior evaluations based on plausible descriptions of active inputs assess interpretability, but not the accuracy of the linearity and sparsity assumptions.Source: §1 · p. 1
  • The paper proposes quantitative metrics to assess sparse-coding success and test how accurately activations can be modeled as sparse linear combinations of feature vectors.Its analysis also considers how sparsity varies across model types and layers.Source: §1 · p. 2

2 Background

The paper frames feature recovery as finding a linear dictionary whose feature vectors reconstruct activations, with few active features per input. Sparse coding optimizes this reconstruction while regularizing coefficient sparsity.

2 Background

  • Under the linear representation hypothesis, an activation is approximately a nonnegative weighted sum of vectors for features present in its input.The dictionary Φ contains the feature vectors; m is its size and d is the embedding size.Source: §2 · p. 2
  • The sparsity hypothesis requires only a small number k of features to represent an activation, with k ≪ d.The paper motivates this by expecting most features to be active on relatively few inputs [17].Source: §2 · p. 2
  • Sparse coding seeks Φ and sparse nonnegative α such that the activation matrix X is approximately Φα, with a small residual ε.The setup can use a fixed set of observed activations or an activation distribution.Source: §2 · p. 2
  • The sparse-coding objective is minimized over α and Φ, using the L1 norm as a continuous relaxation of L0 sparsity and λ to control sparsity regularization.Dictionary columns are constrained to unit L2 norm, and the objective is minimized iteratively, similarly to [48].Source: §2 · p. 2
  • For a fixed Φ, α(Φ) denotes the coefficients minimizing the objective, and L(Φ) is the resulting objective value.In practice, α(Φ) is approximated using the same optimization procedure used for the sparse-coding objective.Source: §2 · p. 2

3 Metrics for Success of Sparse Dictionary Learning

The paper proposes four sparse-coding metrics and tests their behavior on synthetic data with known sparsity. Average coefficient norm and normalized loss best track low sparsity and distinguish sparse-linear data from other datasets, with robustness depending on experimental settings.

3 Metrics for Success of Sparse Dictionary Learning

  • The four metrics are judged by scale invariance, continuity under small activation perturbations, and whether they approximate k for data with k fully active features on average.The paper considers two previously used metrics and two it describes as novel.Source: §3 · p. 2
  • The nonzero-entry metric counts nonzero coefficients on average and is intuitive and scale-invariant, but is not robust in practice.It is denoted N0(Φ).Source: §3 · p. 3
  • Final loss is continuous but not scale-invariant, so it does not support useful comparisons across models or layers within a model.The paper considers the final value of L(Φ) as a metric.Source: §3 · p. 3
  • The average coefficient norm Sp is scale-invariant and, when nonzero coefficients share one value, corresponds to the average number of active features.The paper typically uses p = 1 for simplicity and robustness.Source: §3 · p. 3
  • Normalized loss divides L(Φ) by λ times the average maximum coefficient, restoring scale invariance and adding a penalty for unreconstructed activation components.Under perfect reconstruction it equals S1(Φ); imperfect reconstruction makes it more stable over a range of hyperparameters.Source: §3 · p. 3

3.1 Experimental Verification of Metrics

  • For metric values below 20, normalized loss and average coefficient norm closely approximate true sparsity, while nonzero counts overestimate it and final loss is not scale-invariant.The metrics are compared against true sparsity in Figure 1(a).Source: §3.1 · p. 3, Figure 1 · p. 3
  • Average coefficient norm and normalized loss clearly distinguish sparse-linear data from the other datasets, whereas nonzero entries and final loss perform less well.Figure 1(b) compares metrics for sparse-linear and non-sparse-linear data.Source: Figure 1 · p. 3, §3.1 · p. 4
  • Results are robust to changes in noise, dictionary size, and feature count, and relatively robust to embedding size unless sparsity is near that size.Performance may deteriorate when ground-truth feature count greatly exceeds embedding dimension, as the data starts to look approximately Gaussian.Source: §3.1 · p. 4

4 Demonstrating Sparsity in Language Model Activations

Using normalized loss to assess sparse decompositions, the authors find evidence of sparse structure in token embeddings and across BERT layers, with sparsity greatest in the embedding and latest layers.

4 Demonstrating Sparsity in Language Model Activations

  • The language-model study applies sparse coding to token embeddings and to activations from every layer of four BERT models, using normalized loss as its primary measure.Normalized loss is selected because it is more robust in practice; average coefficient results are reported in appendices.Source: §4 · p. 4, §4.1 · p. 4, §4.2 · p. 5

4.1 Embedding Layers

  • Token embeddings in BERT, TinyStories, and GPT-Neo/GPT-2 have much lower normalized loss than a standard Gaussian reference.Their loss values fall in the range where the metric correctly predicts ground-truth sparsity on synthetic sparse linear data, assuming linearity and sparsity.Source: §4.1 · p. 4, Figure 2 · p. 4
  • The number of active features increases more slowly than embedding size, and BERT representations are slightly sparser than TinyStories representations.The authors also report that discovered feature decompositions often correspond to natural human interpretations.Source: §4.1 · pp. 4–5

4.2 Later Layers

  • All BERT layers are sparse by normalized loss, with sparsity highest in the embedding layer, falling at intermediate depths, then rising in the latest layers.The authors suggest deeper layers may detect more higher-level features before the final layers extract key features for next-token prediction.Source: §4.2 · p. 5, Figure 3 · p. 5

5 Social Impacts

The paper situates sparse coding among feature-extraction and superposition research, formalizes activations as samples from a distribution, and argues that quantitative sparsity measures can support more detailed interpretability studies.

5 Social Impacts

  • The paper reports that activations across model architectures and layers are sparser than several controls, supporting the linearity and sparsity hypotheses, with greatest sparsity in first and final layers.It argues that better quantitative assessment can encourage more rigorous tests of these hypotheses and further study of superposition.Source: §5 · p. 5

A Related Works

  • Related work spans linear representations, feature extraction, superposition, sparse-coding algorithms, and automated interpretability; this paper's sparse-coding method variants an iterative procedure from Yun et al. [48].Prior language-model sparse-coding work assessed features by seeking common interpretations among inputs where each feature was active.Source: A Related Works · p. 9

B Setup for an Activation Distribution

  • The setup treats activations as samples from a distribution and seeks feature directions whose sparse coefficients reconstruct each activation with small expected error.The paper uses fixed activation matrices for notation, replacing row-wise sums with expectations for a distribution.Source: B Setup for an Activation Distribution · p. 9

C Details of Sparse Coding Algorithm

Sparse coding alternates between updating feature directions and greedily assigning nonnegative coefficients until convergence.

C Details of Sparse Coding Algorithm

  • The algorithm alternates several stochastic-gradient steps minimizing the sparse-coding objective over feature directions with a greedy update of coefficients.The procedure is inspired by Yun et al. [48].Source: C Details of Sparse Coding Algorithm · p. 9
  • For each activation, the greedy step repeatedly selects the feature with the largest dot product, assigns its coefficient, and subtracts that feature's contribution while the coefficient remains positive.The coefficient choice corresponds to projection under the L1 norm; once a coefficient is nonpositive, remaining coefficients are set to zero.Source: C Details of Sparse Coding Algorithm · p. 10
  • Feature directions and coefficients are updated sequentially until convergence, with activations resampled each iteration when using a distribution.The authors find an effective regularization choice near 10% of the maximum coefficient magnitude and typically need only a couple of adaptive updates.Source: C Details of Sparse Coding Algorithm · p. 10

D Further Details on Metric Verification Experiments

The verification experiments construct synthetic sparse-linear activations and compare metrics against several non-sparse or differently structured controls.

D Further Details on Metric Verification Experiments

  • Synthetic activations combine randomly sampled unit-sphere feature vectors with sparse coefficients, then center the proto-activations and add Gaussian noise.The coefficient probability is set so each proto-activation sums an average of a activated vectors.Source: D Further Details on Metric… · p. 10
  • Because active feature weights are uniform between zero and one, the expected weighted number of active features is a/2, which the authors treat as the correct metric value.Source: D Further Details on Metric… · p. 10
  • The controls include identity-covariance Gaussian data, heavy-tailed isotropic data, and a d-dimensional Rademacher distribution, each scaled to identity covariance.The Rademacher control is sparse but is not well represented by a sparse linear combination of features.Source: D Further Details on Metric… · p. 10
  • Nonzero-entry counts fail to reliably distinguish heavy-tailed non-sparse data, while final loss fails to reliably distinguish Gaussian data from sparse-linear data.Datasets are scaled to mean zero and average L2 norm one to make final-loss values comparable.Source: D Further Details on Metric… · p. 10, Figure 4 · p. 10

E Ablations for Metric Verification Experiments

Across synthetic-data ablations, the metrics remain useful under changes to dictionary size, noise, embedding dimension, and ground-truth feature count, with limits at low embedding dimension and higher metric values.

E Ablations for Metric Verification Experiments

F Further Details on Language Model Embedding Experiments

The embedding experiments use centered vectors and sparse coding, and the resulting decompositions explain over 90% of activation variance. Using average coefficient rather than normalized loss yields very similar results.

F Further Details on Language Model Embedding Experiments

G Examples of Interpretable Features in Embedding Space

The paper examines sparse decompositions of the GPT-2 embedding vectors for “season” and “physics” and reports that their top features have natural interpretations from maximally activating tokens.

G Examples of Interpretable Features in Embedding Space

H Further Details of Experiments on Later Layers in Language Models

The later-layer experiments sample Wikipedia abstracts and use fixed regularization choices to explain at least 98% of variance. Replacing normalized loss with average coefficient gives qualitatively similar results.

H Further Details of Experiments on Later Layers in Language Models

Related papers

Loading 2310.07837v2…