PaperlayerView exact revision on arXiv

Source-linked AI summary

Finding Neurons in a Haystack: Case Studies with Sparse Probing

Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, Dimitris Bertsimas

arXiv:2305.01610v2cs.LGcs.AI

TL;DR

The paper addresses how interpretable features are represented inside LLM neurons. It trains k-sparse probes to localize relevant neurons across models and scales, finding superposition in early layers, seemingly dedicated contextual neurons in middle layers, and increased average sparsity with scale but varied feature dynamics.

  • Problem

    LLM internal computations are opaque, and how high-level human-interpretable features are represented in neuron activations remains poorly understood.

  • Method

    The authors train k-sparse linear classifiers on internal activations, varying k to localize relevant neurons and study representation sparsity across more than 100 features and multiple models.

  • Results

    Early layers use sparse combinations of polysemantic neurons, middle layers seemingly contain dedicated neurons for higher-level contextual features, and average representation sparsity increases with model scale.

  • Takeaways & Limitations

    Sparse probing reveals rich interpretable structure in LLM neurons, including monosemanticity, polysemanticity, and superposition, while requiring careful follow-up analysis.

  • Takeaways & Limitations

    Sparse probing cannot by itself support strong conclusions, establish causation, detect features built across multiple layers, or easily distinguish superposition from unions of granular features.

Abstract

from arXiv · show

Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train $k$-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of $k$ we study the sparsity of learned representations and how this varies with model scale. With $k=1$, we localize individual neurons which are highly relevant for a particular feature, and perform a number of case studies to illustrate general properties of LLMs. In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.

1 Introduction

The paper introduces sparse probing to localize neurons representing interpretable features and uses it to study superposition, contextual representations, and scaling across language models.

  • Motivation: Superposition compresses more features than available neurons, increasing representational power while producing polysemantic neurons that respond to unrelated stimuli.The paper connects this phenomenon to the challenge of interpreting internal representations.
  • Scope: The study examines over 100 features across 10 categories and 7 models spanning two orders of magnitude in parameter count, up to 6.9 billion parameters.Sparse probing is presented as effective for locating interpretable neurons but dependent on careful follow-up analysis.
  • Findings: Early layers often represent features as sparse combinations of polysemantic neurons, with the first 25% of fully connected layers showing substantially more superposition than later layers.This conclusion combines case-study observations with weight statistics and toy-model insights.
  • Findings: Higher-level contextual and linguistic features are seemingly encoded by monosemantic neurons, predominantly in middle layers.The paper cautions that conclusive claims about monosemanticity remain methodologically out of reach.
  • Findings: As models increase in size, representation sparsity increases on average, while individual features follow different scaling dynamics.Some dedicated neurons emerge, some features become finer-grained, and others remain unchanged or appear somewhat random.

2 Related Work

The paper situates sparse probing within probing, mechanistic interpretability, superposition, and neuroscience research on how neural representations can be understood.

  • Probing: Probing studies whether models represent specific features by training classifiers on internal activations, including work on individual neurons and sparsely represented features.The paper builds on probing’s broader literature while emphasizing its limitations.
  • Mechanistic Interpretability: Mechanistic interpretability seeks to rigorously understand neural-network circuits to support oversight and failure diagnosis in increasingly capable models.The paper is motivated by this goal but focuses on neuron-level representation analysis.
  • Superposition: Superposition represents multiple features in fewer dimensions and is identified as a major obstacle to interpreting neurons and advancing mechanistic interpretability.Related work studies when superposition occurs, how to reduce it, and how to extract features despite it.
  • Connections to Neuroscience: Neuroscience provides an analogue through local coding by monosemantic neurons and sparse coding associated with superposition.The paper presents these parallels as a demonstration of consilience between the domains.

3 Sparse Probing

Sparse probing applies sparse linear classifiers to MLP activations, using feature-aware datasets and evaluation procedures to localize neurons while exposing methodological subtleties.

  • Preliminaries: The study probes transformer language models by applying linear binary classifiers to MLP neuron activations after the elementwise nonlinearity.The MLP activations are treated as a privileged basis because the nonlinearity breaks rotational invariance.
  • Sparse Probes: A k-sparse probe has at most k non-zero coefficients and seeks a single neuron or sparse subset that predicts whether a feature is present.Selecting the most predictive neuron subsets is generally combinatorial, motivating specialized algorithms.
  • Sparse Selection: Optimal sparse probing trains small-k classifiers to provable optimality, while adaptive thresholding handles larger k ranges through iteratively retrained classifiers.Both methods select neurons before retraining logistic regression probes.
  • Probing in Practice: Probe datasets must shape negative examples carefully to separate the target feature from correlated alternatives such as is_politician, is_political, and is_person.The central design tension is creating conceptual separation from possible correlates.
  • Probing in Practice: For sequence-level features, activation aggregation over token spans can provide probe inputs but may fail to distinguish more specific feature variants.The appropriate aggregation strategy depends on the feature and the researcher’s intent.
  • Evaluation and Interpretation: Precision, recall, and F1 are computed from held-out-test confusion counts, with F1 used as the primary metric for identifying neurons associated with the target feature.Precision and recall also indicate whether selected neurons are more general or polysemantic than the probed feature.

4 Empirical Overview

The empirical study spans multiple model scales, feature collections, layers, and sparse-selection methods, with held-out evaluation and follow-up analyses of relevant neurons.

  • Models: The experiments use 7 Pythia autoregressive transformer models ranging from 70M to 6.9B parameters.The models are GPT variants trained on The Pile.
  • Data: The data cover 10 feature collections spanning languages, programming, document sources, grammatical and morphological properties, token features, compounds, LaTeX, and factual features.These collections provide varied linguistic and conceptual targets for probing.
  • Experiments: For each model, feature, layer, and selection method, the authors train probes across k values and evaluate classification on held-out test data.They also analyze activations over larger text datasets for especially relevant neurons.

5 Case Studies

The case studies show superposition, specialized context neurons, scale-dependent sparsity, and the need for follow-up analyses to interpret neurons reliably. They also identify limitations involving ontology, composition, and probing-dataset construction.

  • 5.1 Superposition in the Wild: Compound Word Neurons: 21 compound-word probes found nearly perfect individual-neuron discrimination, but broader-corpus analysis revealed activation across many unrelated n-grams.This polysemanticity illustrates how maximum-activating examples can create interpretability illusions.
  • 5.1 Superposition in the Wild: Compound Word Neurons: Three polysemantic neurons jointly achieved nearly perfect separation for the “social security” bigram from other observed token combinations.The authors propose that linear combinations of polysemantic neurons merge tokens into semantically meaningful n-grams in early layers.
  • 5.3 Effects of Scale: Quantization and Splitting: Representation sparsity increases on average with model scale, but individual features follow different dynamics, including increased sparsity, consistency, or apparent randomness.The authors identify quantization and neuron splitting as two main dynamics behind these patterns.
  • 5.4 Refining and Verifying Interpretations: A low-accuracy 1-sparse probe and high-accuracy k-sparse probe may indicate superposition, a coarse feature, or composition across aligned features.The authors note that composition and superposition can also operate across multiple layers, which their analysis ignores.
  • 5.4 Refining and Verifying Interpretations: Interpretation requires analyses beyond probing because precision and recall depend on labels that may contain imbalance or spurious correlations.For an is_title neuron, F1 > 0.95 nonetheless explained less than half of activations because many others were section titles.

6 Discussion

Sparse probing can precisely localize feature-relevant neurons and accommodate superposition, but interpreting its results requires substantial secondary analysis. The paper presents broad evidence for superposition and neuron types while emphasizing limits in scope, causality, and generalization.

  • Strengths and Weaknesses of Sparse Probing: Sparse probing quickly localizes neurons relevant to specific features while accounting for superposition and composition.Its minimum-capacity probes and optimality guarantees address concerns about probe expressivity and classification-versus-ranking conflation.
  • Strengths and Weaknesses of Sparse Probing: Probing results require detailed secondary analysis because they offer limited causal insight and are sensitive to dataset and implementation choices.Sparse probes may also miss redundant important neurons, struggle with multi-token features, and confuse superposition with unions of granular features.
  • Empirical Findings: The authors corroborate case studies with theoretical predictions, ablations, and vocabulary analyses, finding compelling but not always conclusive evidence.They characterize their findings as especially clear evidence to date for superposition, monosemantic neurons, and polysemantic neurons in full-scale language models.
  • Strengths and Weaknesses of Empirical Findings: The analysis is largely ad hoc, uses only one model family and training dataset, and reaches 6.9 billion parameters rather than frontier scale.The authors also restrict their conclusions to binary features and categorical features converted to binary features.
  • Implications and Future Work: The paper argues that superposition is important for model success and suggests focusing high-leverage interventions on early layers.It also identifies probing datasets for bias, fairness, safety, and high-stakes decision-making as a potential application area.

7 Conclusion

The paper presents sparse probing as a route to studying rich but difficult-to-interpret structure in LLM neurons, while distinguishing superposition, polysemanticity, and distributed representations. Its evidence includes early-layer superposition, seemingly monosemantic middle-layer neurons, and methodological limits on definitive interpretation.

  • Conclusion: Sparse probing reveals interpretable neuron structure in LLMs, including examples of monosemanticity, polysemanticity, and superposition.The authors emphasize that rigorous conclusions require careful use of probing and follow-up analysis.
  • Conceptual distinctions: Superposition represents more features than available dimensions, whereas distributed representations use feature directions that need not align with individual neurons.Polysemanticity concerns neurons representing multiple unrelated concepts, while superposition is a global property requiring more features than neurons.
  • Conceptual distinctions: Polysemanticity and distributed representations can occur without superposition, while superposition necessarily implies polysemantic neurons and likely distributed representations.These distinctions matter because local observations about neurons or features do not by themselves establish global superposition.
  • Methodological limits: Clustering-based identification of polysemanticity remains subjective because deciding whether activation clusters share meaning is not formalized.The paper identifies formalization of faceted and polysemantic neurons as future work.

A.12 Which sparse probing method should I use?

The paper recommends choosing among sparse probing methods according to runtime, sweep breadth, and the need for formal guarantees. It also stresses that probe interpretability depends on feature definitions, causal validation, and the specific role of neurons across transformer layers.

  • Method selection: Maximum mean difference is best for extremely fast or scalable experiments, adaptive thresholding for broad k sweeps, and optimal sparse probing when formal guarantees are required.Optimal sparse probing may require substantially more compute time.
  • Interpretation: Linear probes can falsely suggest nonlinear feature representations because a nonlinear probe may compute complex features from simpler linearly represented features.Probe complexity can therefore obscure what the model itself represents.
  • Layer roles: Neurons may also disambiguate tokens appearing in different contexts and interact with attention-head circuits to clarify edge cases.Examples include language disambiguation and support for induction-head or skip-trigram functions.
  • Layer roles: Early-layer MLPs can be viewed as detokenizing raw tokens into semantic concepts, while late-layer MLPs retokenize semantic features into concrete output tokens.The paper compares these roles to sensory and motor neurons in biological systems.
  • Probe design: The probes are trained on post-GELU neuron activations because a linear map connects them to the residual stream, making their linear combinations meaningful to the network.Pre-GELU histograms are sometimes plotted only for clearer visualization of negative activations.

A.16 Did you have any negative results?

The paper reports negative results for some features and for causal tests of factual neurons. Sparse probes failed to isolate a dedicated falsehood feature, and single-neuron ablations had limited behavioral impact.

  • Feature datasets: Prefix and suffix features did not appear sparsely represented, despite achieving fairly high accuracy with dense probes.Because these properties belong to single tokens, they may already exist in token embeddings rather than dedicated neurons.
  • CounterFact: Sparse probes performed poorly on falsehoods in CounterFact, although increasing k reached 90% accuracy at k = 1024 on the best layer.The authors interpret this as detecting confusion or dissonance around unexpected completions rather than truth or falsehood itself.
  • Causal tests: Ablating individual factual neurons had very limited effect on few-shot classification of gender, occupation, and whether people are alive.Possible explanations are earlier computation of the fact or ensemble redundancy across neurons.
  • Interpretive language: Anthropomorphic descriptions such as models “wanting” to minimize interference are shorthand for stochastic gradient descent selecting lower expected loss.The authors explicitly distinguish this usage from human intentionality.

B Experimental Details

The experiments study EleutherAI’s Pythia language models, whose architectural hyperparameters are summarized in Table 1.

  • B.1 Models: The study uses EleutherAI’s Pythia autoregressive transformer models trained on The Pile.The models use parallel attention and rotary positional encodings.

B.2 Datasets

The paper assembles probing datasets spanning natural language, code, data sources, linguistic structure, text properties, compound words, LaTeX, and factual features. Table 2 summarizes these datasets, while the collection-specific procedures define how examples and labels are constructed.

  • B.2 Datasets: Table 2 summarizes the probing datasets and their feature collections.The paper provides descriptions of dataset design, preprocessing, and motivation.
  • Natural Language: EuroParl sequences are randomly sampled from documents and mean-aggregated for probing.Random contiguous subsequences reduce context clues from document beginnings.
  • Data Subset: The Pile test subsets use randomly selected sequences with labels implied by their source subsets.These data use the same random sub-sequencing and mean aggregation as the natural-language data.
  • Programming Languages: The programming-language dataset filters GitHub source files using a code-recognition prediction above 90% confidence.The procedure also ignores the first 50 tokens during mean aggregation as a coarse cleaning step.
  • Feature Collections: The collections include linguistic, morphological, plain-text, compound-word, LaTeX, and factual features in addition to language, code, and data-source features.Examples include part of speech, verb tense, capitalization, compound words, and properties such as gender or occupation.

B.3 Feature Selection Methods

The paper compares several neuron-selection strategies for k-sparse probes, ranging from simple univariate scores to provably optimal sparse classification and adaptive thresholding.

  • Setup: The feature-selection setup ranks neurons from token activations X ∈ R^n×d and labels y ∈ {−1, 1}^n.Most methods act as scoring algorithms, while optimal sparse probing and adaptive thresholding directly train probes.
  • Scoring Methods: Mean difference scores neurons by the average activation difference between positive and negative classes.
  • Scoring Methods: Mutual information estimates dependence between each continuous neuron activation and the discrete target using a nearest-neighbor approximation.
  • Scoring Methods: L1-regularized logistic regression scores neurons by the absolute values of coefficients from a dense probe.
  • Direct Sparse Probes: Optimal sparse probing trains a cardinality-constrained SVM with hinge loss to provable optimality using cutting planes.Adaptive thresholding repeatedly retrains elastic-net logistic probes while reducing the retained neuron count.
  • Experimental Procedure: All experiments first filter to the top 1024 neurons by mean difference for computational reasons.Class weights are balanced, and hyperparameters are reused across experiments rather than tuned for every trial.

B.5 Feature Selection Results

Feature-selection results compare out-of-sample F1 across sparse probing methods and examine a superposition construction. The methods perform similarly overall, while the construction’s proxy metric decreases monotonically as the number of features grows beyond two.

  • Feature Selection Results: Table 4 reports out-of-sample F1 averaged across models and features at each combination’s best-scoring layer.The comparison evaluates sparse feature-selection methods for k = 1, ..., 8.
  • Feature Selection Results: The top methods are always within 1% of one another, with no method strictly dominating.Even random neurons contain substantial information about the target task.
  • Feature Selection Results: The paper suggests a two-stage design using fast heuristics to find relevant neurons and layers, followed by optimal methods for improvement and verification.Optimal sparse probing often reaches its one-minute timeout, especially for larger k.
  • Superposition Construction: The superposition construction embeds n mutually exclusive one-hot features in two dimensions for later lossless recovery.It places feature representations equally around a circle and applies a ReLU transformation.
  • Superposition Construction: The proxy metric b∥w∥2 = cos(2π/n)(cos(2π/n)−1) decreases monotonically for n > 2.
  • Superposition Construction: The construction concerns residual-stream rather than neuron superposition, and the authors expect the basic interference-cleaning motif to extend when neurons are fewer than features.They also suggest increased residual-stream superposition may be associated with increased neuron superposition.
  • Superposition Construction: The observed weight pattern may have explanations other than superposition, and some superposition mechanisms may not strongly affect weight and bias distributions.

D Additional Results

Additional results examine superposition, neuron-basis alignment, scaling behavior, and examples of monosemantic and factual neurons across Pythia models.

  • Input weight norms and biases are distributed across all Pythia models studied.
  • Figures 9–12 provide additional examples of compound-word superposition and n-gram polysemanticity in Pythia models.
  • Logistic loss for compound-word classification follows power-law scaling until a breakpoint, with the neuron basis achieving lower overall loss than random bases.
  • Figures 15–17 show monosemantic context neurons for language, code, and distribution identification in Pythia models of different sizes.
  • Figures 18–20 examine the highest-performing neuron for each feature, factual-neuron categories, and layerwise top neurons for sex_or_gender and is_alive.
Loading 2305.01610v2…