PaperlayerView exact revision on arXiv

Source-linked AI summary

Toy Models of Superposition

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, Christopher Olah

arXiv:2209.10652v1cs.LG

TL;DR

Reverse engineering neural networks requires decomposability, yet both monosemantic and polysemantic neurons can form. Using toy models to examine feature superposition, the paper finds that whether features are stored in superposition is governed by a phase change.

Abstract

from arXiv · show

Neural networks often pack many unrelated concepts into a single neuron - a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. This paper provides a toy model where polysemanticity can be fully understood, arising as a result of models storing additional sparse features in "superposition." We demonstrate the existence of a phase change, a surprising connection to the geometry of uniform polytopes, and evidence of a link to adversarial examples. We also discuss potential implications for mechanistic interpretability.

KEY RESULTS FROM OUR TOY MODELS · Definitions and Motivation: Features, Directions, and Superposition

The section motivates and defines features as directions in activation space and superposition as encoding more features than dimensions, with toy ReLU models illustrating superposition and related consequences. It distinguishes privileged from unprivileged bases and situates these ideas in a hierarchy of representation properties, while noting that generalization to real networks is unclear.

KEY RESULTS FROM OUR TOY MODELS

  • Toy models exhibit superposition, monosemantic and polysemantic neurons, computation in superposition, a storage phase change, and feature geometries including digons, triangles, pentagons, and tetrahedrons.The authors qualify these findings as demonstrated in simple ReLU networks and say generalization to real networks is unclear.Source: KEY RESULTS FROM OUR TOY MODELS · p. 3

Definitions and Motivation: Features, Directions, and Superposition

  • Interpretability motivates decomposable representations; linearity represents features as directions, while superposition stores more features than dimensions and pushes them away from corresponding to neurons.The paper describes superposition as a way for networks to simulate larger networks.Source: Definitions and Motivation: Features,… · p. 3

Empirical Phenomena

  • Examples motivating feature directions include semantic directions and vector arithmetic in word embeddings, interpretable neurons, and polysemantic neurons responding to unrelated mixtures of inputs.Source: Empirical Phenomena · p. 4

What are Features?

  • The paper considers features as arbitrary functions, interpretable properties, or properties reliably assigned neurons in large models, preferring the last provisionally despite its circularity.The authors note that interpretable properties need not be understandable to humans and caution against attaching too firmly to one definition.Source: What are Features? · pp. 4–5

Features as Directions

  • A representation is linear when each feature corresponds to a direction in activation space; simultaneous feature activations are represented through those directions.Source: Features as Directions · p. 5
  • Linear feature directions can be selected by neurons in the next layer, and may support non-local generalization through linear transformations.The text presents linear representations as a natural neural-network format and contrasts their potential statistical efficiency with local generalization.Source: Features as Directions · p. 6

The Superposition Hypothesis

  • Superposition allows linear representations to store more features than dimensions by encoding sparse features as almost-orthogonal directions, trading interference for additional representational capacity.Sparsity reduces interference because features are rarely active together, while nonlinear activations can filter small amounts of noise.Source: Features as Directions · p. 6, The Superposition Hypothesis · p. 7

Privileged vs Non-privileged Bases

  • Architectural symmetry-breaking can make basis directions special, creating a privileged basis in which basis directions are called neurons and may correspond to interpretable features.A privileged basis does not ensure features align with its directions, but it makes asking whether a neuron is interpretable meaningful.Source: Privileged vs Non-privileged Bases · pp. 6–7

The Superposition Hypothesis

  • Polysemantic neurons that respond to several unrelated features motivate the superposition hypothesis: networks may represent more features than they have neurons.The hypothesis also applies with an unprivileged basis, where superposition means there are more features than dimensions.Source: The Superposition Hypothesis · pp. 7–8

Summary: A Hierarchy of Feature Properties

  • The proposed hierarchy distinguishes decomposable and linear representations, hypothesized to be widespread, from non-superposition and basis alignment, believed to occur only sometimes.Superposition is defined by a non-invertible feature representation; basis alignment means features are one-hot, while partial alignment means they are sparse and requires a privileged basis.Source: Summary: A Hierarchy of Feature… · p. 8

Demonstrating Superposition

The paper asks whether neural networks can represent more features than they have neurons, and argues that slight nonlinearity can enable behavior unlike a linear model’s principal-component storage. It studies this question with synthetic sparse feature vectors and simple models that project and reconstruct them.

Demonstrating Superposition

  • Adding slight nonlinearity lets models behave radically differently from linear models, which store principal components.The paper presents this as its first demonstration of superposition.Source: Demonstrating Superposition · p. 8

Experiment Setup

  • The experiment asks whether a neural network can project a high-dimensional vector into a lower-dimensional vector and recover it.Source: Experiment Setup · p. 9

THE FEATURE VECTOR ( ) X

  • Synthetic feature vectors model sparse features, more features than neurons, and differences in feature importance.The authors use synthetic data because they lack ground truth for features.Source: THE FEATURE VECTOR ( ) X · p. 9

THE MODEL ( )

  • The linear model is a baseline that does not exhibit superposition, whereas the ReLU output model does.Source: THE MODEL ( ) · p. 10

Why these models?

  • The models use a linear projection and its transpose for reconstruction, with an activation function because it is critical to whether superposition occurs.The model also includes a bias, which can set unrepresented features to their expected value.Source: Why these models? · p. 10

THE LOSS

  • The loss is mean squared error weighted by feature importance.Source: THE LOSS · p. 10

Basic Results

  • The basic-results experiment trains ReLU output models at different sparsity levels and includes a linear-model comparison.The supplied passage describes the experiment but does not report its results.Source: Basic Results · p. 10

W W T

The experiments show that ReLU models develop superposition as feature sparsity increases, unlike linear models, and the analysis explains this behavior through competing feature benefits and interference. Under specific assumptions, the ReLU objective also connects to a generalized Thomson problem on the sphere.

W W T

  • Linear models learn the most important features, while ReLU models do so on dense inputs but represent more, non-orthogonal features as sparsity increases.The ReLU model first incorporates less important features and gradually affects the most important ones.Source: W W T · p. 12
  • As ReLU models represent more features, their vectors begin in antipodal pairs and transition to other geometries; larger models show qualitatively similar, rescaled patterns.Source: W W T · p. 12

Mathematical Understanding

  • Linear learning balances feature benefit against interference, so representing more features than dimensions cannot be worthwhile.Representing more features improves loss through feature benefit, but interference worsens loss when the features cannot fit orthogonally.Source: Mathematical Understanding · p. 13
  • With uniform feature importance and a fixed number of important features, the ReLU interference term becomes a generalized Thomson problem of packing points on a sphere.Under these assumptions, the feature-benefit term is constant, leaving a slightly unusual energy function over the sphere.Source: Mathematical Understanding · p. 13
  • ReLU makes negative interference free in the 1-sparse case; features can organize into sparse interference graphs, where only a small number interfere with another feature, reducing the probability of compounding.Source: Mathematical Understanding · pp. 13–14

Superposition as a Phase Change

Feature storage shifts among being unlearned, represented in superposition, or assigned a dedicated dimension, and the transitions between these outcomes appear sharp. Empirical and theoretical diagrams show phase changes as sparsity and relative feature importance vary.

Superposition as a Phase Change

  • With two features in one dimension, the theoretical model compares ignoring either feature with antipodal superposition, which sacrifices representing both features together.The losses for these configurations can be analytically determined.Source: Superposition as a Phase Change · p. 14
  • Sparsity is necessary for superposition, and it interacts in an interesting way with relative feature importance; a phase change appears in both empirical and theoretical diagrams.Source: Superposition as a Phase Change · p. 15
  • The optimal encoding undergoes a phase change in both empirical and theoretical diagrams, discontinuously shifting in magnitude and superposition; theory identifies a first-order crossover.The theoretical crossover causes a discontinuity in the derivative of optimal loss.Source: Superposition as a Phase Change · p. 15
  • For three features in two dimensions, diagrams suggest phase changes among encoding strategies, including pairing the extra feature antipodally or giving it a dedicated dimension.The theoretical model considers four natural solutions; this preliminary view does not capture the more complex structure discussed next.Source: Superposition as a Phase Change · pp. 15–16
  • The preliminary diagrams do not capture the more complex structure of feature-encoding strategies.The authors defer that structure to the next section.Source: Superposition as a Phase Change · p. 16

The Geometry of Superposition

This section examines the geometric structure of superposition in the toy model, including configurations for uniform and non-uniform cases and how higher-dimensional solutions can be built from lower-dimensional ones. It also describes orthogonal factors that prevent interference and a correspondence between polytopes and low-rank matrices.

The Geometry of Superposition

  • The geometric account may be partly idiosyncratic to this toy model, and non-uniform superposition is understood only partly as a deformation of the uniform case.The authors see potential value if any of the geometric structure generalizes to real models.Source: The Geometry of Superposition · p. 16

Uniform Superposition

  • When input features greatly outnumber hidden dimensions, their count matters little, and doubling hidden dimensions doubles the number of learned features.The analysis measures learned features with the Frobenius norm, which remains basis-independent in the dense regime.Source: Uniform Superposition · p. 17
  • A sticky regime features antipodal pairs that pack two features into each hidden dimension, and the model favors them across a wide sparsity range.The graph is also described as sticky at particular dimensions-per-feature values, though those values are missing from the supplied text.Source: Uniform Superposition · p. 17

FEATURE DIMENSIONALITY

  • Feature dimensionalities cluster at specific fractions, revealing distinct geometries and multiple phases within the broader category of superposition.The reported lines correspond to tetrahedra (¾), triangles (⅔), antipodal pairs (½), pentagons (⅖), and square antiprisms (⅜); unlearned features have dimensionality 0.Source: FEATURE DIMENSIONALITY · pp. 18–19

WHY THESE GEOMETRIC STRUCTURES?

  • Observed geometries include uniform-polytope Thomson solutions, while non-uniform solutions can split into configurations such as triangles and antipodal pairs.The model is described as solving a generalized Thomson problem; different features in a non-uniform polyhedron can experience different amounts of interference.Source: WHY THESE GEOMETRIC STRUCTURES? · pp. 19–20
  • Higher-dimensional solutions are often tegum products of 1D, 2D, and 3D solutions, with 3D Thomson solutions combining 2D and 1D solutions.The discussion frames the problem as a higher-dimensional version of the same geometry.Source: WHY THESE GEOMETRIC STRUCTURES? · p. 20
  • Orthogonality between tegum factors prevents features in different factors from interfering, which may benefit the toy model because simultaneous interference could be harmful.The text presents this as an implication of tegum-product orthogonality for superposition.Source: WHY THESE GEOMETRIC STRUCTURES? · p. 20

Aside: Polytopes and Low-Rank Matrices

W W T

The section connects superposition strategies to polytope geometry, examining how feature sparsity, importance, and correlations affect which features are represented together. It describes transitions between superposition and PCA-like representations, including correlated feature sets that form nearly orthogonal local bases.

W W T

  • There is an exact correspondence between polytopes and superposition strategies: three features in two dimensions correspond to a triangle, with an equilateral triangle optimal for equally important, equally sparse features.Projecting basis vectors into a space orthogonal to omitted directions also constructs polytopes; the regular simplex corresponds to minimal possible superposition, and the fully dense vector is the best omitted direction under the stated equal-feature assumptions.Source: W W T · pp. 20–21

Non-Uniform Superposition

  • Non-uniformity from unequal importance, sparsity, or feature correlations distorts uniform superposition geometry, whose theory remains incomplete.The section highlights smooth polytope deformation followed by a snap to another polytope at a critical imbalance.Source: Non-Uniform Superposition · p. 21

PERTURBING A SINGLE FEATURE

  • Sparsifying one feature stretches a regular pentagon’s geometry, and sufficiently strong sparsity causes a first-order transition to two digons with that feature at zero.The transition occurs when the loss curves for the two geometries cross.Source: PERTURBING A SINGLE FEATURE · p. 22
  • Non-uniform superposition can be understood as deformation of uniform configurations and jumps between them, rather than necessarily as a wholly different regime.In the single-feature experiment, a denser feature gains space as other features repel from it, while a sparser feature takes less space.Source: PERTURBING A SINGLE FEATURE · pp. 22–23

Correlated and Anticorrelated Features

  • Correlated features prefer orthogonal representation in separate tegum factors; if that fails, they arrange close together or collapse to their principal component, while anticorrelated features prefer negative interference.When orthogonality is unavailable, correlated features prefer positive over negative interference.Source: Correlated and Anticorrelated Features · p. 23

ORGANIZATION OF CORRELATED AND ANTICORRELATED FEATURES

LOCAL ALMOST-ORTHOGONAL BASES

  • Correlated feature sets can become nearly orthogonal with little superposition, even when the model as a whole is in superposition.If this also holds in real neural networks, local non-superposition could justify using methods such as PCA on selected sub-distributions.Source: LOCAL ALMOST-ORTHOGONAL BASES · pp. 24–25

COLLAPSING OF CORRELATED FEATURES

  • When capacity allows only one of two correlated features, the model represents their principal component—a sparse variable with greater loss impact than either feature alone—and ignores the second component.Source: COLLAPSING OF CORRELATED FEATURES · p. 25
  • In very sparse regimes, correlated features occupy neighboring positions in a hexagon; as sparsity decreases, they collapse into principal components, yielding a PCA-equivalent solution when very dense..Source: COLLAPSING OF CORRELATED FEATURES · p. 25
  • PCA and superposition trade off: greater correlation favors PCA, greater sparsity favors superposition, and sparse, correlated features can produce mixtures of both.Source: COLLAPSING OF CORRELATED FEATURES · p. 26

Superposition and Learning Dynamics

The section reports two observed training-dynamics phenomena, rather than a detailed investigation: feature-dimensionality jumps associated with loss drops, and geometric transformations across learning regimes.

PHENOMENON 1: DISCRETE "ENERGY LEVEL" JUMPS

  • In toy models with many features, training appears dominated by discrete “energy level” jumps between feature dimensionalities, while converged equal-feature solutions occupy a few fractional-dimensionality polytopes.Source: PHENOMENON 1: DISCRETE "ENERGY LEVEL"… · p. 26
  • Feature dimensionalities jump and swap places as loss drops; this suggests apparently smooth loss declines in larger models may comprise many small transitions between feature configurations.Source: PHENOMENON 1: DISCRETE "ENERGY LEVEL"… · p. 27

PHENOMENON 2: LEARNING AS GEOMETRIC TRANSFORMATIONS

  • For correlated features, learning proceeds through loss-visible regimes corresponding to distinct geometric transformations, which in some cases can be understood as simple, independent transformations.Source: PHENOMENON 2: LEARNING AS GEOMETRIC… · p. 27
  • The illustrated path learns a linear PCA solution before a nonlinear one; although the octahedron seems a strong attractor, learning trajectories vary at step C.Source: PHENOMENON 2: LEARNING AS GEOMETRIC… · p. 28

Relationship to Adversarial Robustness

The section links superposition to adversarial vulnerability: interference can expose important features to attack, and vulnerability rises sharply as superposition forms. It also reports that adversarial training reduced superposition only with very large attacks.

Relationship to Adversarial Robustness

  • Superposition interference creates an avenue for attacking the most important feature, even when optimal sparse-data models use superposition to represent more features.Source: Relationship to Adversarial Robustness · p. 29
  • With an L2 attack cap of 0.1 times average input norm, gradient attacks struggled in 99%-zero ReLU regimes, so the authors used analytically derived feature-optimal attacks.Source: Relationship to Adversarial Robustness · p. 29
  • As superposition forms, vulnerability to adversarial examples increases by >3x and closely tracks features per dimension, the reciprocal of feature dimensionality.Source: Relationship to Adversarial Robustness · p. 29
  • They hesitate to claim that superposition significantly contributes to adversarial examples in practice, noting that other explanations exist.Source: Relationship to Adversarial Robustness · p. 30
  • The authors propose that superposition can explain adversarial examples and predict that robust models sacrifice feature capacity, may be more interpretable, and may share transferable attacks when feature arrangements reflect correlations.Source: Relationship to Adversarial Robustness · p. 30
  • Adversarial training with analytic attacks against a random feature reduced superposition, but fully eliminating it required attacks at 80% input L2 norm; stronger attacks were not explored.The authors did not pursue this further because adversarial training increased cost and complexity.Source: Relationship to Adversarial Robustness · p. 30

Superposition in a Privileged Basis

Adding a ReLU hidden layer gives the toy model a privileged basis in which features align with neurons, while increasing feature sparsity shifts solutions from monosemantic to polysemantic neurons. The model has an important limitation: it can circumvent the ReLU and simulate identity behavior.

Superposition in a Privileged Basis

  • Adding a ReLU hidden layer creates a privileged basis where features align with neurons, and many neurons are dedicated to a single feature.This makes the hidden layer directly interpretable in terms of basis-aligned neurons.Source: Superposition in a Privileged Basis · p. 31

VISUALIZING SUPERPOSITION IN TERMS OF NEURONS

  • A per-neuron stacked-weight plot makes polysemantic neurons visible as neurons with weights on more than one feature.Each column represents a neuron, and rectangles encode feature weights; the authors trained 1,000 models per setting and visualized the lowest-loss solution.Source: VISUALIZING SUPERPOSITION IN TERMS OF… · p. 32
  • As feature sparsity increases, solutions shift from monosemantic to polysemantic neurons; both types can coexist, in a transition resembling a neuron-level phase change.The authors note that the phase-change interpretation is not fully formalized.Source: VISUALIZING SUPERPOSITION IN TERMS OF… · p. 34
  • Polysemantic solutions remain structured: features typically map to sets of neurons, while neurons progress from representing a few features to more.The authors say it is unclear how much of this structure generalizes to real models.Source: VISUALIZING SUPERPOSITION IN TERMS OF… · p. 34

LIMITATIONS OF THE RELU HIDDEN LAYER TOY MODEL SIMULATING IDENTITY

  • The toy model can circumvent the ReLU, for example by setting hidden biases positive to make neurons behave linearly, limiting when it shows interesting results.Without biases, enough features can also simulate a bias; the model uses ReLU only when forced.Source: LIMITATIONS OF THE RELU HIDDEN LAYER… · p. 34

Computation in Superposition

This section examines computation in superposition, showing how ReLU networks compute absolute value on superposed features and describing asymmetric superposition and inhibition as a way to manage interference.

Computation in Superposition

  • The section proposes that neural networks can perform computation entirely in superposition and uses this model to study a privileged basis where features align with basis dimensions.Source: Computation in Superposition · p. 34

Experiment Setup

  • The proposed setup treats the input and output as disentangled-model layers and a smaller hidden layer as the observed model, trained to compute absolute value using ReLU.Inputs can take non-positive values, and the two sets of weights are no longer tied.Source: Computation in Superposition · p. 34, Experiment Setup · p. 35

Basic Results

  • Without superposition, the model needs two hidden-layer neurons to implement absolute value on one feature, and the learned model performs the computation as expected.It constructs positive-side and negative-side neurons for each input feature.Source: Basic Results · p. 36

Superposition vs Sparsity

  • Dense inputs yield one feature per neuron, but sufficiently sparse inputs lead the model to represent more features than neurons and approximate absolute value despite mixed-feature activations.The results also show that activation functions can create a privileged basis in which features align with basis dimensions.Source: Superposition vs Sparsity · p. 37
  • In intermediate sparsity regimes, some neurons remain pure while a subset is highly polysemantic, and neurons for the most important features tend to be monosemantic.Many neurons also combine a large-weight primary feature with smaller-weight secondary features, so their strongest activations can appear interpretable while lower-magnitude activations are more polysemantic; the authors say this resemblance to larger networks is suggestive.Source: Superposition vs Sparsity · pp. 37–38

The Asymmetric Superposition Motif

The Strategic Picture of Superposition

The section examines strategies for reducing or solving superposition, including training-time, post-hoc, and hybrid approaches and their costs and difficulty in competitive networks. It frames this work as a route to feature enumeration for safety-oriented interpretability and discusses whether a phase-change regime could allow superposition to be eliminated.

Safety, Interpretability, & "Solving Superposition"

  • Feature enumeration could help rule out certain circuits and cover model behavior, supporting safety claims about harmful behaviors and “unknown unknowns.”The authors describe it as a universal quantifier over fundamental units of neural-network computation.Source: Safety, Interpretability, & "Solving… · p. 40, The Strategic Picture of Superposition · p. 40
  • They define “solving superposition” as enabling feature enumeration—and equivalently, unfolding activations—with methods ranging from changing models to identifying feature directions afterward.With feature directions, compressed sensing could unfold activations into those of a larger non-superposition model.Source: Safety, Interpretability, & "Solving… · p. 40
  • Identifying features would let interpretability decompose activation space into independently understandable components and describe activations in terms of pure features.The authors also argue that understanding weights in general requires connections between understandable features.Source: Safety, Interpretability, & "Solving… · p. 41
  • Superposition can make cosine similarity misleading because unrelated features may have positive dot products; solving it could remove this problem.The proposed routes are neuron-aligned features or compressed sensing to lift features into a space without positive dot products.Source: Safety, Interpretability, & "Solving… · p. 41

Three Ways Out

  • Three routes are creating models without superposition, finding an overcomplete feature basis, or modifying models to help a later analysis find that basis.The authors think each route is possible for their toy models, but difficult for serious networks and modern large language models.Source: Three Ways Out · p. 41
  • Their results suggest superposition and polysemanticity have sharp phase transitions, with a regime in which a model may have neither.The unresolved issue is whether removing or otherwise resolving superposition is too costly.Source: Three Ways Out · p. 41

APPROACH 1: CREATING MODELS WITHOUT SUPERPOSITION

  • In toy models, L1 regularization on hidden activations can remove superposition at higher loss, while superposition may let fixed-neuron models match significantly larger monosemantic ones.The authors expect removal to carry a substantial performance cost when neuron count is fixed.Source: APPROACH 1: CREATING MODELS WITHOUT… · p. 42
  • MoE models can recover neuron sparsity as free flops when sparsity is organized in certain ways, suggesting a possible path to competitive superposition-free models.The authors present MoE as an example of why such models may be possible, not as a recommended path forward.Source: APPROACH 1: CREATING MODELS WITHOUT… · p. 42
  • Under an idealized activation-only compute cost, splitting polysemantic neurons into dedicated feature neurons would cost the same, except where interference already hurts performance.Preliminary comparisons suggest superposition is not optimal for loss reduction per activation frequency, though it asymptotically matches dedicated feature dimensions.Source: APPROACH 1: CREATING MODELS WITHOUT… · p. 42

APPROACH 2: FINDING AN OVERCOMPLETE BASIS

  • Finding an overcomplete basis after training avoids risking model performance, but makes the number of features harder to determine and introduces virtual neurons beyond the surface structure.Those virtual neurons create a further gap between the model’s surface and virtual structures.Source: APPROACH 2: FINDING AN OVERCOMPLETE… · p. 43
  • Post-hoc sparse coding at large scale poses a major engineering challenge, and baked-in superposition lacks the training objective that would push toward less superposition.The paper describes an extremely overcomplete factorization of activations as a different challenge from standard distributed training.Source: APPROACH 2: FINDING AN OVERCOMPLETE… · p. 43

APPROACH 3: HYBRID APPROACHES

  • Hybrid approaches could combine training-time and post-hoc methods, such as reducing superposition to ease decoding or changing architecture to make basis-finding more tractable.The latter could be pursued separately from attempts to reduce superposition.Source: APPROACH 3: HYBRID APPROACHES · p. 43

Additional Considerations

  • A regime exists in which superposition does not occur; pushing models into it may allow superposition to be fully eliminated.The paper argues superposition is useful, but identifies this phase change as grounds for hope.Source: Additional Considerations · p. 44
  • A superposition-free model could provide ground truth for studying superposition in ordinary models, where the features are currently unknown.The authors identify making such models performant, rather than making them at all, as the main research risk; even non-performant models could aid study.Source: Additional Considerations · p. 44
  • Local orthogonal bases may aid understanding of narrow sub-distributions, but are unlikely to support mechanistic safety accounts across the full and off-distribution data.The paper argues that useful safety statements require accounts that hold for the full distribution and off distribution.Source: Additional Considerations · p. 44

Discussion

The toy models reproduce several observed patterns of polysemanticity, but the authors are uncertain how far their other results generalize to real networks. They conclude that superposition raises open questions about its detection, control, and consequences.

To What Extent Does Superposition Exist in Real Models?

  • The third model reproduces polysemantic and non-polysemantic neurons coexisting, while InceptionV1 has a larger fraction of polysemantic neurons in later layers, consistent with the prediction that superposition increases with sparsity.The authors also note that early Transformer MLP neurons can be extremely polysemantic when representing sparse distinctions between interpretations of a token.Source: To What Extent Does Superposition… · p. 45
  • The authors expect some findings—superposition, mixed monosemantic and polysemantic neurons, and perhaps links to adversarial examples—to generalize, but regard the geometry and learning-dynamics results as uncertain.They present consistency with empirical observations of polysemanticity as their best validation of toy models as proxies, not proof that every toy-model phenomenon reflects real networks.Source: To What Extent Does Superposition… · p. 45

Open Questions

  • Open questions include how to detect superposition statistically, prevent it through interventions such as L1 activation regularization, adversarial training, or activation changes, and obtain closed-form models.The paper reports progress toward a closed-form solution for the ReLU output model, while noting that further progress would be useful.Source: Open Questions · p. 46
  • Future work should test how realistic the models are, estimate real models’ feature-importance and sparsity curves, and determine how superposition changes with scale and how many features it can store.The authors also ask for principled definitions of superposition and polysemanticity and tighter bounds on the number of features stored.Source: Open Questions · p. 46
  • They also pose questions about superposition’s relation to non-robust features, computation, correlated features, and nonlinear representations.These questions include whether useful computation can occur in superposition, whether anti-correlated features pack more efficiently, and whether models use nonlinear representations.Source: Open Questions · p. 46

Related Work

The related work situates superposition as a possible explanation for polysemantic neurons and a complication for disentanglement, distinguishing it from distributed coding while relating it to compressed sensing and neural coding. It highlights feature co-occurrence and argues that useful computation can tolerate moderate superposition, so codes need not always be fully sparse.

INTERPRETABLE FEATURES

  • Polysemantic neurons motivate feature-level superposition, distinct from model-level superposition in which parameters represent multiple independent models.The latter line of work is motivated by catastrophic forgetting.Source: INTERPRETABLE FEATURES · p. 47

DISENTANGLEMENT

  • Superposition may occur in generative-model latent spaces, potentially making disentanglement difficult while allowing many sparse, rare features to contribute collectively.Disentanglement often assumes few important latent variables, whereas superposition could make models more effective by representing a large number of individually unimportant features.Source: DISENTANGLEMENT · p. 47

COMPRESSED SENSING

  • Compressed sensing uses general optimization to recover vectors, whereas this toy model uses a neural-network layer and measures sparsity by the probability a dimension is zero.Compressed-sensing algorithms are more powerful in principle; concentration of measure means the toy model’s vectors nevertheless have bounded nonzero entries with high probability.Source: COMPRESSED SENSING · p. 48
  • The learned embedding may ignore input dimensions to ease recovery and favor accurate recovery of important features over recovering less important ones.The model’s feature importances can leave less important features unrecovered altogether.Source: COMPRESSED SENSING · p. 48
  • Compressed-sensing lower bounds act as upper bounds on superposition; here the bound permits feature counts linear in embedding dimension but strongly increasing with sparsity.The authors suggest the sparsity-driven increase may reflect proof techniques rather than an inherent barrier to eliminating superposition.Source: COMPRESSED SENSING · p. 48
  • Compressed sensing and the toy model both exhibit phase changes, though how vector-recovery transitions relate to feature- and neuron-level transitions remains unclear.The authors describe the parallel as suggestive rather than establishing a direct connection.Source: COMPRESSED SENSING · p. 48
  • Although neural networks have been studied as sparse-recovery algorithms, the authors do not expect their toy model to be useful for sparse recovery itself.They see an opportunity to connect understanding of superposition with this and related work.Source: COMPRESSED SENSING · p. 48

THEORIES OF NEURAL CODING AND REPRESENTATION

  • In neural-coding terms, decomposability represents stimuli as compositions of independent features, while superposition represents more features than neurons at the cost of interference.These are distinct senses of distributed representation; decomposability need not assign each feature its own neuron.Source: THEORIES OF NEURAL CODING AND… · p. 49
  • A code can be maximally distributed, with every neuron participating in every input, yet have no more features than dimensions and no superposition.From a linear-readout viewpoint, any decomposable linear code using orthogonal feature vectors is functionally equivalent.Source: THEORIES OF NEURAL CODING AND… · p. 49
  • The authors make the probability that features co-occur a central concept: dense superposition suits items that never co-occur, while jointly needed items can use a code without interference.They contrast representing one item at a time with representing all items at once.Source: THEORIES OF NEURAL CODING AND… · p. 49
  • The absolute-value toy model shows useful computation under moderate superposition, suggesting neural codes may be compressed without being decompressed to fully sparse form.The authors propose this may apply to all neural codes, not only codes used for efficient communication.Source: THEORIES OF NEURAL CODING AND… · p. 49
  • The term “distributed representation” is ambiguous in deep learning, where some works mean independent features are represented independently.The authors distinguish this decomposability meaning from other work that may intend something similar to their usage.Source: THEORIES OF NEURAL CODING AND… · p. 50

Comments & Replications

The comments report independent replications of the paper’s experiments and add analysis of the ReLU toy model’s loss landscape. Replicators also found activation-dependent phase diagrams, while the authors provide notebooks for reproducing some diagrams.

REPLICATION & FORTHCOMING PAPER

  • Redwood researchers replicated plots from the superposition and phase-change sections and found that phase diagrams differ by activation function, suggesting some functions induce more polysemanticity than others.They also replicated the uniform-superposition plot and planned to publish their results separately.Source: REPLICATION & FORTHCOMING PAPER · p. 51

REPLICATION & FURTHER RESULTS

  • An exact expected-loss solution for the basic ReLU output toy model, ignoring biases, makes the loss analytically tractable by representing sparse inputs with delta functions and restricting the integration domain.This permits plotting the full loss surface and solving directly for its minima.Source: REPLICATION & FURTHER RESULTS · p. 51
  • At low sparsity when both features matter, the loss surface has a “confused feature” regime with similar rather than antipodal weights, even without correlated features.The authors note that their solutions can be quantitatively different from the phase-change section’s weights.Source: REPLICATION & FURTHER RESULTS · p. 52
  • As sparsity changes, loss minima move continuously between two configurations, explaining the blurry region around the phase diagram’s triple point.Some loss surfaces have two minima after accounting for symmetry; this pattern may explain discrete energy-level jumps, and near transitions the global minimum can have a smaller attraction basin than local minima.Source: REPLICATION & FURTHER RESULTS · p. 52

REPLICATION

  • OpenAI researchers easily reproduced plots similar to the paper’s basic superposition results, feature dimensionality geometry, and discrete energy-level jumps.The authors said this independent replication increased their confidence that surprising findings were not implementation quirks or bugs.Source: REPLICATION · p. 53

Code

  • The authors provide notebooks for reproducing some core diagrams and separate theoretical phase-change diagrams; other researchers’ replications used clean, independent code based on an early draft.The notebooks are not comprehensive because the experimental code had to be rewritten to run outside the authors’ codebase.Source: Code · p. 54

Citation Information

The paper is Elhage et al.’s 2022 “Toy Models of Superposition,” published in Transformer Circuits Thread. Its footnotes clarify how features are defined and what superposition complicates about interpretation.

Citation Information

  • Elhage et al.’s “Toy Models of Superposition” appeared in Transformer Circuits Thread in 2022.The supplied citation also gives the BibTeX key elhage2022superposition.Source: Citation Information · p. 55

Footnotes

  • Features are defined as patterns that gain a dedicated neuron at a sufficiently large model size, yet arbitrarily large models may still keep many features in superposition.For an individual feature, a non-flat importance curve implies it should eventually receive a dedicated neuron; the definition is less useful for hypothesized or superposed features.Source: Footnotes · p. 56
  • Superposition complicates interpretability: absence claims about behaviors may be insufficient, and knowing a feature direction does not make it easy to remove.The footnotes suggest more nuanced claims about whether features participate in particular circuits and note that superposition obscures interpretable directions without a privileged basis.Source: Footnotes · pp. 56–57
  • A margin may allow reducing superposition without affecting performance, but losses across different superposition levels are not comparable because the task sparsity changes.The footnotes propose L1 regularization as a possible way to reduce superposition and explain that changing sparsity changes the task.Source: Footnotes · pp. 56–57
  • The paper distinguishes its representational challenge from the neuroscience binding problem, which concerns assigning co-occurring features to the correct combinations.The authors explicitly state that their work treats co-occurrence and does not address binding.Source: Footnotes · p. 57

Nonlinear Compression

A simple nonlinear scheme can outperform linear compression on dense inputs, though its costs and practical limits may favor superposition. The toy model’s recovery conditions also imply a compressed-sensing lower bound on projection dimension.

Nonlinear Compression

  • Nonlinear compression may be costly because models must decompress before linear computation, and learning aligned compression and decompression around discontinuities may challenge gradient descent.The authors suggest it may be worthwhile mainly to save residual-stream space across many layers.Source: Nonlinear Compression · p. 60
  • Representing the simple example’s piecewise linear functions with ReLU neurons requires 12 neurons per Z segment; nonlinear compression only starts to beat linear compression at a combined 36 neurons for compression and decompression.Source: Nonlinear Compression · p. 60
  • The authors suspect superposition scales better for sparse features in many dimensions, while allowing nonlinear compression may remain competitive.They describe this scaling judgment as intuition, not a demonstrated result.Source: Nonlinear Compression · p. 60
  • The authors argue nonlinear encodings can be interpreted through their linear feature directions and reverse-engineered as circuits, though pervasive use may call for automated decoding.They see no fundamental interpretability challenge unless the model performs complex computation while remaining in a complicated nonlinear representation.Source: Nonlinear Compression · p. 61
  • On random uniform dense inputs, nonlinear compression outperforms choosing either input dimension or their average, even with values as small as 3.The supplied passage does not specify the epsilon values.Source: Nonlinear Compression · p. 61

Connection between compressed sensing lower bounds and the toy model

Related papers

Loading 2209.10652v1…