PaperlayerView exact revision on arXiv

Source-linked AI summary

Toy Models of Superposition

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, Christopher Olah

arXiv:2209.10652v1cs.LG

TL;DR

Polysemanticity complicates reverse engineering because neural networks may represent multiple features in individual neurons. This paper uses synthetic sparse-feature toy models to study superposition, finding that monosemantic and polysemantic neurons emerge across a phase change.

  • Problem

    Neural-network representations may contain more features than neurons, creating a central challenge for decomposability in reverse engineering.

  • Method

    The paper constructs synthetic data that models important feature properties, including sparse features, and analyzes toy neural-network representations.

  • Results

    Superposition is observed, both monosemantic and polysemantic neurons can form, and feature storage in superposition is governed by a phase change.

  • Takeaways & Limitations

    These toy models provide a framework for studying how neural networks represent multiple features in superposition.

  • Takeaways & Limitations

    The analysis assumes that relevant features can be modeled as sparse and relies on synthetic data because ground-truth features are unavailable.

Abstract

from arXiv · show

Neural networks often pack many unrelated concepts into a single neuron - a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. This paper provides a toy model where polysemanticity can be fully understood, arising as a result of models storing additional sparse features in "superposition." We demonstrate the existence of a phase change, a surprising connection to the geometry of uniform polytopes, and evidence of a link to adversarial examples. We also discuss potential implications for mechanistic interpretability.

KEY RESULTS FROM OUR TOY MODELS

The toy models demonstrate that superposition is real, supports both monosemantic and polysemantic neurons, and can perform at least some computations. Feature storage in superposition undergoes a phase change and organizes features into geometric structures including digons, triangles, pentagons, and tetrahedrons.

  • Superposition is a real, observed phenomenon in the toy models.
  • Both monosemantic and polysemantic neurons can form.
  • At least some kinds of computation can be performed in superposition.
  • Whether features are stored in superposition is governed by a phase change.
  • Superposition organizes features into geometric structures such as digons, triangles, pentagons, and tetrahedrons.

Demonstrating Superposition

The section tests whether neural networks can represent more sparse, variably important features than neurons by comparing a linear baseline with a ReLU output model. Synthetic data and experiments across sparsity levels are used to isolate when superposition emerges.

  • Setup: The model projects high-dimensional feature vectors into a lower-dimensional space and attempts to recover them, testing whether more features can be represented than neurons.Synthetic inputs assign each feature a sparsity and an importance, reflecting assumptions about natural-world representations.
  • Assumptions: The synthetic data assumes features are sparse, potentially more numerous than neurons, and unequal in importance.These assumptions model rarely occurring features, the imbalance between useful features and neurons, and task-dependent contributions to loss.
  • Models: The linear model serves as a baseline without superposition, whereas the ReLU output model exhibits superposition; they differ only in the final activation.The activation function is therefore treated as critical to whether superposition occurs.
  • Experiment: The first experiment trains ReLU output models at different sparsity levels and visualizes the results, alongside a linear model whose optimized solution is sparsity-independent.This comparison is designed to reveal how sparsity affects the nonlinear model while providing a baseline.

W W T

ReLU-output models develop superposition as feature sparsity increases, representing more features through non-orthogonal geometry, unlike linear models that remain limited to orthogonal representations. The analysis attributes this behavior to a tradeoff between feature benefit and interference, with ReLU favoring negative interference and producing a generalized Thomson-problem geometry.

  • Empirical results: Linear models learn the most important features, while ReLU models exhibit superposition as feature sparsity increases.The ReLU model represents more features by allowing their representations to be non-orthogonal.
  • Empirical results: Superposition initially arranges features in antipodal pairs, then transitions to other geometric structures as more features are represented.The observed geometry changes progressively rather than remaining restricted to pairwise opposites.
  • Analytic explanation: Linear models never benefit from representing more features than they can fit orthogonally because feature benefit is outweighed by interference.Representing more features improves loss, but non-orthogonal feature packing creates interference that worsens loss in the linear case.
  • Analytic explanation: Under uniform feature importance and fixed active-feature count, the interference term becomes a generalized Thomson problem that packs points on a sphere.This connects the ReLU model’s feature geometry to an energy-minimizing point-packing problem.
  • Analytic explanation: ReLU makes negative interference free in the 1-sparse case, explaining why learned solutions prefer negative interference when possible.Negative bias can also convert small positive interferences into effectively negative ones.
  • Analytic explanation: Features often form sparse interference graphs, limiting the number of pairwise interactions and reducing compounding interference.This makes the 1-sparse loss term more important relative to losses from less-sparse inputs.

Superposition as a Phase Change

The section shows that feature representations undergo sharp transitions among being unlearned, stored in superposition, or assigned dedicated dimensions. Controlled experiments and analytic toy models reveal a first-order phase change whose regime depends on sparsity and relative feature importance.

  • Superposition as a Phase Change: Features can be unlearned, represented in superposition, or assigned dedicated dimensions, with sharp transitions between these outcomes.The experiment isolates these regimes by varying an extra feature’s importance and overall feature sparsity.
  • Superposition as a Phase Change: For two features in one dimension, the natural strategies are ignoring either feature or storing both in an antipodal superposition.The superposed solution sacrifices the ability to represent the combination of both features simultaneously.
  • Superposition as a Phase Change: The empirical and theoretical diagrams both show a phase change in which the optimal weight configuration discontinuously changes in magnitude and superposition.The theoretical model analytically confirms a first-order phase change caused by crossover between loss functions and a discontinuity in the derivative of optimal loss.
  • Superposition as a Phase Change: With three features in two dimensions, different strategies include ignoring a feature, pairing the extra feature antipodally, or giving it a dedicated dimension.The diagrams again suggest phase changes between encoding strategies, while later sections address structure not captured by this preliminary view.

The Geometry of Superposition

Uniform superposition organizes feature representations into specific geometric configurations, including antipodal pairs and uniform polytopes. Per-feature dimensionalities reveal multiple distinct superposition phases that correspond to different weight geometries and relate the model to generalized Thomson-problem solutions.

  • Uniform and non-uniform superposition: Feature representations organize into geometric structures such as pentagons and tetrahedrons, with uniform superposition linked to uniform polytopes.Non-uniform superposition can be understood, at least partly, as a deformation of uniform superposition.
  • Uniform superposition geometry: Antipodal pairs create a sticky regime by packing two exactly opposite features into each hidden dimension, and the model preferentially uses them across a broad sparsity range.The graph is sticky at configurations corresponding to two features per hidden dimension.
  • Per-feature dimensionality: Specific feature dimensionalities cluster because the model jumps between distinct weight geometries, showing that superposition comprises multiple phases rather than one configuration.Fractional dimensionality spans the interval between not learning a feature and dedicating a full dimension to it.
  • Uniform polytope configurations: The observed dimensionality lines are ¾ for tetrahedrons, ⅔ for triangles, ½ for antipodal pairs, ⅖ for pentagons, ⅜ for square antiprisms, and 0 for unlearned features.A line at 1 would correspond to dedicating a dimension to a feature, but it is obscured in the dense regime.
  • Connection to the Thomson problem: Several configurations match Thomson-problem solutions, while split lines for non-uniform geometries reflect differing interference and feature dimensionalities.The model can be understood as solving a generalized Thomson problem by embedding represented features as points on a sphere.

W W T · PERTURBING A SINGLE FEATURE

The section establishes an exact correspondence between superposition strategies and polytopes, then shows that perturbing one feature deforms the geometry until a first-order phase change switches configurations.

  • W W T: Superposition strategies correspond exactly to polytopes, so geometric properties determine the relationships among represented features.Every strategy for placing three features in two dimensions corresponds to a triangle, and every triangle defines such a strategy.
  • W W T: For three equally important and sparse features, the optimal two-dimensional arrangement is an equilateral triangle.The geometry makes this optimum unsurprising under equal importance and sparsity.
  • W W T: Projecting basis vectors onto a space orthogonal to an omitted direction produces a regular simplex, representing minimal possible superposition.With equally important and sparse features, the best omitted direction is the fully dense vector.
  • PERTURBING A SINGLE FEATURE: Sufficient sparsity causes a phase change in which the pentagon collapses into two digons with the sparse point at zero.The transition occurs when the loss curves for the two geometries cross.
  • PERTURBING A SINGLE FEATURE: The crossing confirms that the geometric transition is genuinely first order.Different geometries’ loss curves cross rather than changing smoothly between a single family of solutions.
  • PERTURBING A SINGLE FEATURE: These results suggest that some non-uniform superposition is a deformation of uniform configurations with jumps between them, rather than a wholly separate regime.This interpretation leverages the comparatively understandable structure of uniform superposition.
  • PERTURBING A SINGLE FEATURE: Pentagonal solutions lie off the unit circle primarily because positive interference is reduced through a negative bias and compensating weights.Distance from the unit circle is interpreted as being primarily driven by the amount of positive interference.

SETUP FOR EXPLORING CORRELATED AND ANTICORRELATED FEATURES … COLLAPSING OF CORRELATED FEATURES

The paper models correlated features as co-occurring bundles and anticorrelated features as mutually exclusive alternatives, then studies their geometric organization in small visualizable toy models. Correlated features form local almost-orthogonal bases, but as sparsity decreases, representations collapse toward principal components, revealing a tradeoff between superposition and PCA.

  • SETUP FOR EXPLORING CORRELATED AND ANTICORRELATED FEATURES: Correlated feature sets activate together as bundles, whereas anticorrelated sets allow at most one active feature at a time.Correlated sets share the same zero-versus-nonzero choice; anticorrelated sets are entirely zero or contain one uniformly sampled active feature.
  • ORGANIZATION OF CORRELATED AND ANTICORRELATED FEATURES: The experiments train small toy models and visualize their weights as points in 2D, where solutions lie on a unit circle after alignment.Rotations and flips are applied to make solutions comparable.
  • LOCAL ALMOST-ORTHOGONAL BASES: For larger models, correlated feature sets arrange nearly orthogonally, producing local almost-orthogonal bases with little superposition despite global superposition.Within these correlated subsets, the features can be treated as nearly orthogonal and as having very little superposition.
  • LOCAL ALMOST-ORTHOGONAL BASES: This structure suggests a local non-superposition assumption for certain sub-distributions, potentially making PCA usable where global superposition would not justify it.The passage presents this as a possibility if the result extends to real neural networks.
  • COLLAPSING OF CORRELATED FEATURES: When capacity represents only one of two correlated features, the model selects their principal component because it affects loss more than either feature individually.The second principal component is ignored.
  • COLLAPSING OF CORRELATED FEATURES: In a six-feature experiment with three correlated pairs, very sparse features form a hexagon with correlated features side-by-side, then progressively collapse into principal components as sparsity decreases.In very dense regimes, the solution becomes equivalent to PCA.
  • COLLAPSING OF CORRELATED FEATURES: PCA and superposition trade off: stronger correlation favors PCA, greater sparsity favors superposition, and sparse correlated features produce mixtures of both strategies.The paper notes that the tradeoff space remains incompletely understood.

Superposition and Learning Dynamics

The toy models’ learning dynamics are often discrete rather than smooth: features jump between geometric configurations, producing corresponding loss drops. In correlated-feature settings, training proceeds through distinct geometric regimes, while the paper limits itself to several striking phenomena rather than a detailed investigation.

  • Scope and limitations: The paper does not provide the detailed investigation these learning-dynamics questions deserve, restricting discussion to a few striking phenomena and leaving deeper analysis for future work.The authors motivate the section by asking how superposition emerges during training, including whether it appears early or later as models fit more features.
  • Energy level jumps: Large-feature models appear dominated by “energy level jumps,” in which feature dimensionalities jump between values and exchange positions.The authors suggest that smooth-looking loss decreases in larger models may comprise many small configuration changes.
  • Energy level jumps: When dimensionality jumps occur, the loss curve drops suddenly, with a very small decrease at the first jump and a larger decrease at the second.The observed jumps occur as features move between different values and swap places.
  • Geometric learning dynamics: Toy-model solutions often correspond to geometric structures, and some learning trajectories can be understood as sequences of simple, independent geometric transformations.This connection is especially accessible when feature embeddings can be visualized directly as points forming a polyhedron.
  • Geometric learning dynamics: For correlated features, training proceeds through distinct loss-visible regimes, each associated with a distinct geometric transformation.An octahedron with features from different correlated sets arranged in antipodal pairs appears to be a strong attractor, although multiple trajectories can occur.
  • Learning trajectory: The toy network first learns a linear PCA solution before moving to a better nonlinear solution.The authors relate this trajectory to prior findings that simple networks tend to learn linear approximations early in training.

Relationship to Adversarial Robustness

Superposition creates interference that can make models vulnerable to adversarial examples, even in the infinite-data limit. Vulnerability increases sharply as superposition forms, while adversarial training reduces superposition only with unreasonably large attacks.

  • Mechanism: Superposition interference creates an attack surface for the model’s most important feature, potentially even when trained on sparse infinite data.The optimal sparse-data model uses superposition to represent more features, leaving it vulnerable to attack.
  • Empirical relationship: >3x: vulnerability to adversarial examples increases sharply as superposition forms and closely tracks the number of features per dimension.The experiments generated L2 adversarial examples with a maximum attack norm of 0.1 of the average input norm.
  • Predicted implications: Superposition predicts that adversarially robust models may perform worse because robustness requires representing fewer features, while greater robustness may improve interpretability.The paper presents these as predictions and remains hesitant about how much superposition contributes to adversarial examples in practice.
  • Predicted implications: Superposition may also help explain why adversarial examples transfer if correlated or anti-correlated features influence their arrangement in superposition.The paper identifies this as a possible prediction for future investigation rather than an established conclusion.
  • Adversarial training: 80% input L2 norm: adversarial training reduced superposition but required attacks of this size to eliminate it fully.The authors found this result unsatisfying and did not explore stronger attacks because adversarial training increased cost and complexity.

Superposition in a Privileged Basis

Adding a ReLU hidden layer creates a privileged, basis-aligned representation in which many neurons dedicate themselves to individual features. As feature sparsity increases, solutions shift from monosemantic to polysemantic neurons, although the toy model often circumvents the ReLU and has important limitations.

  • Privileged basis: Adding a ReLU hidden layer radically changes interpretability by connecting features to basis-aligned neurons.The model’s privileged basis makes its hidden representation directly interpretable.
  • Privileged basis: Many neurons are dedicated to representing a single feature, supporting neuron-focused interpretability approaches.The plotted features align with neurons in a structured way.
  • Sparsity regimes: As sparsity increases, models shift from monosemantic to polysemantic neurons, with both types sometimes coexisting.Some sparsity regimes produce monosemantic neurons, others polysemantic neurons, and the transition resembles a neuron-level phase change.
  • Sparsity regimes: Polysemantic solutions remain structured and neuron-aligned, with neurons progressing from representing a few features to gradually representing more.Features typically correspond to sets of neurons, while monosemantic neurons are the singleton-set case.
  • Limitations: The toy model’s ReLU provides no benefit beyond constraining information encoding, so the model circumvents it whenever possible.With biases, it shifts neurons into a positive linear regime; without biases, it can simulate one by averaging many features.

Computation in Superposition

The toy model shows that neural networks can compute absolute value while representing more features than hidden neurons through superposition. Intermediate sparsity produces polysemantic and monosemantic neuron patterns, while mechanistic analysis identifies asymmetric superposition and inhibition as a central motif.

  • Computation in Superposition: Activation functions can create a privileged basis in which features align with basis dimensions, while sufficiently sparse inputs lead the model to represent more features than neurons.In dense regimes, each neuron represents a single feature; with sufficient sparsity, gradient descent approximates absolute-value computation even when neurons mix multiple features.
  • Qualitative Behaviors: At intermediate sparsity, many neurons encode pure features while a subset becomes highly polysemantic, with the most important features tending to remain monosemantic.This phase change occurs across neurons rather than feature importance: important-feature neurons tend to be monosemantic, while others encode mixtures.
  • Qualitative Behaviors: Many neurons combine a primary feature with lower-weight secondary features, producing strong activations associated with the primary feature and lower-magnitude activations that are more polysemantic.The toy model reproduces qualitative activation patterns reported in larger neural networks, though the passage characterizes this resemblance as suggestive.
  • Mechanistic Interpretation: Mechanistic analysis finds that some neurons implement two instances of asymmetric superposition and inhibition, which the authors identify as the primary pattern.Most weights implement absolute value conventionally, but a few neurons implement this additional motif.

The Strategic Picture of Superposition

Solving superposition is framed as enabling feature enumeration and activation unfolding, which support broad interpretability and safety analyses. The paper outlines three solution strategies while arguing that superposition may be eliminable in principle but useful for performance and difficult to remove from serious models.

  • Definition and motivation: Solving superposition means either enumerating model features or unfolding activations into a larger non-superposition representation.The paper treats both creating models without superposition and identifying feature directions after training as possible solutions.
  • Interpretability consequences: This capability would enable decomposing activation space, describing activations using pure features, analyzing circuits, and avoiding misleading cosine similarities.Superposition can embed unrelated features with positive dot products, whereas solving it either aligns features with neurons or lifts them into a suitable space.
  • Strategic approaches: The three proposed approaches are creating models without superposition, finding an overcomplete feature basis, and modifying models to make later basis discovery easier.The authors regard all three as possible for the toy models, but increasingly difficult for serious neural networks and modern large language models.
  • Phase transition: The results suggest sharp phase transitions, including regimes in which superposition and polysemanticity disappear entirely.The paper presents this as evidence that superposition may be eliminated by pushing models into a non-superposition regime.
  • Performance tradeoff: Removing superposition may impose a large performance cost because superposition effectively makes fixed-neuron models much bigger.The strategic tradeoff is that superposition can provide substantial performance benefits, even though competitive superposition-free models remain plausible.

Discussion

The toy models reproduce several empirical features of polysemanticity, including mixtures of cleanly interpretable and polysemantic neurons and increased polysemanticity with sparsity. The discussion argues that superposition is supported in these models but leaves major questions about control, realism, scaling, and computation unresolved.

  • Empirical validation: The toy models reproduce empirical patterns of polysemanticity observed in neural networks.The third model forms polysemantic neurons, including alongside non-polysemantic neurons in the same layer.
  • Empirical validation: Polysemanticity is predicted to increase as feature sparsity increases, consistent with its greater prevalence in later InceptionV1 layers and potentially early Transformer MLPs.Higher-level features may be rarer and sparser, while sparse distinctions such as token interpretations can produce substantial polysemanticity.
  • Limitations: The paper concludes that the superposition hypothesis holds in certain toy models, while superposition, polysemanticity, and perhaps adversarial-example relationships may generalize more plausibly than the geometry and learning-dynamics results.The authors explicitly characterize the latter results as much more uncertain.
  • Future work: Future work should test toy-model realism, estimate feature-importance and sparsity curves in real models, determine scaling behavior, and clarify computation and feature packing under superposition.The discussion also asks for statistical tests, closed-form solutions, tighter compressed-sensing bounds, principled definitions, and experiments on nonlinear representations.
  • Future work: Controlling whether superposition occurs is identified as the most important pragmatic question, with L1 activation regularization, adversarial training, and altered activation functions suggested as promising approaches.The discussion frames this as changing the phase diagram so features avoid the superposition regime.

Comments & Replications

External researchers independently replicated major experiments and found activation-dependent phase diagrams, while additional analysis exactly solved a basic ReLU case and revealed confused-feature regimes and complex loss landscapes. Further replications reproduced core plots, and accompanying notebooks support reproduction of key diagrams.

  • Replications: Independent researchers replicated plots from the Demonstrating Superposition, Superposition as a Phase Change, and Uniform Superposition sections.These replications used clean code based on the early draft rather than the authors’ notebooks.
  • Replications: Phase diagrams differed substantially across activation functions, suggesting that some functions induce more polysemanticity than others.Redwood Research reported this finding while replicating the relevant phase-change experiments.
  • Additional analysis: The expected loss of the basic ReLU output toy model can be solved exactly, making the full loss surface and its minima analytically tractable.The derivation represents sparse inputs with delta functions and replaces ReLU with a restricted integration domain.
  • Additional analysis: A confused-feature phase appears at low sparsity when both features are important, with similar rather than antipodal weights and sometimes multiple loss minima.The phase transition between antipodal and confused-feature solutions is continuous, helping explain the phase diagram’s blurry triple-point region.
  • Replications: OpenAI researchers easily reproduced plots covering basic results, feature dimensionality, and discrete energy-level jumps, increasing confidence that the findings were not implementation artifacts.The authors said this independent replication strengthened confidence that surprising results were not caused by an idiosyncratic bug.

Citation Information

The paper is titled “Toy Models of Superposition” and was published in 2022 as a Transformer Circuits Thread article. Its citation lists Elhage and collaborators as authors and provides the paper’s URL.

  • Citation: A citation is provided as Elhage et al., “Toy Models of Superposition,” Transformer Circuits Thread, 2022.This compact citation appears separately from the full BibTeX record.
  • Publication: “Toy Models of Superposition” is a 2022 article in the Transformer Circuits Thread.The citation identifies the work as an article published in 2022 in the Transformer Circuits Thread.
  • Authors: The paper is authored by Nelson Elhage, Tristan Hume, Catherine Olsson, and additional collaborators.The BibTeX entry lists 16 authors, beginning with Elhage, Hume, and Olsson.

Nonlinear Compression

Nonlinearities could compress information more compactly than linear superposition, but nonlinear schemes may be difficult to learn and often require decompression before computation. The paper gives a toy example, argues that linear superposition may remain preferable at scale, and formalizes a compressed-sensing connection.

  • Motivation: Nonlinearities make it theoretically possible to compress information more compactly than a linear superposition.The paper frames nonlinear compression as an alternative to representing information with linear feature directions.
  • Limitations: Nonlinear compression may be difficult to learn, require decompression before computation, and offer insufficient savings relative to superposition.Most computation is linear, so the paper suggests nonlinear compression would mainly save residual-stream space across layers before linear decompression.
  • Efficiency: 36 neurons are required before the piecewise-linear example starts to beat linear compression.Using the stated universal-approximation estimate, each line segment takes two neurons and the example requires 12 neurons per Z segment.
  • Toy construction: A toy scheme compresses two [0,1) dimensions into one [0,1) dimension by quantizing one dimension and approximating discontinuities with steep linear segments.The exact construction uses floor and fmod functions conceptually, then replaces discontinuities because models cannot compute discontinuous functions.
  • Compressed-sensing formalization: The toy model is formalized as a compressed-sensing algorithm, with Theorem 1 requiring W_m = Ω(klog(n/k)) for recovery under the stated approximation condition.The formal result connects the model’s restricted-isometry property to a compressed-sensing lower bound, and Theorem 1 follows from Lemma 1 and Theorem 2.
Loading 2209.10652v1…