PaperlayerView exact revision on arXiv

Source-linked AI summary

Weight-sparse transformers have interpretable circuits

Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, Dan Mossing

arXiv:2511.13653v1cs.LGcs.AI

TL;DR

Mechanistic interpretability needs human-understandable accounts of neural-network computations, which dense models often obscure. The paper trains weight-sparse transformers and prunes task-specific circuits, finding compact, rigorously validated circuits with interpretable concepts and connections. It also studies scaling, explores bridges to existing dense models, and identifies substantial efficiency and scaling limitations.

  • Problem

    Mechanistic interpretability seeks to reverse engineer neural networks, but dense-model activations and weights are often difficult to understand as human concepts.

    Source: §1 · p. 1

  • Method

    The paper trains transformers with most weights set to zero and isolates compact task circuits by pruning nodes at fine-grained weight-matrix resolution.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    The models learn compact, interpretable circuits that are validated as necessary and sufficient for task behavior, while sparse models yield roughly 16-fold smaller circuits than comparable dense models.

    Source: §1 · p. 2, §3.1 · p. 5

  • Takeaways & Limitations

    Weight sparsity provides a route to highly understandable circuits and can be preliminarily adapted through bridges to analyze existing dense models.

    Source: §1 · p. 2

    Sparse models require substantially more compute than dense models, and scaling the method beyond small models and simple tasks may produce large, complicated circuits.

    Source: §5 · p. 9

Abstract

from arXiv · show

Finding human-understandable circuits in language models is a central goal of the field of mechanistic interpretability. We train models to have more understandable circuits by constraining most of their weights to be zeros, so that each neuron only has a few connections. To recover fine-grained circuits underlying each of several hand-crafted tasks, we prune the models to isolate the part responsible for the task. These circuits often contain neurons and residual channels that correspond to natural concepts, with a small number of straightforwardly interpretable connections between them. We study how these models scale and find that making weights sparser trades off capability for interpretability, and scaling model size improves the capability-interpretability frontier. However, scaling sparse models beyond tens of millions of nonzero parameters while preserving interpretability remains a challenge. In addition to training weight-sparse models de novo, we show preliminary results suggesting our method can also be adapted to explain existing dense models. Our work produces circuits that achieve an unprecedented level of human understandability and validates them with considerable rigor.

1. Introduction

Mechanistic interpretability seeks to reverse engineer neural networks, but dense-model activations and weights are often difficult to connect to human-understandable concepts. This work trains weight-sparse transformers and prunes task-specific circuits to obtain simpler, more interpretable mechanisms.

Approach

  • The method prunes models for curated simple behaviors, isolating compact circuits whose nodes are defined at the granularity of weight-matrix rows and columns.Nodes outside the task circuit are ablated to their mean activation over the pretraining distribution.Source: Figure 1 · p. 1
  • Weight-sparse transformers constrain most weights to zero, limiting each neuron’s connections and discouraging distributed concept representations.This simplifies computations by reducing how many residual channels neurons can read from or write to.Source: §1 · pp. 1–2

Results

  • Sparse models produce circuits roughly 16-fold smaller than dense models with comparable pretraining loss, averaged across tasks.The comparison sweeps target loss and measures the minimal circuit size needed to achieve each target.Source: Figure 2 · p. 2
  • The resulting circuits contain neurons and residual channels associated with simple concepts, with connections that are often intuitive and interpretable.This addresses the difficulty that dense-model activations and weights often do not correspond to human-understandable concepts.Source: §1 · pp. 1–2

Validation

  • The circuits are validated as necessary and sufficient: mean-ablating outside the circuit preserves task loss, while deleting circuit nodes severely harms it.This provides a behavioral test that the isolated circuits account for the tasks.Source: §1 · p. 2

Scope and limitations

  • Weight-sparse training is inefficient and may not reach frontier capabilities, motivating preliminary bridges for explaining existing dense models.The paper also releases sparse-model weights, pruned circuits, and visualization code to support replication.Source: §1 · p. 2

2. Methods

The paper trains weight-sparse transformers, prunes task-specific circuits, and uses circuit compactness to assess interpretability. It also introduces bridges for translating between dense and sparse model activations.

2. Methods

  • Weight-sparse models are pretrained on Python code and evaluated on 20 manually constructed binary next-token tasks.The tasks include predicting closing quote type and tracking whether a variable is a set or string.Source: §2 · p. 2, §2.2 · p. 4
  • Sparse circuits contain granular neuron, attention-channel, and residual-channel nodes connected by nonzero weight entries.Circuit size is summarized by the geometric mean number of edges across the curated tasks.Source: §2 · pp. 2–3

2.1. Sparse Training

  • The study illustrates sparse task circuits including quote completion and nesting-depth tracking, where attention channels detect, aggregate, and read out task-relevant information.The nesting circuit uses 7 nodes and 4 edges, while the string-closing circuit uses 12 nodes and 9 edges.Source: Figure 4 · p. 3, Figure 5 · p. 4
  • Weights and biases are sparsified during training by retaining only the largest-magnitude entries, while model width can increase at fixed nonzero parameter count.The sparsest models have approximately 1 in 1000 nonzero weights, with 1 in 4 nonzero activations at node locations.Source: §2.1 · pp. 3–4

2.2. Measuring interpretability

  • For each task, structured pruning learns node masks to obtain the smallest circuit meeting a target loss, with deleted nodes mean-ablated.The masks gate node activations using Heaviside functions and are optimized with surrogate gradients alongside circuit size.Source: §2.2 · p. 4

2.3. Bridges

  • Dense-model explanations use bridges that encode dense activations into sparse activations and decode them back at each sublayer.Bridge training combines normalized reconstruction losses with hybrid-pass KL losses against the original dense model.Source: §2.3 · pp. 4–5

3. Results

Weight sparsity yields substantially smaller, more localizable circuits whose components often encode understandable concepts, while scaling improves the capability–interpretability frontier. Detailed circuit analyses recover mechanisms for several tasks, predict adversarial failures, and preliminarily transfer interpretable perturbations to dense models.

3.1. Weight sparsity improves interpretability

  • 16-fold smaller circuits than comparable dense models show that weight-sparse models make simple behaviors substantially more disentangled and localizable.The comparison averages minimal circuit sizes across tasks at comparable pretraining loss.Source: §3.1 · p. 5
  • Ablating the tiny fraction of nodes in the task circuits severely impairs performance, supporting their necessity as well as their sufficiency.This faithfulness check leaves the rest of the network intact.Source: §3.1 · p. 5
  • Larger models improve the capability–interpretability frontier, while changing L0 trades capability against interpretability and fixed-L0 scaling improves both.Increasing weight sparsity also increases residual-stream activation sparsity.Source: §3.1 · p. 5

3.2.1. CLOSING STRINGS

  • The closing-strings circuit uses two MLP neurons and one attention head to detect quote type and copy the corresponding type to the final token.The four interpreted components have 41 total external edges, nine used by the circuit, and some nodes are not monosemantic across pretraining.Source: §3.2.1 · p. 6

3.2.2. COUNTING NESTING DEPTH

  • The bracket-counting circuit averages open-bracket evidence to encode list depth, then thresholds it to select ] or ]].Its simple averaging predicts failures from unmatched distractor brackets and context dilution; the attack generalizes to similarly capable dense models.Source: Figure 8 · p. 6, §3.2.2 · p. 7

3.2.3. TRACKING THE TYPE OF A VARIABLE

  • The variable-type circuit copies the embedding of set() or "" into a token and later retrieves it to complete the task, using four QK and three V channels.The circuit spans two successive attention heads and has 100 total edges.Source: §3.2.3 · p. 7

3.3. Using bridges to extract circuits from existing models

  • Bridges let sparse-model perturbations edit corresponding dense-model representations, increasing dense-model probabilities for single quotes and for a colon after while True.The preliminary demonstrations use interpretable quote-type and line-prefix representations, with the latter effect weaker than the quote task.Source: §3.3 · pp. 7–8

4. Discussion

The method may scale through interpretable model organisms coupled to dense models, or through sparse bridged models focused on narrow safety-relevant task distributions. Sparse circuits may also provide simpler primitives for automated interpretability, but frontier-scale applications remain constrained by efficiency and circuit complexity.

4. Discussion

  • Unstructured weight-sparse networks are unlikely to approach dense-network efficiency, making full interpretation or de novo training of interpretable frontier models infeasible.The paper identifies this as a fundamental barrier motivating alternative scaling strategies.Source: §4 · p. 8
  • Interpretable model organisms scaled toward GPT-3 capability could reveal circuit motifs shared across sparse and dense models and guide frontier-model investigations.Bridges could support comparisons involving superposition and interference weights.Source: §4 · p. 8
  • Training sparse bridged models on narrow but important tasks such as deception, refusal, or goalseeking could reduce compute and support safety cases.The authors caution that this may not enable ambitious reverse engineering of frontier behavior.Source: §4 · pp. 8–9
  • Sparse circuits may complement automated interpretability by providing simpler primitives and a more compact language for expressing model computations.The paper presents this as a suspected bottleneck and a promising role for sparse circuits.Source: §4 · p. 9

5. Limitations and Future Work

The paper identifies limits in computational efficiency, feature disentanglement, faithfulness, interpretability definitions, pruning, and scaling to larger models and more complex tasks.

Compute (in)efficiency

  • Sparse models require 100–1000x more training and inference compute than dense models of comparable capability.The authors cite dead neurons, inefficient sparsity-mask exploration, and missing sparse-kernel systems support as improvement targets.Source: §5 · p. 9

Polysemantic features

  • Complex-task circuits still contain polysemantic nodes, and increasing sparse-model width toward typical SAE dimensions might reduce this.The authors leave open whether limited superposition is advantageous or merely a consequence of current model width.Source: §5 · p. 9

Non-binary-valued features

  • Some features encode information in their magnitude, so explaining them requires accounting for both activation patterns and values.Such features are not fully binarizable.Source: §5 · p. 9

Defining faithfulness

  • Mean ablation is not a perfect faithfulness measure, and causal scrubbing is needed for full confidence in the circuits.Source: §5 · p. 9

Defining interpretability

  • Compact task-specific circuits do not fully capture intuitive human interpretability, despite qualitative evidence for stronger human-understandability.The authors suggest developing an improved interpretability metric.Source: §5 · p. 9

Other interpretable inductive biases

  • Weight sparsity may not suffice to disentangle superposition, motivating additional inductive biases such as expert sparsity.The paper connects expert sparsity with allowing larger weight L0 without using every parameter on every forward pass.Source: §5 · p. 9

Better pruning

  • The pruning method targets nodes rather than edges and often leaves prunable nodes, requiring additional manual pruning for clean circuits.Source: §5 · p. 9

Scaling beyond small models and simple tasks

  • Scaling beyond small models and simple tasks may produce extremely large circuits or behaviors that resist simple description.The authors suggest automated interpretability may become necessary for complex, capable models.Source: §5 · p. 9

6. Related Work

Related work spans sparse autoencoders, circuit discovery, alternative sparsity constraints, sparse-weight training, and pruning, while the paper targets finer and more directly interpretable circuits.

Circuit discovery and SAEs

  • SAE- and transcoder-based approaches identify interpretable features or attribution graphs, but reported circuits can be large, noisy, or incomplete.The cited limitations include incomplete within-head attention explanations and dense head loadings that remain difficult to interpret.Source: §6 · pp. 9–10

Circuit discovery and pruning

  • Prior circuit methods range from coarse attention-head units to gradient-based or iterative pruning, with some producing circuits too large for direct interpretation.Source: §6 · p. 10

Alternatives to activation sparsity

  • Alternative approaches constrain activation, causal attribution, Jacobians, final-layer weights, or computation to obtain simpler representations.Source: §6 · p. 10

Sparse weight training

  • Sparse-weight training methods differ in drop criteria, weight storage, and whether sparsity is introduced by annealing from a dense model.Source: §6 · p. 10

Pruning

  • Pruning is a broad research area including second-order methods, attribution patching, interpretability-oriented edge pruning, and skill localization.Source: §6 · p. 10

A. Method details

Method details vary across experiments because models were trained with slightly different settings at different times.

A. Method details

  • Numbers across different plots are difficult to compare because experiments used slightly different settings over time.The listed method details are general rules rather than exact specifications for every trained model.Source: A. Method details · p. 14

A.1. Architecture

The models use a GPT-2-like architecture with RMSNorm, sparse activations, and several architectural choices intended to preserve simple or interpretable computations.

A.1. Architecture

  • The architecture is close to GPT-2, typically using 8 layers, d_model = 2048, and context length 256, with RMSNorm and untied embeddings.Attention sinks are used in some models and produce cleaner attention circuits without substantially changing loss.Source: §A.1 · p. 14
  • AbsTopK activations zero all but the k largest-magnitude values, usually retaining k = 1/4 of each location’s dimension.Some activation sparsity helps the capability-interpretability frontier, but excessive sparsity hurts it.Source: §A.1 · p. 14
  • Figure 11 specifies a sharkfin learning-rate schedule with 1% warmup and L0 decay during the first 50% of training.Source: Figure 11 · p. 14
  • Activation kurtosis increases with weight sparsity, providing evidence that activation sparsity can emerge naturally from sparse weights.Source: §A.1 · p. 14
  • A dense bigram table supplies prior log probabilities so sparse parameters need not memorize bigram frequencies.This improves loss and preserves MLP and attention capacity for other information.Source: §A.1 · p. 14
  • Positional embeddings are usually omitted because they are roughly loss-neutral, while earlier learned encodings produced sparser attention-head activations.Source: §A.1 · p. 14
  • Small attention-head dimensions, usually d_head = 16, improve monosemanticity anecdotally at a systems-efficiency cost.Source: §A.1 · p. 14

A.2. Weight-Sparse Model Optimization

The paper combines sparse-model optimization, dense–sparse bridges, bridge-based interventions, and learned node pruning to produce compact, faithful task circuits.

A.2. Weight-Sparse Model Optimization

  • The optimization anneals L0 from dense to sparse, scales learning rates with 1/√L0, clips gradients, and preserves at least j = 4 nonzero values per neuron or attention channel.For the largest, sparsest models, L0 annealing was increased from 50% to 80% of training.Source: §A.2 · pp. 14–15
  • A scheduling bug made embeddings, unembeddings, and biases become sparse rapidly; fixing it slightly hurt both capability and interpretability, so the authors retained the buggy schedule.The result is reported as an ablation in Figure 21.Source: §A.2 · p. 15
  • Learning-rate warmup is critical for stability at higher learning rates and substantially improves optimal loss, while longer warmup is slightly beneficial.Warmup typically occupies the first 1% of training.Source: §A.2 · p. 15

A.3. Bridges loss terms

  • Bridge losses combine normalized MSE with bidirectional KL terms so sparse and dense models can translate activations and match hybrid-forward-pass behavior.The KL terms approximate considering sparse–dense transitions at multiple layer combinations.Source: §A.3 · p. 16

A.4. Bridge intervention

  • Bridge interventions convert sparse single-channel activation differences into dense-model interventions using scaled bridge rows and a steering strength from 0.0 to 1.0.Bridges operate on residual-stream locations and compensate for scale differences before intervention.Source: §A.4 · p. 16

A.5. Pruning algorithm

  • Pruning learns uniform boolean node masks across prompts and tokens by optimizing task cross-entropy together with mask sparsity.Nodes include post-RMSNorm, block-output, MLP-activation, and attention q/k/v locations; the learned masks outperform attribution-based selection in circuit compactness.Source: §A.5 · p. 20
  • Discretized pruning circuits require post-hoc logit scale-and-shift calibration, whose general principled status remains unclear.The pruning procedure uses CARBS to retune many hyperparameters separately for each model–task combination.Source: §A.5 · p. 20

A.6. Dataset

The dataset mixes simple repetitive Python with broader-distribution Python, while task tokenization uses tokens chosen to represent quote and bracket conditions reliably.

A.6. Dataset

  • The training data contains 10 billion tokens of simple repetitive Python and 25 billion tokens closer to the full Python-code distribution.The data was generated from a mix of GPT-4-base and GPT-4o-base.Source: §A.6 · p. 20
  • The simpler data component was intended to expose interesting circuits at smaller scale without making the dataset entirely toy-like.This design choice was not ablated, and some smaller experiments used only the simpler component.Source: §A.6 · pp. 20–21

A.7. Task tokenization considerations

  • Quote tasks use dedicated opening and closing quote tokens, while bracket counting accounts for tokenization such as ] being a single token.A 2048-token BPE tokenizer was trained on the dataset.Source: §A.7 · p. 21, §A.6 · p. 21

B. Systems considerations for scaling

The paper examines why weight-sparse training is difficult to scale efficiently, then studies feature binarization and an alternative smooth L0 method. Sparse kernels face fundamental hardware and optimization barriers, while binarizability and task loss depend differently on model size and sparsity.

B. Systems considerations for scaling

  • Unstructured sparsity defeats dense GPU dataflows, wastes memory traffic, and prevents weight-sparse models from matching dense models’ kernel efficiency.Irregular masks disrupt tiled transfers and tensor-core execution, while sparse hardware requires extra routing space and cannot achieve dense compute density.Source: B. Systems considerations for scaling · p. 21
  • Although sparse kernels could provide large theoretical speedups, the current optimization stack is unsuited to training with them, and additional systems research is needed to scale models drastically.Source: B. Systems considerations for scaling · p. 22

C. Binarizing feature magnitudes

  • Binarizability generally improves with more total parameters at fixed L0, whereas increasing L0 at fixed total parameters improves binarizability despite worsening circuit-size interpretability.The result is noisy, and baseline task losses are equal because pruning targets a fixed task loss.Source: C. Binarizing feature magnitudes · p. 23, Figure 34 · p. 24
  • Some understandable features are not binarizable because they represent multiple semantic values or vary continuously with context.Examples include list-nesting depth and attention key channels that increasingly emphasize recent tokens.Source: C. Binarizing feature magnitudes · p. 23

D. Smooth Approximations to L0 Norm

E. Validating circuit hypothesis with mean ablations

The paper validates sparse circuits primarily with mean ablations, which produce more complete task circuits than activation patching. This validation remains incomplete because mean ablation is weaker than full causal scrubbing.

E. Validating circuit hypothesis with mean ablations

  • Mean ablation is not a perfect faithfulness measure, so full confidence ultimately requires causal scrubbing.The authors describe their circuits as substantially improved but still not maximally faithful.Source: E. Validating circuit hypothesis with… · p. 25
  • The method only inconsistently satisfies causal scrubbing’s semantic interchangeability condition, and replacing irrelevant nodes with their mean is weaker than sampling from the pretraining distribution.The paper reports signs of progress but poor performance on the first condition and a strictly weaker test for the second.Source: E. Validating circuit hypothesis with… · p. 25
  • Mean ablations identify more complete circuits than activation patching because they retain components whose activations vary across prompts rather than within prompt pairs.For example, activation patching misses circuitry that copies variable names when names differ between prompt pairs but remain fixed within each pair.Source: E. Validating circuit hypothesis with… · p. 25

F.1. Constant queries

  • Setting the constant query channel to a fixed value increases loss by 2.6e-5 nats per token, versus 1.47e-4 when the query is zeroed.This supports treating the query as constant for the single-double-quote circuit.Source: §F.1 · p. 25

F.2. Rescaling Ablations and Redundancy

Rescaling interventions simplify redundant circuit components, but can conceal superposition if applied indiscriminately. The section also documents qualitative interpretability examples and the project’s contributor roles.

F.2. Rescaling Ablations and Redundancy

  • Rescaling can hide superposition, so the authors restrict it to cases where computation is genuinely redundant.For another residual channel, rescaling also recovers most of the loss after ablating the final MLP.Source: §F.2 · pp. 25–26
  • 7e-4 nats per token versus 4e-3 for zero ablation shows that linearly replacing the redundant layer-3 activation largely preserves performance.The authors therefore ablate layer 3 and rescale layer 2 correspondingly in the bracket-counting circuit.Source: §F.2 · p. 26, Figure 36 · p. 26
  • Layer 3 attention implements highly interpretable computation overlapping with layer 2 and is ablated mainly to simplify the circuit description.The overlap includes both a similar circuit in one head and copying the layer-2 residual change in another.Source: §F.2 · p. 26
  • The scaling frontier improves with more total parameters, while increasing activation sparsity helps initially but eventually becomes Pareto dominated.The figure varies L0, total parameters, and activation sparsity while holding the other two quantities fixed.Source: Figure 37 · p. 27
  • Random residual-stream features are somewhat interpretable, but selected examples can be unusually monosemantic and should not be treated as typical.One highlighted node separates double-quoted and single-quoted contexts, while the caption explicitly notes it was cherry-picked.Source: Figure 38 · p. 28, Figure 39 · p. 29

Related papers

Loading 2511.13653v1…