PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Model Circuits Are Sparse in the Neuron Basis

Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, Sarah Schwettmann

arXiv:2601.22594v2cs.CLcs.AI

TL;DR

The paper addresses whether language-model circuits must be traced in learned feature bases because individual neurons are thought to be uninterpretable. It develops gradient-based tracing on MLP neurons and finds neuron circuits as sparse and faithful as SAE circuits across studied tasks. The approach supports causal circuit analysis without additional training costs, though circuits remain difficult to interpret and the implementation is not fully efficient.

  • Problem

    Existing interpretability work often assumes neuron representations are not an interpretable basis, motivating learned sparse feature bases for circuit tracing.

  • Method

    The paper traces circuits on pre-down MLP activations using gradient-based attribution and evaluates them through mean-ablation interventions.

  • Results

    Neuron-level MLP activation circuits achieve the same performance as SAE circuits with equivalent sparsity and reproduce prior circuit-tracing findings.

  • Takeaways & Limitations

    The neuron basis is a practical avenue for automated circuit tracing that avoids loading SAEs and additional training costs.

  • Takeaways & Limitations

    Comprehensive circuits still contain too many neurons for easy human interpretation, and serial autograd calls reduce compute utilisation.

Abstract

from arXiv · show

The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986). Language model interpretability research has thus turned to techniques which decompose the neuron basis into more interpretable units of model computation, such as sparse autoencoders (SAEs). However, not all neuron-based representations are uninterpretable. For the first time, we empirically show that MLP neurons are as sparse a feature basis as SAEs. We use this finding to develop an end-to-end gradient-based attribution pipeline for circuit tracing on the MLP neuron basis, which surfaces causally effective neurons on a variety of tasks. On a standard subject-verb agreement benchmark (Marks et al., 2025), a circuit of $\approx 10^2$ MLP neurons is enough to control model behaviour. On the multi-hop city-state-capital task from (Lindsey et al., 2025), we find a circuit in which small sets of neurons encode specific latent reasoning steps (e.g. mapping a city to its state), and can be steered to change the model's output. This work thus advances automated interpretability of language models without imposing additional training costs.

1. Introduction

The paper argues that MLP neuron circuits can be as sparse and faithful as SAE circuits, addressing interpretability concerns about the neuron basis. It uses pre-down MLP activations and RelP attribution to close the gap, then applies the approach to prior circuit-tracing tasks.

  • Motivation: Circuit tracing seeks to localise model behaviours to interactions between subcomponents for faithful oversight and behavioural control.Faithful circuits may expose unverbalised reasoning steps and support steering.
  • Contribution: The paper challenges the belief that neuron circuits are not sparse, showing that neuron-basis circuits can match SAE circuits in sparsity and faithfulness.This directly questions the need to replace neurons with learned feature bases for circuit analysis.
  • Method: Using pre-down MLP activations instead of post-down MLP outputs yields significantly sparser circuits and narrows the gap with learned features.The authors identify MLP activations as a privileged basis for this purpose.
  • Method: RelP replaces Integrated Gradients as a stronger, more efficient attribution method, closing the remaining gap between neuron and SAE circuits.The motivation is that Integrated Gradients can be noisy and expensive on deep models.
  • Motivation: Learned bases introduce approximation errors, feature splitting, absorption, and training-time computational overhead, motivating analysis in the original neuron basis.The paper therefore treats neurons as a potentially more faithful and practical basis.
  • Evaluation: The method replicates a cross-layer-transcoder case study in the neuron basis of Llama 3.1-8B-Instruct.This showcases the approach on a task studied in prior work.

2. Related work

Related work frames circuit tracing as either intervention or attribution over coarse components, or analysis using learned sparse dictionaries such as SAEs and transcoders.

  • Causal interpretability: Causal interpretability studies internal representations by intervening on model internals and measuring effects on outputs.These interventions have been used to study how linguistic and other concepts are encoded.
  • Sparse dictionary learning: Sparse dictionary learning decomposes Transformer representations into sparse feature bases, including SAEs and transcoders.SAEs reconstruct representations with sparse latent activations, while transcoders express MLP computation in a similar sparse space.
  • Circuit tracing: Circuit-tracing approaches commonly operate at coarse granularity or use sparse dictionaries because the neuron basis is viewed as uninterpretable.The paper positions neuron-level tracing as an alternative to both families.

3. Background

The background defines Transformer representations, circuits, SAE features, and the intervention-based metrics used to evaluate sparse computational subgraphs.

  • Transformer bases: A Transformer processes token representations through attention and MLP blocks connected by a residual stream.The residual stream accumulates the input embeddings, attention outputs, and MLP outputs across layers.
  • Transformer bases: MLP activations are pre-down-projection hidden activations, whereas MLP outputs are the resulting model-dimensional block outputs.The distinction is central to comparing neuron representations with learned bases.
  • SAE bases: An SAE decomposes a Transformer representation into sparse feature activations using an encoder, biases, and a nonlinearity.The resulting feature basis can have a dimensionality different from the original representation.
  • Circuits: A circuit is a sparse, task-specific computational subgraph whose nodes are units such as MLP neurons and whose edges represent causal influence.Nodes and connectivity may depend on the input, and the same unit at different token positions is treated separately.
  • Evaluating circuits: Circuit evaluation mean-ablates the complement of a selected circuit, replacing excluded node activations with their dataset means.The underlying model is then run while retaining computation for nodes inside the circuit.
  • Evaluating circuits: Faithfulness measures similarity to the original model after complement ablation, while completeness measures similarity to full-model ablation after circuit ablation.A perfect circuit has faithfulness 1 and completeness 0.

4. Methodology

The methodology compares circuit representations and attribution procedures on paired subject-verb agreement tasks, selecting sparse circuits and evaluating their faithfulness and completeness.

  • Dataset and benchmark: The SVA benchmark contains four templatic tasks evaluated through original and counterfactual input-output pairs.Training uses 300 pairs per task and evaluation uses 40 held-out pairs.
  • Metric: The SVA metric is the logit difference between the model’s preferred singular or plural verb for original and counterfactual inputs.The paired outputs differ in grammatical number while referring to the same verb.
  • Attribution: Circuits are formed by greedily selecting the k highest-attribution nodes using training data, then evaluated on a validation subset.The nodes are individual features such as MLP neurons.
  • Attribution: Integrated Gradients interpolates between counterfactual and original inputs to assign attribution scores to individual nodes.These scores determine which nodes enter circuits at different sizes.
  • Evaluation: Figure 1 plots circuit size against faithfulness and completeness for each representation and basis, averaged over the four paired SVA tasks.The evaluation seeks high faithfulness and low completeness at the smallest circuit size.
  • Representations: The study compares neuron and SAE bases across MLP activations, MLP outputs, attention outputs, and the residual stream.The model is Llama 3.1 8B, with 8x-width Llama Scope SAEs.

5. MLP activations are a sparse basis for circuits

The paper finds that MLP activations form a sparse circuit basis, and that RelP improves attribution enough to match or exceed alternatives across paired, unpaired, and edge-based evaluations.

  • MLP activations yield significantly smaller circuits than MLP outputs, by a factor of 100×, while closing the gap with SAEs.
  • RelP uses a replacement model with locally linearized nonlinearities while retaining the original model’s forward pass for evaluation.
  • RelP outperforms IG in almost all settings and reaches near-perfect faithfulness and completeness with only ∼200 MLP-activation neurons.IG uses 10 backward passes in this setup, whereas RelP uses one.
  • In the unpaired setting, MLP activation neurons require considerably smaller circuits for good faithfulness and completeness than other methods.Training uses only original inputs, while evaluation remains based on the original-versus-counterfactual logit difference.
  • RelP improves both faithfulness and completeness over IG in unpaired evaluation, requiring fewer neurons for good performance.
  • For edge-based circuits, RelP with stop grads exceeds 80% faithfulness while maintaining high completeness with only ≈105 edges, or 10% of candidate edges.RelP Pareto-dominates both alternatives.

6. Case study: Multi-hop reasoning

The paper traces Llama 3.1 8B Instruct’s multi-hop state-capitals reasoning in the MLP neuron basis, identifying neuron groups associated with individual reasoning steps. Steering these groups and one neuron changes model outputs in ways matching their hypothesized roles.

  • Circuit structure: The six neuron groups correspond to steps including representing the state, city, Texas, capital-related wording, and Austin-related output.The listed neurons have descriptions associated with these semantic or lexical roles.
  • Task: The state-capitals task asks questions requiring a city-to-state-to-capital reasoning chain, such as Dallas → Texas → Austin.The dataset contains 50 reformatted multi-hop questions.
  • Circuit construction: Automatic tracing identified 257 high-attribution neurons, from which 23 neurons with meaningful descriptions were manually selected.The selected neurons cluster into six groups corresponding to the task’s reasoning categories.
  • Causal tests: Steering almost every cluster changes the model’s top prediction, with effects matching the hypothesized function of each cluster.Suppressing the Texas cluster preserves capital answers but shifts them to other states; only the state cluster leaves the top prediction unchanged.
  • Causal tests: A single neuron, L23/N8079-, can flip the top output from the capital to the state in a majority of examples when steered.The result supports a consistent causal role for this neuron in the state-to-capital reasoning step.

7. Discussion

The discussion argues that MLP neurons are a practical interpretability basis because they can support sparse, faithful circuit tracing without learned-basis overhead. It also identifies remaining interpretability and computational-efficiency limitations.

  • Neurons as a practical avenue for interpretability: MLP neurons should be included as a baseline in sparse-dictionary circuit-tracing studies because their potential uplift over SAEs remains insufficiently explored.The paper notes that recent circuits work has not systematically used MLP activations as a baseline.
  • Neurons as a practical avenue for interpretability: The authors recommend exhausting the neuron basis before assuming that SAE-imposed feature geometry and frequency assumptions are beneficial.They also note compatibility with methods that avoid reconstruction error.
  • Architectural trends favour MLP sparsity: Gated and mixture-of-experts architectures may favor sparse MLP activations, creating opportunities for interpretability methods that use architectural sparsity.Mixture-of-experts models explicitly route inputs to subsets of expert MLPs.
  • Limitations: The circuits remain too large for easy human interpretation, and principled neuron clustering and improved natural-language descriptions are still needed.The implementation also has low compute utilization because it relies on serial torch.autograd.grad calls.

8. Conclusion

The paper concludes that MLP-neuron circuit tracing can match SAE circuit performance with comparable sparsity while recovering interpretable, causally steerable computations. These results renew attention to MLP activations as a basis for circuit tracing.

  • Conclusion: Neuron-level tracing on MLP activations achieves the same performance as SAE circuits with equivalent sparsity.The conclusion presents this as the paper’s central finding.
  • Conclusion: The neuron-basis circuits reproduce prior CLT findings, produce new user-modelling results, and complement automated neuron descriptions.The authors argue that MLP activations merit renewed attention as a potentially interpretable circuit-tracing basis.

Impact Statement

The paper develops attribution methods for tracing internal computations, including a RelP-based approach designed to preserve completeness efficiently. It frames the work as advancing scalable interpretability while noting both societal benefits and misuse risks.

  • Impact: The approach aims to provide efficient, scalable circuit tracing that can support independent AI-safety auditing and mitigation of harmful model behaviours.The paper also acknowledges that the tool could enable interventions producing harmful outputs.
  • Attribution method: The paper contrasts several internal attribution operationalisations, including conductance and IG-inputs, whose computational costs or approximations differ.Conductance has a cheaper equivalent formulation, while IG-inputs approximates conductance when y's step size is relatively constant.
  • Attribution method: RelP linearises model components by treating selected nonlinear effects or normalisation factors as constant multipliers.This includes attention, RMSNorm, and gated-MLP components.
  • Attribution method: RelP replaces numerical integration for attribution while preserving completeness when the half rule is applied to gated MLPs.The half rule divides gradients by two through multiplicative interactions, preventing doubled attributions.

C. Comparing attribution methods on the MIB benchmark

The MIB benchmark evaluates whether attribution methods can localise subnetworks responsible for task behaviour. RelP performs strongly across tasks, extending its usefulness from fine-grained features to edge-based attribution between larger model modules.

  • Benchmark: MIB tests attribution methods by finding subnetworks responsible for specific behaviours, using node- and edge-based circuit localisation.The benchmark operates on larger components such as attention heads and MLPs, unlike the more granular SVA evaluation.
  • Method: The evaluation adapts MIB’s IG baselines to RelP and uses counterfactual ablations for excluded nodes or edges.RelP replaces gradient computation in two of MIB’s strongest attribution methods.
  • Evaluation: Results are averaged across three runs with different random seeds, while existing-method results come from the original MIB paper.The reported comparison therefore combines new RelP runs with previously reported baselines.
  • Results: RelP achieves the best CMD score on three MIB tasks and the second-best score on the remaining two.CMD scores are lower-is-better measures of how closely a circuit matches full-model task behaviour across circuit sizes.

D. Unpaired SVA results with zero ablation

In the unpaired SVA setting with zero ablation, MLP activations remain the sparsest representation and RelP outperforms IG. Gemma-2 evaluations show this pattern across model sizes and SAE widths.

  • MLP activations are the sparsest representation, while RelP outperforms IG in the unpaired setting with zero ablation.
  • The Gemma-2 evaluation compares MLP activations, MLP outputs, residual stream, and SAE bases across 2B and 9B models.
  • Approximately 100x fewer neurons are needed with MLP activations than MLP outputs at comparable faithfulness and completeness on Gemma-2-2B.
  • MLP activations substantially close the gap with standard 16k-width SAEs while remaining competitive with 65k-width SAEs and producing sparser circuits.

E.2. Gemma-2-9B Results

On Gemma-2-9B, MLP activations continue to yield substantially sparser circuits than MLP outputs while maintaining competitive faithfulness and completeness against SAE representations. Their attribution scores are also more spread across neurons and model depth.

  • The Gemma-2-9B evaluation uses 16k-width SAEs on the same SVA benchmark and reports that the findings scale to the larger model.
  • MLP activations produce substantially sparser circuits than MLP outputs while maintaining competitive faithfulness and completeness against SAE representations on Gemma-2-9B.
  • MLP activations remain competitive with SAE-based approaches while delivering significantly sparser circuits than MLP outputs.
  • Across Gemma-2 models, MLP activations provide similar performance to SAE bases, while SAE width does not significantly affect SAE circuit sparsity.
  • MLP activation attribution scores have greater spread in both the bulk and tails than other model representations.
  • MLP activation scores are distributed more evenly across model depth, unlike MLP outputs, which concentrate in the last two layers.

F.1. nounpp: MLP activation neuron labels are task-relevant

Neuron-level analyses show task-relevant and multilingual features across subject-verb agreement, arithmetic, antonym prediction, and user modelling. Automatic tracing and neuron descriptions identify interpretable groups, though some behaviours remain distributed or difficult to access.

  • F.1. nounpp: MLP activation neuron labels are task-relevant: A 500-node nounpp circuit contains neurons associated with number, tense, and agreement features at task-relevant token positions.
  • Case studies: Neuron-level tracing recovers the same feature types in the MLP basis for addition and multilingual antonym prediction, with causal contributions examined rather than only feature presence.
  • Addition problems: All ten modulo-10 addition outcomes have neurons with AUROC ≥0.9 or AUROC ≤0.1, with some near-perfect AUROCs.
  • Addition problems: Attribution scores for modulo-10 neurons form a diagonal pattern, indicating causal roles only when the sum matches a corresponding residue.
  • Addition problems: For most other moduli, high-AUROC neurons are absent or limited, while tens-digit neurons show noisier attribution patterns and may approximate the overall sum.
  • Multilingual antonym prediction: Antonym prediction lacks a single universal antonym feature but contains multiple neurons encoding language, concept, and attribute information.
  • Multilingual antonym prediction: Language neurons are numerous and distributed throughout the model, whereas concept neurons occur mainly early and in middle layers and attribute neurons arise in middle layers.
  • User modelling: The user-modelling analysis finds relevant neurons for three gender classes, but model inferences are not stored at a fixed token position, making the belief state difficult to access.
Loading 2601.22594v2…