PaperlayerView exact revision on arXiv

Source-linked AI summary

The Emergence and Sparsification of a Syntactic Circuit

Qing Yao, Sasha Boguraev, Tiago Pimentel, Kyle Mahowald

arXiv:2610.33299v1cs.CL

TL;DR

Subject–verb agreement is a useful probe of grammatical knowledge, but how its mechanisms change during training is not revealed by behavior alone. The paper tracks agreement circuits across Pythia training and finds that they reorganize and generally sparsify after behavior is learned. In pythia-1b, a sparse circuit captures the primary agreement channel, though the analysis is limited to English and relies on an iterative, manually analyzed method.

  • Problem

    Subject–verb agreement probes grammatical competence and internal representations, and the paper asks whether the mechanism behind stable agreement behavior continues to evolve during training.

    Source: §1 · p. 1, §4.1 · p. 5

  • Method

    The authors track agreement-related neurons and attention heads across checkpoints in ten pythia-410m seeds and one pythia-1b model, using attribution and causal interventions.

    Source: §1 · p. 1, §3.5 · p. 4, §4.2 · p. 5

  • Results

    Agreement circuits were diffuse early and generally became sparser; in pythia-1b, ten subject-position neurons, one head, and eight downstream neurons form a scoped pathway with a one-dimensional causal channel.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that mechanistic reorganization and sparsification can continue beneath stable agreement behavior, with pythia-1b culminating in a sparse, generalizable circuit.

    Source: §1 · p. 2

    The analysis focuses on English subject–verb agreement, and Experiment 2’s circuit-identification method is iterative and manually analyzed, limiting its general procedure across tasks and settings.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.

1 Introduction

The paper traces how subject–verb agreement circuits emerge during language-model training, from early diffuse mechanisms to later sparse, reorganized circuits. It also relates this trajectory to accounts of grokking, where a sparser solution follows saturated accuracy.

1 Introduction

  • A minimal agreement circuit must recognize and store subject number, transport it to the prediction position, and license the corresponding output form.The task requires tracking grammatical subjecthood across potentially unbounded distances.Source: §1 · p. 1
  • Agreement is learned quickly, but its circuit continues to sparsify and reroute while pythia-1b’s performance remains mostly unchanged.The final pythia-1b circuit captures most of the agreement signal.Source: §1 · p. 1
  • The circuit’s trajectory parallels accounts of grokking, in which a sparser, more general solution replaces an earlier one after training accuracy saturates.The authors present this as a parallel, not an explanation of why the transition occurs.Source: §1 · pp. 1–2
  • In pythia-1b, the circuit shifts from two early attention heads to a different head later, and a sparse circuit of 18 neurons and one head captures the primary agreement channel.The sparse circuit generalizes across agreement forms and to WikiText examples.Source: §1 · p. 2

2 Finding agreement in LMs

Prior work has identified agreement-related components and circuits in language models, while longitudinal analyses have examined circuit change across training. This paper studies agreement at finer granularity, across seeds and at the neuron level, and characterizes sparsification and rerouting.

2 Finding agreement in LMs

  • Earlier work identified agreement components in LSTMs and syntax-aware or number-transport heads in transformer models; later studies examined sparse circuits.The cited examples include sparse-autoencoder circuits and sparse circuits over MLP activations.Source: §2 · p. 2
  • Unlike Tigges et al. (2024), who study multiple tasks and localize circuits at coarse component levels, this work focuses on one task, many seeds, and neurons.It tracks which neurons causally control agreement and how they reroute between number-transport heads.Source: §2 · p. 2
  • The paper combines head- and neuron-level interventions to track agreement circuits across checkpoints and random seeds, characterizing sparsification and rerouting as the dominant transport head changes.It also shows that most pythia-1b agreement behavior can be captured by a sparse circuit.Source: §2 · p. 2

3 Methods and materials

The study analyzes Pythia models on controlled subject–verb agreement minimal pairs and evaluates circuits through behavioral comparisons and activation interventions. It localizes heads and neurons, then tests causal relevance and routing.

3.1 Models

  • The study covers pythia-1b and ten pythia-410m seeds, spanning model sizes and random seeds.pythia-1b has 16 layers and 8 heads per layer; the pythia-410m models have 24 layers and 16 heads per layer.Source: §3.1 · pp. 2–3

3.2 Subject–verb agreement task

  • The main task uses minimal pairs with an intervening prepositional phrase, differing only in subject number.The evaluation also includes subject-relative-clause, object-relative-clause, and no-distractor templates.Source: §3.2 · p. 3

3.3 Behavioral validation

  • Behavioral accuracy requires the model to assign higher probability to the correct continuation for both sentences in each minimal pair.For singular subjects, the singular verb must outrank the plural alternative, with the reverse for plural subjects.Source: §3.3 · p. 3

3.4 Circuit evaluation

  • Circuit evaluation measures necessity and sufficiency through changes in the agreement gap under activation interventions.Necessity uses mean ablation; sufficiency swaps component activations across minimal pairs to shift verb predictions toward the source sentence’s predictions.Source: §3.4 · pp. 3–4

3.5 Circuit discovery methods

  • Heads are identified by necessity and sufficiency at the prediction position, while RelP ranks neurons and interventions validate causal relevance and routing.RelP is treated as a hypothesis generator; path-patching-like interventions attribute neuron effects to routes through heads.Source: §3.5 · p. 4

4 Experiment 1: Behavior–mechanism decoupling in training

Across pythia-1b and ten pythia-410m seeds, the study tracks agreement behavior and its circuit through training. Agreement stabilizes early, while the neurons and attention heads involved continue to reorganize.

4 Experiment 1: Behavior–mechanism decoupling in training

  • The experiment tracks agreement behavior, subject-position neurons, and necessary and sufficient prediction-position heads in pythia-1b and ten pythia-410m seeds.The goal is to describe behavior and identify agreement-relevant components across checkpoints, not yet to produce complete circuits.Source: §4 · pp. 4–5

4.1 Agreement behavior is learned within 2% of total training steps

  • By step 2k, all ten pythia-410m seeds reached a median agreement accuracy of 96.9%; their final median accuracy was 91.7%.The pythia-1b model likewise changed from 99.0% accuracy at step 2k to 94.8% at its final checkpoint.Source: §4.1 · p. 5

4.2 Circuit neurons sparsify through training

  • The agreement circuit tends to lose subject-position neurons during early training even as performance remains stable, then levels off around halfway through training.The circuit size is measured by the number of neurons needed for at least 50% necessity and sufficiency under intervention.Source: §4.2 · p. 5, Figure 2 · p. 5
  • Across pythia-1b checkpoints, all 21 neurons in the step-18k k50 set also appear in the step-8k set, and 18 remain in the final set.Across pythia-410m seeds, the intersection of post-8k k50 sets has a median 85% overlap with each seed’s smallest k50 set.Source: §4.2 · pp. 5–6

4.3 Number is carried by a small set of heads

  • In pythia-1b, early agreement circuits rely mainly on L15 H7 and L15 H2, but L13 H7 becomes the leading head by step 18k and stays so through the final checkpoint.L15 H2 persists as a weaker secondary channel; other heads contribute negligibly.Source: §4.3 · p. 6, Figure 3 · p. 6
  • Across the examined models, a small set of heads carries agreement, but none retain the same top three heads from steps 2k to 143k.A leading head often individually achieves over 50% in necessity and sufficiency.Source: §4.3 · p. 6

5 Experiment 2: The sparsification of the pythia-1b circuit

In pythia-1b, agreement-relevant neurons follow the shift from L15 H7 to L13 H7, and a manually constructed sparse circuit captures most agreement information flow within its scope. The circuit generalizes across tested templates and naturalistic text, while its head transport can be localized to one dimension.

5.1 Joint reorganization

  • As L13 H7 becomes dominant, upstream and downstream neurons shift from routing through L15 H7 to aligning primarily with L13 H7.The authors link this neuron rerouting to the heads’ changing relative importance across checkpoints.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 A final sparse circuit

  • The 18-neuron, one-head circuit is 78% necessary and 83% sufficient, close to the full scope’s 88% sufficiency despite using around 1/6000 as many MLP neurons.The evaluation used 192 PP items; the circuit’s complement had little effect on agreement.Source: Table 1 · p. 8, §5.2 · p. 8
  • The final circuit uses one relevant head and selected neurons to represent MLP processing, but its iterative, judgment-based construction is not guaranteed maximally performant or sparse.It contains ten subject-position neurons upstream of L13 H7 and eight prediction-position neurons downstream.Source: §5.2 · pp. 7–8

5.3 Characterizing the final circuit

  • The circuit’s necessity and sufficiency are broadly comparable across novel synthetic templates and WikiText, though performance is slightly worse on SIMPLE.Additional tests report cross-template control and robustness to most tested perturbations, with failures for appended-adverb templates and SIMPLE as a weak donor.Source: §5.3 · pp. 8–9, Table 2 · p. 9
  • A one-dimensional subspace of L13 H7’s output-value circuits captures essentially the head’s total causal effect on agreement.Intervening on or ablating the subspace complement barely affects agreement.Source: §5.3 · p. 9, Table 3 · p. 9

6 Conclusion

The paper characterizes a scoped L13 H7-mediated agreement pathway in pythia-1b and finds that circuits across studied runs generally become sparser over training. It leaves the cause of this transition unresolved, offering a hypothesis that an early diffuse circuit scaffolds a later, more specialized one.

6 Conclusion

  • Within the scoped pythia-1b pathway, ten subject-position neurons propagate number information through a one-dimensional L13 H7 channel to eight downstream neurons controlling verb readout.Across pythia-1b and ten pythia-410m seeds, agreement circuits were diffuse early and generally became sparser over training.Source: §6 · p. 9
  • The authors hypothesize that an early diffuse circuit of useful features may scaffold a later, sparser and more specialized circuit, while leaving the transition’s cause for future work.They frame this longitudinal study of one syntactic phenomenon as complementary to broader investigations of language-model syntactic abilities.Source: §6 · p. 9

Limitations

The paper focuses on subject–verb agreement in English, reports no explanation for the observed circuit sparsification and head rerouting, and notes that its Experiment 2 circuit-identification method was iterative and manual. Its appendices also report experiment compute costs and AI-assistant use.

Limitations

  • The study covers one linguistic task, subject–verb agreement, and English only; extending the work to other constructions, models, and languages remains future work.The authors say a wider investigation would have undermined this project’s particular motivation.Source: Limitations · p. 9
  • Sparsification and head rerouting were common across training runs, but the paper does not explain why they occur.Source: Limitations · p. 9
  • Experiment 2 identified its circuit through iterative manual analyses, so a general procedure for other tasks and settings remains to be developed.The authors say this approach was possible because the work was highly focused.Source: Limitations · p. 10

A Computation cost

  • Experiments used one A40 GPU; a pythia-1b RelP pass took around 40 seconds, while integrated-gradient variants took about two minutes per checkpoint-position.DAS training took around two minutes per layer; a necessity or sufficiency evaluation took about 1–2 seconds.Source: A Computation cost · p. 13

B Use of AI assistants

C Task templates and data

This section describes synthetic and WikiText subject–verb agreement data built around minimal subject-number pairs, including structural and lexical variations used to test generalization beyond the primary template. The WikiText cases pair singular and plural subject phrases with “is” and “are,” respectively.

C Task templates and data

  • Base and source sentences use singular/plural forms of the same subject lemma, differing only in number; the 16 subject pairs and agreement verb labels are single tokens.The templates evaluate predictions on is/was/has for singular and are/were/have for plural subjects.Source: C Task templates and data · p. 13

C.1 Synthetic templates

  • PP is the primary template for finding and evaluating the circuit, while OBJECTRC, SUBJECTRC, and SIMPLE test generalization across structures.PP-based templates use distractor noun phrases; SIMPLE has no distractor, and the prediction position immediately follows the subject.Source: §C.1 · p. 13
  • Structural perturbations add adjectives, an adverb before the predicted verb, prefixes before the subject, or longer distractor phrases.The adverb changes the lexical category immediately before prediction; prefixes shift absolute token positions while preserving dependency structure, and longer distractors increase subject–prediction distance.Source: §C.1 · pp. 13–14
  • Prefix conditions use neutral discourse markers or locative phrases, avoiding tensed prefixes that could independently alter verb-form distributions.The locative-prefix list includes In the park, At the office, At the hospital, At the theater, In the garden, and On the street.Source: §C.1 · p. 13

C.2 Templates for lexical generalizations

  • Lexical-generalization tests keep the PP dependency structure fixed while varying subjects or verb labels, including 13 held-out subject lemmas and 36 verb pairs.Base/source examples remain minimal pairs differing in subject number.Source: §C.2 · p. 14

C.3 WikiText agreement cases

  • The WikiText set contains 80 agreement cases balanced between 40 plural and 40 singular subjects, each paired with a manually constructed opposite-number counterfactual.Subjects and verb continuations tokenize as single tokens; attested past-tense verbs are converted to present tense before scoring singular/plural forms.Source: §C.3 · p. 14
  • Mowing methods, weather’s influence on chick survival, and induction therapy are presented as singular and plural subject pairs.The singular prompts use “is,” while their plural counterparts use “are.”Source: §C.3 · p. 17
  • The cases also pair singular and plural forms for Hispaniola, puzzle consequences, and tolling-point relationships.Each prompt contrasts a singular subject with a plural one and changes the verb from “is” to “are.”Source: §C.3 · p. 17
  • Other paired prompts cover a plan’s purpose, water amounts in a toilet or sink, and poem forms.Their singular and plural variants use “is” and “are,” respectively.Source: §C.3 · p. 17
  • Biology-related examples contrast protein-synthesis rates, in-vivo protein studies, and bioinformatics fields in singular and plural forms.Each pair shifts from “is” with the singular subject to “are” with the plural subject.Source: §C.3 · p. 17
  • The prompts include singular/plural pairs for a Commonwealth force’s task and the definition of leg before wicket.The verb changes from “is” to “are” across each pair.Source: §C.3 · p. 17
  • Further pairs cover an object seated beside a box and a song described as a cheery little tune about death and destruction.Both examples contrast singular subjects with plural counterparts and “is” with “are.”Source: §C.3 · pp. 17–18

D.1 RelP neuron attribution

Agreement-related neuron sets generally shrink over training, while attribution checks and routing analyses track which neurons contribute and which attention heads carry their effects. The section also tests whether the sparsification findings depend on scoring choices.

D.1 RelP neuron attribution

  • Removing the stop gradient changes the broader neuron ranking, but all 18 final-circuit neurons remain within the top 35 and the sparsification conclusion is unchanged.The two rankings overlap on 42 of the top 50 neurons, falling to 356 of the top 1024.Source: §D.1 · p. 18
  • The final pythia-1b circuit’s 18 neurons all rank within the top 35 under both pooled RelP rankings and integrated gradients, which agree on 44 of the top 50 neurons.The integrated-gradient check is used to test whether high-ranked candidates are biased by local linearizations.Source: §D.1 · p. 19

D.2 Neuron–head routing attribution

  • Routed sufficiency measures how much a neuron’s effect disappears when a downstream head is held at its base value, estimating the neuron’s effect routed through that head.For downstream neurons, the analysis instead measures how much a head-intervention effect is mediated through the neuron.Source: §D.2 · p. 19

E.1 Different levels of k

  • Neuron sets generally decrease over training, although pythia-1b’s k70 set rebounds at step 143k, likely because its final circuit uses downstream prediction-position readout neurons.The authors caution that sparsification may not always appear in these figures because a complete circuit can depend on model- and checkpoint-specific mechanisms beyond the subject position.Source: §E.1 · p. 19

E.2 Overlap of neuron sets across training

Across training, neuron sets tend to retain many neurons from earlier checkpoints, while outlier runs and transport heads can reorganize. The appendix also describes how the final circuit’s neuron candidates were selected and pruned.

E.2 Overlap of neuron sets across training

  • Later k50 neuron sets are generally largely contained in the previous checkpoint’s sets, and when set sizes expand, neurons in the smaller set are largely retained.The table reports each k50 set’s size and its overlap with the previous checkpoint, normalized by the smaller of the two sets.Source: §E.2 · pp. 19–20, Table 7 · p. 20

E.3 Late-training outlier trajectories

  • In outlier seeds, k50 rebounds coincide with training setbacks: seed 3 later re-sparsifies after a mechanism setback, while seed 4 relearns agreement but ends without re-sparsifying.The authors interpret the trajectories as consistent with a general pressure to sparsify after agreement is learned, including in outlier runs.Source: §E.3 · p. 20, Figure 7 · p. 21

E.4 Attention-head localization across checkpoints and seeds

  • The dominant transport-head channel changes during training: the top-three set changes in every pythia-410m run, and the leading head changes in pythia-1b and six seeds.Across ten seeds, the end-of-training dominant channel is in layers 16–23; the average head depth is 83% of total layers.Source: §E.4 · pp. 20–21, Table 8 · p. 21, Table 9 · p. 22

F.1 Initial circuit discovery details

  • The initial circuit uses neurons surfaced by output alignment or RelP, then prunes them by repeatedly removing the neuron whose deletion least affects circuit necessity and sufficiency.The described pruning starts from a 21-neuron circuit and removes three neurons using the PP training set.Source: §F.1 · pp. 21–22

F.2 Circuit scope

The pythia-1b circuit is presented as a sparse subject-to-prediction path carrying number information through L13 H7. The authors justify its scope by testing the roles of positions, MLPs, and self-attention.

Circuit scope

  • L13 H7 becomes the dominant late transport channel, reading from both the subject and preposition positions, while the early L15 heads are almost entirely subject-to-prediction channels.The determiner positions are not accessed by these heads.Source: §F.2 · p. 22, Table 10 · p. 23
  • Determiner-position neuron attributions are negligible, supporting their exclusion from the final circuit.The maximum absolute RelP scores at the sentence-initial and distractor determiner positions remain small throughout.Source: §F.2 · p. 22, Table 11 · p. 23
  • Subject-position L0 mean ablation raises cross-entropy by about 0.82 nats on both PP templates and hand-curated WikiText prefixes, whereas L5 ablation leaves it essentially unchanged.Accordingly, the presented circuit path excludes subject L0; the authors suggest L0 may construct the number feature while the circuit transports it to prediction.Source: §F.2 · pp. 22, 24, Table 12 · p. 24
  • Ignoring attention heads at the subject position changes the PP agreement gap only from +3.48 to +3.43, a 1.5% drop; no prediction-position self-attention head under ablation drops it by more than 2.5%.Heads L14 and later have near-zero mean-ablation effects.Source: §F.2 · p. 24

F.3 Circuit components and functional roles

The circuit’s components are described as plural detectors, a number-transport channel, and prediction-side readouts that promote or suppress verb forms. Off-circuit interventions find little effect on agreement within the declared scope.

F.3 Circuit components and functional roles

  • Subject-side CA neurons act primarily as plural detectors, activating strongly on plural nouns but near zero on singular nouns and non-nouns.Most selected CA neurons rank among the top one or two plural-selective neurons in their layer.Source: §F.3 · p. 24, Table 14 · p. 24
  • L13 H7 is characterized as the circuit’s transport channel, moving number information from the subject position to the prediction position through a one-dimensional subspace.The identification combines the final-checkpoint head result with the reported localization of its number channel.Source: §F.3 · p. 24
  • Prediction-side readout neurons are gated by subject number and write toward or away from the corresponding verb-number direction.L14 n1160 promotes plural forms and suppresses singular forms, while L14 n4146 has the opposite output alignment; anti-agreement neurons counteract the agreement gap.Source: §F.3 · pp. 24–25, Table 15 · p. 25

F.4 Off-circuit interventions

  • Replacing relevant non-circuit activations with donor activations leaves agreement intact regardless of donor class, indicating little effect from the circuit complement under this intervention.The intervention tests the declared-scope MLP blocks while leaving circuit neurons untouched.Source: Table 16 · p. 25, §F.4 · p. 25

F.5 Generalization datasets and additional results.

The final circuit’s agreement behavior transfers across templates and novel lexical items, and remains sufficient under most PP-template perturbations. The analysis also isolates a one-dimensional direction in L13 H7’s value-output channel for further ablation evaluation.

F.5 Generalization datasets and additional results.

  • Cross-template interventions achieve sufficiency comparable to within-template interventions, though SIMPLE transfers effectively only as a recipient, not a donor.The circuit’s generalization tests also include novel templates and in-the-wild text.Source: §F.5 · pp. 25–26
  • Across six PP-template perturbations, circuit sufficiency is 81%–88% versus 83% on PP; adding an adverb after the object is the exception.The authors suggest readout components may be specific to predictions after NPs.Source: §F.5 · p. 26, Table 18 · p. 26
  • The circuit generalizes to novel subjects and verbs, extending beyond the original 16 subject-object pairs and six verb forms.The novel-word evaluation uses 13 new subject pairs and 36 verb pairs.Source: §F.5 · p. 26
  • On held-out verbs, the intervention brings the agreement gap 85% of the way to the natural target, despite a lower baseline gap.The authors attribute the lower baseline to weaker model preferences for less common verb completions.Source: §F.5 · p. 26, Table 19 · p. 26

F.6 DAS analysis of the transport channel

  • For the L13 H7 transport channel, the analysis compares mean ablation of the full 256-dimensional slice, the one-dimensional DAS direction, and its orthogonal complement.Training uses a rotation alignment map with a 1D intervention at the prediction position.Source: §F.6 · p. 26

Related papers

Loading 2610.33299v1…