PaperlayerView exact revision on arXiv

Source-linked AI summary

Through the Looking Glass: Directly Reading and Writing Transformers

Mark Oskin

arXiv:2609.10210v1cs.CLcs.LG

TL;DR

The paper asks which identifiable transformer components matter to a particular token prediction and whether they can be traced as a circuit. It introduces an unfitted lens derived from model parameters and activations, finding sufficient sets of two to sixteen components across twelve models trained elsewhere.

  • Problem

    The paper asks which identifiable parts matter to a particular token prediction and whether they can be traced as a circuit.

    Source: §1 · p. 2

  • Method

    An unfitted lens derived from a transformer’s own parameters and activations identifies components that carry, are necessary for, or suffice to produce a prediction.

    Source: §1 · p. 3, §9 · p. 81

  • Results

    2–16 components sufficed across twelve models trained elsewhere; on the baseline, 53 carried 90% of a prediction, 13 it could not survive losing, and 8 sufficed to produce it.

    Source: §1 · p. 3, §9 · p. 81

  • Takeaways & Limitations

    For components operating on directions no token induces, lexical names are unavailable, and their upstream components provide addresses instead.

    Source: §8.3 · p. 80

    Sufficient-circuit sizes are heuristic-search upper bounds, and search cost grows with model size; some predictions on a 24-layer model returned bounds rather than answers.

    Source: §2.5 · p. 13

Abstract

from arXiv · show

How many of a transformer's components decide a token? Counted by the absolute value of each unit's and channel's contribution to the logit, one prediction rests on thousands to hundreds of thousands of them. But contributions are signed, and across eighteen models the mass pushing away from the predicted token is a median of seven times the mass carrying it. Divide by the net and the count is dozens: on the baseline, 53 components carry ninety percent of a prediction, 13 it cannot survive losing, and 8 suffice to produce it alone. Across twelve models trained elsewhere, 124M to 7B parameters, the sufficient set runs from two components to sixteen, and what a prediction draws on, followed all the way back, is one to three percent of the model, a share that does not grow with size. Three quarters of a layer's update is a fixed linear map of the state it received. Everything is read from the model's own parameters and activations, with nothing trained or fitted, and it names a component on both sides: what it writes, from the predictions it drives, reaching close to half of every model; what it reads, from its weights in the frame of its own layer, at 58.9 percent above chance over its eight strongest inputs. Sorting the remainder by upstream source yields grammatical categories the embedding cannot see. A name can be acted on. An association the model does not hold installs into one spare unit, key and value read from the weights, for a quarter of a percent of held-out loss, a fortieth of what a rank-one update costs. An installed attention head and a unit two layers above it make an edit fire only where a token occurred earlier in the context, and a unit the model trained for itself is driven from two layers upstream, 86 percent of the effect passing through it. An order-preserving activation puts a unit's inputs at the instrument's ceiling, at the price of a two-part install.

1 Introduction

The paper introduces a parameter-derived lens for tracing small, token-specific circuits and for naming and editing their components. Its introduction reports baseline results, explains the residual stream’s role, and previews extensions across activations and externally trained models.

Baseline circuit results

  • The standard absolute-value count assigns 3,000–200,000 components to a prediction, but signed attribution yields dozens after accounting for cancellation.Opposing mass is a median of seven times the net across the models and twenty times it on the baseline model.Source: §1 · p. 2

Reading components

  • The lens uses model parameters without training or fitting, naming what a component writes from predictions and what it reads from its layer-frame weights.For a unit’s eight strongest drivers, weights recover the true set at 58.9%, thirty-seven times chance; write names reach close to half of every model. Names must be coherent and behaviorally verified, and traces should connect components end to end to the input and output.Source: §1 · pp. 2–3

Baseline circuit results

  • On the baseline, 53 components carry 90% of a prediction, 13 are necessary, and 8 suffice to produce it alone.Eight random components never produced the same token across 115 tested predictions.Source: §1 · p. 3

Residual stream and circuit scope

  • Three quarters of a layer’s residual update is a fixed linear map, while the remainder of the model manages shared workspace rather than concentrating prediction work.Keeping only the circuit costs a quarter as much at the predicted token as an equally sized random set, but the costs match over the next 512 tokens.Source: §1 · p. 3

Editing components

  • An association absent from the model can be installed in one spare component, moving its target from rank 578 to rank 1 for 0.25% held-out loss.That is a fortieth of the cost of the cited rank-one update; the introduction also previews a two-layer edit and a trained unit receiving 86% of the effect from two layers upstream.Source: §1 · pp. 3–4

Extensions

  • Across 12 externally trained models spanning 124 million to 7 billion parameters, the sufficient set ranges from two to 16 components.Order-preserving sigmoid or softplus units make inputs readable at the instrument’s ceiling but require a two-part install; in gated feed-forward models, nearly half of prediction-carrying components fire negative.Source: §1 · p. 4

2 Reading a prediction: The Looking Glass

This section uses model parameters and activations to attribute predictions to components and examine which are necessary or sufficient to produce them. It finds substantial cancellation and small, structured sufficient sets, while noting that remaining components support other processing and that these attributions have important interpretive limits.

2.1 Models

  • The baseline is a GPT-2-small-architecture model with 12 layers of width 768, trained at context length 2048; the released GPT-2 uses 1024 and is measured separately.Components are feed-forward units or attention head-channel pairs, each counted as a scalar multiplying a write column.Source: §2.1 · pp. 4–5

2.1.1 Training and seeds

  • Five of eight baseline seeds converged at perplexity 19.0–19.7, while the three failures reached 21.6, 24.4, and 39.5; converged seeds span LAMBADA scores of 0.248–0.274.Seeds differ only in initialization and data order, and are separated by quality before legibility is measured; overlapping LAMBADA ranges are treated as equal.Source: §2.1.1 · pp. 5–6

2.1.2 Additional models used

  • The additional scope includes five locally trained models that vary only the feed-forward activation and 12 externally trained models spanning 124 million to 7 billion parameters.GPT-NeoX/Pythia is excluded because its attention and feed-forward sublayers run in parallel, while the readings assume a sequential arrangement.Source: §2.1.2 · p. 6

2.2 The lens

  • Scoring both sides of an inter-layer edge in the reader’s own frame gives correlation +0.28 and beats a size-matched random control on 76% of cases, unlike the mismatched-frame correlation of −0.02.The layer-native token table is averaged over held-out contexts and read from model forward passes, with nothing fitted; the lens uses it to decode intermediate writes, reads, and edges.Source: §2.2 · p. 7

2.3 Identifying components that matter

  • The net denominator yields tens of components for a 90% prediction, versus tens of thousands when signs are discarded; opposing mass is 20 times the net at the baseline median and seven times across 18 models.The comparison holds predictions and attribution fixed while changing only the denominator; the table reports medians over 115 predictions whose top-1 probability is at least 0.3.Source: Table 3 · p. 8, §2.3 · p. 8

2.4 What part of a transformer is necessary

  • Removing components lowers the target logit by only 42–50% of their predicted contribution sum, as remaining components partly repair the loss; the identified components also change nearly every prediction by k = 128.The random-removal control changes at most 5% of predictions. Ablation therefore measures behavior beyond the linear contribution accounting, and the logit shortfall is not an accuracy rate.Source: §2.4 · p. 9, Table 4 · p. 9, Figure 2 · p. 9

2.5 What part of a transformer is sufficient

  • Eight components out of 46,080 reproduce the prediction; eight random components succeed on none of 115 cases, while the eight highest-contributing components succeed on 7%.The 12% floor—predictions already returned by the token embedding alone—is excluded. The reported set size is a heuristic-search result, not a proven minimum.Source: §2.5 · p. 12, Table 7 · p. 12
  • A random-start repair search succeeds on 85% of predictions versus 89% from the required-graph start, and the resulting sets overlap more than random sets; these are context-specific, position-level sets, not proven minima.Sufficient sets have median sizes of 18 against the layer-to-layer map and 16 against zeroing, on a different search and floor from the eight-component result. Sizes are heuristic upper bounds, and search costs grow with model depth.Source: §2.5 · pp. 12–13
  • Sufficient sets are small graphs spanning about five layers; two thirds to three quarters of members are feed-forward units, and most draw only a median tenth of their drive from other members.In Figure 4, seven of the eighteen members are attention channels, the only route by which information from outside the position reaches the set.Source: §2.5 · p. 13, Figure 4 · p. 14
  • On contrastive pairs, losing tokens retain positive net push and 58 to 73 percent of their push; they are out-pushed, not suppressed, by a median of three to five logits.Removing eight winner promoters changes the prediction on 30 to 72 percent of pairs, while removing loser suppressors changes it to the loser on 8 to 26 percent, as it would for any token.Source: §2.5 · p. 13, Table 8 · p. 15
  • At the median, 53 components carry the prediction, 13 cannot be removed without losing it, and 8 suffice to reproduce it.In Figure 4, 18 components preserve one prediction, with n90 = 42 of 46,080 components.Source: §2.5 · p. 13, Figure 4 · p. 14

2.6 What the rest of the model is doing

  • Keeping only a prediction’s circuit does not reveal staging for later tokens: its later-token cost matches an equally sized random set, with median cost ratios of 0.98 to 0.99 from 16 to 512 tokens.The circuit costs a quarter as much as a random set at the predicted token itself.Source: §2.6 · p. 15, Table 9 · p. 15, Table 10 · p. 16
  • Near misses also draw substantial effort: the losing token carries 0.63 to 0.84 of the winner’s net push, while a top-fifty competitor carries 0.43 to 0.79.Counted as positive mass, the runner-up draws 0.94 to 0.99 of the winner’s mass and a top-fifty competitor 0.88 to 0.98; their circuits contain 26 to 48 components on four smaller models and 100 to 200 on Gemma-3 and Qwen.Source: §2.6 · pp. 15–16
  • A fixed map of the incoming state accounts for 74 percent of the baseline layer update and 79 percent on the sigmoid model; the first layer shears the stream three to five times as much as later layers.A rotation accounts for the bulk of the update captured by the fixed map, and the map has positive held-out R2 at every depth.Source: §2.6 · pp. 16–17, Table 11 · p. 17
  • Replacing one layer’s output with the fixed map’s prediction still yields the same token on a median of 92 percent of predictions.The test substitutes the map at the position being read while the rest of the sequence computes normally.Source: §2.6 · p. 17
  • Tracing circuit members’ necessary upstream sources expands the closure to one to three percent of the model, while direct attribution and sufficiency searches can miss components acting only through later components.Removing attributed sources lowers a node’s activation by two thirds; about a dozen signed sources reach 90 percent of its incoming drive, versus many thousands counted without signs.Source: §2.6 · p. 17, §2.7 · pp. 17–18

3 What’s in a Name?

This section characterizes what components write and read, using predictions and parameters in each component’s layer frame. It also traces component inputs through provenance, revealing upstream sources and grammatical categories, and applies the approach to attention channels.

3.1 What a unit writes

  • Close to half of a model’s components have a nameable write, typically promoting grammatical or other related token sets rather than a single token.In the baseline, characterized components are concentrated late in the stack: the share reaches a quarter by layer six, half by layer eight, and 95 percent at the last layer. Only 0.1 percent write a single token.Source: §3.1 · p. 19

3.2 What a unit reads

  • 58.9 percent of a baseline unit’s eight strongest parameter-derived drivers match its measured drivers, but agreement falls below chance by K = 256.The layer-native token table yields 58.9 percent agreement, versus 7 percent when read through the output vocabulary projection. Embedding-visible concepts account for 28 percent of a unit’s response; corpus-coherent token sets capture categories the embedding does not group together. Both naming shares are upper bounds on semantic work because transport is indistinguishable from prediction.Source: §3.2 · pp. 20, 22, 24

3.3 Naming an attention channel

  • At eight tokens, baseline channel read agreement is 0.844 versus 0.598 for units, and at sixty-four it is 0.892 versus 0.279.A channel’s value-projection row is decoded against the layer-native token table; its linear value path preserves the ordering, unlike a unit’s activation function. Channels are characterized mainly in later layers, reaching three quarters to five sixths from layer eight onward.Source: §3.3 · p. 24

3.4 What the graph shows

  • Anonymous unit responses are reproducible across held-out contexts, and graph-based upstream rankings place their driving tokens above the rest of the candidate pool.The two response profiles agree at 0.648, while three null tests return chance or close to it. However, sibling components can mimic a unit’s firing profile without writing into it, so ranking alone does not rule out that source shape.Source: §3.4 · p. 27, Table 17 · p. 27
  • With 24-component ablations, direct weighting tracks damage while sibling weighting stays flat in both families; single-component deletions cannot distinguish the weightings.On the baseline, the two weightings rank a unit’s drivers about equally well, but their predictions diverge for larger ablations.Source: §3.4 · pp. 27–28
  • Embedding terms favor nameable tokens, while terms carried six or more layers favor anonymous tokens, showing that a static token table misses distant contributions.The split appears when Equation 17 is grouped by how far each term was carried.Source: §3.4 · p. 28, Figure 9 · p. 28
  • Sorting a unit’s tokens by their largest source contribution yields a median of four grammatical groups, with source recovery and part-of-speech purity above controls.Groups include sentence-openers, modals, do-support, prepositions, and wh-words; their coherence is not captured by mean pairwise cosine.Source: Table 18 · p. 29
  • The provenance ordering traces attention channels against both nulls across model families, with baseline channels about twice as traceable as baseline units.The channel comparison uses excess over same-layer channels; channels’ linear readout matches the predictor’s form, while unit responses pass through an activation function.Source: §3.4 · p. 29
  • Unlike feed-forward nodes, attention channels draw broadly on the residual: most channels carry a read side, but only a handful carry an incoming edge.Nearly half of model components are named by what they write; provenance sorting names the otherwise hidden remainder, about three quarters of a unit’s inputs, as grammar.Source: §3.4 · p. 30

4 Editing a Model: Say what?

This section tests targeted edits to model components identified by what they write, including turning components up, installing associations, and editing attention. It also examines how writes survive and how cross-layer circuits carry information, including the accuracy limit of reader-frame estimates and a comparison of single-unit installs with rank-one updates.

4.1 Turning a component up

  • In 97 percent of cases, turning a component up moves its named scope more than other components’ scopes, with the named-scope effect exceeding elsewhere by more than an order of magnitude.As α rises from 2 to 8, named-scope movement rises from 0.30 to 1.77 while other scopes stay near zero.Source: Table 20 · p. 31, §4.1 · p. 31

4.2 Installing an association that was not there

  • At β = 1, one spare unit moves B from rank 578 to 1 where A occurs, leaves its elsewhere rank at 778 to 782, and costs a quarter of one percent of held-out loss.The key and value are read from the model’s parameters, and the unit is chosen as the one with the smallest corpus activation; neither is fitted or optimized.Source: §4.2 · p. 32

4.3 Editing attention

  • Attention-channel edits move promoted tokens on 88 to 91 percent of trials, versus 93 to 94 percent for units, but only 10 percent of channel installs succeed versus 32 percent of unit installs.Attention averages values from attended positions, diluting the edit; it needs β = 8 versus β = 1 for a unit and costs a tenth of held-out loss versus a quarter of a percent. Accounting for the channel’s sign is essential: unsigned scoring returns chance, 47 to 49 percent.Source: §4.3 · pp. 32–34

4.4 Where to place an edit

  • A write before layer two survives at 0.11, versus 0.33 to 0.81 before layer eleven; spill falls from 0.40 at layer zero to 0.04 at layer eleven.Depth matters more than direction family at a fixed depth, and the readout-frame write is the only family whose readout visibility rises above chance, from 0.27 to 0.43. The final layer consumes readout-frame content, so the paper’s constructions place readers at layer ten of twelve.Source: §4.4 · pp. 35–36, Figure 12 · p. 35

4.5 What the turn does to a write

  • A write before layer three aligns with itself at 0.70 three layers later, versus 0.60 with its turned image; the turn contributes 0.065.The direct write is larger than the turn’s contribution, while the model’s response grows with distance: alignment with the write falls from 0.89 at one layer to 0.25 at nine. Accordingly, reader inputs are measured at their destination rather than inferred by carrying the write through a map.Source: §4.5 · p. 37

4.6 A circuit across two layers

  • Across layers eight to eleven, the model’s shared working subspace spans 183 dimensions, leaving about 580 of 768 dimensions available for a private carrier direction.The proposed circuit installs an attention head to write that direction where a chosen token occurs, then a downstream feed-forward unit to read it and write a target token’s readout direction. Repeating the construction with different carrier frames leaves the target at median rank 12 and separation at 71.6.Source: §4.6 · p. 38
  • 0.121 nats brings the target into the top ten with the cross-layer circuit, rising to 0.179 nats when specificity is required.Across 24 triples and 2,400 trials, the target reached median rank 29 where the condition occurred versus 4,970 elsewhere; 54% of triples reached the top ten within the 10% loss budget, and the cheapest setting preserving rank elsewhere cost 0.029 nats. Separation rose from 5.1 to 71.6 as the read-row sample grew, making estimate accuracy the constraint.Source: §4.6 · pp. 38–40, Figure 14 · p. 39

4.7 A reader is what makes off-axis content visible

  • 3.5 times: writing a direction and adding a layer-ten reader promoted an unrelated target this much, versus 1.1 times for the write alone and 1.0 for the reader alone.The reader used a measured, thresholded read row to write the target’s readout direction; the largest effect came from the private direction, while the two model-derived directions gave the least.Source: §4.7 · p. 40, Figure 16 · p. 40

4.8 Driving a component the model already has

  • 86% of the target promotion passed through a native unit driven from two layers below in the frame it already reads.The write raised the unit’s activation by 24.1 standard deviations and moved its token from rank 1116 to rank 32, for 0.049 nats. A weight-derived direction mediated 0.86 of the effect, compared with 0.55 for an activation-estimated direction.Source: §4.8 · p. 41, Figure 17 · p. 41

4.9 Scoring against a rank-one editor

  • 0.0016 nats versus 0.0681 nats: the single-unit install reaches the top ten at one-fortieth the cost of the closed-form rank-one update.The target’s rank elsewhere is 1.02 times its starting rank for the install versus 0.81 for the closed-form update. The gradient-valued published method reaches rank one for 0.611 nats and holds specificity at 0.80; unlike the install, it spreads the write across the layer and needs no spare component, but requires a corpus sample and gradient pass.Source: §4.9 · pp. 42–43, Figure 19 · p. 43

5 What the activation function changes

This section examines how order-preserving activation functions affect transformer legibility, circuit size, quality, and the edits components can support. It covers token-set recovery, local association installs, and a two-part context-conditional edit, including the constraints on these edits.

5.1 Why a conventional unit stops being readable

  • Order-preserving functions yield 96–99% reading, versus GELU’s 67%; 59.8–62.0% of baseline pre-activation mass lies in GELU’s decreasing region.This is a reading ablation on baseline pre-activations, not a prediction that the property survives training with another activation.Source: §5.1 · pp. 43–44

5.2 Which activations make sense

  • Sigmoid and softplus are strictly increasing and injective, while ReLU is monotone but not injective; bounded fuzzy set operators provide a fourth order-preserving design [39] [40] [42].The set-operator units use a gate operand to control how much they write and a selector operand to choose between two fixed columns.Source: §5.2 · p. 45

5.3 The construction

  • The arms retain the baseline’s depth, width, heads, parameter budget, data, and schedule, while the set-operator arm replaces the unit operation.Attention values are sigmoid-bounded across the standard arms, and a 0.003 auxiliary term applies only to bounded activations; the two sigmoid variants are equivalent on reported measurements.Source: §5.3 · p. 46

5.4 Quality

  • ReLU trails every other arm by about 10% on perplexity and falls below every other converged seed on accuracy, while the other arms overlap the baseline’s quality range.All eight softplus seeds and all eight set-operator seeds converged; the set-operator arm scored 19.23–19.43 in perplexity and 0.234–0.274 on LAMBADA.Source: §5.4 · pp. 46–47, Table 24 · p. 47

5.5 Circuit size across the arms

  • The arms use 26–36 components versus the baseline’s 53, and their median predictions turn over at one-quarter the baseline’s k.The set-operator arm uses 36 components at 90% of the net, with inflation of 277 times versus the baseline’s 452.Source: §5.5 · pp. 47, 50, Table 26 · p. 48, Figure 23 · p. 49

5.6 The whole dependency graph

  • At closure, activation-function arms separate from the baseline by four to seven times; their required graphs need 3–5 components to change a prediction, versus 13 for baseline.The set-operator arm closes on 120 components in six levels. Signed incoming-drive medians exclude nodes where drive is zero or below, including more than half of the sigmoid model’s nodes.Source: Table 27 · p. 50, Table 28 · p. 50, §5.6 · pp. 50–51

5.7 Legibility

  • At K = 256, baseline agreement falls to −9.5%, while sigmoid, softplus, and set-operator agreement reach 82.4%, 82.4%, and 82.8%, respectively.These three arms are at the causal measurement’s reproducibility ceiling; their curves stay flat across K, unlike the baseline’s decaying curve.Source: Figure 24 · p. 51, §5.7 · pp. 51–52
  • Under sigmoid, effective set size falls by about a third, and upstream-source sorting improves token-source classification and part-of-speech purity over baseline; graph-based component naming improves reliably only for sigmoid and set-operator arms.The set-operator arm’s self-naming margin is 10.0 times but is based on 42 components; softplus’s corrected margin collapses because its repeated edge labels raise the control rate.Source: §5.7 · p. 53, Table 29 · p. 53
  • An incoming-edge description contains a token the component fires on 56% of the time, versus 20% for a control, but covers only 17% of its responses.This description rate is the same for components that can name themselves and those that cannot (60% versus 56%); parameter-derived operand A and B readings score 68.5–72.0 and 64.7–68.9, respectively, versus 33.6–38.4 for the baseline’s single row. Splitting A’s top-40 tokens by B increases coherence by 0.0166, versus 0.0002 for a random split.Source: §5.7 · pp. 53–54

5.8 Which property was responsible

  • Sigmoid and softplus agree on measured quality and have agreement flat at 82, while ReLU scores 75.6–79.5% at K = 8 but loses quality.Across constructions, order-preserving activations land on one agreement curve; ReLU’s units respond to fewer tokens, with arity 2.4–2.7 versus sigmoid’s 5.3–6.7. The same construction yields 18 components on the baseline and 4 on sigmoid in the carriage-background search, though whether that and improved legibility share a mechanism is unresolved.Source: §5.8 · pp. 54–55

5.9 Editability

  • Monotone units reach promote-edit efficacies of 6.3–9.3 versus 3.3 and 3.9 for baseline units, with set-operator units reaching 10.1–14.3.The efficacy ranges do not overlap; at matched efficacy 3.0, set-operator collateral is 0.003–0.005 versus 0.023 and 0.085 for baseline, while monotone models are typically, but not uniformly, more targeted. Baseline association installs at read gain 1 on 48% of trials for 5% held-out loss, while no monotone arm takes an install when only the unit is changed.Source: §5.9 · pp. 55–57, Table 30 · p. 55

5.10 The edit these models will take

  • With compensation, order-preserving arms install associations on 9% of softplus attempts and 16% of ReLU attempts, at held-out loss under 0.05%.A correction subtracts the installed unit’s off-key output from other near-constant units in the layer; thresholding alone leaves shaped arms at 0–9% local installs, while correction plus threshold reaches 27–36%.Source: §5.10 · p. 59
  • Shaped attention channels have little usable key-versus-absent signal: sigmoid’s difference is 0.04, versus 0.44 for baseline, and no compensated edit passes at any strength.The bounded values are flattened toward the activation’s resting level, so the target is delivered away from the key; the most local heads still reach 0.87 and 0.89 versus 0.75 for baseline, and can bring the target to rank 2–5 at the key.Source: §5.10 · pp. 59–60

5.11 Placement, circuits and the rank-one editor across the arms

  • A single-unit install reaches the top ten for less held-out loss than a closed-form rank-one update on all five models, by factors of nine to seventy-two.On sigmoid, a write retains 0.39 of its direction at readout before the last layer versus 0.10 before layer two; a unit trained by the model is driven from two layers upstream with 0.88 of the effect passing through it.Source: §5.11 · p. 60
  • Thresholding the read row lowers sigmoid’s held-out-loss budget to reach the top ten from 0.35 to 0.0020, while the baseline falls from 0.0032 to 0.0016.For sigmoid, the cheapest rank-one route falls from 1.31 nats to 0.0092 with a threshold.Source: §5.11 · p. 60, Figure 26 · p. 61

5.12 Editing through the second operand

  • Editing the set-operator unit’s second operand can install a conjunction and a suppressor in one write, by keying that operand on token C.The unit retains its original trigger through the first operand; the intersection branch fires on its usual trigger and C, while the difference branch fires on its usual trigger and not C.Source: §5.12 · p. 62
  • Held-out loss moved only from 3.156 to 3.157, while the second operand’s bounded gate made suppression exactly zero where C was present.The intersection branch fired five times more often with C present, while the difference branch fired zero times there. Self-cancellation did not improve the conditional edit: 31 attempts passed with the difference column and 31 without.Source: §5.12 · pp. 62–63
  • Five weight writes install an attention channel that carries C from earlier context to the unit’s second operand, enabling the feed-forward block to test the preceding-token condition.Scaling the channel’s key and supplying a bias from the layer’s near-constant direction raised its contrast from 1.02–1.45 to 901 and its firing ratio from 1.2 to 43; these settings came from measured distributions, not target fitting.Source: §5.12 · pp. 62–63
  • 31 of 90 attempts passed while the unit reached the top ten only when A was current and C appeared in the preceding 48 tokens, with held-out loss within 10% of baseline.The sharpest circuit put the target at rank 1 under the condition versus 10,170 without it and 14,215 elsewhere, for 1.6% of loss; the cheapest moved rank 3902 to 9 under the condition for 0.2% of loss. An independent replication passed 17 of 100 attempts.Source: §5.12 · p. 63
  • Removing the installed attention head reduced the conditional ratio from 17.65 to 1.24 and passing attempts from 31 of 90 to one of 90.The head carries the condition; without it, the operand reads a direction nothing writes.Source: §5.12 · p. 63
  • The conventional baseline also performed the edit: 28 of 90 attempts passed versus 31 for the set-operator arm, while its conditional ratio was 194.5 versus 17.65.The second operand’s benefit was locality rather than condition sharpness; within the set-operator model, a single-operand version had a similar ratio (18.49) but promoted the target wherever C occurred, regardless of A.Source: §5.12 · pp. 63–64
  • Order-preserving units read at the instrument’s ceiling, while conventional units reverse on their strongest inputs; order-preserving circuits were smaller at one hop and at closure, and their promote edit reached twice as far before saturation.The order-preserving units’ circuits were half to two thirds the size at one hop and four to seven times smaller at closure; bounded units also require a two-part install because they lack a silent state.Source: §5.12 · p. 64

6 Models trained by other people

The paper applies its analysis unchanged to twelve models trained by other organizations, finding small prediction-carrying sets across diverse models alongside substantial cancellation. Further tests show that these models’ prediction circuits can be traced, isolated, and interpreted, with important limits on what can be inferred from uncontrolled model differences.

6 Models trained by other people

  • 12 off-the-shelf models reach ninety percent of a prediction with a few dozen to a few hundred components, while the absolute-value convention reports two to three orders of magnitude more.The accounting uses the same procedure, confidence floor, and matched random control across the models.Source: §6 · p. 64, Figure 30 · p. 66

6.1 Composition

  • 28 to 76 percent of sufficient-circuit members fall in the last quarter of the stack, and 79 to 100 percent of distinct components appear in only one circuit.Attention contributes 22 to 40 percent of one-hop sets; these shares do not sort by architecture family, size, or activation in this sample.Source: §6.1 · pp. 67–68, Figure 31 · p. 67, Table 36 · p. 67

6.2 The whole dependency graph

  • Across the models, the dependency graph closes at seven to ten levels, and tracing a prediction back terminates on a few percent of model components without growing in share with model size.The required graph is two orders of magnitude smaller than the closure; removing 34 components can change a seven-billion-parameter model’s prediction, but branching factors from 16 to 119 do not sort by activation in this uncontrolled comparison.Source: §6.2 · pp. 68–69

6.3 The smallest sufficient set across models

  • Two to twenty-one components suffice in every searched model, with the largest models at the low end: Mistral 7B and Qwen2.5 7B each require two.A same-sized random set succeeds on none of the sampled predictions, while contribution-ranked controls succeed on 0 to 22 percent; unfinished searches are excluded from the reported sizes and controls.Source: §6.3 · p. 70, Table 40 · p. 70

6.4 Naming a component and acting on the name

  • 94 to 97 percent of edits move the parameter-named scope more than other components’ targets across the three tested off-the-shelf models.In gated models, 47 and 48 percent of prediction-carrying components fire negatively; omitting that sign yields only 4 to 11 percent correct edits on negative-firing components, while including it fixes the scope reading.Source: Table 42 · p. 71, §6.4 · p. 71, Table 41 · p. 71

6.5 Where the instruments are stressed

  • The raw logit drop disagrees with the margin on effect direction in up to 45 percent of cases and is non-monotone in two models.Raw-logit measurements can therefore misstate the effect, including a negative realized-to-predicted effect ratio for GPT-2.Source: §6.5 · p. 71, Table 43 · p. 72

6.6 What the constructions are worth

  • Roughly two thirds of units are token-shaped in models trained for legibility, versus about a third in the remaining models; across nine models, circuits are no better than equally sized random sets for following tokens.The per-prediction ratio stays between 0.73 and 1.07 at every tested horizon from sixteen to five hundred and twelve; conventional transformers can be read without the extra legibility constructions.Source: §6.6 · pp. 72, 74, Figure 33 · p. 72, Table 45 · p. 73

7 Related work

The section separates inherited methods from this paper’s contribution: component-level, signed accounting of predictions and weight-based component selection for edits. It contrasts these choices with prior work and clarifies the scope and limits of the resulting claims.

Decomposing a prediction

  • The residual-stream framework makes logits exact sums of component terms, but opposing writes and downstream compensation mean linear attribution can overstate removal effects.Janiak et al. [26] show opposing writes and recommend pairing direct attribution with activation patching; McGrath et al. [35] and Rushing and Nanda [46] show that other components compensate after ablation. The paper also projects component writes into vocabulary space, an established practice, and makes its substitution protocol specific to this work.Source: §7 · p. 74

Prediction-specific graphs built backward

  • The backward prediction graph’s recursion is not new, but this work uses signed edges between individual components and reports graph size and removal-based necessity.Ferrando and Voita [19] use token-representation nodes and nonnegative proximity scores, while Abnar and Zuidema [1] use maximum flow; Ferrando and Voita report routes rather than graph sizes and validate against previously patched circuits.Source: §7 · pp. 74–75

Circuit discovery

  • Two to three orders of magnitude fewer components reach 90% under signed net normalization than under absolute attribution on the same predictions and attribution.The signed-sum convention itself is established; the paper measures its consequence for circuit size. Chen et al. [10] caution that a signed denominator becomes unsafe when recursively iterated, unlike the single component sum used here.Source: §7 · p. 75

The neuron basis

  • Raw-neuron circuits remain contested: Arora et al. [3] find sparse neuron-basis circuits, whereas Ameisen et al. report weaker interpretability than transcoder features.The closest comparison differs in selecting nodes by attribution threshold and validating by steering rather than removal against a size-matched random control; its attribution also sums several candidate tokens and omits attention as a comparable component.Source: §7 · p. 75

What ablation replaces

  • Necessity is nearly unchanged across zero, mean, and resample ablations, while sufficiency shifts severalfold, motivating the paper’s emphasis on necessity.Li and Janson [29] attribute baseline differences to information in replacement values; the paper also tests removing constant writes, which can dominate the signed sum in a seed.Source: §7 · pp. 75–76

What the feed-forward stack is for; Legibility by construction

  • Prior work debates whether feed-forward layers are load-bearing, while legibility-by-construction methods alter model structure or activations rather than the analysis.The paper asks what one prediction needs from an existing stack; its unit-activation constructions change pre-activations while leaving connectivity dense and use the same evaluation instrument as the baseline.Source: §7 · pp. 76–77

What the feed-forward stack is for

  • 43–81% of components needed to reproduce a prediction sit in the last third of the stack, while the middle holds 12–55% on eight of ten models.Attention makes up 22–54% of retained components in the middle, versus 19–35% in the last third; this is consistent with input-dependent feed-forward memories and does not contradict results about what a stack can represent.Source: §7 · p. 76

Editing a component

  • Section 4.2 combines weight-based site selection with an installed association scored by whether an edit succeeds, unlike prior methods that choose sites from model behavior.The update builds on closed-form unit-write methods, but chooses its site from the read row in the layer’s own frame, with nothing traced or fitted; the reported edits are single-position tests on a 124M-parameter model.Source: §7 · pp. 76–77

8 Discussion

The discussion argues that component-level measurements make prediction circuits auditable and edits checkable, while raising questions about carriage, vocabulary, component naming, and the instrument’s scope.

8 Discussion

  • Named-component edits target stated behavior rather than fine-tuning, but content cannot be installed locally in attention channels because their writes aggregate information from other positions.Edits can use the model’s own frame or an unused frame that requires supplying both circuit ends; the tap’s measured limit is read-row estimation accuracy. The paper’s circuits cover single positions and predictions, not full-context graphs or several architecture types.Source: §8 · pp. 77–78, §8.4 · p. 81

8.1 What the carriage result asks of the architecture

  • Prediction graphs vary four- to sevenfold across activation functions, while closure sizes remain 1.1 to 3.4 percent of model components across seven architecture families with no size trend.The compared models share width, data, schedule, and converged quality. About three quarters of the residual update is predictable from the preceding state, and carriage and content cannot be cleanly separated into subspaces.Source: §8.1 · pp. 78–79

8.2 What a one-component circuit says about the vocabulary

  • On Gemma-3 1B, 49 of 134 verified predictions are carried by one component, and sampled punctuation units suggest some encode word-specific punctuation associations.The punctuation interpretation rests on two comma units, so the passage calls it anecdotal; none of the six tokenizers checked joins a word to a following mark.Source: §8.2 · pp. 79–80

8.3 Where the components have no name

  • A lens cannot lexically name components operating on directions no token induces; the unexplained response is reproducible and unit-specific, while source-sorted categories show grammatical structure.At thirty-two directions, a shared basis explains 68 percent of coherent response versus 55 percent of the remainder on the baseline. Part-of-speech purity within source groups exceeds a random partition, but further feature naming remains open.Source: §8.3 · pp. 80–81

8.4 Is this as good as it gets?

  • Sufficient-set sizes are upper bounds: the search converges on 96 percent of predictions on some models and 44 percent on others, and the instrument does not yet cover several architectures.The method follows dependencies for one position and stops at what an attention channel reads there; extending it to context-wide graphs and routers remains future work.Source: §8.4 · p. 81

9 Conclusion

The paper presents a parameter- and activation-based lens for finding, naming, and editing transformer components, and frames these methods as a toolkit and roadmap for building, training, and using transformers.

9 Conclusion

  • On the baseline, the lens identifies 53 components carrying a prediction, 13 it cannot survive losing, and 8 sufficient to produce it, out of 46,080; sufficient sets range from 2 to 16 across twelve other models.These searched set sizes are upper bounds, not established minima, and are far below what common conventions report for the same predictions.Source: §9 · p. 81
  • Three quarters of a layer’s update is a fixed map of its incoming state, while readout mass pushing away from the prediction is seven times the mass carrying it.After separating out this geometry, predictions name close to half of a model’s components; a unit’s strongest few inputs can be read from its weights in its own layer’s frame.Source: §9 · p. 81
  • For unnamed unit drivers, tracing upstream writers through the dependency graph supplies provenance, and a classifier recovers categories from that sorting.Source: §9 · p. 82
  • A gain stays within the scope named by parameters, while an association the model never held can be installed in one spare component using both halves read from weights.The installation requires neither tracing nor optimization.Source: §9 · p. 82
  • A model-trained component can be driven from upstream layers, and a new circuit can carry intermediate state on an unread channel before committing its answer on the token axis.Source: §9 · p. 82
  • The order-preserving activation puts a unit’s inputs at the instrument’s ceiling at parity quality, but turns an install into a two-part edit.Source: §9 · p. 82

A Gallery · A.1 How to read a page · A.2 How these traces were chosen

The gallery presents complete, verified sufficient sets for predictions, with every component and edge shown. Its pages explain how to read those circuits and distinguish mechanically selected typical traces from examples chosen for their undescribed edges.

A Gallery

  • Each gallery panel shows a verified sufficient set: retaining its components and zeroing all others still produces the same token, though these prefix-and-prune sets are typically larger than Table 7 medians.Components fire from the residual left by the other survivors; nothing is spliced in, and a sufficient set is an upper bound.Source: A Gallery · p. 82
  • Two predictions per model are shown in full, with every set member as a table row and every edge in the graph, even when large traces need separate pages.Smaller traces share a page; larger tables and sideways graphs each take a page of their own.Source: A Gallery · p. 82

A.1 How to read a page

  • Rows trace each component’s three strongest suppliers through the set, while bold marks components whose removal from the intact model changes the prediction.Shared ancestors appear once; the necessary components and the components that reproduce a prediction can differ, possibly because multiple sufficient circuits exist.Source: §A.1 · p. 82
  • Columns report firing strength, tokens that drive a component, signed promoted tokens, and the predicted token’s rank in each write list.The write sign matters: in a gated feed-forward, a negatively firing unit can promote the target when its column points away from it.Source: §A.1 · pp. 82–83, §A.2 · p. 83
  • An attention channel’s read side is measured from its value projection, but the lens does not read the attention pattern that aggregates values across the sequence.Its incoming edges are followed like those of a feed-forward unit.Source: §A.1 · p. 83
  • Edge labels show up to three shared tokens ranked in each side’s first eight, but population-level agreement does not establish that an individual edge’s leading token is meaningful.The shared-token bar is met on between a quarter and three fifths of edges, depending on the arm.Source: §A.1 · p. 83
  • Edges with no shared token are tagged by their writer’s address, making reuse of an internal quantity visible even when no vocabulary describes it.Italic glosses name token categories without replacing the model’s tokens, and the graph shows all members’ relations by depth through to the answer.Source: §A.1 · p. 83

A.2 How these traces were chosen

  • Traces were filtered for confident, non-repetitive, letter-containing predictions, then the two nearest the draw’s median sufficient-set size were kept per model.Up to twenty predictions were drawn randomly from held-out text, stopping early if filters exhausted the sampled contexts; later examples were selected for containing edges no token describes, although most mechanically drawn predictions had no such edges. Models trained for the paper precede the twelve off-the-shelf models, which are ordered by size.Source: §A.2 · pp. 83–84

A.3 What the conventions rest on … B.2 The attention head

The appendices qualify how to read traces, show cases where token labels fail, and specify the set-operator unit and attention-value construction. They pair weak evidence on individual edges with aggregate signal and describe a parameter-matched model design.

A.3 What the conventions rest on

  • Across 23,000 edges, real writers win on 63% against 32%, but only 13% of baseline and 11% of sigmoid edges beat twenty controls at p ≤0.05.The median edge sits at p ≈0.35, so an individual edge token is weak evidence; concept edges have no shared token among the candidates, occurring on a median 2% of edges per model and 65% in OPT-125M.Source: §A.3 · p. 84
  • Required components are often attention channels that read from elsewhere in the sequence, and a source can exceed 100% of a node’s drive when other sources cancel against it.The trace makes visible both the channel read-side caveat and the net-drive accounting convention.Source: §A.3 · p. 84

A.4 Traces where the vocabulary runs out

  • The selected traces are not a sample: they foreground concept-free edges, which make up about two thirds of OPT-125M trace edges versus a few percent elsewhere.OPT-125M is the ReLU model in this set, whose units are silent on the whole negative half-line.Source: §A.4 · p. 125

B The set-operator model and how it is trained

  • The set-operator construction draws on three earlier papers [39] [40] [42], with its every-unit, every-layer ingredient from a fourth [41].This appendix gathers the unit, attention head, training pressure, shaping terms, and recipe; its contribution beyond those papers is their assembly.Source: B The set-operator model and how it is… · p. 146

B.1 The feed-forward unit

  • A set-operator unit reads two rows and writes two columns, computing intersection and difference over two fuzzy memberships.Memberships lie in [0, 1] so complement expresses negation; product t-norms are used because they train while lattice operations saturate. One operand gates the contribution, while the other selects between the two write columns.Source: §B.1 · pp. 146–147
  • With 1,536 operand pairs, every set-operator layer matches the baseline’s parameter count, but every unit uses the new construction and none retain conventional units as a gradient path.The write projection remains the same 3072 × d matrix as the baseline, split between intersection and difference columns; earlier versions used conventional units alongside the construction because its pure form did not train.Source: §B.1 · p. 147

B.2 The attention head

  • Each attention head reinterprets its 64 value channels as 32 intersections and 32 differences, leaving queries, keys, positional softmax, output projection, and parameter count unchanged.Attention therefore averages bounded quantities; the baseline configuration is twelve layers of width 768 with twelve heads, learned positional embeddings, pre-norm blocks, no biases, and tied input/output embeddings, apart from one learned scalar per layer.Source: §B.2 · p. 147

B.3 The pressure … C.1 The models can rig the comparison

These sections explain how an auxiliary schedule makes the set-operator arm trainable, describe its shaping terms and recipe, and document ways model quality and comparison choices can distort legibility results.

B.3 The pressure

  • The schedule asks the first half of the stack to keep layer decodes orthogonal to the final prediction and the second half to point toward it.Its target angle steps over depth; gradients can move either the layer decode or final prediction. The loss uses 128 shared random positions per batch, α = 3, and is switched off at evaluation; it adds no parameters or forward-pass operations.Source: §B.3 · p. 148
  • All eight set-operator seeds converged within 0.2 nats of one another and within the baseline’s converged quality range, unlike the conventional model’s loss-only training lottery.The comparison is with earlier papers’ runs, not a paired ablation: no set-operator run without the schedule is reported. Oskin [41] found six of eight conventional-model seeds diverged outright under the loss alone.Source: §B.3 · p. 148

B.4 Two shaping terms

  • Two auxiliary terms, each at coefficient 0.003, encourage operands toward 0 or 1 and penalize operand variance across batch tokens below τ.Both apply to feed-forward operands and attention memberships. The crispness term requires bounded activations and is zero for softplus and ReLU; sigmoid models trained with and without it agree on the measurements supporting that comparison, but differ on context delivery over depth.Source: §B.4 · pp. 148–149

B.5 The recipe

  • The recipe keeps the baseline’s learning rate and warmup unchanged; a unit is read through operand A and written through two columns, with edits landing on one branch.All eight seeds passed the screening. Operand B is also nameable at roughly twice the baseline’s single-row rate, and the second operand is unique to this construction.Source: §B.5 · p. 149

C Ash heap

  • The appendix groups failed approaches into five families: rigged comparisons, biased estimators, arbitrary cuts and summaries, failed ablation searches, and standard reading methods that fail on conventional units.It also examines instrument-caused negative results, conditions for the reported numbers, fitting drivers upstream, reading write columns against the vocabulary, naming units by where they fire, and tests of unnameable remainders.Source: C Ash heap · p. 149

C.1 The models can rig the comparison

  • Every excluded seed scored 40.7–49.2% on ov@8 versus 33.6–38.4% for converged seeds, despite carrying only 30.3–45.7 versus 98.4–106.7 effective directions.Two excluded seeds collapsed to 2.2 and 1.1 directions at layer 1, out of 768; including one would shrink the later reported gap by about a third, so quality is reported beside every legibility figure.Source: §C.1 · pp. 149–150
  • Dropping units with no clear trigger lifts the baseline by about six points, so this fairness correction rewards whichever model has more such units; the uncorrected number is reported.Units without a clear trigger have noisy causal rankings and score low, but removing them can favor one model over another.Source: §C.1 · p. 150

C.2 The estimators are biased

Equation 45’s dimensionality estimate is biased in opposite directions by isotropic noise and sample-eigenspectrum dispersion; cross-covariance measurement removes the ridge with little change. The reported estimates and naming results also depend on context-table size and reproducibility.

C.2 The estimators are biased

  • Equation 45 overstates dimensionality through an isotropic ridge but understates it through over-dispersed sample eigenspectra.The ridge varies with within-token contextual variance, which differs seven hundredfold across seeds and requires the largest correction where the measurement is used to argue for better models.Source: §C.2 · p. 150
  • Less than one unit of PR separates every seed after cross-covariance measurement removes the isotropic ridge.The method splits contexts into disjoint halves and uses their symmetrized cross-covariance; independent noise contributes nothing in expectation.Source: §C.2 · p. 150
  • Baseline ov@8 rises from 34.6 to 40.2 percent as the context table grows from 4 to 32 contexts.Because the estimand is a model property, larger tables improve its estimate; absolute figures should state the table size.Source: §C.2 · pp. 150–151
  • Evenly splitting contexts between table construction and testing halves instrument quality unnecessarily; disjoint sets avoid overlap without that hygiene cost.The overlap flag also doubles the table used to build the naming side, while every reported number uses disjoint sets.Source: §C.2 · p. 151
  • At sixteen contexts, overlapping versus disjoint sets yield 39.2 versus 38.9 under the ranked summary and 59.7 versus 59.5 under the mean.The leakage is about a third of a point; the paper uses disjoint sets throughout.Source: §C.2 · p. 151
  • At K = 8, a baseline unit’s top-K list reproduces itself 64 percent at eight contexts per half and 86 percent at ninety-six.Agreement between procedures should be interpreted against this reproducibility floor; several large-looking disagreements fall within the ground truth’s own variation.Source: §C.2 · p. 151

C.3 The cuts and summaries are arbitrary · C.4 Which summary the causal set is taken under

Causal token sets depend on the cutoff and summary used: fixed cuts lose resolution, and defensible summaries can disagree. The disagreement is smaller for bounded models than for the baseline, while threshold choices can also hide or distort signal.

C.3 The cuts and summaries are arbitrary

  • A baseline unit’s effective set size is a participation ratio; its causal response spans a median 115 of 512 candidates (quartiles 82 and 148), so the paper reports c ov@K over a sweep.In Figure 95, the top eight capture 18% of total response and the top thirty-two capture 42%; rank-K membership is close to a coin flip at the boundary.Source: §C.3 · p. 151, Figure 95 · p. 152
  • Four threshold choices failed: midpoint activation ranges measured silence, percentile ranges discarded selective-unit signal, quantile bins misplaced resolution, and a three-standard-deviation rule lost half of symmetric evidence.The proposed defense is to inspect distributions before choosing cutoffs and require checks such as quantization error falling as bins are added.Source: §C.3 · p. 151
  • Two summaries of the same measurements move baseline ov@8 from 34.6% to 52.4% and change the curve over K as well as its level.Averaging activation across contexts measures strongest response; ranking tokens within each context and averaging ranks measures most reliable response.Source: §C.3 · p. 152
  • Across six baseline summary statistics, four magnitude-based measures agree, while the single ordering-based measure is an outlier.The two-set disagreement alone does not establish that a unit has two unique token sets.Source: §C.3 · p. 152

C.4 Which summary the causal set is taken under

  • Mean activation across contexts selects tokens driving a unit most strongly, while averaging within-context ranks selects those driving it most consistently.The mean summary is used for the paper’s per-seed tables; a unit’s linear write means a token that spikes once still contributes on that occasion.Source: §C.4 · p. 152
  • At K = 8, mean-to-ranked agreement falls 17.8 points on baseline and 1.7 on set-operator; their top-8 lists overlap on 63% of baseline and 75–79% of bounded-model units, versus 1.5% chance.The overlap compares the conventions’ own top-8 lists, without decoding.Source: §C.4 · pp. 152–153
  • The architecture legibility ratio shifts from about twice to about a third depending on the summary convention, so it is not portable across architectures or unstated conventions.The shift is architecture-specific, and the paper therefore does not frame claims using that ratio.Source: §C.4 · p. 153
  • Under ranked summaries, strictly monotone units match their ceiling by construction; under mean summaries, baseline reaches 0.81 of its ceiling and sigmoid reaches 0.98.Set-operator and sigmoid models return identical ceiling and realized figures under the ranked summary.Source: §C.4 · p. 153

C.5 How the standard reading methods fail on a conventional unit

Max-activating rankings can give every unit a plausible name while the underlying inputs remain non-distinctive; attempts to align layer frames sharply harm quality, and fitted autoencoders add a second model to interpret.

C.5 How the standard reading methods fail on a conventional unit

  • Across two seeds, max-activating ranking gives 36,864 baseline units distinct-looking answers, yet winners span all 512 candidates and the most popular accounts for only 1.1% and 2.8% of units.The top K = 8 inputs agree 59% above chance, but reading only that part misses the grab-bag of inputs behind each apparent name.Source: §C.5 · p. 153
  • The concentration score gives a nearly rank-one model 16.9 times its null, but a distinct-name metric yields nine names across 1,536 units and scores below random null.When most units share one dominant component, high concentration can coexist with individually meaningless names, so distinctness must be measured alongside concentration.Source: §C.5 · p. 153
  • At the strongest auxiliary alignment setting, LAMBADA quality falls by up to 48%; learned orthogonal transforms worsen perplexity by roughly an order of magnitude and increase measured rotation [41].Auxiliary alignment saturates near fourteen degrees, while the only cost-free setting leaves the frame essentially where it started.Source: §C.5 · pp. 153, 155
  • A sparse autoencoder recovers nameable features with good reconstruction and few dead units, but it is a separately trained explainer that adds another model to understand [12].A hand-tuned ℓ1 penalty activated more than 200 of 256 units; fixing the active count with top-k activation avoided blind coefficient search.Source: §C.5 · p. 155

C.6 The ablation searches fail their own checks … C.12 Naming a unit by where it fires

The appendices show where the measurement instruments fail, why some apparent failures reflect measurement rather than model structure, and what can and cannot be inferred about component contributions and inputs. They also distinguish naming a component by the contexts where it fires from identifying what it reads.

C.6 The ablation searches fail their own checks

  • Bisection’s monotonicity assumption fails on 11–28% of predictions, and when it misses the true minimum it overshoots by up to 25 components on Llama-3.2-1B.The dependency graph has heavily shared sources, so ablating a larger prefix changes the remaining components’ computations; keeping a pruned graph preserves predictions only 0–77% of the time, and often less as more is kept. Keeping subsets also evaluates the model off-distribution, so sufficiency cannot be established by enlarging a set.Source: §C.6 · pp. 155–156

C.7 Whether the instrument or the model is at fault

  • Three apparent failures in Section 3 trace to the instrument, while the fourth reflects genuinely non-lexical model structure.Scoring edges in one frame gives correlation −0.02 against a random control, versus +0.28 in one frame; fitting unit drivers gives negative held-out variance despite reproducible profiles, and attention’s same-position probe misses information assembled from other positions. The remaining case is a component operating on a direction no token induces, so it has no vocabulary name.Source: §C.7 · p. 157

C.8 The opposing side of the accounting

  • A losing token’s negative logit mass matches control tokens, rather than indicating a token-specific “not this” signal.Across models, its negative-to-positive mass ratio is within 0.02 of the competitor’s; the share of components needed to reach 90% of negative mass is within a point of the shuffled-sign control. About three quarters of baseline mass on both sides comes from position-mean activations, with the bounded arms’ mass effectively all constant.Source: §C.8 · p. 158, Table 50 · p. 159

C.9 Conditions for trusting these numbers

  • Section 3 measurements can return confident wrong answers when conditions are dropped; Table 51 pairs each condition with the number that changes without it.The passage emphasizes that these conditions distinguish the two outcomes rather than merely reflecting a preference about measurement.Source: §C.9 · p. 158, Table 51 · p. 160

C.10 Fitting a unit’s drivers from its upstream components

  • All four fitted-driver pools produce negative held-out variance, with the strongest reaching −0.306 versus −0.009 for the carried embedding alone.Selecting three atoms from 1,440 costs about 29 bits against 30 observations, consuming the sample before coefficients are estimated. Yet profiles reproduce across context halves at 0.648 (CI [0.643, 0.654], n = 14,123), and a ranking that selects and fits nothing recovers the target.Source: §C.10 · p. 159

C.11 Reading a write column against the vocabulary

  • A write column alone cannot identify its credited token: retrieval from column geometry reaches only 1.23 times chance on the baseline and 1.35 on the sigmoid model.On the baseline, credited tokens lie at the column’s 86th percentile and a third are within its top 1%, but firing carries 52% of contribution variance, leaving context to choose the token. Other tests find write-column alignment with individual unembedding rows at chance through two thirds of each model, and morphology axes at chance per column.Source: §C.11 · pp. 161–162

C.12 Naming a unit by where it fires

  • Naming a unit by its firing-position token describes where it activates, not what it reads, because the seed token rapidly disappears from the residual.Agreement is three times stronger at layer zero than at layer eleven and declines throughout depth; sweeping earlier-token offsets cannot solve the problem because the relevant token may lie hundreds of positions back. Attention uses few columns, but only the first block shows a large firing-weighted readability gap; at the last layer, baseline heads straddle chance while every other model is above it.Source: §C.12 · p. 162, Table 54 · p. 162

C.13 What the tokens outside the concept are … D.3 A pruned graph is still the reason

Most unit response outside embedding-space concepts forms a reproducible remainder, while the dependency graph traces upstream causes and remains consequential after pruning. Together, these sections characterize what the embedding misses and how the model’s components contribute to predictions.

C.13 What the tokens outside the concept are

  • Across four models, 61–72% of unit response lies outside embedding-space concepts; that remainder is reproducible for most units, single-direction, and no more context-bound than coherent tokens.The remainder’s half-context rank correlations are 0.52–0.64; 11–20% of units fall below 0.2, while no unit shows more than one direction above the null.Source: §C.13 · p. 163
  • Carriage and untrained-read accounts fail: write-column carriage correlates with nameability at +0.14 and +0.22, while nameability is flat across read-row training movement.No read row among 36,864 is static between checkpoints; median cosine is 0.58 and 0.64, and read-row carriage lift does not predict nameability (r = +0.007 and +0.177).Source: §C.13 · pp. 163–164

D The dependency graph

  • The dependency graph supplies causal edges, and multiplying the one-hop count by branching overstates the measured two-level closure by 1.6–9.8× across eighteen models.The median overstatement is 3.6×; removing a node’s eight highest-ranked sources leaves 0.34 of its activation, versus 1.0004 after removing eight random sources.Source: D The dependency graph · p. 164, §D.1 · pp. 165–166

D.1 Edges, branching, and the closure

  • The one-hop set exactly accounts for the logit, while the backward closure recursively follows 90% of each node’s incoming drive; attention is counted and ablated but not followed across positions.The graph is acyclic and ordered by depth; between one-third and three-fifths of a closure is attention.Source: §D.1 · pp. 165–166
  • The closure grows from 53 to 336 to 736 components over six levels on the baseline, ending at 1.6% of model components; every model closes within four to ten levels.The closure is three to six times the one-hop set on shaped arms, fourteen times on the baseline, and twelve to sixty times on models trained elsewhere.Source: §D.1 · p. 165

D.2 Aggregating influence along paths

  • For signed, locally linear contributions, path sums handle overlapping paths, but maximum flow does not transfer because contributions are signed and bottleneck capacity measures routing, not delivery.Abnar and Zuidema [1] use maximum flow to aggregate attention across layers; here the reported prefix ranking is checked by ablation and stated as a bound, not a minimum.Source: §D.2 · pp. 166, 168

D.3 A pruned graph is still the reason

  • 56 components—about one-thousandth of the model—still change the prediction 95% of the time; pruning to a fifth changes every prediction tested.The table describes pruning by lowering the per-node threshold θ and notes that rows use the same paired predictions.Source: Table 56 · p. 168, §D.3 · p. 168

D.4 Why sufficiency needs a search rather than a closure · D.5 What the replacement choice does to a keep-only number

The appendix finds that closure is not a reliable substitute for searching when measuring sufficiency: its results depend on sample size and model structure, and keeping more of a graph does not necessarily improve preservation. It also shows that replacement choices barely affect necessity measurements but can sharply change keep-only sufficiency results.

D.4 Why sufficiency needs a search rather than a closure

  • The sufficiency test must use a graph-based search because replacing everything outside a one-hop set also removes the drive that makes its members fire.The test keeps a set, replaces every other component, and asks whether the same token still comes out; this is a fair test for closure, but not for the one-hop set.Source: §D.4 · p. 168
  • Two controls test whether recursion helps: a size-matched random set and the top-|C| components ranked by direct contribution.The controls ask whether closure beats an arbitrary set of the same size and whether it beats simply taking more of the one-hop ranking.Source: §D.4 · p. 168
  • Raising the kept set from none to 2,048 components increases preservation by 63 points, validating that the keep-only test can discriminate.The text says the closure’s advantage over the controls is a real difference rather than noise around a ceiling.Source: §D.4 · pp. 168–169
  • At 150 predictions, closure preserves less often than the size-matched set on the baseline (60% vs. 65%), sigmoid model (63% vs. 78%), and GPT-2 (65% vs. 87%).At 28 predictions, closure leads 54% to 32%, but the 95% interval is ±19 points at n = 28 versus ±8 at n = 150. The value of recursion also depends on branching factor; when it is small, closure adds little beyond the one-hop set.Source: §D.4 · p. 168, Table 57 · p. 169
  • Tightening recursion from threshold 0.9 to 0.99 and beyond grows the baseline closure from 640 to 786 components while preservation falls from 59% to 56%.Across tested models, preservation stays flat or slightly lower as the set grows; the authors conclude that ancestry is the wrong currency because closure spends its budget on ancestry rather than components that write toward the prediction.Source: §D.4 · p. 169
  • Pruned-graph preservation peaks at θ = 0.7 and falls at θ = 0.9 despite retaining strictly more of the graph.Appendix C.6 reports this non-monotonicity as a property of the instrument, not just this model.Source: §D.4 · p. 169
  • The necessity measurements do not show sufficiency: the separate search finds a different eight-component set, two orders of magnitude smaller than closure.Neither necessity number establishes that its components can produce the answer alone; Section 2.5’s search addresses reproduction.Source: §D.4 · p. 169

D.5 What the replacement choice does to a keep-only number

  • Replacement choice shifts necessity by at most seven points, but keep-only sufficiency preserves predictions in 33% of cases with zero replacement and 3% with resampling.Zero replacement is the most flattering yet least informative: it removes opposing logit mass along with other components, so the target logit can exceed the intact model’s. The three necessity replacements are zero, mean output, and resampling; they agree within seven points.Source: §D.5 · p. 169, Table 58 · p. 170

Related papers

Loading 2609.10210v1…