PaperlayerView exact revision on arXiv

Source-linked AI summary

Through the Looking Glass: Directly Reading and Writing Transformers

Mark Oskin

arXiv:2609.10210v1cs.CLcs.LG

TL;DR

The paper asks how to identify and understand the components responsible for a transformer’s individual predictions without relying on a separately trained explainer. It derives a lens from the model’s own parameters and activations, then finds small sufficient circuits, names components and their inputs, and applies those names to targeted edits. Across the baseline and twelve externally trained models, the resulting circuits are far smaller than conventional attribution counts, while the lens supports both tracing and editing.

  • Problem

    Published component counts attribute single behaviors to thousands or tens of thousands of units, leaving it unclear which components actually decide an individual prediction.

  • Method

    The paper derives an untrained lens from a transformer’s own parameters and activations to identify prediction-carrying, necessary, and sufficient components and read their upstream inputs.

  • Results

    Sufficient prediction circuits contain 8 components on the baseline and 2–16 across twelve other models, far below conventional counts.

  • Takeaways & Limitations

    The lens makes transformer predictions traceable to small component circuits and supports naming and acting on components without fitting a second model.

  • Takeaways & Limitations

    Reported circuit sizes are upper bounds from heuristic searches, so the minimal sufficient circuit remains unknown and search scaling is unresolved for larger models.

Abstract

from arXiv · show

How many of a transformer's components decide a token? Counted by the absolute value of each unit's and channel's contribution to the logit, one prediction rests on thousands to hundreds of thousands of them. But contributions are signed, and across eighteen models the mass pushing away from the predicted token is a median of seven times the mass carrying it. Divide by the net and the count is dozens: on the baseline, 53 components carry ninety percent of a prediction, 13 it cannot survive losing, and 8 suffice to produce it alone. Across twelve models trained elsewhere, 124M to 7B parameters, the sufficient set runs from two components to sixteen, and what a prediction draws on, followed all the way back, is one to three percent of the model, a share that does not grow with size. Three quarters of a layer's update is a fixed linear map of the state it received. Everything is read from the model's own parameters and activations, with nothing trained or fitted, and it names a component on both sides: what it writes, from the predictions it drives, reaching close to half of every model; what it reads, from its weights in the frame of its own layer, at 58.9 percent above chance over its eight strongest inputs. Sorting the remainder by upstream source yields grammatical categories the embedding cannot see. A name can be acted on. An association the model does not hold installs into one spare unit, key and value read from the weights, for a quarter of a percent of held-out loss, a fortieth of what a rank-one update costs. An installed attention head and a unit two layers above it make an edit fire only where a token occurred earlier in the context, and a unit the model trained for itself is driven from two layers upstream, 86 percent of the effect passing through it. An order-preserving activation puts a unit's inputs at the instrument's ceiling, at the price of a two-part install.

1 Introduction

The paper presents a parameter- and activation-derived lens showing that individual predictions depend on small, traceable circuits rather than the thousands of components identified by absolute attribution. It also uses the lens to name components, edit them, and trace their upstream connections without training a separate explainer.

  • Motivation: Thousands to hundreds of thousands of components appear responsible under absolute attribution, but signed cancellation reduces a prediction’s relevant count to dozens.The opposing mass is a median of seven times the mass carrying the prediction across eighteen models.
  • Prediction circuits: 53 components carry 90% of a baseline prediction, 13 are necessary, and 8 suffice to reproduce the same token.The sufficient set is tested by retaining selected units and attention channels while zeroing the rest at that position.
  • Prediction circuits: Three quarters of a layer’s update is a fixed linear map of the received state, applied independently of the model’s upcoming output.This work is distributed across components as workspace management rather than concentrated prediction computation.
  • Legibility: The lens names what components write from the predictions they drive and reads their strongest inputs from weights in the layer’s own frame.Weight-based recovery identifies a unit’s eight strongest drivers at 58.9 percent, or thirty-seven times chance; close to half of model components can be named by their driven predictions.
  • Legibility: Sorting unnamed upstream sources separates grammatical categories that the embedding cannot distinguish.The remaining three quarters of a unit’s drives are concepts built by the model rather than categories visible in the vocabulary.
  • Editability: A missing association can be installed into one spare component using weights-derived key and value vectors, moving the target from rank 578 to rank 1 for 0.25% of held-out loss.This costs a fortieth of the rank-one update needed to reach the top ten.
  • Editability: An installed attention head and a unit two layers above it can make an edit fire only where an earlier token occurred, while a trained unit can be driven from two layers upstream with 86% of the effect passing through it.The paper also examines order-preserving activations that improve input readability but require a two-part install because they are never silent.

2 Reading a prediction: The Looking Glass

The paper reads predictions through signed attribution, ablation, and sufficient-circuit searches, finding that small, context-dependent graphs matter despite extensive distributed computation. Its lens uses the model’s own parameters and activations, but sufficient-circuit sizes remain heuristic upper bounds.

  • Reading components: 58.9 percent agreement over a unit’s eight strongest inputs shows that layer-native weights can recover what components read.The lens decodes inputs in each component’s own layer frame rather than through the final vocabulary.
  • Identifying components that matter: Seven times the net is the median opposing contribution mass, so signed normalization reduces apparent prediction support from thousands to dozens of components.The comparison uses the same predictions and attribution while changing the denominator.
  • What part is necessary: 42 to 50 percent of predicted logit loss is realized after removal because surviving components recompute and recover the remainder.This measures logit mass, not whether the prediction changes.
  • What part is sufficient: 85 percent of predictions reach a sufficient set from a random start versus 89 percent from the required graph, with the same median size of six on that draw.The random-start procedure requires about half as much work again.
  • Limitations: Every reported sufficient-circuit size is a heuristic upper bound, and larger models can exhaust a substantially larger search budget without converging.The minimal sufficient circuit remains open, making scaling the search a separate problem.
  • What part is sufficient: Two thirds to three quarters of surviving members are feed-forward units, with attention channels providing the only route for information from outside the position.The surviving sets span about five layers and form graphs in which many feed-forward units draw on other selected members.

3 What’s in a Name?

The paper names components from both sides: what they write is inferred from the predictions they drive, while what they read is decoded from their weights and checked against behavior. These names reveal grammatical, often embedding-invisible categories and show that naming becomes more available deeper in the model.

  • What a unit writes: Only a tenth of one percent of components write a single token; their token sets are usually grammatical categories rather than semantic topics.Examples include negated auxiliaries, first names, units of measurement, and sentence-openers.
  • Depth and breadth: Characterized feed-forward units rise from one or two percent in early layers to ninety-five percent in the last, while attention lags by about two layers.Coherence remains between eighty and ninety-nine percent across depth, so depth changes how many components can be named rather than how well.
  • What a unit writes: The predicted token appears in the top hundred of two thirds of weighted component contributions and in the top thousand of ninety-three percent, so components usually aim near the answer.Components promote related tokens and the sum selects the final result rather than any one component selecting it alone.
  • What a unit reads: 58.9 percent agreement recovers a unit’s eight strongest drivers from its parameters, or thirty-seven times chance.The agreement is strongest at the head of the ranking and falls below random selection by 256 requested tokens.
  • What a unit reads: Nine sets in ten are corpus-coherent, but only one in three is coherent in the token embedding, exposing grammatical categories the vocabulary does not group.Embedding-based concept tests and corpus-based naming therefore measure different kinds of coherence.
  • Naming attention channels: About a third of channels and feed-forward units are characterized at matched budget, with coherent sets differing by only a few points.The activation function makes unit naming harder, whereas a channel’s linear value path is easier to decode.

4 Editing a Model: Say what?

The paper turns component names into interventions: it identifies what components write and read, then tests whether targeted edits are strong, local, and architecturally placeable. Feed-forward units are easier to edit than attention channels, while successful cross-layer edits depend on depth and the frame used to carry information.

  • Editing components: Three edit types test amplification, installing an absent association, and driving a trained unit from two layers below.The section also compares the installation against a rank-one editor.
  • Editing components: 97% of edits move the parameters’ named tokens while leaving other components’ targets alone, with more than tenfold greater effect on the named scope.The evaluation controls for other edited components and vocabulary-wide distribution shifts.
  • Editing components: Doubling α increases named-scope movement from 0.30 to 0.89 to 1.77 while other scopes remain near zero.The sweep shows strength changes the intended effect without comparable collateral movement.
  • Installing associations: A single spare unit moves B from rank 578 to rank 1 after A, preserves its elsewhere rank, and costs 0.25% of held-out loss.Neither the key nor value is fitted, traced, or optimized; the result is a fortieth of the rank-one editor’s cost to reach the top ten.
  • Placing edits: Readout-frame writes become increasingly visible near commitment layers, rising from 0.27 to 0.43, while depth dominates preservation and spill falls from 0.40 to 0.04.Late writes are more preserved and localized; the last layer consumes readout-frame content, so constructions place readers at layer ten of twelve.

5 What the activation function changes

Replacing GELU with order-preserving activations makes parameter-derived inputs broadly readable and produces smaller, more localized circuits, while ReLU sacrifices quality by discarding information. The gain comes from preserving the ordering supplied by the weights, but bounded units complicate local editing.

  • Readability: 96 to 99 percent: order-preserving activations read flat across K, whereas GELU reads at 67 percent and decays.The baseline falls from 58.9 percent at K = 8 to −9.5 at K = 256, while sigmoid, softplus, and the set-operator arm rise or remain high.
  • Readability: 59.8 to 62.0 percent of pre-activation mass lies below GELU’s x∗≈−0.752, where larger weighted inputs produce smaller activations.This reversal explains why GELU’s strongest inputs are readable but its broader input set becomes incoherent.
  • Caveats: Order-preserving activations do not guarantee training generalization: the reading ablation does not establish that another activation’s trained pre-activation distribution will share the property.The reported predictor is the pre-activation mean rather than w_u · Ẽ, placing these numbers on a different scale from the rest.
  • Quality: ReLU trains stably but sits about ten percent behind other arms on perplexity and below every converged seed on accuracy.The injective sigmoid and softplus arms remain inside the baseline’s converged quality range, unlike ReLU.
  • Circuit structure: 26 to 36 components versus 53: order-preserving arms produce smaller one-hop circuits, and closure separates them from baseline by four to seven times.The arms also require 3 to 5 indispensable components rather than the baseline’s 13, while closure reaches 99 to 182 components depending on the arm.

6 Models trained by other people

Applying the lens unchanged to twelve externally trained models shows that prediction-specific circuits remain small, concentrated, and traceable across architectures and scales. The results also expose boundaries: search convergence is incomplete, and uncontrolled model differences prevent attributing branching-factor variation to activation functions alone.

  • Cross-model concentration: 124 million to 7 billion parameters: every model reaches 90% of a prediction with a few dozen to a few hundred components, versus two to three orders of magnitude more under the common count.This gap appears across seven independent transformer implementations.
  • Circuit structure: 28 to 76% of sufficient-set members lie in the last quarter of the stack, while 79 to 100% of distinct components appear in exactly one prediction set.Attention contributes 22 to 40% of one-hop sets, so feed-forward-only traces omit substantial machinery.
  • Whole dependency graphs: Following dependencies backward closes on a few percent of each model, and that fraction does not grow with size.The required graph is much smaller than the closure; removing it usually changes the prediction, unlike removing a size-matched random set.
  • Scope boundary: Branching factors range from 16 to 119 across conventional activations, with overlapping intervals that do not support inferring the value from activation function alone.The controlled comparison isolates activation effects, but these stock models vary in architecture and training.
  • Sufficient sets: Two to twenty-one components suffice across models, with Mistral 7B and Qwen2.5 7B requiring two each, or three parts per million of the model.The largest models occupy the small end of the observed range.
  • Search limits: The search fails to converge on some predictions, which are excluded from downstream measurements; random same-sized sets never reproduce a prediction, while contribution rankings reach 0 to 22%.The fixed round limit means incompletely searched predictions provide no sufficient set to report.

7 Related work

The paper combines established ideas from component reading, prediction decomposition, backward graph construction, and model editing, while changing the accounting and applying it at single-component resolution. Its claimed novelty is the conjunction of weight-based site selection, weight-based writing, and intervention-based validation.

  • Circuit discovery: Backward prediction-specific graphs already exist, but this work uses single units and attention channels rather than whole operations or token representations.The comparison also distinguishes signed edges from non-negative proximity or flow scores.
  • Prediction decomposition: The paper’s accounting preserves signed component contributions, unlike non-negative importance conventions that can change circuit-size conclusions by orders of magnitude.Its graph measures single components, signed edges, closure size, and what survives removal.
  • Basis choice: The neuron basis remains unsettled: one line reports worse interpretability than learned dictionaries, while another finds sparse neuron circuits comparable to feature-basis circuits.These conflicting findings motivate treating basis choice as an open question.
  • Evaluation choices: The paper chooses counterfactual baselines carefully because replacement values can alter sufficiency severalfold, whereas necessity is nearly unchanged across the tested counterfactuals.Accordingly, the paper frames its claims primarily as necessity claims.
  • Editing a component: Prior work reads component functions from weights and edits components, while this paper joins those lines through weight-based site selection, association installation, and behavioral scoring.The edit can therefore fail a criterion rather than merely produce a target change.
  • Evaluation caveats: Related critiques caution that causal-tracing locations may predict editing success poorly and that headline edit success can fall under deployment-like evaluation conditions.The paper’s own edit results are measured on single edits and inherit this evaluation caution.

8 Discussion

The discussion presents the lens as both a measurement tool for prediction-specific reading and a route toward targeted writing, while emphasizing structural limits and open design questions. It argues that most layer computation is carriage rather than decision-making, yet that carriage remains functionally entangled with content.

  • Reading: A prediction-specific circuit turns model failures and capabilities into auditable component-level measurements with an error bar.The paper also frames circuits as comparable across architectures when traced for individual tokens.
  • Writing: An edit to a named component can change a stated behavior and check its collateral, unlike fine-tuning, which accepts whatever internal change follows from showing data.The distinction is operational: the edit is evaluated against the behavior it was intended to install.
  • Writing: Edit placement is architectural: writes can use the model’s own frame, or require both ends of a circuit in an otherwise unused frame.The available unused space and how it changes with scale remain open questions.
  • Control: Driving a trained component from two layers upstream turns a weight-based legibility claim into a control surface, limited chiefly by read-row estimation accuracy.The discussion suggests better instruments may improve the construction without changing the architecture.
  • Architectural constraints: Attention cannot receive a locally installed association because its output is gathered from other positions, but it can carry conditions for downstream units.This structural constraint explains why installed content is directed mainly to feed-forward layers while attention remains useful for markers and existing-scope gains.
  • Scope limits: The proposed programming direction remains limited to single components and single predictions, while vocabulary and architecture implications rest on small or incomplete measurements.For example, the comma-unit observation is explicitly described as an anecdote rather than a measurement.
  • Carriage and content: Most contribution mass cancels, leaving a few dozen components to carry the prediction while the remainder organizes information in the residual stream.The discussion treats this carriage as neither idle nor separable from content: a fixed operator reproduces 74 to 79% of a layer update, but prediction lies inside its acted-on span.

9 Conclusion

The paper presents a parameter- and activation-derived lens that identifies, traces, and edits the small component sets behind individual transformer predictions. It argues that finding, naming, and editing these components form a practical toolkit for understanding and modifying transformers.

  • 53 components carry 90% of a baseline prediction, 13 are necessary, and 8 suffice to reproduce it.The counts are reported for the baseline model’s 46,080 components.
  • Three quarters of a layer’s update is a fixed map of its incoming state, while opposing logit mass is seven times the mass supporting the prediction.These effects help distinguish prediction-relevant computation from architectural workspace management.
  • Close to half of each model’s components can be named from the predictions they drive, while upstream sources provide provenance for vocabulary-invisible computations.The dependency graph separates unnamed internal quantities into categories by their upstream writers.
  • Named components can be edited by installing associations, driving trained units, or constructing circuits from components and channels.The paper also tests activation-function replacement, which improves input legibility but makes an install a two-part edit.
  • Finding, naming, and editing together provide a toolkit and roadmap for building, training, and using transformers.

A Gallery

The gallery presents complete prediction circuits as tables and graphs, showing every sufficient-set component, its strongest suppliers, and its relation to the predicted token. The displays preserve both readable token-level links and source tags for concepts without vocabulary descriptions.

  • Each gallery page shows the full sufficient set for a prediction, with every component represented in the table and graph.Nothing is trimmed to fit, even when a set occupies a page and its graph is rotated.
  • The header reports the context, predicted token, prediction probability, 90%-mass count, necessary-component count, and cancellation share.
  • Each row identifies a component’s strength, drivers, promoted tokens, answer rank, and strongest suppliers, allowing chains to be read from prediction back through upstream sources.The graph preserves every edge, not only the three strongest suppliers shown under each row.
  • Attention-channel rows measure reads from attended positions through value projections, while their attention patterns remain outside this lens.
  • Dashed concept edges identify internal quantities whose writer and reader share no highly ranked token, tagging them by source to expose reuse.A concept edge occurs when the writer and reader share no token among the candidate lists used for the display.
  • The examples are mechanically sampled and typically contain no vocabulary-invisible concepts, while the gallery’s ordering and model coverage are fixed by the sampling procedure.The six models trained for the paper appear before twelve externally trained models, ordered by size.

A.3 What the conventions rest on

The gallery illustrates how conventional component counts and traces can be misleading because signed contributions cancel and many individual edges carry weak evidence. Its figures therefore show circuit structure without treating every displayed edge or selected example as a strong semantic claim.

  • Concept edges occur on a median of 2% of edges per model, but OPT-125M reaches 65%, making it an outlier.Across all eighteen models, the pooled figure is 11% and is almost entirely driven by OPT-125M.
  • 13% of baseline edges and 11% of sigmoid edges beat twenty random controls at p ≤0.05, while the median edge has p ≈0.35.Across 23,000 edges, real writers win 63% against 32%, but most individual edges remain weakly informative.
  • The displayed traces use signed contribution accounting, so component shares can exceed 100% when other sources cancel against the net drive.This accounting is distinct from simply counting absolute contribution mass.

B.3 The pressure

The paper’s pressure experiments expose how training objectives and representation quality affect apparent legibility. A model can look easier to interpret when its representation collapses, so legibility measurements must be reported alongside model quality and method-specific caveats.

  • The pure construction did not train reliably under language-modeling loss alone, with earlier work finding divergence above roughly half conversion.An auxiliary objective prescribes layerwise geometry relative to the final prediction.
  • The auxiliary pressure trains the first half of the stack to remain orthogonal to the answer and the second half to point toward it.The objective is added to cross-entropy with α = 3, uses 128 sampled positions per batch, and is disabled at evaluation.
  • Six of eight seeds diverged under the loss-only prescription in prior work, whereas a fixed orthogonal rotation supplied the requested frame in that work’s remedy.The set-operator arm uses no such device.
  • Two auxiliary operand terms, each weighted 0.003, encourage crisp values and prevent operands from becoming constant across batch tokens.They apply to feed-forward operands and attention memberships; the crispness term defines the shaping condition for bounded activations.
  • The shaped construction changes how components are read and edited: both operand rows can be nameable, while an edit to one column affects one branch.
  • A collapsed representation can appear more legible: excluded seeds have 30.3–45.7 effective directions and score 40.7–49.2% on ov@8, versus 98.4–106.7 directions and 33.6–38.4% for converged seeds.Including one collapsed seed would reduce the later reported legibility gap by about a third.
  • Dropping units without clear triggers raises the baseline by about six points, so this proposed fairness correction favors models with more such units.

C.2 The estimators are biased

The section identifies sampling and estimator biases in dimensionality and agreement measurements, then tests corrections using independent context halves. The correction removes the main noise bias with little change, while agreement estimates remain dependent on table size and context overlap.

  • Dimensionality bias: A factor of seven hundred variation in σ2_within makes the ridge correction model-dependent rather than a shared measurement adjustment.The isotropic ridge otherwise pulls participation ratios toward the ambient dimension.
  • Dimensionality bias: Less than one unit of PR movement after cross-covariance correction shows that removing the ridge does not materially change the reported seed results.The correction uses disjoint context halves so independent noise contributes nothing in expectation.
  • Table construction: 40.2% baseline ov@8 at 32 contexts, versus 34.6% at 4, shows that agreement estimates are still rising with table size.Absolute figures therefore require the context-table size that produced them.
  • Table construction: Overlapping context sets share sampling noise between pre-activation and activation means, changing the scoring comparison and doubling the naming table.The overlap is presented as a cheap way to improve the instrument rather than splitting contexts between construction and testing.
  • Reproducibility: 64% agreement at K = 8 rises to 86% with ninety-six contexts per half, establishing a reproducibility floor for top-K lists.Procedure comparisons must be interpreted relative to this self-reproducibility.

C.3 The cuts and summaries are arbitrary

The section shows that effective response sets, thresholds, and summary statistics can make apparent legibility depend on arbitrary measurement choices. It therefore favors distributions and sweeps over single cuts, while finding that summary dependence differs across model constructions.

  • Effective set size: ν_u = 115 at the baseline median, with quartiles 82 and 148, makes a fixed K a poor proxy for causal response breadth.The causal profile spreads across roughly 115 candidate tokens, so neighboring rank boundaries are unstable.
  • Summary statistics: 34.6% versus 52.4% baseline ov@8 shows that averaging activation and averaging within-context ranks summarize different properties.The first emphasizes strongest drives; the second emphasizes consistent drives.
  • Summary statistics: 17.8 points of agreement are lost on the baseline when moving from mean to ranked summaries, compared with 1.7 on the set-operator model.The same convention change is therefore architecture-dependent.
  • Summary statistics: 63% of baseline top-8 lists and 75 to 79% of bounded-model lists overlap across conventions, against 1.5% chance.The two conventions agree substantially more for bounded units than for conventional baseline units.
  • Interpretation: A legibility ratio can move from about twice to about a third depending on an unstated convention, so cross-architecture comparisons are not portable.The section treats this as a limitation of the measurement definition rather than a model result.

C.5 How the standard reading methods fail on a conventional unit

Standard max-activation reading can produce individualized names even when a conventional unit’s underlying response set is a grab-bag. The section also shows that fitted or rotated readouts do not resolve the broader frame and representation constraints.

  • Max-activation reading: 1.1% and 2.8% are the largest shares held by any single strongest candidate across baseline units, despite every unit receiving an apparently individual name.Distinct top examples can therefore conceal broad, weakly structured response sets.
  • Max-activation reading: 16.9 times null for a nearly rank-one model can coexist with only nine names across 1536 units under a metric counting distinct names.Nameability scores depend strongly on what concentration statistic counts.
  • Frame alignment: Up to 48% LAMBADA degradation occurs before auxiliary frame-alignment pressure reaches zero-angle alignment.The learned alignment objective saturates near fourteen degrees while quality falls sharply.
  • Fitted explainers: A sparse autoencoder recovers nameable features with good reconstruction and few dead units, but it is set aside because it is a separately trained explanatory model.The paper distinguishes the method’s parameter-derived lens from the viability of fitted explainers.

C.6 The ablation searches fail their own checks

The section exposes failure modes in ablation searches, dependency tracing, and component reading when their invariants or coordinate systems do not match the model. These checks show that plausible aggregate results can conceal invalid individual explanations.

  • Ablation searches: 11 to 28% of predictions violate the monotonicity required for bisection over ablation-prefix length.Larger removals can restore a prediction that a smaller removal changed because components share upstream sources.
  • Ablation searches: 84 to 100% agreement with the true minimum still hides overestimates of up to 25 components on Llama-3.2-1B.Median tables can look correct while individual bisection results are wrong.
  • Sufficiency checks: 0 to 77% prediction preservation and non-monotone sufficiency curves show that keeping more of a pruned graph can preserve less.The failure arises because the subset ablation evaluates the model far off its training distribution.
  • Component connections: −0.02 in the reader’s frame versus +0.28 in a shared frame shows that edge scores depend on coordinate alignment.The model and edges are unchanged between the two scoring frames.
  • Attention: Attention supplies 23% of trace components but ranks same-position driver probes near chance because its outputs assemble information from other positions.The probe’s fixed position cannot order contributions that arrive from elsewhere in the context.
  • Model representation: A component operating on a direction no token induces has no vocabulary name, limiting lexical reading even when the instrument itself is correctly applied.The write column remains near chance through much of the stack, and fitted rotations recover nothing beyond no rotation.
  • Driver reconstruction: 29 bits of atom-selection cost against thirty observations explains negative held-out variance without implying that the model lacks upstream structure.The search can fit noise before estimating coefficients, while the underlying profile reproduces across contexts at 0.648.

C.13 What the tokens outside the concept are

The embedding cannot explain much of a unit’s response, but the remainder is reproducible, single, and not simply context-bound or architectural carriage. Reading upstream components turns that remainder into a traceable graph whose necessity survives substantial pruning, although closure size is not a sufficiency guarantee.

  • Unnameable response: 72 percent of the baseline response lies outside the embedding-space concept, rising to 61–72 percent across four measured models.This remainder is the portion that cosine-based embedding analysis cannot name.
  • Unnameable response: 0.52–0.64 split-half rank correlations show that the remainder’s ordering reproduces across contexts, despite being less reliable than coherent tokens at 0.59–0.75.This rules out sampling accident as a general explanation.
  • Unnameable response: 0 of 1,444 units showed more than one substructure direction above the matched random null, while 15–40 percent showed none.The result argues against the remainder being several concepts packed into each unit.
  • Unnameable response: 31–50 percent of units run opposite the context-dependence pattern, and the remainder varies slightly less across contexts than the coherent part.The remainder therefore lacks the expected signature of a relational, syntagmatic class.
  • Upstream sources: 1.26 and 1.34 times a matched random direction describe read-row alignment with carriage, but nameability correlations are only +0.007 and +0.177.Reading the carried stream does not explain the unnameable response.
  • Causal graph: 53, 336, and 736 components appear across six baseline closure levels, ending at 1.6 percent of the model; a tenth of the closure still changes 95 percent of predictions.The recursive graph is acyclic and overlapping, and necessity remains after aggressive pruning.
  • Causal graph: 1.6–9.8×, with median 3.6×, is how much a branching-factor product overstates measured two-level closure across eighteen models.Heavy overlap makes multiplication an unreliable proxy for graph size.

D.4 Why sufficiency needs a search rather than a closure

A closure is not sufficient for the keep-only test because replacing everything outside it can remove the activity that drives its members. Direct contribution ranking performs better in the reported sufficiency comparisons, so producing a prediction requires a separate search.

  • Evidence for a separate search: 54 percent versus 32 percent at 28 predictions favors the closure, but at 150 predictions direct-contribution controls lead on the baseline, sigmoid model, and GPT-2.The reported comparison reverses with more predictions: 60 versus 65, 63 versus 78, and 65 versus 87 percent, respectively.
  • Evidence for a separate search: 63 points of preservation are gained when the kept set grows from nothing to 2,048 components, establishing that the keep-more control can discriminate.A flat keep-more curve would make the sufficiency measurement uninformative.
  • Why closure fails: 59 to 56 percent preservation accompanies growth from 640 to 786 closure components as the threshold rises from 0.9 to 0.99.Tightening the recursion enlarges the set without improving preservation.
  • Why closure fails: θ = 0.7 gives the peak preservation, while θ = 0.9 retains strictly more graph but performs worse.This non-monotonicity is reported as a property of the instrument rather than only this model.
  • Evidence for a separate search: Two necessity levels are distinguished from sufficiency: the closure contains far more components than the eight-component search result that can reproduce a prediction.The paper explicitly treats the two measurements as different questions.
  • Why closure fails: 33 percent under zero replacement and 3 percent under resampling are the sufficiency rates when only the identified circuit is kept.The same necessity circuit therefore does not independently reproduce the prediction, and replacement choice moves the keep-only result substantially.
Loading 2609.10210v1…