PaperlayerView exact revision on arXiv

Source-linked AI summary

Reader--Writer Coupling in Transformer MLPs

Giyeong Oh, Sunghee Ahn, Youngjae Yu

arXiv:2610.29340v1cs.LGcs.CL

TL;DR

The paper asks whether a transformer MLP’s fixed reader–writer bindings reflect learned organization, which single-factor measurements cannot reveal. It compares the paired factors through the MLP update and its re-pairings, finding a predominantly opposed core whose sign appears early in training.

  • Problem

    Single-factor measurements cannot reveal whether an MLP’s fixed reader–writer assignment is learned or merely an index choice.

    Source: §1 · p. 1

  • Method

    The paper reads readers and writers jointly through the square MLP-update Jacobian and compares the trained assignment with re-pairings of the same factors.

    Source: §3.1 · p. 3, §3.2 · p. 4

  • Results

    The strongest pairs are predominantly opposed, and their sign appears within the first few percent of training before coupling grows.

    Source: §1 · p. 2, §5.5 · p. 8

  • Takeaways & Limitations

    Most channels return contributions against their firing readers for later-block readers to take up, while few appear in logits.

    Source: §6 · p. 9

    For gated models, the gate value can make the weight-level sign differ from the effective reading, so the paper reports elementwise models first.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

A transformer MLP gives each hidden channel two roles: a reader that decides when the channel responds, and a writer that decides what it adds back to the residual stream. Read as a key--value memory, the layer can be measured one factor set at a time: the pairing of readers to writers is carried only by their shared index, invisible to any quantity computed from one factor set alone. Yet training couples the two factors through their joint update, so we ask what reading them together shows. The block supplies the joint instrument: the MLP writes back into the very stream it reads, so the exact Jacobian of its update is square, and its agreement with its own transpose resolves into reader--writer terms whose inner products give each channel, read as a couple, a signed pairing coordinate. Ranked by this coordinate, each block we measure has a small core of strongly coupled channels; where initialization gives either sign equally, the core is nearly all opposed: each writer points against the direction its reader responds to. This opposed core recurs from the smallest models trained from scratch to released checkpoints; its sign is in place within a few percent of training and holds while the coupling grows. Read as couples, most channels return their contribution against the reader that fired it, and the readers of later blocks take it up; few show in the logits. Training holds the couples in place: flipping the opposed writers moves the output toward common tokens, and a writer permuted onto another reader regrows to fit that reader while the reader stays.

1 INTRODUCTION

Transformer MLP channels pair a reader that selects when a feature fires with a writer that determines what it adds to the residual stream. The paper measures these pairs jointly and finds that trained assignments have a distinctive, predominantly opposed coupling.

1 INTRODUCTION

  • Training leaves a learned reader–writer assignment that single-factor measurements cannot distinguish from any re-pairing of the same factors.The channel index binds the factors, and joint quantities preserve that correspondence.Source: §1 · p. 1
  • The strongest channel pairs are predominantly opposed, with writers pointing against their readers through most of the stack.The models trained here release toward both stack ends, while OLMo-1B releases only at the output.Source: Figure 1 · p. 2, §1 · p. 2
  • A square MLP-update Jacobian and its transpose yield a signed channel-level pairing coordinate from reader–writer inner products.The coordinate records whether a writer points with or against its reader.Source: §1 · p. 2
  • Across trained and released models, the opposed sign appears within the first few percent of training, before coupling grows.Most channels return contributions against their firing readers for later-block readers to take up; few appear in logits.Source: §1 · p. 2

2 RELATED WORK

Prior work studies MLP readers and writers separately, their same-index relationships, or update geometry. This paper instead treats the whole reader–writer assignment as the object of comparison.

2 RELATED WORK

  • Key–value accounts describe MLP expansion rows as context selectors and contraction columns as concept writers; editing methods update the contraction map as an associative memory.Neuron censuses also classify writers by trigger opposition, token frequency, or prediction and suppression behavior.Source: §2 · pp. 2–3
  • Prior same-index weight cosines and mismatched-pair comparisons do not test the whole assignment against its bijective re-pairings.The paper uses the operator statistic to make that whole-assignment comparison.Source: §2 · p. 3
  • Hidden-unit permutation work studies equivalent parameterizations and model matching; here, common permutations serve as a negative control.The one-sided re-pairing ensemble probes relations within a model.Source: §2 · p. 3
  • Prior local-update studies analyze symmetry or residual-stream geometry, whereas this work measures what the trained assignment contributes to update symmetry.Source: §2 · p. 3

3 PRELIMINARIES

A pre-norm MLP is represented by hidden-feature readers and writers whose indexed correspondence determines the Jacobian. The paper tests whether that correspondence is organized beyond exchangeable alternatives.

3.1 TWO FACTORS, ONE OPERATOR

  • A pre-norm MLP is a bank of hidden features, with each feature’s reader and the block Jacobian defined at the residual-stream input.The feature gradient is the reader in input coordinates, while the writer gives the output direction; the output bias is absent from the Jacobian.Source: §3.1 · p. 3
  • Commonly permuting both factor sets preserves the Jacobian, but permuting writers alone changes their correspondence and can change the operator.Separate unordered factor sets cannot reveal that correspondence.Source: §3.1 · p. 3
  • For the elementwise models studied, panel-mean activation derivatives are positive on nearly every channel, allowing weight signs to indicate the effective sign.Gated readers can differ because their panel-mean gate value may be negative when the gate rests closed.Source: §3.1 · p. 4

3.2 EXCHANGEABLE ALTERNATIVES

  • At independent, channel-exchangeable initialization, readers and writers are independently sampled, making writer re-pairings an exchangeable reference.Source: §3.2 · p. 4
  • At trained checkpoints, the test asks whether the observed assignment is uniform among re-pairings, conditional on the factor sets and inputs.Source: §3.2 · p. 4

4 MLP COUPLING

The paper measures MLP coupling through update geometry, re-pairing tests, signed channel coordinates, and vocabulary-facing channel effects. These views distinguish learned reader–writer correspondence from properties of either factor set alone.

4 MLP COUPLING

  • The coupling analysis treats the whole update, its assignment among re-pairings, each channel’s signed coordinate, and channel effects on later readers and logits.Source: §4 · p. 4

4.3 SEEN AS PAIRS

  • The signed coordinate mh is the panel-mean reader–writer inner product; negative values mark opposed channels, and the core selects the 16 largest |mh|.In three medium seeds, five-sixths of channels are opposed, and cores are more opposed and larger than the bulk.Source: §4.3 · p. 5

4.1 THE UPDATE AGAINST ITS TRANSPOSE

  • Trained updates are more symmetric than sampled writer re-pairings, and negative eigenvalues carry most symmetric-part energy except in specified stack-end blocks.The exceptions are OLMo-1B’s last block and the small model’s first block.Source: Figure 3 · p. 6

4.3 SEEN AS PAIRS

  • Figure 2’s four core examples include logit couples that move next-token logits and stream couples that affect later readers without moving them.Both kinds occur inside and outside the norm-product top sixteen.Source: Figure 2 · p. 5, §4.3 · p. 6

4.4 THE VOCABULARY LENS

  • The vocabulary lens projects a centered, final-norm-scaled writer through the unembedding, optionally removing its leading singular directions.The raw lens uses k = 0; larger k removes leading unembedding directions.Source: §4.4 · p. 6

5 EXPERIMENTS

Across trained models and released checkpoints, reader–writer assignments show excess coupling, with the strongest pairs predominantly opposed. The opposed sign appears early and remains as coupling grows.

5.1 EXPERIMENTAL SETUP

  • The experiments train pre-LayerNorm GELU decoders on FineWeb-EDU at three sizes, with three seeds, corpus controls, and released checkpoints for comparison.The setup also includes a parameter-matched SwiGLU family and multiple alternative training streams.Source: §5.1 · p. 6

5.2 LEARNED PAIRING AND UPDATE GEOMETRY

  • Every trained seed’s symmetry score exceeds sampled writer re-pairings; from medium blocks five through twenty-one, over nine tenths of S’s eigenvalue energy is negative.The trained assignment makes the spectrum one-sided, unlike re-pairing; mutually reading channel pairs account for most of the update’s pushback.Source: §5.2 · pp. 6–7

5.4 READER–WRITER PAIRING ACROSS DEPTH

  • Weight-level opposed coupling deepens toward mid-depth, then amplifying share rises in the back half; its sign matches the effective coordinate on 96–99% of small- and medium-model channels.The input blocks of OLMo-1B are an exception where the weight-level coordinate can diverge from the effective coordinate.Source: Figure 5 · p. 7, §5.4 · pp. 7–8

5.3 THE WRITER FOLLOWS ITS READER

  • After writer permutation, writers regrow toward their control pairing while readers barely turn; loss can recover before writers reach halfway to their plateau.At earlier breaks, loss returns to control even though writers stop near a cosine of one half.Source: §5.3 · p. 7, Figure 4 · p. 7

5.5 EMERGENCE OF THE FINAL CORE

  • The final top-16 core turns opposed within a few hundred steps, while coupling is only a few percent of its final size; later growth leaves the sign unchanged.This pattern is tracked across medium training states and public OLMo checkpoints.Source: Figure 6 · p. 8, §5.5 · p. 8

5.6 THE CORE SIGN ACROSS CORPORA

  • Across web, stories, news, and books, opposed cores recur; order-destroyed web and frequency-matched Zipf controls end near zero core sums and unigram-entropy loss.The specific property of ordered text associated with the sign remains open.Source: Figure 6 · p. 8, §5.6 · p. 8

5.7 THE COUPLE CENSUS

  • Seven in ten channels return contributions against their firing reader and are taken up downstream, while one in twelve moves logits beyond the census cut, mostly in the last quarter.One in thirteen is fixed by its subject; the remaining channels are nudges. The vocabulary projection describes the logit couples, while later blocks read most other contributions.Source: §5.7 · pp. 8–9, Figure 7 · p. 9

6 CONCLUSION AND DISCUSSION

The paper treats an MLP channel as a trained reader–writer couple, with a sign that organizes how contributions move through the residual stream. It identifies practical research uses for this view while leaving the cause and downstream role of the opposed sign open.

6 CONCLUSION AND DISCUSSION

  • Most channels oppose their firing reader and are taken up by later-block readers; relatively few appear in next-token logits, chiefly in the final blocks.The conclusion frames the reader–writer pair, including its sign, as the unit of analysis.Source: §6 · p. 9
  • A few percent of channels carry half a block’s squared coupling, making reader–writer pairs candidate units for pruning or editing.The paper also suggests tracking early cores across checkpoints as vocabulary content turns over.Source: §6 · p. 9
  • The work is diagnostic of existing transformer language models, not a deployment-oriented capability; its expected effect is on analysis practice.The stated analysis implication is to treat adjacent MLP factors jointly when relative assignments may be missed by marginal statistics.Source: ETHICS STATEMENT · p. 10

A NOTATION

  • Reader notation distinguishes raw rows, feature-gradient readers, effective readers, and panel means; m_h is the signed trace contribution from a reader–writer inner product.The notation also specifies vector and operator dimensions and names the within-block writer permutation.Source: A NOTATION · p. 15
  • Local C separates paired-channel and signed collective terms; the notation distinguishes seed-level loss contrast, vocabulary projection, and conditional mean response.These quantities are defined separately from the signed contribution and its top-16 core.Source: A NOTATION · p. 15

B REPRODUCIBILITY DETAILS

The appendix records the training cohorts, data and measurement protocols, then defines the Jacobian conventions and conditional reassignment test used to evaluate learned assignments.

B.1 HYPERPARAMETERS

  • Three pre-norm decoder cohorts use FineWeb-EDU for next-token prediction, with tiny ablations and small and medium runs trained to specified token budgets.The small and medium cohorts use three seeds; small is tied and medium untied.Source: §B.1 · p. 15
  • Table 1 gives cohort training hyperparameters, while Table 2 specifies streams, evaluation panels, and validation sets.Set B contains 65,280 scored transitions, and the released-label panel provides 128 observations per block.Source: Table 1 · p. 16, Table 2 · p. 17
  • The corpus controls include TinyStories, C4 RealNews-like, Gutenberg, shuffled web text, and Zipf i.i.d. tokens.Small seeds use document-disjoint thirds; small and medium stay within one pass, while tiny and grid schedules may revisit their streams.Source: §B.1 · p. 16

B.2 BUDGETS, TOLERANCES, AND PROTOCOLS

  • Jacobian measurements use analytic adapter formulas checked against autodifferentiation or reverse-mode autodifferentiation, with precision depending on the study.Table 3 specifies measurement, draw, and selection budgets.Source: §B.2 · p. 17, Table 3 · p. 18
  • Reinitialization bands are descriptive reference intervals, not confidence intervals, and training seeds are the replication units for replicated-cohort tests.Bands ordinarily use pooled means plus or minus two population standard deviations; the study’s trigger can expand released surveys to ten draws.Source: §B.2 · pp. 17–19

C.3 THE CONDITIONAL TEST

  • The assignment test compares trained channel assignments with writer-only permutations, retaining identity draws, fixed points, and duplicates.Independent block permutations form each seed-level draw; inclusive ties and the Monte Carlo +1 rule yield a finite-sample-valid upper-tail test conditional on fixed inputs and unordered factor sets.Source: §B.2 · p. 18, §C.3 · p. 21

C.1 OPERATOR CONVENTIONS; C.2 IDENTITY AND EXPANSION; C.3 THE CONDITIONAL TEST

  • The Jacobian conventions distinguish the MLP update from the full residual map, and the reported operator is ordinarily the update or branch Jacobian.For a square Jacobian, the symmetry score is tied exactly to the Frobenius cosine between the operator and its transpose; the conditional gap splits into paired-channel organization and signed collective reciprocity as an accounting identity.Source: §C.1 · p. 19, §C.2 · p. 20, §C.3 · p. 21

D THE LEARNED ASSIGNMENT

The learned-assignment tests find trained Jacobian symmetry beyond writer-reassignment controls across objectives and depths. The operator’s scale qualifies its eigenvalue contrast, while the assignment signal persists under corpus shuffling and coarse scale constraints.

D THE LEARNED ASSIGNMENT

  • The tiny endpoint exceeds all 999 re-pairings in each seed, and every objective-grid pre-norm cell exceeds all 999 stored re-pairings.The tiny endpoint has Holm-adjusted p = 0.003; every block of each pre-norm objective-grid cell exceeds the re-pairings.Source: D THE LEARNED ASSIGNMENT · p. 21
  • The assignment gap peaks inside the stack in every measured cohort, while the smallest symmetric eigenvalue’s separation largely disappears after RMS normalization.After normalization, only the first five and last two medium blocks retain a contracting direction extreme for its size.Source: Figure 8 · p. 21, D THE LEARNED ASSIGNMENT · p. 22, Figure 9 · p. 22
  • After training-stream token shuffling, both tiny GELU and SwiGLU families retain positive assignment gaps, though opposed cores do not form in the GELU corpus controls.The shuffled arms are collective-dominant, whereas the natural-corpus SwiGLU arm is diagonal-dominant.Source: D THE LEARNED ASSIGNMENT · p. 22
  • Writer reassignment within reader-activity and writer-norm scale cells retains nearly the full gap in both small context-256 families.The reported constraints therefore do not remove the signal, although within-cell orientation and collective structure remain possible contributors.Source: D THE LEARNED ASSIGNMENT · p. 22, Table 4 · p. 22
  • Across the measured operator readouts, trained assignments exceed their re-pairing ensembles on symmetry score, Frobenius norm, and branch-output norm, and fall below them on three other statistics.These differences are at the draw floor in both families and every seed (two-sided p = 0.002; Holm-adjusted p = 0.006).Source: D THE LEARNED ASSIGNMENT · p. 23
  • A single-hidden-layer untied MNIST autoencoder also has a positive assignment gap in every tested cell, while the paper confines its functional claims to transformers.The gap is at the draw floor across two optimizers, two widths, and three seeds.Source: D THE LEARNED ASSIGNMENT · p. 23

E WEIGHT-LEVEL AND EFFECTIVE READINGS

  • The study frames weight-level and effective pairing coordinates as distinct readings whose applicability depends on the MLP family.The subsequent comparison contrasts elementwise and gated MLPs, including released models.Source: E WEIGHT-LEVEL AND EFFECTIVE READINGS · p. 23

E.1 WEIGHT-LEVEL PAIRING ON RELEASED MODELS

In released checkpoints, the weight-level pairing coordinate tracks depth in elementwise models but not in OLMo-1B, where effective and weight-level readings diverge in the input block.

E.1 WEIGHT-LEVEL PAIRING ON RELEASED MODELS

  • In OLMo-1B’s input block, about half a percent of channels are opposed by the weight-level coordinate, while the effective coordinate marks all but about half a percent as opposed.The sixteen effective-core channels lie in the amplifying weight-level half, with values 0.16–0.23 against a block median of 0.06.Source: §E.1 · p. 23, Figure 11 · p. 24
  • GPT-2 small’s effective core is opposed at the weight level in every block, with sixteen of sixteen core channels opposed through the middle of the stack.Its weight-level share below zero rises along the stack; the gap from the effective share is 35 percentage points in the input block and averages 23 points over blocks 2–9.Source: §E.1 · p. 24, Figure 11 · p. 24
  • The weight-level coordinate reads depth order in elementwise models but not in the gated OLMo-1B checkpoint.In OLMo-1B, the weight-level share below zero reaches the effective level only in the last two blocks.Source: §E.1 · pp. 23–24

E.2 WHERE THE WEIGHT-LEVEL AND EFFECTIVE READINGS PART

For gated MLPs, the effective reader includes input-conditioned gate and normalization effects absent from the weight-level coordinate. Their divergence tracks negative-gate prevalence, and removing negative-gate reader contributions restores high sign agreement.

E.2 WHERE THE WEIGHT-LEVEL AND EFFECTIVE READINGS PART

  • At four measured OLMo-1B states, the effective and weight-level coordinate signs agree for 0.79–1.00 of channels by block and 0.92–0.97 by state.When the gate rests closed on most of the panel, its negative mean can make weights that amplify read as opposed.Source: §E.2 · pp. 24–25
  • Across activation families, agreement between weight-level and effective readings follows the share of channels with a negative mean gate or derivative.Elementwise families show 99.2–99.3% agreement at small scale and 96.3% in the medium model, while gated SwiGLU and GEGLU lose agreement in and beyond the input quarter.Source: §E.2 · p. 25, Figure 12 · p. 25
  • Removing reader contributions at negative-gate positions raises gated-model agreement to 0.90–0.92, while the small GELU control remains unchanged.The change concentrates in input blocks; matched random deletions leave signs where the full reader has them.Source: §E.2 · p. 25, Figure 12 · p. 25
  • The removed T−k term carries the sign of −m_w,h in nearly every channel, and removing it alone recovers most of the agreement change.Removing T−q alone changes nothing.Source: §E.2 · p. 25
  • How the measured geometry divides between learned gating and learned weights remains open, and the authors do not build on the post-norm OLMo-2-1B case.Its gates rest closed through most of the stack at two trillion tokens.Source: §E.2 · p. 25

F FORMATION AND TRAINING TRAJECTORIES

Across training histories, opposed reader–writer signs form before coupling grows and persist across model families, while writer reassignment can rebuild the pairing. Across corpora, the opposed core depends on token order and remains largely distinct from the vocabulary content channels project.

F.1 FORMATION AND RECOVERY

  • After three earlier breaks, endpoint loss returns within 0.004 nats of control although writers end only 0.2–0.6 cosine aligned with control partners; reassignment still rebuilds part of the original pairing.The latest break does not close its gap within the resumed schedule.Source: §F.1 · p. 26, Figure 13 · p. 26
  • At step 100, the medium-model loss pushes 0.55–0.56 of writers against their readers while only 0.56 of channel signs are opposed; medium blocks are all-negative from step 100 onward.Small models can briefly regain positive blocks after their first all-negative state.Source: §F.1 · p. 26
  • Among OLMo-1B’s 256 endpoint-core channels, 36 change sign across saved states, and only three are positive at both the first and last checkpoints.The endpoint-selected cohort’s final-block overlap with the contemporaneous core rises from zero through 41B tokens to all sixteen channels.Source: §F.1 · p. 27, Figure 15 · p. 28, Figure 16 · p. 28

F.2 TRAJECTORIES ACROSS RELEASED STEPS

  • Across released histories, the coupling gap emerges early and plateaus, while GPT-2 small’s core turns opposed deepest-first and input-block last, with seed-consistent timing.GPT-2 small’s shallow blocks turn by 2 × 10^8 tokens after an amplifying dip; the input block turns after 10^11 tokens spent amplifying.Source: §F.2 · pp. 27, 29, Figure 17 · p. 29, Table 7 · p. 31
  • Interior-block core signs become majority opposed at a few hundred steps, when summed coupling is only a few percent of its final size; coupling later grows without changing the sign.Table 6 reports the sign and coupling milestones across model histories.Source: §F.2 · p. 30, Table 6 · p. 30
  • Most channels keep their opposed pairing from formation to endpoint even as their promoted-token sets have no overlap and writer directions remain far from final at half of training.Promoted sets also turn over by more than half between consecutive checkpoints throughout training.Source: §F.2 · p. 30

F.3 ACROSS CORPORA AND TOKEN ORDER

  • Natural corpora form negative cores across depth, while order-removed controls retain little coupling; deeper runs preserve the pattern with small positive exceptions near stack ends.On the natural stream, core channels leave zero and open downward while other channels remain near zero; without order, all channels stay near zero.Source: §F.3 · pp. 31–33, Figure 20 · p. 32, Figure 21 · p. 33
  • With order intact within four tokens, the core sum exceeds half its natural value; natural-stream cores generally promote more frequent tokens, while couple readings find few recurring relations beyond frequency.With order intact within thirty-two tokens, the core sum reaches its natural value; frequency dominates readings even in order-removed controls, where coupling is much smaller.Source: §F.3 · pp. 32–33

G CORES AND THEIR DOWNSTREAM EFFECTS

The coupling core is concentrated in a small set of channels, especially large-reader, opposed channels. Its contributions are broadly taken up by later readers, while direct logit effects are less consistent.

Concentration across channels

  • 133–471 of 4,096 channels (3.2–11.5%) account for at least half of cumulative |mh| across 72 medium seed–block pairs.The block nearest the median half-share count needs 384 channels for half and 1,475 (36.0%) for 90%.Source: G CORES AND THEIR DOWNSTREAM EFFECTS · p. 34

Channel face

Transported-path check of the stream register

  • At the next block, transported response has cosine 0.99 with the stream-register prediction, then its coefficient falls to about 0.6 four blocks on and a fifth at twelve.The response direction does not turn negative; intervening computation is consistent with net attenuation, but the attenuating blocks are not identified.Source: G CORES AND THEIR DOWNSTREAM EFFECTS · p. 36

Returns and pushes across the population

  • About 1,800 of the next block’s 4,096 readers carry the transported response, while five in six logit couples lie in the last quarter of blocks.Under a tenth of the core’s direct output effect survives later blocks, compared with about three quarters of the strongest pushers’ effect.Source: G CORES AND THEIR DOWNSTREAM EFFECTS · p. 36

Direct path to the logits

  • Direct logit attribution is exploratory: the core carries 1.6–7.8% of immediate absolute logit mass, but direct-path and full-removal estimates have opposite signs in eight cells.The correct token’s write projection is at the 46th to 49th percentile, and target-label shuffling finds no consistent preferential promotion.Source: G CORES AND THEIR DOWNSTREAM EFFECTS · p. 37

H FUNCTIONAL INTERVENTIONS

Interventions distinguish effects tied to reader–writer pairing from those carried by factor sets or shared directions. Pair breaking changes spectral shape, while parallel-component removals become costly later in training.

Pair breaking

  • Pair breaking returns controlled and released pre-norm models toward matched references, but moves the unit-normalized singular profile in every measured group.Stable rank remains 15–28 after breaking, while a reciprocity-only interpretation is not licensed by the separate direction check.Source: H FUNCTIONAL INTERVENTIONS · p. 37, Table 11 · p. 37, Table 12 · p. 37

Bare branch on cached states

  • On identical cached states, pair breaking restores the controlled GELU, GPT-2, and SmolLM2 bare branch to within two percent of its reference.The factorization Jbranch = JgJN holds to numerical precision in these pre-norm groups.Source: H FUNCTIONAL INTERVENTIONS · p. 38

Re-pairing at equal disturbance

  • At equal disturbance, re-pairings that damage symmetry more incur small but positive loss differences in every seed, significant after correction.The comparison is a conditional association after matching, not a fraction of function explained.Source: H FUNCTIONAL INTERVENTIONS · p. 38, Table 13 · p. 38

Decomposing the writer

  • At the medium endpoint, parallel removal costs 0.004–0.59 nats per block against at most 0.01 for matched random directions; jointly it costs 21–22 nats, four to five times the sum.Across training, perpendicular deletion becomes costly first, while parallel deletion grows costly later; OLMo shows the same order across checkpoints.Source: H FUNCTIONAL INTERVENTIONS · p. 38, Figure 24 · p. 39
  • The parallel components sum to a symmetric operator with over 99.6% of its eigenvalue energy negative; its leading eigenvector aligns with mean reader direction at cosine 0.89–1.00.The leading part holds a fifth to a half of A’s energy but only 2–4% of its trace.Source: H FUNCTIONAL INTERVENTIONS · p. 38

Re-pairing separates the two parts

  • Re-pairing leaves under 1% of native trace but preserves about six tenths of shared-direction damping, separating pairing effects from effects in the factor sets.The opposed sign appears before parallel-removal costs grow, so it is not a by-product of the shared direction.Source: H FUNCTIONAL INTERVENTIONS · p. 39

Targeted moves on named couples

  • Opposed-writer flips shift output toward common tokens by 0.52 in expected centered log-frequency, with 0.20 nats/token excess loss over matched flips.Other broad edits have sparse effects relative to comparators and isolate no semantic family.Source: H FUNCTIONAL INTERVENTIONS · p. 39

I COUPLE CENSUS PROTOCOL

This section sets out a couple-census protocol that combines replay-checked channel evidence with model annotations, while deriving numeric classes and denominators from measurements alone. It freezes and audits a taxonomy across all channels, distinguishing annotations of individual factors from the relation revealed by reading matched factors together.

I.1 EVIDENCE PER CHANNEL

  • Each channel’s input, stream, and output registers combine ranked firing contexts with downstream-reader and logit effects, drawn from replayed profiles.The evidence uses an exact replay of the first 107 evaluation-panel tokens in 256-token windows; register arrays are checked against replayed activations before annotation.Source: I COUPLE CENSUS PROTOCOL · p. 39, §I.1 · pp. 40–41, Figure 26 · p. 40

I.2 STAGE 1: OPEN READING

  • One anonymized Qwen3.6-35B-A3B call per channel produces ten constrained fields; responses that fail to parse, 1.2–1.3% per seed, remain UNPARSED.The call uses BF16, vLLM 0.27.1, temperature zero, and at most 512 output tokens.Source: §I.2 · p. 41

I.3 STAGE 2: TAXONOMY

  • Numeric classes are computed from measurements, then a stratified discovery, synthesis, validation, and revision process freezes a single-label taxonomy of 24 classes plus OTHER and AMBIGUOUS.Output push relative to the decision gap is negligible below 0.10, weak from 0.10 to 0.25, material from 0.25 to 1, and decisive above 1; moving the 0.25 threshold to 0.20 or 0.30 changes the above-threshold share from 10.6% to 16.8% or 7.0%. The taxonomy groups classes into stream couples, logit couples, nudges, and subject-fixed couples.Source: §I.3 · p. 41

I.4 STAGE 3: APPLICATION AND AUDIT

  • Population labels agree with reference labels on 76% of channels at the class level and 92% at the family level.Fixed denominators are 98,304 channels per seed and 294,912 pooled; NOT APPLICABLE and AMBIGUOUS remain table rows, while OTHER has no channels in the three seeds.Source: §I.4 · p. 41, Table 15 · p. 42

I.5 ENGINES AND COST

  • Evidence replay ran in PyTorch on one H200 per seed, while open reading, taxonomy construction, application, and reference labeling used Qwen and Claude agents.The open-reading workload used 6.0 × 10^9 prompt-plus-completion tokens; taxonomy construction used 6.4 × 10^6 tokens, and application and flagged-row rereading used 7.0 × 10^7 tokens.Source: §I.5 · p. 42

I.6 READING READERS ALONE

  • Reader labels come from reading each reader alone, and a second-model check finds 92–95% same-or-compatible agreement among jointly informative cases.About one first-pass label in twenty is judged uninformative; population counts retain unlabelled cases, while comparisons requiring an informative reader exclude them.Source: §I.6 · p. 43
  • The reference audit uses Claude Opus 5.5 at one query per 250 held-out records, with each record independently assigned one frozen-taxonomy class.The protocol specifies a single-label rule and returns a confidence value for each record.Source: §I.6 · pp. 46–47

J ONE-FACTOR READINGS AND THE COUPLE

  • A single-factor reading shows what a writer or reader carries, while the reader–writer relation is assessed from both factors at the same positions.The corpus-cohort cores are read in the same way in Section F.3.Source: J ONE-FACTOR READINGS AND THE COUPLE · p. 47

J.1 WHAT THE CORE WRITERS PROJECT

The core’s vocabulary projections are compared with several control sets using frequency rankings and taxonomy readings. The strict surface rules identify only a few qualifying core channels.

What the core writers project

  • The medium core promotes more frequent tokens than its controls, and the contrast narrows when leading unembedding directions are removed.The frequency comparison uses median promoted-token rank; lower ranks indicate more frequent tokens.Source: §J.1 · p. 47, Figure 27 · p. 47, Table 16 · p. 48

Surface census

  • Only two of 1,152 medium core channels pass the common-token rule, and none pass the repeated-separator rule.Across the three control sets, three channels pass the common-token rule and none pass the separator rule.Source: §J.1 · p. 48, Table 17 · p. 48
  • Morphemes dominate early core readings, while semantic fields concentrate in later amplifying channels.The broader census compares four sixteen-channel sets against a fixed vocabulary taxonomy.Source: §J.1 · p. 48

J.2 WHAT THE READERS RESPOND TO

The readers respond to distinct kinds of contexts, while their triggers are not apparent in the corresponding writers’ vocabulary. Across the medium population, reader-promoted tokens and writer-promoted tokens rarely overlap.

J.2 WHAT THE READERS RESPOND TO

  • One reader responds to microbiology and biochemistry vocabulary, while another responds to the first pieces of long words; neither subject appears in its writer’s vocabulary.Document, line, list, and heading conditions shift neither reader by more than 0.25 population SDs.Source: §J.2 · p. 49
  • Using the reader direction in place of the writer predicts the same vocabulary ranking for exactly parallel directions and the reverse ranking for exactly antiparallel directions.This is the angle-alone prediction from the raw vocabulary lens.Source: §J.2 · p. 49
  • In just under one channel in four, the reader’s promoted sixteen share a token with the writer’s suppressed sixteen; the two promoted lists overlap in fewer than one channel in thirty-five.On average the lists share a third of one token, and three channels in four share none.Source: §J.2 · p. 50

Related papers

Loading 2610.29340v1…