PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane

arXiv:2610.21025v1cs.CLcs.AIcs.LG

TL;DR

Behavioral dates based on model outputs may conflate what hidden states encode with what a still-learning readout expresses. The paper tracks readout development directly and finds an early period of reorganization and gaps between availability and native expression. Its results indicate that early behavioral dates can reflect readout development as well as hidden-state development.

  • Problem

    Developmental studies often date capability acquisition from outputs, although those outputs pass through a readout that is itself learned during pretraining.

    Source: §1 · p. 1

  • Method

    The paper tracks readout development with parameter-trajectory crosscoding and separates hidden-state and readout contributions using probes and readout swaps.

    Source: §1 · p. 2

  • Results

    The readout’s learning rate changes when distinctions become expressible in 31M runs, while final losses remain within 0.06 nats.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Early behavioral dates can reflect readout development as well as hidden-state development, so measuring both helps separate what a model encodes from what its logits express.

    Source: §6 · p. 10

    Evidence that readout learning rate shifts timing is limited to one 31M setting, with one run per condition stopped at 10B tokens; larger models remain untested.

    Source: §6 · p. 10

Abstract

from arXiv · show

When does a language model acquire a capability? Developmental studies usually answer this by tracking when models start producing the correct token. But that prediction passes through the unembedding matrix, or readout, which is itself still being learned during pretraining. We show that a model's hidden states can encode task information before its own readout expresses it, so timelines read from outputs conflate what the model encodes with what its readout expresses. To track the readout directly, we introduce parameter-trajectory crosscoding, which fits one sparse dictionary to every token's readout row at every checkpoint. Unlike crosscoders on activations, it needs only checkpoint weights. We use it to locate a narrow early window, 0.4-1.1% of the way through Pythia's pretraining, in which the readout reorganizes and WordNet word categories become decodable from its rows. By then, linear probes on the hidden states already decode task information that the model's own outputs do not yet show. To test whether such gaps depend on how fast the readout learns, we pretrain small models from scratch that differ only in the readout's learning rate. We find that raising it makes the readout develop earlier and the gap close sooner, with little effect on final loss. Together, these results imply that early in training, where many developmental transitions are dated, a behavioral date can reflect when the readout learns to express task information and not when the model learns it.

1 Introduction

Behavioral timelines read through logits can conflate information in hidden states with what a changing readout expresses. The paper introduces a way to track readout development and finds that it can lag behind available task information, especially early in training.

1 Introduction

  • The readout maps last-layer hidden states to vocabulary logits, so output-based measurements depend on a matrix that is still being learned during pretraining.For the same early hidden state, the model’s own readout favors the incorrect “is,” while the final readout favors “are.”Source: §1 · p. 1
  • Logit changes can reflect hidden-state changes, readout changes, or their interaction, so logits alone cannot distinguish these sources.The same ambiguity affects checkpoint comparisons using the logit lens, vocabulary projections, and attention-head scores computed through the readout.Source: §1 · pp. 1–2
  • The paper introduces parameter-trajectory crosscoding to decompose readout development into sparse features followed across checkpoints, using only checkpoint weights and covering the full vocabulary.Unlike activation crosscoders, the method fits readout weights directly.Source: §1 · p. 2
  • 0.4–1.1% of training is a narrow window in which the Pythia readout reorganizes and WordNet categories become decodable from its rows.The paper argues that a behavioral date in this early period can reflect readout development as well as hidden-state development.Source: §1 · p. 2
  • Pythia-1B and 6.9B hidden states encode subject number before their own or any intermediate readout expresses it.In controlled small-model runs, raising only the readout learning rate makes it develop earlier and native expression reach availability sooner, with little effect on final loss.Source: §1 · pp. 2–3

2 Method

The method tracks shared changes in readout rows across checkpoints, then uses feature trajectories and paired probe/readout measurements to characterize development and separate encoding from expression.

2.1 Parameter-Trajectory Crosscoding

  • Parameter-trajectory crosscoding fits one sparse dictionary to readout rows across checkpoints, giving each token a shared feature code while allowing feature directions and active tokens to change.The approach needs only checkpoint weights, not labels or an activation corpus.Source: §2.1 · p. 3
  • The reconstruction objective combines row reconstruction error with a saturating firing-rate penalty that encourages sparse use without pushing active contributions toward zero.Decoder norm trajectories are normalized by their maximum to compare when features grow or decay rather than their absolute size.Source: §2.1 · pp. 3–4, §2.2 · p. 4

2.2 Measuring Readout Development

  • A reorganization window is selected where the geometric mean of six normalized change measures most exceeds circular-shift nulls that break their temporal alignment.For controlled runs, the paper also reports median feature peak step and peak turnover step.Source: §2.2 · p. 4

2.3 Distinguishing Hidden States and Readout Expression

  • Readout swaps hold a hidden state fixed while changing the readout, and the oracle sweep reports the best intermediate-readout contrast accuracy as an upper bound.A linear probe measures availability directly from the hidden state, while native expression is the checkpoint’s own readout accuracy.Source: §2.3 · p. 4

3 Experimental Setup

The study compares readout development across pretrained language models and controlled small models, using task probes and readout-feature analyses to measure lexical structure and task margins.

Models and dictionaries

  • Four pretrained models cover Pythia-160M, 1B, and 6.9B plus OLMo-2-7B; four 31M models vary readout learning rate or warmup from a shared initialization.The 31M runs use the first 10B tokens of Pythia’s preshuffled training data.Source: §3 · p. 5

Tasks and scoring

  • Binary tasks compare a correct answer token with a foil at the final prompt position, using contrast accuracy over the two answers with 50% chance.Tasks include subject–verb agreement, indirect object identification, and number comparison.Source: §3 · p. 5

Probes and controls

  • Linear probes on final hidden states measure task-information availability, with step-0, prompt-token, random-feature, and shuffled-label controls testing alternative explanations.The broader 20-family suite uses a shared-label shrinkage LDA direction; SVA and IOI use logistic regression at each checkpoint.Source: §3 · p. 5

Sparse localization

  • Sparse localization selects eight high-attribution features on one item split, then tests ablation and isolation on another against matched random features.The reconstructed final readout is decomposed into feature contributions for SVA, IOI, and greater-than completion.Source: §3 · pp. 5–6

Lexical organization

  • WordNet lexical organization is tested by one-vs-rest probes on centered readout rows for 35 categories, with metadata and matched-token controls.The categories include broad supersenses such as animal nouns; performance is held-out balanced accuracy.Source: §3 · p. 6

4 Results

Readout structure changes early, and readout rows gain lexical organization as hidden states become capable of encoding task information before native expression. Varying readout learning rate shifts both readout development and the timing of expression, while final validation losses remain similar.

4 Results

  • The dictionaries explain 0.808–0.920 of readout-trajectory variance across four models, using 1.17–2.26% of features per row per checkpoint and keeping feature indices shared across checkpoints.They nearly match separately fitted sparse autoencoders with fewer active features; dense PCA needs more components for the same fit in Pythia.Source: §4 · p. 6

4.1 The Readout Reorganizes Early

  • Pythia’s readout reorganizes between steps 512 and 1.6k (0.4%–1.1% of training), while feature trajectories differ by model and OLMo-2-7B’s similar-fraction window does not clear nulls.The Pythia window contains the end of warmup; 20 of 21 Pythia dictionaries select that window, and one selects the adjacent window.Source: §4.1 · p. 6

4.2 Lexical Organization Develops Alongside Structural Change

  • In Pythia-160M, all 35 WordNet categories become decodable from readout rows by step 1000, with 33 categories’ largest gains between steps 128 and 1000.By step 512, 33 categories exceed both matched-random and shuffled-label nulls; lemma-based splits preserve the early rise.Source: §4.2 · p. 7

4.3 Hidden State Availability Precedes Native Readout Expression

  • On Pythia-6.9B agreement, hidden-state probes approach ceiling by step 512 while native and best swept readouts remain at chance; related gaps recur across subject–verb agreement sets and model sizes.Later readouts also lower next-token loss on earlier Pythia-1B states, though the early-state task-accuracy gain depends on readout frame.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 A Few Readout Directions Carry the Agreement Margin

  • Eight features carry the reconstructed final readout’s SVA margin: ablating them lowers accuracy from 0.900 to 0.487, while keeping only them gives 0.938.Ablating matched random features leaves accuracy at 0.931.Source: §4.4 · p. 8

4.5 Readout Learning Rate Shifts Readout Development

  • Raising the readout learning-rate multiplier moves median feature peak steps from 128 to 64 to 32, while probe–expression gaps close by step 1024 except at 0.25×, which closes by 2048.At 0.27B tokens, the 4× run’s validation loss is 1.20 nats below baseline and 2.08 nats below the 0.25× run; final losses agree within 0.06 nats.Source: Figure 5 · p. 9, §4.5 · p. 9

5 Related Work

Prior work studies language-model development through outputs, projected hidden states, and embedding geometry. This paper adds direct tracking of readout rows across checkpoints and relates their development to what fixed hidden states express.

Reading development through the readout

  • Developmental analyses often date linguistic representations and concepts through outputs or vocabulary projections, which depend on a readout learned during training.The paper frames its contribution as adding readout tracking to those comparisons.Source: §5 · p. 9

Weight-space analyses

  • Prior weight-space work studies output embeddings and their geometry, while sparse-feature analyses explain individual readouts at a single model state.Papadimitriou and Prince [2026] relate input/output embedding geometry to semantic, syntactic, and frequency structure over training.Source: §5 · p. 9

Shared dictionaries over checkpoints

  • Checkpoint crosscoders share features across activations sampled from text, whereas this approach applies shared codes to token-aligned readout parameters and covers every readout row using checkpoint weights.Behavioral tests still require text, forward passes, and task labels.Source: §5 · p. 10

6 Discussion

Readout development can shape behavioral timelines, especially early in training, so output-based analyses should distinguish what hidden states encode from what the readout expresses. The readout measurements also depend on how normalization and LayerNorm scales are represented.

6 Discussion

  • Readout-based development measures should be paired with native expression, same-task probes with initialization baselines, and readout sweeps on the same states.The authors also suggest testing whether these measures sharpen forecasts from early checkpoints and small, briefly trained models, including for safety-relevant capabilities.Source: §6 · p. 10
  • Behavioral dates during the early readout-reorganization period can reflect readout development as well as hidden-state development.The discussion places this period near other dated transitions and leaves the causes of differing feature trajectories open.Source: §6 · p. 10
  • Readout learning-rate changes alter early loss and expression timing in 31M runs, while final losses remain within 0.06 nats.The authors caution that recipe comparisons on partial runs may reflect readout speed rather than final training outcomes.Source: §6 · p. 10
  • The learning-rate timing evidence is limited to one 31M setting, with one run per condition stopped at 10B tokens; availability probes cover two-token contrasts.The readout is about 42% of parameters in that setting, versus about 10% in Pythia-1B, and larger models remain untested.Source: §6 · p. 10
  • Future work proposes testing early recipe rankings with probes or swapped readouts, training new vocabulary rows with their own learning rate, and tracking drift across layers.These are proposed directions, not reported results.Source: §6 · p. 10

A.1 Normalization and Reconstruction

  • Reconstruction restores a normalized readout row to original space, and interventions retain the stored mean and decoder bias unless specified otherwise.The normalization centers vocabulary rows and scales them to prevent checkpoints with larger entries from dominating the loss.Source: §A.1 · p. 20
  • Readout geometry and swaps depend on how LayerNorm gain and bias scales are split with the readout, although logits and probes remain invariant to that symmetry.The analysis uses each released checkpoint’s stored split; centering and scaling remove only a vocabulary-wide offset and one global scale.Source: §A.1 · p. 20

A.2 Gate and Training Objective

The appendix specifies how trajectory crosscoding and behavioral probes are fit and reported, then records the model, checkpoint, and unembedding choices underlying those analyses.

A.2 Gate and Training Objective

  • A shared feature index follows one vocabulary row across checkpoints, while checkpoint-specific gates and decoders reconstruct each view.The reconstruction objective averages squared errors over vocabulary rows and checkpoints; its sparsity penalty trades against reconstruction, with linear warmup of the effective weight.Source: Figure A.1 · p. 20, §A.2 · p. 21

A.3 Behavioral Probe Fitting and Reporting

  • Behavioral analyses use grouped held-out logistic probes for SVA and IOI, task-supervised one-direction probes for broader binary tasks, and readout sweeps with distinct selection procedures.The 32-checkpoint SVA/IOI sweep is a post hoc upper envelope; the broader-suite sweep selects on one half of items and evaluates on the other.Source: §A.3 · p. 21

B.1 Code and Data Release

  • The code repository provides training and evaluation implementations, while trained dictionaries, evaluation outputs, and controlled-run checkpoints are released separately.Instructions cover external model and data inputs.Source: §B.1 · p. 22

B.2 Models, Tokenizers, and Checkpoint Schedules

  • The study uses Pythia-160M, 1B, and 6.9B plus OLMo-2-7B, with 32 selected checkpoints per trajectory.Pythia checkpoints span steps 0–143k; OLMo-2 uses stage-1 checkpoints spanning steps 150–928k.Source: §B.2 · pp. 22–23

B.3 Checkpoint-Suite Selection

  • Pythia supplies a controlled family with a shared tokenizer, data order, and untied readout; OLMo-2 is a cross-architecture check, not a Pythia scaling point.The checkpoint suites were selected to provide accessible readout weights and sufficient early checkpoints around step 1000.Source: §B.3 · p. 23

B.4 Unembedding Convention and Tied Embeddings

  • Pythia’s readout is the untied embed_out.weight matrix, whose vocabulary rows give token output directions; OLMo-2 uses lm_head.weight.The analyses use the output readout, not Pythia’s separate input embedding.Source: §B.4 · p. 23

B.5 Sparse Dictionary Architecture and Loss

The sparse dictionaries use checkpoint-specific heads to track shared readout features across training, while centered and scaled rows support reconstruction analyses that remove the vocabulary-wide mean offset.

B.5 Sparse Dictionary Architecture and Loss

  • Each dictionary uses 32 checkpoint-specific encoder and decoder heads to rebuild every token row from sparse features, with training settings specified for the main fits.The dictionaries are trained on centered-and-scaled vocabulary rows for 300 epochs with batches of 1,024 rows and no dead-feature auxiliary loss.Source: §B.5 · p. 23, Table B.2 · p. 24

B.6 Existing Assets, Licenses, and Terms

  • The released dictionaries and derived measurements identify upstream assets, but the original Pythia and OLMo checkpoints and raw Wikipedia dumps are not redistributed.The paper lists licenses and terms for the checkpoints, datasets, and released artifacts.Source: §B.6 · pp. 24–25

C Dictionary Validation and Quality

  • Selected dictionaries explain 0.808–0.920 of readout-trajectory variance while activating 1.17–2.26% of features per row.The validation defines EV on centered and scaled rows and reports a narrower Pythia-160M dictionary separately from the selected fit.Source: Figure C.1 · p. 25, C Dictionary Validation and Quality · p. 25

C.1 Mean-Direction and Preprocessing Check

  • Centering removes the positive floor in mean pairwise row cosine and lowers the late-checkpoint top spectral gap across Pythia-1B, Pythia-6.9B, and OLMo-2-7B.The raw mean row aligns closely with the top right singular vector, inflating spectral and cosine geometry.Source: §C.1 · p. 25, Figure C.2 · p. 26

C.2 Next-Token Distribution Fidelity of the Reconstructed Readout

The appendix checks whether sparse dictionaries preserve readout behavior, outperform dense trajectory baselines, and generalize across checkpoints and settings. It also documents how readout features and their timing vary across models and dictionary fits.

C.2 Next-Token Distribution Fidelity of the Reconstructed Readout

  • At terminal checkpoints, Pythia-1B and Pythia-6.9B reconstructions match 0.70 and 0.69 of native top-1 predictions, with ∆NLL of 0.23 and 0.35 nats, respectively.Fidelity varies over training and is least stable before step 2k; the narrower Pythia-160M fit loses scale calibration at the final checkpoint.Source: §C.2 · p. 26, Figure C.3 · p. 26

C.3 Sparse Autoencoder Baselines

  • On Pythia-160M, checkpoint-spanning WU dictionaries reconstruct checkpoints with one shared feature index at low active-code density, whereas per-snapshot SAEs are much denser and the final-snapshot SAE fails on early checkpoints.They are not parameter-efficient because they use checkpoint-specific encoder and decoder heads to preserve the shared index.Source: §C.3 · p. 27, Figure C.4 · p. 27

C.4 Linear and Trajectory Baselines

  • Rank-1024 per-snapshot PCA falls short of selected dictionaries at the stated Pythia-1B and 6.9B checkpoints and OLMo checkpoints from step 4k onward; dominant Pythia subspaces move most before the reorganization window.The dense trajectory subspace also performs worse than dictionaries at every tested rank from 16 to 256.Source: §C.4 · p. 28, Figure C.5 · p. 28, Figure C.6 · p. 28

C.5 Held-out Checkpoint Reconstruction

  • From step 10k onward, the Pythia-160M crosscoder tracks the in-sample ceiling within 0.011 EV at all eight held-out checkpoints, with no sharp anchor-memorization failure.For Pythia-1B, deviations bracket the step-116k anomaly, which is also present in the in-sample ceiling and direct WU interpolation.Source: §C.5 · p. 29, Figure C.7 · p. 29

C.6 Sparsity-Penalty Sensitivity

  • At fixed Pythia-160M width, raising λ1 from 0.4 to 1.8 traces a single reconstruction–sparsity tradeoff, lowering mean L0 from about 160 to about 23 and EV from 0.75 to 0.53.The sweep adds an auxiliary loss on dead features, unlike the λ1 = 0.3 fits.Source: §C.6 · p. 29, Figure C.8 · p. 30

D.1 Rule-Based Lifecycle Profiles; D.2 Aggregate Timing Summaries

  • Pythia-160M and 1B features include early-decaying, persistent, and late-emerging profiles, while nearly all Pythia-6.9B features persist and most OLMo-2-7B features emerge late.In the Pythia dictionaries, the 512–1.6k window is where early-peaking features decay, not where features peak; at most 2% peak inside it.Source: §D.2 · pp. 31–32, Table D.1 · p. 31, Figure D.1 · p. 32

D.3 Late-Start Control for OLMo

  • With early checkpoints removed from the Pythia-1B fit, early-decaying features fall to 0.02% versus 24.1% in the selected dictionary, linking the profile to the checkpoint range used in fitting.The authors interpret OLMo’s scarcity of early-decaying features as more consistent with its missing early checkpoints than with a qualitatively different lifecycle.Source: §D.3 · pp. 32–33

D.4 Temporal Localization and Null Models

  • Twenty of 21 Pythia dictionary fits select the 512–1.6k reorganization window; the score combines six change measures and tests their alignment against circular-shift nulls.The Pythia-6.9B window is the least stable under feature resampling.Source: §D.4 · p. 33, Table D.2 · p. 33, Figure D.2 · p. 34

D.5 Stability Across Dictionary Fits

Reorganization timing is stable across most Pythia dictionary fits, although lifecycle-profile shares depend on dictionary settings. The accompanying WordNet analysis probes centered readout rows across a defined category inventory.

D.5 Stability Across Dictionary Fits

  • Twenty of 21 Pythia fits select steps 512–1.6k, while lifecycle-profile shares and median peak steps vary with dictionary settings and should be read qualitatively.The exception selects steps 1k–3.2k; the reported profile fractions should not be attributed to models alone.Source: §D.5 · p. 34, Table D.3 · p. 35

E.1 Probe Hyperparameters; E.2 WordNet Supersense Inventory

  • Each category is tested one-vs-rest on centered WU rows using balanced held-out accuracy, with a fixed row split across checkpoints and a minimum of 20 clean token ids.The row split can place surface variants of the same lemma on opposite sides; a grouped-by-lemma analysis is reported separately.Source: §E.1 · p. 35, §E.2 · p. 35

E WordNet Supersense Probe Analyses; E.2 WordNet Supersense Inventory

  • In Pythia-160M, all 35 probed WordNet supersense categories reach sustained decodability between steps 128 and 1000, and 33 have their largest adjacent-checkpoint gain in that interval.The categories are the lexicographer files with at least 20 clean token ids; probes use mean-centered WU rows.Source: §E.2 · p. 35, Figure E.1 · p. 36

E.3 Matched-Control Analysis

The WordNet timing analysis compares observed labels against metadata and matched-label controls, then checks timing across models and under lemma-grouped splits. The early decodability pattern largely persists under the grouped split.

E.3 Matched-Control Analysis; E.4 Probe Timing Trajectories

  • In Pythia-160M, category onsets fall between steps 128 and 1000, 19 occur at step 256, and 33 of 35 categories have their largest gain in that interval.Observed accuracy exceeds the median null threshold by 0.170 at step 512 and 0.272 at the final checkpoint; controls use 20 permutations per category and null type.Source: §E.3 · p. 36, §E.4 · p. 37

E.5 Pythia-1B Replication

  • On Pythia-1B, onsets range from step 128 to 2000 with a median of 512, and 33 of 35 categories have their largest gain between steps 128 and 2000.The metadata-only baseline and matched-random and label-shuffle nulls were not run for this replication.Source: §E.5 · p. 37

E.6 Lemma-Grouped Splits

  • With lemma-grouped splits, 34 of 35 Pythia-160M categories reach sustained decodability by step 1000, and the median onset remains step 256.The noun.relation category never separates under the grouped split; row-split accuracy had been supported by variants of the same lemma crossing splits.Source: §E.6 · p. 37
  • Under lemma-grouped splits, 32 of 35 categories exceed both Pythia-160M row-split null thresholds at step 512, and 34 do so from step 1000 through the final checkpoint.The nulls and metadata-only baseline were not rerun under the grouped split.Source: §E.6 · p. 37

F Availability and Expression Across Contrastive Task Families

Across the task suite, hidden-state probes can detect information before the model’s readout expresses it, but some gaps are already present at initialization. The appendix tests how these gaps vary across task families and languages, and how contrast accuracy differs from full-vocabulary prediction.

F Availability and Expression Across Contrastive Task Families

  • The appendix extends probe and readout comparisons to 20 task families across 24 checkpoints of Pythia-1B and Pythia-6.9B, using fixed-template contrast tasks and separate localization items.Localization splits share answer tokens across halves and therefore do not test transfer to new tokens.Source: F Availability and Expression Across… · p. 37, §F.1 · pp. 38–39

F.3 Initialization, Probe, and Answer-Token Controls

  • Across eight numeric families, step-0 probes already reach 0.86–0.89, so their persistent expression gaps do not show that the model learns the comparison.The folds group by number pair but still share individual numbers, allowing probes to learn values from token identity.Source: §F.3 · p. 40

F.4 Transient and Persistent Gaps

  • In Pythia-6.9B, simple and across-PP agreement and IOI close their gaps, whereas relative-clause and numeric gaps persist to the final checkpoint.For agreement tasks whose probes start near chance, probes reach near-ceiling by step 512 while native and swept expression remain at or below chance; the simple and across-PP gap closes by step 1000.Source: §F.4 · pp. 40–41, Figure F.2 · p. 41

F.5 Contrast Accuracy and Full-Vocabulary Prediction

  • A subject-relative-clause contrast accuracy of 0.75 coexists with a gold-verb rank-1 rate of zero, showing that two-token contrast does not establish full-vocabulary prediction.Function-vector families also reach E = 1.00 with rank-1 rate 0.00.Source: Table F.3 · p. 40, §F.5 · p. 41

G.1 Swap Grid Setup and Aggregate Measures

  • In Pythia-1B, early hidden states often achieve lower NLL through an intermediate readout, while later states prefer nearby or same-checkpoint readouts.The pattern also appears qualitatively in a Pythia-160M replication, which lacks the step-116k spike.Source: §G.1 · p. 42

G.2 Corpus Composition and Per-Language Swaps

  • The step-1000 readout raises English NLL by 0.726 nats [0.677, 0.779] over the native readout for step-256 states, despite being the aggregate optimum.English supplies 2.3% of scored targets, and seven languages written in neither Latin nor Cyrillic script supply 97.2% of the aggregate gain at that readout.Source: §G.2 · p. 43
  • For step-256 hidden states, the best readout varies by language, though every language prefers an intermediate readout for both step-256 and step-512 states.For step-512 states, all nine languages share the step-1000 optimum; from step 1000 onward, each language prefers its native or next readout, with gains at most 0.109 nats.Source: §G.2 · p. 43, Figure G.2 · p. 44

G.3 Step-1000 Swap Optimum and Late Checkpoint Anomaly

Later hidden states become more readout-specific, while early states remain well decoded through many later readouts. A step-116k readout anomaly produces a sharp, temporary NLL spike, and the early-state optimum is robust to several alignment choices.

G.3 Step-1000 Swap Optimum and Late Checkpoint Anomaly

  • For hidden states from steps 2000 and 3000, the next checkpoint’s readout is narrowly preferred in the aggregate, while early states remain close to their same-checkpoint distribution under many later readouts.Late states stay close only under nearby late readouts, so the terminal readout is not a neutral decoder for early checkpoints.Source: §G.3 · p. 44
  • For fixed step-47k states, NLL rises from 2.50 nats under the 102k readout to 5.55 under 116k, then returns to 2.95 under 130k.The spike also appears for fixed step-143k states, disappears under the next readout, and is absent in the Pythia-160M replication.Source: §G.3 · p. 44
  • The early-state readout optimum persists under raw scoring and mean, scale, row-norm, or orthogonal-frame alignment, except for a small one-checkpoint shift under Frobenius scaling.That shift is for step-1000 states and has an NLL gap of 0.031 nats.Source: §G.3 · p. 45, Figure G.3 · p. 45

G.4 Contrastive Task Margin Localization

At the fixed localization configuration, selected readout features affect some task margins more selectively than others, while hidden-state projection also implicates hidden-state directions. In controlled small-model runs, faster readout learning shifts feature development earlier; shortening warmup leaves the main timing statistics unchanged.

G.4 Contrastive Task Margin Localization

  • Subject–verb agreement and greater-than completion localize to selected features, while indirect object identification shows no selective positive-margin effect from a compact set at this configuration.Relational facts show an exploratory preserve-only effect on 12 of 13 scoring items.Source: §G.4 · pp. 46–47, Table G.2 · p. 46
  • Projecting out eight decoder directions removes at most 1% of hidden-state norm yet costs subject–verb agreement far more accuracy than greater-than completion.Under the native readout, the agreement effect depends on hidden-state directions used to separate the two answers, so readout change is not its only source.Source: §G.4 · p. 47

H.1 Setup and Conditions

  • The controlled recipe compares four 31M-parameter models trained on 10B tokens, varying readout learning-rate multiplier or warmup length while holding initialization seed and data order fixed.Each condition has one pretraining run; timing statistics use three crosscoder fits.Source: §H.1 · p. 47

H.2 Lifecycle Response to the Readout Learning Rate

  • Faster readout learning shifts feature trajectories toward earlier peaks and moves the localized decoder-norm-turnover peak earlier.Feature profiles shift from mostly persistent at 0.25× to mostly early-decaying at 4×; the timing claim rests on norm turnover because rotation peaks late in three conditions.Source: §H.2 · pp. 47–48, Figure H.1 · p. 48, Table H.1 · p. 48

H.3 Warmup Length Leaves the Crosscoder Timing Statistics Unchanged

  • Shortening warmup from 1430 to 715 steps leaves crosscoder timing statistics and maximum turnover aligned with baseline, though the frequency-class gap shifts slightly toward the faster-readout condition.The warmup-short validation loss at peak turnover is 0.52 nats lower; profile fractions are treated separately because their classifications are threshold-sensitive.Source: §H.3 · p. 48, Table H.1 · p. 48

H.4 Agreement Across Views on the Same Checkpoints

Across geometry, readout swaps, and probe-versus-native accuracy, diagnostics agree along the readout learning-rate axis, with larger early expression gaps at lower rates.

H.4 Agreement Across Views on the Same Checkpoints

  • 0.93 at 0.25× and 1.85 at 4× are the final mean WU row norms; final LayerNorm gain moves oppositely.Warmup_short tracks baseline; its final geometry measurement is missing.Source: §H.4 · p. 49, Figure H.2 · p. 49
  • At 4×, hidden states are best decoded by their native readout; at 0.25×, early states favor later readouts, with t=256 gains of 0.91–1.07 nats across alignments.For baseline and warmup_short, the gain over the native readout is at most 0.05 nats.Source: §H.4 · p. 49, Figure H.3 · p. 50
  • At t=256 and 512, early frequency-class probe-minus-native gaps order 0.25× > 1× > 4×; from t=2048, native accuracy exceeds probes in every condition.The early comparison measures a coarse next-token frequency property, and its absolute gap is not interpreted after t=2048.Source: §H.4 · p. 49, Figure H.4 · p. 50

H.5 Early Loss Differences and Similar Final Loss

  • At step 128 (268,435,456 training tokens), the 4× run’s validation loss is 1.202 nats below baseline and 2.079 nats below 0.25×; final losses are reported in Table H.1.Similar final losses do not establish that independently trained models differ only by the final-normalization/readout scalar symmetry.Source: §H.5 · p. 50

I.1 Extended Related Work

I.2 Extended Limitations

The analyses have bounded temporal and probe resolution and narrow comparison scope; proposed readout control and safety implications remain incompletely evaluated.

I.2 Extended Limitations

  • Checkpoint schedules limit temporal resolution; linear probes inspect only the final hidden state at the answer position, and comparisons have narrow scope, including one cross-family check and one sparse-intervention configuration.The reorganization window holds in 20 of 21 fits, while profile shares and median feature peak steps depend on dictionary width and sparsity.Source: §I.2 · p. 51

I.3 Implications

  • How task directions couple to the readout and how hidden-state/readout updates coordinate remain open; readout-edit side effects and persistence also remain unmeasured.Source: §I.3 · p. 51

I.4 Social Impact

  • No language model is deployed or validated safety evaluation proposed; methodological transparency supports auditing, while targeted readout manipulation is the main negative-use concern.Source: §I.4 · p. 51

Related papers

Loading 2610.21025v1…