PaperlayerView exact revision on arXiv

Source-linked AI summary

Erased, Rerouted, or Rescaled? Post-Training and the Causal Quotient of a Language Model's Belief State

Weihan Li, Tianshi Zheng, Junhao Wu, Xinlei Chen

arXiv:2610.05292v1cs.LG

TL;DR

The paper asks what happens to pretrained distinctions when post-training no longer rewards using them. It defines reward-null distinctions and measures whether they remain represented, affect decisions, or occupy variance. They are usually retained but may be rerouted or rescaled; anchoring and sustained weight decay distinguish continued use from erasure.

  • Problem

    A coarse reward can ignore distinctions that prediction needs, leaving unclear whether post-training erases them, reroutes them from decisions, or rescales them while retaining their use.

    Source: §1 · p. 1

  • Method

    The paper defines an exact reward-null kernel in controlled belief-state models and separately measures recoverability, causal decision use, and activation variance.

    Source: §1 · pp. 1–2

  • Results

    Reward-null information usually remains decodable after post-training, while the anchor determines whether decisions keep using it; erasure appears under sustained weight decay for distinctions needed by neither reward nor prediction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Post-training selects a causal quotient: reward defines decision-equivalence, the anchor and state update protect some ignored distinctions, and optimization determines the fates of the rest.

    Source: §7 · p. 8

    Exact ground truth for identifying the three fates is available only in small hidden-Markov worlds with one-step decisions whose actions do not feed back into context.

    Source: Limitations · p. 9

Abstract

from arXiv · show

What happens to information a pretrained model already encodes when post-training no longer rewards using it? The common language of representation compression conflates three fates: information may be erased, rerouted away from the decision while still represented, or rescaled to occupy less variance while still represented and used. We make these fates identifiable in models whose pretraining recovers Bayesian belief states. A reward that reads only a coarse function of the hidden state defines an exact reward-null kernel. The kernel lets us separately measure whether the information remains recoverable, whether decisions causally depend on it, and how much activation variance it occupies. Theory says what is protected: KL-anchored reinforcement learning preserves the reference policy's log-odds among equally rewarded outputs, supervised and unanchored objectives carry no such constraint, and spectral compression implies neither erasure nor loss of use. In controlled worlds, post-training mostly reroutes or rescales reward-null information and leaves it decodable. Without an anchor decisions can stop using it although the representation survives, and with one they keep using it. Erasure appears only under prolonged weight decay, for distinctions that neither reward nor next-token prediction can see. Open language models show the same dissociation: in-context belief geometry stays decodable under late-layer spectral compression, and within-class behavior depends on the anchor. Post-training thus selects a causal quotient of the pretrained belief state: the reward defines decision-equivalence, the anchor and the state update protect part of what it ignores, and optimization decides whether the rest is erased, rerouted, or rescaled.

1 Introduction

Post-training can leave pretrained distinctions erased, rerouted away from decisions, or rescaled while still represented and used. The paper makes these fates identifiable and finds that reward, anchoring, and optimization dynamics shape which distinctions survive and how they are used.

1 Introduction

  • Because prediction needs distinctions that a coarse reward may ignore, post-training leaves open whether those distinctions are erased, rerouted from decisions, or rescaled while still represented and used.Existing representational-change measurements cannot distinguish these fates from shrinking spectra alone.Source: §1 · p. 1
  • Belief states and components invisible to the next token are measurable in hidden-Markov models and open language models, enabling the paper’s tests of what post-training preserves.Prior work reports linear belief-state representations in residual streams and in-context reconstruction of belief geometry.Source: §1 · p. 1
  • A reward that reads only a coarse hidden-state function defines a reward-null kernel of distinctions pretraining may encode but the reward need not use.The paper separately asks whether each distinction remains represented, causally affects decisions, and occupies activation variance.Source: §1 · p. 2
  • KL-anchored reinforcement learning preserves the reference policy’s log-odds among equally rewarded outputs, whereas supervised and unanchored objectives impose no such constraint.Spectral statistics alone are not invariants of information or its use.Source: §1 · p. 2
  • Reward-null information usually remains decodable after post-training, while the anchor determines whether decisions keep using it; sustained weight decay can erase distinctions needed by neither reward nor prediction.Open language models also retain decodable in-context belief geometry under late-layer spectral compression, with within-class preferences depending on the anchor.Source: §1 · p. 2

2 Quotients of a Belief State

The section defines reward-equivalence as a partition of beliefs and causal-equivalence as a partition of histories by their policy decisions. It separates representation, causal use, and variance allocation to distinguish erasure, rerouting, and rescaling.

2 Quotients of a Belief State

  • Reward-equivalent beliefs have the same expected reward and differ by a direction in the reward-null kernel, which the reward map computes from the world parameters.The reward depends only on the task-class marginal.Source: §2 · p. 3
  • The kernel splits into next-token-visible and next-token-invisible distinctions, separating reward-ignored information that changes the next-token distribution from information that does not.The invisible component changes neither reward nor next-token distribution and may matter only for later tokens.Source: §2 · p. 3
  • The causal quotient groups histories that induce the same policy distribution, tested by intervening on history rather than decoding activations.Post-training selects a new causal quotient constrained by the objective, with optimization choosing among the allowed policies.Source: §2 · p. 3
  • R measures represented information, U measures causal effect on decisions through interventions, and A measures variance allocation; these coordinates distinguish erasure, rerouting, and rescaling.Erasure lowers R; rerouting preserves R while lowering U; rescaling preserves R and U while lowering A.Source: §2 · p. 3, Table 1 · p. 3
  • Invertible changes to activations can alter covariance and spectral statistics while preserving probe-based information and intervention responses, so spectral compression alone cannot identify information loss.Within an architecture, allocation remains a separately measured property of the trained solution.Source: §2 · pp. 3–4

3 Which Reward-Null Distinctions Are Protected

The reward defines which distinctions are decision-equivalent, but does not determine the fate of every reward-null distinction. The reference policy and state update protect some distinctions; optimization resolves the fate of the rest.

Which Reward-Null Distinctions Are Protected

  • For β > 0, KL-regularized reinforcement learning preserves the reference policy’s log-odds between equally rewarded actions, keeping decisions sensitive to reward-null directions when the reference distinguishes them.In the tied-token design, the reference is the model’s own next-token distribution and its within-class preferences follow the nuisance belief.Source: §3 · p. 4
  • In the tied-token design, the anchored policy’s causal quotient merges beliefs differing along Ninv, while Nvis remains represented and used, so the kernel’s share of output variance falls.The optimal policy depends on belief only through (bκ, b⊤E); centered log-probability variation within a fiber is untouched while between-fiber variation grows as 1/β.Source: §3 · p. 4
  • Supervised and unanchored objectives leave within-class behavior unconstrained, making the whole reward-null kernel free at the decision except for distinctions required by the recurrent filter.Without the anchor, any policy supported on the best class is optimal, and its within-class behavior may differ from the reference.Source: §3 · pp. 4–5
  • When observations entangle task and nuisance variables, a recurrent filter must carry reward-null distinctions needed to update task beliefs, whereas a transformer can recompute beliefs from context.The filter-closed partition is then finer than the reward quotient.Source: §3 · p. 4
  • Figure 2 separates kernel variance allocation, decision use, and representation; leftward movement marks rescaling, downward movement rerouting, and an empty marker erasure.Its marker halves show representation of next-token-visible and next-token-invisible components.Source: Figure 2 · p. 5

4 Experiments in Exact Worlds

In exact worlds, the experiments separately track whether reward-null information remains represented, is used by decisions, and occupies activation variance. Across objectives, compression usually changes allocation rather than erasing information, while anchors, recurrent updates, and optimization dynamics determine which distinctions remain protected or used.

4.1 Compression Is Not Loss

  • The kernel’s variance share falls by up to three orders of magnitude, while almost no representations disappear and next-token-visible information remains decodable above 0.9 of base without weight decay.The next-token-invisible component erodes from 0.57 to 1.00 of base as policy drift proceeds, stopping under an anchor and continuing slowly without one.Source: §4.1 · p. 5

4.2 The Anchor Decides Use

  • With an anchor, KL-RL, GRPO, and DPO preserve reference within-class log-odds; without one, policies can collapse each class onto one token while the component remains decodable.The unanchored GRPO use threshold lies within (0.003, 0.02], while representation and allocation vary smoothly through it.Source: §4.2 · pp. 5–6

4.3 Erasure Needs Pressure Beyond the Objective

  • Under 50,000-step weight decay, SFT and fresh-token objectives reduce the next-token-invisible component to 0.001–0.012 of base, while an anchored policy retains 0.64.The anchored policy also retains the visible component and reference log-odds; the stress test erases distinctions that no objective term needs.Source: §4.3 · p. 6

4.4 A Recurrent State Spreads Freedom and Protection

  • A GRU’s shared recurrent state spreads decision constraints across positions: it retains less next-token-invisible information under free-kernel objectives, but an anchored stress-test GRU retains 0.99–1.02 of it.After SFT, the GRU filter nearly stops using the invisible component although about half remains decodable; in the entangled world, the filter uses kernel information needed by future reward-relevant beliefs.Source: §4.4 · p. 6, Figure 3 · p. 6

4.5 What Allocation Costs

  • Of 172 runs whose kernel share fell below one-tenth of base, 153 retained R ≥0.9; lower nuisance variance nevertheless required more steps for a new readout.At a 200-fold variance reduction, the nuisance belief remained linearly encoded (R2 = 0.93) but was indistinguishable from an untrained network under the reported adaptation setting.Source: Figure 4 · p. 6, §4.5 · p. 7

4.6 Optimization Settles What the Reward Leaves Open

  • In separable worlds, the collective policy-gradient shift overtakes history-wise amplification before reward saturation; which mode wins depends on the pretrained representation at the query.Training only the head removes the collapse, while the exact expected gradient still collapses, implicating shared-representation updates rather than sampling noise; outcomes vary in entangled worlds and Mess3.Source: §4.6 · p. 7

5 Open Language Models

Open language models retain decodable in-context belief geometry despite late-layer spectral contraction. In controlled coarse-class GRPO, within-class behavior returns to the base policy with an anchor, while without one it is amplified rather than collapsing.

Released checkpoints

  • Across three base/post-trained checkpoint pairs, belief-state peak R2 changed by at most 0.05 and next-token-invisible-component R2 by at most 0.06 in all but three smallest-model cases, while later-half effective rank fell 12–19%.The invisible component’s variance share rose despite spectral contraction.Source: §5 · p. 7

Controlled coarse-class GRPO

  • With an anchor, within-class log-odds overshoot and return to reference within 1,000 steps at β ≥0.3, or within 10,000 steps at β = 0.04 in the continued run.Without an anchor, within-class log-odds are amplified ten to twenty times, with no run collapsing across both models and tested parameterizations and training lengths.Source: §5 · p. 7, Figure 5 · p. 8
  • Effective rank falls in every controlled run while kernel variance share rises under amplification and falls under the anchor, separating allocation changes from information loss.Unlike small transformers, the language model amplifies its existing in-context computation without an anchor; both systems return to reference log-odds with one.Source: §5 · pp. 7–8

6 Related Work

Prior work measures belief-state geometry and post-training changes, but often cannot identify which objective-irrelevant distinctions were lost, rerouted, or rescaled. This paper uses an exact reward quotient to assign those changes and examines anchored and unanchored behavior among tied outputs.

Belief states in sequence models

Post-training and representation change

KL-regularized reinforcement learning and tied outputs

  • The paper derives implications of KL-regularized preservation of reference proportions among equally rewarded outputs (Korbak et al., 2022; Rafailov et al., 2023) and contrasts unanchored outcomes with prior claims about the closest optimal policy (Shenfeld et al., 2026).Its unanchored runs leave that policy in different directions in small transformers and language models.Source: §6 · p. 8

7 Conclusion

The conclusion frames post-training as selecting a causal quotient of the pretrained belief state: reward defines decision-equivalence, while the anchor and state update protect some reward-null distinctions and optimization resolves the rest.

7 Conclusion

  • The reward defines decision-equivalence, the anchor preserves distinctions the reference policy uses, and the state update preserves distinctions needed by future reward-relevant beliefs.Optimization usually rescales reward-null information, reroutes decisions around it without an anchor, and erases it only under sustained pressure when no objective term needs it.Source: §7 · p. 8

Limitations

The three-fate ground truth is established only in small hidden-Markov worlds with one-step decisions that do not affect the context. Language-model evidence covers three checkpoint pairs from one family on synthetic streams, while effects on natural-task post-training remain unmeasured.

Limitations

  • The identifiable ground truth is limited to small hidden-Markov worlds with one-step decisions whose actions do not alter subsequent context.Language-model experiments cover three checkpoint pairs from one family and synthetic in-context streams; the size of effects on natural-task post-training is unmeasured.Source: Limitations · p. 9

A.1 The anchor identity and what it protects

The section distinguishes what KL anchoring protects from what supervised objectives and spectral statistics leave unconstrained. It also shows that filtering can require distinctions beyond the immediate reward quotient, while shared-parameter dynamics shape how tied actions change.

A.1 The anchor identity and what it protects

  • For equally rewarded actions, KL anchoring preserves reference-policy log-odds, including under groupwise standard-deviation normalization such as GRPO.The protection extends to objectives whose action signal enters only through reward.Source: §A.1 · p. 10
  • At the optimum, within-fiber log-probability variance matches the reference, while total output variance follows a + 2c/β + c′/β².The kernel’s share of output variance falls as β decreases in the main world.Source: §A.1 · p. 10
  • The supervised optimum depends on belief only through the class marginal and lies 19 to 33 times farther from the reference in KL than the anchored optimum in the canonical-token setting.The same class-marginal dependence holds for labels given by the Bayes action.Source: §A.1 · p. 11

A.2 Filter closure

  • Filter-closed equivalence requires reward vectors to agree now and after every continuation, and is strictly finer than reward equivalence in the entangled example.Exchanging the kernel component changes the reward-relevant statistic by 0.14 on average and flips the best class in 17% of cases.Source: §A.2 · p. 11

A.3 The spectrum

  • Scaling the kernel component by 0.03 with compensated readout leaves least-squares R2 at 0.954 but lowers ridge R2 to 0.139; projecting a direction can raise or lower effective rank.The probe comparison motivates whitening without a penalty, and projection’s effect depends on the removed direction’s spectral share.Source: §A.3 · p. 11

A.4 Tied actions under shared parameters

  • Without an anchor, separate-parameter gradient flow amplifies the reference-preferred tied token, whereas an anchor returns the within-class slope to 1.Natural policy gradient keeps the slope at 1.Source: §A.4 · p. 11
  • With shared parameters, only interaction features are driven and the collective same-symbol mode grows fastest; its base-model rate exceeds the history-wise rate by 104 times.A small KL anchor can undo collective collapse, with the intercept recovering before the slope.Source: §A.4 · pp. 11–12

B Worlds, Models and Training

The methods use controlled hidden-Markov worlds and pretrained models to measure belief information, policy use, variance allocation, and filter use separately. They specify post-training conditions and operational tests for each measurement.

B Worlds, Models and Training

  • The worlds pair a task variable with a nuisance variable, including separable and entangled settings, and insert queries without making the hidden state depend on the query or action.The main separable world includes a next-token-invisible nuisance direction; the control makes the whole kernel next-token-visible.Source: B Worlds, Models and Training · p. 12
  • Four-block transformers and two-layer GRUs were pretrained on fresh streams, reaching within 5 × 10^-4 nat of Bayes-optimal loss while belief and kernel components remained linearly decodable above baselines.Post-training compared SFT, DPO, GRPO, Dr. GRPO, KL-RL, and auxiliary conditions including weight decay and next-token loss.Source: B Worlds, Models and Training · p. 12
  • Kernel subspaces, fate criteria, and batch predictions were preregistered; two directional predictions were overturned and reported as findings.The overturned predictions concerned unanchored policy-gradient sharpening and recurrent-model retention of the free kernel.Source: B Worlds, Models and Training · p. 12

C Measurements

  • Representation was measured by whitened, unpenalized least-squares probes on belief coordinates, with held-out R2 and checks under random invertible activation maps.A frozen base-model probe distinguishes re-encoding from intact information recoverable by a refit probe.Source: C Measurements · pp. 12–13
  • Decision use was measured by twin histories with matching class marginals, using centered-log-policy differences and activation-subspace interchange interventions.The twin set contains 16,384 pairs; class marginals agree to 10^-9.Source: C Measurements · p. 13
  • Variance allocation was measured as the kernel-coordinate share of activation variance, alongside effective rank.The effective-rank measure is RankMe on the same activations.Source: C Measurements · p. 13
  • Filter use was tested by replacing a state’s kernel component with a twin’s and comparing later next-token distributions against the exact Bayes filter.The recurrent state is patched and rerun for GRUs; transformers are patched across all positions and depths up to the test position.Source: C Measurements · p. 13

D Additional Results in Exact Worlds

Across the tested worlds and architectures, the anchor preserves reward-null information’s decision use, while unanchored objectives can reroute it and prolonged weight decay can erase distinctions unseen by reward and prediction. Additional controls separate representation, use, and variance, and show that anchored training can re-encode retained information.

Other worlds and architectures

  • Anchored objectives keep R ≥0.97 across eight further world–architecture combinations, with γ near 1 across KL-RL, GRPO, and DPO.Across 144 runs, γ ranges from 0.93 to 1.02 for KL-RL and GRPO and 0.86 to 1.13 for DPO.Source: D Additional Results in Exact Worlds · p. 13

Erosion and weight decay

  • SFT reroutes the kernel in all 48 runs while retaining decodability; prolonged weight decay instead drives the next-token-invisible component toward erasure.Under SFT, Rinv falls from 0.69 after 10,000 steps to 0.001 after 50,000; adding next-token loss raises anchored retention from 0.64 to 0.79.Source: D Additional Results in Exact Worlds · pp. 13, 16

The anchor identity in trained networks

  • At β ≥0.04, trained anchored policies largely preserve within-class log-odds slopes, whereas unanchored slopes fall to 0.43 under GRPO and 0.12 under the exact gradient.The unanchored policies retain Rvis of 0.96–0.97 despite the changed within-class preferences.Source: D Additional Results in Exact Worlds · p. 13

Re-encoding under an anchor

  • Anchored training can rotate the kernel representation by 20–77 degrees, so base-model probes can miss information that refitted probes and interventions still detect.Refitted subspaces and probes recover 0.94–0.97 of the information and use.Source: D Additional Results in Exact Worlds · p. 13

Matched KL

  • At matched KL, unanchored runs have lower within-class preference than every anchored run despite similar next-token-invisible decodability.The comparison uses the KL reached by anchored runs for β between 0.01 and 0.1.Source: D Additional Results in Exact Worlds · p. 16

The recurrent model

  • Under objectives that leave the kernel free, the GRU retains less next-token-invisible information than the transformer, but next-token loss restores GRU decodability to 0.96–0.99.In the separable worlds, that loss does not prevent collapse in the transformer because ordinary-position next-token constraints cannot reach the query through attention.Source: D Additional Results in Exact Worlds · p. 16

Gauge control

  • A gauge rescaling changes the kernel’s variance share twenty- to a hundredfold without changing probe R2 or the interchange effect.The effective rank changes by a factor of 0.9 to 1.7.Source: D Additional Results in Exact Worlds · p. 18

Revival

  • A new nuisance-variable readout reaches 95% of attainable reward in 154–244 steps on the base model, but takes 943–1,932 steps or exceeds 2,000 after unanchored or fresh-action objectives.After SFT or when kernel share is below about one hundredth of base, it does not reach 95% within 2,000 steps.Source: D Additional Results in Exact Worlds · p. 18

D.1 The direction without the anchor

Without the anchor, policy-gradient training can shift within-class preferences even when reward is unchanged. The direction and timing of this shift depend on the pretrained representation, while controls locate the effect in shared-representation training rather than sampling noise.

Parameterization and budget

  • Low-rank adapters of ranks 4 and 16 collapse in all nine runs, and at one-tenth the learning rate the slope still falls to 0.50–0.60.At the lower learning rate, reward saturation occurs eight times later.Source: §D.1 · pp. 18–19

World dependence

  • Where the two modes grow at comparable rates at the base model, early training sharpens, but outcomes after 5,000 steps range from sharpening to partial collapse.The comparable-rate settings are the entangled world and Mess3.Source: §D.1 · p. 19

Post-training seeds

  • Across eight further seeds for each of three pretrained models, unanchored runs repeat the same outcome 48 times; anchored controls return to slopes of 0.96–1.00.In GRPO, the slope peaks at 1.09–1.18 before falling below 1, ahead of reward saturation.Source: §D.1 · p. 19

Selected token across seeds

  • In 17 of 18 pretrained-model, class, and estimator combinations, all eight seeds select the same token; GRPO and the exact gradient agree in all nine model–class combinations.The selected token is unrelated to the reference’s average preference in four of nine model–class combinations.Source: §D.1 · p. 19

Pretrained-model dependence

  • Seeds from one pretrained model end together while different pretrained models diverge, linking the final direction to the starting model rather than seed variation.Across seeds, the final slope varies by at most 0.26 within a model.Source: §D.1 · p. 19

E Language-Model Experiments

The language-model experiments assess in-context belief representations in released checkpoints and test anchor effects using controlled coarse-class GRPO. Belief information remains decodable, while within-class behavior and representation variance differ with the anchor and training setup.

Released checkpoints

  • The checkpoint analysis probes every residual-stream layer on streams from six hidden Markov models and tests the next-token-invisible component on held-out sequences.The protocol follows Balcells et al. (2026).Source: E Language-Model Experiments · p. 19
  • Peak belief-state R2 is 0.85–0.99 across the Qwen checkpoint pairs, and the next-token-invisible component’s variance share rises by 9–40%.A base model from another family was excluded because it did not converge to the Bayes predictor in context.Source: E Language-Model Experiments · p. 20

Controlled GRPO

  • In controlled GRPO, six single-token letters form three classes, and the reward depends only on class; the base model’s next-letter distribution serves as reference.Decisions occur at query markers in the second half of 2,048-token contexts.Source: E Language-Model Experiments · p. 20
  • The reported value returns to between 1.00 and 1.08 within 1,000 steps; at β = 0.04, it overshoots six- to ninefold before returning.Source: E Language-Model Experiments · p. 20
  • Both kernel components remain decodable in every run, while kernel variance share rises 1.1–2.8 times under amplification and falls to 0.33–0.49 under β = 0.3.Decodability is reported as R ≥1.0.Source: E Language-Model Experiments · p. 22

Controls

  • Unanchored runs preserve the marker representation’s mean while its class component grows 31–110 times; the small transformers instead remove most of the mean within 300 steps.In the 1.7B model, effective rank falls by 16–61%.Source: E Language-Model Experiments · p. 22

Related papers

Loading 2610.05292v1…