PaperlayerView exact revision on arXiv

Source-linked AI summary

Decodable In-Context State and Model Output Across Training

Manas Venkata Sai Ravulapalli, Samrath Singh Chadha

arXiv:2609.31401v1cs.LG

TL;DR

Prior work showed that probes can decode in-context bindings on model errors and that probe-guided steering can repair some errors, but the evolution of these measures across checkpoints was not established. This paper follows probe accuracy, model output, and steering across public checkpoints; probe accuracy rises during Pythia pretraining, while steering benefit grows later at two sizes. The results do not establish that these patterns explain downstream omissions.

  • Problem

    Prior work found decodable bindings on some incorrect outputs, but did not identify where ordinary model computation fails or how decodability, output, and intervention response change across checkpoints.

    Source: §1 · p. 1

  • Method

    The paper tracks binding probes, model outputs, and steering responses across frozen public pretraining and post-training checkpoints.

    Source: §1 · pp. 1–2

  • Results

    Probe accuracy rises across several Pythia pretraining sweeps, and probe-guided steering gains more all-trial benefit later in pretraining at two model sizes.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Whether the observed patterns explain omitted downstream actions remains an open question.

    Source: §1 · p. 2

    On some executed-code tests, the protocol cannot separate computed-state decoding from surface evidence, so the supported scope is declared bindings and a tracked-state control.

    Source: §7.5 · p. 10

Abstract

from arXiv · show

Prior work established that a probe can decode an in-context binding on model errors and that probe-guided steering can repair some of them. We follow probe accuracy, model output, and steering response across public pretraining and post-training checkpoints. Probe accuracy rises during Pythia pretraining, while probe-guided steering moves from negligible all-trial benefit to a larger benefit at two model sizes. Saved scores distinguish probe-correct errors with low and above-uniform model probability for the correct candidate. Oracle-target steering already repairs many early errors, but saved aggregates cannot separate target quality from intervention sensitivity. A held-out comparison of decoders trained on the final state or candidate logits finds no detected final-state advantage on late-checkpoint model errors. An information-theoretic counterexample explains why decodability on errors alone cannot establish discarded output information. The connection to downstream omissions remains open.

1 Introduction

The paper tracks whether in-context bindings are decodable, reflected in model output, and responsive to steering across checkpoints. It tests this distinction on binding tasks while treating downstream repository omissions as motivation, not an established consequence of binding errors.

1 Introduction

  • Earlier work found that a separate probe can decode the correct binding on some incorrect responses and that probe-derived steering can change the choice.Those results do not locate where ordinary model computation fails.Source: §1 · p. 1
  • Binding keeps an entity–attribute association intact while other tokens compete for representational resources, a structure relevant to coding agents and planners.Examples include keeping lock→balance distinct from mutex→queue and tracking which subtask belongs to which actor.Source: §1 · p. 1
  • Fixed-middle-layer probe accuracy rises across several Pythia pretraining sweeps, while probe-guided steering gains more all-trial benefit later at two model sizes.The connection between these patterns and omitted downstream actions remains open.Source: §1 · pp. 1–2
  • Repository-fix failures found gold files nearly as often as passing attempts (0.68 vs 0.81 recall) but edited fewer (0.55 vs 0.74), while controlled binding trials separately showed probe accuracy of 0.73 on gpt-oss-20B errors.The repository episodes and binding trials are separate experiments and do not establish a causal link.Source: §1 · p. 1, Figure 1 · p. 2

2 Setup

The study measures binding-task probe accuracy and model output on frozen checkpoints, using a present-set baseline and distinct probe protocols. It distinguishes decodability from output behavior, then tests whether final-state decoders outperform candidate-logit decoders on held-out model errors.

2 Setup

  • Probe accuracy is measured at the final query token against the present-set baseline 1/K, and the binding task resamples entities and obligations independently on each trial.The model is correct only when its top obligation-token logit matches the queried obligation.Source: §2 · pp. 2–3
  • The fixed-middle-layer Pythia probes use cross-validated regularization and disjoint test folds; above-baseline probe accuracy shows classifier-recoverable binding information, not why the model chose incorrectly.The older Pythia phase-plane sweep selected its best layer using test accuracy and is excluded from the developmental claim.Source: §2 · p. 2

2.1 Decodability and output are different tests

  • The paper distinguishes probe accuracy from output information: a fitted probe-versus-argmax comparison does not estimate information absent from candidate logits.The proposed comparison tests matched-trial decoder log loss; even a state-decoder advantage would remain subject to decoder capacity and sample size.Source: §2.1 · p. 3

2.2 Held-out decoder comparison

  • The candidate-logit decoder has lower log loss on early-checkpoint model errors, while the late all-trial difference is 0.096 with paired interval [−0.051, 0.238] (n = 250).Decoder capacity, 20 training examples per class, and template shift limit inference about the Bayes-optimal information difference.Source: §2.2 · p. 3
  • The late-checkpoint held-out comparison detects no final-state decoder advantage on model errors; its interval includes zero.The test uses one model seed and one held-out test template, and the interval resamples test episodes rather than training seeds or templates.Source: §2.2 · p. 3, Table 1 · p. 4
  • At the late checkpoint, the correct candidate ranks second in 43/193 native errors versus 12/232 at step 1,000, while median winner-minus-correct logit gap falls from 2.935 to 1.629.The error subsets differ across checkpoints, and rank and margin do not measure information absent from the full score vector.Source: §2.2 · p. 4

3 Probe accuracy changes over pretraining

Probe accuracy rises over pretraining at several larger Pythia scales, while the smallest scales stay near the present-set baseline. OLMo-2 summaries suggest a similar rise, but their unavailable generator limits reproducibility.

3 Probe accuracy changes over pretraining

  • The probe-correct, low-score group grows from 64 to 147 trials, while the probe-correct, above-uniform group grows from 11 to 78.These are group counts across checkpoints, not paired transitions; low score does not mean the full logits lack label information.Source: §3 · p. 4

3.1 Across the Pythia ladder

  • Full-state accessibility exceeds 1/K=0.167 at 1B (ρ=+0.89), 1.4B (ρ=+1.000), and 6.9B (ρ=+0.96), but is flat at 160M and 410M, ending at or below baseline.The full-state-minus-gradient-subspace contrast has ρ from +0.83 to +0.98, with exact p < 0.05 at all five usable scales.Source: §3.1 · p. 4

3.3 OLMo-2 summary rows

  • OLMo-2-1B and OLMo-2-13B probe accuracy rises and clears baseline on most late checkpoints, but the unavailable generator prevents protocol checks.OLMo-2-7B is under-powered and does not reach significance; these summaries do not establish an independently reproducible cross-family result.Source: §3.3 · p. 5

3.2 Scale dependence

  • At scales of 1B and above, Figure 3 shows curves separating from near-chance 160M/410M curves, with similar final accuracy within the larger group.Final probe accuracy among scales at or above 1B is 0.65–0.71.Source: §3.2 · p. 5, Figure 3 · p. 6

4 Depth and subspace comparisons

At Pythia-1.4B, the best- and final-layer probe trajectories separate during pretraining. At a fixed middle layer, gradient- and class-mean-subspace probe contrasts vary by scale and do not identify the model’s natural readout.

4.1 Depth separation

  • 0.317 is the monotonically opening separation between best-layer and final-layer probes at Pythia-1.4B.Final-layer decoding reaches 2.3× the 1/K present-set baseline without layer selection; the distinct trajectories do not show that information disappears at the final layer.Source: §4.1 · p. 6

4.2 Gradient and class-mean subspaces

  • At Pythia-1.4B, ρ = −0.893 for gradient-subspace minus class-mean-subspace probe accuracy, a falling contrast that does not replicate at 6.9B.The gradient directions are conditioned on the gold label and pass through the remaining network, so this comparison cannot identify the natural output mechanism.Source: §4.2 · p. 6

5 Repairability develops during pretraining

Probe-guided repair grows across Pythia pretraining checkpoints, including at 1.4B and 6.9B, while an independent implementation reproduces a rising trajectory. Saved arm aggregates show that early errors can already be repaired by oracle steering but do not separate target quality from intervention sensitivity.

5.1 Self-gated repair across checkpoints

  • At Pythia-1.4B, probe-target repair rises from 0.019 of failures early to 0.668 late, while a matched-norm wrong direction repairs 0.077 at the final checkpoint.Oracle-target steering repairs 239/298 early failures versus 56/298 for probe-target steering, and 241/271 versus 174/271 late; aggregates do not identify why probe-target repair grows.Source: §5.1 · p. 7
  • At Pythia-1.4B, all-trial accuracy changes from 0.255 to 0.238 at step 8,000 and from 0.323 to 0.753 at step 143,000; at 6.9B, it changes from 0.201 to 0.190 and from 0.321 to 0.597.These are descriptive checkpoint contrasts without paired trial-level uncertainty; the steering target is probe-derived and the selected layer changes across checkpoints.Source: §5.1 · p. 7
  • The saved arm-rate identity is an accounting constraint, and the files omit the paired targets and outcomes needed to separate incorrect probe targets from their conditional repair rates.The interventions coincide when the probe target is correct.Source: §5.1 · p. 7

5.2 Replication at 6.9B

  • At 6.9B, failure-set repair rises from 0.158 to 0.511 (ρ = +0.929, p = 6.7 × 10−3, n = 7), while at 410M probe accuracy stays near chance and repair remains limited.The reported 410M repair range is 0.058–0.150; probe accuracy limits information available to the intervention.Source: §5.2 · p. 7

5.3 Independent re-implementation

  • An independent implementation finds accessibility rising from 0.261 to 0.972 and a final-checkpoint recovery rate of 0.953 versus a 1/K = 0.167 baseline.Its onset-lag interval includes zero, and onset is left-censored because accessibility exceeds baseline at the first checkpoint, so the developmental claim concerns slope and asymptote.Source: §5.3 · p. 8

6 Post-training changes output and probe compatibility

Across post-training observations, output behavior and probe accuracy do not always move together, and frozen-probe performance falls more than refitted-probe performance. The checkpoint contrasts are observational and do not isolate individual training objectives.

6 Post-training changes output and probe compatibility

  • These post-training comparisons are observational contrasts across stages, not isolated estimates of individual training-objective effects.Source: §6 · p. 8

6.1 SFT, DPO, and RLVR on OLMo-2-7B

  • On OLMo-2-7B, failures fall from 194 to 58 per 400 at SFT and then to 49 and 48, while probe decodability jumps from 0.787 to 0.953 and plateaus.Probe and model posterior rankings become more correlated, with Spearman 0.367 → 0.667; the comparison does not establish which stage formed the representation.Source: §6.1 · p. 8

6.2 Per-step RL

  • Across 2,400 RL steps on an Olmo-3-7B-RL-Zero-Code branch, recall at K=10 falls from 0.590 to 0.510 while a held-out probe shows no trend.At late steps the model no longer prefers the correct obligation at that load; a non-significant probe trend does not establish constant encoded information.Source: §6.2 · p. 8

6.3 Frozen and refitted probes

  • Across GRPO steps, the frozen probe falls from 0.864 to 0.714, versus 0.864 to 0.792 for a probe refitted at each checkpoint.The 0.150 versus 0.072 declines show reduced compatibility of a fixed probe, but do not decompose geometric rotation and information loss.Source: §6.3 · p. 8

7 Computed state: the boundary of the protocol

The computed-state tests mostly do not establish probe accuracy above the present-set baseline on model errors, although tracked Python state provides a positive control. Surface-visible targets and low error counts constrain what this protocol can conclude.

7 Computed state: the boundary of the protocol

  • Most tested computed-state instruments do not establish probe accuracy above the present-set baseline on model errors.The substrates and controls specify a target that is internally produced, behaviorally evaluable, and absent from the prompt surface as the desired test conditions.Source: §7 · p. 8, Table 5 · p. 10

7.1 Tracked Python state (positive control)

  • On six models tracking Python variable values, probes recover the correct value on model errors at 5.8–9.3× the 1/K baseline and distinguish binding from recency by 5.5×.The recovered value is the model’s second-ranked token, and the probe never beats model logits (0 of 6).Source: §7.1 · pp. 8–9
  • During code-reward training, output accuracy falls while a base-checkpoint probe loses more accuracy than probes refitted at each step; post-training panels also show OLMo-2 failures falling and probe decodability jumping at SFT.The latter then plateaus; on the per-step RL branch recall at K=10 falls while a held-out probe shows no trend.Source: Figure 4 · p. 9, Figure 5 · p. 9

7.2 Executed code

  • On four frontier code models, traced final-value probes clear the vocabulary baseline in three models but remain at 0.9–1.2× the stricter 1/K present-set baseline, with all four established 95% intervals straddling it.The probe tracks executed value over recency (0.25 versus 0.12).Source: §7.2 · p. 9

7.3 Wrong-content control on executed code

  • On executed Python, the wrong-content control that fired on all 18/18 synthetic runs does not replicate because wrong content itself repairs on every executed model.The decoded arm still beats wrong content on the cleanest model, deepseek (0.49 versus 0.33), but the probe cannot separate what the model holds from what the prompt shows.Source: §7.3 · p. 9

7.4 Entity tracking and CRUXEval-O

  • On Boxes, probes decode initial declarations (0.54–0.87) better than tracked states (0.00–0.29); CRUXEval-O establishes no gap on any model and is censored by only 7–12 wrong trials.The surfaceliteral control decodes given input at 12.7–16.0× chance but computed output at only 2.7–5.1×.Source: §7.4 · p. 10

7.5 Surface presence

  • The supported scope is declared bindings and the tracked-state control; a decisive computed-state test needs surface-absent targets, adequate error counts, and a calibrated candidate-logit baseline.Some targets have a value token on the prompt surface, while some confound-clean targets have too few distinct outputs.Source: §7.5 · p. 10

8 Related work

Related work distinguishes internal decodability from model output across factual recall, reasoning, and action tasks. This paper’s contribution is to track probe-guided repair across training, while emphasizing that decodability and steering alone do not establish the unmodified model’s computation.

8 Related work

  • Prior studies find answers decodable but absent from output in factual QA and reasoning, while this task supplies new prompt bindings rather than testing stored-fact recall.Cheang et al. (2025) argue that some internal signals track knowledge recall rather than truthfulness.Source: §8 · p. 10
  • Earlier binding work established the probe gap and repair, while this paper adds checkpoint trajectories; Kwon (2026) instead tracks communicative intent and response across fixed models.Ravulapalli et al. (2026) also found no reduced output sensitivity to the binding direction on wrong trials.Source: §8 · pp. 10–11
  • Concurrent studies cover binding dissociation, context-defined representations, other probe-output gaps, silent agent-memory failures, and probe-based reasoning-step selection.Mao et al. (2026) examine different task trajectories; Schmied et al. (2026) compare a generated bandit calculation with a generated choice, which neither measurement alone identifies as the model’s natural route to action.Source: §8 · pp. 10–11
  • A probe alone does not show how the unmodified model uses decoded information, and unsupervised elicitation can select an unrelated prominent feature.Gurnee et al. (2026) use a context-averaged Jacobian lens, concept swaps, and ablations to test representational influence; that J-space method measures a different object from the supervised binding probe.Source: §8 · p. 11
  • Probe accuracy can precede behavior during training, but other reported trajectories are non-monotone, making the rises in this paper an empirical finding.The paper adds intervention and repairability measurements over the developmental axis.Source: §8 · p. 11
  • Fine-tuning-style updates can make frozen probes stale, so monitoring requires validation against behavior after updates; steering gains do not establish the unmodified model’s computation.The paper’s developmental control addresses part of the distinction by measuring whether probe-guided steering becomes more effective across sampled checkpoints.Source: §8 · p. 11

9 Conclusion

Across checkpoints, probe accuracy, output accuracy, and steering response follow distinct patterns, and the held-out decoder comparison detects no late final-state advantage on model errors. These results do not identify why the unmodified model errs or explain omitted repository actions.

9 Conclusion

  • The held-out decoder comparison detects no final-state information advantage over candidate logits on sampled late-checkpoint model errors.The conclusion also notes that an information-theoretic counterexample prevents inferring discarded output information from decodability on errors alone.Source: §9 · p. 11
  • Fixed-middle-layer probe accuracy rises across several Pythia pretraining sweeps, while self-gated steering grows from little early all-trial benefit to larger late benefit at two sizes.The binding-error gap and probe-guided repair were established in earlier work (Ravulapalli et al., 2026).Source: §9 · p. 11
  • Probe-correct failures can have low model probability for the correct candidate, and oracle-target steering repairs many early failures without separating target quality from intervention sensitivity.Saved aggregates do not establish that the probability threshold measures information loss.Source: §9 · p. 11
  • The intermediate-layer subspace contrast changes at Pythia-1B and 1.4B but trends oppositely and non-significantly at 6.9B.During code-reward training, output accuracy and frozen-probe accuracy fall on the sampled branch; OLMo-2 summaries suggest a similar pretraining trajectory but lack a committed generator.Source: §9 · p. 11
  • The checkpoint tests neither identify why the unmodified model chooses incorrectly nor explain omitted repository edits; the latter requires a causal intervention on a computed downstream task.OLMo-2 summaries lack a committed generator.Source: §9 · p. 11

A Supporting figure

  • Across sampled GRPO steps, the refitted probe shows no detected trend, while probe-guided steering recovers part of the measured output-accuracy loss.This intervention does not identify the cause of that loss.Source: Figure 6 · p. 13

B Compute resources

Experiments used frozen public checkpoints on a single node with eight H100-80GB GPUs and a CPU control-plane host.

B Compute resources

  • Experiments ran on one node with 8×NVIDIA H100-80GB GPUs and a CPU control-plane host.The paper reports wall-clock time rather than instrumented GPU-hours.Source: B Compute resources · p. 13
  • The paper trains nothing; probing, steering, and per-checkpoint clocks use forward/backward passes over frozen public checkpoints.Wall-clock time is dominated by weight download rather than compute.Source: B Compute resources · p. 13
  • Reported timing is wall-clock rather than instrumented GPU-hours, with weight downloads dominating compute time.Source: B Compute resources · p. 13

C Code and data availability

The source package has several provenance and artifact gaps that limit independent reproduction. The paper also notes that post-training can stale a probe and that the steering machinery requires white-box access to a frozen model.

C Code and data availability

  • Several analyses and data sources are absent from the public source package, limiting independent reproduction until those gaps are closed.Missing items include the DeepSWE-v11 attempt-level records, OLMo-2 developmental summary generator, post-training batteries and aggregator, and held-out decoder artifacts.Source: C Code and data availability · p. 13

D Broader impacts

  • A probe trained at one checkpoint can lose accuracy after model updates, so a monitor based on it needs re-validation against output behavior.Source: D Broader impacts · p. 13
  • The repair machinery requires white-box access to a frozen model and does not expand what an adversary with weights can already do.Source: D Broader impacts · pp. 13–14

Related papers

Loading 2609.31401v1…