What Pretraining and Midtraining Make Learnable from Rewards?

Chiwun Yang; Xiaoyu Li

Source-linked AI summary

What Pretraining and Midtraining Make Learnable from Rewards?

Chiwun Yang, Xiaoyu Li

arXiv:2609.38446v1Sep 29, 2026cs.LGcs.AIcs.CL

TL;DR

Rewards can identify correct answers without determining the computation needed for new inputs. The paper characterizes this ambiguity and follows source prediction and reward adaptation in the same parameters; its experiments distinguish acquired computation from task use, including an 82.61% versus 44.15% Sequential comparison.

Problem
Training rewards can leave different mechanisms indistinguishable even when they require opposite held-out answers, leaving unclear what prediction stages contribute to learning from rewards.
Source: §1 · pp. 1–2
Method
The paper characterizes reward-equivalent worlds and constructs finite sampled Adam paths from random initialization through task-independent source prediction and reward adaptation.
Source: §5 · p. 12, §8 · p. 22
Results
82.61% Sequential success with correct source and first-operation supervision versus 44.15% with a private-random source control; Memory retrieval preparation also supports task learning in an independent confirmation.
Source: §1 · p. 3, §7 · p. 21
Takeaways & Limitations
Understanding reward learning requires tracking both the computation available when rewards begin and how adaptation uses it.
Source: §7 · p. 21, §8 · p. 22
The no-source bounds assume a world-independent learner and initialization, while the experiments study continued source prediction from inherited weights.
Source: §7 · p. 21

Abstract

from arXiv · show

A reward can identify a correct answer while leaving the computation needed for new inputs undetermined. We study how pretraining and midtraining supply the information and computation that make reward adaptation effective. In sequential state computation and contextual memory, we characterize mechanisms that agree on every training reward yet demand different held-out answers. Task-independent source observations resolve this ambiguity. We construct finite sampled Adam paths from specified random initializations through source prediction and reward adaptation in the same parameters, proving how prediction acquires execution or retrieval and rewards learn their task-specific use. Experiments with pretrained Qwen2.5 checkpoints test this division of labor. Across eight worlds, Sequential models trained with correct source and first-operation supervision reach 82.61% success, versus 44.15% for a private-random source control. Memory replay preserves retrieval during reward adaptation, and an independent eight-world confirmation achieves 75.32% task success versus 49.86% after matched alternative-retrieval training. GSM8K and HotpotQA separate accuracy at reward entry, subsequent gain and final performance. Together, these results connect information acquisition, executable computation and reward-guided task learning.

1 Introduction

The paper asks what prediction stages contribute when rewards leave the computation for new inputs undetermined. It characterizes that ambiguity, constructs finite source-prediction and reward-adaptation paths, and measures the stages’ distinct roles in pretrained models.

1 Introduction

Source prediction trains composition or retrieval, while reward adaptation learns the task-specific use of that shared computation.Figure 1 depicts states and operations for composition, keys, values, and a query for retrieval, with rewards selecting task use.Source: §1 · p. 2, Figure 1 · p. 2
A two-parameter family of Sequential transformations preserves every training reward while half reverse held-out answers; in contextual memory, unobserved contexts impose an exact error floor despite additional rewards.Local source observations distinguish the Sequential possibilities.Source: §1 · p. 2
At least 7/8 held-out success follows Sequential source prediction and two REINFORCE updates; Memory reaches at least 19/20, while frozen random query/key attention stays at most 1/64 above chance.The finite sampled Adam paths start from specified random initializations and use the same parameters across source prediction and reward adaptation.Source: §1 · pp. 2–3
82.61% Sequential success versus 44.15% for a private-random source control; independent Memory confirmation reached 75.32% versus 49.86% after alternative-retrieval training.The experiments also distinguish accuracy at reward entry, reward-stage gain, and final accuracy on GSM8K and HotpotQA.Source: §1 · p. 3
Qwen2.5 experiments separately probe acquired computation, retention, and task-specific use: correct-source training acquires execution or retrieval, and Memory replay preserves retrieval during reward adaptation.The Sequential comparison uses correct source and first-operation supervision; an independent single-world comparison begins reward learning at zero task success.Source: §1 · p. 3

2 Related Work

Related work is organized around information supplied by training stages, optimization into computation, and reward use of acquired computation. The paper distinguishes its observation interfaces, learning paths, and evaluation targets from those in prior studies.

2 Related Work

The paper frames related work around information supplied by each training stage, how optimization turns information into computation, and how rewards use acquired computation.These are the three organizing questions for the section.Source: §2 · p. 3

2.1 Information supplied by observations and feedback

Unlike query-based identification and other observation models, the paper characterizes reward-equivalent transition and task transformations and the source observations that distinguish held-out predictions.Terminal correctness feedback differs from expert labels; in the contextual binary-action task, correctness plus the chosen action reveals the binary label.Source: §2.1 · pp. 3–4

2.2 Learning across source and adaptation stages

Prior staged-training work varies in what its intermediate data reveal; here, source laws leave the downstream task rule unspecified, and finite sampled updates follow both training stages from random initialization.The cited comparisons include parity learning with chain-of-thought verification and graph backtracking from a near-world-law model.Source: §2.2 · pp. 4–5

2.3 Acquisition and reuse of representations

Related studies examine transferable features, sequential computation, and memory, while this work tracks source-acquired computation into reward learning and tests execution on generated prefixes.Its contextual result follows acquisition and retention of the information needed during feedback.Source: §2.3 · pp. 5–6

2.4 Reward learning, empirical comparisons, and capability interfaces

Reward-learning studies use varied entry conditions, objectives, and success events; this paper separates access to successful trajectories, acquisition of computation, and reward-dependent adaptation.The comparison also distinguishes task-specific computation and verification interfaces such as solvers, search, theorem proving, and tools.Source: §2.4 · pp. 6–7

3 Framework and Preliminaries

The framework separates a reusable mechanism from a task rule and gives the learner task-independent source records before terminal-reward adaptation. It specifies the information, feedback, and evaluation conditions for sequential and contextual tasks.

3 Framework and Preliminaries

A world pairs a reusable mechanism with a task rule; the learner first predicts from task-independent source records and then adapts the same parameters using terminal correctness rewards.In the sequential family the mechanism is a transition table and the task rule is an initial operation; in contextual memory they are a retrieval map and a context–cue decision rule.Source: §3 · p. 7
The learner and initialization are chosen independently of the fixed world; source observations reveal the mechanism but not the task rule, and feedback scores sampled candidates on fresh training prompts.The learner receives neither test feedback nor intermediate correctness labels.Source: §3 · p. 7
Sequential source records place the displayed suffix before the first control and state sequence, but states evolve in execution order; records are unfiltered by downstream success and carry no success labels.The task binding may appear as a token, but its downstream role is hidden from the source law.Source: §3 · p. 8
Source training minimizes next-token cross-entropy, while the verifier returns terminal reward only when the generated answer satisfies grammar, public-coordinate, and endpoint conditions.The verifier does not require the chosen first control to equal the hidden task binding or the private history to be faithful.Source: §3 · pp. 8–9
Evaluation distinguishes endpoint success, task utility, first-transition execution, and public-coordinate or grammar violations; memory evaluation tests delayed retention of an initial bit across a long suffix.The resource vector counts source tokens, verifier queries, generated tokens, and optimizer updates.Source: §3 · p. 9

4 Information and a finite neural learning path

Training rewards can leave mechanisms indistinguishable even when they require opposite held-out answers, while source observations expose the transitions that separate them. A finite neural learning path uses source prediction to acquire reusable execution and reward updates to learn its task-specific binding.

4.1 What training feedback cannot identify

Reward-equivalent worlds can demand opposite answers on every test prompt, limiting reward-only learners to at most 1/2 average test success even after unlimited queries.Source records distinguish the worlds by exposing single transitions before their effects cancel along a training sequence.Source: §4.1 · pp. 9–10

4.2 A complete neural learner in the same family

Two reward updates raise all three metrics by at least 3/4 while keeping format violations at the reset-noise level.The source model already executes, but its success is of order 1/q because it guesses the binding.Source: §4.2 · pp. 10–11
An unregularized source near-minimizer can preserve exact conditional transitions yet emit EOS on a chosen suffix, so low source loss alone need not yield a usable executor.In the canonical class, the ridge objective has a unique optimum that controls transition errors and token-class masses together.Source: §4.2 · p. 11
Source prediction teaches execution of transitions, while reward training selects the initial operation and transfers the learned transitions to held-out suffixes.The finite path starts from specified random initializations and uses the same parameters across stages.Source: §4.2 · pp. 10–11
Finite source-minibatch SGD followed by sampled policy-gradient updates also reaches 7/8 final success and improves all three metrics by 3/4.The result uses unregularized source training, a predetermined source endpoint, and separately chosen horizons, block rates, and budgets.Source: §4.2 · p. 11
For the specified two-world gauge experiment, initial-law overlap and the chance of observing odd training prompts determine the optimal endpoint error.The passage distinguishes these information bounds from the separate optimization guarantee for the finite neural path.Source: §4.2 · p. 12

5 Learning the representation that rewards use

The contextual memory results separate source-learned retrieval from a task rule learned through rewards. Finite training acquires retrieval and achieves strong held-out success, while freezing attention under matched schedules leaves performance near chance.

5.1 A contextual memory task

Source prediction trains retrieval without touching the downstream Boolean rule, while rewards train the decision made with retrieved information.The source law is independent of the contextual signs and cue vector that specify the downstream rule.Source: §5 · p. 12, §5.1 · p. 13

5.2 What the two observation channels identify

Rewards identify the contextual-sign and orientation products and the shared cue vector up to a common sign, while source geometry identifies each orientation.Flipping an orientation reverses labels on equal-sign test-amplitude configurations in that context.Source: §5.2 · pp. 13–14

5.3 Finite acquisition and feedback learning

At least 19/20 held-out success follows source training that raises matching from at most 0.51 to at least 0.9; freezing query/key parameters instead limits success to at most 1/2 + 1/64.The intervention uses the same schedules and budgets, with every head factor still trained.Source: §5.3 · pp. 14–15
The learned policy is input-dependent: reversing the cue sign reverses the correct action at the same context and query, unlike a constant action with 1/2 success.The learned coefficient shares the cue rule across contexts and combines it with the contextual sign.Source: §5.3 · p. 15

5.4 Training resources

The training-resource accounting tracks source records, verifier queries, sampled actions, and optimizer updates, pairing finite budgets with information bounds.The learned and frozen runs receive the same budget.Source: §5.4 · p. 16

6 Experiments

The experiments use source midtraining of pretrained Qwen2.5 checkpoints before reward adaptation, measuring computation acquisition, retention, task selection, and public-task performance. Correct-source Sequential training and task-relevant Memory retrieval support stronger reward learning in their respective comparisons, while public-task results distinguish reward-entry accuracy, subsequent gain, and final performance.

6 Experiments

The Qwen2.5 experiments compare correct source with matched structural controls and measure acquisition, retention, and task-specific use separately.A/B reward branches share source checkpoints and are evaluated on fixed held-out records; source costs differ between Figure 4 panels.Source: §6 · p. 16, §6.2 · p. 17, Figure 4 · p. 18

6.1 What computation does source training acquire?

Correct-source Sequential models generate 87.74% strictly complete trajectories versus 1.93% for the private-random control; correct-source Memory reaches 99.76% first-field accuracy.The Memory wrong-retrieval comparison is 51.95% first-field accuracy; its joint original/value-flip accuracy is 99.41% versus 5.81%.Source: Figure 3 · p. 17, §6.1 · p. 17

6.2 Do rewards select and retain the computation?

With first-operation supervision, correct-source Sequential success rises from 4.83% to 82.61%, versus 44.15% for control.The paired final-score advantage is 38.46 pp; task-selectivity advantage is 67.57 pp.Source: §6.2 · pp. 17–18, Figure 4 · p. 18
In eight new Memory worlds, selected retrieval reaches 75.32% success versus 49.86% for alternative retrieval after a shared correct-source start.The paired accuracy advantage is 25.46 pp, positive in every world.Source: Figure 4 · p. 18, §6.2 · p. 18
With correct-source Memory, replay retains 92.21% joint original/flip-pair accuracy versus 0% under pure RL after 8,192 queries.Replay adds 131,072 source targets in 128 CE updates; the task-selectivity interval under replay spans zero.Source: Figure 3 · p. 17, §6.2 · p. 18

6.3 How does source training change the subsequent reward trajectory?

On GSM8K, source+RL reaches 62.05% versus 62.32% for direct RL; on HotpotQA, it reaches 45.59% versus 47.60%.Source training changes reward-entry accuracy and subsequent gains in different directions across the two tasks.Source: §6.3 · p. 20, Table 2 · p. 20

7 Discussion

The discussion connects source-acquired computation to its retention and reward-guided use, while distinguishing these stages from accuracy already present at reward entry. It also clarifies which information and training settings the interventions address.

7 Discussion

Source observations distinguish mechanisms left indistinguishable by training rewards, and sampled rewards then teach their task-specific use of acquired execution or retrieval.The sequential construction uses two updates, while the contextual construction has an explicit finite budget; acquired computation applies to held-out inputs.Source: §7 · pp. 20–21
The interventions isolate information access, representation learning, or continued source training, while differing in source data and initialization assumptions.The no-source bounds assume a world-independent learner and initialization; public-task domain-related text without downstream labels differs from the controlled families’ rule-independent source laws.Source: §7 · p. 21
75.32% task success with task-relevant retrieval versus 49.86% after matched alternative-retrieval training in the independent Memory confirmation.Both routes inherit correct source information and receive the same added training budget.Source: §7 · p. 21
Final accuracy combines accuracy at reward entry and gain during reward adaptation, and these effects can move in different directions.Reporting entry, gain, and final scores locates improvement in the training sequence; controlled-task probes identify available computation at each stage.Source: §7 · p. 21

8 Conclusion

The paper concludes that source training can reveal mechanisms left ambiguous by rewards and build the computation needed to use them. Its experiments distinguish computation acquisition, retention, and task-specific use.

8 Conclusion

The conclusion is that pretraining and midtraining can reveal reward-ambiguous mechanisms and build computation that reward adaptation uses for task learning.The paper connects this account to sampled-Adam paths for execution and retrieval.Source: §8 · p. 22

A.1 Output supervision and source replay

Memory replay adds 1,024 targets from complete source records previously observed by that route after each reward update, sharing its Adam state and learning rate.The replay branches use 128 cross-entropy updates and 131,072 additional source targets.Source: §A.1 · p. 30

A Experimental details and supplementary results

Across the experiments, worlds are the independent units; A/B branches are averaged within world before paired effects and intervals are computed.Evaluation uses fixed held-out records, with reward outcomes measured through 8,192 queries.Source: §A.1 · pp. 30–31

A.1 Output supervision and source replay

82.61% versus 44.15% success: correct-source Sequential action models outperform the private-random source control across eight worlds.The paired difference is 38.46 percentage points (95% interval [19.78, 57.15]).Source: §A.1 · p. 31, Table 5 · p. 32
92.21% joint original/flip-pair accuracy is retained under correct-source Memory replay, versus 0% under pure RL.First-field accuracy is 96.24% versus 49.66%; task-specific advantage remains unresolved at this budget.Source: §A.1 · p. 32

A.2 Independent Memory confirmation: retrieval that rewards can use

The independent Memory confirmation changes which retrieval computation receives additional source training while both routes inherit correct source information.The selected route targets the task-relevant memory slot; the other route targets the other slot.Source: §A.2 · p. 37
75.32% versus 49.86% held-out success: selected retrieval outperforms other retrieval, with both routes entering reward training at 50.00%.The paired accuracy difference is 25.46 percentage points (95% CI 20.10–30.81).Source: §A.2 · p. 38, Table 6 · p. 39

A.3 Independent Sequential parameter-plane measurement

This section measures how source training and reward adaptation affect sequential execution and task selection, and how well a shared parameter plane reconstructs trained models, using controlled experiments and evaluation protocols. It also defines secondary HotpotQA and external-transfer result scopes and uncertainty conventions without reporting outcome values.

A.3 Independent Sequential parameter-plane measurement

48.24 percentage points over direct RL and 48.73 over the private-random source control were the correct-source route’s final advantages on 1,024 held-out records in world 508.The correct-source route had 0% task success before RL and 100% after 1,024 queries; this single-world comparison is separate from the eight-world tests.Source: §A.3 · pp. 39–40, Figure 15 · p. 40
On 256 common inputs, actual/reconstructed success was 0.00%/100.00% at the correct-source endpoint, 0.00%/52.73% at the private-random endpoint, and 53.91%/19.14% at the direct-RL endpoint.The rank-two plane uses displacements from the shared format checkpoint across source and reward checkpoints; because later RL checkpoints enter the basis, source reconstructions include directions learned during reward adaptation.Source: §A.3 · pp. 40–41

A.4 Eight-world source acquisition and reward adaptation

Across eight worlds, correct-source minus control differences were 87.50 percentage points for Sequential trajectories, 44.48 for Memory original records, and 88.28 for Memory intervention pairs.The 95% paired t intervals were [62.89, 112.11], [31.92, 57.05], and [60.57, 115.99], respectively; these source probes are distinct from downstream task selectivity.Source: §A.4 · p. 41, Figure 16 · p. 42
Neither eight-world correct-source versus matched-control task-selectivity interval excludes zero after 8,192 reward queries.The comparison averages eight world-level differences; Table 7 reports the effects and intervals, and Figure 18 visualizes the contrasts.Source: §A.4 · p. 41, Table 7 · p. 42, Figure 18 · p. 43
Correct-source Sequential complete-trajectory accuracy changed from 89.60% to 88.94% during reward adaptation, while correct-source Memory first-field accuracy fell from 94.38% to 50.34%.Memory relevant-pair accuracy fell from 89.01% to 0%, despite public-format accuracy remaining 100%; 26 of 32 Sequential reward branches made no positive-reward updates, while all 32 Memory branches made nonzero updates.Source: §A.4 · p. 42, Figure 19 · p. 44

A.6 Task definitions and observation permissions

The controlled task definitions pair sequential hidden-binding tasks with contextual-memory retrieval tasks, and use Qwen2.5-0.5B and Qwen2.5-1.5B models with full-parameter training.Sequential and memory source objectives differ, while reward branches share source checkpoints and use reset Adam moments; the implementation specifies sampled REINFORCE updates and fixed held-out evaluations.Source: §A.6 · p. 46, §A.7 · p. 46

A.8 Alternative routes and controls

In the earlier three-world shared-source comparison, paired task selectivity was −0.10 percentage points with interval [−0.42, 0.22].The comparison tests task-specific selection across opposite-reward branches; together with conditional private accuracy, the passage characterizes shared public execution without measurable preference for the branch’s reward task.Source: Figure 20 · p. 45, §A.5 · p. 45, §A.8 · p. 48

A.10 Public-task adaptation and feedback

The public-task comparisons use five paired seeds per task for GSM8K and HotpotQA, comparing direct, domain-source, and shuffled-source reward routes.The passages specify fixed endpoint evaluations, paired seed-level uncertainty, conditional item-bootstrap intervals, and secondary metrics, but provide no public-task outcome values in this part.Source: §A.10 · p. 49, §A.11 · p. 49, Figure 21 · p. 50

A.11 Secondary public-task metrics and external transfer

Table 12 reports HotpotQA greedy success by question-type subset, using normalized exact match percentages with seed-level 95% Student-t half-widths.The supplied caption does not include the table’s outcome values.Source: Table 12 · p. 51
Table 13 covers external-transfer and verification-selection results; Bo16 means best-of-16 selection with test-time verification, and BBH logic/tracking denote three-object logical deduction/object tracking.Its caption specifies training-seed 95% Student-t half-widths and item/template-bootstrap intervals for a fixed model.Source: Table 13 · p. 52
Bracketed base intervals bootstrap prompts or GSM-Symbolic templates conditional on one fixed model; GSM-Symbolic averages template means equally across ten variants per template.A dagger marks a fixed-base best-of-16 estimate.Source: §A.11 · p. 52

A.12 Endpoint decomposition and complete controlled-task results

Correct-source reward adaptation improves greedy success mainly through acquired public execution, while conditional private-bit accuracy remains near chance; direct and format controls produce no reward-bearing trajectories.

A.12 Endpoint decomposition and complete controlled-task results

The controlled Sequential comparisons used eight paired worlds, 220 source-prediction tokens, and 8,192 reward queries per route.Correct-source models obtained 242–584 positive training rewards per world, whereas direct RL and format controls obtained none.Source: §A.12 · p. 53
Correct-source greedy success rose from 3.30% to 41.67%, while wrong-source success rose from 3.94% to 38.15%.The two routes share public transition structure, but the wrong source has a different private transition table.Source: §A.12 · p. 53
Correct-source outputs had valid public paths on 96.22% of prompts and correct public endpoints on 83.25%, but conditional private-bit accuracy was 50.12%.The private bit was the largest correct-source error category at 41.58% of all prompts; wrong-source conditional private-bit accuracy was 49.87%.Source: §A.12 · p. 53
Direct-RL outputs were malformed, while format-source outputs were syntactically valid on 97.75% of prompts but lacked consistent public trajectories; both controls received zero positive rewards.Across the reported direct, format-source, and Memory direct runs, all 128 reward-loss values were zero.Source: §A.12 · p. 53
Under sampled decoding, source-trained sequential models exceeded direct RL by 5.48 ± 1.68 points at the endpoint, while correct-source reward-stage gain was 0.31 ± 0.55 percentage points.The corresponding correct-source gain under greedy decoding was 38.38 ± 5.21 percentage points; sampled evaluation used one answer per prompt at temperature one.Source: §A.12 · p. 57, Figure 22 · p. 57

A.13 Alternative routes, selection and external transfer

Results vary with source budget, initialization and adaptation route, while acquisition probes show that format-stage timing affects Sequential state prefixes and Memory intervention sensitivity.

A.13 Alternative routes, selection and external transfer

41.99% greedy success is reached at 220 source tokens, when every run receives positive reward; at 218 and 219 tokens, none does.The access to successful trajectories switches on within a factor of two in source budget.Source: §A.13 · pp. 57, 59, Figure 24 · p. 59
At 1.5B parameters, correct-source and wrong-source routes reach 26.43% and 22.27%, while direct RL stays at zero; interleaving source updates with RL retains 38.57%.In the three-world comparison, no random-initialized route attains sequential success under this budget and protocol.Source: §A.13 · p. 59
On separate 256-question sets, best-of-16 scores 22.1% after domain source versus 44.5% for the fixed base on GSM8K, and 50.7% versus 45.7% on HotpotQA.Source-plus-RL reaches 53.0% on GSM-Symbolic versus 54.5% after direct RL; BBH results vary by task and route.Source: §A.13 · p. 59

A.14 Source-acquisition probes and public source dose

The acquisition study compares Sequential state targets with private-bit-weighted targets and Memory all-token with future-bit objectives at 222–224 source tokens.An alternative Sequential representation adds an explicit length boundary and local-to-complete-trajectory curriculum; an alternative Memory representation uses discrete keys and values.Source: §A.14 · p. 62
Format training supervises complete public-format answers on records with independent random private labels, so it reveals neither the target mechanism nor task binding.Sequential receives 128 format updates per stage and Memory 32; Adam moments reset at stage transitions.Source: §A.14 · p. 62
At 224 source tokens, every Sequential model places END correctly on all 512 prompts after the format stage, while state-prefix counts fall to 6–14/512.Before format training, seven of eight original-representation 222-token endpoints already generate at least 511/512 correct prefixes, but no Sequential model emits END.Source: §A.14 · p. 62, Table 31 · p. 64
After the format stage, Memory accuracy means fall to 52.54% for original-representation all-token training and 3.91% for successful relevant-flip pairs.Discrete future-bit means change from 85.35%/68.16% to 64.45%/33.79%; the passage links random-label format supervision to input-independent private predictions as a consistent explanation.Source: §A.14 · p. 64

B Reading map and theorem interfaces

This section maps the paper’s information and optimization results and gives information-theoretic boundaries for identifying sequential worlds and mechanisms. It distinguishes what source observations, terminal observations, and reward feedback can reveal, including how contextual source observations resolve reward-equivalent ambiguity for held-out inputs.

B Reading map and theorem interfaces

The appendix maps reward equivalence and source-information results to their theorem and proof locations, then identifies separate sections for shared network bounds, finite Adam paths, SGD, costs, and experiments.It distinguishes source records, prediction targets, verifier queries, and optimizer updates as separate resources.Source: B Reading map and theorem interfaces · pp. 64–65

C Information supplied by source and reward observations

In both task families, a world combines a reusable mechanism with a downstream rule, and learnability depends on initial information, reward-feedback transcripts, and the held-out decision.For no-source lower bounds initialization is world-independent; arbitrary pretrained weights instead count as initial information.Source: C Information supplied by source and… · p. 65

C.1 Mechanisms with identical training rewards

In the sequential model, two worlds are reward-equivalent exactly when their transition and binding differences have the stated two-bit form, and their targets differ on every test prompt exactly when d = 1.The endpoint equality also matches the full terminal-verification oracle.Source: §C.1 · p. 65, §C.2 · pp. 66–67

C.2 Exact identification from source and terminal observations

At fixed δ = 1/16, the low-source learner uses O(log m) raw source tokens and O(m^2) queries, with zero gradient updates; its query coordinate is minimax-optimal in the stated regime.The result assumes L = ⌈216 ln(2m)⌉ and m ≥ 512 for the lower bound; the source-token charge includes a mandatory whole record, distinct from the constant-order transitions needed to decode the two residual bits.Source: §C.2 · p. 66
The exact-identification procedure first identifies the binding coordinate, uses fresh reward replies to form binary equations, then uses independently stored source records to recover both residual gauge bits.On successful recovery it yields canonical execution and the target first state on every test prompt; the source records are retained but not used in the feedback calculation.Source: §C.2 · pp. 67, 70
With O(log m) source records, the sequential learner uses O(m²) queries, while worst-world risk at most 1/16 requires Q ≥ 5m²/896 − nrecH − 1.This matches the query order in the stated logarithmic-source regime.Source: §C.2 · p. 72

C.3 A finite-budget boundary for a known pair of worlds

In the known-pair experiment, even prompts give no information, while one odd prompt identifies the world; the exact risk boundary is attained.The all-even event has probability (1 − ε)^Q; the boundary accounts for initial observations and applies to charged, exogenous prompts, not arbitrary unknown-table learning.Source: §C.3 · pp. 72, 73, 76

C.4 Information supplied by contextual source observations

Contextual source observations resolve reward-equivalent worlds that differ by independent context orientations, although a global sign leaves predictions unchanged.With source observations alone, minimax error remains 1/2 for every Nobs; ⌈K log(K/ϵ)⌉ iid reward queries attain error at most RNobs + ϵ/2, where K = nctx2^k.Source: §C.4 · p. 76

C.5 The information supplied by source observations

In contextual memory, worlds with identical training rewards can require different held-out answers, while exact source observations reveal visited context orientations. The section characterizes the resulting minimax risk and gives finite feedback-query attainment, with exact risk defined as an infimum over query budgets.

C.5 The information supplied by source observations

An exact source record reveals information about its own context, not other contexts’ orientations or the task parameters; visited orientations are recovered almost surely.Source: §C.5 · pp. 77–78
Training-reward-equivalent worlds can reverse equal-sign held-out labels, while opposite-sign test labels remain invariant.Source: §C.5 · p. 77
Worst-world test error is at least F_Nobs = 1/4(1−1/nctx)^Nobs, while finite feedback budgets achieve at most F_Nobs + ε/2, making minimax risk the infimum F_Nobs.No fixed finite budget is asserted to attain the infimum exactly.Source: §C.5 · pp. 77–79
With ⌈K log(K/ε)⌉ fresh iid binary correctness queries, a finite-table learner reaches expected test error at most F_Nobs + ε/2 without optimizer updates.Source: §C.5 · p. 78
Source observations alone leave source-only minimax expected test error at 1/2 for every source count.Source: §C.5 · p. 78

D.1 What additional source data can replace

The section examines what source data can replace in learning sequential mechanisms, distinguishing feedback-replaceable mechanisms from task information that source observations cannot provide. It also establishes optimizer and derivative foundations for finite Gaussian-network models, including initialization and local feature and derivative controls.

D.1 What additional source data can replace

With little source, feedback learns the observable transition class while source resolves remaining ambiguity; abundant source can instead teach the mechanism first, leaving binding search.These routes distinguish task identification from learning a neural computation.Source: §D.1 · p. 79

D.2 What remains for feedback and its alternatives

Source-only decoding has at most 1/q sequential endpoint success under a uniform binding and source-only minimax error 1/2 in the contextual family; task labels or expert SFT can supply missing information.More source prediction cannot identify the task parameters absent from source observations.Source: §D.2 · p. 80
Reward-weighted sampled likelihood matches the baseline-zero REINFORCE update on the same batch and optimizer state, whereas off-policy passes or accepted-sample normalization define different updates.Verification-based selection at deployment additionally requires test-verifier access and generation cost.Source: §D.2 · pp. 80–81
A frozen source-informed controller can search bindings with fresh-prompt verifications and then execute exactly, while Adam adapts the source model’s parameters using sampled trajectories.Both use the same acquired transition information but have different query and computation costs.Source: §D.2 · p. 81

E Shared Gaussian network foundations

The shared Gaussian-network initialization and derivative statements support both sequential optimizers, whose objectives, rates, stopping rules, and probability allocations are established separately.The complete raw architecture includes every displayed matrix, embedding, and bias with positive rates specified in the optimizer applications.Source: E Shared Gaussian network foundations · p. 81, §E.2 · p. 86

E.1 Finite Gaussian initialization

Finite Gaussian initialization yields a hard-lag encoder frame and a concentrated realized coefficient operator; coverage and separated bias winners enable finite-temperature soft attention to approximate hard lag selection.The soft-versus-hard feature estimate holds uniformly over input strings, including repeated padding and wrong generated tokens.Source: §E.1 · pp. 82, 83, 85

E.2 Uniform derivatives of the complete block

When block matrices, biases, and embedding columns have norms at most M, the complete raw block has feature norm at most Araw and parameter-derivative norm at most D.The bounds use the concatenated Euclidean/Frobenius metric.Source: §E.2 · p. 86
The complete-block derivative bound includes direct-factor and through-z terms, while the feature bound uses the feature and block norm bounds.The accompanying bound also states ∥˙α∥1 ≤ N∥˙score∥∞.Source: §E.2 · p. 86

E.3 The nonlinear normalized member

The normalized member retains the raw attention and bilinear blocks, adds a trainable gain initialized to ones, and updates backbone blocks and gain at a shared positive rate.The readout uses a faster rate; the nonlinear RMS-type normalization has a positive denominator offset and changes features and derivatives away from the constant-norm ideal feature manifold.Source: §E.3 · p. 87
Within a radius-one displacement from initialization, the gain satisfies ∥g∥∞≤2, and the motion bound includes both gain and original blocks.The feature and concatenated derivative bounds combine normalization bounds with raw bounds for the original parameters and the coordinate gain.Source: §E.3 · p. 87

F.1 Finite source learning and sampled Adam improvement

This section develops finite source learning with sampled Adam and bounds how source acquisition and feedback updates support held-out behavior. It connects source prediction to execution through transition accuracy and establishes source-policy, resource, parameter, and probability guarantees using shared-initialization and derivative bounds, width, and realized-row results.

F.1 Finite source learning and sampled Adam improvement

With probability at least 1 − δ, source stopping succeeds and the theorem gives a simultaneous held-out-prompt guarantee after exactly two feedback Adam updates.The source stage uses at most Ts batches of Bs records and Ts − 1 updates; feedback uses two updates on sampled trajectories.Source: §F.1 · p. 88

F.2 Applying the shared bounds to Adam

The shared initialization bounds give the normalized member an initial feature discrepancy of at most 3εsrc/8, while raw or normalized derivative envelopes apply throughout the Adam-controlled radius-one backbone ball.The derivative bounds cover the full ball maintained by the Adam displacement bounds.Source: §F.2 · p. 88

F.3 Finite source acquisition by coupled-regularized Adam

When source excess loss is at most c^2(q − 1)^2/(32q^2H^2N), both learned transition parameters are within c/(8H) of a = 1 − c/H.This links source prediction accuracy to execution of the learned transitions.Source: §F.3 · p. 91
The sampled source Adam procedure stops within Ts inspections with probability at least 1 − δbatch, achieving source-logit error below h/2 on legal patterns without verifier queries.It uses at most BsTs fresh records, NBsTs targets, and Ts − 1 updates, with the error guarantee covering unseen words and generated state histories.Source: §F.3 · p. 94
The width requirements make feature and head widths instantiable without self-reference, with source-parameter envelopes independent of d.The lemma states that the explicit bounds ensure the feature and Gram-accuracy conditions.Source: §F.3 · pp. 96–97
Conditioned on coverage and the actual Gram event, the realized maximum feature-row norm is bounded by Crow except with conditional probability δ0.The row bound does not assert independence of the realized feature rows; the readout-coordinate event receives a separate δ0 failure allocation.Source: §F.3 · p. 97

F.4 Two sampled Adam updates after source training

This section studies how source training supplies parity and correct-slot matching that sampled reward-feedback updates use to learn a task-specific rule, characterizing the guarantees and conditions under which two updates remain close to a fixed-executor reference. It also compares recovery routes and source-fit limitations, distinguishes source observations from reward feedback, and analyzes the nonmonotone contrast signal used in the convergence argument.

F.4 Two sampled Adam updates after source training

Lemma F.17 bounds every Adam coordinate direction by K_mom without an upper gradient bound or a positive lower second moment.The bound applies to every coordinate and finite history.Source: §F.4 · p. 98
The raw initial hard coefficient operator is left-orthogonally invariant, and the identity holds for coefficient columns including their constant and linear terms.This is a property of the raw initial operator; the passage disclaims trajectory equivariance of coordinatewise Adam.Source: §F.4 · p. 99
The correlated finite-feature operator approximates the Gaussian operator uniformly over adaptive second-moment matrices, conditional on the stated covariance event and with a bounded failure probability.The second-moment matrices may depend on realized features and training batches.Source: §F.4 · pp. 99–100
The learned executor’s bound-operation reward exceeds its hidden-bit opposite by at least 1/2, while the opposite reward is below r*/q, conditional on correct output classes.The reward bounds include trajectories with repaired hidden-bit errors.Source: §F.4 · p. 102
After two Gaussian-reference updates, the bound operation is chosen with probability at least 15/16 on every training and held-out choice prompt.The canonical choice geometry makes reference choice probabilities prompt-independent.Source: §F.4 · pp. 102–103
Sampled feedback uses full-vocabulary decisions and the original batch denominator for every trajectory, while the fixed-executor choice gradient and two-step Gaussian row fields have Lipschitz bounds.The row fields are Lipschitz in maximum Frobenius history error.Source: §F.4 · p. 104
On matching valid histories, conditional choice and state laws differ from the fixed-executor laws by at most 8h, and two fresh batches concentrate while history-aware feature comparisons control the Adam updates.The batch guarantee holds with probability at least 1 − δR; the two-batch trajectories must be independent within each fresh batch, conditional on its past.Source: §F.4 · p. 105
After two updates, the iterates stay within E*/2 of the reference, with final choice-logit error at most AE* + 2h and complete-path discrepancy at most n(8h + h exp(2Bexp + 4)).The reference task utility and first-state retention are at least (15/16)(1 − 9c/8), and the procedure uses exactly two updates and 2BR verifier queries.Source: §F.4 · p. 106

F.5 An explicit joint source and feedback budget

The joint budget orders horizon and source-precision choices so the signed-source requirement and both feedback bounds hold without requiring source tolerance exponentially small in the horizon.The horizon is chosen before H is set, and the choices depend only on declared task parameters, not realized lag frequencies, the hidden mechanism, or test scores.Source: Appendix F.4 · pp. 107–108

Appendix F.4. All logarithms are natural unless their base is displayed. Fix

At the selected finite widths and positive rates, source stopping succeeds, total backbone displacement across both stages is at most one, and the two feedback iterates remain within E*/2 of the reference.Source displacement is at most rψ/2 ≤ 1/2; source and feedback together maintain total feature error at most εf and base, slow-row, and feature-motion logit discrepancy at most 2h.Source: Appendix F.4 · pp. 110, 112
With probability at least 1 − δ, all held-out prompts have source metrics E, M, S at most 1/q + 2nh, final metrics above 7/8, and gains above 3/4.The grammar or public-coordinate violation probability is at most 9c/8 + 1/128.Source: Appendix F.4 · pp. 112, 116
The ordered construction has polynomial real-arithmetic bounds in expanded q, H, and δ−1, up to logarithms, and uses exactly two feedback updates.Source uses at most NBsTs targets and Ts − 1 updates; feedback uses exactly 2BR fresh prompts and terminal queries.Source: Appendix F.4 · p. 113
With probability at least 1 − δtab, raw-source estimation recovers all q2 opaque transition rows; binding selection then achieves perfect deployment utility without gradient updates or a test-time oracle.The recovery procedure charges at most 6M0 + 2H + 2 source tokens and uses at most q − 1 fresh-prompt verifier queries.Source: Appendix F.4 · pp. 115–116

F.6 Output format and regularized source learning

A finite coefficient choice can make within-state predictions match noisy true transitions yet assign EOS probability at least 1 − δf to a forbidden held-out first-state prefix.The perturbation targets one forbidden suffix, increases coefficient norm with its amplitude, and isolates the role of regularization in extending source prediction to the required format.Source: §F.6 · pp. 116–117

G.1 Acquisition, contextual transfer and intervention

Source matching rises from at most 0.51 to at least 0.9; feedback success reaches at least 19/20, while freezing query/key parameters caps success at 1/2 + 1/64.The feedback improvement over the source checkpoint is at least 2/5, and the paired gap over the frozen-parameter intervention exceeds 2/5.Source: §G.1 · p. 118

G.2 Source learning in the radial class

The source-learning reference has attention core M = βU and a head with only C12 = C21 = t/2 in the source-active content block.Its population head gradient has no entries outside those same two positions.Source: §G.2 · p. 118

G.3 Contextual feedback from the acquired state

From the acquired source state, contextual feedback preserves an invariant attention core and positive task-rule drift, reaching a target coefficient with a held-out signed-logit margin of at least log 19 + 1.The invariant region has B ≥ 0 and c ≥ cmin; the endpoint margin applies across contexts, queries, Boolean cues, and specified amplitude pairs.Source: §G.3 · pp. 122–124

G.4 The frozen intervention and source-information bound

With query/key updates frozen, the reference target logit stays zero; separately, after Npre iid source observations, every world-independent learner has worst-world expected test error at least 1/4(1 − 1/nctx)^Npre, regardless of adaptive feedback queries.The bound is for the stated iid-observation or worst-case-cap experiment; the passage distinguishes it from expected stopping budgets and chosen-context queries.Source: §G.4 · pp. 124–125

G.5 A sufficient contrast region and its signal scale √

For ψ(u)=u−1, w(c)=c/(1+kc^2) rises on [0,1/√k], falls on [1/√k,1], peaks at 1/(2√k), and remains positive for c>0.Here k>1 and 0≤c≤1; the signal is positive below the turning point, while the convergence argument uses the invariant region above it.Source: §G.5 · pp. 125–126

G.6 Actual sampled Adam and the cubic moment map

This section develops finite-run and finite-precision guarantees for source acquisition and reward transfer, relating sampled Adam, SGD, and neural training paths to reference behavior. It bounds trajectory and sampling effects and specifies when source-acquired predictions or computation can persist through encoding and reward adaptation, without making a full-history density-ratio claim.

G.6 Actual sampled Adam and the cubic moment map

Actual bias-corrected Adam follows the exact normalized-coordinate update ∓ηtDt, and every scalar gradient history has |Dt| ≤ KA, including zero rewards and nonzero old momentum.Moments reset at each stage; the finite-run error analysis bounds sample, lag, and Euler errors and couples actual cores to population references across the stage switch.Source: §G.6 · pp. 126, 130
For p ≥17, source variables Y and X have uniform moment bounds through order eight without assuming independence between the Gaussian maximum W and h.The analysis also bounds source and reward gradient regularity on the core region and establishes a unique self-consistent population reference on both stages.Source: §G.6 · pp. 128–129

G.7 Explicit budgets and functional outcomes

Source matching rises from below 0.51 initially to at least 0.9 at the endpoint, with endpoint greedy source-bit error below 0.1.These guarantees hold uniformly over source contexts, frames, and queries on the joint high-probability event.Source: §G.7 · pp. 131–132
Feedback success exceeds 19/20, versus source-checkpoint success at most 1/2 + 1/128; the gain exceeds 2/5, while frozen-attention success is at most 1/2 + 1/64.The frozen branch's gap below the trained branch also exceeds 2/5 on held-out inputs.Source: §G.7 · p. 132

G.8 Finite coordinate alphabets

A uniform lower bound is established for all source attention pools under the finite-alphabet construction.The supplied passage does not provide the bound's value or event probability.Source: §G.8 · p. 132
65δ/128 < δ bounds the total failure probability for encoded on-policy runs, which retain the matching and target endpoint margins with combined core discrepancy at most 5κ/16.The bound combines the continuous two-branch, coordinate, QR, and action-mismatch events.Source: §G.8 · p. 135
Rounding a fresh encoded source record adds at most ϵdec = 1/100 to Gaussian-proxy greedy source-bit error, keeping the resulting error below 0.1.The guarantee is specific to quadratic heads; unrestricted finite-grid decoders are not covered.Source: §G.8 · p. 136
The finite-coordinate resource accounting specifies source records, generated actions, verifier queries, updates, coordinate precision, and computation costs, with no validation or selection queries.The neural optimizer retains real-valued states; coordinate bits are charged according to the chosen serialization.Source: §G.8 · p. 136

H.1 Supplementary SGD learning path

Theorem H.1 gives a finite source-SGD and sampled-REINFORCE path on the same parameters, with its stated conclusions holding for every test prompt in each fixed world with probability at least 1−δ.Feedback uses complete-trajectory terminal REINFORCE scores with baseline zero and fresh minibatches.Source: §H.1 · p. 137

H.2 Finite neural source acquisition

The sequential source family contains 2m2 labeled permutation tables, and its invariant source dynamics retain semantic coefficients on every prescribed class/PAD prefix, including incorrect state histories.The signed Koopman operator has norm 2^-r for every unknown table.Source: §H.2 · p. 138
A source comparator with excess at most ϵ yields final excess E ≤ 2ϵ when η ≤ 1/K and Tsrc ≥ D*/(2ηϵ), with first and repeated transition errors bounded by the stated Pinsker bounds.The comparator has a squared-norm bound, and the repeated-transition bound uses denominator H−1.Source: §H.2 · p. 139
For either transition type, the source reference’s last-iterate excess is at most 2ϵ when the comparator CE excess is at most ϵ and Tsrc ≥ D∗/(2ηϵ).Source: §H.2 · p. 140
With failure probability at most 7δ/20, the actual source endpoint has uniform logit error at most 6e0 ≤ h, feature error at most 2e0/Rsrc, readout norm at most Rsrc, and backbone displacement at most rball/2.The bounds hold on all legal-pattern prefixes, including incorrect generated histories; with the conditional feedback allowance δ/2, total failure is at most 17δ/20.Source: §H.2 · p. 143
The actual ancestral source sampler satisfies max{E(ws, X), M(ws, X), S(ws, X)} ≤ 1/q + nh simultaneously for every test prompt X.Source: §H.2 · p. 143

H.3 SGD readout and moving-feature bounds

For shared batches, the deterministic coupling compares actual SGD with a fixed-feature reference, and its feature-motion term contains D^2; no source-sampling or test-word-distribution premise is needed.The stated coupling applies when the actual path stays in the specified radius-r ball and satisfies the required movement bound.Source: §H.3 · pp. 144–145
The normalized member’s prescribed source schedule has ssrc ≤ CsH6, and its finite feedback proof requires no normalization second-derivative assertion.The proof uses a fixed-backbone readout Hessian and bounds backbone updates through a first-derivative trajectory bound.Source: §H.3 · p. 146
The conditional-versus-unconditioned trajectory comparison controls local observables, not a full-history density ratio.The comparison also bounds reference-logit error; fresh target sampling is conditioned on the source event.Source: §H.3 · pp. 147–149
The joint source and feedback event has failure probability at most 17δ/20, without requiring independence among the resulting events.The bound allocates δ/20 to each of seven source estimates and δ/2 to feedback sampling conditionally on the source event.Source: §H.3 · p. 149

H.4 Finite neural feedback and information comparisons

This section establishes information limits and finite-path guarantees for reward adaptation, showing how source prediction and execution can provide structure that source-free feedback may leave ambiguous. It also describes finite sampled training procedures in which feedback learns task-specific control while preserving source computation, with performance guarantees and explicit resource accounting.

Finite reference reward improvement

With the source feature mean and fixed-executor rewards, the reference reward path reaches correct-control probability 15/16 with cumulative coefficient displacement O(log q).This finite improvement guarantee assumes the actual/reference discrepancy stays at most 1/16 and the correct control remains maximal; the fixed-executor equalities do not hold exactly for the moving network.Source: §H.4 · pp. 150–152

Actual feedback path stays in the reference tube

Under the same source event and stopped feedback path in Lemma H.36, independent fresh batches of size (H.86) yield projected fast-score batch noise at most ν with probability at least 1−δR, simultaneously for all T predictable iterates.The original batch denominator is retained, including for zero-reward batches.Source: §H.4 · p. 154

Final SGD metrics and source-to-reward gain

Final task utility, memory success, and endpoint success all exceed 7/8; each final-minus-source gain exceeds 3/4, while violation risk is at most 9c/8 + 4nh.These bounds hold simultaneously on every test prompt at the predetermined source and reward endpoints.Source: §H.4 · p. 155

Source-only decoding with an independent binding

For a binding chosen uniformly and independently of source observations, initialization, learner randomness, and evaluation prompts, every source-only decoder has binding-averaged endpoint success at most 1/q.This is an average-over-binding or minimax guarantee, not a pointwise bound for every binding.Source: §H.4 · p. 156

Proposition H.41

Without source observations, paired worlds have identical adaptive training transcripts for any number of queries, yet average endpoint success on a common tag-one test prompt is at most 1/2.The pairing preserves uniform η; the bound does not imply a first-state memory lower bound because paired first targets agree.Source: §H.4 · p. 156

Singleton-query resource bound

With uniform independent binding and no binding-dependent prompt information, success of at least 7/8 requires at least Q ≥7q/8 −1 adaptive pre-test candidate-verification queries, even when θ is given.The bound excludes expected-reward oracles, intermediate labels, binding-dependent prompt side information, and test-time verification.Source: §H.4 · p. 157

Packed-mechanism information bound

For m ≥512 and H ≥5, a separate packed-mechanism result bounds information from independent reset source records and at most Q adaptive binary pre-test replies.It permits a fixed known binding and adaptive prompt choice; nrecH + Q counts information bits, not monetary or computational costs.Source: §H.4 · p. 157

Propositions H.43–H.44

At most 6M0 + 2H + 2 raw tokens recover all q2 opaque successors with probability at least 1 −δtab, enabling binding identification with at most q −1 verifications.The rowwise empirical-mode estimator needs no downstream goal labels; on recovery, fresh uniform training prompts yield perfect test utility and public-coordinate compliance.Source: §H.4 · pp. 157–158

Proposition H.45

With an external binding controller, greedy source execution follows the exact trajectory and EOS, and at most q −1 fresh-prompt verifications identify a binding yielding correct first states and endpoints with probability at least 1 −7δ/20.After binding identification, every test prompt has the correct first state and endpoint.Source: §H.4 · pp. 158–159

Proposition H.46

Complete SGD accounting gives Q = BRT feedback queries, nQ generated decisions, Tsrc + T optimizer updates, and T additional inference-only rate inspections.The accounting separates prompt, context, parameter-storage, and verifier costs; no test score selects a checkpoint.Source: §H.4 · p. 159

Proposition H.47

For a fixed sampled batch, baseline-zero terminal REINFORCE equals the negative gradient of on-policy reward-weighted likelihood, producing identical paths under matched rates, inspection, and coupled fresh batches.Independently sampled fresh batches give equality in law; the equivalence does not extend to arbitrary expert SFT, filtered-denominator training, or repeated off-policy replay.Source: §H.4 · pp. 159–160

Theorem H.1

The finite sampled path achieves 239/256 correct-choice probability, executor success at least 503/512, coupling loss below 1/1024, gain at least 3/4, and violation at most 9c/8 + 4nh.Together these yield success exceeding 7/8; the endpoints are predetermined, so checkpoint or test selection adds no query.Source: §H.4 · p. 160