PaperlayerView exact revision on arXiv

Source-linked AI summary

$T^5$: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training

Nan Qiao, Yebin Yang, Weinong Wang, Shuning Wang, Shangpin Peng, Fengyuan Lu, Xinming Wang, Zhehan Kan, Ruixu Zhang, Songyang Zhang, Sheng Yue, Yonglong Tian, Ju Ren

arXiv:2609.32791v2cs.AI

TL;DR

Training–inference mismatch and PPO clipping can make a common advantage offset contribute to policy-update drift, complicating reliable token-level credit assignment. T5 uses two qualified critics and a conditional-moment objective to calibrate advantages from a single trajectory. Compared with RLP, it improves mean benchmark performance by 7.8% and reduces mean training-step time by up to 63.4%.

  • Problem

    Training–inference mismatch or PPO clipping can prevent a common advantage offset from cancelling, adding drift to advantage-based policy updates.

    Source: §1 · p. 1

  • Method

    T5 qualifies two critics and combines their single-rollout advantage estimates with action-dependent weights learned by a conditional-moment objective constrained to retain signal.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    Compared with RLP, T5 improves mean benchmark performance by 7.8% and reduces mean training-step time by up to 63.4%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis characterizes optimal mixing under the signal-retention constraint and bounds residual mean-induced drift in clipped PPO updates.

    Source: §5.3 · p. 8, §7 · p. 10

    Without seed-level uncertainty, the reported point estimates establish observed ordering, not statistical significance or component-level causality.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

Reinforcement mid-training lets language models learn internal thoughts from unlabeled text, but efficient token-level credit assignment remains challenging. Existing group-relative methods require costly repeated generation. Learned critics offer single-rollout feedback, but accurate return prediction alone does not ensure reliable policy updates. Our analysis shows how training--inference mismatch and PPO clipping prevent a common offset in advantage estimates from cancelling out, introducing additional update drift. We propose \tfour{}, a twin-critic method that calibrates token-level advantages from a single generated trajectory. After warmup and held-out qualification, the critics provide two advantage estimates, combined using action-dependent weights learned through a conditional-moment saddle-point objective. This objective brings the average advantage at each prefix toward zero, while a signal-retention constraint prevents the correction from erasing the learning signal. Sharing information across text positions avoids repeated sampling of each prefix. Theoretically, we characterize optimal mixing under the signal-retention constraint and establish an upper bound on residual mean-induced drift. Experiments show that, compared with the state-of-the-art critic-free method, \tfour{} improves mean benchmark performance by 7.8\% and reduces mean training-step time by up to 63.4\%.

1 INTRODUCTION

T5 addresses unreliable token-level credit assignment in reinforcement mid-training by calibrating advantages from two qualified critics on a single rollout. Its analysis identifies how mismatch and PPO clipping can make a common advantage offset contribute to policy-update drift.

1 INTRODUCTION

  • Group-relative methods such as GRPO, RMT, and RLP use multiple responses to construct a baseline, whereas learned critics can provide token-level feedback along one trajectory.The comparison uses groups of eight for GRPO; critic predictions of remaining return combine with prefix rewards through GAE.Source: §1 · p. 1
  • Training–inference mismatch or PPO’s sign-dependent clipping can prevent a common advantage offset from cancelling, adding mean-induced drift to the expected update.The analysis separates this component from the update component reflecting variation across actions.Source: §1 · pp. 1–2
  • T5 mixes two critics’ advantage estimates with action-dependent weights learned by a conditional-moment saddle-point objective.The objective reduces residual prefix-conditioned offset using critic disagreement, while a constraint limits deviation from the fixed twin average to retain aggregate signal.Source: §1 · p. 2
  • Against RLP, T5 improves mean benchmark performance by 7.8% and reduces mean training-step time by up to 63.4%.The paper also proves strong duality, characterizes optimal mixing under the signal-retention constraint, and bounds residual mean-induced drift.Source: §1 · p. 2

2 RELATED WORK

Related work develops reinforcement signals from ordinary text and compares group-relative estimation, learned critics, and policy-stabilization methods. T5 focuses on combining qualified critics’ advantages while retaining clipped PPO.

Latent reasoning and reinforcement mid-training

  • QUIET-STAR, RPT, RLPT, RLP, PretrainZero, and RMT use different ways to connect internal thoughts or masked spans to learning from pretraining text.This paper follows QUIET-STAR’s continuation-utility objective at the intermediate training stage and studies learned token-value baselines across thought prefixes.Source: §2 · p. 2

Group-relative estimation and token-level actor–critic learning

  • GRPO constructs a baseline from sampled response groups, while learned critics share state information to support GAE along each trajectory.VAPO, SAO, and offline-RL work address other aspects of value-based or single-rollout training; T5 uses two state-value critics and learns how to combine their advantages.Source: §2 · pp. 2–3

What policy stabilization controls

  • Policy-stabilization methods constrain policy movement, alter normalization or clipping, or correct sampling-related drift, while T5 acts on the conditional advantage mean.T5 retains clipped PPO after qualifying the predictors; under exact unclipped score weighting, inaccurate state baselines can still cancel.Source: §2 · p. 3

3 PRELIMINARIES AND PROBLEM SETUP

The setup turns a thought’s effect on next-token prediction into token rewards, then uses a learned critic and PPO to assign token-level advantages. It distinguishes the actor’s observations from the scoring context and defines the rollout and critic quantities used in optimization.

3.1 REINFORCEMENT MID-TRAINING FROM UNLABELED TEXT

  • A thought’s utility is the reduction in prediction loss on the same observed continuation, and successive token-level loss changes add up to that utility.An individual change can be negative even when the full thought helps.Source: §3.1 · p. 3
  • The method clips and rescales cumulative gain into a potential, then assigns each thought token the change in that potential as its reward.The scorer and reward scale remain frozen within each training window.Source: §3.1 · p. 3
  • A separate prediction gate combines thought-conditioned and no-thought predictions, while ordinary next-token loss anchors language modeling.This gate is distinct from the advantage weight introduced later.Source: §3.1 · p. 3

3.2 PPO WITH A LEARNED TOKEN-LEVEL CRITIC

  • A learned critic predicts remaining return from each prefix, and GAE combines temporal-difference residuals into token-level advantages.The setup uses full-return targets and a length-adaptive trace; values and advantages vanish after termination.Source: §3.2 · pp. 3–4
  • PPO optimizes normalized, detached advantages using token probability ratios and clipping bounds, with recorded rollout quantities fixed during actor updates.The rollout law q, recorded policy πold, and current actor πθ are distinct; critic mismatch includes stale targets or hidden scoring information.Source: §3.2 · p. 4

4 CRITIC MISMATCH AND PREDICTIVE QUALIFICATION

A learned critic can add fit error and target lag even when rollout and actor policies match, so T5 tests both critics on current held-out returns. Qualification is refreshed as the actor changes before their advantages are combined.

4.1 THE CRITIC ADDS ITS OWN MISMATCH

  • Critic fit error and target lag can remain even when rollout, recorded, and current actor policies match, so policy-ratio control alone does not ensure current return predictions.Stale value references can distort credit assignment across a thought because GAE propagates later value errors to earlier tokens.Source: §4.1 · p. 4

4.2 TRAINING AND QUALIFYING BOTH CRITICS

  • T5 stabilizes each critic’s return target, tests both critics on held-out returns, and repeats qualification as the actor changes.Prediction quality, rather than a fixed warmup duration, determines readiness; both critics must qualify before actor updates.Source: §4.2 · pp. 4–5
  • Qualification requires each critic to improve on the constant held-out-return predictor by at least η, and the weaker critic’s score determines readiness.The score accounts for continuation variability, so prediction quality is tested despite irreducible return variation.Source: §4.2 · p. 5
  • Fresh held-out tests recur at fixed intervals, and sustained qualification failure pauses actor updates while critic fitting resumes.Changes to rollout or scoring targets trigger new checks.Source: §4.2 · p. 5
  • A historical naive PPO run improved CommonsenseQA thought gain from about −4.8 to −3.0 nats and mixed gain from about −0.18 to −0.10 nats after 20 critic-only steps.Both gains remained below the no-thought reference; this fixed-warmup diagnostic did not evaluate T5 or its R² qualification rule.Source: §4.2 · p. 5, Figure 1 · p. 5

5 LEARNING COMPLEMENTARY ADVANTAGES FROM SINGLE ROLLOUTS

Training–inference mismatch or PPO clipping can keep a common advantage offset from canceling, adding mean-induced drift to policy updates. T5 mixes two critic estimates to reduce that offset while retaining an aggregate learning signal.

5.1 ACTION-DEPENDENT MIXING AND CONDITIONAL DRIFT

  • A common advantage offset can contribute to policy-update drift when training–inference mismatch or clipping prevents its cancellation.The conditional update separates into an action-variation component and a mean-induced component.Source: §5 · p. 5, §5.1 · p. 6
  • T5 uses an action-dependent weight based on visible-prefix and critic features to mix two estimates before applying PPO clipping.The weight does not use realized returns or GAEs.Source: §5.1 · p. 6

5.2 A SINGLE-ROLLOUT SADDLE-POINT OBJECTIVE

  • The saddle-point objective minimizes squared conditional offset while constraining displacement from the fixed-average advantage to retain signal.A maximizing auxiliary mean predictor exposes residual offset, while the minimizing mixture weight reduces it.Source: §5.2 · p. 7
  • Each selected position supplies one thought to both critics, avoiding repeated prefix continuations for the advantage estimates and objective.The training window warms up critics, checks them on fresh tests, learns and validates the mixture, then freezes it for actor updates.Source: §5.2 · pp. 7–8

5.3 BOUNDING CONDITIONAL DRIFT

  • A small measured objective bounds mean-induced drift tightly when mean-prediction error and validation uncertainty are also small.The bound uses the actual PPO mask to control the mean-induced component, not the entire update.Source: §5.3 · p. 8

6 EXPERIMENTS

The experiments compare T5 with matched controls and critic-free and critic-based baselines across benchmarks and model scales. T5 leads the retained seven-benchmark comparison, while the reported training-time comparison shows lower mean step times than RLP.

6 EXPERIMENTS

  • The experiments address downstream performance, cross-task and scale consistency, mechanism diagnostics, and performance–time trade-offs.The 2B and 9B models continue pretraining on a fixed corpus with disjoint data splits.Source: §6 · p. 8, §6.1 · p. 8

6.1 EXPERIMENTAL SETUP

  • The study compares Base, NTP, single-critic Quiet-STaR with PPO, Quiet-STaR with GRPO, RLP, and T5 at 2B; 9B measures scaling.All GRPO-based baselines use group size 8; the comparisons use matched settings.Source: §6.1 · pp. 8–9

6.2 MAIN RESULTS

  • 69.56 versus 64.52 for RLP and 60.15 for Quiet-STaR (w/ GRPO): T5 leads all seven retained benchmarks on unweighted mean.The largest margin over RLP is on MATH-500 (85.45 versus 63.65); gains on the other six tasks are 0.90–4.51 percentage points.Source: §6.2 · p. 9, Figure 3 · p. 9
  • In an earlier shared-architecture comparison, T5 exceeds Quiet-STaR (w/ GRPO) on all seven displayed tasks, but is slightly below Base on GPQA pass@1.These point estimates do not isolate predictive qualification or complementary weighting.Source: §6.2 · p. 9, Figure 3 · p. 9
  • Against RLP, T5 reduced mean step time by 41.1% at 2B and 63.4% at 9B in the reported comparison, with timings measured on the same hardware.Source: §6.2 · p. 9

7 CONCLUSION

T5 separates return prediction from advantage calibration, using qualified twin critics and a conditional-moment saddle objective to mix estimates from one rollout per selected position. A retention constraint preserves aggregate signal strength as the analysis bounds mean-induced clipped PPO drift.

7 CONCLUSION

  • T5 separates return prediction from advantage calibration, combining qualified twin-critic estimates through a conditional-moment saddle objective using one rollout per selected position.The objective’s population optimality conditions relate the mixture to residual offset, critic disagreement, and the retention constraint.Source: §7 · p. 10
  • The retention constraint preserves aggregate signal strength while the analysis bounds the mean-induced component of clipped PPO drift.Source: §7 · p. 10
  • The method makes both return prediction and the use of advantage estimates part of the learning process.Source: §7 · p. 10

REPRODUCIBILITY STATEMENT

The statement directs readers to the main text and appendices for method, derivations, implementation, evaluation provenance, and training-time plotting data.

REPRODUCIBILITY STATEMENT

FULL PAPER CONTENTS

The appendix covers additional experimental results, model and data configuration, calibration implementation, and evaluation and comparison protocols. It also describes compute accounting and diagnostics relevant to interpreting and reproducing the comparisons.

A ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS

A.1 MODEL, DATA, AND IMPLEMENTATION CONFIGURATION

  • The target setup uses Qwen3.5 Base checkpoints and a recent corpus; calibration separates training, calibration, development, validation, and actor-optimization roles.One thought per selected item is used for head learning, independent validation, and actor optimization; predictive qualification alone does not certify conditional-moment risk.Source: §A.1 · p. 15

A.2 TARGET EVALUATION AND COMPARISON PROTOCOL

  • The protocol defines a seven-benchmark primary suite, excludes those benchmarks from training and selection, and reports point estimates without independent-seed variation.Compute accounting distinguishes corpus tokens, sampled thought tokens, and total compute; it includes critic and calibration costs.Source: §A.2 · pp. 16–17

A.3 QWEN3.5-2B RESULTS AND DIAGNOSTICS

On Qwen3.5-2B, T5 leads the seven retained benchmarks, while additional diagnostics describe pass@4 gains, reasoning length, and warmup trade-offs.

A.3 QWEN3.5-2B RESULTS AND DIAGNOSTICS

  • 69.56 for T5 versus 64.52 for RLP on the retained-seven mean, with T5 leading all seven tasks.The mean is recomputed from unrounded Table 7 values; T5 exceeds RLP by 5.04 points.Source: §A.3 · p. 17, Table 7 · p. 17
  • Pass@4 exceeds pass@1 by 15.83 points on each AIME set, 8.38 on GSM8K, and 4.15 on MATH-500.Coverage is 100% on every benchmark; reasoning length ranges from 548.2 tokens on PIQA to 3922.4 on AIME 2026.Source: §A.3 · p. 18, Table 8 · p. 18, Figure 4 · p. 18
  • The AIME 2026 score rises from 29.9 to 35.0 as warmup increases from 1.2 to 23.1 minutes.Increasing τ from 0.20 to 0.40 adds only 0.2 percentage points for 12.4 additional minutes; τ = 0.00 still takes 1.2 minutes.Source: §A.3 · p. 19, Figure 5 · p. 19

A.4 EARLIER CROSS-TASK COMPARISON AND PROVENANCE

The appendix documents an earlier cross-task comparison and training-time records, then examines performance across model scales and a historical naive-PPO diagnostic. It also derives how old rollouts, trajectory distributions, and PPO surrogates relate to policy updates.

A.4 EARLIER CROSS-TASK COMPARISON AND PROVENANCE

  • The earlier Qwen3-1.7B comparison reports T5 above Quiet-STaR (w/ GRPO) on seven displayed tasks, but below Base on GPQA pass@1.The provenance record lacks the training corpus, inference settings, seeds, and exact T5 implementation, so these point estimates are not attributed to the analyzed single-rollout protocol.Source: §A.4 · p. 20, Table 9 · p. 20

A.6 PERFORMANCE AND EFFICIENCY ACROSS MODEL SCALES

  • On Qwen3.5-9B, T5 scores 92.71 mean versus 86.86 for Base and 87.00 for NTP, leading both on every task.The 2B-to-9B T5 mean rises by 23.15 points, while its margin over NTP narrows from 13.26 to 5.71 points.Source: §A.6 · pp. 22–23, Figure 7 · p. 22, Figure 8 · p. 23

A.5 RECORDED PER-STEP TRAINING TIME

  • At 2B and 9B, T5’s mean step-time reductions versus RLP are 41.1% and 63.4%, respectively.The recorded means are 31.79 versus 54.01 seconds at 2B and 182.18 versus 497.83 seconds at 9B; these compare per-step costs, not time to a common score.Source: Figure 6 · p. 21, §A.6 · p. 23

A.7 HISTORICAL NAIVE-PPO DIAGNOSTIC

  • The historical naive-PPO diagnostic shows prediction mixing narrowing the average raw-thought deficit and becoming slightly positive at the final checkpoint.The reconstructed seven-point curves are figure-derived values, not raw run logs, and this contrast does not evaluate T5.Source: §A.7 · p. 24, Figure 9 · p. 24

B WHY PREDICTIVE ACCURACY DOES NOT REMOVE DRIFT

  • The drift analysis distinguishes policy lag, probability mismatch, and value error as different changes affecting policy updates.Its derivations assume a finite undiscounted horizon, fixed text distribution, transitions, scorer, and reward normalizer.Source: B WHY PREDICTIVE ACCURACY DOES NOT… · p. 25, §B.1 · p. 25

B.1 WHAT CHANGES WHEN OLD ROLLOUTS TRAIN A NEW POLICY?

  • The trajectory-likelihood derivation expresses the expected-return gradient using sampled trajectories and their action log-probability gradients.With a behavior law q covering the actor’s trajectory support, full-trajectory likelihood ratios restore the exact gradient under integrability; long products can concentrate weight on few trajectories.Source: §B.1.1 · pp. 25–26

B.1.1 A FIXED REWARD, BUT A CHANGING DISTRIBUTION OF THOUGHTS

  • For fixed rewards and transitions, the same sampled thought retains its score while its likelihood and continuation quality under the policy can change.The return sample is generated under a policy; a fixed trajectory is not assigned a different reward for each policy.Source: §B.1.1 · p. 25

B.1.2 WHY AN OLD-POLICY ADVANTAGE IS INTENTIONAL IN PPO

  • The old-policy surrogate matches the policy gradient at its origin, but normalized GAE, action-dependent weights, and clipping alter that estimator.Connecting the actual mixed PPO gradient to the exact identity requires matching the occupation measure and time weights.Source: §B.1.2 · p. 26

B.1.3 VALUE-LEARNING ERROR AND A MOVING VALUE TARGET

These parts analyze sources of value and advantage error and how they affect PPO updates, including limits of cancellation and critic-based calibration. They characterize twin-critic mixing through a conditional-moment objective, one-trajectory gradients, and conditions for optimal signal-retaining weights, while clarifying what signal retention does not guarantee.

B.1.3 VALUE-LEARNING ERROR AND A MOVING VALUE TARGET

  • Value disagreement with a current return can reflect policy change or historical target lag rather than failed fitting, while a visible-prefix critic also faces error from withheld scoring-item information.More regression on old returns does not move the old-policy optimum; predictive qualification therefore tests useful out-of-sample return structure, not exact item-level oracle reconstruction.Source: §B.1.3 · p. 27

B.1.4 BASELINE CANCELLATION UNDER EXACT ACTION RATIOS

  • Baseline cancellation requires exact action reweighting and full support, while an inexact denominator or sample-dependent clipping can leave a nonzero baseline contribution.The baseline need not equal either policy’s value; this cancellation does not remove old-state or old-continuation errors, and GAE can add future value errors.Source: §B.1.4 · p. 27

B.2.1 CLIPPING DOES NOT AMPLIFY ADVANTAGE PERTURBATIONS

  • Clipped PPO is nonexpansive in the advantage, so a convex mixture’s update error is bounded by the less accurate critic under the same rollout distribution and normalizer.The comparison permits detached action-dependent weights and does not require the clipping masks to remain equal; prediction risk from another distribution needs coverage control.Source: §B.2.1 · pp. 28–29

B.2.2 PREDICTIVE SCORES AND INDEPENDENT VALIDATION

  • Return-prediction risk includes both value error and continuation randomness, but aggregate predictive scores can hide rare-state errors and require independent validation with appropriate concentration bounds.Critics should be qualified separately: a collapsed predictor can have vanishing loss and twin disagreement as return scale shrinks while learning no return variation.Source: §B.2.2 · p. 29

B.2.3 HOW VALUE ERRORS PROPAGATE THROUGH GAE

  • GAE value error includes the current-prefix error and nonnegative, decaying contributions from later prefixes, so accurate current values alone need not suffice.With uniform value-error bound ϵi, the GAE error is bounded by 2ϵi; at λ = 1 only the current error remains, and the matched oracle comparison does not assume zero conditional mean.Source: §B.2.3 · pp. 29–30

B.3 TWO DISTINCT CONSISTENCY CONDITIONS FOR CANCELLATION

  • Either exact action reweighting or a zero conditional mean advantage can eliminate the mean-induced term, but practical clipping, recorded probabilities, frozen normalization, and GAE complicate those conditions.A fixed pilot mean need not be zero, and length-adaptive GAE estimates the conditional mean of the actual signal rather than assuming it equals current value error.Source: §B.3 · pp. 30–31

B.4 WHY A FIXED MINIMUM DOES NOT CALIBRATE ADVANTAGES

  • Minimum aggregation shifts the conditional mean downward rather than calibrating it, and global prediction scores can pass thresholds while a local harmful update remains.The constructed example gives ∆µ,min = −0.21 despite exact rollout probabilities; it is an example, not a measured result.Source: §B.4 · pp. 31–32

C CONDITIONAL-MOMENT LEARNING AND THE DRIFT BOUND

  • Conditional-moment learning trains a separate conditional-mean function, while action-dependent weights can introduce covariance and twin disagreement alone cannot identify a shared offset.The correction is limited by a signal-retention constraint; freezing the rule does not remove its dependence on action, and the dual result alone does not show that two critics estimate the mean better than one.Source: C CONDITIONAL-MOMENT LEARNING AND THE… · p. 33, §C.1 · p. 33

C.1 ACTION DEPENDENCE AND THE LIMITS OF TWIN DISAGREEMENT

  • Action-dependent mixing can turn a state baseline into an action-dependent term, so the method does not claim baseline invariance or oracle policy-gradient recovery.In the Monte Carlo case, critic disagreement reveals relative prediction differences rather than their shared value error; its covariance vanishes in that special case, unlike general GAE’s later-prefix randomness.Source: §C.1 · p. 33

C.2 THE CONDITIONAL-MOMENT DUAL

  • The conditional-moment dual targets the population conditional mean of mixed advantages, not an exact mean from one observed advantage.Fitting one label per context can memorize noise, and error against a sampled advantage additionally includes within-context variance; compressed inputs leave risk equal to unresolved conditional-mean variance.Source: §C.2 · p. 34

C.3 ONE-ROLLOUT STOCHASTIC GRADIENTS

  • Under the stated integrability conditions, one complete trajectory gives unbiased trajectory-summed gradient estimates, while independent fresh trajectories are needed for the validation concentration bound.Tokens within a trajectory may be correlated; reusing a fixed trajectory set supports gradients of its empirical objective but does not provide independent population validation.Source: §C.3 · p. 35

C.4 POPULATION SADDLE POINTS AND OPTIMAL WEIGHTS

  • Under Proposition 1’s assumptions, the population constrained problem has a saddle point at the optimal feasible risk, yielding an optimal gate based on visible input.The gate projects a correction determined by the conditional association between the auxiliary mean estimate and critic disagreement; it does not use realized GAE at actor-update time.Source: §C.4 · pp. 35–36

C.5 SIGNAL RETENTION AND ATTAINABLE RISK

  • Signal retention preserves aggregate energy and alignment with the fixed twin mean, but not every state’s action ordering, conditional covariance, or gradient direction.The constant gate is feasible and serves as a risk comparator; a failed empirical check falls back to it, guaranteeing feasibility but not improved drift correction. Shared critic bias, restricted inputs, and retention constraints can prevent exact centering.Source: §C.5 · p. 37

C.6 ACTUAL PPO GRADIENT AND ERROR BOUNDS

The bounds analyze the actual mixed PPO signal and specify when calibration can control mean-induced drift. Their scope depends on validation assumptions, clipping and sampling factors, and using the sampled GAE rather than treating one-step TD as its mean.

C.6 ACTUAL PPO GRADIENT AND ERROR BOUNDS

  • Clipping the mixed advantage with its own PPO mask need not match clipping each critic’s advantage before mixing their gradients.The conditional-moment objective trains the weight before actor sampling; it is not an additional penalty inside the actor loss.Source: §C.6 · p. 37

C.7 INDEPENDENT TRAJECTORY VALIDATION AND ITS LIMITS

  • The drift certificate requires valid bounds on approximation and optimization error as well as an independent validation bound; the base algorithm reports diagnostics without claiming this certificate.A low empirical saddle value or small auxiliary-head gradient alone does not bound the residual objective.Source: §C.7 · p. 38

C.8 POLICY SENSITIVITY, CLIPPING, AND SAMPLING MISMATCH

  • Mean-induced drift is bounded through offset size, policy sensitivity, and clipping imbalance, but clipping alone supplies no uniform bound on the mismatch or actor-sensitivity factors.A guarantee across the update window requires common bounds on these factors.Source: §C.8 · pp. 38–39

D SCOPE OF CONDITIONAL CALIBRATION

  • The guarantees are examined beyond the frozen-window setting by varying the target, withholding scoring information, and moving the prefix distribution.These comparisons specify which assumptions must remain fixed for the guarantees to apply.Source: D SCOPE OF CONDITIONAL CALIBRATION · p. 40

D.1 WHY ONE-STEP TD IS NOT SUBSTITUTED FOR GAE

  • One-step TD residuals are not generally unbiased observations of the GAE mean, and the length-selected trace parameter can correlate with future residuals.The moment objective uses actual sampled GAE; TD-based mean learning is a separate fixed-trace variant.Source: §D.1 · p. 40

D.2 TRAINING-TIME SCORING INFORMATION AND VISIBLE-STATE LIMITS

The appendix identifies limits from hidden scoring information, prefix-distribution changes, and neighboring actor–critic methods. It specifies where calibration applies and how T5 differs from those methods.

D.2 TRAINING-TIME SCORING INFORMATION AND VISIBLE-STATE LIMITS

  • A visible-state baseline leaves a variance floor from scoring-context variation, and even a full-context auxiliary model may not represent the needed mean or yield a feasible zero-offset gate.The auxiliary mean function receives the full scoring context during training, while the actor and return critics do not.Source: §D.2 · p. 40

D.3 WHY QUALIFICATION MUST FOLLOW THE PREFIX DISTRIBUTION

  • An early action can shift more later prefixes than a final action, so qualification risk may change with the prefix distribution even when the fitted gate is frozen.The bound holds under fixed-horizon, frozen-score conditions; changing the continuation law, scorer, or normalizers requires fresh validation.Source: §D.3 · p. 41

D.4 RELATION TO NEIGHBORING ACTOR–CRITIC METHODS

  • Unlike SAO, T5 trains on unlabeled text with hidden thought tokens as actions, qualifying two critics and learning a frozen action-dependent advantage mixture.T5 and SAO both use VAPO’s length-adaptive GAE trace, which T5 does not claim as new; unlike TD3 and SAC, T5 uses locally on-policy discrete latent tokens rather than replay-buffer action values.Source: §D.4 · p. 42

E TRAINING PROTOCOL AND COMPUTATIONAL COST

The protocol fixes data separation and scoring conventions within training windows, then fits, validates, and freezes the components used for actor updates. It also details reward construction, prediction gating, computation, trajectory accounting, and recovery conditions.

E TRAINING PROTOCOL AND COMPUTATIONAL COST

  • The protocol assumes a fixed sampling law, independent training and validation data, and detached advantages within each training window.Changes to the rollout, scorer, critics, features, or normalizer require renewed fitting and independent evaluation.Source: E TRAINING PROTOCOL AND COMPUTATIONAL… · p. 43, §E.8 · p. 47

E.1 REWARD SCALE AND CHECKPOINT CONSTRUCTION

  • Checkpoint rewards telescope to the terminal potential, so intermediate scoring changes credit assignment but not total reward; clipping can merge distinct high utilities.The authors claim only the within-iteration telescoping identity, not stationary-MDP policy invariance across model updates.Source: §E.1 · p. 43

E.2 VALUE REGRESSION AND SNAPSHOT REFERENCES

  • Critic fitting excludes predictive holdout, pilot, head-learning, head-validation, and actor sets, with complete items or parent trajectories split before token extraction.Return targets retain their absolute shaped scale, and the run configuration specifies the full-critic warmup loss.Source: §E.2 · pp. 43–44

E.3 ADVANTAGE SNAPSHOTS AND MASKED GAE

  • Actor advantages use frozen post-regression critic snapshots and shared pilot normalizers, with terminal conventions setting the bootstrap and subsequent advantage to zero.Pilot normalization constants remain fixed through weight learning, validation, and PPO.Source: §E.3 · p. 44

E.4 TRAJECTORY COLLECTION AND ACTOR REPLAY

  • Each selected item supplies one complete thought, and the weight head uses pre-action features plus the sampled action while the auxiliary head receives full context through a training-only path.The gate is independently validated for signal retention, with a failed test falling back to equal weights; an empirical pass is not a population certificate.Source: §E.4 · pp. 44–45

E.5 THE PREDICTION-MIXTURE GATE

  • The prediction-mixture gate is distinct from the critic-mixture weight, and its utility-aware loss can be outweighed by no-harm or mixed-NLL objectives.An exactly zero final gate layer blocks upstream gradients and can leave the mixture weight constant.Source: §E.5 · p. 45

E.6 STATE, ACTION, AND COMPUTATIONAL COST

  • Sampling scales with hidden actions, while continuation scoring scales with selected positions, checkpoints, and token evaluations; execution assigns separate process groups to actor and critics.Replay tensors and model synchronization dominate communication rather than external environment interaction.Source: §E.6 · p. 45

E.7 TRAJECTORY COUNTS AND COMPUTATIONAL COST

  • Removing repeated-prefix generation does not guarantee equal-accuracy sample savings: estimation error still depends on coverage, function capacity, and optimization.The trajectory-count comparison also does not establish time or samples needed to reach a target quality, and total computation includes critic and head training plus scoring and actor updates.Source: §E.7 · p. 46

Related papers

Loading 2609.32791v2…