PaperlayerView exact revision on arXiv

Source-linked AI summary

TEMPO: Scaling Test-time Training for Large Reasoning Models

Qingyang Zhang, Xinke Kong, Haitao Wu, Qinghua Hu, Minghao Wu, Baosong Yang, Yu Cheng, Yun Luo, Ganqu Cui, Changqing Zhang

arXiv:2604.19295v1cs.LG

TL;DR

Existing LRM test-time training methods plateau and collapse output diversity because self-generated rewards drift as policies evolve without external calibration. TEMPO alternates unlabeled-policy refinement with periodic labeled-data critic recalibration under an EM formulation, improving accuracy across models and reasoning domains while preserving diversity. Its main scope limits are added actor-critic overhead, dependence on labeled calibration data, unvalidated applicability beyond tested reasoning tasks, and absent formal convergence guarantees.

  • Problem

    Existing LRM TTT methods plateau and lose output diversity because self-generated reward signals drift without ground-truth labels at test time.

  • Method

    TEMPO alternates policy refinement on unlabeled test questions with periodic critic recalibration on labeled data, interpreting these stages as the EM M-step and E-step.

  • Results

    TEMPO consistently outperforms baselines across model scales and reasoning domains while preserving output diversity; on AIME 2024, OLMO3-7B rises from 33.0% to 51.1%.

  • Takeaways & Limitations

    Periodic critic recalibration restores the missing EM step, keeps the ELBO tight, and supports sustained self-improvement over extended training.

  • Takeaways & Limitations

    TEMPO requires extra actor-critic memory and computation, labeled calibration data, and remains unvalidated for domains such as code generation without formal convergence guarantees.

Abstract

from arXiv · show

Test-time training (TTT) adapts model parameters on unlabeled test instances during inference time, which continuously extends capabilities beyond the reach of offline training. Despite initial gains, existing TTT methods for LRMs plateau quickly and do not benefit from additional test-time compute. Without external calibration, the self-generated reward signal increasingly drifts as the policy model evolves, leading to both performance plateaus and diversity collapse. We propose TEMPO, a TTT framework that interleaves policy refinement on unlabeled questions with periodic critic recalibration on a labeled dataset. By formalizing this alternating procedure through the Expectation-Maximization (EM) algorithm, we reveal that prior methods can be interpreted as incomplete variants that omit the crucial recalibration step. Reintroducing this step tightens the evidence lower bound (ELBO) and enables sustained improvement. Across diverse model families (Qwen3 and OLMO3) and reasoning tasks, TEMPO improves OLMO3-7B on AIME 2024 from 33.0% to 51.1% and Qwen3-14B from 42.3% to 65.8%, while maintaining high diversity.

1 Introduction

Existing LRM test-time training methods plateau and lose output diversity because self-generated rewards drift as policies evolve. TEMPO alternates policy refinement with labeled critic recalibration, achieving sustained, diverse gains across models and reasoning tasks.

  • Motivation: Existing TTT methods plateau as heuristic self-reward signals become bounded by the model’s initial capabilities and increasingly overestimate narrow self-generated reasoning patterns.The lack of test-time ground-truth labels lets reward estimates drift as the policy becomes more confident.
  • TEMPO: TEMPO alternates actor policy refinement on unlabeled test prompts with periodic critic recalibration using verifiable rewards from labeled data.Its EM formulation treats critic recalibration as the E-step and policy optimization as the M-step.
  • Results: 33.0% to 51.1%: TEMPO improves OLMO3-7B accuracy on AIME 2024 while maintaining high pass@K scores where baselines suffer diversity collapse.The framework also improves Qwen3-14B on AIME 2024 from 42.3% to 65.8%.
  • Contribution: The paper characterizes prior TTT methods as incomplete EM procedures that omit posterior recalibration, motivating periodic calibration on labeled data.This perspective identifies the missing E-step as the structural source of scalability failures.
  • Results: TEMPO demonstrates superior accuracy and preserved output diversity across OLMO3 and Qwen3 model scales and five math, logic-puzzle, and STEM benchmarks.These experiments test whether the alternating design generalizes beyond mathematical reasoning.

2 Related Work

Prior work uses self-generated rewards to avoid ground-truth labels in reinforcement learning and reasoning-model TTT. These approaches share a structural reliance on policy-derived signals without reward calibration.

  • Self-rewarding reinforcement learning: Self-rewarding reinforcement learning replaces labeled rewards with intrinsic signals such as entropy, self-certainty, or reasoning topology.These methods aim to support self-training without external supervision.
  • Test-time training for reasoning models: Test-time training originated in computer vision as parameter updates on test instances to address distribution shifts.The reasoning-model literature later adapted TTT to inference-time reinforcement learning.
  • Test-time training for reasoning models: Reasoning-model TTT methods use self-generated signals including majority-vote pseudo-labels and entropy-based rewards.TTRL uses majority voting, while Intuitor and EMPO use entropy-based rewards.
  • Test-time training for reasoning models: The surveyed TTT methods share a structural deficiency: they refine the policy while omitting reward calibration.This deficiency motivates TEMPO’s alternating critic-policy design.

3 Method

TEMPO frames test-time training as an EM-style alternation between critic recalibration on labeled data and policy refinement on unlabeled questions. The critic estimates correctness-related rewards, while the policy uses those rewards to continuously adapt through weighted likelihood or policy-gradient optimization.

  • Alternating procedure: TEMPO initializes actor and critic with RLVR on labeled data, then alternates critic calibration with policy refinement on unlabeled test questions.The procedure follows the EM algorithm, with critic calibration as the E-step and policy optimization as the M-step.
  • Problem formulation: The objective maximizes the expected log-probability of generating a correct response when correctness is unobserved for unlabeled questions.The marginal correctness probability is obtained by summing over possible generated responses, and the policy πθ(y|x) represents the model’s output distribution.
  • Variational objective: The ELBO decomposes optimization into expected likelihood of likely-correct responses and a KL term measuring mismatch between the auxiliary distribution and the true correctness posterior.Dropping the non-negative KL term yields a lower-bound objective whose equality condition is exact posterior matching.
  • E-step: critic recalibration: The critic is trained on labeled responses with correctness indicators to approximate the posterior over correct responses and provide a grounded reward signal.Its token-level predictions are fit by mean squared error, and the critic’s terminal value estimates response correctness.
  • E-step: posterior estimation: q(y|x) ∝ Vϕ(x, yT)πθ0(y|x) reweights current policy outputs by critic scores to identify high-quality responses without ground-truth labels.The resulting auxiliary distribution serves as a surrogate for the missing correctness labels.
  • M-step: policy refinement: The M-step updates θ by weighted maximum likelihood or policy gradient on reasoning trajectories sampled from unlabeled questions.The terminal critic value supplies the external reward, while intermediate predictions provide token-varying baselines and advantages for variance reduction.

4 Experiments

TEMPO is evaluated across mathematical and general reasoning tasks, model families, benchmarks, scalability, diversity, and ablations. It consistently improves accuracy while preserving diversity, extends gains beyond RLVR training ceilings, generalizes beyond mathematics, and benefits from alternating critic recalibration.

  • Experimental setup: TEMPO is evaluated across diverse base models, datasets, benchmarks, and baselines, including mathematical and general-domain reasoning tasks.The experiments cover OLMO3 and Qwen3 models, five mathematical benchmarks, and non-mathematical tasks such as BBH, AGI Eval, ZebraLogic, and GPQA-Diamond.
  • Scalability: TEMPO significantly outperforms baselines across model scales and benchmarks, including AIME 24 avg@16 gains from 33.0% to 51.1% for OLMO3-7B-Base and from 42.3% to 65.8% for Qwen3-14B-Base.Periodic critic recalibration prevents reward drift as the policy evolves, enabling informative gradients for challenging test-time questions.
  • Scalability: TEMPO continues converting additional test-time compute into gains after convergence, while further supervised PPO on the labeled dataset produces negligible improvement.Starting from a converged OLMO3 model, TEMPO produces a consistent performance surge over 200 iterations.
  • Diversity: TEMPO maintains output diversity across benchmarks, whereas representative baselines suffer diversity degradation and declining pass@k despite short-term avg@k gains.On Qwen3-14B-Base, TEMPO reaches pass@k values of 73.0 on AIME 24 and 64.3 on AIME 25, compared with TTRL’s 56.7 and 43.3.
  • Versatility: TEMPO generalizes beyond mathematics, achieving substantial gains on logic, broad reasoning, and STEM-oriented benchmarks across OLMO3 and Qwen3.For OLMO3-7B, the reported absolute gains are +21.4 on BBH and +24.5 on AGI Eval; Qwen3-8B improves particularly on ZebraLogic and GPQA-Diamond.
  • Ablation: Ablations show that novel unlabeled test problems and periodic critic recalibration are both important: supervised continuation saturates, while a frozen critic plateaus after about 100 iterations.The frozen critic becomes misaligned with the evolving policy, whereas full TEMPO sustains improvement through recalibration.

5 Discussion

The EM framework interprets prior TTT methods as incomplete procedures that omit posterior recalibration, while TEMPO restores this step with a dynamically calibrated critic.

  • Unified EM perspective: EMPO and TTRL effectively perform only policy optimization with self-generated pseudo-labels, omitting the posterior-calibration E-step.This omission makes them degenerate EM instances.
  • Prior methods: TTRL’s majority-consensus signal is determined solely by the current policy, making confident reasoning patterns increasingly self-reinforcing.The resulting feedback loop is identified as the root cause of TTRL’s performance plateaus and diversity collapse.
  • TEMPO’s calibrated critic: TEMPO uses a continuous, quality-aware critic score instead of binary majority votes to differentiate generated responses.The critic is periodically recalibrated on labeled data during the E-step.
  • TEMPO’s calibrated critic: Periodic labeled-data recalibration keeps the critic grounded in external supervision and the ELBO tight throughout training.This allows meaningful self-improvement over hundreds of iterations without the self-reinforcement trap limiting prior methods.

6 Limitations

TEMPO’s alternating actor-critic procedure introduces memory and compute overhead, depends on labeled calibration data, and has unvalidated scope and theory boundaries.

  • Resource and data requirements: TEMPO requires both actor and critic models, increasing GPU memory and computational overhead relative to single-model TTT methods such as TTRL.The limitation follows from maintaining the alternating E/M-step procedure.
  • Resource and data requirements: Critic quality may depend on the size and distribution of the labeled calibration dataset when generalizing to out-of-domain test questions.The critic recalibration specifically relies on access to a labeled dataset D_L.
  • Scope and theory: Experiments cover math, STEM, and puzzle reasoning, leaving applicability to domains such as code generation unvalidated.The paper identifies broader-domain evaluation as an open boundary.
  • Scope and theory: The EM framing does not provide formal convergence guarantees for the alternating optimization.Formal convergence analysis is left for future work.

7 Conclusion

TEMPO frames LRM test-time training as alternating actor-critic optimization and identifies periodic critic recalibration as the missing E-step in prior methods. Across model scales and reasoning domains, it outperforms baselines while preserving output diversity.

  • Conclusion: TEMPO is a scalable LRM test-time training framework based on alternating actor-critic optimization.Its procedure is framed as EM-style test-time training.
  • Conclusion: Periodic critic recalibration on labeled data is identified as the key deficiency missing from prior TTT baselines.The conclusion links this missing E-step to their performance plateaus and diversity collapse.
  • Conclusion: TEMPO consistently outperforms baselines across model scales and reasoning domains while preserving output diversity.The reported conclusion spans the paper’s empirical evaluation rather than a single benchmark.
  • Future work: Formal convergence guarantees, agentic-task extensions, and critic-calibration efficiency trade-offs remain future-work directions.These are the specific extensions and analyses named in the conclusion.
Loading 2604.19295v1…