PaperlayerView exact revision on arXiv

Source-linked AI summary

Not only where, But when: Temporal Scheduling for RLVR

Jinghao Zhang, Ruilin Li, Feng Zhao, Jiaqi Wang

arXiv:2605.25381v1cs.LG

TL;DR

RLVR’s globally broadcast reward and stagnant credit allocation overlook heterogeneous behaviors along response trajectories. The paper introduces temporal scheduling, which shifts optimization from targeted tokens toward broader training, using trajectory percentiles to organize this progression. Across models, algorithms, and benchmarks, the approach consistently improves results while mitigating policy-entropy loss.

  • Problem

    RLVR optimizes all sampled tokens under a global reward, while existing credit allocation criteria remain stagnant despite heterogeneous trajectory behaviors.

  • Method

    Temporal scheduling evolves credit allocation during training, using trajectory percentiles to prioritize later tokens before progressively incorporating earlier tokens.

  • Results

    Across model scales, RL algorithms, and credit allocation strategies, temporal scheduling consistently improves performance and substantially mitigates sacrificed policy entropy.

  • Takeaways & Limitations

    Temporal scheduling provides a promising optimization dimension that complements advantage reweighting and sparse token optimization.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become a core technique for post-training of Large Language Models (LLMs). While policy optimization is driven by all sampled tokens under a globally broadcast scalar reward, the heterogeneous policy behaviors exhibited along trajectories are largely overlooked without differentiation. Existing works address this by credit allocation, including token-level advantage reweighting, and selective token optimization, however, the allocation criterion are principally stagnant throughout training, limiting resilient policy evolution. In this work, we argue that \textit{when} learning signals are scheduled can be as important as \textit{where} they are allocated across tokens, and introduce the temporal dimension that scheduling the credit allocation criteria over the course of RLVR optimization. We find that prioritizing targeted tokens emphasized with specific policy behaviors, and gradually attenuating toward general optimization leads to more stable and efficient learning dynamics. Furthermore, we show that simple trajectory percentiles provide a natural perspective for distinguishing policy behaviors, and works effectively with temporal scheduling. Our analysis reveals that standard optimization substantially sacrifices policy entropy when simultaneously accommodating heterogeneous behaviors, whereas temporal scheduling yields healthier policy evolution dynamics. Experiments across mathematical and general reasoning benchmarks demonstrate consistent improvements, suggesting that temporal scheduling constitutes a promising optimization dimension.

1 Introduction

RLVR broadcasts one scalar reward across all sampled tokens, overlooking heterogeneous trajectory behaviors. The paper introduces temporal scheduling to evolve credit allocation from targeted tokens toward general optimization.

  • RLVR broadcasts a single scalar reward across the full sequence, so heterogeneous behaviors such as reasoning scaffolding and answer convergence receive undifferentiated optimization.
  • Existing credit allocation methods identify where learning signals should concentrate, but their allocation criteria remain stagnant throughout training.
  • Temporal scheduling varies credit allocation criteria over optimization, initially prioritizing targeted tokens before gradually attenuating toward general optimization.
  • Trajectory percentiles provide a simple perspective for distinguishing heterogeneous policy behaviors and scheduling optimization.
  • Figure 2 analyzes position-specific token distributions and entropy dynamics, showing that temporal scheduling preserves higher entropy throughout training.

2 Preliminaries

RLVR optimizes responses with group-relative advantages and token-level policy gradients, while credit allocation methods reweight or sparsify token contributions using policy-derived proxies.

  • RLVR samples responses from a policy and uses a verifier-assigned global scalar reward to optimize expected response quality.
  • GRPO removes the critic model, estimates each response’s advantage relative to its sampled group, and retains a PPO-style clipped surrogate objective.
  • The importance-sampling ratio corrects distributional differences, while the group-relative advantage is shared across all tokens in a response.
  • Token-level policy optimization allocates globally distributed advantages across tokens using a policy proxy and a wrapper function that determines contribution magnitude.
  • Advantage reweighting uses continuous proxy functions, whereas sparse optimization selects only critical tokens with an indicator-based binary mask.
  • TP-Score estimates where a token most likely occurs along a normalized trajectory by weighting position ranges according to token occurrence frequencies.

3 Method

The method schedules token credit over training, using trajectory percentiles to distinguish policy behaviors and progressively broaden optimization from later to earlier tokens.

  • 3 Method: The method adds a temporal dimension that schedules token optimization throughout training while complementing advantage reweighting and sparse optimization.
  • 3.1 Policy Behavior Analysis: Trajectory percentiles distinguish policy behaviors because sampled token distributions differ substantially between early, middle, and later positions.
  • 3.1 Policy Behavior Analysis: Early percentiles have the lowest entropy, middle percentiles the highest, and later percentiles intermediate entropy, aligning with distinct reasoning behaviors.
  • 3.2 Temporal Scheduling of Credit Allocation: Temporal scheduling starts with targeted credit and gradually attenuates it toward general optimization through a monotonically decreasing schedule.
  • 3.2 Temporal Scheduling of Credit Allocation: For sparse optimization, the method schedules the threshold over training, progressively relaxing it to include more tokens.
  • 3.3 Scheduling over Trajectory Percentiles: Trajectory-percentile scheduling initially focuses on reliable downstream continuations, then incorporates earlier behaviors after partial policy stabilization.

4 Experiments

Experiments evaluate temporal scheduling across models, benchmarks, credit-allocation proxies, entropy dynamics, and ablations. Results show consistent gains, preserved entropy, and sensitivity to scheduling range but not schedule shape.

  • Experimental Setup: Experiments use Qwen3-4B and Qwen3-8B on mathematical and general reasoning benchmarks, comparing GRPO with token-level and trajectory-based allocation methods.The training data comprise 30K filtered problems with verified synthetic reasoning trajectories.
  • Main Results: 2.2% and 2.7%: TP-Schedule outperforms vanilla GRPO on mathematical and general reasoning benchmarks, respectively, for Qwen3-4B.Qwen3-8B shows overall 1% improvements over vanilla GRPO.
  • Temporal Scheduling Analysis: Temporal scheduling preserves higher full-response policy entropy than standard optimization across credit-allocation strategies.Compared with entropy-based advantage reweighting, scheduling avoids rapid entropy collapse and preserves policy entropy over 33.9%.
  • Temporal Scheduling Analysis: Temporal scheduling shifts KL divergence from later toward earlier trajectory percentiles during training, indicating progressively differentiated policy updates.The analysis compares checkpoints at scheduled training progress τ = 0.3 and τ = 0.8 with the initial model.
  • Ablations: Schedule-function choice has a minor performance effect, with sigmoid, gamma, and linear schedules producing consistent improvement over basic GRPO.The linear schedule is adopted as the default setting.
  • Ablations: Extending scheduling to later training stages is more effective, with ε ∈[0.6, 0.8] recommended as a performance sweet spot.This range balances allocation criteria over training for effective policy optimization.
  • Ablations: Trajectory percentiles improve temporal scheduling across reasonable proxies, whereas random token selection and prefix ordering degrade or destabilize optimization.Entropy scheduling improves over its stagnant counterpart by extending optimization from high-entropy tokens toward all sampled tokens.

5 Related Work

Related work frames RLVR as a credit-assignment problem because scalar rewards provide weak token-level attribution. Existing approaches use dense process feedback or token-selective policy signals to address this non-uniformity.

  • Credit Assignment: RLVR broadcasts a single scalar reward across long sequences, making it difficult to attribute advantages to specific actions.Process reward models provide step-level feedback, while policy proxies provide token-level credit estimates.
  • Token-selective Optimization: Token-selective optimization targets the small fraction of tokens that disproportionately dominate gradient updates.This non-uniformity is supported by token-level divergence, ranking-shift statistics, and related analyses.

6 Conclusion

The paper frames temporal scheduling as an optimization dimension beyond where credit is allocated, extending common allocation methods by changing their emphasis over training. It reports preserved policy entropy and consistent improvements across model scales and RLVR algorithms.

  • Trajectory percentiles provide a simple perspective for evolving policy optimization under temporal scheduling.
  • Temporal scheduling extends advantage reweighting and sparse token optimization by prioritizing criterion-selected tokens before gradually broadening optimization.
  • Temporally scheduling policy optimization largely reserves the policy entropy sacrificed by standard optimization.
  • Experiments across model scales and RLVR algorithms demonstrate consistent improvements.

A.1 Training Configurations

The experiments use fixed training infrastructure and on-policy optimization, with linear temporal scheduling as the default and sigmoid and gamma variants additionally explored.

  • Experiments use 8 H200 GPUs, AdamW with weight decay 0.1, a constant learning rate, and gradient clipping at maximum norm 1.0.
  • Each policy update uses 2048 sampled trajectories with on-policy rollout generation throughout training.
  • Linear scheduling is the default temporal schedule, with sigmoid and gamma variants explored additionally.
  • Trajectory percentile is the default credit-allocation setting, starting from the strongest criterion and using the maximal temporal range.

A.2 Scheduling Functions.

Scheduling functions control how optimization expands from stricter early selection toward broader token inclusion later in training. The framework applies these schedules to token-level proxies, including trajectory position, entropy, and random selection.

  • Scheduling values decrease with normalized training progress, so stricter selection early expands toward broader sampled-token optimization later.
  • Scheduling functions: The linear schedule S(τ) = 1 − τ increases optimization scope linearly throughout training.
  • Scheduling functions: The gamma schedule S(τ) = (1 − τ)^γ delays expansion when γ is larger and uses γ = 2 in the experiments.
  • Credit-allocation proxies: Trajectory percentile scheduling expands suffix optimization from later toward earlier tokens, or prefix optimization in the reverse direction.
  • Credit-allocation proxies: Entropy-based scheduling begins with high-entropy tokens and progressively expands toward broader sampled-token optimization.
  • Credit-allocation proxies: Random scheduling selects tokens probabilistically but does not preserve coherent policy behaviors and typically produces unstable optimization.

B Further Analysis on Temporal Scheduling

Temporal scheduling produces stronger, more concentrated early optimization signals and progressively broadens the participating trajectory region. TP-Schedule is reported as especially stable and expansive, whereas random scheduling is unstable.

  • Gradient Norm Dynamics: Temporal scheduling yields larger and more concentrated early gradient norms than standard GRPO, with TP-Schedule showing the most stable trajectory.
  • Gradient Norm Dynamics: Random scheduling produces gradient spikes that disrupt coherent token organization and corrupt accumulated policy gradients.
  • Gradient Norm Dynamics: Standard GRPO maintains smaller gradient norms because optimization is uniformly averaged across sampled tokens.
  • Scheduled Response Length Dynamics: All temporal schedules progressively increase optimization scope from specified token subsets toward broader trajectory regions.
  • Scheduled Response Length Dynamics: TP-Schedule reaches the largest final scheduled response length among the temporal scheduling methods.

C Case Study

The case study contrasts standard GRPO with temporal scheduling on a rotated-parabola problem, showing that temporal scheduling produces more coherent reasoning despite both approaches addressing the same target answer.

  • The derivation formulates the intersection condition with the wrong rotation direction, reversing the signs of the cubic and linear terms in the correct counterclockwise quartic.
  • The solution then switches to clockwise rotation, obtains the point and parameters a = 3, b = 57, c = 2, and reports a + b + c = 62.
  • The task asks for the fourth-quadrant intersection of y = x2 −4 and its 60° counterclockwise rotation, with the y-coordinate expressed using positive integers a, b, and c.
  • The response derives the rotated parabola, substitutes the original equation, and reduces the intersection condition to a quartic equation in x.
  • Standard GRPO can reach the correct final answer while producing inconsistent or logically incorrect intermediate derivations, whereas temporal scheduling more often yields coherent reasoning trajectories.
Loading 2605.25381v1…