Near-Future Policy Optimization

Chuanyu Qin; Chenxu Yang; Qingyi Si; Naibin Gu; Dingyu Yao; Zheng Lin; Peng Fu; Nan Duan; Jiaqi Wang

Source-linked AI summary

Near-Future Policy Optimization

Chuanyu Qin, Chenxu Yang, Qingyi Si, Naibin Gu, Dingyu Yao, Zheng Lin, Peng Fu, Nan Duan, Jiaqi Wang

arXiv:2604.20733v1Apr 22, 2026cs.LG

TL;DR

RLVR needs auxiliary trajectories that are stronger than the current policy without incurring excessive variance. NPO uses near-future checkpoints from the same run, improving average performance from 57.88 to 62.84, while AutoNPO reaches 63.15.

Problem
RLVR’s pure on-policy exploration is limited by sparse early correct trajectories and late-stage plateaus, while existing auxiliary sources trade signal quality against variance cost.
Method
NPO guides the current policy with verified trajectories from a near-future checkpoint, while AutoNPO adaptively times interventions and selects the guide checkpoint.
Results
NPO improves average multi-modal performance from 57.88 to 62.84 (+4.96), and AutoNPO further reaches 63.15 (+5.27) on Qwen3-VL-8B with GRPO.
Takeaways & Limitations
NPO delivers faster early convergence and a higher late-stage performance ceiling, with AutoNPO achieving the best overall score across the reported benchmarks.
Takeaways & Limitations
The core operation ignores minor policy drift accumulated between rollout collection and gradient updates.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become a core post-training recipe. Introducing suitable off-policy trajectories into on-policy exploration accelerates RLVR convergence and raises the performance ceiling, yet finding a source of such trajectories remains the key challenge. Existing mixed-policy methods either import trajectories from external teachers (high-quality but distributionally far) or replay past training trajectories (close but capped in quality), and neither simultaneously satisfies the strong enough (higher $Q$ , more new knowledge to learn) and close enough (lower $V$ , more readily absorbed) conditions required to maximize the effective learning signal $\mathcal{S} = Q/V$. We propose \textbf{N}ear-Future \textbf{P}olicy \textbf{O}ptimization (\textbf{NPO}), a simple mixed-policy scheme that learns from a policy's own near-future self: a later checkpoint from the same training run is a natural source of auxiliary trajectories that is both stronger than the current policy and closer than any external source, directly balancing trajectory quality against variance cost. We validate NPO through two manual interventions, early-stage bootstrapping and late-stage plateau breakthrough, and further propose \textbf{AutoNPO},an adaptive variant that automatically triggers interventions from online training signals and selects the guide checkpoint that maximizes $S$. On Qwen3-VL-8B-Instruct with GRPO, NPO improves average performance from 57.88 to 62.84, and AutoNPO pushes it to 63.15, raising the final performance ceiling while accelerating convergence.

1 Introduction

Pure on-policy RLVR is constrained by sparse early correct trajectories and late-stage plateaus, motivating auxiliary trajectories that balance signal quality against variance cost. NPO uses near-future checkpoints for this balance, while manual interventions and AutoNPO improve convergence and final performance.

Motivation: Pure on-policy RLVR faces sparse correct trajectories early and rollout-distribution plateaus later, motivating mixed-policy auxiliary trajectories.The effective signal depends on balancing trajectory quality Q against variance cost V.
NPO: NPO learns from later checkpoints of the same run, using checkpoint distance ∆ to trade increasing quality Q against increasing variance V.Its effective learning signal is S(∆) = Q(∆)/V(∆), with a unique optimal checkpoint distance.
Manual interventions: Manual interventions show near-future guidance helps both early training and late-stage plateaus, accelerating convergence and breaking through the on-policy ceiling.Early guidance uses a slightly-ahead scout-run checkpoint; late guidance uses a stronger checkpoint from continuing past the plateau.
NPO: NPO is plug-and-play and objective-preserving, and it theoretically and empirically dominates far-future, past, and external-teacher trajectories.These are identified as existing mixed-policy baselines.
Results: 57.88 to 62.84 (+4.96) average multi-modal performance on Qwen3-VL-8B with GRPO; AutoNPO further reaches 63.15 (+5.27).NPO and AutoNPO provide faster early convergence and a higher late-stage performance ceiling.

2 Motivation: The Quality--Variance Trade-off

The section frames auxiliary-trajectory selection as a quality–variance trade-off: guides must be stronger than the current policy yet close enough for their signal to be absorbed. Because quality saturates while variance rises sharply with checkpoint distance, the effective signal S has an interior optimum.

Motivation: Pure on-policy RLVR is constrained by sparse early correct trajectories and collapsed late-stage diversity, limiting capability expansion beyond the base model’s solution space.An effective auxiliary source should provide correct trajectories the current policy cannot reliably discover while remaining close enough to absorb.
Quality–Variance Trade-off: Signal quality Q(∆) increases with a guide checkpoint’s optimization distance, while variance cost V(∆) rises from the resulting gradient mismatch.Q measures verified-correct source trajectories on prompts where the current policy fails; V measures the induced gradient variance.
Quality–Variance Trade-off: As Q saturates and V grows near-exponentially, their ratio S first rises and then falls, producing a unique interior optimum ∆∗.Guides below the optimum provide too little new information, whereas guides above it impose variance costs that exceed their marginal quality gains.
Candidate Guide Policy: A later checkpoint from the same run balances strength and closeness because it shares initialization, architecture, and optimization history with the current policy while differing by bounded gradient steps.This shared training lineage lets it escape current failure modes without the distributional gap associated with external teachers or fully trained-model replay.
Empirical Validation: S peaks at ∆⋆≈20 steps from the base policy (T=0) and ∆⋆≈70 steps from a mid-training policy (T=50), confirming a clear interior maximum.Measurements on Qwen3-VL-8B-Instruct with vanilla GRPO show Q rising quickly before flattening, while V grows sharply and accelerates with ∆.

3 Near-Future Policy Optimization

NPO replaces one rollout in struggling prompts with a verified-correct trajectory from a near-future checkpoint of the same training run, leaving the rest of RLVR unchanged. Manual interventions demonstrate usefulness across training stages, while AutoNPO automates intervention timing and checkpoint selection using online signals and an effective-signal estimate.

NPO mechanism: NPO uses a near-future checkpoint from the same run to provide auxiliary trajectories for prompts the current policy struggles with.It changes only the source of one trajectory within each rollout group.
NPO mechanism: A future checkpoint is rolled out offline, verified, and cached; when the current policy struggles and a correct trajectory exists, it replaces the n-th on-policy slot.Prompts without a correct cached trajectory retain their original on-policy groups.
NPO mechanism: NPO preserves the standard clipped objective, verifier, and group-relative advantage, with importance sampling needed only for the guidance slot.The near-future checkpoint keeps the correction ratio close to one, and omitting the correction performs nearly identically in the cited ablation.
Manual interventions: Early-stage bootstrapping and late-stage plateau breakthrough apply the same guidance mechanism to cold-start and plateau regimes, respectively.Both interventions improve over pure on-policy RLVR without changing the reward, verifier, or optimizer.
AutoNPO: AutoNPO monitors exploration-collapse signals, probes a mistake pool, and selects the rollback distance maximizing an empirical effective signal estimate.Its variance proxy uses per-token KL between the current and rolled-back policies, while all candidate quality estimates come from one confirmation rollout.

4 Experiments

Experiments evaluate NPO and AutoNPO on Qwen3-VL-8B-Instruct using a shared GRPO-style RLVR backbone across eight multimodal reasoning benchmarks. Results support NPO’s quality–variance rationale, show broad gains from manual and adaptive interventions, and indicate improved exploration and training dynamics without requiring exact importance-sampling correction.

Experimental setup: All methods use Qwen3-VL-8B-Instruct with the same GRPO-style group-based RLVR backbone, isolating trajectory-source choice as the performance variable.Baselines span pure on-policy GRPO, external-teacher LUFFY, and replay-based ExGRPO and RLEP.
Experimental setup: Evaluation covers eight multimodal reasoning benchmarks, with Table 1 reporting their unweighted average.The table identifies blue rows as NPO variants and bolds the best result in each column.
Main results: LUFFY is the weakest RL-trained model despite the strongest trajectory source, while ExGRPO and RLEP recover some performance but remain below NPO.LUFFY even regresses below the base on WeMath, illustrating variance costs overwhelming signal quality; replay methods are bounded by their producing checkpoints.
Main results: Early-stage NPO surpasses every baseline on average and leads on ZeroBench, while adding late-stage intervention produces the largest further gains on WeMath and MathVerse.The early intervention helps where correct trajectories are scarce, whereas the late intervention targets reasoning-depth-driven plateaus.
Training dynamics: AutoNPO accelerates initial reward growth, remains above GRPO throughout optimization, and widens its advantage step-wise after intervention windows.Its interventions also re-expand exploration and preserve substantially higher policy entropy than GRPO, which undergoes steady entropy collapse.
Ablations: Both NPO variants stay clearly above vanilla GRPO during the common step 200–510 interval, supporting the practical omission of exact importance-sampling correction.NPO’s near-policy relationship and on-policy pass-rate gating motivate this comparison on the late-stage intervention.

5 Related Work

Related work spans stronger external supervision, replay-based trajectory mixing, and self-distillation, while NPO combines mixed-policy RLVR with self-taught learning by using later-checkpoint trajectories as privileged optimization-time guidance. Its verifier-filtered, prompt-aligned sequence-level guidance differs from same-step context conditioning and token-level distillation.

RLVR is a dominant post-training recipe, and GRPO variants improve on-policy optimization while trajectory-source choice increasingly matters.
External-supervision methods mix teacher traces, expert prefixes, or other outside-policy trajectories into training.
NPO regenerates verifier-verified correct trajectories from a later checkpoint on the same optimization path, unlike replaying past trajectories.
Self-distillation methods use privileged context, successful self-generated traces, or token-level teacher guidance on the student’s rollout distribution.
NPO instead uses optimization time as privileged information, guiding the current policy with verifier-filtered, prompt-aligned later-checkpoint trajectories at the sequence level within standard RLVR.

6 Conclusion and Future Work

The paper proposes Near-future Policy Optimization (NPO), in which a model’s temporal future self guides its present self with strong and readily learnable auxiliary signals. It also situates NPO within the broader Self-Taught RLVR framework and suggests On-Policy Distillation as a future direction.

Conclusion: NPO uses near-future policy trajectories to guide the current policy with sufficiently strong signals and low variance cost.The approach mixes near-future policy trajectories into the current rollout within a mixed-policy framework.
Future Work: Future work could explore alternative mechanisms for injecting learnable signals from the near-future self, including On-Policy Distillation.
Research Program: NPO is the second contribution in the Self-Taught RLVR program and studies temporal self-guidance, complementing prior work on the informed self.The present work examines a near-future self teaching its past self, while prior work studied a self augmented by privileged information teaching the base self.