PaperlayerView exact revision on arXiv

Source-linked AI summary

Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO

Hyun Bin Park, Du-Seong Chang

arXiv:2609.03941v1cs.LGcs.AIcs.CL

TL;DR

Repeated fresh rollouts make reasoning-model RL expensive, while existing replay methods often entangle reuse with broader training machinery. Headroom-Drift Replay isolates replay selection in GRPO by ranking groups by Headroom and gating them by Policy Drift. Across mathematical, multimodal, and Agentic Search reasoning, it outperforms naive replay and matches or exceeds broader replay methods; its transfer beyond GRPO remains open.

  • Problem

    The paper asks how far principled replay selection alone can go, because replay's contribution is difficult to isolate when combined with auxiliary training mechanisms.

  • Method

    Headroom-Drift Replay ranks stored groups by remaining learning value, gates them by current-policy compatibility, and adds accepted full groups to GRPO's unchanged fresh stream.

  • Results

    Across mathematical reasoning, multimodal reasoning, and Agentic Search, Headroom-Drift outperforms naive replay and matches or exceeds broader replay methods; in Agentic Search, it delivers a Pareto improvement over on-policy scaling.

  • Takeaways & Limitations

    Replay controlled along learning-value and policy-compatibility axes functions as a capable optimization primitive in its own right.

  • Takeaways & Limitations

    Held-out evaluation across PPO-, GSPO-, and CISPO-style objective families remains open; current CISPO evidence is preliminary training-score evidence.

Abstract

from arXiv · show

RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost. Replay can reduce this burden by reusing past trajectories, but existing methods typically embed it within larger training pipelines involving exploration, experience restructuring, or mixed-policy optimization. This makes replay's own contribution difficult to isolate. We ask a focused question: how far can principled replay selection alone go? We introduce Headroom-Drift Replay, a group-level replay control primitive for GRPO that separates reuse into two decisions. Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The fresh on-policy stream remains unchanged, and the method adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, this single intervention outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32. In Agentic Search, where environment interaction dominates cost, it delivers comparable quality at materially lower wall-clock time.

1 Introduction

The paper isolates replay selection in GRPO as a two-part control problem: retaining learning value while ensuring compatibility with the current policy. Headroom-Drift Replay addresses this with a single replay-side intervention evaluated across reasoning settings.

  • Repeated fresh rollout generation makes GRPO-style post-training costly, especially when agentic training adds substantial environment-interaction overhead.
  • Reliable replay requires separate judgments of whether stored groups retain learning value and whether they remain compatible with the current policy.
  • Headroom-Drift Replay ranks groups by remaining learning value and gates them by current-policy compatibility, while leaving the fresh on-policy stream unchanged.
  • Headroom-Drift outperforms naive replay across mathematical, multimodal, and Agentic Search reasoning, matching or exceeding methods with additional training machinery.
  • The study compares this isolated primitive with on-policy budgets, fresh-data scaling, naive replay, non-replay alternatives, and broader replay methods across three reasoning domains.
  • The analysis examines replay dynamics through counterfactual comparisons, age compatibility, lifetime exposure, reward ingress, and entropy collapse.

2 Related work

Prior work establishes replay's value for reasoning-oriented RL but generally combines it with broader mechanisms. This paper instead treats replay-side control as a separable, independently evaluable primitive.

  • Prioritized Experience Replay motivates selecting stored experience by expected training utility rather than randomly.
  • Off-policy correction methods address policy mismatch by weighting or truncating the reuse of past experience under the current policy.
  • RePO incorporates replay into GRPO and improves sample efficiency beyond fully on-policy updates in reasoning-oriented language-model training.
  • EFRame, ExGRPO, and BAPO strengthen replay through exploration, filtering, trajectory organization, or adaptive batch construction.
  • The paper positions its two-axis replay control as a composable layer that can stand alone and whose effects can be independently evaluated.

3 Method

Headroom-Drift Replay adds group-level selection to standard GRPO: stored groups are prioritized by Headroom, filtered by Policy Drift, and merged with unchanged fresh groups for the update.

  • 3.1 Method overview.: The method reuses full groups rather than individual responses, preserving GRPO's within-group comparison structure.
  • 3.2 GRPO setup and replay state.: Each replay group stores its actions, generation-time log-probabilities, and response-level advantages under an immutable reference policy.
  • Headroom: Headroom measures remaining directional correction room and provides the stored group's learning-value priority.
  • Policy Drift: Policy Drift measures mismatch between the current policy and the generation policy, and groups pass the compatibility gate only when Drift is at most τ.
  • Replay selection: At each step, fresh groups are generated and scored, while pre-existing buffered groups are ordered by Headroom and scanned under the current policy.
  • Batch and buffer update: Accepted replay groups, up to Krep, are merged with fresh groups for the GRPO update; newly identified ingress groups enter the FIFO buffer only afterward.
  • Efficiency: Replay adds no fresh autoregressive generation or environment interaction because stored trajectories are reevaluated with teacher-forced computation.
  • Design scope: The fresh on-policy rollout path remains unchanged, so changes in training dynamics are attributed to replay-side control.

4 Experiments

The experiments evaluate Headroom-Drift Replay across mathematical, agentic, and multimodal reasoning, using role-aligned baselines and quality–cost analyses. Results show stronger replay selection, broader replay-source diversity, and advantages over naive replay or fresh-data scaling.

  • Experimental setup: Experiments cover mathematical reasoning, Agentic Search, and multimodal reasoning with fixed, verifiable rewards and Mean@32 evaluation.The benchmark suite includes five mathematical, seven agentic-search, and three multimodal datasets.
  • Experimental setup: Role-aligned baselines separate replay reuse, fresh-data scaling, naive replay volume, non-replay alternatives, and broader replay methods.The comparisons include matched and larger on-policy training, GRPO + replay, DAPO, ExGRPO, and BAPO where directly comparable.
  • Mathematical reasoning: Headroom-Drift leads mathematical reasoning on Avg Mean@32 against matched and larger on-policy training, naive replay, DAPO, ExGRPO, and BAPO.The result holds under the fullest comparison set and uses fewer fresh responses than the larger on-policy baseline.
  • Agentic Search: 0.3577 vs. 0.3548: Headroom-Drift’s Agentic Search Avg Mean@32 advantage over matched naive replay is small, while Avg Best@32 is 0.4879 vs. 0.4623.The paper presents the latter gap as evidence that principled selection shifts the upper tail of trajectory quality more than the average.
  • Agentic Search: 0.3577 vs. 0.3212: Headroom-Drift exceeds larger on-policy training while costing 166.3 s vs. 197.2 s per step.The comparison uses 1.5× the fresh rollout budget for GRPO on-policy larger.
  • Multimodal reasoning: 0.4137 vs. 0.3986, 0.4005, and 0.4056: Headroom-Drift leads Avg Mean@32 in the simplified multimodal comparison.The baselines are GRPO on-policy matched, GRPO on-policy larger, and DAPO.
  • Replay-control analysis: Headroom-Drift selects replay subsets with gains in both compatibility and learning value across most matched-buffer training steps.Compatibility gain is plotted on x and learning-value gain on y; most steps fall in the jointly improved quadrant.
  • Replay-control analysis: Groups with initial stored Headroom below 0.8 are replayed rarely, whereas groups above 0.8 average approximately 2.0 replays and exceed 56% replay probability.Lower-Headroom groups remain eligible when the Policy Drift gate finds them sufficiently compatible.

5 Conclusion and future directions

The paper concludes that Headroom-Drift Replay is a capable replay-side optimization primitive across evaluated reasoning settings. It also identifies integration with broader replay methods and validation across other objective families as future directions.

  • Conclusion: Headroom-Drift decomposes replay into learning-value prioritization and current-policy compatibility control.The primitive is studied across mathematical reasoning, Agentic Search, and multimodal reasoning.
  • Conclusion: Across the evaluated domains, the single intervention outperforms naive replay, matches or exceeds broader replay methods on mathematical Avg Mean@32, and improves the Agentic Search quality–cost trade-off.The Agentic Search result is described as a Pareto improvement over on-policy scaling.
  • Future directions: Held-out evaluation across PPO-, GSPO-, and CISPO-style objective families remains open.Preliminary CISPO-style training-score evidence suggests transfer beyond GRPO, but does not establish held-out generalization.
  • Future directions: The primitive is designed to compose with exploration, filtering, and adaptive batch-construction strategies in replay-centric methods.The paper frames it as a separable control layer rather than a competing pipeline.

Ethics Statement

The ethics statement scopes the work to replay selection for reinforcement-learning post-training of reasoning models and reports risks related to capability misuse and compute consumption.

  • Scope: The work studies replay selection mechanisms for reinforcement-learning post-training of reasoning models using standard datasets and simulated training environments.No new human-subject data were collected.
  • Risks: Potential risks include misuse of improved reasoning capability and increased compute consumption in large-scale training.The authors state that they report explicit scope limits and quality–cost trade-offs under the evaluated settings.
  • Disclosure: During camera-ready preparation, a large language model assisted with prose revision and drafting, and the authors reviewed the resulting text.The authors state that they take responsibility for the paper’s content.

A Formal interpretation, mixed-batch objective, and replay-step formalization

The appendix formalizes Headroom-Drift Replay as a constrained augmentation to GRPO: cached Headroom ranks buffered groups, Policy Drift filters incompatible groups, and accepted replay is combined with unchanged fresh on-policy data.

  • A.1 Notation and replay-state conventions: Headroom-Drift distinguishes reference, cached, current-policy, and runtime Headroom, with cached Headroom determining current-step replay ordering.Cached priorities start from generation-time Headroom and are refreshed only for groups actually scanned.
  • A.2 Mixed-batch objective as constrained replay augmentation: The mixed-batch objective preserves the fresh on-policy stream and adds a constrained replay term over accepted buffered groups.Replay augments rather than replaces fresh rollout training.
  • A.3 Headroom and policy drift: Headroom measures remaining directional correction room for stored actions, while Policy Drift measures current-policy mismatch on those actions.Policy Drift uses a non-cancelling squared aggregation of sampled tokenwise log-ratio magnitudes.
  • A.3 Headroom and policy drift: The policy-drift bound provides a conservative lower bound on current-policy Headroom for replay groups that pass the Drift gate.A separate ranking certificate shows when reference-Headroom margins preserve current-policy ordering.
  • A.4 Replay multiplicity and lifetime exposure: Replay exposure counts include each group’s initial fresh use and later replay uses, with cumulative objective exposure tracking its total training mass.Lifetime reuse depends on ingress, FIFO residence, Headroom ordering, and Policy Drift admissibility.
  • A.5 Replay-step formalization: Replay selection scans pre-existing buffered groups in descending cached Headroom order and accepts only Drift-feasible groups within the replay budget.Current-step ingress is appended after replay selection and the GRPO update.
  • A.5 Replay-step formalization: The accepted replay set is the cardinality-constrained subset with maximum total cached Headroom among Drift-feasible candidates.This is equivalent to taking the top Krep feasible groups under cached-Headroom ordering.

B.1 Metric definitions

The section defines Best@n and Mean@n, clarifies that Avg Mean@n is usually macro-averaged across benchmarks, and motivates answer-aligned replay ingress for multi-reward settings.

  • B.1 Metric definitions: Best@n takes each input’s maximum score across n samples, whereas Mean@n averages its n sample scores before averaging over inputs.Scores are scalar values in [0, 1].
  • B.1 Metric definitions: Table 3 reports held-out mathematical-reasoning Best@32 / Mean@32 values and compares Headroom-Drift with recency-only replay baselines.The recency-only baselines use the latest 128 or 256 buffered groups without Headroom ranking or Policy Drift gating.
  • B.1 Metric definitions: Macro Avg Mean@n is the unweighted average of benchmark-level Mean@n values, while Weighted Mean@n weights benchmarks by size.The paper’s Avg Mean@n notation ordinarily means Macro Avg Mean@n.
  • B.1 Metric definitions: In multi-reward settings, replay ingress should target within-group variation that reflects answer correctness rather than auxiliary formatting signals.Geometry3K combines answer correctness and formatting quality, so final-reward positivity is not an exact correctness indicator.

C.2 Engineering observation: replay ingress drying

The engineering analysis shows that replay supply and compatibility evolve during training: coarse multi-reward success labels can dry ingress, while Headroom and Drift jointly concentrate and constrain replay selection.

  • C.2 Engineering observation: replay ingress drying: Using the original success indicator in Geometry3K causes replay ingress to decrease sharply as training progresses.The indicator can classify format-only positives as successful and exclude groups that still mix answer-correct and answer-incorrect responses.
  • C.2 Engineering observation: replay ingress drying: Many groups labeled all-success under the original indicator still contain answer-incorrect samples rewarded positively by formatting.Thus all-success labels do not uniquely identify answer-correct groups in the multi-reward setting.
  • C.2 Engineering observation: replay ingress drying: Answer-based ingress is associated with better training-score dynamics while preserving sufficient replay-buffer supply.The stricter criterion does not cause replay ingress to dry out prematurely within the observed window.
  • C.2 Engineering observation: replay ingress drying: Older replay groups are less likely to pass Policy Drift, but age alone does not determine compatibility because young groups can also be rejected.Finite FIFO residence also mechanically limits the large-age tail.
  • C.2 Engineering observation: replay ingress drying: Accepted replay is concentrated near the top of the Headroom-ordered buffer, while later training requires deeper scans and may fail to fill the replay budget.The pattern reflects a tightening set of groups that are both high-priority and near-policy.

D.3 Initial stored headroom and lifetime replay pressure

Initial stored Headroom is associated with stronger lifetime replay exposure, while replay remains highly concentrated among a small subset of groups. This selection pressure is substantial but not deterministic because compatibility and buffer constraints also shape reuse.

  • Higher initial stored Headroom is associated with both more frequent reuse and a higher probability of replay at least once.This indicates that Headroom drives actual selection pressure rather than serving only as a definitional score.
  • Higher initial stored Headroom does not fully determine reuse because Policy Drift admissibility, scan-prefix competition, budget truncation, and buffer residence also intervene.The strongest Headroom deciles are favored, but the relationship is not strictly monotone and some lower-decile groups are replayed.
  • Most buffered groups are never replayed, while repeatedly reused groups account for a disproportionately large share of total replay exposure.Cohort share and exposure-mass share therefore describe different aspects of replay concentration.
  • Headroom-ranked, Drift-gated replay redistributes lifetime training exposure toward a comparatively small subset rather than spreading reuse uniformly.A sparse ever-replayed cohort can still exert substantial optimization influence when its repeated users receive much of the replay mass.

E.2 Experimental setting and comparison axes

The entropy-collapse analysis compares matched on-policy and replay configurations using shared training-score anchors to separate replay effects from differences in optimization speed. The selected analysis tracks low-entropy entry and rebound across comparable score levels.

  • Experimental setting: The comparison uses standard on-policy, Headroom-Drift Replay, and larger-batch on-policy runs sharing the same model, framework, and dataset.The larger-batch control uses six mini-batches like the replay run, helping separate replay structure from simply processing more mini-batches.
  • Matched-score analysis: Matched-score analysis defines delays from a shared training-score anchor to low-entropy entry and from low-entropy entry to rebound.The analysis uses consecutive-log-point criteria to reduce sensitivity to short-lived fluctuations.
  • Matched-score analysis: The representative criteria use S ≥0.40, low-entropy entry threshold HL = 0.03, and exit threshold HU = 0.04.These thresholds are used to summarize score-conditioned delays and visualize entropy trajectories.
  • Observed patterns: Headroom-Drift Replay reaches S ≥0.40 earlier than larger-batch on-policy training but enters the low-entropy regime later.This pattern indicates that score improvement and entropy compression are not strictly synchronized.
  • Observed patterns: At matched scores, replay shows a longer low-entropy-entry delay than larger-batch on-policy training, 237 versus 207, and rebounds later.Replay therefore reaches the shared high-score regime earlier while entering and exiting the low-entropy regime later.
  • Late-stage transition: The used replay subset combines multiple recent source ages rather than trivial one-step reuse, but this mixture contracts sharply near collapse.Late-stage replay volume falls despite continued ingress, and accepted groups become increasingly concentrated in very recent sources.

F Implementation details for headroom-drift replay

The implementation preserves standard rollout, reward, advantage, ingress, and FIFO operations while adding group-level selection that ranks by cached Headroom and gates by Policy Drift. Selection refreshes cached scores during the same forward pass and stops once the replay budget is filled.

  • Replay state: The buffer stores generation-time log-probabilities and response-level advantages as immutable replay state, while only current-policy numerators are recomputed during PPO updates.The importance-ratio denominator remains anchored to the generation-time policy.
  • Selection: Replay selection scans candidates in descending cached-Headroom order and accepts groups satisfying Drift(g; θt) ≤τ until Krep is filled.Early termination makes selection depend on the replay budget and candidates scanned rather than the full buffer size.
  • Selection: A parallelized current-policy forward pass computes Policy Drift and refreshes cached Headroom for every scanned group.The refresh occurs whether a group passes or fails the gate, without an additional forward pass.
  • Gate behavior: The L2 gate outperforms L1 through selection quality rather than replay volume: at step 350, Mean@4 is 0.3636 versus 0.3565 and replay KL is 8.5 × 10−5 versus 5.9 × 10−4.The two gates fill similar replay budgets through the middle phase, so the difference is not explained by quantity alone.
  • Gate behavior: L2 selection concentrates early on low-success, high-Headroom groups and broadens as the high-value region is depleted.The effective selectivity adapts to the available candidate pool over training.
  • Threshold selection: MODERATE balances replay supply and policy mismatch, whereas TIGHT accepts 6.4% of candidates and LOOSE accepts 94.4%.MODERATE keeps replay KL approximately 0.002, while the permissive gate allows replay KL to rise over training.

G.3 Component ablation on mathematical reasoning

The component ablation shows that combining Headroom with Policy Drift gives the strongest mathematical-reasoning endpoint performance, while sign-only Headroom is retained because it outperforms advantage-weighted alternatives in a preliminary replay comparison.

  • Component ablation: The full Headroom-Drift combination records the highest endpoint Best@32 and Mean@32 values on MATH-500.Headroom-only and Policy-Drift-only are both lower under the same training configuration.
  • Component ablation: The full variant remains above Headroom-only and Policy-Drift-only in the late-stage training-score trajectories.The full-to-Headroom-only margin is modest, while the gap to Policy-Drift-only is larger later in training.
  • Headroom weighting: Sign-only Headroom achieves a higher average training score than advantage-weighted Headroom over steps 1–38, 0.3288 versus 0.3022.Sign-only leads on 32 of 38 steps and on every step from 19–38, when both runs use replay.
  • Headroom weighting: Advantage weighting deprioritizes failure-heavy groups that sign-only Headroom ranks highly because those groups can carry useful corrective signal.The two variants therefore induce qualitatively different ranking geometries rather than minor weighting changes.

H Preliminary portability to a CISPO-style objective

Preliminary CISPO-style experiments reuse the same Headroom-Drift components and hyperparameters while changing only the policy-optimization objective. The replay trajectory outperforms matched on-policy training, exceeds the larger on-policy baseline late, and remains more stable after step 400.

  • The CISPO-style test changes only the policy-optimization objective, keeping Headroom ranking, Policy Drift gating, buffer mechanics, and replay hyperparameters unchanged.
  • Headroom-Drift Replay outperforms the on-policy matched baseline (b256) across nearly the entire training trajectory.Figure 25 compares full Headroom-Drift Replay (b256+r128) with the matched on-policy control.
  • 0.522 vs. 0.510: Headroom-Drift Replay reaches a higher raw peak score than the on-policy larger baseline (b384) despite fewer fresh responses.The peaks occur at steps 380 and 343, respectively.
  • After step 400, the on-policy larger baseline shows a sharp score collapse, whereas the replay trajectory remains more stable through the same region.This pattern mirrors the delayed entropy-collapse observation reported under GRPO.
Loading 2609.03941v1…