PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Rollouts Are Worth Learning: On Trajectory Valuation for Post-Training Reinforcement Learning

Xuesong Jia, Ziao Yang, Zhanhe Huang, Hongfu Liu

arXiv:2609.35072v1cs.LG

TL;DR

Reinforcement-learning trajectories are generated online, without the fixed validation signals used in classification, and their utility depends on the policy and reward dynamics. The paper proposes DTV, which values and filters training units through mini-batch gradient alignment. Experiments across PPO, GRPO, and DPO report improved performance, data efficiency, and optimization stability.

  • Problem

    Online reinforcement learning lacks fixed validation signals for valuing dynamically generated, policy-dependent trajectories, making conventional influence-based valuation difficult to apply.

    Source: §1 · p. 1, §4.1 · p. 4

  • Method

    DTV uses mini-batch gradient alignment to identify and filter detrimental training units, without a predefined validation set or influence-function approximation.

    Source: §1 · p. 2, §4.2 · p. 5, §6 · p. 12

  • Results

    Across PPO, GRPO, and DPO, experiments report consistent performance improvements, enhanced data efficiency, and more stable optimization.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    The reported evaluations support dynamic trajectory valuation across diverse reinforcement-learning and LLM post-training paradigms.

    Source: §1 · p. 2, §6 · p. 12

    DTV relies on the assumption that most sample gradients in a mini-batch provide a locally informative reference direction for the current update.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

We consider the problem of trajectory valuation in reinforcement learning: how to identify and mitigate detrimental trajectories during online training. Unlike classification, where data valuation relies on fixed training and validation sets, reinforcement learning involves dynamically generated trajectories without explicit validation signals, making conventional influence-based methods inapplicable. We propose Dynamic Trajectory Valuation (DTV), a simple and efficient framework that estimates trajectory utility at the mini-batch level and filters detrimental trajectories based solely on gradient information. By operating at the optimization level, DTV integrates seamlessly with existing reinforcement learning pipelines with minimal overhead. Extensive experiments across diverse settings, including PPO, GRPO, and DPO, demonstrate that DTV consistently improves performance, enhances data efficiency, and stabilizes optimization.

1 Introduction

Online reinforcement-learning trajectories are generated dynamically, and their policy-dependent value is difficult to assess without fixed validation signals. The paper formulates trajectory valuation and proposes DTV to identify and filter detrimental training units using mini-batch gradients.

1 Introduction

  • Unlike classification, reinforcement learning generates trajectories online without a fixed validation set, while trajectory utility depends on policy, temporal credit assignment, and reward dynamics.These differences make consistent impact measures and efficient online valuation difficult.Source: §1 · p. 1
  • Prior valuation methods in reinforcement learning use replay buffers, surrogate impact functions, verifiable feedback, or trusted validation gradients, raising concerns about generalizability.The paper frames trajectory valuation as a response to these setting-specific designs.Source: §1 · pp. 1–2
  • The paper formulates trajectory valuation for general reinforcement learning, addressing dynamic data generation, absent validation signals, and policy-dependent utility.The formulation aims to unify trajectory valuation across diverse reinforcement-learning pipelines.Source: §1 · p. 2
  • DTV dynamically identifies and filters detrimental trajectories at the mini-batch level using gradient information, integrating with existing reinforcement-learning pipelines.The method operates purely on gradients.Source: §1 · p. 2
  • Across PPO, GRPO, and DPO, DTV consistently improves performance, data efficiency, and optimization stability.The experiments span diverse reinforcement-learning settings.Source: §1 · p. 2

2 Related Work

The related work reviews data selection in reinforcement learning and influence-function approaches to sample valuation. It identifies the lack of a unified, principled framework for valuing complete trajectories across reinforcement-learning settings.

2.1 Data-Centric Learning in Reinforcement Learning

  • Existing reinforcement-learning valuation methods use replay buffers and surrogate objectives [9], verifiable feedback [10], or trusted validation gradients [11], potentially limiting applicability across paradigms.The paper positions its approach as a principled alternative to these auxiliary-signal-dependent methods.Source: §2.1 · p. 2, §2.2 · p. 3
  • Transition-level heuristics and coarse trajectory filtering leave a gap in principled trajectory valuation for reinforcement learning.Local signals such as temporal-difference error or immediate rewards may miss long-range dependencies, while output filtering relies on coarse signals.Source: §2.1 · p. 2

2.2 Influence Function

  • Influence functions estimate how training samples affect validation performance; efficient approximations address the computational cost of inverse-Hessian estimation.Examples include LiSSA [4], EKFAC [7], DataInf [5], TRAK [6], and Hessian-free methods [8] [26] [27].Source: §2.2 · pp. 2–3
  • Optimization-aware methods estimate sample effects along training paths, while dynamic valuation estimates value at training steps rather than aggregating checkpoints.The reviewed approaches include SGD-Influence [28], MoSo [29], Z0-Inf [30], and dynamic methods [31] [32].Source: §2.2 · p. 3

3 Preliminaries

The preliminaries establish the reinforcement-learning objectives and optimization methods used in the paper, then introduce influence functions as a way to estimate training-sample effects on validation performance.

3.1 Reinforcement Learning

  • Reinforcement learning models an agent interacting with an environment to maximize expected discounted return, with trajectories representing sequences of states and actions.The framework uses a Markov Decision Process and a policy that maps states to action distributions.Source: §3.1 · p. 3
  • Policy-gradient methods estimate the objective gradient using policy log-probability gradients weighted by the advantage function.The advantage measures an action's relative contribution compared with a baseline policy.Source: §3.1 · p. 3
  • PPO stabilizes policy-gradient training with a clipped surrogate objective that limits excessively large policy updates.Its objective clips the importance-sampling ratio around the old policy.Source: §3.1 · p. 3
  • GRPO normalizes rewards within sampled groups to form relative advantages, whereas DPO learns from preferred and rejected response pairs without explicit reward modeling.GRPO reduces reliance on value-function estimation; DPO uses a reference policy and bypasses explicit reward-model training.Source: §3.1 · pp. 3–4

3.2 Influence Function

  • Influence functions approximate how removing a training sample changes a validation objective by perturbing its contribution to the training objective.The formulation uses the sample gradient and Hessian to estimate the effect on model behavior.Source: §3.2 · p. 4

4 Method

Section 4 frames trajectory valuation as an online reinforcement-learning problem and presents DTV as a unified gradient-based framework, followed by a decomposition and leave-one-out extension.

4.1 Challenges of Influence Functions in Reinforcement Learning

  • Conventional influence valuation is difficult to transfer to reinforcement learning because trajectories are generated online, lack a fixed validation set, and require efficient dynamic valuation.Trajectory usefulness is also policy-dependent and shaped by temporal credit assignment and reward dynamics.Source: §4.1 · p. 4

4.2 Dynamic Trajectory Valuation

  • Under majority gradient alignment, DTV treats a training unit’s negative alignment with the mini-batch reference direction as detrimental and filters negative-scoring units before the optimizer update.A training unit is a PPO trajectory segment, a GRPO completion trajectory, or a DPO prompt with a chosen–rejected response pair; DTV uses current mini-batch gradients rather than a separate impact function or validation set.Source: §4.2 · p. 5

4.3 DTV Decomposition and Leave-One-Out Extension

  • DTV decomposes each score into self-alignment and aggregate gradient agreement with the remaining training units; a sufficiently large self-term can offset a negative cross-term.The self-term’s effect depends on the optimization setting and is not universally detrimental.Source: §4.3 · p. 5
  • DTV-Loo excludes the evaluated unit from its reference direction, removing the self-term while preserving exact leave-one-out normalization.The DTV-λ family weights the self-term: λ=1 recovers DTV and λ=0 recovers DTV-Loo; the paper retains the two as practical instantiations without additional λ tuning.Source: §4.3 · pp. 5–6
  • DTV-Loo performs better than DTV in most evaluated settings, while DTV can be more effective in specific regimes.The two methods therefore offer complementary strengths without requiring tuning of the broader DTV-λ parameter.Source: §4.3 · p. 6

5 Experiments

Across PPO, GRPO, and DPO, the experiments test adaptive gradient-based trajectory valuation on different training units and corruption regimes. DTV-Loo leads in many settings, while DTV remains competitive or stronger when conflicts are weak; both variants show benefits in performance, convergence, or data efficiency.

5 Experiments

  • The experiments span PPO trajectory segments, GRPO completion groups, and DPO preference pairs, with evaluation on MiniGrid, GSM8K, AIME, and UltraFeedback.AIME uses one training run per method; the other experiments described in the setup use five matched seeds.Source: §5 · p. 6, §5.2 · pp. 7, 9, §5.3 · p. 10

5.1 DTV for Online Trajectory Valuation with PPO

  • Across both MiniGrid tasks, DTV-based filtering accelerates PPO convergence; on DoorKey-8x8, both variants maintain an advantage over IIF and Vanilla PPO.DTV-Loo converges substantially earlier on Empty-8x8, where all methods ultimately attain comparable returns.Source: §5.1 · p. 6, Figure 2 · p. 7

5.2 DTV for Group-Relative Completion Valuation with GRPO

  • On GSM8K, DTV improves exact accuracy over the shared pre-trained model by 0.0760 under Clean and 0.0690 under Mismatch-20%, outperforming the fixed-filter baselines.DTV also achieves the best exact and partial accuracy in both settings.Source: Table 1 · p. 8, §5.2 · p. 9
  • On AIME, DTV-Loo improves Pass@1 from 0.1063 to 0.1333, Pass@16 from 0.3333 to 0.4333, and Maj@16 from 0.2333 to 0.3667.It also attains the highest extractable-answer rate (0.7521) and accuracy given an extractable answer (0.1773).Source: §5.2 · pp. 9–10, Table 2 · p. 9

5.3 DTV for Pairwise Preference Valuation with DPO

  • DTV-Loo achieves the best AUC and final-checkpoint accuracy across all three DPO conditions, with its margins over Reward 10% widening under heavier corruption.Under Mismatch-40%, the AUC margin is +0.0274 and the Acc. margin is +0.0339.Source: §5.3 · p. 10, Table 3 · p. 11
  • DTV-Loo's advantage over Vanilla DPO becomes larger and more persistent as preference corruption increases, with the strongest separation under Mismatch-40%.Under Clean, its early-stage advantage narrows as training proceeds.Source: §5.3 · p. 10, Figure 4 · p. 11
  • Under corrupted preference settings, DTV-Loo reaches the T95 target in approximately 30 minutes, while Random 10% misses it under Mismatch-40% within the training budget.The target is defined from the five-seed best validation performance; the Figure 4 bottom row reports seed-0 wall-clock time.Source: §5.3 · pp. 10–11, Figure 4 · p. 11
  • DTV's self-term favors conservative filtering, while DTV-Loo's cross-unit-only scoring is more robust when conflicts are consequential; DTV can remain preferable when conflicts are weak.On GSM8K, weak negative cross-terms accompany similar performance, with DTV slightly ahead; the DPO analysis describes stronger cross-unit conflicts.Source: §5.2 · p. 9, §5.3 · pp. 11–12

6 Conclusion

The conclusion presents DTV and DTV-Loo as gradient-alignment methods for identifying detrimental training units, with experiments spanning several reinforcement-learning and post-training settings. The appendix adds related work, implementation details, and extended DPO results.

6 Conclusion

  • DTV identifies detrimental training units through mini-batch gradient alignment without a predefined validation set or influence-function approximation.DTV-Loo removes the self-term, whose utility can depend on the optimization setting.Source: §6 · p. 12
  • Experiments cover PPO, GRPO, and DPO across trajectory segments, completion trajectories, and preference pairs.The paper reports improved optimization efficiency, robustness, and final performance across these settings.Source: §6 · p. 12

Appendix

  • The appendix provides additional related work, implementation details for PPO, GRPO, and DPO, and extended DPO experimental results.Source: Appendix · p. 16

A Additional Related Work on on Data-Centric RL

This section compares gradient-based trajectory-valuation methods by their requirements and applicability. DTV and DTV-Loo perform online valuation from current batch gradients without auxiliary signals, a fixed dataset, or extra rollouts.

A Additional Related Work on on Data-Centric RL

  • DTV and DTV-Loo use current batch gradients online, a fixed zero threshold, and no explicit filtering-rate or selection-budget hyperparameter.Their filtering rule applies across GRPO, DPO, and PPO.Source: A Additional Related Work on on… · p. 16, Table 4 · p. 16
  • IIF requires an algorithm-specific surrogate target and filtering parameter, while LearnAlign relies on verifiable feedback and a fixed problem dataset.IIF reuses the current rollout buffer; LearnAlign requires additional rollouts to estimate learnability.Source: A Additional Related Work on on… · p. 16
  • GradAlign requires a trusted validation set, is developed for GRPO, and uses additional rollouts to estimate validation and candidate gradients.Table 4 compares methods by valuation signals, dataset and validation requirements, selection budgets, extra rollouts, online use, and generalizability.Source: A Additional Related Work on on… · p. 16, Table 4 · p. 16

B Additional DTV Experimental Details for PPO

This appendix details the PPO experiments, including MiniGrid environments, comparison methods, filtering, configurations, and evaluation. It also introduces additional GRPO experimental details.

B.1 Environments

  • PPO is evaluated on partially observable, sparse-reward MiniGrid environments: Empty requires reaching the goal, while DoorKey also requires collecting a key and unlocking a door.The policy receives a symbolic 7 × 7 × 3 image observation.Source: §B.1 · p. 16
  • DTV and DTV-Loo value trajectory units, whereas Vanilla PPO has no filtering and IIF filters transition-level records using a return-based impact target.IIF removes the bottom 12.5% of negatively influential records and uses the current rollout buffer.Source: §B.1 · p. 16
  • PPO filters units with negative policy-loss scores and removes their transitions from the full objective, reusing the mask across epochs and mini-batches.Filtering starts immediately on Empty and after five unfiltered warmup rounds on DoorKey; IIF uses the same schedule.Source: §B.1 · p. 17
  • All methods share a convolutional actor–critic and base PPO configuration, with learning curves averaged over five seeds and final Worst/Best subsets drawn from 1,000 episodes.Table 5 summarizes PPO training and evaluation configuration.Source: §B.1 · p. 17, Table 5 · p. 17

C Additional DTV Experimental Details for GRPO

C.1 Datasets and Benchmarks

This section describes the GSM8K and AIME data partitions and the Mismatch-20% corruption protocol used in the GRPO experiments.

C.1 Datasets and Benchmarks

  • GSM8K uses 691 training batches and 77 clean evaluation batches from 3,072 shuffled examples, with its official test split reserved for final evaluation.The selected partition yields 691 optimizer updates.Source: §C.1 · p. 17
  • Mismatch-20% reverses the four observed rewards within selected prompt groups, preserving their reward multiset, mean, and standard deviation while disrupting completion–reward associations.Ties can leave selected groups partly or entirely unchanged, and evaluation data remain clean.Source: §C.1 · pp. 17–18
  • AIME evaluation uses all 30 AIME 2024 problems, while training uses 40,192 selected prompts in 314 updates and one seed.Training prompts come from the DeepScaleR-Preview-Dataset; the group reward is binary mathematical correctness.Source: §C.1 · p. 18

C.2 Experimental Setup

The experiments compare adaptive trajectory filtering with fixed-rate and other selection baselines across GRPO and DPO settings. The setup specifies valuation units, training protocols, and clean evaluation; the added DPO experiments also test corrupted training preferences and external instruction-following benchmarks.

C.2 Experimental Setup

  • DTV and DTV-Loo use adaptive completion-level filtering with a zero valuation threshold, while comparison methods include vanilla, fixed-rate, and prompt-level selection baselines.The GRPO setup evaluates GSM8K and AIME; the AIME comparison focuses on Vanilla GRPO, DTV, and DTV-Loo.Source: §C.2 · p. 18
  • GSM8K updates value four completions per prompt group from policy-only gradients, then apply the selected mask to the full policy-plus-KL update without recomputing advantages.DTV uses all four completion gradients as its reference; DTV-Loo uses the other three.Source: §C.2 · p. 18
  • AIME updates use 128 prompts with eight completions each; DTV compares each gradient with all eight, while DTV-Loo uses the other seven.A shared mask removes groups with eight zero advantages; AIME DTV-Loo has no minimum-retention rule.Source: §C.2 · p. 19
  • The GSM8K evaluation uses five training seeds and final step 691 on the official 1,319-example test split, while the AIME protocol samples 16 responses per problem.AIME reports Pass@1, Pass@16, Maj@16, extractability, and accuracy conditioned on extraction.Source: §C.2 · p. 19
  • DTV and DTV-Loo training takes approximately 168–192 hours versus 120–144 hours for Vanilla GRPO, motivating a single-training-seed AIME protocol.Evaluating all four checkpoints sequentially takes 32.1 hours, excluding initialization and restoration.Source: §C.2 · p. 19

D.1 Dataset and Mismatch Construction

  • DPO uses prompt-level partitions, holds out clean prompts, and corrupts selected training pairs by swapping responses across donors while inverting preference labels.The corrupted pairs are shared across methods matched by seed; evaluation data remain clean.Source: §D.1 · pp. 19–20

D.2 Experimental Setup

External instruction-following results are mixed: DTV remains broadly competitive, but DTV-Loo’s in-domain gains do not uniformly transfer. The DPO setup compares adaptive pair filtering with fixed-rate baselines and evaluates clean held-out preference accuracy.

D.2 Experimental Setup

  • DTV and DTV-Loo value each prompt–response pair using gradients of the complete DPO loss, retaining non-negative finite scores rather than prescribing a filtering rate.DTV uses the accumulation-window average gradient as its reference; DTV-Loo uses the average over the remaining pairs.Source: §D.2 · p. 20
  • Figure 6 plots absolute preference-accuracy trajectories for Clean, Mismatch-20%, and Mismatch-40% training conditions, with five-seed means and ±1 sample standard deviation.The supplied caption specifies the plotted conditions and variability, not comparative outcomes.Source: Figure 6 · p. 21
  • All methods use the final step-115 checkpoint, with five matched seeds; AUC measures normalized area under the clean held-out preference-accuracy curve over training.Held-out pairs are excluded from DTV’s reference gradient.Source: §D.2 · p. 21

D.3.1 Downstream Instruction-Following Results

  • DTV remains broadly competitive on three clean external instruction-following benchmarks, while DTV-Loo’s stronger in-domain gains do not uniformly transfer across external metrics.The benchmarks are LiveBench-IF, RewardBench 2 Precise IF, and IFBench P-Strict.Source: §D.3.1 · p. 21

D.3.2 Training Curves and Filtering-Rate Ablation

Under mismatch training, DTV-Loo improves faster and more strongly, with the largest separation at Mismatch-40%. Increasing fixed filtering rates does not reproduce that trajectory, underscoring the importance of which pairs are removed.

D.3.2 Training Curves and Filtering-Rate Ablation

  • DTV-Loo maintains faster and stronger improvement under both mismatch settings, with the largest separation under Mismatch-40%.Figure 6 shows absolute preference-accuracy trajectories with five-seed means and ±1 sample standard deviation.Source: §D.3.2 · p. 21
  • Under Mismatch-40%, DTV-Loo optimizes faster and performs better overall than fixed-rate variants, even as higher Reward-based Filtering rates improve on lower-rate variants.Figure 7 compares DTV and DTV-Loo with Random and Reward-based Filtering at 5%, 10%, 20%, and 40%.Source: §D.3.2 · p. 21, Figure 7 · p. 22
  • The advantage is not explained solely by filtering more pairs: which pairs are removed matters, and a fixed zero valuation threshold lets DTV’s effective filtering rate adapt during training.The ablation tests fixed-rate sensitivity, not the theoretical DTV-λ family.Source: §D.3.2 · p. 22
Loading 2609.35072v1…