PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning

Shixuan Liu, Cheng Tang, Yuzheng Hu, Fan Wu, Han Zhao, Jiaqi W. Ma

arXiv:2610.28883v1cs.LGcs.AI

TL;DR

Online-RL attribution has largely focused on single-round influence, leaving data’s effects across later training rounds uncharacterized. This paper defines trajectory-level RL-LOO and estimates it with RL-IF, then uses an optimization-only approximation in RINSE, which improves efficiency and final performance in the reported settings.

  • Problem

    Existing online-RL attribution is primarily local to one training round, leaving the influence of data across multiple rounds unexplored.

    Source: §1 · p. 2

  • Method

    The paper defines trajectory-level RL-LOO and derives RL-IF to propagate a sample’s influence through later optimization and policy-induced sampling, then builds RINSE using an optimization-only approximation.

    Source: §1 · p. 2, §3.1 · p. 4, §4 · p. 7

  • Results

    RINSE improves training efficiency and final performance over standard PPO and local filtering baselines on standard benchmarks and in an RLHF-style toxicity-mitigation setting.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The optimization-only approximation is reported as effective enough to support practical filtering, while RL-IF provides a counterfactual perspective on data value beyond local attribution.

    Source: §6 · p. 10

    The framework is specialized to PPO-style on-policy training and a first-order approximation, and its derivation assumes SGD-style updates despite empirical use with Adam.

    Source: §6 · p. 10

Abstract

from arXiv · show

Online reinforcement learning (RL) has achieved significant success in decision-making tasks, but modern online RL remains highly sensitive to the quality of training experience. Understanding the contribution of individual training instances is therefore central to data-centric RL, with applications ranging from experience selection to policy interpretation. Existing data attribution methods for online RL primarily focus on local influence within a single training round, overlooking the non-local data influence across multiple training rounds. In this work, we formalize data attribution in online RL through a trajectory-level leave-one-out quantity, RL-LOO, which measures how one training sample influences a downstream target after subsequent online updates have taken place. We then derive RL-IF, a first-order estimator that propagates data influence through both optimization effects and policy-induced sampling effects, and provide theoretical guarantees on its approximation accuracy under smoothness assumptions. Our analysis further disentangles these two effects and shows both theoretically and empirically that the full RL-IF estimator can often be well approximated using the optimization effect alone. Building on this observation, we develop RINSE, a practical data filtering method for online RL. Experiments on standard control benchmarks and an RLHF-style toxicity-mitigation setting show that RINSE improves training efficiency and final performance over standard PPO and local attribution baselines. These results suggest that RL-IF provides a practical and principled framework for understanding and improving online RL.

1 Introduction

Online RL depends on variable-quality experience, making individual training-instance attribution relevant to data selection and interpretation. This section introduces trajectory-level RL-LOO and RL-IF, then presents RINSE as a practical filtering method with reported gains across benchmarks and an RLHF-style setting.

1 Introduction

  • Training experience quality can vary across time and trajectories, motivating instance-level attribution for data selection, experience replay, and interpretation.The introduction connects this need to sample inefficiency, high variance, and training instability.Source: §1 · p. 1
  • Hu et al. [2025] introduced local PPO attribution that scores rollout records to identify harmful time steps and support data filtering.Their framework measures influence on a local policy update within a round.Source: §1 · p. 2
  • Existing online-RL attribution methods are described as heuristic and limited to one round, while inter-round influence can propagate through later optimization and policy-induced sampling.The paper frames this inter-round influence as a counterfactual data-value question.Source: §1 · p. 2
  • RL-LOO formulates non-local online-RL attribution as a leave-one-out objective defined over the training trajectory.It is intended to measure data influence beyond a single training round.Source: §1 · p. 2
  • RL-IF estimates RL-LOO influence through both optimization and policy-induced sampling effects, while RL-IF-opt uses the optimization effect alone as an efficient approximation.The paper reports theoretical and experimental support for the optimization-only approximation.Source: §1 · p. 2
  • RINSE improves training efficiency and final performance over standard PPO and local filtering baselines across standard benchmarks and an RLHF-style toxicity-mitigation setting.It is presented as a non-local attribution-based filtering method.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe PPO’s round-based rollout and update process, explain how local attribution scores training samples, and motivate non-local attribution by showing that short- and long-horizon influence can differ. They also distinguish optimization and sampling pathways through which a sample affects later learning.

2.1 Background

  • PPO proceeds in rounds: the current policy collects a rollout buffer, then the policy is updated for multiple gradient steps.The rollout-buffer distribution is induced by the policy used to collect that round’s data.Source: §2.1 · p. 2
  • A training sample can affect learning through parameter updates and through changes to the policy-induced distribution of future rollout buffers.The figure distinguishes the optimization path in blue from the sampling path in orange; the pathways are coupled through the data distribution.Source: §2.1 · p. 3, Figure 1 · p. 3
  • The background assumes SGD-style policy updates, with learning rate η_t, minibatch size b, and PPO loss ℓ.The parameter is updated through successive gradient steps within each round.Source: §2.1 · p. 3
  • Local online-RL attribution scores samples using gradient inner products with a downstream target, and positive scores indicate a push toward higher surrogate return.The cited local framework uses a policy-gradient surrogate defined on a reference validation distribution.Source: §2.1 · p. 3
  • Two LunarLander samples that appear harmful locally can be beneficial in the long run, and local attribution tends to overestimate the long-term value of late-episode samples.Figure 2 tracks the fraction of late-episode samples labeled harmful as attribution horizon H grows.Source: §2.1 · p. 3, Figure 2 · p. 4

3 Measuring Data Influence in Online RL

The section defines non-local online-RL attribution as the effect of removing a training occurrence on a downstream target after later updates. It derives a first-order estimator that propagates this influence through optimization and future sampling, then analyzes when the optimization-only approximation is sufficient.

3.1 Problem Formulation

  • RL-LOO measures how removing one minibatch occurrence changes a downstream target after the remaining training trajectory, including induced future sampling, is replayed.The attribution unit specifies both the sample identity and the minibatch where it appears.Source: §3.1 · p. 4

3.2 Approximating Leave-One-Out in RL

  • RL-IF approximates RL-LOO without retraining by propagating a first-order perturbation through later optimization and policy-dependent sampling updates.The recursion separates optimization and sampling effects; the sampling effect vanishes within the same training round, and the matrix products can be computed with Hessian-vector and vector products.Source: §3.2 · pp. 5–6

3.3 Sampling vs. Optimization Effects in RL-IF

  • Under smoothness assumptions and η0Λ ≪ 1, the accumulated gap between full RL-IF and RL-IF-opt grows only logarithmically with the horizon, supporting the optimization-only approximation.RL-IF-opt is obtained by dropping the sampling terms; the bound characterizes sampling as a controlled perturbation to optimization dynamics.Source: §3.3 · p. 6
  • RL-IF-opt and RL-IF-full perform nearly identically and both substantially outperform local TracIn attribution in LDS evaluation.LDS correlates aggregated attribution scores with observed subset-removal effects under retraining; the evaluation uses KL-regularized PPO over two rounds with future rollouts resampled.Source: Table 1 · p. 6, §3.3 · p. 7

4 Data Selection with RL-IF

RINSE uses RL-IF attribution over training windows to filter samples for online RL. Across six control tasks it improves sample efficiency and peak return, and in an RLHF-style toxicity-mitigation task it reduces toxicity with reported training speedups over standard training and IIF.

4 Data Selection with RL-IF

  • RINSE ranks samples by RL-IF-opt scores, removes the most negative-influence samples, and retrains on filtered datasets within a finite horizon window.It uses the final policy in the window as the reference; RL-IF-opt is chosen because it achieves comparable empirical performance at lower cost.Source: §4 · p. 7
  • The window strategy balances local relevance against non-local influence: short horizons are myopic, while excessively long horizons introduce severe off-policy mismatch.The method also uses an adaptive procedure that extends the horizon until the maximum window size or a decline in mean episode return.Source: §4 · pp. 7–8
  • RINSE achieves the best sample efficiency and peak return across all six tasks and uses 42.8% fewer rollouts on average than standard training to reach the same performance.Figure 3 presents test returns over training rounds and relative efficiency and peak-return comparisons across environments.Source: Figure 3 · p. 8, §4 · p. 9

4.1 Extension to RLHF

  • On the RLHF-style toxicity-mitigation task, RINSE has lower toxicity probability than baselines across most of training and averages over 3× speedup versus standard training and over 2× versus IIF.The task starts from GPT-Neo 2.7B and uses toxicity-based rewards with KL-regularized PPO.Source: §4.1 · p. 9

5 Related Work

Related work spans training-data attribution and valuation, data-centric RL, and online-RL attribution. This work distinguishes itself by tracking how a sample affects future optimization and policy-induced data collection across rounds.

5 Related Work

  • Prior training-data attribution and valuation work includes influence functions, trajectory gradient tracing, representer-point decompositions, and scalable estimators.Related work also studies sequential attribution, data valuation, cleaning, reweighting, and selection.Source: §5 · p. 9
  • Unlike fixed-distribution attribution, this work tracks sample influence through future optimization and data collection induced by the evolving policy.Its target is non-local influence over an extended online training trajectory, rather than only an immediate update.Source: §5 · pp. 9–10
  • Data-centric RL studies experience replay and prioritization, curriculum design, trajectory filtering, selective reuse, and replay-buffer management.These approaches seek to improve learning efficiency or robustness in RL training.Source: §5 · p. 10

6 Conclusion and Limitations

The paper frames online-RL attribution as trajectory-level counterfactual influence, estimates it with RL-IF, and reports that an optimization-only approximation supports RINSE. Its appendix describes the propagation recursion, a single-pass adjoint computation, and a closed-form derivation; the framework remains scoped to PPO-style on-policy training and first-order, SGD-style analysis.

6 Conclusion and Limitations

  • RL-LOO measures a training occurrence’s influence over the policy trajectory, and RL-IF propagates that effect through later optimization and future sampling.The conclusion reports that RL-IF-opt supports RINSE, which improves efficiency and final performance on standard benchmarks and an RLHF-style toxicity-mitigation setting.Source: §6 · p. 10
  • The framework is specialized to PPO-style on-policy training and first-order leave-one-out approximation, with an SGD-style derivation despite Adam-based experiments.The authors also note that sampling effects may not remain smaller than optimization effects in other regimes.Source: §6 · p. 10

A.1 Derivation of RL Leave-One-Out and RL-IF

  • The recursion injects a sample perturbation at its update, then propagates it through later updates and policy-induced rollout distributions.Fixed minibatches apply within the injection round; future stages use rollout distributions induced by the perturbed policy.Source: §A.1 · pp. 13–14

A.2 Efficient Computation through Adjoint Formulation

  • A backward adjoint pass computes trajectory-wide RL-IF scores without materializing each forward perturbation recursion.The procedure uses Hessian–vector products and score-function vector products; the passage states that all influence scores can be computed in one backward pass.Source: §A.2 · pp. 14–15

A.3 Closed-Form RL-IF: Proof of Equation (8)

  • The appendix proves a compact closed form for the RL-IF recursion by induction over minibatch updates and subsequent stages.The supplied proof passages describe the induction and its application at the final update.Source: §A.3 · pp. 15–16

A.4 Construction and Justification of the Reference Target

The reference target uses the terminal checkpoint and its on-policy rollout to evaluate the policy at the end of an attribution window. The authors report that this default construction achieves the highest final mean return in the tested reference ablation.

A.4 Construction and Justification of the Reference Target

  • The terminal checkpoint is the reference because RL-LOO evaluates the policy obtained at the end of the attribution window.The construction uses that checkpoint’s on-policy rollout, critic, and PPO-matched GAE procedure, with reference estimates held fixed during attribution.Source: §A.4 · p. 17
  • The reference target connects locally to terminal-return change through a first-order parameter-perturbation approximation.The proposition assumes the policy-gradient identity and twice-continuous differentiability near the reference checkpoint.Source: §A.4 · p. 17
  • The empirical target replaces the population target and exact advantages with a reference rollout and GAE estimates, adding sampling and advantage-estimation error.Sensitivity to reference construction is evaluated in the appendix.Source: §A.4 · p. 18

A.4.1 Choice of Reference Policy and Rollout

  • The default terminal-checkpoint reference achieves the highest final mean return on both Ant and HalfCheetah.The alternatives use the initial checkpoint with its rollout, or the terminal checkpoint with rollouts collected across the window.Source: §A.4.1 · p. 18, Figure 5 · p. 18

B Detailed Theorem Derivations

  • The appendix section introduces detailed theorem proofs, but the supplied passage does not state a further result.This section’s stated role is to provide detailed derivations.Source: B Detailed Theorem Derivations · p. 18

B.1 Uniform Approximation Error of First-Order RL-IF

The appendix states a uniform approximation-error theorem for first-order RL-IF under smoothness, bounded-gradient, and learning-rate assumptions. It also treats stage-wise and accumulated relative gaps between sampling-aware and optimization-only effects, then gives experimental setup details.

B.1 Uniform Approximation Error of First-Order RL-IF

  • Theorem B.1 bounds the uniform error of first-order RL-IF under differentiability, bounded gradients and Hessians, Lipschitz Hessians, bounded rollout scores, and a learning-rate condition.The proof analyzes population updates and excludes minibatch noise.Source: §B.1 · p. 18

B.2 Relative Bound between Stage-Wise Operators

  • Theorem B.2 gives a stage-wise relative-gap result between sampling-aware and optimization-only operators under smoothness, bounded-gradient, KL-smoothness, and learning-rate assumptions.The supplied theorem passage states the assumptions but not the bound’s explicit expression.Source: §B.2 · p. 21

B.3 Accumulated Relative Gap

  • Theorem B.3 extends the relative-gap analysis to accumulated effects and includes a harmonic-learning-rate refinement.The theorem is stated for T > k; the supplied passages do not expose its full bound.Source: §B.3 · p. 22

C Detailed Experimental Setup

  • The experimental setup reports AWS c7a CPUs and four NVIDIA A40 GPUs with 46 GB of memory each.The appendix section covers experimental setup, hyperparameter selection, and implementation details.Source: C Detailed Experimental Setup · p. 24

C.1 Experimental Setup for Traditional RL

The experiments use PPO-based setups for traditional RL benchmarks and an RLHF-style toxicity-mitigation task, with separate training and evaluation protocols.

C.1 Experimental Setup for Traditional RL

  • Traditional RL uses Stable-Baselines3 with KL-regularized PPO and selects KL coefficients by grid search for standard training performance.The search spans β from 0.0 to 5.0; the IIF discard ratio is selected for highest and most stable baseline performance.Source: §C.1 · p. 24
  • Traditional-RL evaluation averages policy performance over 32, 100, or 1,000 episodes depending on the environment, with three seeds and mean ± standard error.Continuous-control tasks generally use 32 episodes, BipedalWalker 100, and LunarLander and FrozenLake 1,000.Source: §C.1 · p. 25

C.2 Experimental Setup for RLHF

  • The RLHF setup fine-tunes GPT-Neo 2.7B with LoRA on toxic Real Toxicity Prompts examples and uses a hate-speech classifier logit as reward.The reward is the raw nothate-class logit, where a higher value indicates less toxic output.Source: §C.2 · p. 25
  • RLHF training uses fixed-KL PPO with β = 0.4, batches of 64, and 300 rollout steps; attribution uses H = 2 and ρ = 10%.The KL penalty uses the k1 approximation; the PPO update uses mini-batches of 8 over 4 epochs.Source: §C.2 · p. 25
  • RLHF evaluation uses 200 toxic Wiki-Toxic prompts and a separate hate-speech classifier, reporting mean toxicity probability across three seeds.The test prompts are longer than training prompts to elicit toxic continuations; results are reported as mean ± standard error.Source: §C.2 · pp. 25–26

D.1 Further Details on Linear Data Modeling Scores and Evaluation

The section evaluates attribution quality by relating scores to retrained outcomes after removing subsets of rollout data, using LDS as a linear-prediction test.

D.1 Further Details on Linear Data Modeling Scores and Evaluation

  • LDS tests whether summed attribution scores predict performance changes after removing subsets, assuming influence is approximately linearly additive across samples.It compares aggregated scores with actual retraining effects; individual-sample LOO correlation is difficult to validate because effects are small relative to run-to-run variance.Source: §D.1 · p. 26
  • The evaluation uses KL-regularized PPO over two consecutive rounds, 100 subsets of 4,096 samples, and 32 retraining runs per subset across three seeds.The benchmark environments are Ant-v4 and HalfCheetah-v4.Source: §D.1 · p. 26
  • RL-IF-opt and RL-IF-full substantially outperform TracIn and random baselines across Ant-v4 and HalfCheetah-v4 and the reported correlation metrics.The results indicate stronger capture of rollout data’s long-term effects on downstream policy behavior.Source: Table 4 · p. 26, §D.1 · p. 26

D.1.1 Direct Leave-One-Out Evaluation

These additional evaluations test direct LOO agreement, the relevance of sampling effects, and RINSE’s horizon and filtering choices.

D.1.1 Direct Leave-One-Out Evaluation

  • RL-IF-full has the highest mean correlation with direct LOO effects on FrozenLake, outperforming RL-IF-opt and TracIn when future rollouts can change with the perturbed policy.The evaluation uses H = 2 and 256 retraining runs.Source: §D.1.1 · pp. 26–27, Table 5 · p. 26

D.1.2 Empirical Characterization of the Sampling Effect

  • FrozenLake’s diagnostic ratio is about four orders of magnitude above Ant’s, while full and optimization-only RL-IF have nearly identical LDS on Ant and HalfCheetah.The measurements include loss curvature, rollout-score second moment, and learning rate; the comparison identifies FrozenLake as a setting where retaining the sampling correction is useful.Source: §D.1.2 · p. 27, Table 6 · p. 27

D.2 Additional Results for LOO

  • Under fixed rollouts, RL-IF remains highly correlated with ground-truth LOO as horizon grows, whereas TracIn’s high horizon-1 correlation does not capture long-horizon effects.Figure 6 compares both methods with ground-truth LOO across attribution horizons H = 1 to 5.Source: §D.2 · p. 27, Figure 6 · p. 27

D.3.1 Ablation on Adaptive Horizon

  • The adaptive horizon matches or exceeds fixed-horizon RINSE on all four environments, with the largest gap on harder-exploration tasks.Fixed-horizon runs progress slowly and erratically on Ant-v4 and BipedalWalker-v3; adaptive horizon reaches higher return with lower variance on LunarLander-v2.Source: §D.3.1 · p. 28, Figure 7 · p. 28

D.3.2 Effect of Horizon in RINSE

  • Across Ant-v4 and HalfCheetah-v4, selection quality improves as H grows but saturates around H = 5 and can slightly decline at the largest horizons.The sweep tests H ∈ {1, 2, 4, 5, 8, 10}; H = 5 is suggested as a robust default.Source: §D.3.2 · p. 28, Figure 8 · p. 29

D.3.3 Effect of the Discard Ratio

  • Filtering improves Ant-v4 performance across tested discard ratios, but HalfCheetah-v4 performs worse than no filtering at ρ = 50% and ρ = 100%.These results support a moderate discard ratio; the sweep includes ρ from 0% to 100%.Source: §D.3.3 · pp. 28–29, Figure 9 · p. 29

D.4 Further Details on Time and Sample Efficiency

The section details how time and sample efficiency are measured relative to PPO and compares those metrics across evaluation protocols, then reports per-round computational overhead.

D.4 Further Details on Time and Sample Efficiency

  • Time and sample efficiency are normalized to PPO at 100%, with lower values indicating better efficiency; the main protocol averages resource use to reach 90%, 95%, and 100% of PPO peak performance.A separate 100%-only threshold focuses on the final training stage.Source: §D.4 · p. 29
  • RINSE improves mean sample efficiency over IIF on all six environments and mean time efficiency on five, with similar overall conclusions under both evaluation protocols.Gains are especially pronounced for sample usage on Ant and HalfCheetah; the 90/95/100% protocol yields slightly more stable estimates than the 100%-only variant.Source: §D.4 · pp. 29–30, Table 7 · p. 30

D.4.1 Wall-Clock Runtime Breakdown

  • Attribution and replay add 62%–128% of baseline per-round cost, and whether faster learning offsets this overhead depends on the environment.This per-round overhead is distinct from time efficiency, which measures total time to reach target performance.Source: §D.4.1 · p. 30, Table 8 · p. 30

D.5 Additional Results for RLHF

This section presents additional RLHF toxicity-reduction results for RINSE and reports the paper’s declaration that LLMs are not an important, original, or non-standard component of its core methods.

D.5 Additional Results for RLHF

  • RINSE achieves more toxicity reduction and higher reward than Standard and IIF under the same number of training steps.Standard and IIF show similar learning dynamics, with IIF slightly faster than Standard.Source: Figure 10 · p. 31, §D.5 · p. 31
  • RINSE reaches the best test toxicity achieved by Standard on each seed substantially faster, showing that non-local data filtering accelerates convergence.With the Generalized Gauss–Newton matrix Martens [2020] approximation, RINSE is only ~1.5× slower per training step; this overhead is compensated by fewer required steps, improving overall wall-clock efficiency.Source: Figure 11 · p. 31, §D.5 · p. 31, Figure 12 · p. 31

D.5.1 Generation Quality at Matched Toxicity

  • At matched toxicity on held-out Wiki-Toxic prompts, RINSE has lower median perplexity, higher Distinct-2 and Distinct-3, and a lower repeated-4-gram rate than the unperturbed baseline.Fluency is measured with GPT-2 Large perplexity, diversity with Distinct-2 and Distinct-3, and repetition with Seq-rep-4.Source: §D.5.1 · pp. 31–32, Table 9 · p. 32

NeurIPS Paper Checklist

  • The paper marks LLM usage N/A because its core method development does not involve LLMs as an important, original, or non-standard component.The checklist defines N/A in those terms.Source: NeurIPS Paper Checklist · p. 39

Related papers

Loading 2610.28883v1…