Source-linked AI summary
Dynamic Important Example Mining for Reinforcement Finetuning
Haoru Tan, Sitong Wu, Yanfeng Chen, Shizhen Zhao, Yang-Tian Sun, Tianjia Liu, Chirui Chang, Shaofeng Zhang, Samm Sun, Xiuzhe Wu, Ruobing Xie, Xiaojuan Qi
TL;DR
RFT data-centric methods often treat sample value as fixed, despite non-stationary policy learning and the need to quantify examples’ policy-update contributions. DIEM estimates dynamic importance from gradient alignment and reweights batches under a gradient-magnitude constraint. Across multimodal reasoning benchmarks, it consistently outperforms static and dynamic baselines while adding only 1.2% training overhead.
Problem
Existing RFT data-selection methods rely on static or heuristic importance measures that overlook non-stationary dynamics and do not quantify samples’ marginal policy-update impact.
Method
DIEM uses a gradient-based alignment estimator for sample marginal contribution and constrained batch reweighting that preserves overall gradient magnitude.
Results
Across large-scale multimodal reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines, with approximately 1%–6% performance uplift and 1.2% additional training overhead.
Takeaways & Limitations
DIEM makes data utilization an adaptive, model-driven curriculum component of RFT and is broadly compatible with diverse RFT algorithms.
Abstract
from arXiv · showhide
Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.
1 Introduction
DIEM addresses the mismatch between dynamic RFT learning and mostly static or heuristic data selection by measuring and reweighting examples during optimization. It reports consistent benchmark gains with minimal overhead.
- Static and heuristic selectors treat sample importance as fixed despite RFT’s non-stationary learning dynamics.
- The framework adds Dynamic Data Importance Measuring and Dynamic Data Reweighting as two lightweight steps to the vanilla RFT pipeline.
- A gradient-alignment estimator quantifies sample importance, while constrained reweighting preserves overall gradient magnitude and emphasizes important samples.
- DIEM measures each sample’s marginal contribution to current policy improvement and adaptively reweights samples within the optimization loop.
- DIEM consistently achieves the best performance with a minimal additional 1.2% training overhead and induces a self-organizing curriculum.
2 Related Works
Prior RFT data-selection methods use static reward-based screening or dynamic heuristic curricula. Their shared limitation is that these proxies do not directly measure policy-specific sample impact.
- Static Selection: Static selection curates subsets before RFT using reward trends or variability, assuming sample importance remains constant during training.
- Dynamic Selection: Dynamic selection methods reorder samples during training, often prioritizing intermediate difficulty through auxiliary models or pass-rate heuristics.
- Heuristic metrics are externally detached from the policy and cannot quantify each sample’s true marginal impact on policy updates.
3 Preliminaries
RFT treats a language model as a stochastic policy that generates actions for prompts and receives scalar rewards. Policy optimization updates parameters using advantage-weighted gradients, with refined methods improving stability.
- RFT Formulation: The training corpus contains input-output pairs, with prompts as states, generated text as actions, and rewards evaluating response quality.
- Vanilla Policy Optimization: Vanilla policy optimization maximizes expected empirical advantage through iterative stochastic gradient ascent.
- Vanilla Policy Optimization: The advantage A(s,a)=r(s,a)-v measures an action’s merit relative to a baseline value.
- Beyond the Vanilla Policy Gradient: Contemporary RFT methods such as PPO, QPO, GPG, and GRPO address instability in simple gradient ascent with refined optimization schemes.
- Beyond the Vanilla Policy Gradient: GRPO standardizes rewards within groups, clips extreme relative advantages, and uses KL regularization to limit divergence from a reference model.
4 Dynamic Important Example Mining
DIEM measures each sample’s step-specific contribution to policy improvement and dynamically reweights samples within every RFT optimization step. Its estimator uses gradient alignment, while constrained reweighting prioritizes utility without changing the aggregate update magnitude.
- Overview: DIEM assesses sample importance at each RFT step and adjusts each sample’s weight accordingly.This makes data utilization adaptive throughout training rather than fixed before optimization.
- Dynamic Data Importance Measuring: The true importance score measures the change in total batch reward caused by removing a sample from the update calculation.Positive, negative, and near-zero scores indicate beneficial, detrimental, and negligible marginal influence, respectively.
- Efficient Estimation: The gradient-alignment estimator approximates each sample’s marginal contribution by comparing its individual policy gradient with the aggregate batch gradient.The required gradient terms are already computed during standard RFT backpropagation, making the estimator computationally lightweight.
- Algorithm: DIEM computes individual sample gradients, aggregates them into a gradient matrix, and derives sample-wise influence scores for dynamic reweighting.The algorithm then forms a Gram matrix, solves for weights, enforces non-negativity, and computes the reweighted total gradient.
- Interpretation: Positive alignment marks samples whose optimization direction agrees with the batch direction, whereas negative divergence can slow convergence or introduce noise.These directional scores provide an interpretable basis for prioritizing or downweighting samples at a specific step.
- Theoretical Analysis: Under Lipschitz log-likelihood and bounded-advantage assumptions, the estimator’s approximation error is bounded and depends on learning rate and reward smoothness.The estimator does not require convexity or near-stationarity, supporting its use in non-convex, non-stationary early RFT.
- Dynamic Data Reweighting: The reweighting objective maximizes weighted sample utility while constraining the reweighted aggregate gradient to retain the original L2 norm.A quasi-closed-form solution uses Lagrange multipliers and requires one inversion of a minibatch-sized Gram matrix.
5 Experiments
Experiments compare DIEM with RFT and data-selection baselines across language and multimodal reasoning settings. DIEM generally improves performance, with low training overhead and dynamic weights that evolve into a curriculum-like pattern.
- LLM Results: DIEM consistently outperforms GRPO on average across four model families, with gains ranging from +1.68 to +3.36.Reported averages improve by +1.74 on Qwen-3-1.7B, +2.50 on Qwen2.5-3B, +3.36 on Qwen3-4B, and +1.68 on Qwen2.5-7B.
- LLM Results: DIEM improves high-difficulty AIME25 results over GRPO by +2.1 on Qwen-3-1.7B and +5.3 on Qwen2.5-7B.These correspond to relative gains of 62% and 96%, respectively.
- VLM Results: DIEM achieves 61.8% average performance across six 7B multimodal benchmarks, exceeding Vanilla RFT by 3.6 percentage points and SPEED-RL by 1.8 points.It obtains the highest score in five of six benchmarks, including MathVista, MathVerse, MathVision, MMStar, and AI2D.
- VLM Results: DIEM reaches a 67.3% average on Qwen2.5-VL-32B, improving over Vanilla RFT by 2.4 percentage points and SPEED-RL by 1.7 points.For the 32B model, DIEM achieves the highest score across all six individual benchmarks.
- Ablation Study: Replacing DIEM’s dynamic influence score or specialized reweighting function causes substantial performance drops in the MathVerse ablation.Random replacement lowers performance to 53.0, raw Pass@k to 53.2, raw PCL Difficulty to 52.1, removing reweighting to 55.4, and Softmax normalization to 56.4, versus 58.0 for the full model.
- Speed Test: DIEM adds only 0.9 hours, or about 1.28%, to Vanilla RFT training time, requiring 71.2 hours versus the baseline’s 70.3 hours.This overhead is lower than the reported times for PCL, SPEED-RL, LIMR, and HVS.
- Training Process Visualization: DIEM’s normalized weights form a smooth curriculum-like trajectory: Easy and Medium samples receive relatively high initial importance, while Easy-sample weight later declines rapidly.Difficulty is proxied by Pass@k, with lower Pass@k indicating greater difficulty.
6 Conclusion
DIEM enhances RFT by replacing fixed-value assumptions with dynamic, gradient-based sample importance and adaptive batch reweighting. Its constrained optimization seeks greater collective sample contribution while maintaining stable policy updates, and experiments show superior performance and efficiency.
- DIEM dynamically quantifies each sample’s marginal contribution to policy improvement using a theoretically grounded gradient-based importance estimator.
- Its constrained optimization procedure adaptively reweights batches to maximize collective sample contribution while ensuring stable policy updates.
- Extensive experiments demonstrate DIEM’s superior performance and high efficiency.