Source-linked AI summary
Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation
Jiaze Li, Hao Yin, Haoran Xu, Boshen Xu, Wenhui Tan, Zewen He, Jianzhong Ju, Zhenbo Luo, Jian Luan
TL;DR
TVG post-training methods must preserve the benefits of on-policy optimization without GRPO’s sparse rewards and high rollout cost. Video-OPD uses on-policy trajectories and teacher-based reverse-KL supervision, with TVDF selecting reliable, informative samples. It consistently outperforms GRPO while converging faster and using less computation.
Problem
GRPO-based TVG post-training is limited by sparse sequence-level rewards and substantial computational overhead from multiple rollouts.
Method
Video-OPD distills dense token-level supervision from a fixed frontier teacher on current-policy trajectories, while TVDF prioritizes teacher-reliable trajectories with large disagreement.
Results
Video-OPD consistently outperforms GRPO across TVG and broader video benchmarks, with faster convergence and lower computational cost.
Takeaways & Limitations
On-policy distillation provides an effective alternative to conventional reinforcement learning for TVG.
Takeaways & Limitations
Video-OPD assumes access to a high-capacity teacher that can provide conditional log-probabilities, raising practical questions about teacher availability and reliability.
Abstract
from arXiv · showhide
Reinforcement learning has emerged as a principled post-training paradigm for Temporal Video Grounding (TVG) due to its on-policy optimization, yet existing GRPO-based methods remain fundamentally constrained by sparse reward signals and substantial computational overhead. We propose Video-OPD, an efficient post-training framework for TVG inspired by recent advances in on-policy distillation. Video-OPD optimizes trajectories sampled directly from the current policy, thereby preserving alignment between training and inference distributions, while a frontier teacher supplies dense, token-level supervision via a reverse KL divergence objective. This formulation preserves the on-policy property critical for mitigating distributional shift, while converting sparse, episode-level feedback into fine-grained, step-wise learning signals. Building on Video-OPD, we introduce Teacher-Validated Disagreement Focusing (TVDF), a lightweight training curriculum that iteratively prioritizes trajectories that are both teacher-reliable and maximally informative for the student, thereby improving training efficiency. Empirical results demonstrate that Video-OPD consistently outperforms GRPO while achieving substantially faster convergence and lower computational cost, establishing on-policy distillation as an effective alternative to conventional reinforcement learning for TVG.
1. Introduction
TVG requires precise temporal localization, but existing post-training methods trade off on-policy alignment, dense supervision, and computational efficiency. Video-OPD addresses these tensions with on-policy distillation, while TVDF further targets reliable, informative trajectories.
- TVG localizes video segments corresponding to natural-language queries, supporting precise temporal reasoning and downstream video applications.
- GRPO-based TVG methods suffer from sparse sequence-level rewards, ineffective credit assignment, and costly multiple rollouts.
- Video-OPD preserves on-policy training while replacing sparse rewards with dense token-level supervision from a frontier teacher through reverse KL.
- TVDF validates teacher reliability with annotations and prioritizes trajectories exhibiting large teacher–student disagreement.
- Video-OPD improves TVG performance by over 17% on average, compared with roughly 12% for GRPO, while converging faster at lower computational cost.
2. Motivation
SFT suffers from off-policy distribution mismatch and compounding errors, while GRPO restores on-policy optimization but remains limited by sparse rewards and multi-rollout overhead. These limitations motivate dense, step-wise supervision without abandoning on-policy training.
- TVG formulation: TVG autoregressively predicts temporal actions conditioned on video, query, and prior actions to localize a temporal boundary.
- Failure of SFT: SFT trains on fixed expert trajectories rather than inference-induced states, creating distributional mismatch and compounding errors after early deviations.
- On-policy reinforcement learning: GRPO samples on-policy trajectories and aligns training with inference states, helping mitigate the compounding-error problem of off-policy methods.
- Sparse rewards: GRPO supplies only O(1) trajectory-level feedback per episode, impairing credit assignment across long autoregressive sequences.
- Multi-rollout overhead: Multiple rollouts reduce reward-estimation variance but impose prohibitive costs because each requires long-context autoregressive video generation.
- Design requirements: The target framework retains on-policy optimization while adding fine-grained signals and eliminating multiple rollouts to improve convergence and reduce training cost.
3. Method
Video-OPD combines strictly on-policy trajectory optimization with dense teacher-derived token supervision, while TVDF selects reliable, informative trajectories to improve training efficiency.
- Video-OPD: Video-OPD samples trajectories from the current student policy and uses a fixed teacher only to evaluate those on-policy tokens.This preserves alignment between training and inference distributions and avoids teacher-generated or ground-truth-conditioned trajectories.
- Video-OPD: Reverse-KL supervision converts sparse episode-level feedback into per-token learning signals for fine-grained temporal credit assignment.The teacher-derived signal penalizes tokens assigned low probability by the teacher, identifying intermediate reasoning steps that diverge from expected behavior.
- Video-OPD: A single rollout per training sample reduces the grouped-rollout and reward-normalization overhead required by GRPO.Dense token-level supervision also reduces gradient variance, supporting more stable optimization.
- TVDF: TVDF uses ground-truth temporal annotations only to validate teacher reliability rather than directly supervise the student.Its Teacher Reliability Pre-Validation component assesses whether teacher-predicted temporal boundaries satisfy a consistency criterion.
- TVDF: After validation, TVDF prioritizes on-policy trajectories with large aggregated reverse-KL disagreement between the student and fixed teacher.These trajectories are treated as more informative because they combine a reliable teacher signal with greater mismatch from the student’s current behavior.
- TVDF: TVDF is applied iteratively so its selected high-disagreement trajectories adapt as the student policy evolves.The curriculum targets samples that remain challenging for the student while improving sample efficiency and accelerating convergence.
4. Experiment
The experiments evaluate Video-OPD across TVG and broader video-understanding benchmarks, with ablations examining TVDF, multi-round training, and teacher choice.
- Experimental Setup: Experiments cover TVG and broader video-understanding benchmarks, including datasets evaluated with corrected temporal annotations.The setup includes Charades-STA, ActivityNet, QVHighlights, and broader video-understanding tasks.
- Main Results: Video-OPD achieves state-of-the-art performance among open-source models across the evaluated TVG benchmarks.It approaches Gemini-2.5-Flash on most datasets and surpasses it on several benchmarks.
- Ablation Study: TVDF adds approximately 2% performance gain to Video-OPD on three TVG benchmarks.The ablation attributes positive contributions to both TRPV and DBTP.
- Ablation Study: Video-OPD gains consistently as TVDF training rounds increase and surpasses the teacher after three rounds.The curriculum repeatedly prioritizes trajectories challenging for the student and reliable for the teacher.
- Ablation Study: Stronger teacher models produce larger Video-OPD gains, while final students generally match or surpass their corresponding teachers.This pattern is reported on QVHighlights-TimeLens under different teacher settings.
5. Extended Analysis
Extended analysis attributes Video-OPD’s advantages to dense supervision and eliminating GRPO’s multiple rollouts, improving convergence and reducing training cost.
- Analysis Motivation: The comparison targets GRPO’s sparse-reward optimization and multi-rollout computational limitations.These are identified as the two limitations that the extended analysis evaluates.
- Dense Supervision: Video-OPD converges substantially faster than GRPO while achieving superior overall performance.The analysis links this behavior to precise, dense, fine-grained learning signals and improved credit assignment.
- Computational Overhead: Video-OPD requires approximately 20% of GRPO’s training cost under the same number of training steps.Avoiding multi-rollout reward estimation reduces computational overhead while maintaining stable optimization.
6. Related Work
Related work spans TVG training and architectural methods, while on-policy distillation combines distributional alignment with dense token-level supervision.
- MLLMs for TVG: Prior TVG work improves multimodal models through TVG-specific supervised objectives and verifiable reinforcement-learning rewards.A complementary direction develops token compression and timestamp encoding for long-video processing and temporal alignment.
- On-Policy Distillation: On-policy distillation aligns a student with specialized teachers using reverse KL divergence over trajectories sampled from the student policy.This combines on-policy reinforcement learning’s distributional alignment with knowledge distillation’s token-level supervision.
- On-Policy Distillation: Video-OPD is presented as the first application of on-policy distillation to TVG.The stated target is long-horizon temporal reasoning requiring distributional alignment and fine-grained credit assignment.
7. Conclusion
The paper introduces Video-OPD and TVDF for efficient TVG post-training, reporting stronger performance than GRPO with faster convergence and lower computational cost.
- Conclusion: Video-OPD samples current-policy trajectories and replaces sparse trajectory-level rewards with dense token-level supervision.TVDF further prioritizes teacher-reliable, informative trajectories using annotated temporal data as validation.
- Conclusion: Experiments show that Video-OPD consistently outperforms GRPO while converging faster and costing less computationally.The conclusion states these results across the paper’s evaluated settings.
Limitations
Video-OPD assumes a high-capacity teacher that provides conditional log-probabilities, creating practical concerns about teacher availability and reliability. The framework can also use domain-specific expert models, and the student may surpass its teacher after repeated training.
- Video-OPD requires a high-capacity teacher capable of providing conditional log-probabilities for student-generated trajectories.
- The teacher requirement introduces practical considerations concerning teacher availability and reliability.
- Video-OPD can distill online from multiple domain-specific expert models rather than an extremely large general-purpose language model.
- After multiple Video-OPD training rounds, the student consistently surpasses its teacher, supporting the framework’s practical feasibility and broad applicability.
Impact Statement
Video-OPD improves the efficiency, stability, and scalability of temporal video grounding post-training through on-policy distillation. By eliminating sparse rewards and costly rollouts, it reduces training costs and lowers barriers for resource-constrained practitioners.
- Video-OPD improves the efficiency, stability, and scalability of post-training frameworks for temporal video grounding.
- By eliminating sparse rewards and computationally expensive rollouts, Video-OPD significantly reduces multimodal language-model training costs.
- Lower training costs reduce barriers for academic researchers and resource-constrained practitioners.
A. Proof Sketch: Dense Rewards and Optimization Advantages of Video-OPD over GRPO
The proof sketch explains why Video-OPD supplies denser, lower-variance supervision than GRPO while retaining on-policy optimization. These properties support more precise credit assignment, faster convergence, and improved final performance.
- Reward Density: GRPO assigns one sequence-level reward to every trajectory, providing no explicit guidance for individual intermediate decisions.
- Reward Density: Video-OPD provides T distinct, step-specific rewards per trajectory, strictly increasing reward density along the temporal dimension.
- Variance Reduction and Credit Assignment: GRPO’s shared trajectory reward induces strong cross-time correlations, causing gradient variance to grow rapidly with trajectory length T.
- Variance Reduction and Credit Assignment: Video-OPD uses local state–action rewards whose reduced cross-time covariance yields a lower-variance gradient estimator.
- Relation to KL Minimization: Video-OPD’s expected update is equivalent to minimizing expected reverse KL divergence between student and teacher policies.
- Video-OPD performs on-policy stochastic gradient descent on a smooth objective at each visited state, supporting stable optimization.
- Implications for Convergence: Lower gradient variance supports faster convergence under the same sampling budget, while dense rewards enable accurate credit assignment and improved final performance.
B. Additional Implementation Details of the Teacher-Validated Disagreement Focusing.
The implementation details define disagreement-based data selection for TVDF and describe off-policy reverse- and forward-KL distillation baselines. These baselines use fixed ground-truth trajectories, creating distributional mismatch with student on-policy states, while Video-OPD-8B matches a stronger baseline with fewer samples.
- Teacher-Validated Disagreement Focusing: TVDF measures teacher reliability and student performance using mean IoU against annotated temporal intervals, then uses their discrepancy for data selection.
- Difference-Based Sampling: The sampling objective selects k samples from matched data, with difference-based strategies operating on teacher–student IoU disagreement.
- Difference-Based Sampling: The adopted strategy directly selects samples with the largest teacher–student IoU discrepancies because they are treated as the most informative cases.
- Off-Policy Distillation Baselines: Off-policy reverse- and forward-KL distillation use fixed ground-truth trajectories instead of student-generated on-policy rollouts.
- Off-Policy Reverse KL Distillation: Because OP-RKD evaluates fixed corpus trajectories, its supervision is aligned with the corpus rather than the student’s on-policy state distribution, making it vulnerable to distributional mismatch.
- Off-Policy Reverse KL Distillation: In OP-RKD, teacher-derived reverse-KL signals provide dense per-token supervision and identify intermediate reasoning steps that diverge from teacher-preferred behavior.
- Off-Policy Forward KL Distillation: Forward KL is mode-covering, encouraging student probability mass wherever the teacher assigns non-negligible probability, including low-probability modes.
- Experimental Comparison: Video-OPD-8B achieves comparable performance across all reported measures to GRPO-trained TimeLens-8B while using 2.5k rather than 12k training samples.
D.3. Complete Ablation Results of Video-OPD under Different Teacher Models
Video-OPD remains effective across teacher models, training rounds, and curriculum choices. Students can surpass teachers after one to three OPD rounds, while Top-k sampling performs best and explicit thinking can reduce performance.
- A single OPD round lets the 8B student consistently surpass 4B and 8B teachers across all benchmarks.
- 62.5 mIoU on QVHighlights-TimeLens exceeds the 4B teacher by 7.6 points, while 47.0 mIoU on ActivityNet-TimeLens exceeds the 8B teacher by 2.3 points.
- Across three benchmarks, performance improves monotonically over multiple OPD rounds, with students surpassing teachers after three rounds.
- Top-k sampling achieves the best performance across all three TVG benchmarks and evaluation metrics, outperforming DSUS, BBDS, and GWDS.
- Adding a thinking process lowers Charades-TimeLens mIoU from 52.0 to 48.7 during Video-OPD training.