PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation

Jiaze Li, Hao Yin, Haoran Xu, Boshen Xu, Wenhui Tan, Zewen He, Jianzhong Ju, Zhenbo Luo, Jian Luan

arXiv:2602.02994v3cs.CV

TL;DR

TVG post-training methods must preserve the benefits of on-policy optimization without GRPO’s sparse rewards and high rollout cost. Video-OPD uses on-policy trajectories and teacher-based reverse-KL supervision, with TVDF selecting reliable, informative samples. It consistently outperforms GRPO while converging faster and using less computation.

  • Problem

    GRPO-based TVG post-training is limited by sparse sequence-level rewards and substantial computational overhead from multiple rollouts.

    Source: §1 · pp. 1–2

  • Method

    Video-OPD distills dense token-level supervision from a fixed frontier teacher on current-policy trajectories, while TVDF prioritizes teacher-reliable trajectories with large disagreement.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    Video-OPD consistently outperforms GRPO across TVG and broader video benchmarks, with faster convergence and lower computational cost.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    On-policy distillation provides an effective alternative to conventional reinforcement learning for TVG.

    Source: §7 · p. 9

    Video-OPD assumes access to a high-capacity teacher that can provide conditional log-probabilities, raising practical questions about teacher availability and reliability.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement learning has emerged as a principled post-training paradigm for Temporal Video Grounding (TVG) due to its on-policy optimization, yet existing GRPO-based methods remain fundamentally constrained by sparse reward signals and substantial computational overhead. We propose Video-OPD, an efficient post-training framework for TVG inspired by recent advances in on-policy distillation. Video-OPD optimizes trajectories sampled directly from the current policy, thereby preserving alignment between training and inference distributions, while a frontier teacher supplies dense, token-level supervision via a reverse KL divergence objective. This formulation preserves the on-policy property critical for mitigating distributional shift, while converting sparse, episode-level feedback into fine-grained, step-wise learning signals. Building on Video-OPD, we introduce Teacher-Validated Disagreement Focusing (TVDF), a lightweight training curriculum that iteratively prioritizes trajectories that are both teacher-reliable and maximally informative for the student, thereby improving training efficiency. Empirical results demonstrate that Video-OPD consistently outperforms GRPO while achieving substantially faster convergence and lower computational cost, establishing on-policy distillation as an effective alternative to conventional reinforcement learning for TVG.

1. Introduction

TVG requires precise temporal localization, but existing post-training methods trade off on-policy alignment, dense supervision, and computational efficiency. Video-OPD addresses these tensions with on-policy distillation, while TVDF further targets reliable, informative trajectories.

  • TVG localizes video segments corresponding to natural-language queries, supporting precise temporal reasoning and downstream video applications.Source: §1 · p. 1
  • GRPO-based TVG methods suffer from sparse sequence-level rewards, ineffective credit assignment, and costly multiple rollouts.Source: §1 · pp. 1–2
  • Video-OPD preserves on-policy training while replacing sparse rewards with dense token-level supervision from a frontier teacher through reverse KL.Source: §1 · p. 2
  • TVDF validates teacher reliability with annotations and prioritizes trajectories exhibiting large teacher–student disagreement.Source: §1 · p. 2
  • Video-OPD improves TVG performance by over 17% on average, compared with roughly 12% for GRPO, while converging faster at lower computational cost.Source: §1 · p. 2

2. Motivation

SFT suffers from off-policy distribution mismatch and compounding errors, while GRPO restores on-policy optimization but remains limited by sparse rewards and multi-rollout overhead. These limitations motivate dense, step-wise supervision without abandoning on-policy training.

TVG formulation

  • TVG autoregressively predicts temporal actions conditioned on video, query, and prior actions to localize a temporal boundary.Source: §2.1 · p. 3

Failure of SFT

  • SFT trains on fixed expert trajectories rather than inference-induced states, creating distributional mismatch and compounding errors after early deviations.Source: §2.1 · p. 3

On-policy reinforcement learning

  • GRPO samples on-policy trajectories and aligns training with inference states, helping mitigate the compounding-error problem of off-policy methods.Source: §2.2 · p. 3

Sparse rewards

  • GRPO supplies only O(1) trajectory-level feedback per episode, impairing credit assignment across long autoregressive sequences.Source: §2.2 · p. 3

Multi-rollout overhead

  • Multiple rollouts reduce reward-estimation variance but impose prohibitive costs because each requires long-context autoregressive video generation.Source: §2.2 · p. 4

Design requirements

  • The target framework retains on-policy optimization while adding fine-grained signals and eliminating multiple rollouts to improve convergence and reduce training cost.Source: §2.3 · p. 4, Figure 2 · p. 4

3. Method

Video-OPD combines strictly on-policy trajectory optimization with dense teacher-derived token supervision, while TVDF selects reliable, informative trajectories to improve training efficiency.

Video-OPD

  • Video-OPD samples trajectories from the current student policy and uses a fixed teacher only to evaluate those on-policy tokens.This preserves alignment between training and inference distributions and avoids teacher-generated or ground-truth-conditioned trajectories.Source: §3.1 · pp. 4–5
  • Reverse-KL supervision converts sparse episode-level feedback into per-token learning signals for fine-grained temporal credit assignment.The teacher-derived signal penalizes tokens assigned low probability by the teacher, identifying intermediate reasoning steps that diverge from expected behavior.Source: §3.1 · p. 5
  • A single rollout per training sample reduces the grouped-rollout and reward-normalization overhead required by GRPO.Dense token-level supervision also reduces gradient variance, supporting more stable optimization.Source: §3.1 · p. 5

TVDF

  • TVDF uses ground-truth temporal annotations only to validate teacher reliability rather than directly supervise the student.Its Teacher Reliability Pre-Validation component assesses whether teacher-predicted temporal boundaries satisfy a consistency criterion.Source: §3.2 · pp. 5–6
  • After validation, TVDF prioritizes on-policy trajectories with large aggregated reverse-KL disagreement between the student and fixed teacher.These trajectories are treated as more informative because they combine a reliable teacher signal with greater mismatch from the student’s current behavior.Source: §3.2 · p. 6
  • TVDF is applied iteratively so its selected high-disagreement trajectories adapt as the student policy evolves.The curriculum targets samples that remain challenging for the student while improving sample efficiency and accelerating convergence.Source: §3.2 · p. 6

4. Experiment

The experiments evaluate Video-OPD across TVG and broader video-understanding benchmarks, with ablations examining TVDF, multi-round training, and teacher choice.

Experimental Setup

  • Experiments cover TVG and broader video-understanding benchmarks, including datasets evaluated with corrected temporal annotations.The setup includes Charades-STA, ActivityNet, QVHighlights, and broader video-understanding tasks.Source: §4 · p. 6, §4.1 · p. 7

Main Results

  • Video-OPD achieves state-of-the-art performance among open-source models across the evaluated TVG benchmarks.It approaches Gemini-2.5-Flash on most datasets and surpasses it on several benchmarks.Source: §4.2 · p. 7

Ablation Study

  • TVDF adds approximately 2% performance gain to Video-OPD on three TVG benchmarks.The ablation attributes positive contributions to both TRPV and DBTP.Source: §4.3 · pp. 7–8
  • Video-OPD gains consistently as TVDF training rounds increase and surpasses the teacher after three rounds.The curriculum repeatedly prioritizes trajectories challenging for the student and reliable for the teacher.Source: §4.3 · p. 8
  • Stronger teacher models produce larger Video-OPD gains, while final students generally match or surpass their corresponding teachers.This pattern is reported on QVHighlights-TimeLens under different teacher settings.Source: §4.3 · p. 8, Table 3 · p. 8

5. Extended Analysis

Extended analysis attributes Video-OPD’s advantages to dense supervision and eliminating GRPO’s multiple rollouts, improving convergence and reducing training cost.

Analysis Motivation

  • The comparison targets GRPO’s sparse-reward optimization and multi-rollout computational limitations.These are identified as the two limitations that the extended analysis evaluates.Source: §5 · p. 8

Dense Supervision

  • Video-OPD converges substantially faster than GRPO while achieving superior overall performance.The analysis links this behavior to precise, dense, fine-grained learning signals and improved credit assignment.Source: §5 · p. 8

Computational Overhead

  • Video-OPD requires approximately 20% of GRPO’s training cost under the same number of training steps.Avoiding multi-rollout reward estimation reduces computational overhead while maintaining stable optimization.Source: §5 · p. 8

6. Related Work

Related work spans TVG training and architectural methods, while on-policy distillation combines distributional alignment with dense token-level supervision.

MLLMs for TVG

  • Prior TVG work improves multimodal models through TVG-specific supervised objectives and verifiable reinforcement-learning rewards.A complementary direction develops token compression and timestamp encoding for long-video processing and temporal alignment.Source: §6 · pp. 8–9

On-Policy Distillation

  • On-policy distillation aligns a student with specialized teachers using reverse KL divergence over trajectories sampled from the student policy.This combines on-policy reinforcement learning’s distributional alignment with knowledge distillation’s token-level supervision.Source: §6 · p. 9
  • Video-OPD is presented as the first application of on-policy distillation to TVG.The stated target is long-horizon temporal reasoning requiring distributional alignment and fine-grained credit assignment.Source: §6 · p. 9

7. Conclusion

The paper introduces Video-OPD and TVDF for efficient TVG post-training, reporting stronger performance than GRPO with faster convergence and lower computational cost.

Conclusion

  • Video-OPD samples current-policy trajectories and replaces sparse trajectory-level rewards with dense token-level supervision.TVDF further prioritizes teacher-reliable, informative trajectories using annotated temporal data as validation.Source: §7 · p. 9
  • Experiments show that Video-OPD consistently outperforms GRPO while converging faster and costing less computationally.The conclusion states these results across the paper’s evaluated settings.Source: §7 · p. 9

Limitations

Video-OPD assumes a high-capacity teacher that provides conditional log-probabilities, creating practical concerns about teacher availability and reliability. The framework can also use domain-specific expert models, and the student may surpass its teacher after repeated training.

  • Video-OPD requires a high-capacity teacher capable of providing conditional log-probabilities for student-generated trajectories.Source: Limitations · p. 9
  • The teacher requirement introduces practical considerations concerning teacher availability and reliability.Source: Limitations · p. 9
  • Video-OPD can distill online from multiple domain-specific expert models rather than an extremely large general-purpose language model.Source: Limitations · p. 9
  • After multiple Video-OPD training rounds, the student consistently surpasses its teacher, supporting the framework’s practical feasibility and broad applicability.Source: Limitations · p. 9

Impact Statement

Video-OPD improves the efficiency, stability, and scalability of temporal video grounding post-training through on-policy distillation. By eliminating sparse rewards and costly rollouts, it reduces training costs and lowers barriers for resource-constrained practitioners.

  • Video-OPD improves the efficiency, stability, and scalability of post-training frameworks for temporal video grounding.Source: Impact Statement · p. 9
  • By eliminating sparse rewards and computationally expensive rollouts, Video-OPD significantly reduces multimodal language-model training costs.Source: Impact Statement · p. 9
  • Lower training costs reduce barriers for academic researchers and resource-constrained practitioners.Source: Impact Statement · p. 9

A. Proof Sketch: Dense Rewards and Optimization Advantages of Video-OPD over GRPO

The proof sketch explains why Video-OPD supplies denser, lower-variance supervision than GRPO while retaining on-policy optimization. These properties support more precise credit assignment, faster convergence, and improved final performance.

Reward Density

Variance Reduction and Credit Assignment

Relation to KL Minimization

Implications for Convergence

B. Additional Implementation Details of the Teacher-Validated Disagreement Focusing.

The implementation details define disagreement-based data selection for TVDF and describe off-policy reverse- and forward-KL distillation baselines. These baselines use fixed ground-truth trajectories, creating distributional mismatch with student on-policy states, while Video-OPD-8B matches a stronger baseline with fewer samples.

Teacher-Validated Disagreement Focusing

Difference-Based Sampling

  • The sampling objective selects k samples from matched data, with difference-based strategies operating on teacher–student IoU disagreement.Source: B. Additional Implementation Details… · p. 13
  • The adopted strategy directly selects samples with the largest teacher–student IoU discrepancies because they are treated as the most informative cases.Source: §B.2 · p. 13

Off-Policy Distillation Baselines

Off-Policy Reverse KL Distillation

  • Because OP-RKD evaluates fixed corpus trajectories, its supervision is aligned with the corpus rather than the student’s on-policy state distribution, making it vulnerable to distributional mismatch.Source: §C.1 · pp. 14–15
  • In OP-RKD, teacher-derived reverse-KL signals provide dense per-token supervision and identify intermediate reasoning steps that diverge from teacher-preferred behavior.Source: §C.1 · p. 14

Off-Policy Forward KL Distillation

  • Forward KL is mode-covering, encouraging student probability mass wherever the teacher assigns non-negligible probability, including low-probability modes.Source: §C.2 · p. 15

Experimental Comparison

  • Video-OPD-8B achieves comparable performance across all reported measures to GRPO-trained TimeLens-8B while using 2.5k rather than 12k training samples.Source: §D.2 · p. 16

D.3. Complete Ablation Results of Video-OPD under Different Teacher Models

Video-OPD remains effective across teacher models, training rounds, and curriculum choices. Students can surpass teachers after one to three OPD rounds, while Top-k sampling performs best and explicit thinking can reduce performance.

  • A single OPD round lets the 8B student consistently surpass 4B and 8B teachers across all benchmarks.Source: §D.3 · p. 16
  • 62.5 mIoU on QVHighlights-TimeLens exceeds the 4B teacher by 7.6 points, while 47.0 mIoU on ActivityNet-TimeLens exceeds the 8B teacher by 2.3 points.Source: §D.3 · p. 16
  • Across three benchmarks, performance improves monotonically over multiple OPD rounds, with students surpassing teachers after three rounds.Source: §D.4 · p. 17
  • Top-k sampling achieves the best performance across all three TVG benchmarks and evaluation metrics, outperforming DSUS, BBDS, and GWDS.Source: §D.5 · p. 17, Table 7 · p. 17
  • Adding a thinking process lowers Charades-TimeLens mIoU from 52.0 to 48.7 during Video-OPD training.Source: §D.6 · p. 18, Table 8 · p. 18

Related papers

Loading 2602.02994v3…