PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Attention Policy Distillation for Temporal Video Grounding

Shaobo Ju, Haiyang Yu, Xuecheng Wu, Qiong Wu, Jiacong Wang, Fan Shi, Jun Peng, Yiyi Zhou

arXiv:2609.34581v2cs.CV

TL;DR

Precise temporal boundaries remain difficult when target events appear amid repeated actions and similar contexts, while next-token distillation does not identify which video regions support each timestamp. CAPD uses temporal-group masking to reweight distillation, improving average recall by 12.0% over GRPO on TimeLens while preserving general video understanding.

  • Problem

    Accurate event boundaries remain difficult amid repeated actions and similar contexts, and OPD does not guide which temporal regions support each timestamp.

    Source: Introduction · p. 1

  • Method

    CAPD masks contiguous temporal groups to measure their effects on teacher predictions, then uses the resulting influence to calibrate attention and reweight distillation.

    Source: Introduction · p. 2

  • Results

    12.0% higher average recall than GRPO across three TimeLens-corrected benchmarks, while preserving general video understanding and remaining comparable to the base model on several general video benchmarks.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    CAPD provides more effective temporal-grounding supervision by integrating temporal evidence into on-policy distillation.

    Source: Conclusion · p. 7

    CAPD uses fixed temporal groups, which may not fully capture events with diverse durations and temporal structures.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Temporal video grounding is a key capability of advanced Multimodal Large Language Models (MLLMs) for the thorough understanding of video events, which is however often limited by repeated actions and visually similar contexts in long videos. In this paper, we study this issue from the perspective of On-policy distillation (OPD) and propose a new training regime for MLLMs termed Counterfactual Attention Policy Distillation (CAPD). In particular, OPD is a viable solution for MLLMs via providing dense teacher supervision on student-generated trajectories. But its next-token based teacher-student distillation is hard to identify the specific video segments supporting each predicted timestamp, which is critical for temporal grounding. In this case, CAPD measures how masking each temporal group changes the teacher's output distribution. The resulting counterfactual influence calibrates the teacher's attention and weights token-level distillation, allowing the student to learn the temporal evidence that affects boundary prediction. To validate CAPD, we trained it on Qwen3-VL-8B-Instruct using only 2,500 samples for one epoch, and evaluated it on the TimeLens and multiple general video benchmarks. Experimental results show that CAPD improves average recall by 12.0% relative to GRPO on TimeLens while preserving general video understanding, achieving comparable accuracy to the base model.

Introduction

CAPD measures which temporal groups change teacher predictions, using their influence to calibrate attention and output distillation. The introduction reports higher TimeLens average recall while retaining comparable general video performance.

  • Temporal video grounding localizes a query-described interval, but precise boundaries remain difficult when short events occur amid repeated actions, similar contexts, or irrelevant salient content.MLLM grounding generates start and end timestamps directly from video and query, while on-policy distillation provides dense teacher supervision on student-generated trajectories.Source: Introduction · p. 1
  • CAPD masks contiguous temporal groups under the same student-generated context, measures changes in teacher output distributions, and uses those changes to calibrate attention and weight output-level distillation.Groups causing larger output changes receive higher target attention; the student matches both corrected attention and teacher output distributions on OPD rollouts.Source: Introduction · p. 2
  • With 2,500 examples for one epoch, CAPD improves average recall by 12.0% over GRPO across three TimeLens-corrected benchmarks and remains comparable to the base model on general video benchmarks.Under the same training settings, CAPD also consistently outperforms OPD and RAL on TimeLens.Source: Introduction · p. 2
  • OPD transfers teacher next-token distributions but not supporting temporal regions, while RAL transfers attended regions that may not affect predictions (Li et al. 2026b; Li et al. 2026a).The introduction frames CAPD as addressing this gap by identifying decision-relevant evidence rather than relying on raw attention.Source: Introduction · pp. 1–2

Related Work

Prior work advances temporal grounding through video MLLMs, temporal reasoning, and on-policy distillation, while CAPD uses temporal-group counterfactual influence to calibrate teacher attention for distillation.

Temporal Video Grounding

  • CAPD measures temporal-group influence on teacher outputs to calibrate attention and distill decision-relevant evidence, unlike prior grounding and attention-policy approaches.The framework masks each temporal group, estimates counterfactual influence from teacher-output changes, and combines output-level OPD with calibrated attention-policy learning.Source: Temporal Video Grounding · p. 2, On-Policy Distillation · p. 3, Figure 2 · p. 3
  • Grounding methods evolved from segment ranking and boundary prediction to video MLLMs, temporal reasoning methods, and improved annotation and post-training.Examples include TimeChat, VTimeLLM, Time-R1, VTimeCoT, TAR, and TimeLens.Source: Temporal Video Grounding · p. 2
  • Earlier counterfactual grounding methods alter video-query samples or construct counterfactual sequences for moment-bias mitigation and weakly supervised contrastive learning.These approaches include Zhai et al. (2022) and Xu, Xu, and Miao (2025); Wang, Chen, and Shen (2025) is also cited in this context.Source: Temporal Video Grounding · p. 2

On-Policy Distillation

  • OPD supervises student-generated trajectories; prior methods vary in divergence objectives, data reuse, target adjustment, prefix selection, and token-importance estimation.Examples include MiniLLM, GKD, DistiLLM, Veto, Prefix OPD, and TIP; Video-OPD applies OPD to temporal video grounding, while RAL distills attention distributions as latent policies.Source: On-Policy Distillation · pp. 2–3

Method

CAPD extends on-policy distillation with counterfactual estimates of which temporal evidence affects teacher predictions, using those estimates to calibrate attention supervision and weight output-token distillation.

Overview

  • Unlike OPD and RAL (Li et al. 2026a), CAPD uses counterfactual interventions to identify temporal evidence that affects predictions.OPD transfers next-token distributions, while attention alone can reveal where the teacher looks without showing whether a region changes its prediction.Source: Overview · p. 3

On-Policy Distillation

  • The student samples an on-policy response, and the frozen teacher evaluates shared student-generated prefixes to provide next-token supervision without identifying supporting video regions.The student-generated timestamps are part of the response being distilled.Source: Overview · p. 3, On-Policy Distillation · pp. 3–4

Temporal-Group Attention Policy

  • CAPD aggregates final-layer attention over frame-based temporal groups and separately tracks prompt and prior-response tokens, avoiding visual-only renormalization.Unlike the RAL-AD baseline, direct attention matching does not establish whether an attended visual group changes the teacher’s prediction.Source: Temporal-Group Attention Policy · p. 4

Counterfactual Group Importance

  • For each temporal group, CAPD replaces its visual features with same-location means from other temporal positions and measures the resulting change in the teacher’s next-token distribution.The counterfactual preserves input shape, spatial layout, positional indices, query, and student prefix; this estimates prediction influence rather than attention mass.Source: Counterfactual Group Importance · p. 4

Counterfactual Attention Calibration

  • CAPD uses standardized counterfactual group importance to increase calibrated teacher-attention mass for above-average influential groups and decrease it for below-average groups.When total visual dependence is low, the gate keeps the calibrated policy close to the original attention policy.Source: Counterfactual Attention Calibration · p. 4

Counterfactual Token Weighting

  • CAPD combines calibrated attention matching with OPD weighted toward outputs whose predictions depend more strongly on video, normalizing token weights to an average of one.The counterfactual signal is used to emphasize video-dependent output tokens.Source: Overview · p. 3, Counterfactual Token Weighting · p. 4

Experiments

CAPD achieves the best average temporal-grounding performance among the compared post-training methods, while maintaining comparable general video understanding. Component, parameter, and qualitative analyses examine how counterfactual calibration supports temporal evidence selection and boundary prediction.

Implementation Details

  • Experiments initialize the student from Qwen3-VL-8B-Instruct and train for one epoch on 2,500 Video-OPD examples, with Qwen3-VL-32B as the frozen teacher.CAPD uses G = 8, last-layer student and teacher policies, λatt = 0.25, and maximum token weight wmax = 5; Vanilla OPD disables attention distillation and counterfactual token weighting.Source: Implementation Details · p. 5

Benchmarks and Metrics

  • Temporal grounding is evaluated on three TimeLens-corrected benchmarks using Recall at IoU thresholds 0.3, 0.5, and 0.7; general understanding is assessed on Video-MME, LongVideoBench, and LVBench.The grounding sets contain 3,363, 4,500, and 1,541 queries, respectively; the general benchmarks include videos ranging from short clips to an average length of 68 minutes on LVBench.Source: Benchmarks and Metrics · p. 5

Performance Comparison with Other Methods

  • On three TimeLens-corrected benchmarks, CAPD reaches an average of 55.1, improving over GRPO by 5.9 points (12.0% relative) and achieving the best average among compared post-training methods.It improves most recall metrics, especially under the stricter IoU threshold; gains are not concentrated in a single dataset.Source: Performance Comparison with Other… · p. 5
  • CAPD maintains comparable performance to Qwen3-VL-8B and baselines on Video-MME, LongVideoBench, and LVBench, preserving general video understanding.The comparison is reported in Table 2; the passage attributes this preservation to a training signal targeted at temporal grounding.Source: Performance Comparison with Other… · p. 6, Table 2 · p. 6
  • CAPD requires 10.2 hours versus 6.4 hours for OPD, a 1.60× training cost, while peak GPU memory increases by only 5.21 GiB.The additional computation mainly comes from teacher forward passes for counterfactual temporal analysis.Source: Performance Comparison with Other… · p. 6

Ablation Study

  • Both counterfactually calibrated attention supervision and counterfactual token weighting improve over OPD; combining them performs best across benchmarks.Token-Only gives a larger gain than Attention-Only, while the two components provide complementary output- and evidence-level supervision.Source: Ablation Study · pp. 6–7, Table 3 · p. 6
  • G = 8 achieves the best overall result, and performance first improves then declines as λatt increases; the study selects λatt = 0.25.Finer temporal grouping improves group precision but reduces evidence coverage, preference gap, and selection stability; moderate attention weighting balances evidence and output-level supervision.Source: Ablation Study · pp. 6–7, Table 4 · p. 6, Figure 3 · p. 6

Qualitative Analysis

  • In the visualized example, CAPD recovers the event tail and predicts a more complete interval than OPD and RAL-AD, whose predictions end before the action is complete.Across two additional examples, counterfactual influence highlights groups containing the queried action while raw attention peaks on visually salient but irrelevant content.Source: Qualitative Analysis · p. 7, Figure 4 · p. 7

Conclusion

CAPD uses counterfactual influence from temporal groups to calibrate teacher attention and output-level distillation for video temporal grounding. Experiments on TimeLens show improved localization over OPD and RAL-AD while preserving general video understanding.

Conclusion

  • CAPD improves temporal localization over OPD and RAL-AD on TimeLens benchmarks while preserving general video understanding ability.The paper attributes this to integrating temporal evidence into on-policy distillation.Source: Conclusion · p. 7
  • CAPD estimates temporal groups’ counterfactual influence to calibrate teacher attention and output-level distillation.The method is introduced as an on-policy distillation approach for temporal grounding in video MLLMs.Source: Conclusion · p. 7
  • CAPD uses fixed temporal groups that may not fit diverse event durations, motivating event-adaptive finer partitions for short actions and coarser partitions for extended events.Multiple teacher forward passes increase post-training cost over OPD, but batched inference can reduce the overhead because the teacher is frozen.Source: Conclusion · p. 7

Related papers

Loading 2609.34581v2…