Source-linked AI summary
COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models
Chenghua Zhu, Zhaolu Kang, Qifan Shi, Siyan Wu, Kehan Jiang, Lei Wei, Lianyu Hu, Guangyuan Dong, Mingbo Yang, Rui Lu, Guibo Luo
TL;DR
Video MLLMs remain fragile on fine-grained temporal reasoning because sparse, independently encoded frames and training objectives do not fully represent or reward temporal change and direction. COMET adds explicit Taylor-based motion representation, appearance-motion fusion, and forward-reverse direction-aware optimization, yielding its strongest gains on action and temporal reasoning while transferring across model families.
Problem
Video MLLMs struggle with motion direction, action order, and state changes because temporal evidence is weakly represented and temporal sensitivity is not directly optimized.
Method
COMET uses a Taylor-style temporal motion branch, TAB-enhanced appearance-motion fusion, temporal-prior distillation, and forward-reverse TC-GRPO.
Results
COMET shows an Action > Reasoning > Perception pattern, with +4.9 pp Action and +2.1 pp Reasoning over BL-GRPO at 8B, while Perception changes by −0.4 pp.
Takeaways & Limitations
The Taylor-based temporal signal and direction-aware optimization transfer from Qwen3-VL to InternVL2.5, including across different visual-LLM architectures.
Abstract
from arXiv · showhide
Video multimodal large language models have advanced significantly, yet fine-grained motion-temporal understanding remains fragile. The core bottleneck is not only sparse frame sampling, but also the lack of a complete temporal modeling pipeline for explicitly representing frame-to-frame change, enabling appearance-motion interaction, and optimizing temporal direction sensitivity. We propose COMET, a temporally grounded framework that systematically strengthens video MLLMs through explicit temporal representation, appearance-motion fusion, and direction-aware optimization. Architecturally, COMET introduces a temporal motion branch built on Taylor frame differences and injects its motion evidence into the appearance stream via temporal attention bias-enhanced cross-attention. For optimization, COMET combines temporal prior distillation with a forward-reverse TC-GRPO stage that turns temporal order into a direct learning signal and strengthens the model's use of directional motion patterns encoded by the temporal motion branch. The method achieves consistent overall improvements with a pronounced motion-temporal bias: on Qwen3-VL-8B, action-centric tasks (STAR, SSv2) improve by 4.9% on average, temporal reasoning tasks (NExT-QA, CLEVRER, LLaVA-178K) by 2.1% over BL-GRPO, while static perception tasks (PerceptionTest) remain on par. The same gain pattern also transfers to InternVL2.5-8B, indicating that COMET generalizes across model families.
1 Introduction
COMET addresses fragile temporal reasoning in video MLLMs by making frame-to-frame change explicit, coupling motion with appearance, and optimizing sensitivity to temporal direction and order. Its framework combines a Taylor-style motion branch, attention-based fusion, and forward-reverse contrastive training.
- Motivation: Video MLLMs often struggle when answers depend on motion direction, action order, or state changes rather than static appearance cues.Temporally grounded evidence can be weaker than the model’s final answer confidence.
- Motivation: Sparse, independently encoded RGB frames leave inter-frame interaction absent, weakening temporal evidence and encouraging appearance-dominated shortcuts.This limits video-data utilization and temporal coherence.
- Motivation: COMET treats temporal reasoning as both an architectural and optimization problem requiring explicit motion representation and direction-sensitive training.The framework targets the mismatch between available temporal signals and the objectives that reward their use.
- COMET: COMET uses a lightweight Taylor-style motion signal and TAB-enhanced cross-attention to expose motion evidence before language-model reasoning.The motion branch preserves directionality while temporal attention bias guides appearance features toward motion-relevant evidence.
- COMET: COMET combines temporal-prior distillation with forward-reverse TC-GRPO, turning temporal order into a direct directional learning signal.The training design encourages more faithful use of directional motion patterns.
- COMET: The framework’s dual-branch encoder separates spatial appearance from explicit frame-to-frame motion while using inter-frame fusion for temporal reasoning.The RGB branch models spatial content, and the temporal motion branch models explicit change.
2 Related Work
Related work spans sparse-frame and temporal-fusion video MLLMs, explicit motion representations, dual-branch fusion, and GRPO-style reasoning optimization. COMET combines these directions through a dedicated motion pathway and explicit temporal-order training.
- Video MLLM temporal modeling: Video MLLMs commonly extend image-pretrained encoders with sparse sampling, spatiotemporal tokens, temporal adapters, connectors, or temporal-attention modules.These approaches strengthen temporal processing through sampling, encoding, adaptation, or fusion mechanisms.
- Explicit motion representations: Explicit motion representations such as optical flow and frame differencing complement RGB features when dynamic evidence is difficult to recover from static frames.Flow4Agent uses optical-flow priors to refine frame hierarchy and prune redundant visual tokens.
- Dual-branch fusion: Dual-backbone, multi-branch, and cross-attention fusion paradigms motivate COMET’s separation of static appearance and explicit motion pathways.COMET connects the two branches through aligned inter-frame cross-attention before language reasoning.
- Dual-branch fusion: COMET differs from single-stream temporal enhancement by preserving a dedicated motion pathway and injecting its evidence into visual reasoning before the LLM.This design retains distinct appearance and motion roles while enabling their interaction.
- Optimization: GRPO-style optimization commonly follows supervised cold-start training, but vanilla GRPO does not directly encourage sufficient temporal-information use in video reasoning.COMET targets temporal-order sensitivity more explicitly within this training lineage.
3 Method
COMET combines explicit Taylor-based motion representation, appearance–motion fusion, and direction-aware training to make temporal evidence available throughout video MLLM reasoning. Its pipeline uses a dual-branch encoder with temporally biased cross-attention, followed by temporal prior distillation and TC-GRPO.
- 3.2 Explicit Temporal Representation: COMET uses Taylor-style grayscale frame differences as an explicit, lightweight, estimator-free motion signal that preserves temporal direction.Reversing a video flips the Taylor motion pattern while largely preserving appearance.
- 3.3 Dual-Branch Visual Encoding: A dual-branch encoder assigns RGB frames to appearance and Taylor inputs to temporal evidence, preserving distinct roles before aligned fusion.ViT-A preserves spatial appearance priors, while ViT-B processes Taylor inputs and the branches are fused at multiple DeepStack-aligned levels.
- 3.4 Appearance-Motion Fusion: Cross-attention fusion lets RGB queries retrieve Taylor-based temporal evidence across frames at corresponding spatial positions.The fusion block injects motion into the appearance stream through a residual path rather than simply concatenating or averaging the branches.
- 3.4 Appearance-Motion Fusion: Temporal attention bias prioritizes motion-salient moments using Taylor-derived motion strength, temporal activity, and frame distinctiveness.The bias is computed from features already present in the fusion path, without an auxiliary scorer or extra visual branch.
- 3.5 Direction-Aware Temporal Optimization: COMET trains in two stages: TPD distills temporal reasoning traces, then TC-GRPO directly addresses reliance on temporal direction.TPD updates the temporal branch, fusion modules, and LLM while keeping the appearance ViT frozen; TC-GRPO exploits Taylor directionality under video reversal.
4 Experiments
COMET is evaluated across multiple video MLLM backbones, benchmarks, temporal baselines, ablations, and optimization settings. Its gains are strongest on action and reasoning tasks, while the framework remains lightweight and transfers across architectures.
- Evaluation Setup: The evaluation covers Qwen3-VL at three scales and InternVL2.5-8B across Action, Reasoning, and Perception benchmarks.Action includes STAR and SSv2; Reasoning includes NExT-QA, CLEVRER, and LLaVA-178K; Perception uses PerceptionTest.
- Overall Results: 5.9–10.2 pp cumulative gains over pretrained models are obtained with TC-GRPO across scales and backbones.BL-SFT contributes 2.4–4.2 pp, while TPD adds a further 1.4–2.6 pp.
- Cross-Architecture Results: COMET performs best across all three Qwen3-VL scales and achieves the best average accuracy on InternVL2.5-8B.The temporal advantage transfers despite InternVL2.5 using a standard ViT-MLP-LLM design instead of Qwen3-VL’s DeepStack pathway.
- Category-Level Results: +4.9 pp on Action and +2.1 pp on Reasoning over BL-GRPO at 8B, versus −0.4 pp on Perception.COMET therefore concentrates improvements on motion- and temporal-relevant tasks rather than producing uniform gains.
- Component Ablation: The temporal branch with appearance-motion fusion supplies most ablation benefit, while TAB provides a modest additional refinement.The authors characterize COMET’s value as the combination of components in an estimator-free, relatively lightweight temporal pipeline.
- Direction-Aware Optimization: 6+1 TC-GRPO reaches 78.8 average accuracy, exceeding standard GRPO’s 78.0 and the 5+2 and 4+3 variants.A small reversed-sample fraction improves direction-sensitive exploration, whereas more reversed samples weaken the forward-video objective.
5 Conclusion
COMET addresses the underuse of temporal evidence in video MLLMs by making frame-to-frame change explicit, enabling appearance-motion interaction, and optimizing temporal directionality. Experiments show these temporal signals transfer across Qwen3-VL and InternVL2.5 model families, while future work targets more efficient fusion and broader evaluation.
- COMET improves video reasoning by explicitly modeling frame-to-frame change, appearance-motion interaction, and temporal directionality.The framework targets appearance-dominated shortcuts and sparse temporal evidence in current pipelines.
- Taylor features and direction-aware optimization transfer across Qwen3-VL and InternVL2.5, spanning DeepStack-based and standard ViT-MLP-LLM architectures.
- Future work will explore deeper fusion of the two ViT branches to reduce parameter footprint and strengthen inter-frame interaction.
- The framework will also be evaluated on broader benchmarks and longer-form video reasoning settings.