Source-linked AI summary
Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning
Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, Siliang Tang
TL;DR
Existing Video-LLMs lack precise temporal representation and segment-level modeling for fine-grained video comprehension and localization. Momentor combines temporal modeling with Moment-10M, a segment-annotated instruction dataset, and outperforms previous Video-LLMs across several temporally precise tasks. The paper concludes that this training enables comprehensive segment-level reasoning and fine-grained video understanding.
Problem
Existing Video-LLMs capture coarse semantics but lack effective temporal representation and segment-level modeling for precise video comprehension and localization.
Method
Momentor uses a Temporal Perception Module and Grounded Event-Sequence Modeling, trained with Moment-10M generated from segment-level video information and annotations.
Results
Momentor outperforms previous Video-LLMs on temporal grounding, dense captioning, action segmentation, and highlight moment retrieval.
Takeaways & Limitations
Momentor performs segment-level reasoning and localization for fine-grained temporally grounded video understanding.
Takeaways & Limitations
The dataset is sourced from internet videos and may contain personal information, requiring privacy safeguards.
Abstract
from arXiv · showhide
Large Language Models (LLMs) demonstrate remarkable proficiency in comprehending and handling text-based tasks. Many efforts are being made to transfer these attributes to video modality, which are termed Video-LLMs. However, existing Video-LLMs can only capture the coarse-grained semantics and are unable to effectively handle tasks related to comprehension or localization of specific video segments. In light of these challenges, we propose Momentor, a Video-LLM capable of accomplishing fine-grained temporal understanding tasks. To support the training of Momentor, we design an automatic data generation engine to construct Moment-10M, a large-scale video instruction dataset with segment-level instruction data. We train Momentor on Moment-10M, enabling it to perform segment-level reasoning and localization. Zero-shot evaluations on several tasks demonstrate that Momentor excels in fine-grained temporally grounded comprehension and localization.
1. Introduction
Existing Video-LLMs capture coarse video semantics but lack precise temporal representation and segment-level modeling. Momentor addresses these gaps with temporal modeling, segment-level instruction data, and zero-shot performance across fine-grained video tasks.
- 1. Introduction: Existing Video-LLMs lack precise temporal representations and segment-level modeling, limiting responses to specified segments and precise content localization.They encode sampled frames independently and focus mainly on global visual semantics.
- 1. Introduction: Momentor introduces fine-grained temporal awareness and segment-level reasoning to address these limitations.Its Temporal Perception Module represents temporal positions and injects temporal information into frame features.
- 1. Introduction: Moment-10M supplies extensive segment-level instruction data generated from tracked instances, detected event boundaries, and structured video information.The dataset includes single-segment and cross-segment reasoning tasks and comprises 1.5 million segments and 451.5 thousand instance tracks.
- 1. Introduction: Momentor outperforms previous Video-LLMs on temporal grounding, dense captioning, action segmentation, and highlight moment retrieval.It provides responses based on user-indicated segments and locates target segments matching user requirements.
2. Related Work
Related work covers multimodal understanding and fine-grained video tasks such as segmentation, grounding, and highlight retrieval. Unlike task-specific prior methods, this work targets a unified Video-LLM that solves these tasks zero-shot.
- 2. Related Work: Fine-grained video understanding treats videos as interconnected events requiring temporal comprehension or localization.Representative tasks include action segmentation, temporal grounding, and highlight moment retrieval.
- 2. Related Work: Momentor aims to solve multiple fine-grained video tasks with one unified Video-LLM in a zero-shot setting, unlike task-specific prior methods.The targeted tasks include identifying segments, assigning action labels, and locating related moments.
- 2. Related Work: Video-LLMs extend LLM capabilities to video by combining visual encoders and temporal information aggregation with language modeling.Related systems seek video understanding, reasoning, and interaction through multimodal alignment.
3. Momentor
Momentor combines temporal-aware architecture with grounded event-sequence training to support fine-grained video comprehension and localization. Its Temporal Perception Module represents precise positions continuously, injects them into frame features, and is trained through temporally grounded event sequences.
- 3. Momentor: Momentor combines a frame encoder, projection layer, Temporal Perception Module, and LLM to process videos with injected temporal information.Frames are encoded independently, projected into the LLM feature space, temporally processed, and combined with tokenized instructions.
- 3.2. Temporal Perception Module (TPM): Continuous temporal tokens avoid the quantization error of discrete timestamp tokens, enabling precise temporal position representation.The approach constructs a continuous temporal token space over temporal tokens rather than using discrete tokens directly on continuous timelines.
- 3.2. Temporal Perception Module (TPM): Neighboring token propagation strengthens continuity by sharing exponentially distance-weighted parameter updates among adjacent temporal-token embeddings.The mechanism preserves the current token value while allowing adjacent tokens to receive related updates.
- 3.2. Temporal Perception Module (TPM): Temporal embeddings corresponding to sampled-frame positions are added to projected frame features as fine-grained temporal position encodings.The embeddings and projected frame features share dimensionality, allowing direct temporal-information injection.
- 3.3. Grounded Event-Sequence Modeling: Grounded Event-Sequence Modeling pre-trains temporal tokens to align with video timelines and decode sequences of timestamped events and captions.The event sequence concatenates events containing continuous start and end temporal embeddings with event captions, and is trained with a language-modeling loss.
- 3.3. Grounded Event-Sequence Modeling: The automatic instruction-data engine extracts structured video information and generates diversified instruction data for segment-level training.The generated event information supports instruction construction, while event-sequence data is used for grounded event-sequence modeling.
4. Moment-10M
Moment-10M is a large-scale video instruction dataset designed for segment-level reasoning, built from structured instance and event information. Its generation pipeline detects coherent events, organizes instance-event relationships, and uses an LLM to produce diverse instruction data.
- Moment-10M generates segment-level reasoning instructions from automatically extracted instance, event, and relationship information in videos.The engine is designed to support fine-grained annotations and comprehensive segment-level reasoning.
- Structured Information Extraction: Instance tracks capture object dynamics over time, while event-boundary detection combines frame differences with semantic merging to produce coherent event segments.Grounding DINO supplies instance information, and adjacent sub-segments are merged when their consistency exceeds a threshold.
- Structured Information Extraction: The Instance-Event Matrix represents instance tracks as rows and events as columns, organizing visual clues and spatio-temporal associations for instruction generation.The matrix includes the video itself as a track and emphasizes complex associations between events.
- Instruction Data Generation: Vicuna converts structured matrix information into instruction data spanning five single-segment tasks and three multi-segment reasoning tasks.All eight task types support instruction fine-tuning, while chronological segment captions support Grounded Event Sequence Modeling.
- Instruction Data Generation: Moment-10M includes varied task types whose distribution is summarized in Figure 4.
5. Experiments
Momentor is evaluated zero-shot across fine-grained temporal tasks, Video QA, ablations, dataset validation, scaling, qualitative cases, and temporal-token visualizations. It performs strongly on segment localization, event description, highlight retrieval, and coarse-grained Video QA, while analyses examine the contributions of its components and data.
- 5. Experiments: Momentor outperforms previous Video-LLMs across temporal grounding, dense captioning, action segmentation, and highlight moment retrieval.The evaluation covers Breakfast, 50 Salads, ActivityNet Captions, Charades-STA, and QVHighlights.
- 5.2. Dense Video Captioning: Momentor provides more detailed event descriptions, more accurate boundaries, and complete event coverage in dense video captioning.Grounded Event-Sequence Modeling is reported to support precise timestamps and event descriptions.
- 5.3. Temporal Grounding / 5.5. Highlight Moment Retrieval: Momentor achieves the highest mean IoU among existing Video-LLMs for temporal grounding and state-of-the-art highlight moment retrieval.The reported gains are attributed respectively to its continuous temporal token space and cross-segment multi-event reasoning.
- 5.6. Video QA: Momentor achieves state-of-the-art or comparative performance across ActivityNet-QA, MSRVTT-QA, and MSVD-QA.These results demonstrate capability in coarse-grained video understanding in addition to fine-grained temporal tasks.
- 5.7. In-Depth Analysis: Removing continuous interpolation or neighboring token propagation decreases overall performance, with interpolation primarily affecting localization metrics.The ablation results are reported on ActivityNet Captions, Breakfast, and QVHighlights.
- 5.7. In-Depth Analysis: Video-ChatGPT trained on Moment-10M improves on fine-grained temporal reasoning tasks, while Momentor performance generally increases with more data before slowing at million-level scale.Qualitative cases also show localization from visual and textual inputs, vague descriptions, and referred instances.
6. Conclusion
The paper proposes Momentor and Moment-10M for segment-level video comprehension and localization. Its Temporal Perception Module and Grounded Event-Sequence Modeling support fine-grained temporal understanding across evaluated tasks.
- 6. Conclusion: Momentor combines fine-grained temporal representation, Grounded Event-Sequence Modeling, and Moment-10M to perform segment-level reasoning.Moment-10M contains 10 million diversified instructions with segment-level annotations.
Impact Statement
The dataset is sourced from internet videos and is curated with privacy safeguards. The authors acknowledge potential personal information and describe measures intended to protect it.
- Impact Statement: Moment-10M is sourced from internet videos and includes privacy safeguards addressing potential personal information.The authors also state that the model is developed to avoid social harm and ethical breaches.
A. Overview
The appendix provides supplementary implementation details, task descriptions, and instruction-generation prompts. These materials correspond to Sections B, C, and D.
- A. Overview: The appendix covers Momentor implementation details, Moment-10M task descriptions, and prompts used for instruction generation.The listed materials are organized in Sections B, C, and D.
B. Implementation
Momentor uses CLIP ViT-L/14 for frame encoding and LLaMA 7B as its language model, with temporal tokens and uniformly sampled frames supporting fine-grained reasoning.
- CLIP ViT-L/14 encodes frames, while LLaMA 7B serves as the language model.
- The model incorporates N = 300 temporal tokens and uniformly samples M = 300 frames per video.
- Training freezes the frame encoder and LLM, updating only the linear projection layer and TPM.
C. Task Formats
Momentor supports segment- and instance-level comprehension, multiple forms of temporal localization, activity summarization, and reasoning across segments.
- Segment captioning, segment QA, and instance QA assess content description and question answering at localized temporal or instance scopes.
- Direct, inferential, and composed segment localization require timestamp retrieval from explicit queries, hypothetical scenarios, or source-target differences.
- Instance activity summarizing asks the model to summarize an instance’s activities throughout the entire video.
- Cross-segment QA requires combining information from multiple segments to answer questions.
D. Prompts
The prompts generate segment-level captions, factual dialogue, multi-clip reasoning, hypothetical retrieval questions, and composed retrieval queries for Momentor’s task formats.
- Captioning prompts ask models to synthesize noisy frame- or clip-level descriptions into concise, informative captions under a 20-word limit.
- Segment QA prompts generate concise factual questions and answers directly inferred from segment captions.
- Inferential localization prompts create hypothetical scenario questions asking which scene or clip should be watched.
- Cross-segment prompts combine descriptions from different timestamps to produce factual questions and answers about shared instances or activities.
- Composed retrieval prompts describe differences between source and target clips to guide target-segment retrieval.