Source-linked AI summary
LITA: Language Instructed Temporal-Localization Assistant
De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, Jan Kautz
TL;DR
Video LLMs struggle with temporal localization, especially for “When?” questions, because of limitations in time representation, architecture, and data. LITA addresses these issues with relative time tokens, SlowFast tokens, and temporal-localization training including RTL and ActivityNet-RTL. It doubles baseline temporal metrics on RTL and improves video-based generation, including a 36% relative gain in Temporal Understanding.
Problem
Existing Video LLMs support video instruction following but cannot accurately localize time periods in response to “When?” questions.
Method
LITA combines relative time tokens, SlowFast tokens, and temporal-localization training data including the RTL task and ActivityNet-RTL dataset.
Results
LITA doubles baseline temporal metrics on RTL and achieves a 36% relative improvement in Temporal Understanding over existing Video LLMs.
Takeaways & Limitations
Emphasizing temporal localization improves both temporal reasoning and broader video-based text generation.
Abstract
from arXiv · showhide
There has been tremendous progress in multimodal Large Language Models (LLMs). Recent works have extended these models to video input with promising instruction following capabilities. However, an important missing piece is temporal localization. These models cannot accurately answer the "When?" questions. We identify three key aspects that limit their temporal localization capabilities: (i) time representation, (ii) architecture, and (iii) data. We address these shortcomings by proposing Language Instructed Temporal-Localization Assistant (LITA) with the following features: (1) We introduce time tokens that encode timestamps relative to the video length to better represent time in videos. (2) We introduce SlowFast tokens in the architecture to capture temporal information at fine temporal resolution. (3) We emphasize temporal localization data for LITA. In addition to leveraging existing video datasets with timestamps, we propose a new task, Reasoning Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for learning and evaluating this task. Reasoning temporal localization requires both the reasoning and temporal localization of Video LLMs. LITA demonstrates strong performance on this challenging task, nearly doubling the temporal mean intersection-over-union (mIoU) of baselines. In addition, we show that our emphasis on temporal localization also substantially improves video-based text generation compared to existing Video LLMs, including a 36% relative improvement of Temporal Understanding. Code is available at: https://github.com/NVlabs/LITA
NVIDIA
The passage provides NVIDIA author contact information.
- The authors’ NVIDIA email addresses are listed.
1. Introduction
Existing Video LLMs follow instructions but struggle with temporal localization, motivating LITA’s changes to time representation, architecture, and training data. LITA adds reasoning temporal localization and reports improved temporal metrics and video-based generation.
- Video LLMs often fail to accurately localize time periods when answering “When?” questions, despite supporting video instruction following.
- Three limitations are identified: plain-text timestamps, insufficient temporal resolution, and sparse or unverified timestamp data.
- LITA addresses these issues with relative time tokens, SlowFast tokens, and temporal-localization training data including RTL and ActivityNet-RTL.
- Relative time tokens divide each video into T equal chunks, allowing timestamps to be encoded and decoded from video length.
- RTL requires reasoning and temporal localization because answers depend on world knowledge, temporal reasoning, and explanations alongside timestamps.
- LITA doubles baseline temporal metrics on RTL while improving explanations, and raises Correctness of Information by 22% and Temporal Understanding by 36%.
2. Related Work
Related Video LLMs extend image-based multimodal models but remain limited in temporal localization, while LITA adds reasoning to this capability.
- Prior multimodal LLMs adapt language models through cross-attention, adapters, projection layers, or Q-Formers.
- Existing Video LLMs mainly project visual tokens into LLM input space but still lack temporal localization capabilities.
- LITA’s architecture overview represents video through Fast and Slow pathways, relative time tokens, and language-based task instructions.
- LITA differs from concurrent temporal-localization models by introducing a reasoning aspect to temporal localization.
3. Language Instructed Temporal-Localization
LITA combines relative time tokens, SlowFast visual-token pooling, and multitask training to support fine-grained temporal localization and reasoning-oriented video tasks.
- LITA enables temporal localization through relative time tokens, fine-resolution SlowFast tokens, and multitask training with accurate timestamps.
- 3.1. Architecture: SlowFast pooling reduces T × M visual tokens to T + M tokens by combining dense temporal Fast tokens with spatially richer Slow tokens.
- 3.2. Time Representation: Relative time tokens <1> to <T> encode timestamps within T video chunks and can be converted to and from timestamps using video length.
- 3.4. Training Tasks: Dense captioning and event localization express video tasks as language instructions containing temporal start and end tokens.
- 3.4. Training Tasks: LITA jointly trains on dense captioning, event localization, video question answering, natural-language visual question answering, and RTL.
- 3.4. Training Tasks: RTL answers contain both timestamps and explanations, requiring the model to learn reasoning and localization together.
4. Reasoning Temporal Localization
Reasoning Temporal Localization extends temporal localization by requiring Video LLMs to infer an indirectly described event, locate its timestamps, and explain the reasoning. ActivityNet-RTL is built from captioned videos with generated training examples and manually curated evaluation questions, assessed on localization and explanation quality.
- Reasoning Temporal Localization: RTL requires inferring an indirectly described target event, predicting its start and end timestamps, and explaining the reasoning.Unlike standard temporal localization, the query does not directly describe the target event and may require world knowledge.
- ActivityNet-RTL Dataset: ActivityNet-RTL is curated from ActivityNet Captions, whose videos contain multiple sentence-described events with start and end timestamps.The caption annotations provide the temporal context used to construct reasoning questions.
- Training Set Generation: GPT-4 generates training questions and timestamp-plus-explanation answers from caption context, yielding 33,557 pairs across 10,009 training videos.The generated questions are generally valid, although some directly ask about events already described in the captions and therefore require little reasoning.
- Evaluation Set Curation: The evaluation set removes non-reasoning questions manually and verifies answer timestamps and explanations, producing 229 pairs from 160 videos.This curation is intended to avoid an evaluation set dominated by questions that do not require reasoning.
- Evaluation: ActivityNet-RTL evaluates localization with mIOU and Precision@0.5, and explanation quality with GPT-4 Relative Scores.The localization metrics average per-video results before averaging across evaluation videos, while explanation scores assess helpfulness, relevance, accuracy, and detail against ground-truth explanations.
5. Experiments
LITA is evaluated on temporal localization, video-based text generation, and training-task ablations, with comparisons against Video LLM baselines and model variants. Across these evaluations, LITA improves temporal localization, explanation quality, and video understanding.
- 5.2. Reasoning Temporal Localization Evaluation: LITA substantially outperforms all baselines on ActivityNet-RTL across temporal localization and explanation metrics.The evaluation compares LITA variants with Video-LLaMA-v2 and Video-ChatGPT; Table 1 reports results averaged over three trials.
- 5.2. Reasoning Temporal Localization Evaluation: LITA provides more detailed and accurate explanations than Video-LLaMA-v2 in qualitative ActivityNet-RTL examples.Examples include identifying the second arm wrestling, roasting marshmallows with timing, and explaining resilience after a fall from a beam.
- 5.2. Reasoning Temporal Localization Evaluation: Temporal localization accuracy improves when scaling LITA from 7B to 13B.The paper reports this scaling trend on ActivityNet-RTL in Table 1.
- 5.3. Video-Based Generation Evaluation: 36% relative improvement in Temporal Understanding and 22% improvement in Correctness of Information over existing Video LLMs.LITA achieves 2.68 versus 1.98 for Temporal Understanding and 2.94 versus 2.40 for Correctness of Information.
- 5.4. Evaluating the Effects of Training Tasks: Training-task ablations show that RTL alone is insufficient, standard video tasks improve all metrics, and NLVQA improves reasoning and natural-language responses.“RTL Only” struggles with timestamps, explanations, and non-temporal questions; “No NLVQA” answers non-temporal questions but produces short answers, while LITA produces natural-language responses.
6. Conclusion
LITA combines model-design and data-strategy changes to improve temporal localization with Video LLMs and strengthen video-based text generation, including for non-temporal questions.
- LITA enables accurate temporal localization and improves video-based text generation compared with existing Video LLMs, including for non-temporal questions.
- Its model design uses time tokens for temporal representation and SlowFast tokens for efficient video processing.
- Its data strategy emphasizes temporal localization through the Reasoning Temporal Localization task and ActivityNet-RTL dataset.