LITA: Language Instructed Temporal-Localization Assistant

Motivation

• Existing Video LLMs struggle to answer “When?” questions, in part because of inadequate time representation, temporal resolution, and timestamped training data.Plain-text timestamps depend on frame rate, sparse sampling can miss temporal detail, and timestamps in instruction-tuning data are scarce and unverified.Source: §1 · pp. 1–2

Time representation

• LITA represents relative video positions with time tokens <1> to <T>, allowing timestamps to be encoded and decoded using video length rather than frame rate.The tokens divide the video into T equal-length chunks, making start and end positions well-defined from the input video.Source: §1 · p. 2

Architecture

• LITA uses fast tokens at high temporal resolution and few tokens per frame, alongside slow tokens at low temporal resolution and more tokens per frame for spatial information.This SlowFast design supports dense frame sampling while addressing the LLM’s context-length limitation.Source: §1 · p. 2

Data and task

• LITA trains on timestamped dense video captioning and event localization data, and introduces ActivityNet-RTL for reasoning temporal localization that requires world knowledge, temporal reasoning, timestamps, and explanations.RTL evaluates temporal understanding together with the reasoning needed to identify and localize an event.Source: §1 · p. 2

Results

• LITA doubles baseline performance on RTL temporal metrics (mIOU, Precision@0.5) and improves Correctness of Information by 22% and Temporal Understanding by 36% relative to existing Video LLMs.The latter improvements are on a benchmark for video-based question answering [27].Source: §1 · p. 2

Multimodal Large Language Models

• Multimodal LLMs adapt to visual inputs with gated cross-attention or adapter layers, or project visual-encoder outputs into LLM input space; LISA [18] extends referring to reasoning segmentation.LITA shares LISA’s reasoning-task spirit by proposing Reasoning Temporal Localization to jointly evaluate reasoning and temporal understanding.Source: §2 · p. 2

Video Large Language Models

• Video LLMs project visual tokens through projection layers or Q-Formers and support descriptive questions and instructions, but lack temporal localization; LITA targets this gap.Concurrent VTimeLLM [13], TimeChat [32], and Momentor [30] also address temporal localization, while LITA adds reasoning to it.Source: §2 · pp. 2–3

Temporal Localization in Videos

• Temporal localization identifies activities in untrimmed videos, either as predefined action classes or natural-language events; instruction-following Video LLMs like LITA can unify diverse temporal-task frameworks.Related tasks include dense video captioning [14] [16] [40] and action segmentation [26] [34].Source: §2 · p. 3

3.1. Architecture

• LITA encodes selected video frames as visual tokens, combines them with text tokens, and fine-tunes the full model on temporal-localization data and other video tasks.It builds on LLaVA [24] but is not dependent on that base architecture; prompts and responses use time tokens that can be converted to timestamps using video length.Source: §3.1 · p. 3

3.2. Time Tokens

• Relative time tokens represent timestamps as positions within the video, avoiding dependence on an unknown frame rate while introducing discretization error.For T steps and video length L, timestamp τ maps to token <t> using t = round(τ(T −1)/L) + 1; the token maps back to τ = L(t −1)/(T −1).Source: §3.2 · pp. 3–4

3.4. Training Tasks

• Time-token instructions support timestamped dense captions and “When?” event-localization questions, alongside standard video question answering.Dense captioning represents each sentence with start and end times, while event localization returns the event’s start and end times.Source: §3.2 · p. 4, §3.4 · p. 4

3.3. SlowFast Visual Tokens

• SlowFast pooling combines dense temporal tokens with sparse spatial tokens, reducing T × M frame tokens to T + M video tokens.Fast tokens average all tokens within each frame; slow tokens spatially pool uniformly selected frames, using s = 2 in the experiments.Source: §3.3 · p. 4

3.4. Training Tasks

• LITA is trained on five tasks: dense video captioning, event localization, video question answering, natural-language visual question answering, and reasoning temporal localization.RTL answers include timestamps and an explanation; the paper reports that training data helps LITA learn to provide both. Visual instruction tuning improves conversation quality while maintaining good video understanding.Source: §3.4 · p. 4

4. Reasoning Temporal Localization

• RTL is proposed to combine Video LLMs’ temporal understanding with their reasoning capabilities.The paper argues that standard temporal localization does not fully use Video LLMs’ reasoning abilities.Source: §4 · p. 5

4.1. Problem Definition

• RTL asks for timestamps of an indirectly described event and an explanation derived through reasoning and model world knowledge.The target event is not directly named in the question; the response gives its start and end times and explains how they were inferred.Source: §4.1 · p. 5

4.2. ActivityNet-RTL Dataset

• ActivityNet-RTL uses GPT-4-generated questions from ActivityNet-Captions, with 33,557 training pairs and a manually curated 229-pair evaluation set across 160 videos.Training uses generated data from 10,009 videos; non-reasoning questions are removed from the evaluation set, whose timestamps and explanations are also checked.Source: §4.2 · pp. 5–6

4.3. Metrics

• ActivityNet-RTL evaluates temporal localization with mIOU and Precision@0.5, and explanations with GPT-4 Relative Scores.Localization metrics are averaged per video before averaging across videos; GPT-4 scores explanation helpfulness, relevance, accuracy, and detail, with predictions normalized by ground-truth scores.Source: §4.3 · p. 6

5.1. Implementation Details

• The implementation uses 100 sampled frames and relative time tokens, with slow and fast visual tokens, and trains across five tasks.It uses CLIP-L-14 and Vicuna, with a LLaVA-style projection; training samples 100K examples per task with replacement for 4K iterations. Datasets include ActivityNet-Captions, YouCook2, NExT-QA, LLaVA-150K, and ActivityNet-RTL.Source: §5.1 · p. 6

5.2. Reasoning Temporal Localization Evaluation

• LITA nearly doubles mIOU and P@0.5 versus “Slow Tokens Only” on ActivityNet-RTL, and localization quality improves from 7B to 13B.All metrics are averaged over three trials; explanation Score is GPT-4-assisted and may be slightly affected by timestamps.Source: Table 1 · p. 6, §5.2 · p. 7
• LITA’s qualitative examples show more accurate event localization and detailed explanations than Video-LLaMA-v2.Examples include locating a second arm wrestling, identifying marshmallow roasting, and describing a gymnast falling off the beam and getting back up.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3. Video-Based Generation Evaluation

• LITA improves Correctness of Information by 22% (2.94 vs. 2.40) and Temporal Understanding by 36% relative (2.68 vs. 1.98) on video-based text generation.The benchmark uses GPT to assess correctness, detail orientation, contextual understanding, temporal understanding, and consistency.Source: Table 2 · p. 7, §5.3 · p. 8

5.4. Evaluating the Effects of Training Tasks

• “No NLVQA” improves all ActivityNet-RTL metrics over “RTL Only,” while adding NLVQA further supports reasoning and natural-language answers.RTL-only training lacks sufficient supervision for timestamp accuracy and explanation quality; standard video tasks enable answers to non-temporal questions, but they tend to be short.Source: §5.4 · pp. 8–9, Table 3 · p. 8

• LITA enables accurate temporal localization and demonstrates the ability to answer complex temporal-localization questions.Source: §6 · p. 9
• Its time tokens represent time, while SlowFast tokens efficiently process video; experiments show the importance of both components.Source: §6 · p. 9
• The Reasoning Temporal Localization task and ActivityNet-RTL dataset support training with temporal-localization data, enabling localization and improving general video understanding.Source: §6 · p. 9
• LITA substantially improves video-based text generation over existing Video LLMs, including for non-temporal questions, through its model design and data strategy.Source: §6 · p. 9
• Analysis also finds benefits from incorporating standard video tasks and image instruction-tuning data when training Video LLMs.Source: §6 · p. 9