PaperlayerView exact revision on arXiv

Source-linked AI summary

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, Lu Hou

arXiv:2312.02051v2cs.CVcs.AIcs.CL

TL;DR

Long videos call for an assistant that can identify meaningful moments and locate them in time. TimeChat combines timestamp-aware and variable-length video representations with instruction tuning, and substantially outperforms previous video LLMs across zero-shot tasks, including +27.5 R@1 (IoU=0.5) on Charades-STA.

  • Problem

    Developing an assistant for long-form video comprehension that can localize timestamps, detect key moments, and summarize meaningful segments remains a longstanding research pursuit.

    Source: §1 · p. 2

  • Method

    TimeChat combines a timestamp-aware frame encoder and sliding video Q-Former with instruction tuning on TimeIT, a dataset of 6 tasks and 125K instances.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    TimeChat substantially outperforms previous VidLLMs across zero-shot tasks, including +27.5 R@1 (IoU=0.5) on Charades-STA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TimeChat demonstrates strong temporal localization capabilities, identifying significant events, pinpointing their start and end times, and generating concise summaries.

    Source: §5 · p. 8

    TimeChat is not intended to achieve state-of-the-art performance on every task, and specialized models may use more training data and downstream fine-tuning.

    Source: §4.7 · p. 8

Abstract

from arXiv · show

This work proposes TimeChat, a time-sensitive multimodal large language model specifically designed for long video understanding. Our model incorporates two key architectural contributions: (1) a timestamp-aware frame encoder that binds visual content with the timestamp of each frame, and (2) a sliding video Q-Former that produces a video token sequence of varying lengths to accommodate videos of various durations. Additionally, we construct an instruction-tuning dataset, encompassing 6 tasks and a total of 125K instances, to further enhance TimeChat's instruction-following performance. Experiment results across various video understanding tasks, such as dense captioning, temporal grounding, and highlight detection, demonstrate TimeChat's strong zero-shot temporal localization and reasoning capabilities. For example, it achieves +9.2 F1 score and +2.8 CIDEr on YouCook2, +5.8 HIT@1 on QVHighlights, and +27.5 R@1 (IoU=0.5) on Charades-STA, compared to state-of-the-art video large language models, holding the potential to serve as a versatile video assistant for long-form video comprehension tasks and satisfy realistic user requirements.

1. Introduction

TimeChat targets long-video assistance that locates meaningful events with accurate timestamps, addressing VidLLMs’ difficulty preserving long-video semantics and aligning visual content with time. It combines duration-adaptive video-token compression with explicit frame-timestamp binding and is tuned for timestamp-related instructions.

1. Introduction

  • Existing VidLLMs, including VideoLLaMA [61] and VideoChat [19], struggle with untrimmed-video localization because fixed token compression degrades long-video semantics and timestamps are not explicitly bound to frames.The introduction notes that fixed compression can use a set number of tokens, such as 32, regardless of video duration.Source: §1 · p. 2
  • The sliding video Q-Former compresses frames in moving windows into variable-length token sequences, while a time-aware frame encoder binds each frame’s visual context to its timestamp.The adaptive sequence is intended to accommodate videos of different durations while preserving significant visual semantics.Source: §1 · p. 2
  • TimeIT contains 125K instances across 6 tasks and 12 benchmarks, with average video length of 190.8 seconds, reformatted as dialogs with manually written instructions.The paper presents TimeIT as the first time-sensitive, video-centric instruction-tuning dataset.Source: §1 · p. 2
  • +9.2 F1 score and +2.8 CIDEr on YouCook2 [64], +5.8 HIT@1 on QVHighlights [16], and +27.5 R@1 (IoU=0.5) on Charades-STA [7] over previous VidLLMs in zero-shot settings.Qualitative results in movie [58] and egocentric video domains demonstrate generalization toward a versatile, practical video assistant.Source: §1 · p. 2

2. Related Work

Related work identifies fixed video-token compression, instruction data without time-aware video requests, and task-specific temporal systems as gaps that TimeChat addresses through flexible compression, time-aware instruction tuning, and an LLM-based generalist approach.

2.1. Video Large Language Models

  • TimeChat contrasts with video-LLM approaches that compress videos to fixed token counts, offering adjustable compression and explicit frame-level vision-timestamp associations.Representative approaches include VideoChat [19] and VideoLLaMA [61]; fixed compression can degrade visual semantics in lengthy videos, while TimeChat’s design targets adaptability to varying video lengths and temporal localization.Source: §2.1 · p. 2

2.2. Vision-Language Instruction Tuning

  • Prior vision-language instruction-tuning data neglects time-aware video requests, motivating TimeChat’s time-aware instruction-tuning dataset.Existing approaches either convert multimodal benchmark datasets into instruction format [3] [55] or use LLMs to create image- or video-centric dialogue data [19] [25] [30].Source: §2.2 · pp. 2–3

2.3. Video Temporal Localization

  • Temporal video tasks require linking semantics to timestamps, but prior studies often address tasks separately and LLM-based generalist approaches remain under exploration.Tasks include temporal grounding, dense captioning, summarization, highlight detection, and step localization; TimeChat unifies a wide range of time-sensitive tasks in language-modeling format.Source: §2.3 · p. 3

3. Method

TimeChat combines timestamp-aware frame encoding and sliding-window temporal aggregation to support long-video understanding, and uses TimeIT, a 125K-instance instruction-tuning dataset spanning six tasks, to improve instruction following across time-sensitive video tasks.

3.1.1 Overview

  • TimeChat independently encodes frames, aggregates them into video tokens, then combines those tokens with optional transcribed speech and user instructions for LLM response generation.The architecture comprises a time-aware frame encoder, a sliding video Q-Former, and a large language model.Source: §3.1.1 · p. 3

3.1.2 Timestamp-aware Frame Encoder

  • Timestamp descriptions condition the image Q-Former to bind each frame’s visual content to its time, addressing prior methods’ limited precision about when events occur.The encoder uses a pretrained ViT for frame features, then cross-attention between learnable queries and those features to produce visual tokens.Source: §3.1.2 · p. 4

3.1.3 Sliding Video Q-Former

  • Sliding windows aggregate temporal relations across frames and produce (T/S) × NV video tokens, avoiding the excessive compression of fixed-token approaches on long videos.Stride S controls the final token count and computation budget while retaining richer semantics for long videos.Source: §3.1.3 · p. 4

3.1.4 Large Language Model

  • The LLM receives video tokens and text-query tokens in its embedding space and is instruction-tuned from existing pretrained checkpoints using language-modeling loss and LoRA.Text queries can include optional transcribed speech and user instructions.Source: §3.1.4 · p. 4

3.2. Instruction Data TimeIT

  • TimeIT is a timestamp-focused video instruction-tuning dataset built from timestamp-associated datasets and characterized by long-form videos.The paper describes it as comprising 6 tasks and 125K instances.Source: §3 · p. 3, §3.2 · p. 5

3.2.1 Task Coverage

  • TimeIT covers six timestamp-related tasks, including dense captioning, temporal grounding, step localization and captioning, summarization, highlight detection, and transcribed speech generation.It incorporates 12 datasets from different domains to accommodate prevalent timestamp-related video requests.Source: §3.2.1 · p. 5

3.2.2 Data Construction

  • TimeIT converts source datasets into instruction-following examples through instruction writing and answer formatting.Instructions are manually initialized, expanded with GPT-4, and manually selected and refined; outputs are reformulated as user-friendly natural-language responses. The paper reports that all six tasks benefit model performance.Source: §3.2.2 · p. 5

4. Experiments

In zero-shot tests, TimeChat outperforms prior video LLMs across three long-video tasks. Ablations and frame-scaling results support its architectural choices, while fine-tuned comparisons show remaining gaps to specialized models.

4.1. Implementation Details

  • TimeChat uses ViT-G/14 and LLaMA-2 (7B), tunes its Q-Formers and linear layer while freezing ViT and LLM, and trains for 3 epochs on TimeIT and Valley.Training uses batch size 32 on one 8-V100 (32G) machine; LoRA rank, window size, stride, and video tokens per window are 32, with 96 input frames.Source: §4.1 · p. 5

4.2. Evaluation Setups

  • The zero-shot evaluation covers dense captioning on YouCook2, temporal grounding on Charades-STA, and highlight detection on QVHighlights.Compared baselines include multi-model pipelines and end-to-end video LLMs; the reported table excludes transcribed speech.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3. Zero-shot performance

  • +9.2 F1, +2.8 CIDEr, and +1.0 SODA_c on YouCook2; +5.8 HIT@1 and +1.4 mAP on QVHighlights; +27.5 R@1 (IoU=0.5) over Valley on Charades-STA.TimeChat (7B) outperforms prior VidLLMs across all three zero-shot tasks; the YouCook2 gains are over the previous SOTA, and the QVHighlights gains are over previous VidLLMs.Source: Table 2 · p. 6, §4.3 · pp. 6–7

4.4. Qualitative Evaluation

  • Qualitative cases show improved temporal localization and summarization over prior models, with fewer hallucinations, though generated summaries still lack richness and detail.Appendix examples also demonstrate generalization to movie and egocentric video domains.Source: §4.4 · p. 7

4.5. Ablation Study

  • On YouCook2, removing the sliding video Q-Former lowers SODA_c by 1.0, CIDEr by 2.8, and F1 by 3.0; removing the timestamp-aware encoder lowers F1 by 2.3.Without the sliding Q-Former, visual tokens fall from 96 to 32, a 3× compression rate.Source: Table 3 · p. 6, §4.5 · p. 7

4.6. Further Analysis

  • When fine-tuned only on YouCook2, TimeChat exceeds prior models by +2.2 CIDEr and +3.9 F1 across metrics.The comparison is intended to test architectural gains apart from the TimeIT dataset.Source: Table 4 · p. 6, §4.6 · p. 7
  • As input frames increase, TimeChat’s performance improves linearly, while Video-LLaMA and VideoChat show minimal impact from increasing frames from 32 to 96.The sliding video Q-Former decouples the number of frames from the compression rate.Source: §4.6 · p. 8, Figure 5 · p. 8

4.7. Comparison with Specialized Models

  • After fine-tuning, TimeChat gains +6.9 F1 on YouCook2, +16.9 HIT@1 on QVHighlights, and +16.4 R@1 (IoU=0.5) on Charades-STA, but remains behind specialized models.The authors attribute specialized-model advantages to task-specific designs, extra fine-tuning, and, for some systems, additional pretraining or objectives.Source: §4.7 · p. 8, Table 5 · p. 8

5. Discussion and Conclusion

TimeChat combines time-aware frame encoding, a sliding video Q-Former, and TimeIT instruction tuning to support long-video understanding. It identifies significant events, pinpoints their start and end times, and produces concise summaries, while future work targets denser video semantics and broader time-related applications.

5. Discussion and Conclusion

  • TimeChat combines a time-aware frame encoder, a sliding video Q-Former, and TimeIT instruction tuning for long-video understanding.The authors identify these components as the basis for its temporal localization capabilities.Source: §5 · p. 8
  • TimeChat identifies significant events in lengthy videos, pinpoints their start and end times, and generates concise summaries.The authors describe these temporal localization capabilities as absent in previous VidLLMs.Source: §5 · p. 8
  • Future work will improve video semantic density while reducing spatial-temporal redundancy and collect more diverse, high-quality instruction data for broader time-related applications.Source: §5 · p. 8

A. Task Coverage in TimeIT · B. Instructions for Each Task

TimeIT covers six timestamp-related video tasks across 12 datasets from different domains, targeting prevalent real-world timestamp requests to AI assistants. Its instructions are manually initialized, diversified with GPT-4, then selected and refined by hand.

A. Task Coverage in TimeIT

  • TimeIT covers 6 timestamp-related video tasks and 12 datasets from different domains to address prevalent real-world timestamp requests to AI assistants.The dataset is designed to accommodate common user requests involving video timestamps.Source: A. Task Coverage in TimeIT · p. 12
  • DVC unifies event localization and captioning, returning timestamps and descriptions for events using ActivityNet Captions [15], ViTT [12], and YouCook2 [64].It supports narration of significant events in long videos.Source: A. Task Coverage in TimeIT · p. 12
  • TVG predicts start and end timestamps for a natural-language query using DiDeMo [10], QuerYD, HiRESTgrounding [59], and Charades-STA [7].The task supports accurate moment localization through natural-language interaction.Source: A. Task Coverage in TimeIT · p. 12
  • SLC segments and describes significant steps in long untrimmed instructional videos, using COIN [46] and HiRESTstep for cooking, repairing, and furniture-assembly scenarios.It is intended to detect key steps in noisy instructional videos.Source: A. Task Coverage in TimeIT · p. 12
  • Video summarization selects compressed frames or clip shots to represent informative content, using TVSum and SumMe to provide efficient overviews.The overviews are intended to save busy stakeholders time.Source: A. Task Coverage in TimeIT · p. 12
  • VHD identifies exciting, impressive, or emotional moments that may not cover the full video, and uses QVHighlights [16] to assess highlight recommendations.Unlike summarization, highlight detection need not represent the video’s full scope.Source: A. Task Coverage in TimeIT · p. 12
  • TSG predicts speech content and start/end timestamps from visual signals using YT-Temporal-1B [60], sampling 31.6K videos from its 18 million narrated videos.Following Vid2Seq [56], the authors use Whisper-timestamped [38] to transcribe speech automatically as the target answer; TSG is framed as weakly supervised event localization and description.Source: A. Task Coverage in TimeIT · p. 12

B. Instructions for Each Task

  • For each task, the authors manually draft instructions, use GPT-4 [35] to diversify expressions, then manually select and refine them into six instructions per task.This follows the M3IT observation that around five instructions per task suffice; Tab. 6 shows template examples and formatted answers.Source: B. Instructions for Each Task · p. 12

C. Contribution Analysis of Each Task to Model Performance · D. Hyper-parameters for Instruction Tuning

The analysis finds that all six TimeIT tasks benefit model performance, while instruction tuning uses window size and stride 32 for efficiency. Non-overlapping windows outperform overlapping ones, and a more thorough search may improve performance at size and stride 16.

C. Contribution Analysis of Each Task to Model Performance

D. Hyper-parameters for Instruction Tuning

E. Details of Evaluation Datasets and Metrics

TimeChat evaluates representative tasks from TimeIT’s three format-based groups: dense video captioning, highlight detection, and temporal video grounding. The datasets and metrics cover caption quality, event localization, and query-based moment retrieval.

E. Details of Evaluation Datasets and Metrics

  • YouCook2 [64] provides 1,790 untrimmed cooking videos for dense captioning, evaluated with CIDEr, SODA_c, and F1 across IoU thresholds 0.3, 0.5, 0.7, and 0.9.Videos average 320 seconds and 7.7 temporally localized imperative sentences; 1,333 videos are for training and 457 for validation.Source: E. Details of Evaluation Datasets and… · pp. 12–13
  • QVHighlights [16], with over 10,000 videos annotated with human-written text queries, is evaluated using mAP at IoU thresholds 0.5 and 0.75 and HIT@1.HIT@1 measures the hit ratio of the highest-scored clip.Source: E. Details of Evaluation Datasets and… · pp. 12–13
  • Charades-STA [7] supports temporal video grounding evaluation with 6,670 videos and 16,124 queries, using R@1 at an IoU threshold µ to measure retrieval of ground-truth moments.It includes 12,404 training pairs and 3,720 test pairs; videos average 30.59 seconds, and moments average 8.09 seconds.Source: E. Details of Evaluation Datasets and… · pp. 12–14

F. Details of Multi-model Pipelines

The multi-model pipelines use specialized visual models to turn video content into textual descriptions, then rely on ChatGPT to process the inputs and solve tasks. The section compares VideoChat-Text and InstructBLIP+ChatGPT, which differ in their frame sampling and visual-description approaches.

F. Details of Multi-model Pipelines

  • VideoChat-Text [19] and InstructBLIP [3]+ChatGPT serve as the multi-model pipeline baselines.Source: F. Details of Multi-model Pipelines · p. 14
  • Both pipelines convert video semantics into textual descriptions with specialized visual models, then use ChatGPT to process inputs and solve the target task.Source: F. Details of Multi-model Pipelines · p. 14
  • VideoChat-Text extracts key frames at FPS=1, uses visual tools to gather video information, and applies task-related prompts for timestamp-sensitive tasks.Its visual information includes action labels, frame summaries, video tags, descriptions, object positional coordinates, video narratives, timestamps, and segment details.Source: F. Details of Multi-model Pipelines · pp. 14–15
  • InstructBLIP [3]+ChatGPT [34] uses InstructBLIP to describe uniformly sampled 50-frame videos in exhaustive, semantically detailed paragraphs, with prompts for ChatGPT to solve each task.Source: F. Details of Multi-model Pipelines · p. 15

G. Generalized to New Domains

Figure 7 presents qualitative results showing TimeChat generalizes to novel movie [58] and egocentric video scenarios. The paper frames this generalization as important for a practical video assistant and as a difference from specialized models tailored to downstream datasets.

G. Generalized to New Domains

H. More Qualitative Results

Figures 8–10 present qualitative results for dense video captioning, temporal video grounding, and highlight detection; the paper describes TimeChat as proficient across these intricate temporal localization tasks.

H. More Qualitative Results

  • For dense video captioning, Figure 8 shows TimeChat identifying a series of events and outputting their timestamps and descriptions.Source: Figure 8 · p. 16
  • For temporal video grounding, Figure 9 shows predicted start and end timestamps for two specified queries alongside their corresponding segments.Source: Figure 9 · p. 17
  • For highlight detection, Figure 10 shows TimeChat identifying query-relevant moments, assigning saliency scores, and plotting predictions against ground truth over time.The ground-truth curve is orange and the model-prediction curve is blue.Source: Figure 10 · p. 18

Related papers

Loading 2312.02051v2…