PaperlayerView exact revision on arXiv

Source-linked AI summary

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, Lu Hou

arXiv:2312.02051v2cs.CVcs.AIcs.CL

TL;DR

Long-form video assistants need to identify meaningful events and connect them to accurate timestamps, while existing Video LLMs struggle with this requirement. TimeChat addresses the gap with timestamp-aware frame encoding, sliding-window temporal compression, and timestamp-focused instruction tuning, achieving strong zero-shot results across temporal video tasks. The model is presented as a step toward a versatile video assistant, while leaving state-of-the-art performance on every task for future work.

  • Problem

    Existing Video LLMs capture global semantics for short clips but struggle to associate significant content in untrimmed videos with accurate timestamps.

  • Method

    TimeChat uses a time-aware frame encoder, a sliding video Q-Former, and the 6-task, 125K-instance TimeIT instruction-tuning dataset.

  • Results

    +27.5 R@1 (IoU=0.5) on Charades-STA, alongside improvements on YouCook2 and QVHighlights, demonstrates strong zero-shot temporal localization across tasks.

  • Takeaways & Limitations

    TimeChat identifies significant events, pinpoints their start and end times, and generates concise summaries toward a versatile video assistant.

  • Takeaways & Limitations

    Achieving state-of-the-art performance on every task is not the paper’s major goal and remains future work.

Abstract

from arXiv · show

This work proposes TimeChat, a time-sensitive multimodal large language model specifically designed for long video understanding. Our model incorporates two key architectural contributions: (1) a timestamp-aware frame encoder that binds visual content with the timestamp of each frame, and (2) a sliding video Q-Former that produces a video token sequence of varying lengths to accommodate videos of various durations. Additionally, we construct an instruction-tuning dataset, encompassing 6 tasks and a total of 125K instances, to further enhance TimeChat's instruction-following performance. Experiment results across various video understanding tasks, such as dense captioning, temporal grounding, and highlight detection, demonstrate TimeChat's strong zero-shot temporal localization and reasoning capabilities. For example, it achieves +9.2 F1 score and +2.8 CIDEr on YouCook2, +5.8 HIT@1 on QVHighlights, and +27.5 R@1 (IoU=0.5) on Charades-STA, compared to state-of-the-art video large language models, holding the potential to serve as a versatile video assistant for long-form video comprehension tasks and satisfy realistic user requirements.

1. Introduction

TimeChat addresses the difficulty of understanding lengthy videos with accurate temporal associations by combining time-aware video representation, adaptive temporal compression, and timestamp-focused instruction tuning. It outperforms previous video language models across zero-shot captioning, grounding, and highlight detection.

  • Motivation: Existing Video LLMs capture global semantics for short clips but struggle to associate meaningful events in untrimmed videos with accurate timestamps.This limits their performance on temporal localization and description tasks.
  • Proposed approach: TimeChat combines a time-aware frame encoder with a sliding video Q-Former for long-video understanding and accurate temporal localization.The frame encoder binds visual context to each frame’s timestamp, while the Q-Former creates varying-length video representations for different video durations.
  • Instruction tuning: TimeIT contains 6 tasks, 12 academic benchmarks, and 125K instruction-tuning instances derived from timestamp-associated long-video datasets.The data are reformatted into dialog-style examples with manually written instructions.
  • Experimental results: +9.2 F1 score and +2.8 CIDEr on YouCook2, compared with previous VidLLMs.These results are reported under zero-shot evaluation settings.

2. Related Work

Prior work combines video encoders with language models and uses instruction tuning, but generally neglects time-aware video requests and often treats temporal tasks separately. TimeChat frames these tasks in a language-modeling paradigm with a unified time-sensitive architecture.

  • Video-language models: Video-language models differ mainly in how they encode video into vision tokens compatible with open-source large language models.Representative systems combine video encoders with Query Transformers to compress video features.
  • Vision-language instruction tuning: Vision-language instruction tuning converts multimodal benchmark data or LLM-generated descriptions into instruction-format conversations.These approaches improve instruction following but focus on general multimodal interaction.
  • Vision-language instruction tuning: Existing instruction-tuning approaches neglect time-aware user requests for video understanding.The paper positions timestamp-focused instruction tuning as a response to this gap.
  • Video temporal localization: Temporal video tasks require explicit associations between video semantics and corresponding timestamps, especially in untrimmed long videos.These tasks include grounding, dense captioning, summarization, highlight detection, and step localization.
  • Video temporal localization: TimeChat unifies a wide range of time-sensitive video tasks in language-modeling format as an initial LLM-based generalist approach.Its architecture processes timestamped frames, models temporal relations, and generates responses through an LLM.

3. Method

TimeChat combines timestamp-aware visual encoding with sliding-window temporal feature fusion, then uses TimeIT to support instruction following across timestamp-related video tasks.

  • Model Architecture: TimeChat combines a timestamp-aware frame encoder, a sliding video Q-Former, and a large language model for long-video understanding.The frame encoder extracts visual and timestamp features independently, the Q-Former models temporal relations, and the language model generates responses.
  • Model Architecture: Existing Video LLMs often capture global semantics in short clips but fail to associate meaningful events with accurate timestamps.Separately modeling visual semantics and timestamps can identify frame order without precisely binding visual events to their occurrence times.
  • Model Architecture: The time-aware frame encoder conditions the Q-Former on each frame’s timestamp to bind temporal information directly to visual tokens.It first encodes frames with a pretrained image encoder, compresses frame tokens with an image Q-Former, and adds timestamp descriptions during extraction.
  • Model Architecture: The sliding video Q-Former compresses frames within moving windows into video tokens, producing variable-length sequences for videos of different durations.Its stride controls the final token count, while a linear layer maps token dimensions into the LLM embedding space.
  • Sliding Video Q-Former: Fixed-token approaches can excessively compress long videos, whereas the sliding design makes the compression rate constant and retains richer semantics.The stride S controls the number of final video tokens according to the computation budget.
  • TimeIT Dataset: TimeIT reformats timestamp-associated video datasets into instruction-following data spanning six tasks and twelve datasets, with 125K instances.The construction includes instruction writing and answer formatting, and covers dense captioning, grounding, step localization and captioning, summarization, highlight detection, and speech generation.

4. Experiments

TimeChat is evaluated on long-video understanding tasks in zero-shot and fine-tuned settings, with comparisons spanning general-purpose VidLLMs, pipelines, and specialized models. Results show strong temporal localization and broad-domain generalization, while specialized models retain advantages after task-specific fine-tuning.

  • Zero-shot performance: TimeChat (7B) outperforms previous VidLLMs (7B/13B) across dense captioning, highlight detection, and temporal grounding in zero-shot evaluation.The evaluation uses YouCook2, QVHighlights, and Charades-STA; higher metrics indicate better performance.
  • Zero-shot performance: +9.2 F1 score, +2.8 CIDEr, and +1.0 SODA_c over the previous SOTA demonstrate gains on YouCook2 dense video captioning.The task requires localizing approximately 8 cooking steps and describing them faithfully in videos averaging 320 seconds.
  • Zero-shot performance: +5.8 HIT@1 points over previous VidLLMs gives TimeChat 23.9 HIT@1 on QVHighlights.The task evaluates frame-level highlight detection; the authors associate the result with timestamp-aware frame representations.
  • Zero-shot performance: +27.5 R@1 (IoU=0.5) over Valley gives TimeChat 32.2 points on Charades-STA temporal grounding.This task identifies the timestamp corresponding to a query sentence.
  • Qualitative Evaluation: Qualitative comparisons report improved temporal localization, video-content matching, and reduced hallucinations, although generated summaries remain less rich and detailed.The comparisons are conducted in zero-shot settings against Video-LLaMA and VideoChat.

5. Discussion and Conclusion

TimeChat combines time-aware video modeling and TimeIT instruction tuning to support long-video understanding with temporal localization and concise summarization. The authors identify future work in improving semantic density, reducing redundancy, and broadening time-related applications.

  • 5. Discussion and Conclusion: TimeChat integrates a time-aware frame encoder, sliding video Q-Former, and TimeIT instruction tuning for long-video understanding.These components support identifying significant events, pinpointing their start and end times, and generating concise summaries.
  • 5. Discussion and Conclusion: TimeChat demonstrates temporal localization capabilities that previous Video LLMs lacked.The conclusion attributes this capability to the model’s architecture and instruction tuning.
  • 5. Discussion and Conclusion: TimeChat identifies significant events in lengthy videos, pinpoints their start and end times, and generates concise summaries.
  • 5. Discussion and Conclusion: Future work targets higher video semantic density, lower spatial-temporal redundancy, and more diverse, higher-quality instruction-tuning data.The stated goal is to broaden time-related applications.

A. Task Coverage in TimeIT

TimeIT covers six timestamp-related video tasks across twelve datasets and is designed around prevalent timestamp-oriented requests for video assistants. Its instructions combine manual design, GPT-4 diversification, and manual refinement.

  • A. Task Coverage in TimeIT: TimeIT covers 6 timestamp-related video tasks and incorporates 12 datasets from different domains.
  • A. Task Coverage in TimeIT: Dense Video Captioning detects multiple events and outputs their timestamps and descriptions.It uses ActivityNet Captions, ViTT, and YouCook2.
  • A. Task Coverage in TimeIT: Temporal Video Grounding predicts a queried video moment’s start and end times from a natural-language query.The task includes DiDeMo, QuerYD, HiRESTgrounding, and Charades-STA.
  • A. Task Coverage in TimeIT: Step Localization and Captioning segments and describes significant steps in long untrimmed instructional videos.Its scenarios include cooking, repairing, and furniture assembly.
  • A. Task Coverage in TimeIT: Video Summarization selects a compressed set of frames or shots representing informative video content.TVSum and SumMe support efficient video overviews.
  • A. Task Coverage in TimeIT: Video Highlight Detection identifies exciting, impressive, or emotional moments rather than covering the full video scope.QVHighlights evaluates highlight recommendation.
  • A. Task Coverage in TimeIT: Transcribed Speech Generation predicts speech content and its start and end timestamps from visual signals.The task uses 31.6K sampled videos from YT-Temporal-1B for instruction tuning.
  • A. Task Coverage in TimeIT: TimeIT accommodates prevalent real-world requests involving video timestamps when users interact with AI assistants.

C. Contribution Analysis of Each Task to Model Performance

The contribution analysis finds that adding tasks to TimeIT generally improves model performance, with especially positive effects from related tasks. The sliding-window configuration is chosen for efficiency, while a more thorough search may improve performance.

  • C. Contribution Analysis of Each Task to Model Performance: Adding similar tasks to TimeIT improves related performance; adding SLC to DVC raises YouCook2 F1 from 5.9 to 12.1.
  • C. Contribution Analysis of Each Task to Model Performance: All 6 TimeIT tasks are reported as beneficial overall.
  • C. Contribution Analysis of Each Task to Model Performance: A window size=stride=32 is adopted for efficiency because it provides a higher compression rate and fewer video tokens.
  • C. Contribution Analysis of Each Task to Model Performance: A more thorough sliding-window search using window size=stride=16 may improve performance.
  • C. Contribution Analysis of Each Task to Model Performance: Non-overlapping sliding windows outperform overlapping windows in the reported ablation.

E. Details of Evaluation Datasets and Metrics

Evaluation selects representative tasks from TimeIT’s format-based groups and measures dense captioning, highlight detection, and temporal grounding on established video datasets. The section also documents instruction templates, frame sampling, and task-specific metrics.

  • E. Details of Evaluation Datasets and Metrics: TimeIT groups tasks into dense-captioning, highlight-detection, and temporal-grounding formats, evaluating DVC, HD, and TVG as representative tasks.
  • E. Details of Evaluation Datasets and Metrics: YouCook2 contains 1,790 untrimmed cooking videos averaging 320 seconds, with 7.7 temporally localized imperative sentences per video.The split contains 1,333 training videos and 457 validation videos.
  • E. Details of Evaluation Datasets and Metrics: Dense captioning is evaluated with CIDEr, SODA_c, and F1 across IoU thresholds of 0.3, 0.5, 0.7, and 0.9.
  • E. Details of Evaluation Datasets and Metrics: QVHighlights contains over 10,000 videos with human-written text queries, while highlight detection uses mAP at IoU 0.5 and 0.75 and HIT@1.
  • E. Details of Evaluation Datasets and Metrics: Charades-STA contains 6,670 videos and 16,124 queries, evaluated with R@1 at specified IoU thresholds for timestamp retrieval.The average video duration is 30.59 seconds and the average moment duration is 8.09 seconds.
  • E. Details of Evaluation Datasets and Metrics: Instruction templates define task goals and guidelines, while prompts provide frame descriptions with timestamps; videos are uniformly sampled at 50 frames.Table 6 presents template examples and formatted outputs for each task.

F. Details of Multi-model Pipelines

The multi-model pipelines combine specialized visual processing with ChatGPT to solve timestamp-sensitive video tasks. VideoChat-Text extracts key frames and structured video information, while InstructBLIP+ChatGPT uses exhaustive frame descriptions.

  • VideoChat-Text and InstructBLIP+ChatGPT integrate specialized visual models with ChatGPT through textual video descriptions.These pipelines first convert video semantics into text, then use ChatGPT to process the inputs for the target task.
  • VideoChat-Text samples key frames at FPS=1 and obtains labels, summaries, tags, descriptions, coordinates, narratives, timestamps, and segment details.
  • A qualitative movie example asks for the interval when a person stares at a clown puppet and reports 225.6–231.6 seconds.
  • Task-related prompts are designed to give VideoChat-Text the capability to solve timestamp-sensitive tasks.
  • InstructBLIP+ChatGPT uniformly samples 50 frames and uses exhaustive paragraph descriptions plus task-specific prompts.

G. Generalized to New Domains

TimeChat is evaluated qualitatively on movie and egocentric videos, demonstrating generalization to novel scenarios. The authors identify this generalization as important for a practical video assistant.

  • TimeChat demonstrates qualitative generalization to new domains including movies and egocentric videos.
  • The authors describe generalization to novel scenarios as a key characteristic for a practical video assistant.

H. More Qualitative Results

Figures 8–10 provide qualitative results across dense captioning, temporal grounding, and highlight detection. Together, these results show TimeChat performing diverse temporal localization tasks.

  • Figures 8–10 cover qualitative results for dense video captioning, temporal video grounding, and video highlight detection.
  • Overall, the qualitative results demonstrate proficiency across diverse intricate temporal localization tasks.
  • The dense captioning results ask TimeChat to detect event series and output corresponding timestamps and descriptions.
  • The temporal grounding results ask TimeChat to estimate starting and ending timestamps for specified queries.
  • The highlight detection results plot predicted and ground-truth saliency scores against timestamps, using blue for predictions and orange for ground truth.
Loading 2312.02051v2…