PaperlayerView exact revision on arXiv

Source-linked AI summary

MovieChat: From Dense Token to Sparse Memory for Long Video Understanding

Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, Gaoang Wang

arXiv:2307.16449v4cs.CV

TL;DR

Long-video understanding lacks systems and standardized benchmarks that address its computation, memory, and temporal-connection challenges. MovieChat uses token-based short- and long-term memory with sliding-window visual processing, and reports state-of-the-art long-video performance alongside MovieChat-1K. The benchmark contains 1K videos and 14K manual annotations, while the method still has limited perception and imprecise temporal processing.

  • Problem

    Existing systems had not explored videos over one minute, and standardized benchmarks for long-video understanding were lacking despite challenges in computation complexity, memory cost, and long-term temporal connection.

  • Method

    MovieChat uses sliding-window visual features represented as tokens, sequential short-term memory, and consolidation into compact long-term memory before LLM interaction.

  • Results

    MovieChat achieves state-of-the-art long-video understanding performance and supports videos exceeding 10K frames; MovieChat-1K contains 1K videos with 14K manual annotations.

  • Takeaways & Limitations

    The proposed token-based memory mechanism enables a video understanding system designed for long videos and evaluated through a dedicated benchmark.

  • Takeaways & Limitations

    MovieChat is limited by the pretrained short-video model’s perception capacity and provides only rough event-duration proportions without precise temporal details.

Abstract

from arXiv · show

Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing systems can only handle videos with very few frames. For long videos, the computation complexity, memory cost, and long-term temporal connection impose additional challenges. Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose the MovieChat to overcome these challenges. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video and 14K manual annotations for validation of the effectiveness of our method.

1. Introduction

Existing multimodal video systems lacked long-video exploration and standardized evaluation. MovieChat addresses these gaps with a token-based memory mechanism, support for >10K-frame understanding, and the MovieChat-1K benchmark.

  • Under 224 × 224 visual-only inference without frame sampling, MovieChat handles >10K frames on a 24GB graphics card, compared with around 100 frames for the previous method.The reported average VRAM increase per frame is 21.3KB for MovieChat versus ∼200MB for other methods.
  • Existing systems had not explored videos longer than one minute, and standardized benchmarks for long-video understanding were lacking.
  • MovieChat integrates vision models and LLMs for long-video understanding using rapidly updated short-term and compact long-term memories.Video features are extracted with a sliding window, converted to tokens, and consolidated from short-term into long-term memory when the buffer reaches its limit.
  • MovieChat is designed to support long-video understanding beyond 10K frames.
  • Its memory management mechanism reduces computation complexity and memory cost while enhancing long-term temporal connection.

2. Related Works

Related work spans multimodal language models, long-video representations and benchmarks, and memory models for video tasks. MovieChat builds on these directions by reducing visual-token redundancy and passing information across a large temporal range.

  • Multimodal large language models combine language models with visual components to support perception, commonsense reasoning, code reasoning, and interactive multimodal understanding.
  • Prior long-video methods use feature banks, motion representations, or efficient attention, while long-form datasets remain challenging and rarely explored.
  • Earlier video benchmarks often target grounding, event-boundary detection, metadata prediction, or captioning rather than long-term question-answering understanding.
  • MovieChat uses global mode with long-term memory and breakpoint mode combining long-term and current short-term memory for moment-specific understanding.
  • Video memory models have been applied to segmentation, tracking, and action understanding, including systems handling thousands of frames.
  • MovieChat focuses on reducing redundant visual tokens and building memory to pass information across a large temporal range.

3. MovieChat

MovieChat combines frame-wise visual features with short- and long-term memory to support understanding of ultra-long videos. Its memory consolidation and inference modes reduce redundant representations while preserving temporal context for global and moment-specific questions.

  • 3. MovieChat: MovieChat targets ultra-long video understanding beyond 10K frames using visual features, memory modules, projection, and an LLM.A sliding-window approach avoids concurrently storing vast numbers of frames in GPU memory and RAM.
  • 3.2. Visual Feature Extraction: Frame-wise visual tokens are extracted with pretrained EVA-CLIP ViT-G/14 and BLIP-2 Q-former models in sliding windows.Each frame produces N visual tokens of feature dimension D, grouped into clips spanning C frames.
  • 3.3. Short-term Memory: Short-term memory stores a fixed-length buffer of K frames and updates through a FIFO queue as new visual-token batches arrive.When full, stored frames enter memory consolidation, and the resulting feature reinitializes short-term memory while augmenting long-term memory.
  • 3.4. Long-term Memory: Memory consolidation greedily merges adjacent frames with the highest similarity until the token count reaches RL, transforming dense tokens into sparse long-term memories.The output feature has shape v′ ∈ ZRL×3×H×W; RL controls the performance-efficiency trade-off, and the parameter-free procedure adds negligible overhead relative to savings.
  • 3.4. Long-term Memory: MovieChat extends positional encoding from length n to n^2 and offers global and breakpoint inference modes for whole-video or moment-specific understanding.Breakpoint mode aggregates long-term memory, short-term memory, and the current frame feature, while global mode uses long-term memory as the video representation.
  • 3.5. Inference: The video representation passes through a Q-former and linear projection before reaching the LLM, mapping visual features into text space for answering questions.The formulation identifies Q as the question, A as the answer or instruction, and P as the visual-to-text projection.

4. A New Benchmark: MovieChat-1K

MovieChat-1K is a benchmark for long-video comprehension built from diverse movie and television clips with dense captions and question-answer annotations. Its evaluation covers global and breakpoint question answering, while supporting short-video comparisons through reported quantitative tables.

  • 4. A New Benchmark: MovieChat-1K: MovieChat-1K addresses the lack of established long-video question-answering benchmarks with 1K high-quality clips sourced from movies and television series.The benchmark was created to evaluate MovieChat on long-video understanding tasks.
  • 4. A New Benchmark: MovieChat-1K: The benchmark spans 15 popular video categories, and over 90% of videos contain 10K–12K frames.Another 14.6% extend beyond 12K frames, while 8.6% are shorter than 10K frames.
  • 4. A New Benchmark: MovieChat-1K: Each video includes one dense whole-video caption, three global question-answer pairs, and ten timestamped breakpoint question-answer pairs.Most questions are open-ended, with approximately one quarter classified as multiple-choice.
  • 4. A New Benchmark: MovieChat-1K: Table 1 reports short-video question-answering results with GPT-3.5, where MovieChat is described as achieving comparable performance despite its long-video focus.The table marks the best result in bold and the second-best result with underlining.
  • 4. A New Benchmark: MovieChat-1K: Table 2 reports short-video generation performance with GPT-3.5 across correctness, detail, contextual, temporal, and consistency dimensions.The caption defines CI, DO, CU, TU, and CO as the five evaluation dimensions.

5. Experiments

Experiments evaluate MovieChat on short- and long-video question answering and generation, compare it with prior methods, and analyze memory and hyperparameter choices. MovieChat maintains or improves performance while processing longer videos, with memory mechanisms and consolidation choices affecting results.

  • Short video question-answering: MovieChat achieves comparable performance to previous methods on short video question answering despite not being specifically designed for that task.
  • Long video question-answering: MovieChat’s long-video question-answering performance gains persist in global and breakpoint modes across LLM-assisted and human evaluations, including open-ended and true-false questions.The method reads more video frames than the compared baselines.
  • Long video generative performance: MovieChat generates higher-quality answers than previous methods as long-video content becomes more extensive.Evaluation averages scores from GPT-3.5, Claude, and human blind rating.
  • Memory ablation: The memory-enabled variant significantly outperforms the memory-independent variant on long-video question answering and generation.The ablation evaluates both tasks using GPT-3.5, Claude, and human blind rating.
  • Hyper-parameter ablations: Performance initially rises and then declines as short- and long-term memory buffers grow, because larger buffers expand information while increased compression loses detail.Merged-token initialization outperforms using the last few tokens or uniform sampling, and merged-token length interacts with memory-buffer size.
  • Qualitative evaluation: MovieChat provides responses about both the current moment and entire videos over 10K frames with less hallucination in the reported case study.The case study covers breakpoint and global question modes on open-ended long videos.

6. Limitation

MovieChat remains an early-stage prototype with limitations in visual perception and temporal precision.

  • MovieChat’s perception capacity is limited by the pretrained short-video understanding model, and its time processing provides only rough event-duration proportions.The paper states that precise temporal details are lacking.

7. Conclusion

MovieChat integrates video foundation models and LLMs with token-based memory to address long-video analysis challenges. The paper reports state-of-the-art long-video understanding performance compared with systems limited to few-frame videos.

  • MovieChat integrates video foundation models and large language models into a video understanding system using Transformer tokens as memory carriers.
  • MovieChat is reported to achieve state-of-the-art performance in long video understanding while surpassing systems limited to videos with few frames.

Understanding

The supplementary material documents MovieChat’s memory consolidation, benchmark statistics, evaluation procedures, hyperparameters, filtering strategy, category analyses, and additional understanding examples.

  • Supplementary contents: The supplementary material presents a schematic diagram of MovieChat’s memory consolidation algorithm.
  • Supplementary contents: It provides detailed statistical information for the MovieChat-1K benchmark and evaluates question categories and score-method correlations.
  • Supplementary contents: It documents the prompt template, hyperparameter settings, and LLM-assisted evaluation procedures used in the experiments.
  • Supplementary contents: It compares results using a different LLM and introduces manual filtering to reduce the impact of inconsistent LLM-assisted judgments.
  • Supplementary contents: It includes additional quantitative results for breakpoint-mode generation, hyperparameter analyses, and examples of scene and temporal understanding.
  • Memory consolidation: During consolidation, adjacent frames with the greatest similarity are merged repeatedly until short-term memory reaches its predefined size.

B. MovieChat-1K Statistics Information

MovieChat-1K is introduced as a long-video understanding benchmark containing diverse videos, dense captions, and visual question-answer pairs. Its statistics characterize question, answer, caption, category, and evaluation-format distributions.

  • Dataset overview: The benchmark includes videos from 15 popular categories, with every video comprising multiple alternating scenes.
  • Dataset overview: MovieChat-1K contains 1K long videos, 1K dense captions, and 13K visual question-answer pairs.
  • Question-answer statistics: Questions mostly contain 5–15 words, while answers generally contain fewer than 10 words across global and breakpoint modes.
  • Caption statistics: Approximately two-thirds of segmented clips have captions of 100–149 words, one-fifth have fewer than 100 words, and about 11% exceed 150 words.
  • Caption statistics: Caption vocabulary covers objects, attributes, locations, scenes, and actions or events, while MovieChat-1K contains 102,988 unique verb instances versus 109,485 in WebVid10M.
  • Benchmark comparison: MovieChat-1K is presented as having the longest average duration for movie clips among the compared benchmarks.

F. Ablation study on large language models.

This ablation examines how the selected large language model affects MovieChat’s long-video evaluation. Llama generally outperforms Llama2 under the reported evaluation conditions, while the study also discusses evaluator reliability and long-video question types.

  • LLM comparison: MovieChat with Llama provides answers more aligned with video content than the Llama2 version in the illustrated cooking example.
  • LLM comparison: The memory buffer enables dense frame sampling, allowing Llama2 to estimate scene-time proportions from adjacent similar frames despite lacking specific time information.
  • Evaluation reliability: GPT-3.5 evaluations can produce inconsistent yes/no judgments and scores, such as yes with 0 or no with 5.
  • Evaluation reliability: Manual screening retains evaluation outcomes whose judgments and scores are consistent, and the reported results outperform baselines across long-video question types and breakpoint-mode generation.

J. Pearson correlation coefficient of different score methods.

The paper uses Pearson correlation analysis to compare GPT-3.5, Claude, and human blind rating as evaluation methods. The reported results indicate substantial agreement among these methods and support the reliability of the assessment.

  • Analysis procedure: The analysis computes correlations between GPT-3.5, Claude, and human blind rating using mean scores across score dimensions for MovieChat and prior methods.
  • Correlation measure: Pearson correlation coefficients quantify the linear relationship between pairs of evaluation methods, ranging from -1 to +1.
  • Findings: The reported correlations indicate substantial agreement among the three evaluation methods.
  • Findings: The paper states that score alignment across evaluation methods strengthens the reliability of its assessment.

L. Analysis on hyperparameter ablations.

The hyperparameter ablations show that MovieChat’s performance depends on balancing memory capacity, compression, and initialization. More retained information can help, but excessive compression or memory expansion can reduce performance, while merged-token initialization performs best.

  • Memory-buffer lengths: Increasing short- and long-term memory lengths first improves and then reduces performance.Larger buffers acquire more information, but compression can lose detail while merged-token length remains constant.
  • Memory-buffer lengths: The ablations identify a trade-off between dense information extraction and information compression in long-video understanding.
  • Short-term memory initialization: Merged-token initialization produces the best performance compared with using the last few tokens or uniform sampling.
  • Short-term memory initialization: Using only the last few tokens can yield repetitive or incoherent merged tokens, while uniform sampling captures representative frames but remains inferior to merged-token initialization.
  • Long-video case study: MovieChat provides responses to questions about both the current moment and entire video content for long videos exceeding 10K frames.
Loading 2307.16449v4…