PaperlayerView exact revision on arXiv

Source-linked AI summary

MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding

Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, Ser-Nam Lim

arXiv:2404.05726v2cs.CV

TL;DR

Existing multimodal models struggle to process long videos because framewise tokens exceed LLM context lengths and GPU memory budgets. MA-LMM processes frames online while storing historical information in a memory bank, and reports state-of-the-art results across multiple video tasks. Its main scope boundary is that online processing trades lower memory use for longer processing time on extremely long videos.

  • Problem

    Prior large multimodal models can process only limited video frames because LLM context lengths and GPU memory constrain framewise inputs.

  • Method

    MA-LMM processes video frames sequentially and stores historical visual features in a long-term memory bank that the Q-Former can reference.

  • Results

    MA-LMM achieves state-of-the-art performance across long-term video understanding, video question answering, and video captioning tasks.

  • Takeaways & Limitations

    The memory bank provides a plug-and-play way to add long-term video modeling to existing multimodal models while addressing context-length and GPU-memory constraints.

  • Takeaways & Limitations

    Online processing reduces GPU memory usage but increases video processing time, especially for extremely long videos.

Abstract

from arXiv · show

With the success of large language models (LLMs), integrating the vision model into LLMs to build vision-language foundation models has gained much more interest recently. However, existing LLM-based large multimodal models (e.g., Video-LLaMA, VideoChat) can only take in a limited number of frames for short video understanding. In this study, we mainly focus on designing an efficient and effective model for long-term video understanding. Instead of trying to process more frames simultaneously like most existing work, we propose to process videos in an online manner and store past video information in a memory bank. This allows our model to reference historical video content for long-term analysis without exceeding LLMs' context length constraints or GPU memory limits. Our memory bank can be seamlessly integrated into current multimodal LLMs in an off-the-shelf manner. We conduct extensive experiments on various video understanding tasks, such as long-video understanding, video question answering, and video captioning, and our model can achieve state-of-the-art performances across multiple datasets. Code available at https://boheumd.github.io/MA-LMM/.

1. Introduction

Existing multimodal models are constrained by LLM context length and GPU memory when processing long videos, while simpler temporal pooling or added video modules have important drawbacks. MA-LMM addresses this with online processing and a long-term memory bank, achieving strong results across video understanding tasks.

  • Motivation: LLM context limits and GPU memory costs restrict prior multimodal models to relatively few video frames.LLaMA has a context length of 2048, while LLaVA and BLIP-2 use 256 and 32 tokens per image, respectively.
  • Motivation: Average temporal pooling reduces input size but performs worse because it lacks explicit temporal modeling.
  • Motivation: Video-LLaMA adds a video Q-Former to model temporal dynamics, but increases model complexity and training parameters and is unsuitable for online analysis.
  • MA-LMM: MA-LMM sequentially processes video frames and stores historical features in a long-term memory bank for later reference.This online strategy reduces GPU memory usage and addresses LLM context-length constraints for long video sequences.
  • MA-LMM: The memory bank aggregates past features auto-regressively, serves as key and value in Q-Former attention, and can integrate with existing multimodal models.A compression method selects and averages similar adjacent frame features to keep memory length constant and reduce temporal redundancy.
  • Results: MA-LMM reports new state-of-the-art performance on long-term video understanding, video question answering, and video captioning tasks.

2. Related Work

Prior video-language models extend image-language architectures or add temporal modeling, but are primarily designed for short videos. Long-term video methods instead reduce computation through pre-extracted features, sparse sampling, or specialized temporal models.

  • Image-language models: Image-language models connect pretrained visual encoders with language models, with BLIP-2 using a lightweight Q-Former to bridge their embedding spaces.
  • Video-language models: Video inputs are often flattened into one-dimensional spatio-temporal feature sequences, which does not effectively capture video temporal dynamics.
  • Video-language models: Video-LLaMA adds a video Q-Former, Video-ChatGPT uses spatial-temporal average pooling, and VideoChat forwards perception annotations to an LLM.
  • Video-language models: These video-language models are primarily designed for short videos, motivating temporal token reduction through averaging highly similar features.
  • Long-term video models: Long-term video methods commonly use pre-extracted features, sparse frame sampling, or streamlined transformer decoders to reduce computational demands.

3. Method

MA-LMM processes video frames online while maintaining visual and query memory banks for long-term temporal modeling. It compresses redundant historical features and reduces the sequence passed to the LLM.

  • Overview: MA-LMM processes video frames autoregressively through visual encoding, temporal modeling, querying, and frozen-LLM decoding.The framework uses a pretrained visual encoder, Q-Former, long-term memory modeling, and text decoding.
  • Memory Banks: The visual memory bank stores concatenated raw features from past frames and supplies them as keys and values to Q-Former cross-attention.This lets the Q-Former attend explicitly to historical visual information across processing steps.
  • Memory Banks: The query memory bank stores timestep-specific Q-Former queries, preserving the model’s evolving understanding of the video for self-attention.Unlike static visual features, queries evolve through cascaded Q-Former blocks and each self-attention layer has its own query memory bank.
  • Memory Compression: When memory exceeds threshold M, Memory Bank Compression merges temporally redundant adjacent features to retain earlier historical information.The method selects highly similar adjacent tokens, averages the selected features spatially, and applies the same procedure to the query memory bank.
  • LLM Interface: Autoregressive processing reduces the LLM input from N ∗T text tokens to N, easing context-length and GPU-memory constraints.The final Q-Former output contains historical information accumulated across timesteps; training updates Q-Former parameters while freezing the visual encoder and language model.

4. Experiments

MA-LMM is evaluated across long-term video understanding, video question answering, video captioning, and online action prediction, with strong results across the reported benchmarks. Ablations and off-the-shelf evaluations further examine the memory banks, memory length, model compatibility, and online reasoning behavior.

  • Long-term Video Understanding: MA-LMM improves LVU average top-1 accuracy by 3.8% over S5 while outperforming prior long-term video models on content understanding and metadata prediction.It processes frames online and directly outputs text labels for the classification tasks.
  • Long-term Video Understanding: 2.3% and 2.4% top-1 accuracy gains over S5 are reported on Breakfast and COIN, respectively.Both datasets evaluate challenging long-term video activity classification.
  • Video Question Answering: MA-LMM achieves state-of-the-art performance on MSRVTT and MSVD question answering, but falls short of VideoCoCa on ActivityNet.The authors attribute VideoCoCa’s advantage on ActivityNet to its large-scale video-text pre-training, whereas MA-LMM uses image-text pre-training.
  • Video Captioning: MA-LMM ranks among the top two methods across video captioning metrics and surpasses Video-LLaMA on the evaluated captioning and question-answering datasets.Captioning is evaluated with METEOR and CIDEr on MSRVTT, MSVD, and Youcook2.
  • Ablation Studies: Combining visual and query memory banks boosts performance by 14.7%, 18.4%, and 20.9% on LVU, Breakfast, and COIN, respectively.The ablation finds that either memory bank improves performance, the visual memory bank is stronger individually, and the two are complementary.

5. Conclusion

MA-LMM augments multimodal models with a long-term memory bank for efficient long-term video modeling. It processes frames sequentially, stores historical data, and addresses context-length and GPU-memory constraints.

  • 5. Conclusion: MA-LMM introduces a long-term memory bank that augments existing large multimodal models for long-term video modeling.The module is designed as a plug-and-play component for current multimodal models.
  • 5. Conclusion: Sequential frame processing stores historical video data in the memory bank while addressing LLM context-length and GPU-memory constraints.
  • 5. Conclusion: The memory bank can be integrated into existing large multimodal models in an off-the-shelf manner.
  • 5. Conclusion: Experiments across various tasks demonstrate the superior advantages of the proposed method.

Appendix

The appendix contains additional experiments, qualitative captioning results, related-work discussion, implementation details, and limitations with future directions.

  • Appendix: Additional ablation experiments are presented in Section A.
  • Appendix: Further qualitative results for video captioning are provided in Section B.
  • Appendix: The appendix also covers concurrent works, dataset-specific implementation details, hyper-parameters, limitations, and future research directions.

A. Additional Experiments

Additional experiments examine memory-bank compression and inference-time scaling. MA-LMM’s inference time grows linearly with frame length, unlike simultaneous feature concatenation, which takes longer and consumes more GPU memory.

  • A. Additional Experiments: Memory-bank compression is compared at frame-level and token-level spatial granularities on LVU, Breakfast, and COIN.Frame-level compression averages adjacent frame features with the highest cosine similarity; token-level compression compares same-position tokens.
  • A. Additional Experiments: MA-LMM inference time increases linearly with video frame length because it processes frames sequentially in an auto-regressive design.
  • A. Additional Experiments: Directly concatenating frame-level features takes longer and consumes more GPU memory because all video frames are processed simultaneously.

B. More Qualitative Results

Additional qualitative results show that MA-LMM produces more detailed and accurate video captions than Video-LLaMA in the illustrated examples. The section also includes Figure 6 and Table 10 as supporting visual materials.

  • B. More Qualitative Results: MA-LMM significantly outperforms Video-LLaMA in generating detailed and accurate sentence descriptions.
  • B. More Qualitative Results: In one example, MA-LMM describes removing onion rings and placing them on a paper towel while capturing the complete action sequence.
  • B. More Qualitative Results: Video-LLaMA misses the powder detail in the illustrated example, whereas MA-LMM recognizes and describes fine-grained details.
  • B. More Qualitative Results: Table 10 compares memory-bank compression at different spatial levels.
  • B. More Qualitative Results: Figure 6 presents inference time as a function of input frame length.

C. Relations to Concurrent Works

Concurrent methods differ in temporal modeling, token merging, model foundations, and memory-bank design. MA-LMM performs fine-grained temporal token merging and adds a dynamic query memory bank for evolving past-video representations.

  • Temporal Modeling: Temporal modeling ranges from implicit LLM sequence processing to global self-attention and segmented video processing.Chat-UniVi lacks explicit temporal modeling, whereas TESTA and MovieChat use global self-attention in different ways.
  • Token Merging Application: Concurrent methods reduce video redundancy through token merging but differ in how they aggregate spatial and temporal information.TESTA uses cascaded aggregation, while Chat-UniVi operates in parallel; all methods modify the core idea of merging similar tokens.
  • Token Merging Application: MA-LMM condenses video length through temporal token merging at a fine-grained spatial level.It independently compresses visual and query tokens across different spatial areas.
  • Based Model: The compared approaches use different multimodal foundations, including video-based encoders and image-based adaptations.TESTA uses TimeSFormer, MovieChat uses Video-LLaMA, and Chat-UniVi adapts LLaVA; the passage does not complete the Chat-UniVi description.
  • Memory Bank Design: MA-LMM differs from MovieChat by adding a query memory bank that captures dynamic memory from the evolving understanding of past frames.MovieChat primarily consolidates raw and static visual features, while MA-LMM adds dynamic query memory.

D. Experiment Details

The experiments build MA-LMM on InstructBlip and use cosine learning-rate decay, with dataset-specific frame sampling for long-term video understanding.

  • Implementation: MA-LMM is built on InstructBlip following the referenced codebase.The paper reports dataset-specific hyperparameters for different tasks and datasets.
  • Implementation: All experiments use cosine learning-rate decay.
  • Long-term Video Understanding: For LVU, Breakfast, and COIN, the long-term video understanding setup samples 100 frames per video clip.LVU uses 100 frames at 1 fps; the passage states uniform sampling for Breakfast and COIN but is truncated before completing their settings.

E. Limitation and Future Work

MA-LMM reduces GPU memory usage through online processing, but extremely long videos increase processing time. Future work targets stronger temporal representation, pre-training, and language decoding.

  • Limitation: Online frame processing reduces GPU memory usage but increases video processing time, especially for extremely long videos.The authors suggest hierarchical segment processing to balance memory efficiency and speed.
  • Future Work: Replacing the image-based visual encoder with a video- or clip-based encoder could improve short-term temporal dynamics.The authors identify this as a future enhancement to video representation.
  • Future Work: Large-scale video-text pre-training is proposed as a way to enhance video understanding and generalization capabilities.
  • Future Work: Incorporating a more advanced LLM as the language decoder is proposed as another route to improve final performance.
Loading 2404.05726v2…