PaperlayerView exact revision on arXiv

Source-linked AI summary

ST-LLM: Large Language Models Are Effective Temporal Learners

Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, Ge Li

arXiv:2404.00308v1cs.CV

TL;DR

Video LLMs still struggle to encode and understand temporally dynamic content efficiently. ST-LLM feeds spatial-temporal tokens into the LLM, using dynamic masking and global-local input to manage sequence length and long videos, and reports state-of-the-art performance across multiple video benchmarks. Its main scope limits are fine-grained understanding, occasional hallucinations, and exceptionally long videos where performance and computational cost remain difficult to balance.

  • Problem

    Existing video LLMs have limited temporal understanding, while feeding all video tokens creates context-length and variable-video-length challenges.

  • Method

    ST-LLM models raw spatial-temporal video tokens inside the LLM, with dynamic masking and a global-local input mechanism for efficiency, stability, and long videos.

  • Results

    ST-LLM achieves state-of-the-art performance across multiple video LLM benchmarks, including MVBench and VideoChatGPT-Bench.

  • Takeaways & Limitations

    The results support a concise video-LLM design that leverages the LLM’s sequence modeling capacity without additional modules or expensive alignment pre-training.

  • Takeaways & Limitations

    ST-LLM remains unsatisfactory on fine-grained and detailed long-video understanding, with some hallucinations and no especially effective strategy for exceptionally long videos at low computational cost.

Abstract

from arXiv · show

Large Language Models (LLMs) have showcased impressive capabilities in text comprehension and generation, prompting research efforts towards video LLMs to facilitate human-AI interaction at the video level. However, how to effectively encode and understand videos in video-based dialogue systems remains to be solved. In this paper, we investigate a straightforward yet unexplored question: Can we feed all spatial-temporal tokens into the LLM, thus delegating the task of video sequence modeling to the LLMs? Surprisingly, this simple approach yields significant improvements in video understanding. Based upon this, we propose ST-LLM, an effective video-LLM baseline with Spatial-Temporal sequence modeling inside LLM. Furthermore, to address the overhead and stability issues introduced by uncompressed video tokens within LLMs, we develop a dynamic masking strategy with tailor-made training objectives. For particularly long videos, we have also designed a global-local input module to balance efficiency and effectiveness. Consequently, we harness LLM for proficient spatial-temporal modeling, while upholding efficiency and stability. Extensive experimental results attest to the effectiveness of our method. Through a more concise model and training pipeline, ST-LLM establishes a new state-of-the-art result on VideoChatGPT-Bench and MVBench. Codes have been available at https://github.com/TencentARC/ST-LLM.

1 Introduction

Existing video LLMs struggle with temporally dynamic scenes, motivating ST-LLM’s use of the LLM itself for spatial-temporal sequence modeling. Dynamic masking and global-local input address token-length, variable-length, and long-video challenges while supporting strong benchmark performance.

  • 1 Introduction: Existing video LLMs often handle static-context actions better than motion-dependent content, including basic movement direction.Mean pooling is efficient but inadequate for dynamic temporal sequences.
  • 1 Introduction: ST-LLM delegates spatial-temporal sequence modeling to the LLM by feeding it raw video tokens.The approach avoids introducing additional temporal-modeling modules.
  • 1 Introduction: Dynamic video-token masking reduces LLM sequence length and improves robustness across videos with varying numbers of frames.The strategy is paired with masked video modeling during training.
  • 1 Introduction: The global-local input mechanism processes particularly long videos by combining pooled information from many frames with a smaller subset of frames.This preserves video-token modeling inside the LLM while controlling input length.
  • 1 Introduction: ST-LLM achieves state-of-the-art performance across contemporary video benchmarks, especially on tasks requiring temporal understanding.The reported benchmarks include MVBench, VideoChatGPT-Bench, and zero-shot video QA evaluation.

2 Related Works

Prior video-language approaches adapt image models, add temporal modules, or use dedicated video encoders, often requiring intensive training. ST-LLM instead entrusts visual sequence modeling to the LLM through a more concise training approach.

  • 2 Related Works: Existing video-specific adaptations improve over mean pooling but typically add modules that require intensive training.VideoChat2’s dedicated video encoder required three-stage alignment training with 30 million visual-language samples.
  • 2 Related Works: ST-LLM directly uses the LLM for visual sequence modeling, combining conciseness and reduced training requirements with reported superior effectiveness.The approach builds on prior joint spatial-temporal modeling while applying it inside an LLM.

3 Methodology

ST-LLM feeds spatial-temporal video tokens directly into the LLM, then adds dynamic masking and global-local input to improve efficiency, robustness, and long-video handling.

  • 3.1 Video Tokens Inside LLM: ST-LLM directly feeds the spatial-temporal sequence into the LLM, using a visual encoder, projection layer, and concatenated video-text input.Each frame is encoded as an image into K visual tokens, which are concatenated with text tokens before entering the LLM.
  • 3.1 Video Tokens Inside LLM: The method favors simple token organization without additional separators or visual position embeddings, reporting that simplicity often performs best.ST-LLM avoids additional modules and expensive alignment pre-training, enabling reuse of existing image conversational models.
  • 3.2 Training with Dynamic Masking: Uncompressed video tokens increase context length and computational burden, while mismatched training and testing frame counts can noticeably degrade performance.Dynamic masking reduces sequence length but does not fully resolve impractical context lengths for extremely long videos.
  • 3.2 Training with Dynamic Masking: Dynamic masking varies the spatial-temporal sequence length during training while maintaining an average masking rate of 50%, improving inference robustness.The method masks video tokens while leaving text tokens unchanged and uses masked video modeling with the LLM decoder loss.
  • 3.3 Global-Local Input: Global-local input combines a pooled global representation with sampled local frame tokens, allowing long videos to retain global information within a limited context.A simple MLP projector integrates information from the high-fps branch into low-fps spatial-temporal sequences.

4 Experiments

Experiments evaluate ST-LLM across video understanding benchmarks, token-input designs, masking strategies, and qualitative cases. Results show strong temporal modeling, with dynamic masking and MVM improving robustness to varying sequence lengths.

  • Quantitative Results: ST-LLM consistently performs well across VideoChatGPT-Bench tasks despite considerable variation from unstable GPT-based evaluation.The benchmark measures correctness, detail, contextual understanding, temporal understanding, and consistency.
  • Token Input and Training Ablations: Joint spatial-temporal input surpasses mean pooling after instruction tuning, while dynamic masking and MVM further improve performance.Without instruction tuning, joint input is less effective and can produce severe hallucinations from training-testing mismatch.
  • Token Input and Training Ablations: Dynamic masking improves overall performance and robustness to varying test frame counts, while MVM yields larger gains as sequence length increases.Fixed-frame training and static 50% masking remain less robust when training and testing frame counts differ.
  • Global-Local Input: Global-local input does not improve MVBench because its short videos may not require many frames, whereas longer VideoChatGPT-Bench videos motivate the design.The comparison combines global features from more frames with local features from fewer frames.
  • Qualitative Results: Qualitative results show stronger instruction adherence, response precision, and sensitivity to temporal sequences and actions.The paper presents these observations as validation of its spatial-temporal modeling approach.

5 Conclusion

ST-LLM is presented as a robust video large language model that uses the LLM to model video tokens for spatial-temporal comprehension. Dynamic masking and global-local input address efficiency, stability, and long-video challenges, yielding state-of-the-art performance across multiple video LLM benchmarks.

  • ST-LLM uses an LLM to model video tokens for joint spatial-temporal-text video comprehension.The method is designed to encode and comprehend spatial-temporal sequences within the LLM.
  • Dynamic masking and a global-local input module address efficiency, stability, and lengthy-video modeling concerns.
  • Extensive experiments report state-of-the-art performance across multiple video LLM benchmarks with reduced training resource requirements.

Temporal Learners ——–Supplementary Material——–

The supplementary material provides additional details and experiment analysis for the main paper.

  • The supplementary material contains additional details of the main paper.
  • It also provides more experiment analysis.
  • The material complements the main paper with further experimental information.

1 Detailed Experiment Setup

The experiments use InstructBLIP and MiniGPT4 image models initialized by BLIP-2, with task-specific prompts for MVBench and GPT-based evaluation. BLIP-2 compresses CLIP visual tokens through a Q-former, while InstructBLIP’s Q-former receives only the user instruction here.

  • InstructBLIP and MiniGPT4 are initialized by BLIP-2 and undergo additional image instruction tuning.
  • BLIP-2 uses a 39-layer EVA-CLIP and Q-former to compress 256 CLIP tokens into 32 visual tokens.
  • MVBench and GPT-based evaluation use distinct prompt designs tailored to their respective tasks.

2 Additional Ablations

Additional ablations examine image-video joint training, input-frame scaling, and inference strategy. Results indicate that training and testing consistency matters, while ST-LLM benefits from more frames mainly when the frame count is low.

  • Image-Video Joint Pretraining: Image-video joint training improves MiniGPT4 but decreases InstructBLIP performance.The reported discrepancy is attributed to InstructBLIP’s prior training on rich image data and repeated training.
  • Scale Up Input Frames: ST-LLM shows noticeable gains from increasing input frames at low counts, but very high counts provide no further improvement.Mean-pooling inputs are relatively insensitive to frame-count increases; the high-count plateau may reflect short MVBench videos or difficulty modeling extended visual contexts.
  • Inference Strategy: Fixed inference frame counts generally outperform a frame rate of 1, underscoring the importance of training and testing consistency.The ablation reports average MVBench scores for the testing strategies.

3 MVBench Leaderboards

ST-LLM achieves the highest rank across 11 MVBench tasks, with particularly strong performance on action and movement tasks.

  • ST-LLM achieves the highest rank across 11 MVBench tasks.
  • ST-LLM generally outperforms VideoChat2 on action-related tasks and shows significant advantages across movement-related tasks.

4 Additional Qualitative Results

Additional visualizations show ST-LLM handling summaries, detailed descriptions, inference, and fine-grained descriptions of long videos while following user instructions and capturing dynamic information.

  • ST-LLM demonstrates exceptional performance across video summaries, detailed descriptions, content inference, and fine-grained descriptions of long videos.
  • ST-LLM follows user instructions strictly while accurately describing video content and capturing dynamic information.

5 Limitations and Future Work

The paper identifies weaknesses in fine-grained spatiotemporal understanding and acknowledges remaining hallucinations and efficiency challenges for exceptionally long videos, while proposing further work on modeling, efficiency, and scaling.

  • Limitations: ST-LLM does not excel at fine-grained tasks, indicating limitations in particularly fine-grained spatiotemporal modeling.
  • Limitations: For exceptionally long videos, ST-LLM lacks effective strategies that preserve both performance and low computational cost.
  • Limitations: Some hallucinations may still occur, and overall video understanding remains short of satisfactory despite outperforming VideoChat2.
  • Future Work: Future work targets improved spatiotemporal modeling, greater efficiency and long-video adaptability, and scaling through larger models, more data, and broader pre-training.
Loading 2404.00308v1…