PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, Lidong Bing

arXiv:2406.07476v3cs.CVcs.CL

TL;DR

Video-LLMs remain limited in temporal feature fusion and audio integration, motivating broader multimodal video understanding. VideoLLaMA 2 addresses these gaps with an STC connector and jointly trained Audio Branch, achieving competitive video results and improvements on audio-oriented benchmarks. The authors conclude that it advances multimodal comprehension and can support further specialized research.

  • Problem

    Video-LLMs have limited temporal feature fusion and frequently omit audio, restricting comprehensive understanding of multimodal video data.

  • Method

    VideoLLaMA 2 combines a Spatial-Temporal Convolution connector for video features with a jointly trained Audio Branch for multimodal modeling.

  • Results

    VideoLLaMA 2 consistently improves multimodal comprehension across video and audio-oriented tasks, outperforming similarly sized open-source models and approaching proprietary-model performance in several aspects.

  • Takeaways & Limitations

    VideoLLaMA 2 provides a publicly available foundation for video-language research and further development on specialized problems.

  • Takeaways & Limitations

    The main comparison excludes preference-optimized LLaVA-NeXT-Video as a baseline, with full comparisons deferred to Appendix B.

Abstract

from arXiv · show

In this paper, we present the VideoLLaMA 2, a set of Video Large Language Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio understanding in video and audio-oriented tasks. Building upon its predecessor, VideoLLaMA 2 incorporates a tailor-made Spatial-Temporal Convolution (STC) connector, which effectively captures the intricate spatial and temporal dynamics of video data. Additionally, we integrate an Audio Branch into the model through joint training, thereby enriching the multimodal understanding capabilities of the model by seamlessly incorporating audio cues. Comprehensive evaluations on multiple-choice video question answering (MC-VQA), open-ended video question answering (OE-VQA), and video captioning (VC) tasks demonstrate that VideoLLaMA 2 consistently achieves competitive results among open-source models and even gets close to some proprietary models on several benchmarks. Furthermore, VideoLLaMA 2 exhibits reasonable improvements in audio-only and audio-video question-answering (AQA & OE-AVQA) benchmarks over existing models. These advancements underline VideoLLaMA 2's superior performance in multimodal comprehension, setting a new standard for intelligent video analysis systems. All models are public to facilitate further research.

1 Introduction

VideoLLaMA 2 addresses Video-LLMs’ difficulty modeling temporal dynamics and incorporating audio by combining a Spatial-Temporal Connector with a jointly trained Audio Branch. The resulting system is presented as a multimodal model for video-language understanding and is described as achieving strong performance across video-analysis tasks.

  • Research gap: Video-LLMs struggle to fuse information across frames and often overlook audio, limiting temporal reasoning and comprehensive multimodal analysis.These limitations motivate models that handle multimodal video data while preserving processing efficiency and contextual integrity.
  • Proposed model: VideoLLaMA 2 uses a specially designed Spatial-Temporal Connector to process temporal dynamics for video-language tasks.The model is designed for tasks ranging from video captioning to complex question answering.
  • Proposed model: A jointly trained Audio Branch integrates auditory signals into VideoLLaMA 2’s multimodal understanding.The branch is intended to incorporate audio cues that are often underutilized in video-language models.
  • Reported significance: VideoLLaMA 2 is presented as a development for video-language analytics, with evaluations intended to illustrate performance relative to existing models.The paper frames these enhancements as improving intelligent video understanding systems.

2 Method

VideoLLaMA 2 uses independent vision-language and audio-language branches that connect pretrained encoders to an instruction-tuned LLM. Its STC connector preserves token order while reducing and aggregating spatial-temporal features, and its audio branch aligns encoded spectrogram features through an MLP.

  • Architecture: VideoLLaMA 2 has independent vision-language and audio-language branches, with cross-modal interactions occurring within the language model.Each branch connects a pretrained modality encoder to an instruction-finetuned large language model in a modular framework.
  • Vision-Language Branch: The vision branch samples fixed-size video frames, encodes them with image-level CLIP, and processes them using an STC Connector instead of VideoLLaMA 1’s Q-former.Frames are padded and resized to 336x336 before image encoding.
  • Audio-Language Branch: The audio branch transforms audio into 128-bin fbank spectrograms, encodes them with BEATs, and passes the features through a two-layer MLP block.The MLP processes the encoded audio features for alignment with the language model.
  • Language backbone: The experiments use Mistral-Instruct, Mixtral-Instruct, and Qwen2-Instruct as language decoders, while other LLM backbones are left for future exploration.The paper states that it does not extensively search for the optimal LLM backbone.
  • STC Connector: The STC connector uses two spatial interaction modules and one spatial-temporal aggregation module, implemented with RegStage and 3D convolution.The design aims to preserve spatial-temporal order, reduce token count, and alleviate information loss during downsampling.
  • Connector design study: The STC design search evaluates spatial interaction and spatial-temporal aggregation choices on Video-LLaVA data using Egoschema, MV-Bench, and ActivityNet-QA.The reported best average performance comes from 3D convolution combined with the RegStage block; the study defaults to eight sampled frames.

3 Training

VideoLLaMA 2 is trained through staged video-language, audio-language, and joint multimodal learning, combining broad weakly labeled data with task-specific annotations.

  • Training stages: VideoLLaMA 2 training comprises video-language pre-training, multimodal fine-tuning, audio-language training, and joint audio-visual integration.The training process explicitly covers the Video-Language and Audio-language branches followed by joint training.
  • Video-language training: The pre-training stage uses large-scale web-crawled image-text and video-text pairs, while freezing the vision encoder and language model and optimizing only the connector.Video frames are evenly sampled and resized to 336 × 336 pixels, with text-token cross-entropy as the objective.
  • Video-language training: Multi-task fine-tuning combines high-quality annotations for video captioning, classification, and visual question answering while optimizing the language model and spatial-temporal connector.The visual encoder remains frozen during this stage.
  • Audio-language training: Audio-language training begins with approximately 400,000 WavCaps audio samples for captioning, then expands to question answering and sound-event and vocal-sound classification.The audio projector is optimized initially, while later audio-language training optimizes the audio encoder and projector with the LLM frozen.
  • Joint multimodal training: The joint stage uses audio-visual question-answering datasets alongside diverse video-text, audio-text, and instruction-following text datasets.Examples include AVQA, AVQA-music, SthSthv2, EgoQA, AudioCaps, Wavcaps, and evol-instruct.
  • Joint multimodal training: Joint multimodal training synchronizes audio tracks with video clips and samples audio-visual, visual, and audio data in a 2:1:1 ratio.The audio/video projector, audio encoder, and language model are optimized while the video encoder remains frozen.

4 Implementation

VideoLLaMA 2 uses modular visual, audio, and language components, with later variants favoring stronger encoders and multiple instruction-tuned language backbones.

  • Encoders: All VideoLLaMA 2 variants use CLIP-Large-336 as the primary visual encoder, while later variants prefer SigLIP-So400m-384 for its superior performance.Audio encoding uses fine-tuned BEATs_iter3+(AS2M)(cpt2).
  • Language decoders: Language decoders are initialized with Mistral-7B-Instruct, Mixtral-8x7B-Instruct, Qwen2-7B-Instruct, or Qwen2-72B-Instruct.The model is built upon the LLaVA 1.5 library.

5 Model Evaluation

VideoLLaMA 2 is evaluated across multiple-choice and open-ended video question answering and video captioning, using zero-shot comparisons with proprietary and open-source models.

  • Evaluation setup: Evaluation covers MC-VQA, OE-VQA, and VC tasks across established video benchmarks, with quantitative and qualitative analyses of multimodal understanding.MC-VQA uses EgoSchema, Perception-Test, MV-Bench, and VideoMME; OE-VQA uses MSVD-QA, ActivityNet-QA, and Video-ChatGPT.
  • Evaluation caveats: VideoMME results are reported without subtitles because of unknown issues retrieving them, limiting that benchmark setting.The table caption also notes that explicit chain-of-thought prompting was not used for Gemini 1.5 on EgoSchema.
  • Baselines and protocols: Comparisons include proprietary systems such as Gemini, GPT4-V, GPT4-O, Reka, and Pegasus-1, alongside open-source models including LLaVA-NeXT-Video and VideoChat2.All experiments are conducted zero-shot, using greedy decoding except for MSVC, which uses sampling at temperature 0.2.
  • MC-VQA and VC results: VideoLLaMA 2-7B reaches 51.7% accuracy on EgoSchema, 51.4% on Perception-Test, 53.9% on MV-Bench, and 48.4% on VideoMME.On EgoSchema, 51.7% exceeds LLaVA-NeXT-Video’s 43.9%; on MV-Bench, the model also exceeds GPT4-V’s 43.7%.
  • MC-VQA and VC results: VideoLLaMA 2 scores 2.57 in correctness and 2.61 in detailedness on MSVC, exceeding all other open-source models but trailing GPT4-V’s 2.70 and 2.76.MSVC contains 500 videos with human-annotated captions from MSVD, MSRVTT, and VATEX.
  • OE-VQA results: On MSVD, VideoLLaMA 2-7B achieves 71.7% accuracy with a score of 3.9, outperforming LLaVA-NeXT-Video and VideoChat2 among cited open-source comparisons.The reported comparison is 71.7%/3.9 versus 67.8%/3.5 and 70.0%/3.9.
  • OE-VQA results: On ActivityNet, VideoLLaMA 2-7B obtains 49.9% accuracy with a score of 3.3, slightly below LLaVA-NeXT-Video’s 53.5% accuracy and 3.2 score.The evaluation uses GPT-assisted binary correctness judgments for open-ended answers.
  • OE-VQA results: On Video-ChatGPT, VideoLLaMA 2-7B scores 3.09 correctness, 3.09 detail, 3.68 context, 2.63 temporal understanding, and 3.25 consistency.LLaVA-NeXT-Video outperforms it on correctness, detail, and context in the cited comparison.

5.2 Audio Understanding

VideoLLaMA 2 is evaluated on audio-only and audio-video question answering using established benchmarks and comparisons with specialized audio and multimodal models. The results show strong AQA performance with substantially less training data and competitive OE-AVQA performance across multiple datasets.

  • Evaluation setup: The evaluation covers open-ended Clotho-AQA, multiple-choice TUT2017 and VocalSound, and open-ended VGGSound, AVSD, and Music-AVQA benchmarks.GPT-assisted evaluation is used for the open-ended audio-video question-answering assessment.
  • AQA: VideoLLaMA 2-A (7B) achieves 68.90% on Clotho-AQA and 75.19% on TUT2017, surpassing Qwen-Audio’s 57.90% and 64.90%, respectively.VideoLLaMA 2-A uses 4k hours of audio data, compared with Qwen-Audio’s 137k hours.
  • AQA: VideoLLaMA 2-A (7B) reaches 92.73% on VocalSound, closely matching Qwen2-Audio (7B)’s 93.92%.The comparison is reported despite VideoLLaMA 2-A using 4k hours versus Qwen2-Audio’s 520k hours.
  • AQA: VideoLLaMA 2-AV (7B) records 70.11% on Clotho-AQA and 78.40% on TUT2017, the highest scores reported for those benchmarks in the comparison.The model uses about 5k hours of multimodal audio-visual data.
  • OE-AVQA: VideoLLaMA 2-AV (7B) scores 79.2% on MUSIC-QA, exceeding CREAM’s 75.6% specialized-model result.The benchmark evaluates musical and audio cues in audio-visual question answering.
  • OE-AVQA: VideoLLaMA 2-AV (7B) scores 57.2% on AVSD and 70.9% on VGGSound, exceeding AV-LLM’s 47.6% on VGGSound.The reported benchmarks assess dialogue-based audio-visual scene description and integration of visual and auditory scene information.

6 Cases

Figure 3 presents qualitative cases showing VideoLLaMA 2’s ability to understand scenes, track spatial-temporal information, reason with commonsense, and recognize fine-grained details across video moments.

  • Global Scene Understanding: VideoLLaMA 2 describes objects, atmosphere, and a boy’s movements in an arcade scene, illustrating global scene understanding.The case identifies a game console and dancing movements as part of the scene description.
  • Spatial-temporal Orientation Awareness: VideoLLaMA 2 correctly determines the yellow cab’s turning direction after observing the entire video, demonstrating spatial-temporal orientation awareness.The qualitative answer states that the cab turns left at the intersection.
  • Commonsense Reasoning: VideoLLaMA 2 infers that a scene is likely morning from reading, drinking, and natural-light cues, applying commonsense reasoning.The response preserves uncertainty rather than presenting the inferred time as certain.
  • Spatial-Temporal Fine-grained Recognition: VideoLLaMA 2 locates the moment and position of a red sticky note and recognizes its text without being distracted by earlier video information.The example targets fine-grained object recognition across both time and space.

7 Related Works

The paper situates VideoLLaMA 2 within Video-LLM architectures and audio-integration research, while Figure 3 illustrates its multimodal video-understanding behaviors through qualitative cases.

  • Video-LLM architectures: Existing Video-LLMs commonly combine a pretrained visual encoder, a vision-language adapter, and an instruction-tuned language decoder.The adapter aggregates frame features into representations usable by the language model.
  • VideoLLaMA 2 architecture: VideoLLaMA 2 introduces an STC Connector using 3D convolution for spatial-temporal aggregation while maintaining a reasonable number of visual tokens.RegStage blocks are used before and after the 3D convolution to preserve local visual patterns during spatial compression.
  • Audio integration: Prior multimodal models integrate audio through universal modality alignment, separate visual and audio branches, or audio-visual training data.The related work describes PandaGPT, VideoLLaMA, MacawLLM, AV-LLM, and AVicuna as examples of these approaches.

8 Conclusion

VideoLLaMA 2 combines an STC connector with a jointly trained Audio Branch to improve multimodal comprehension across video and audio-oriented tasks. It outperforms similarly sized open-source models across multiple benchmarks and approaches proprietary-model performance in several aspects.

  • VideoLLaMA 2 combines a Spatial-Temporal Convolution connector with a jointly trained Audio Branch to improve multimodal comprehension.The conclusion links these components to modeling temporal dynamics with synchronous audio streams.
  • Across multiple benchmarks, VideoLLaMA 2 outperforms open-source models of similar size and achieves performance comparable to proprietary models in several aspects.The reported scope includes video-language and audio-oriented tasks.
  • The model is presented as a foundation for further work on long-video understanding, video agents, autonomous driving, motion understanding, and robotic manipulation.These applications are identified as specialized and challenging problems for future development.

A The Prompt for GPT-aided AVQA Task Evaluation

Table 9 presents the prompt used for GPT-aided AVQA task evaluation.

  • Table 9 contains the prompt for GPT-aided AVQA task evaluation.
  • The evaluation described is for an AVQA task.
  • GPT is used to aid the evaluation process.

B Comparision Results with LLaVA-NeXT-Video Series

Tables 10 and 11 report comparison results across video question-answering and captioning benchmarks, including MV-VQA, VC, MSVD, ActivityNet, and Video-ChatGPT.

  • Table 10 compares results on MV-VQA and VC tasks.
  • Table 11 compares results on MSVD, ActivityNet, and Video-ChatGPT.
  • Table 11 distinguishes officially reported results from reproduced results.♥ denotes officially reported results, while ♠ denotes reproduced results.

C Results on More Video Benchmarks

Table 12 reports VideoLLaMA 2 model results on four additional video benchmarks: MLVU, VideoVista, MMBench-Video, and DREAM-1k.

  • Table 12 reports VideoLLaMA 2 results on the MLVU benchmark.
  • Table 12 reports VideoLLaMA 2 results on VideoVista and MMBench-Video.
  • Table 12 reports VideoLLaMA 2 results on the DREAM-1k benchmark.
Loading 2406.07476v3…