Source-linked AI summary
MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens
Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, Mohamed Elhoseiny
TL;DR
Video LLMs must capture temporal dynamics and textual context, but prior approaches can lose information when compressing frame sequences. MiniGPT4-Video concatenates compressed visual tokens with subtitles and instructions, and reports improvements across four video benchmarks, while remaining limited by the LLM context window.
Problem
Video understanding requires language models to capture temporal dynamics alongside textual input, a capability that remains underexplored and difficult for existing approaches.
Method
MiniGPT4-Video concatenates compressed visual tokens from sampled frames with subtitle tokens and instruction text, trained through staged image-text, video-text, and question-answer instruction learning.
Results
4.22%, 1.13%, 20.82%, and 13.1% gains are reported on MSVD, MSRVTT, TGIF, and TVQA, respectively.
Takeaways & Limitations
MiniGPT4-Video combines visual and conversational comprehension to support video question answering and understanding of temporal video content.
Takeaways & Limitations
The model’s usable video length is constrained by the LLM context window, supporting fewer than 1.5 minutes with Llama 2 and fewer than 3 minutes with Mistral.
Abstract
from arXiv · showhide
This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM) designed specifically for video understanding. The model is capable of processing both temporal visual and textual data, making it adept at understanding the complexities of videos. Building upon the success of MiniGPT-v2, which excelled in translating visual features into the LLM space for single images and achieved impressive results on various image-text benchmarks, this paper extends the model's capabilities to process a sequence of frames, enabling it to comprehend videos. MiniGPT4-video does not only consider visual content but also incorporates textual conversations, allowing the model to effectively answer queries involving both visual and text components. The proposed model outperforms existing state-of-the-art methods, registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF, and TVQA benchmarks respectively. Our models and code have been made publicly available here https://vision-cair.github.io/MiniGPT4-video/
1. Introduction
MiniGPT4-Video addresses the underexplored challenge of adapting language models to videos, whose temporal structure must be understood alongside textual input. It concatenates visual and subtitle tokens while reducing token count, and reports gains over existing methods across four benchmarks.
- Motivation and contributions: Video understanding requires modeling temporal information across frame sequences together with textual input, unlike static image understanding.
- Motivation and contributions: Prior pooling and frame-token compression methods can lose information and fail to fully exploit video temporal dynamics.
- Motivation and contributions: MiniGPT4-Video concatenates adjacent visual tokens and incorporates subtitles so the LLM processes visual and textual information jointly.
- Motivation and contributions: 4.22%, 1.13%, 20.82%, and 13.1% gains are reported on MSVD, MSRVTT, TGIF, and TVQA, respectively.These evaluations use zero-shot open-ended and multiple-choice benchmarks.
2. Related work
Related work develops vision-language models by connecting visual encoders with language models, then extends these approaches to video understanding. Video systems vary in their use of frame embeddings, temporal modules, and separate audio-visual streams.
- Large Vision-Language Models: Vision-language models align image and language representations, while newer systems use large language models and instruction tuning for multimodal understanding.
- Video Understanding: Video-LLaMA and VideoChat extend BLIP-2-style architectures with video processing, using separate streams or tools for visual and audio information.
- Video Understanding: Otter adapts OpenFlamingo with instruction tuning so it can process multiple video frames as input.
3. MiniGPT4-Video
MiniGPT4-Video maps sampled video frames and subtitles into the LLM input space through staged visual-language, video-text, and question-answer instruction training. Its architecture concatenates compressed visual tokens with subtitle and instruction tokens.
- Training Pipeline: The training pipeline progresses from image-text alignment to video-text pretraining and then video question-answer instruction fine-tuning.The final stage uses video-question-answering data to improve responses to video questions.
- Context and implementation: 45 frames are sampled for Llama 2 and 90 for Mistral because the available context window determines the video input length.The model represents each image with 64 tokens and reserves 1000 tokens for subtitles in the described prompt allocation.
- Architecture: Each frame is represented by compressed EVA-CLIP visual tokens projected into the LLM space, alongside subtitle tokens and an instruction sequence.The architecture concatenates adjacent visual tokens before projection and appends the instruction tokens after all sampled frames.
- Context and implementation: EVA-CLIP remains frozen while the projection layer is trained and LoRA fine-tunes selected language-model components.
4. Experiments
The experiments evaluate MiniGPT4-Video on Video-ChatGPT, open-ended, and multiple-choice benchmarks using GPT-3.5-based comparison and accuracy measures. The model is competitive without subtitles and achieves stronger reported results when subtitles are included.
- Evaluation: Evaluations use GPT-3.5 Turbo to compare model outputs with ground truth, producing accuracy and a 0-to-5 alignment score.The same prompt is used for comparison with Video-ChatGPT results.
- Results: 4.22%, 1.13%, 20.82%, and 13.1% improvements are reported on MSVD, MSRVTT, TGIF, and TVQA, respectively.These results come from zero-shot open-ended and multiple-choice evaluations.
- Results: TVQA accuracy rises from 33.9% to 54.21% when subtitle information is integrated with visual cues.
- Results: With subtitles, MiniGPT4-Video achieves state-of-the-art performance across all five Video-ChatGPT evaluation dimensions.Without subtitles, it is reported as comparable with previous methods.
5. Qualitative Results
MiniGPT4-Video is illustrated through qualitative examples demonstrating its ability to answer different questions about general videos.
- 5. Qualitative Results: Qualitative examples demonstrate MiniGPT4-Video answering different questions about video content.The examples are presented as demonstrations of the model’s performance and question-answering ability.
- 5. Qualitative Results: Figure 3 presents a qualitative result for understanding general videos.
- 5. Qualitative Results: Figure 4 presents another qualitative result of MiniGPT4-Video.
- 5. Qualitative Results: Figure 5 presents another qualitative result of MiniGPT4-Video.
6. Conclusion
MiniGPT4-Video combines visual and conversational comprehension by directly processing visual and textual tokens, while its usable video length is constrained by the LLM context window.
- 6. Conclusion: MiniGPT4-Video combines visual and conversational comprehension for video question answering.The conclusion describes this combination as a solution within the video domain.
- 6. Conclusion: Figures 6 and 8 provide qualitative results and a comparison between MiniGPT4-Video and VideoChatGPT.
- 6. Conclusion: Directly inputting visual and textual tokens helps the LLM model relationships between video frames and temporal dynamics.
- 6. Conclusion: The LLM context window limits videos to 45 frames with Llama 2 and 90 frames with Mistral.These correspond to less than one and a half minutes and less than three minutes respectively at the stated sampling rate.