Source-linked AI summary
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, Li Yuan
TL;DR
Existing LVLMs commonly place image and video features in separate spaces, making unified multimodal learning difficult. Video-LLaVA aligns both modalities in the language feature space before shared projection and jointly trains on images and videos, achieving strong results across image and video benchmarks. The authors conclude that unified representation and joint training mutually benefit image-video understanding.
Problem
Most existing approaches encode images and videos in separate feature spaces, limiting an LLM's ability to learn their interactions through projection layers.
Method
Video-LLaVA pre-aligns image and video representations in the language feature space, feeds them through a shared projection layer, and jointly trains on both modalities.
Results
Video-LLaVA outperforms models specialized for images or videos across broad image and video evaluations, including gains of 5.8%, 9.9%, 18.6%, and 10.1% over Video-ChatGPT on MSVD, MSRVTT, TGIF, and ActivityNet.
Takeaways & Limitations
Unified visual representation and joint image-video training mutually benefit multimodal understanding and enable an LLM to handle images and videos simultaneously.
Takeaways & Limitations
Video-LLaVA performs moderately on long videos because uniformly sampling 8 frames can lose detailed information, and training requires 3-4 days on 8 A100-80G GPUs.
Abstract
from arXiv · showhide
The Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in visual-language understanding. Most existing approaches encode images and videos into separate feature spaces, which are then fed as inputs to large language models. However, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it becomes challenging for a Large Language Model (LLM) to learn multi-modal interactions from several poor projection layers. In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. As a result, we establish a simple but robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images and videos, mutually enhancing each other. Video-LLaVA achieves superior performances on a broad range of 9 image benchmarks across 5 image question-answering datasets and 4 image benchmark toolkits. Additionally, our Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive experiments demonstrate that Video-LLaVA mutually benefits images and videos within a unified visual representation, outperforming models designed specifically for images or videos. We aim for this work to provide modest insights into the multi-modal inputs for the LLM. Code address: \href{https://github.com/PKU-YuanGroup/Video-LLaVA}
1 Introduction
Video-LLaVA addresses the difficulty of jointly understanding images and videos by aligning their representations before projection into an LLM. It reports strong image and video results and finds that unified representations and joint training benefit both modalities.
- Motivation: Existing LVLMs often handle only one visual modality, while shared-encoder approaches struggle to learn unified image-video representations.The paper attributes this challenge to inherent differences between images and videos.
- Motivation: Separate image and video feature spaces make it difficult for LLMs to learn cross-modal interactions through multiple projection layers.The paper frames this as a lack of alignment before projection.
- Approach: Video-LLaVA aligns images and videos into a unified visual feature space before using a shared projection layer for the LLM.It jointly trains on images and videos, completing training in 1 epoch.
- Results: Video-LLaVA surpasses mPLUG-owl-7B and InstructBLIP-7B on 5 image benchmarks and exceeds IDEFICS-80B by 6.4% in MMBench.These results are reported across image benchmarks and four benchmark toolkits.
- Results: 5.8%, 9.9%, 18.6%, and 10.1% are Video-LLaVA's reported gains over Video-ChatGPT on MSVD, MSRVTT, TGIF, and ActivityNet, respectively.The paper reports these comparisons for video question-answering datasets.
- Findings: Ablations show that alignment before projection and joint image-video training benefit unified visual representation learning.The experiments support complementary benefits from the two modalities.
2 Related Work
The related work distinguishes LVLMs that treat the LLM as a scheduler from methods that use it as a decoder. Video-LLaVA belongs to the decoder-oriented line, extending alignment and instruction tuning to jointly handle images and videos.
- Comparison: Table 1 compares large vision-language models according to their treatment of the LLM and whether they use pre-alignment and joint training.The caption specifically notes that scheduler-based methods do not require either procedure.
- LLM as scheduler: Scheduler-based LVLMs use visual models as plug-and-play modules selected by the LLM for particular visual tasks.These methods do not require end-to-end training, pre-alignment, or joint training of each modality.
- LLM as decoder: Decoder-oriented methods align visual tokens with the LLM, often through projection layers and instruction tuning.The paper identifies this family as its primary focus.
- Prior multimodal models: Video-ChatGPT uses a 100k video instruction dataset to enable video comprehension, while VideoChat and Video-LLaMA jointly train on images and videos.These approaches represent prior efforts to extend LLMs beyond image-only inputs.
3 Video-LLaVA
Video-LLaVA maps images and videos into a shared visual-language space, projects them with shared layers, and trains the LLM on visual-language conversations. Its pipeline combines visual encoding, multimodal token construction, and staged instruction tuning.
- Architecture: The framework extracts visual features with LanguageBind, applies shared visual projection layers, and combines them with tokenized textual queries for the LLM.The pipeline accepts raw images or videos as visual signals.
- Architecture: LanguageBind maps image and video representations into the language feature space, producing a unified visual representation before shared projection.Its encoders are initialized from OpenCLIP and video-text alignment using VIDAL-10M.
- Training pipeline: Equation 1 converts textual input and visual signals into a sequence of tokens used for response generation.The model then maximizes the likelihood objective described in Equation 2.
- Training pipeline: Joint training places image and video samples in the same batch, supporting multimodal learning with a single training process.The paper states that this improves computational efficiency and achieves results with 1 training epoch.
- Understanding training: Stage 1 uses single-turn image/video-text pairs to teach basic visual interpretation with autoregressive training.The model learns concise visual understanding while other model parameters are frozen.
- Instruction tuning: Stage 2 uses multi-turn conversations and prior dialogue context to train responses to more complex visual instructions and reasoning tasks.The LLM participates in training at this stage.
4 Experiments
Video-LLaVA is evaluated across video reasoning, image understanding, and object hallucination benchmarks using mixed image-video training and shared visual components. The experiments compare its performance with existing LVLMs and video or image specialists.
- Video Understanding: Video-LLaVA is evaluated on four video question-answering datasets, with accuracy and GPT-Assistant-assessed scores reported.The evaluation follows the Video-ChatGPT pipeline.
- Video Understanding: 5.8%, 9.9%, 18.6%, and 10.1% are the reported gains over Video-ChatGPT on MSRVTT, MSVD, TGIF, and ActivityNet, respectively.
- Video Understanding: Video-LLaVA surpasses Chat-UniVi on MSVD, MSRVTT, and TGIF despite Chat-UniVi using more datasets.
- Object Hallucination: The experiments include comparisons on zero-shot object hallucination evaluation using random, popular, and adversarial POPE subsets.
- Image Understanding: Video-LLaVA outperforms InstructBLIP-7B across all five image question-answering benchmarks.It also surpasses InstructBLIP-13B by 14.7% on VisWiz.
Evaluation under Image Benchmark Toolkits
Video-LLaVA is evaluated with visual instruction-tuning benchmark toolkits and shows strong performance against both smaller and larger LVLM baselines. These evaluations cover general visual question answering and object hallucination-related consistency.
- Benchmark Toolkits: 24.9%, 12.2%, and 5.8% are Video-LLaVA's reported gains over InstructBLIP-7B on MMBench, LLaVA-Bench, and MM-Vet, respectively.
- Benchmark Toolkits: Video-LLaVA-7B surpasses InstructBLIP-13B by 6.4% on MM-Vet and IDEFICS-80B by 6.4% on MMBench.
- Object Hallucination: Video-LLaVA is competitive across random, popular, and adversarial POPE subsets and outperforms MM-GPT and Mini-GPT4 across the reported comparisons.The comparisons use a polling-based query evaluation pipeline for zero-shot object hallucinations.
4.3 Ablation Results
The ablations show that aligning image and video representations before projection, and jointly training on both modalities, improves unified visual understanding across image and video evaluations.
- Alignment Before Projection: Unified visual representations significantly improve video question-answering accuracy and answer scores across four datasets compared with separated representations.The comparison replaces the image encoder while keeping the video encoder fixed, isolating representation alignment before projection.
- Alignment Before Projection: Unified visual representations outperform separated representations across five image question-answering datasets and four benchmark toolkits.The largest gains appear on MMBench, LLaVA-Bench, and MM-Vet, with benefits also reported for object hallucination and OCR capabilities.
- Joint Training: The ablations support complementary benefits from unified representations and joint image-video training for LLM visual understanding.The joint-training analysis explicitly evaluates whether images and videos mutually enhance comprehension through a unified visual representation.
- Joint Training: Joint training on images and videos improves performance across all four evaluated video question-answering datasets versus training without image data.Video-LLaVA* denotes the comparison model trained only with video data.
- Joint Training: Joint video-image training improves Video-LLaVA over a matched LLaVA-1.5† configuration on 8 out of 9 image benchmarks.The comparison uses the LanguageBind-Image encoder and the same training configuration for fairness; POPE results indicate reduced object hallucination.
5 Limitation and Future Directions
Video-LLaVA remains limited in long-video understanding and requires substantial computational resources for training. Future work targets more efficient token compression, broader modalities, and temporal reasoning.
- Limitation: Video-LLaVA performs moderately on long videos because it uniformly samples only 8 frames, losing detailed information.Chat-UniVi surpasses 0.5 on ActivityNet-QA in the cited comparison.
- Limitation: Training Video-LLaVA requires 3–4 days on 8 A100-80G GPUs.
- Future Directions: Future work could explore more efficient shared projection that compresses tokens while preserving data features for improved long-video understanding.
- Future Directions: The framework could be extended to depth and infrared images, with timestamp embeddings supporting questions about temporal relationships.
6 Conclusion
Video-LLaVA addresses misalignment before projection by pre-binding image and video signals into a shared language-feature space and jointly training on both modalities. The resulting framework supports unified image-video comprehension and outperforms specialized image or video experts in the reported experiments.
- Conclusion: Video-LLaVA uses a LanguageBind encoder to pre-bind visual signals into the language feature space, addressing misalignment before projection.
- Conclusion: Joint image-video training lets the LLM learn multimodal interactions from a unified visual representation.
- Conclusion: Extensive experiments show that joint training mutually benefits image and video performance, while alignment before projection aids LLM learning.
- Conclusion: Video-LLaVA simultaneously engages with images and videos and surpasses models designed specifically for images or videos.
A.1 Training Setting
The training setting keeps the image and video encoders frozen and uses a shared two-layer projection module for both modalities.
- Training Setting: The video and image encoders are not trained in either training stage.
- Training Setting: The shared projection layer contains two linear layers with a GeLU activation between them.
- Training Setting: Image and video representations share the projection layer.
A.2 Exhibition Board
This section presents unselected samples of Video-LLaVA in video understanding, sourced from Video-ChatGPT.
- The videos shown in these samples are sourced from Video-ChatGPT.
- The figures present unselected Video-LLaVA samples for video understanding.