Source-linked AI summary
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, Deli Zhao
TL;DR
VideoLLaMA3 addresses the challenge of building strong image and video understanding amid temporal complexity and limited video-text data quality. It uses a vision-centric training paradigm with high-quality image-text data, flexible image tokenization, and compact video representations. The model achieves state-of-the-art performance across image and video understanding benchmarks, while real-time processing remains constrained by computational overhead.
Problem
Video understanding requires temporal modeling, while large-scale video-text datasets are often lower quality and harder to annotate than image-text data.
Method
VideoLLaMA3 prioritizes image understanding through four training stages and uses dynamic-resolution vision encoding plus compact video-token representations.
Results
VideoLLaMA3 achieves state-of-the-art performance on both image and video understanding benchmarks, including chart, mathematical, long-video, temporal-reasoning, and grounding tasks.
Takeaways & Limitations
High-quality image-text data and vision-centric designs support a single model’s strong performance across diverse image and video understanding tasks.
Takeaways & Limitations
High-resolution and lengthy video inputs create computational overhead that can hinder real-time performance.
Abstract
from arXiv · showhide
In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation model for image and video understanding. The core design philosophy of VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the vision-centric training paradigm and vision-centric framework design. The key insight of our vision-centric training paradigm is that high-quality image-text data is crucial for both image and video understanding. Instead of preparing massive video-text datasets, we focus on constructing large-scale and high-quality image-text datasets. VideoLLaMA3 has four training stages: 1) Vision Encoder Adaptation, which enables vision encoder to accept images of variable resolutions as input; 2) Vision-Language Alignment, which jointly tunes the vision encoder, projector, and LLM with large-scale image-text data covering multiple types (including scene images, documents, charts) as well as text-only data. 3) Multi-task Fine-tuning, which incorporates image-text SFT data for downstream tasks and video-text data to establish a foundation for video understanding. 4) Video-centric Fine-tuning, which further improves the model's capability in video understanding. As for the framework design, to better capture fine-grained details in images, the pretrained vision encoder is adapted to encode images of varying sizes into vision tokens with corresponding numbers, rather than a fixed number of tokens. For video inputs, we reduce the number of vision tokens according to their similarity so that the representation of videos will be more precise and compact. Benefit from vision-centric designs, VideoLLaMA3 achieves compelling performances in both image and video understanding benchmarks.
1 Introduction
VideoLLaMA3 is a vision-centric multimodal foundation model designed to improve both image and video understanding by prioritizing high-quality image-text data and adapting its vision framework to varied visual inputs.
- Video-centric MLLMs face temporal modeling challenges and often rely on lower-quality, harder-to-annotate video-text data.
- The vision-centric paradigm prioritizes image understanding because videos are sequences of temporally correlated images and image-text data is easier to collect with higher quality.
- Training proceeds through Vision Encoder Adaptation, Vision-Language Alignment, Multi-task Fine-tuning, and Video-centric Fine-tuning.The stages adapt resolution handling, establish multimodal alignment, add downstream and video data, and enhance video question answering.
- The framework uses dynamic-resolution image encoding and compact video-token representations to better capture image details and reduce video redundancy.
- VideoLLaMA3 achieves state-of-the-art performance on image and video benchmarks, including chart understanding, vision-related mathematics, long-video understanding, temporal reasoning, and grounding.
- VideoLLaMA3 targets both image and video understanding, addressing the need for multimodal intelligence beyond language.
2 Methodology
The methodology combines flexible visual tokenization, video-token pruning, and a high-quality image re-captioning pipeline to support efficient multimodal training.
- VideoLLaMA3 uses Any-resolution Vision Tokenization and Differential Frame Pruner as its two key technical points.AVT handles variable-resolution inputs, while DiffFP removes minimally changing video content.
- AnyRes methods can increase token counts but remain inflexible and neglect positional relationships when extracting image tokens.
- AVT converts images and videos of different resolutions into one-dimensional token sequences and is supported by fine-tuning with scene and document data.
- The image re-captioning pipeline filters aspect ratios, aesthetic quality, text-image similarity, and visual clusters before generating short and detailed captions.
- DiffFP prunes video tokens using patch similarities in pixel space, removing patches with smaller distances to the previous frame.
- 7 million image-caption pairs comprise the VL3-Syn7M dataset used to support training across visual tasks.
3 Training
VideoLLaMA3 uses a four-stage training pipeline centered on image understanding, with multimodal data mixtures and distinct formats for images, videos, and streaming videos.
- Training stages: VideoLLaMA3’s training comprises Vision Encoder Adaptation, Vision-Language Alignment, Multi-task Fine-tuning, and Video-centric Fine-tuning.The first three stages primarily improve image understanding, while the final stage focuses on video understanding.
- Vision Encoder Adaptation: Vision Encoder Adaptation fine-tunes the vision encoder and projector to process variable-resolution images and capture fine-grained visual details.The stage uses scene images, document images, and scene-text images, while the language decoder remains frozen.
- Multi-task Fine-tuning: Multi-task Fine-tuning uses diverse instruction-following image and video question-answering data and introduces a video compressor to reduce video-token counts.The image data covers general, document, chart/figure, OCR, grounding, and multi-image tasks.
- Video-centric Fine-tuning: Video-centric Fine-tuning trains all parameters with video-text, image-only, and text-only data to enhance video understanding.The stage targets video understanding and video question answering.
- Data formats: Image sequences separate image tokens with newlines, video sequences add timestamps and comma-separated frames, and streaming sequences interleave video and text tokens.Video data uses “Time: xxs” before each frame, with newlines separating videos from subsequent text.
- Vision-Language Alignment: Vision-Language Alignment jointly trains the multimodal model with high-quality image-text data spanning scenes, text images, documents, charts, fine-grained data, and text-only data.The data mixture is designed to cover diverse everyday scenarios and visual information types.
4.1 Image-based Evaluation
VideoLLaMA3 is evaluated against diverse baselines across document, chart, scene-text, mathematical, multi-image, and general knowledge benchmarks, with improvements reported for both model sizes.
- Evaluation dimensions: Image evaluation covers document, chart, scene-text, mathematical reasoning, multi-image understanding, and general knowledge question answering.The benchmark suite includes DocVQA, ChartQA, InfoVQA, MathVista, MathVision, MMMU-Pro, MMMU, BLINK, RealWorldQA, AI2D, and GQA.
- 2B results: 69.4% on InfoVQA for the 2B model exceeds the previous best score of 65.5%.The result is reported as an OCR benchmark comparison.
- 2B results: 59.2% on MathVista for the 2B model surpasses the state-of-the-art method by 7.9%.The comparison is reported for mathematical reasoning tasks.
- 2B results: 67.3% on RealWorldQA for the 2B model exceeds the prior score of 62.9%.This is reported as the highest performance in the real-world knowledge QA comparison.
- 7B results: The 7B model surpasses the previous best by 6.5% on MathVision, improves InfoVQA by 1.3%, and outperforms prior models by 2.0% on RealWorldQA.These results are summarized from the 7B image-benchmark evaluation.
- Overall findings: Overall, the reported results show consistent advancements across broad image benchmarks, including OCR, mathematical reasoning, and general knowledge.The paper characterizes the model as effective across complex multimodal tasks.
4.2 Video-based Evaluation
VideoLLaMA3 is evaluated across general, long-form, and temporal video understanding tasks using consistent benchmark protocols. The 2B model achieves leading results across most evaluated benchmarks, while the 7B model remains competitive.
- Evaluation scope: The evaluation covers general video understanding, long-video comprehension, temporal reasoning, and temporal sentence grounding.General evaluation includes both multiple-choice and open-ended question answering, while long-video evaluation spans videos from minutes to more than two hours.
- Evaluation protocols: The protocol expands the visual-token budget to 16K and allows up to 180 frames during video evaluation.These settings keep per-frame tokenization reasonable and fit the total context within the base LLM’s maximum range.
- 2B results: The 2B model achieves the highest scores on VideoMME w/o sub (59.6%), VideoMME w/ sub (63.4%), ActivityNet-QA (58.2%), PerceptionTest-test (68.0%), MVBench (65.5%), and MMVU (37.6%).It consistently outperforms or competes strongly with the selected 2B baselines in general video understanding.
- 7B results: VideoLLaMA3-7B leads on 5 of 7 general video benchmarks, matches InternVL2.5-8B comparably on MVBench, and ranks highest on MLVU-dev.It also achieves second-best results on LongVideoBench-val and LVBench.
4.3 Case Study
The case studies illustrate VideoLLaMA3’s capabilities across chart, document, multi-image, general image, and video tasks. Examples include fine-grained visual reasoning, temporal understanding, long-video grounding, and video-image integration.
- Chart Image Understanding: VideoLLaMA3 analyzes stock trends and compares multimodal models while recognizing tradeoffs between parameter count and performance.These examples are presented as chart-image understanding cases.
- OCR and Document Understanding: VideoLLaMA3 parses poster text, suggests design improvements, and performs OCR on document images.The document example demonstrates recognition of words in dense textual visual content.
- Multi-Image Understanding: VideoLLaMA3 distinguishes animals, locates answers in multi-image documents beyond simple OCR, and understands comic-strip storylines.The examples span visual comparison, document reasoning, and sequential narrative interpretation.
- General Image and Video Understanding: VideoLLaMA3 answers general image questions using visual knowledge and comprehends video content through temporal dimensions rather than static-image inference alone.The case studies cover visual question answering, knowledge-based responses, and video understanding.
- Long video and joint understanding: VideoLLaMA3 handles long-video question answering, identifies specified times for temporal grounding, and captures relationships between videos and images.These examples cover complex long-video grounding and video-image joint understanding.
4.4 Ablation Study
The ablation compares CLIP, DFN, and SigLIP vision encoders under fixed-resolution conditions. SigLIP performs best, especially on fine-grained text-related understanding tasks.
- Vision Encoder Ablation: The ablation compares CLIP, DFN, and SigLIP pretrained transformer-based vision encoders.The study uses a subset of the full dataset because of computational limitations.
- Vision Encoder Ablation: The encoders are evaluated with frozen pretrained weights and fixed input resolutions matching their pretrained settings.The resolutions are 336×336 for CLIP, 378×378 for DFN, and 384×384 for SigLIP.
- Vision Encoder Ablation: SigLIP outperforms CLIP and DFN, especially on fine-grained understanding tasks involving text.The result motivates choosing pretrained SigLIP as the base vision encoder.
5 Related Work
Related work spans multimodal models for native video understanding and general vision understanding. Prior studies also report that strong image understanding can transfer effectively to video tasks, while newer systems incorporate longer contexts, audio, and streaming capabilities.
- Multimodal LLMs for Native Video Understanding: Native video MLLMs address token and long-form challenges through expanded context windows and methods that connect visual encoders with language models.Earlier systems used sparse frames and connectors such as MLPs, discrete visual tokenizers, and Q-formers.
- Multimodal LLMs for General Vision Understanding: General MLLMs process images and videos, but dedicated video-specific training may be needed for optimal performance.Models designed for multiple images can process video in principle, yet video training remains relevant to best performance.
- Multimodal LLMs for General Vision Understanding: Prior studies show that robust image-understanding MLLMs can achieve strong video performance with minimal or no dedicated video training data.Other recent systems extend multimodal inputs with audio and speech or target real-time streaming video through adaptive memory and incremental processing.
6 Discussion, Limitations, and Future Work
VideoLLaMA3’s vision-centric framework supports strong performance across image and video understanding, while its remaining limitations concern video data, real-time processing, and generalization beyond image and video modalities.
- Discussion: Dynamic resolution adaptation and vision token compression provide flexible, efficient representations for varied image and video inputs.These designs support handling different visual formats with minimal information loss.
- Discussion: Multi-task fine-tuning exposes the model to interactive question answering, video captioning, and other downstream tasks.The paper associates this varied training with robust generalization across static and dynamic visual information.
- Limitations: Video-text data quality and diversity remain constrained, limiting generalization across different video domains and genres.The paper identifies lower annotation quality and limited diversity as the relevant boundary.
- Limitations: High-resolution and lengthy video inputs impose computational overhead that can hinder real-time performance.This constrains applications requiring real-time video processing, such as autonomous driving and live video analytics.
- Limitations and Future Work: Generalization to audio or speech remains unexplored, while adding modalities would pose architectural and training challenges.Future work proposes higher-quality video-text data, real-time inference optimization, multimodal expansion, and advanced post-training techniques.