Source-linked AI summary
Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, Junyang Lin
TL;DR
Existing vision-language models often use fixed image resolutions and limited positional representations, restricting sensitivity to visual detail, scale, spatial structure, and temporal dynamics. Qwen2-VL addresses these gaps with dynamic-resolution processing and multimodal positional embeddings, matching leading models across multimodal scenarios while understanding videos over 20 minutes.
Problem
Fixed image resolutions and one-dimensional positional representations limit LVLMs’ sensitivity to visual detail, scale, spatial structure, and temporal dynamics.
Method
Qwen2-VL uses naive dynamic resolution with 2D-RoPE and M-RoPE to process variable-resolution images and fuse spatial, temporal, and multimodal positional information.
Results
Qwen2-VL matches GPT-4o and Claude3.5-Sonnet across multimodal scenarios, surpasses other open-weight LVLMs, and understands videos over 20 minutes.
Takeaways & Limitations
Qwen2-VL provides a versatile open-weight LVLM for visual benchmarks, extended-duration video understanding, device operation, and multilingual image-text comprehension.
Abstract
from arXiv · showhide
We present the Qwen2-VL Series, an advanced upgrade of the previous Qwen-VL models that redefines the conventional predetermined-resolution approach in visual processing. Qwen2-VL introduces the Naive Dynamic Resolution mechanism, which enables the model to dynamically process images of varying resolutions into different numbers of visual tokens. This approach allows the model to generate more efficient and accurate visual representations, closely aligning with human perceptual processes. The model also integrates Multimodal Rotary Position Embedding (M-RoPE), facilitating the effective fusion of positional information across text, images, and videos. We employ a unified paradigm for processing both images and videos, enhancing the model's visual perception capabilities. To explore the potential of large multimodal models, Qwen2-VL investigates the scaling laws for large vision-language models (LVLMs). By scaling both the model size-with versions at 2B, 8B, and 72B parameters-and the amount of training data, the Qwen2-VL Series achieves highly competitive performance. Notably, the Qwen2-VL-72B model achieves results comparable to leading models such as GPT-4o and Claude3.5-Sonnet across various multimodal benchmarks, outperforming other generalist models. Code is available at https://github.com/QwenLM/Qwen2-VL .
1 Introduction
The introduction identifies fixed-resolution vision processing, static visual encoders, limited video-aware positional modeling, and underexplored LVLM scaling as key limitations. It presents Qwen2-VL as an open-weight series addressing these challenges through broad visual understanding, long-video comprehension, device-operation agents, and multilingual support.
- 1 Introduction: Fixed image input sizes limit LVLMs’ ability to capture information across varying resolutions and aspect ratios.Standard models commonly resize images to a fixed resolution such as 224×224.
- 1 Introduction: Static frozen CLIP-style vision encoders may produce inadequate representations for complex reasoning and intricate image details.Recent work has explored fine-tuning the vision transformer during LVLM training to address this limitation.
- 1 Introduction: One-dimensional positional embeddings limit models’ ability to understand the three-dimensional, dynamic nature of real-world video.Qwen2-VL uses separate temporal and spatial components in Rotary Position Embedding to represent dynamic content such as videos and streaming data.
- 1 Introduction: Scaling laws for LVLM training data and model parameters remain an open research area.The introduction emphasizes that LVLM scaling is less explored than scaling in large language models.
- 1 Introduction: Qwen2-VL comprises three open-weight models with total parameter counts of 2 billion, 8 billion, and 72 billion.The series is introduced as the newest addition to the Qwen family of large vision-language models.
- 1 Introduction: Qwen2-VL provides leading visual benchmark performance, understands videos over 20 minutes, supports device-operation agents, and offers multilingual image-context understanding.Its multilingual coverage extends beyond English and Chinese to European languages, Japanese, Korean, Arabic, Vietnamese, and others.
2 Approach
Qwen2-VL combines scalable Qwen2 language models with a fixed 675M-parameter ViT and introduces dynamic-resolution, multimodal-position, and unified image-video processing. It is trained through staged multimodal pre-training and instruction fine-tuning on diverse data, totaling 1.4 trillion pre-training tokens.
- Model Architecture: Qwen2-VL scales across 2B, 7B, and 72B models while keeping the ViT fixed at 675M parameters.The fixed ViT keeps its computational load constant across language-model scales.
- Naive Dynamic Resolution: Naive Dynamic Resolution processes images at any resolution by converting them into variable numbers of visual tokens.The ViT removes absolute position embeddings and adds 2D-RoPE to capture image position.
- Multimodal Rotary Position Embedding: M-RoPE decomposes rotary embeddings into temporal, height, and width components to model positional information across text, images, and video.For text, the components share position IDs; for images, temporal IDs remain constant while height and width IDs vary by token location.
- Unified Image and Video Understanding: Unified Image and Video Understanding trains on both image and video data, sampling videos at two frames per second and processing depth-two 3D tubes.This regimen is designed to preserve video information while supporting image understanding and video comprehension.
- Training: Qwen2-VL uses three-stage training: ViT-only training, full-parameter training, and final fine-tuning with the ViT locked.The pre-training data includes image-text pairs, OCR, interleaved articles, visual question answering, video dialogues, and image knowledge datasets.
- Training: 1.4 trillion tokens are processed across pre-training, including text and image tokens, with supervision provided only for text tokens.The first phase uses around 600 billion tokens, followed by an additional 800 billion tokens of image-related data.
3 Experiments
Section 3 evaluates Qwen2-VL through comparative visual benchmarks, capability-specific analyses, and ablation studies. It also includes performance comparisons with state-of-the-art models and internal multilingual OCR results involving GPT-4o.
- Comparative evaluation: The experiments compare Qwen2-VL against other models across a variety of visual benchmarks.The section frames this comparison as demonstrating the advantages of the proposed approach.
- Capability analysis: The capability analysis covers general visual perception, document understanding, multilingual image recognition, video comprehension, and agent abilities.These analyses examine specific aspects of the model’s visual and multimodal performance.
- Ablation study: The section presents an ablation study investigating several key components of the approach.The study is designed to examine the contribution of selected components.
- Comparative evaluation: Table 2 reports a performance comparison of Qwen2-VL models with state-of-the-art models.The table is titled “Performance Comparison of Qwen2-VL Models and State-of-the-art.”
- Capability analysis: Table 3 reports the performance of Qwen2-VL and GPT-4o on internal multilingual OCR benchmarks.The comparison specifically concerns multilingual optical character recognition.
3.1 Compare to SOTAs
Qwen2-VL is evaluated across visual benchmarks, video tasks, and agent-based assessments, achieving highly competitive same-scale performance and new state-of-the-art results. Its 72B model delivers top-tier results across most metrics and frequently surpasses GPT-4o and Claude 3.5-Sonnet.
- Qwen2-VL achieves highly competitive performance at the same scale, including new state-of-the-art results.
- The evaluation covers visual benchmarks, video tasks, and agent-based assessments.
- The 72B model consistently delivers top-tier performance across most evaluation metrics.
- Qwen2-VL frequently surpasses closed-source models including GPT-4o and Claude 3.5-Sonnet.
3.2 Quantitative Results
Qwen2-VL demonstrates strong quantitative performance across general visual question answering, OCR and document comprehension, mathematical reasoning, localization, video understanding, and interactive agent tasks. The 72B model is particularly competitive with or superior to leading baselines, while larger models generally improve video performance.
- General visual question answering: 77.8 surpasses the previous state-of-the-art (72.2) and GPT-4o (75.4), while Qwen2-VL-72B reaches 68.3 on MMStar versus the previous best of 67.1.These results demonstrate strong understanding of physical environments and integration of visual and textual information.
- OCR and document comprehension: Qwen2-VL achieves SoTA level on DocVQA, InfoVQA, TextVQA, and OCRBench, demonstrating textual comprehension across multiple image domains.The evaluation covers documents, high-resolution infographics, charts, naturalistic images, and mixed OCR tasks.
- OCR and document comprehension: Qwen2-VL surpasses general-purpose LVLMs in multilingual OCR, outperforming GPT-4o on the internal benchmark across all foreign languages except Arabic.It also outperforms existing LVLMs, including GPT-4o and Claude 3.5 Sonnet, on the publicly available MTVQA dataset.
- Mathematical reasoning: Qwen2-VL is evaluated on MathVista’s 6,141 examples and MathVision’s 3,040 visual math problems spanning 16 disciplines and five difficulty levels.These benchmarks assess mathematical reasoning in visual contexts.
- Visual localization: Qwen2-VL attains top-tier referring-expression results, with high-resolution detail perception producing significant improvements over Qwen-VL.Evaluation uses RefCOCO, RefCOCO+, and RefCOCOg, with performance compared against generalist and specialist models.
- Video understanding: Qwen2-VL-72B achieves the best performance on MVBench, PerceptionTest, and EgoSchema, while scaling improves video understanding; Video-MME evaluation extracts at most 768 frames per video.The video benchmarks span clips from a few seconds to one hour, and the frame limit may affect longer-video performance.
- Interactive agent tasks: 93.1 vs. 90.2 in Type Match and 53.2 vs. 50.0 in Exact Match show Qwen2-VL outperforming GPT-4o on function calling.Qwen2-VL’s OCR, particularly for Chinese characters, also outperforms GPT-4o, while VLN performance is comparable between the models.
3.3 Ablation Study
The ablations show that dynamic resolution, M-RoPE, and scaling improve Qwen2-VL performance, while appropriate image resolution matters more than simply increasing image size. M-RoPE also supports robust length extrapolation, and scaling benefits capabilities unevenly across tasks.
- Image Dynamic Resolution: Dynamic resolution lets image-token counts depend primarily on native image resolution, using min_pixels=100 × 28 × 28 and max_pixels=16384 × 28 × 28.Fixed resolution instead keeps a constant number of image tokens while preserving the original aspect ratio.
- Image Dynamic Resolution: Appropriate image resolution improves perceptual-task performance more reliably than simply increasing image size, with upscaling benefiting InfoVQA, HallusionBench, and OCRBench.The gains are attributed to increased computational load, whereas excessive resolution can hurt OCRBench.
- M-RoPE: M-RoPE outperforms 1D-RoPE on downstream tasks, particularly video benchmarks.The ablation uses Qwen2-1.5B and ViT-L backbones with pretrained models.
- M-RoPE: 80K tokens: M-RoPE maintains exceptional performance at maximum inference length despite training videos being limited to 16K tokens.Figure 5 evaluates Qwen2-VL-72B on Video-MME Medium Video and reports robust performance beyond the 16384-token maximum training length.
- Model Scale and Training Data: Larger models consistently improve performance, especially mathematical abilities, while smaller-scale models remain relatively strong on OCR-related tasks.Overall capability averages scores across benchmarks grouped into problem-solving, mathematics, document and table comprehension, general question-answering, and video comprehension.
- Model Scale and Training Data: More training tokens improve performance overall, although VQA results fluctuate during Qwen2-VL-7B’s second-stage pretraining.AI2D and InfoVQA are identified as tasks involving textual and graphical information.
4 Conclusion
Qwen2-VL presents three open-weight vision-language models that match top-tier multimodal models while surpassing other open-weight LVLMs, enabled by dynamic resolution and M-RoPE. The model weights are openly accessible for research and applications.
- 4 Conclusion: Qwen2-VL includes three open-weight models with 2, 8, and 72 billion parameters.The series comprises versatile large vision-language models spanning these total parameter counts.
- 4 Conclusion: Qwen2-VL matches GPT-4o and Claude3.5-Sonnet across multimodal scenarios while surpassing all other open-weight LVLMs.The comparison applies across a range of multimodal scenarios.
- 4 Conclusion: Naive dynamic resolution and M-RoPE enable Qwen2-VL to fuse information across modalities effectively.These mechanisms are introduced as core components of the series.
- 4 Conclusion: Qwen2-VL model weights are openly accessible for researchers and developers across applications and research projects.The authors state that open access is intended to support broader use and beneficial effects on society.
A Model Capabilities and Qualitative Examples
This section presents practical examples demonstrating the capabilities of Qwen2-VL.
- The section provides practical examples of Qwen2-VL.
A.1 General Chat and OCR
Qwen2-VL improves visual understanding by accurately describing, identifying, and contextualizing complex image information. Its text recognition is also substantially stronger, especially for Chinese and English, with demonstrated multilingual and dense-text capabilities.
- General Chat: Qwen2-VL accurately describes and identifies complex information in images while providing detailed background and answering related questions.The models’ general visual understanding is presented as more capable across image-based description, identification, and question answering.
- OCR: Qwen2-VL shows improved recognition of Chinese and English text within images.The passage specifically highlights significant gains in recognizing both Chinese and English visual text.
- General Chat: The models recognize layouts and object attributes, including the arrangement and colors of cubes and flowers in photographs.Figures illustrate cube-layout and color identification, alongside flower recognition.
- OCR: The model recognizes multilingual text in images and can convert lengthy, densely packed Chinese characters into accurate, fluent English.Examples include multilingual text recognition and direct conversion of dense Chinese text into standard English.
A.2 Information extraction and Visual Reasoning
Qwen2-VL enhances visual reasoning by interpreting complex representations such as flowcharts, diagrams, and other symbolic systems. The model also demonstrates capabilities in mathematical, algorithmic, web-content, and OCR-related tasks.
- Qwen2-VL interprets complex visual representations, including flowcharts, diagrams, and other symbolic systems.
- The model solves mathematical problems involving surface area and volume with step-by-step explanations.
- Algorithmic Problem Solving: The model demonstrates adeptness in solving algorithmic problems, including executing matrix-movement commands to determine a snake’s final position.
- The model recognizes content from web pages and performs OCR with mathematical reasoning and format following.
A.3 Video Understanding
Qwen2-VL demonstrates adeptness in recognizing multi-round video chat and understanding multiple videos.
- The model displays adeptness in recognizing multi-round video chat.
- The model displays adeptness in multi-video understanding.
A.4 Visual Agent Capability
Qwen2-VL demonstrates visual-agent capabilities spanning precise image localization, webpage understanding, multimodal task execution, code generation and verification, and step-by-step UI operations. The examples show it can connect visual information to actions and external tools to complete user requests.
- Visual localization: Qwen2-VL accurately locates specific elements within images, including identifying a red car.This demonstrates the model’s location-task capability.
- Visual agent tasks: The model annotates webpage screenshots and poses questions about their content, demonstrating visual-agent potential.It interprets screenshot content rather than only recognizing isolated objects.
- Tool-using task execution: The model identifies a destination and arrival time, calls a 24-hour weather query function, enters the destination, and extracts weather for the arrival time.This workflow enables it to answer the user’s question from image-derived information and query results.
- Code generation and verification: The model analyzes flowcharts and tables, drafts or writes code, executes it through the code interpreter, and successfully produces the requested simulation or bar-chart visualization.For the table task, it calculates each row’s mean and plots the results as a bar chart.
- UI operations: Qwen2-VL understands UI-operation queries, uses predefined actions step by step, and can recognize cards and use Hit and Stand to play blackjack.These examples illustrate interaction with interfaces through agent actions.