Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

Visual Resolution

• Qwen2-VL uses dynamic-resolution training with 2D RoPE in the ViT to preserve visual information across varying spatial scales.This targets fixed-size image processing, which can lose detail in high-resolution inputs and limit scale-sensitive perception.Source: §1 · p. 2

Video Understanding

• M-RoPE separates temporal and spatial position components, enabling unified modeling of video dynamics and three-dimensional real-world content.The approach addresses limitations of one-dimensional positional embeddings for temporal and spatial reasoning.Source: §1 · pp. 2–3

Qwen2-VL Series

• Qwen2-VL introduces 2B, 8B, and 72B open-weight models to investigate scaling with model parameters and training data.LVLM scaling laws remain comparatively underexplored relative to LLM scaling.Source: §1 · p. 3

Visual Benchmarking

• Qwen2-VL achieves leading performance across visual benchmarks including DocVQA, InfoVQA, RealWorldQA, MTVQA, and MathVista.The reported coverage spans varying image resolutions and aspect ratios.Source: §1 · p. 3

Video Capabilities

• Qwen2-VL understands videos longer than 20 minutes, supporting video question answering, dialogue, and content creation.This extends video comprehension to extended-duration content.Source: §1 · p. 3

Agents and Multilingual Support

• Qwen2-VL supports multilingual image-text understanding and robust visual device operation for phones, robots, and other devices.Supported image languages extend beyond English and Chinese to include European languages, Japanese, Korean, Arabic, Vietnamese, and others.Source: §1 · p. 3

2.1 Model Architecture

• Qwen2-VL uses a 675M-parameter ViT across 2B, 7B, and 72B language-model scales, with dynamic resolution, 2D-RoPE, and visual-token compression.Images are packed into sequences under GPU-memory limits; a 224 × 224 image compresses to 66 tokens before entering the LLM.Source: §2 · p. 3, §2.1 · p. 4
• M-RoPE decomposes positional encoding into temporal, height, and width components, modeling text, images, and videos while enabling longer-sequence extrapolation.Videos increment temporal IDs across frames, while image and video height-width IDs follow spatial token positions.Source: §2.1 · pp. 4–5, Figure 3 · p. 5
• Qwen2-VL jointly trains on images and videos, sampling videos at two frames per second and using depth-two 3D convolutions while limiting each video to 16384 tokens.Each image is treated as two identical frames, balancing long-video comprehension with training efficiency.Source: §2.1 · p. 5

2.2 Training

• Training follows three stages: ViT-only learning, full-parameter training, and LLM-only instruction fine-tuning over diverse multimodal data totaling 1.4 trillion tokens.The corpus includes image-text pairs, OCR, interleaved articles, visual question answering, video dialogues, and image knowledge data; supervision is applied only to text tokens.Source: §2.2 · pp. 5–6

2.2.1 Data Format.

• Instruction data uses ChatML and supports visual grounding and visual-agent interaction through normalized bounding boxes, object references, and iterative tool actions.Agent tasks are framed as sequential decision-making cycles of observation, reasoning, planning, action execution, and new observations.Source: §2.2.1 · pp. 6–7

2.3 Multimodal Model Infrastructure

• Multimodal training infrastructure combines CPFS and OSS storage with 3D parallelism, memory-saving techniques, flash-attention, and fused operators.The system uses caching for long-video decoding and 1F1B pipeline parallelism for Qwen2-VL 72B training.Source: §2.3 · p. 8

Capability Evaluation

• The capability analysis examines general visual perception, document understanding, multilingual image recognition, video comprehension, and agent abilities.The study also evaluates Qwen2-VL and GPT-4o on internal multilingual OCR benchmarks.Source: §3 · p. 8, Table 3 · p. 9

Ablation Study

• An ablation study investigates several key components of the proposed approach.This analysis is presented after the comparative and capability evaluations.Source: §3 · p. 8

Comparative Evaluation

• The study compares Qwen2-VL models with state-of-the-art systems across visual benchmarks.Table 2 presents the performance comparison of Qwen2-VL models and state-of-the-art methods.Source: §3 · p. 8, Table 2 · p. 9

3.1 Compare to SOTAs

• Qwen2-VL-72B delivers top-tier performance across most evaluation metrics, frequently surpassing GPT-4o and Claude 3.5-Sonnet, with a significant advantage in document understanding.The model achieves new state-of-the-art results at the same scale across visual benchmarks, video tasks, and agent-based assessments.Source: §3.1 · p. 9
• On MMMU, Qwen2-VL-72B still lags behind GPT-4o, indicating room for improvement on more complex and challenging problem sets.Source: §3.1 · p. 9

3.2.1 General Visual Question Answering

• Qwen2-VL-72B achieves leading results across diverse VQA benchmarks, including 77.8 on RealWorldQA, 74.0 on MMVet, 71.7 on MMT-Bench, and 2482.7 on MME.It also reaches 86.5 on English MMBench and 86.6 on Chinese MMBench, matching or surpassing reported state-of-the-art results.Source: §3.2.1 · p. 10

3.2.2 Document and Diagrams Reading

• Qwen2-VL reaches state-of-the-art levels on DocVQA, InfoVQA, TextVQA, and OCRBench, demonstrating broad comprehension of textual content in images.The evaluation covers documents, infographics, charts, naturalistic images, mathematical formulas, information extraction, and scientific diagrams.Source: §3.2.2 · p. 11

3.2.3 Multilingual Text Recognition and Understanding

• Qwen2-VL surpasses existing general-purpose LVLMs on multilingual OCR, including GPT-4o and Claude 3.5 Sonnet on MTVQA and GPT-4o across foreign languages except Arabic internally.The internal multilingual comparison is reported in Table 3.Source: §3.2.3 · p. 11

3.2.4 Mathematical Reasoning

• Qwen2-VL achieves 70.5 on MathVista and establishes a new open-source benchmark on MathVision with 25.9.MathVision contains 3,040 visually embedded competition problems spanning 16 disciplines and five difficulty levels.Source: §3.2.4 · p. 11

3.2.5 Referring Expression Comprehension

• Qwen2-VL attains top-tier referring-expression results on RefCOCO, RefCOCO+, and RefCOCOg, with high-resolution perception improving over Qwen-VL.The comparison is presented in Table 6 and includes generalist and specialized models.Source: §3.2.5 · p. 11, Table 6 · p. 12

3.2.6 Video Understanding

• Qwen2-VL-72B achieves the best performance on MVBench, PerceptionTest, and EgoSchema across evaluated video-understanding models.For Video-MME, evaluation limited each video to 768 frames, potentially affecting longer videos of up to one hour.Source: Table 4 · p. 10, §3.2.6 · pp. 11–12

3.2.7 Visual Agent

• Qwen2-VL exceeds GPT-4o in function calling, scoring 93.1 versus 90.2 in Type Match and 53.2 versus 50.0 in Exact Match.The evaluation uses an internally constructed dataset because public LVLM function-calling benchmarks are unavailable; GPT-4o’s weaker performance is attributed partly to conservative tool use and inferior Chinese OCR.Source: Table 5 · p. 10, §3.2.7 · pp. 12–13
• Across visual-agent tasks, Qwen2-VL surpasses GPT-4 and prior state-of-the-art on UI operations, exceeds ThinkBot on ALFRED, and outperforms alternatives across card games.Its vision-language navigation performance is comparable to GPT-4o but trails specialized VLN models because generated 3D maps remain incomplete and unstructured.Source: §3.2.7 · p. 13

3.3.1 Dynamic Resolution

• Dynamic resolution matches top-tier fixed-resolution performance across benchmarks while using fewer tokens, and appropriate image sizing matters more than simply increasing image size.Small-image upscaling improves InfoVQA, HallusionBench, and OCRBench, but excessive enlargement severely harms OCRBench; increasing min_pixels has negligible effect on MMMU.Source: §3.3.1 · pp. 14–15

3.3.2 M-RoPE

• M-RoPE outperforms 1D-RoPE on downstream tasks, particularly video benchmarks, and supports robust Video-MME performance up to 80K inference tokens despite 16K-token training.The length-extrapolation evaluation uses Qwen2-VL-72B on medium-length videos.Source: Table 8 · p. 15, §3.3.2 · p. 15, Figure 5 · p. 15

3.3.3 Model Scaling

• Performance consistently improves with model size, especially for mathematical abilities, whereas smaller models remain relatively strong on optical character recognition tasks.Capability scores average benchmarks covering college-level problem solving, mathematics, document and table comprehension, general question answering, and video comprehension.Source: §3.3.3 · pp. 15–16, Figure 6 · p. 16
• More second-stage pretraining tokens improve performance overall, with steady gains on AI2D and InfoVQA but fluctuations on visual question-answering tasks.The training-token analysis concerns Qwen2-VL-7B and distinguishes textual-graphical understanding tasks from broader VQA tasks.Source: §3.3.3 · p. 16, Figure 6 · p. 16

• Qwen2-VL matches GPT-4o and Claude3.5-Sonnet across multimodal scenarios and surpasses other open-weight LVLMs.Source: §4 · p. 16
• The series offers open-weight models with 2, 8, and 72 billion parameters, enabling broad research and application use.The model weights are openly accessible to researchers and developers.Source: §4 · p. 16
• Naive dynamic resolution and M-RoPE fuse multimodal information and support understanding videos over 20 minutes long.Source: §4 · p. 16
• Qwen2-VL combines advanced reasoning and decision-making with integration opportunities for mobile phones, robots, and other devices.Source: §4 · p. 16
• The models understand multilingual text in images, including most European languages, Japanese, Korean, Arabic, and Vietnamese.Source: §4 · p. 16

A Model Capabilities and Qualitative Examples

• The section introduces practical examples of Qwen2-VL.Source: A Model Capabilities and Qualitative… · p. 25
• The examples focus on demonstrating Qwen2-VL in practice.Source: A Model Capabilities and Qualitative… · p. 25
• Qwen2-VL is the subject of the practical examples presented.Source: A Model Capabilities and Qualitative… · p. 25

General Chat

• Qwen2-VL accurately describes and identifies complex image information, provides detailed background, and answers related questions.Figure 7 shows identification of cube layouts and colors, while Figure 8 shows flower recognition in photographs.Source: §A.1 · p. 25, Figure 7 · p. 25, Figure 8 · p. 26

OCR

• Qwen2-VL improves recognition of Chinese and English text within images, including multilingual and densely packed Chinese text.Figures 10, 12, and 13 depict multilingual text recognition, while Figure 11 shows conversion of dense Chinese characters into fluent standard English.Source: §A.1 · p. 25, Figure 10 · p. 28, Figure 11 · p. 29

General Chat

• Qwen2-VL generates literary writing in multiple languages from visual stimuli.Figure 9 illustrates multilingual literary generation based on images.Source: Figure 9 · p. 27

Visual Reasoning

• Qwen2-VL interprets complex visual representations, including flowcharts, diagrams, and other symbolic systems.This enhanced visual reasoning capability supports comprehension beyond ordinary image recognition.Source: §A.2 · p. 32

Mathematical Reasoning

• The model solves mathematical problems involving geometric surface area and volume with step-by-step explanations.The examples also show mathematical problem-solving aptitude.Source: Figure 14 · p. 32, Figure 15 · p. 33

Algorithmic Problem Solving

• The model solves algorithmic tasks by translating a matrix-and-command specification into a step-by-step solution for the Snake in Matrix problem.The task requires returning the snake’s final position after executing commands.Source: Figure 16 · p. 35

Information Extraction

• The examples demonstrate visual information extraction through web-page content recognition, OCR, format following, and large-image OCR.These capabilities are represented across the web-page, OCR, formatting, and large-image examples.Source: Figure 17 · p. 36, Figure 18 · p. 37, Figure 19 · p. 38

A.3 Video Understanding

• The model displays adeptness in recognizing multi-round video chat.Source: Figure 22 · p. 41
• The model displays adeptness in recognizing multi-video understanding.Source: Figure 23 · p. 42

• Qwen2-VL accurately locates specific elements in images, such as identifying a red car.Source: §A.4 · p. 43, Figure 24 · p. 43
• The model annotates webpage screenshots, poses questions about their content, and retrieves weather information by identifying a destination, arrival time, and query results.Source: Figure 25 · p. 44, Figure 26 · p. 45
• Qwen2-VL interprets flowcharts and tables, drafts and executes code, and produces visualized results through the code interpreter.Source: Figure 27 · p. 46, §A.4 · p. 47, Figure 28 · p. 48
• As a visual agent, Qwen2-VL follows predefined UI actions step by step and can interact with interfaces, including playing blackjack using Hit and Stand.Source: Figure 30 · p. 51, Figure 31 · p. 52