Source-linked AI summary
Qwen2.5-VL Technical Report
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, Junyang Lin
TL;DR
Qwen2.5-VL addresses limitations in multimodal models through improved visual understanding, interaction, and training-data quality. It combines windowed visual attention, dynamic frame-rate processing, absolute-time encoding, and curated data, achieving advances across visual question answering, document parsing, object grounding, video understanding, and agent functionality.
Problem
Existing vision-language models remain competent across tasks but fall short of exceptional fine-grained performance, while noisy training data can undermine robust multimodal task handling.
Method
Qwen2.5-VL combines window attention, dynamic FPS training, absolute-time encoding, and extensive pre-training and supervised fine-tuning data curation.
Results
Qwen2.5-VL advances visual recognition, object localization, document parsing, long-video comprehension, and visual question answering, with the 72B model matching or surpassing leading models, particularly on document and diagram understanding.
Takeaways & Limitations
The model supports applications ranging from edge AI to high-performance computing and interactive visual-agent tasks on computers and mobile devices.
Takeaways & Limitations
Optimal alignment between visual and textual modalities in chain-of-thought reasoning remains an ongoing challenge despite rule-based and model-driven filtering.
Abstract
from arXiv · showhide
We introduce Qwen2.5-VL, the latest flagship model of Qwen vision-language series, which demonstrates significant advancements in both foundational capabilities and innovative functionalities. Qwen2.5-VL achieves a major leap forward in understanding and interacting with the world through enhanced visual recognition, precise object localization, robust document parsing, and long-video comprehension. A standout feature of Qwen2.5-VL is its ability to localize objects using bounding boxes or points accurately. It provides robust structured data extraction from invoices, forms, and tables, as well as detailed analysis of charts, diagrams, and layouts. To handle complex inputs, Qwen2.5-VL introduces dynamic resolution processing and absolute time encoding, enabling it to process images of varying sizes and videos of extended durations (up to hours) with second-level event localization. This allows the model to natively perceive spatial scales and temporal dynamics without relying on traditional normalization techniques. By training a native dynamic-resolution Vision Transformer (ViT) from scratch and incorporating Window Attention, we reduce computational overhead while maintaining native resolution. As a result, Qwen2.5-VL excels not only in static image and document understanding but also as an interactive visual agent capable of reasoning, tool usage, and task execution in real-world scenarios such as operating computers and mobile devices. Qwen2.5-VL is available in three sizes, addressing diverse use cases from edge AI to high-performance computing. The flagship Qwen2.5-VL-72B model matches state-of-the-art models like GPT-4o and Claude 3.5 Sonnet, particularly excelling in document and diagram understanding. Additionally, Qwen2.5-VL maintains robust linguistic performance, preserving the core language competencies of the Qwen2.5 LLM.
1 Introduction
Qwen2.5-VL addresses bottlenecks in multimodal models with improved perception, grounding, document parsing, video understanding, and device interaction. Its contributions combine efficient visual processing, temporal modeling, and expanded training data.
- Current vision-language models face computational complexity, limited contextual understanding, weak fine-grained perception, and inconsistent performance across sequence lengths.
- Qwen2.5-VL uses window attention, dynamic FPS sampling, absolute-time MRoPE, and expanded pre-training data to improve multimodal processing.The pre-training corpus grows from 1.2 trillion to 4.1 trillion tokens.
- The model parses multilingual and multi-scene documents containing handwriting, tables, charts, chemical formulas, and music sheets.
- Qwen2.5-VL grounds objects through detection, pointing, and counting with absolute coordinates and JSON outputs.
- Dynamic temporal processing supports understanding videos lasting hours and extracting event segments at second-level resolution.
- Grounding, reasoning, and decision-making support agent functionality on smartphones and computers.
2 Approach
The approach section outlines the Qwen2.5-VL architecture updates and summarizes its data and training details.
- The section introduces the architectural updates and provides an overview of the data and training details.
2.1 Model Architecture
Qwen2.5-VL combines a Qwen2.5 language model, a redesigned native-resolution vision encoder, and an MLP-based feature merger. Dynamic spatial and temporal processing preserves input scale while reducing visual computation and improving video-time alignment.
- Large Language Model: Qwen2.5-VL uses a Qwen2.5 language model initialized from pretrained weights and modified with MRoPE aligned to absolute time.
- Vision Encoder: The redesigned ViT uses 2D-RoPE and window attention to support native resolutions while accelerating visual-encoder computation.Input dimensions are resized to multiples of 28 before patch processing.
- MLP-based Vision-Language Merger: Adjacent groups of four patch features are concatenated and projected through a two-layer MLP into the LLM text-embedding dimension.This compresses variable-length image feature sequences and reduces computational costs.
- Fast and Efficient Vision Encoder: Windowed attention makes most visual computation scale linearly with patch count, while only four layers use full self-attention.The maximum window size is 112×112, corresponding to 8×8 patches.
- Fast and Efficient Vision Encoder: For video, 14×14 patches from two consecutive frames are grouped to reduce the number of tokens sent to the language model.
- Fast and Efficient Vision Encoder: Dynamic native-resolution sampling trains the model on diverse image sizes and aspect ratios across staged ViT pre-training, alignment, and fine-tuning.
- Native Dynamic Resolution and Frame Rate: Actual image dimensions represent spatial features, while dynamic FPS training and timestamp-aligned MRoPE encode scale and temporal dynamics without additional computational overhead.
- Multimodal Rotary Position Embedding Aligned to Absolute Time: Aligning temporal MRoPE with absolute time lets the model learn consistent temporal alignment across videos sampled at different FPS rates.
2.2 Pre-Training
Qwen2.5-VL’s pre-training combines expanded multimodal data with native-resolution grounding, document parsing, OCR, video, and agent-focused data construction. These datasets are cleaned, scored, and formatted to support broad visual understanding and interaction.
- Pre-training data expanded from 1.2 trillion to approximately 4 trillion tokens across diverse multimodal sources.Sources include image captions, interleaved image-text data, OCR, visual knowledge, academic questions, localization, and document parsing data.
- Interleaved Image-Text Data: Interleaved image-text data is cleaned and scored for text quality, image-text relevance, complementarity, and information-density balance.The pipeline applies standard cleaning followed by four scoring dimensions using an internal evaluation model.
- Grounding Data with Absolute Position Coordinates: Grounding data uses absolute coordinates based on actual image dimensions to represent bounding boxes and points across resolutions.The training corpus includes bounding boxes, referring expressions, open-vocabulary categories, and point-based grounding data.
- Document Omni-Parsing Data: Document omni-parsing data represents layouts, text, charts, formulas, captions, images, music sheets, and chemical formulas in structured formats.This design integrates multimodal document elements for more efficient and accurate document understanding and transformation.
- Training data includes multilingual OCR, dynamically sampled-FPS video data, long-video captions, and screenshots with UI grounding for agent capabilities.Agent data covers mobile, web, and desktop interfaces, while video data includes captions and grounding timestamps.
2.3 Post-training
Qwen2.5-VL uses a two-stage post-training process combining supervised fine-tuning and preference optimization over diverse, quality-filtered multimodal data. Rejection sampling further emphasizes correct, coherent reasoning while modality alignment remains challenging.
- The SFT dataset contains approximately 2 million entries split evenly between pure text and multimodal data.It covers single- and multi-turn interactions, multiple image scenarios, and specialized VQA, OCR, grounding, video, and agent tasks.
- Data Filtering Pipeline: A two-stage filtering pipeline categorizes QA pairs into domains and applies domain-specific rule-based and model-based quality checks.Filtering removes repetition, incomplete responses, improper formatting, irrelevant content, and answers lacking correctness, completeness, clarity, relevance, or helpfulness.
- Rejection Sampling for Enhanced Reasoning: Rejection sampling retains ground-truth-matching outputs for multi-step reasoning tasks and removes code-switching, excessive length, and repetitive responses.The resulting data emphasizes accurate and coherent reasoning for mathematics, code generation, and domain-specific VQA.
- Rejection Sampling for Enhanced Reasoning: Visual-language chain-of-thought filtering verifies whether intermediate reasoning steps integrate visual and textual information, but optimal modality alignment remains unresolved.The authors identify modality alignment as an ongoing challenge requiring further advances.
- Post-training combines Supervised Fine-Tuning and Direct Preference Optimization while keeping Vision Transformer parameters frozen.SFT adapts the model to diverse multimodal tasks, and DPO refines behavior through preference optimization.
3 Experiments
The experiments evaluate Qwen2.5-VL against state-of-the-art models across multiple sub-capabilities. The section introduces the overall comparison before presenting capability-specific results.
- Experiments first compare Qwen2.5-VL with current state-of-the-art models, then evaluate performance across various sub-capabilities.
- The evaluation is organized around the model overview and subsequent capability-specific performance analyses.
- The section frames results as a broad assessment rather than a single-task evaluation.
3.1 Comparison with the SOTA Models
Qwen2.5-VL-72B is compared with leading closed- and open-source models across college-level, mathematical, and general visual question-answering benchmarks. The reported results show competitive or state-of-the-art performance across these evaluations.
- 70.2 on MMMU and 51.1 on MMMU-Pro show strong college-level performance, with MMMU-Pro comparable to GPT-4o.
- 74.8 on MathVista surpasses the previous open-source state-of-the-art score of 72.3.
- 88.6 on MMbench-EN slightly exceeds the previous best score of 88.3, alongside strong results on MuirBench and BLINK.
3.2 Performance on Pure Text Tasks
Qwen2.5-VL is evaluated on pure text benchmarks spanning general, mathematics and science, coding, and alignment tasks against similarly sized language models.
- Pure Text Evaluation: The evaluation compares Qwen2.5-VL with similarly sized large language models across general, mathematics and science, coding, and alignment benchmarks.The supplied passage introduces the benchmark categories and comparison setup but does not provide the individual scores.
3.3 Quantitative Results
Qwen2.5-VL demonstrates broad capability across visual question answering, document understanding, grounding, video comprehension, and GUI-agent tasks. Reported results include strong benchmark scores, state-of-the-art OCR performance, precise localization, and competitive interaction with real-world interfaces.
- General Visual Question Answering: 88.4% accuracy on MMBench-EN-V1.1 surpasses InternVL2.5 (78B) and Claude-3.5 Sonnet-0620.Qwen2.5-VL-72B achieves this result in visual detail comprehension and reasoning.
- General Visual Question Answering: 63.2 on MME-RealWorld and 70.7 on MuirBench demonstrate strong performance in high-resolution real-world and multi-image understanding.The passage describes both results as state-of-the-art or leading scores in their respective evaluations.
- Document Understanding and OCR: Qwen2.5-VL-72B sets new state-of-the-art results on CC-OCR and OmniDocBench for multilingual, multi-scene, and built-in document-element parsing.The evaluated document elements include handwriting, tables, charts, chemical formulas, and mathematical expressions.
- Grounding and Counting: 43.1 mAP on ODinW-13 shows leading open-vocabulary object detection performance, while box and point grounding support detailed spatial reasoning.The model is reported to lead across box-grounding, point-grounding, and counting benchmarks.
- Video Understanding and Grounding: 50.9 mIoU on Charades-STA surpasses GPT-4o for timestamp-based event localization in video.Synchronized MRoPE is associated in the passage with timestamp referencing, temporal grounding, and dense captioning capabilities.
- Agent: 87.1% accuracy on ScreenSpot and 43.6% on ScreenSpot Pro establish strong GUI grounding, with competitive online agent performance.Qwen2.5-VL-72B outperforms listed baselines on AndroidWorld and MobileMiniWob++ and is comparable on OSWorld without auxiliary marks.
4 Conclusion
Qwen2.5-VL is presented as a vision-language model series advancing multimodal understanding and interaction across static, dynamic, and agentic tasks. Its 72B model matches or surpasses leading models, while smaller variants offer competitive performance across use cases.
- Conclusion: Qwen2.5-VL advances visual recognition, object localization, document parsing, and long-video comprehension across static and dynamic tasks.The conclusion highlights native dynamic-resolution processing and absolute time encoding for diverse inputs.
- Conclusion: Window Attention reduces computational overhead without sacrificing native-resolution fidelity.The conclusion identifies this design as part of the model’s input-processing innovations.
- Conclusion: The Qwen2.5-VL-72B flagship matches or surpasses GPT-4o and Claude 3.5 Sonnet, particularly in document and diagram understanding.The model also maintains strong performance on pure text tasks.
- Conclusion: Qwen2.5-VL-7B and Qwen2.5-VL-3B outperform similarly sized competitors, supporting applications from edge AI to high-performance computing.The conclusion characterizes the model series as offering efficiency and versatility.
5 Authors
The paper lists core contributors and additional contributors to the Qwen2.5-VL work.
- Core Contributors: The core contributors include Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, and other listed authors.The passage provides the complete core-contributor list.
- Contributors: Additional contributors include An Yang, Binyuan Hui, Bowen Yu, Chen Cheng, and other listed authors.The passage provides the complete additional-contributor list.