PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen2.5-VL Technical Report

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, Junyang Lin

arXiv:2502.13923v1cs.CVcs.CL

TL;DR

Large vision-language models remain competent across tasks but fall short of exceptional performance, with fine-grained visual tasks identified as a foundational capability area. Qwen2.5-VL combines window attention, dynamic-resolution processing, and absolute-time encoding; its 72B flagship matches or surpasses leading models, particularly in document and diagram understanding.

  • Problem

    Existing large vision-language models are competent across tasks but still fall short of exceptional performance, with fine-grained visual tasks identified as a foundational capability area.

    Source: §1 · p. 2

  • Method

    Qwen2.5-VL combines window attention in the vision encoder with dynamic-FPS video sampling, absolute-time-aligned multimodal position encoding, and curated training data.

    Source: §1 · p. 2, §2.1 · p. 3, §2.1.1 · p. 4

  • Results

    Qwen2.5-VL-72B matches or surpasses leading models like GPT-4o and Claude 3.5 Sonnet, particularly in document and diagram understanding.

    Source: §4 · p. 15

  • Takeaways & Limitations

    The model series caters to applications ranging from edge AI to high-performance computing.

    Source: §4 · p. 15

    Despite filtering intermediate reasoning steps, optimal alignment between visual and textual modalities remains an ongoing challenge.

    Source: §2.3.3 · p. 10

Abstract

from arXiv · show

We introduce Qwen2.5-VL, the latest flagship model of Qwen vision-language series, which demonstrates significant advancements in both foundational capabilities and innovative functionalities. Qwen2.5-VL achieves a major leap forward in understanding and interacting with the world through enhanced visual recognition, precise object localization, robust document parsing, and long-video comprehension. A standout feature of Qwen2.5-VL is its ability to localize objects using bounding boxes or points accurately. It provides robust structured data extraction from invoices, forms, and tables, as well as detailed analysis of charts, diagrams, and layouts. To handle complex inputs, Qwen2.5-VL introduces dynamic resolution processing and absolute time encoding, enabling it to process images of varying sizes and videos of extended durations (up to hours) with second-level event localization. This allows the model to natively perceive spatial scales and temporal dynamics without relying on traditional normalization techniques. By training a native dynamic-resolution Vision Transformer (ViT) from scratch and incorporating Window Attention, we reduce computational overhead while maintaining native resolution. As a result, Qwen2.5-VL excels not only in static image and document understanding but also as an interactive visual agent capable of reasoning, tool usage, and task execution in real-world scenarios such as operating computers and mobile devices. Qwen2.5-VL is available in three sizes, addressing diverse use cases from edge AI to high-performance computing. The flagship Qwen2.5-VL-72B model matches state-of-the-art models like GPT-4o and Claude 3.5 Sonnet, particularly excelling in document and diagram understanding. Additionally, Qwen2.5-VL maintains robust linguistic performance, preserving the core language competencies of the Qwen2.5 LLM.

1 Introduction

Qwen2.5-VL targets bottlenecks in efficiency, contextual understanding, fine-grained perception, and sequence-length robustness by combining visual perception with language reasoning for real-world interaction. It builds on the Qwen2.5 LLM and multimodal QA data to strengthen perception and reasoning.

Introduction

  • Qwen2.5-VL continues Qwen’s open-source approach and is reported to match or surpass top-tier closed-source models across various benchmarks.Its technical contributions include visual-encoder window attention, dynamic-FPS sampling, absolute-time MRoPE, and scaling pretraining data from 1.2 trillion to 4.1 trillion tokens.Source: §1 · p. 2
  • The model extends text recognition to multilingual, multi-scene document parsing, including handwriting, tables, charts, chemical formulas, and music sheets.The report presents this as an upgrade to omni-document parsing.Source: §1 · p. 2
  • Qwen2.5-VL improves object detection, pointing, and counting, with grounding in absolute-coordinate and JSON formats for spatial reasoning.Source: §1 · p. 2
  • Dynamic resolution extended across time enables understanding of videos lasting hours and grounding event segments to within seconds.Dynamic FPS sampling supports video understanding across varied sampling rates, while temporal MRoPE aligns time IDs to absolute time.Source: §1 · p. 2
  • Advanced grounding, reasoning, and decision-making support agent functionality on smartphones and computers.Source: §1 · p. 2

2 Approach

Section 2 outlines Qwen2.5-VL’s architectural updates and summarizes its data and training details.

  • The approach section covers Qwen2.5-VL’s architectural updates, data, and training details.Source: §2 · p. 3

2.1 Model Architecture

Qwen2.5-VL combines a pretrained Qwen2.5 language model, a redesigned vision encoder, and an MLP merger that compresses visual features. Its architecture supports native-resolution image processing and video inputs with spatial and temporal information encoded directly.

2.1 Model Architecture

  • The architecture pairs a Qwen2.5-pretrained LLM with a redesigned ViT and an MLP that groups adjacent patches to compress image features before projection into the LLM’s text-embedding dimension.This reduces computational costs while allowing image-feature sequences of varying lengths to be dynamically compressed.Source: §2.1 · pp. 3–4

2.1.1 Fast and Efficient Vision Encoder

  • Windowed attention makes most ViT layers scale linearly with patch count; four layers retain full attention, preserving native resolution without padding regions smaller than 112×112.The ViT uses 2D RoPE and 14×14 patches; video processing groups consecutive frames, and training from scratch includes CLIP pre-training, vision-language alignment, end-to-end fine-tuning, and dynamic native-resolution sampling.Source: §2.1.1 · pp. 4–5

2.1.2 Native Dynamic Resolution and Frame Rate

  • Images become variable-length token sequences, while spatial coordinates use actual image dimensions; video training adapts to variable frame rates and aligns MRoPE IDs to timestamps without extra computational overhead.Using actual dimensions lets the model learn scale information inherently; intervals between temporal IDs encode video tempo.Source: §2.1.2 · p. 5

2.1.3 Multimodal Rotary Position Embedding Aligned to Absolute Time

  • Unlike Qwen2-VL’s frame-count-based temporal IDs, Qwen2.5-VL aligns MRoPE’s temporal component to absolute time, supporting consistent temporal alignment across videos sampled at different frame rates.MRoPE separates temporal, height, and width components; intervals between temporal IDs carry timing information.Source: §2.1.3 · p. 5

2.2 Pre-Training

Qwen2.5-VL pre-training uses a substantially expanded, diverse multimodal dataset and a three-phase training recipe. The data and schedule target document understanding, grounding, video, and agent capabilities.

2.2.1 Pre-Training Data

  • Pre-training data grew from 1.2 trillion to approximately 4 trillion tokens, spanning image-text, OCR, grounding, documents, video, and agent interactions, with composition adjusted across stages.The dataset combines cleaned web data and synthesized data.Source: §2.2.1 · p. 5
  • Interleaved image-text data is cleaned and scored for text quality, image-text relevance, complementarity, and information-density balance to support reasoning and coherent multimodal generation.The pipeline first applies standard data cleaning, then uses an internal evaluation model for four-stage scoring.Source: §2.2.1 · p. 6
  • Grounding uses actual input-image dimensions for box and point coordinates, with referring-expression and pointing data plus over 10,000 categories for open-vocabulary detection.The training set draws on public and proprietary data, synthetic examples, and queries containing non-existent categories for extreme detection.Source: §2.2.1 · p. 6
  • Document training represents tables, charts, equations, images, music sheets, and chemical formulas in HTML with layout, reading order, and module coordinates.This unified format encodes document elements, layout, text, charts, and illustrations in structured form for processing and transformation.Source: §2.2.1 · pp. 6–7
  • OCR training combines synthetic, open-source, and in-house data, including multilingual text images; the corpus also includes 1 million synthesized chart samples and 6 million filtered real-world tables.The OCR data covers languages including French, Arabic, Japanese, Korean, and Vietnamese; low-confidence, overlapping, and sparse tables are filtered out.Source: §2.2.1 · p. 7
  • Video training dynamically samples frame rates, adds synthesized multi-frame captions for videos over half an hour, and uses second-based and hmsf timestamps for grounding.These data choices target videos with varying FPS, long-video understanding, and timestamp output in multiple formats.Source: §2.2.1 · p. 7
  • Agent data pairs mobile, web, and desktop screenshots with captions and UI grounding, while function-call trajectories include annotated step-by-step reasoning to improve real-world robustness.Low-quality reasoning annotations are filtered using a model-based process.Source: §2.2.1 · pp. 7–8

2.2.2 Training Recipe

  • Training proceeds through three phases: ViT-only alignment, full-model multimodal training, then longer-sequence training with more video and agent data.The vision encoder starts from scratch using DataComp and in-house data, while the LLM starts from pre-trained Qwen2.5.Source: §2.2.2 · p. 8

2.3 Post-training

Qwen2.5-VL post-training combines supervised fine-tuning and preference optimization with curated instruction data, quality filtering, and rejection-sampled reasoning examples. Its training recipe keeps ViT parameters frozen and uses DPO to align the model with human preferences.

2.3.1 Instruction Data

  • SFT draws on approximately 2 million entries split evenly between pure-text and multimodal data, with image-text and video-text examples and primarily Chinese and English content.The data covers single- and multi-turn dialogue, single-image and multi-image scenarios, and specialized tasks including VQA, captioning, math, coding, security, document/OCR, grounding, video analysis, and agent interactions; multimodal entries consume more tokens and compute. Queries come primarily from open-source repositories, supplemented by purchased and online data.Source: §2.3.1 · p. 9

2.3.2 Data Filtering Pipeline

  • A two-stage SFT-data filter categorizes QA pairs into eight domains and 30 subcategories, then applies domain-tailored rule- and model-based quality checks.Rules remove repetitive, incomplete, truncated, malformed, irrelevant, or potentially harmful entries; reward models assess query complexity and relevance and answer correctness, completeness, clarity, relevance, helpfulness, and visual grounding.Source: §2.3.2 · pp. 9–10

2.3.3 Rejection Sampling for Enhanced Reasoning

  • Rejection sampling retains ground-truth-matching outputs for multi-step math, code-generation, and domain-specific VQA tasks, supporting CoT reasoning (Wei et al., 2022; DeepSeek-AI et al., 2024).It also removes code-switching, excessive length, and repetition; rule- and model-driven checks validate visual-text integration in intermediate reasoning, though optimal modality alignment remains an ongoing challenge. The authors report improved reasoning proficiency from iteratively refining the data with accurate, coherent examples.Source: §2.3.3 · p. 10

2.3 Post-training

  • Post-training uses SFT followed by DPO with ViT parameters frozen; SFT uses diverse multimodal data, while DPO uses image-text and pure-text preference samples once.SFT structures instruction-following data in ChatML, diverging from the pretraining schema while retaining architectural consistency with Qwen2-VL; it tags dialogue roles, injects visual embeddings, and preserves cross-modal positional relationships.Source: §2.3 · pp. 8–9, §2.3.4 · pp. 10–11

3 Experiments

The experiments compare the overall model with current state-of-the-art models and assess its performance across various sub-capabilities.

  • The section compares the overall model with current state-of-the-art models and evaluates performance across various sub-capabilities.Source: §3 · p. 11

3.1 Comparison with the SOTA Models

Qwen2.5-VL-72B achieves competitive results across academic, mathematical, and visual question-answering benchmarks. It exceeds prior open-source or best scores on MMMU-Pro, MathVista, and MMbench-EN.

3.1 Comparison with the SOTA Models

  • On MMMU, Qwen2.5-VL-72B scores 70.2; on MMMU-Pro, it scores 51.1, surpassing previous open-source SOTA models and performing comparably to GPT-4o.Source: Table 3 · p. 11, §3.1 · p. 11
  • On MathVista, Qwen2.5-VL-72B scores 74.8 versus the previous open-source SOTA score of 72.3; it scores 38.1 on MATH-Vision and 57.6 on MathVerse.Source: §3.1 · p. 11
  • On MMbench-EN, Qwen2.5-VL-72B scores 88.6, slightly above the previous best of 88.3; it scores 70.7 on MuirBench and 64.4 on BLINK.Source: §3.1 · p. 11
  • Qwen2.5-VL-72B scores 31.7 on MTVQA, demonstrating multilingual text recognition, and scores 76.2 on MMVet and 7.6 on MM-MT-Bench, indicating conversational experience and user satisfaction.Source: §3.1 · p. 11

3.2 Performance on Pure Text Tasks

Qwen2.5-VL was evaluated on representative pure-text benchmarks against similarly sized language models and showed leading performance across diverse task domains.

3.2 Performance on Pure Text Tasks

  • The evaluation compares Qwen2.5-VL with several large language models of similar size.Source: §3.2 · p. 11
  • Benchmarks cover general tasks, mathematics and science, coding, and alignment.Source: §3.2 · p. 11
  • Qwen2.5-VL exhibits leading performance on pure-text tasks while also achieving state-of-the-art performance on multimodal tasks.The report presents the pure-text results in Table 4.Source: §3.2 · p. 11, Table 4 · p. 12

3.3 Quantitative Results

Section 3.3 reports benchmark results for Qwen2.5-VL across visual question answering, document and OCR understanding, spatial grounding, video understanding, and GUI-agent tasks. The reported findings include strong performance across these tasks and competitive results from smaller model sizes.

3.3.1 General Visual Question Answering

  • Qwen2.5-VL-72B scores 88.4% on MMBench-EN-V1.1, surpassing InternVL2.5 (78B) and Claude-3.5 Sonnet-0620, and 70.8% on MMStar.It also scores 63.2 on MME-RealWorld and 70.7 on MuirBench, leading on the latter benchmark.Source: §3.3.1 · p. 12
  • On MMStar, Qwen2.5-VL-7B scores 63.9% and Qwen2.5-VL-3B scores 55.9%, showing competitive performance from smaller models.The passage characterizes this as evidence that the architecture maintains strong performance with fewer parameters.Source: §3.3.1 · p. 12

3.3.2 Document Understanding and OCR

  • On OCRBench_v2, Qwen2.5-VL models exceed Gemini 1.5-Pro by 9.6% on the English track and 20.6% on the Chinese track.Qwen2.5-VL-72B sets new state-of-the-art results on CC-OCR and OmniDocBench; it also significantly outperforms InternVL2.5-78B on OCRBench, InfoVQA, and SEED-Bench-2-Plus.Source: Table 5 · p. 13, §3.3.2 · p. 13

3.3.3 Spatial Understanding

  • Qwen2.5-VL-72B achieves 43.1 mAP on ODinW-13 and 93.6 accuracy on CountBench using a “detect then count”-style prompt.The section reports leading performance across box grounding, point grounding, and counting; point grounding enables precise localization of object details that bounding boxes can poorly represent.Source: §3.3.3 · p. 13, Table 6 · p. 14, Table 7 · p. 14

3.3.4 Video Understanding and Grounding

  • On Charades-STA, Qwen2.5-VL-72B achieves 50.9 mIoU, surpassing GPT-4o, and it significantly outperforms GPT-4o on long-form video QA benchmarks LVBench and MLVU.Across evaluated video benchmarks, the maximum was 768 analyzed frames per video and 24,576 video tokens; synchronized MRoPE supports timestamp referencing and temporal grounding.Source: Table 8 · p. 14, §3.3.4 · p. 14

3.3.5 Agent

  • Qwen2.5-VL-72B scores 43.6% on ScreenSpot Pro, exceeding Aguvis-72B (23.6%) and Qwen2-VL-72B (1.6%), and outperforms baselines on offline evaluations.It scores 87.1% on ScreenSpot; online, it outperforms baselines on AndroidWorld and MobileMiniWob++ and is comparable on OSWorld without auxiliary marks.Source: Table 9 · p. 15, §3.3.5 · p. 15

4 Conclusion

Qwen2.5-VL advances multimodal understanding and interaction across static and dynamic tasks, combining broad visual capabilities with efficient processing. Its 72B flagship matches or surpasses leading models, while smaller variants outperform similarly sized competitors.

4 Conclusion

  • Qwen2.5-VL-72B matches or surpasses GPT-4o and Claude 3.5 Sonnet, particularly in document and diagram understanding, while maintaining strong pure-text performance.Qwen2.5-VL-7B and Qwen2.5-VL-3B outperform similarly sized competitors.Source: §4 · p. 15
  • Qwen2.5-VL advances visual recognition, object localization, document parsing, and long-video comprehension across static and dynamic tasks.The model is described as demonstrating generalization and task execution across domains.Source: §4 · p. 15
  • Native dynamic-resolution processing and absolute time encoding support diverse inputs, while Window Attention reduces computational overhead without sacrificing resolution fidelity.Source: §4 · p. 15
  • The model series targets applications ranging from edge AI to high-performance computing and aims to bridge perception with real-world application.Source: §4 · p. 15

Related papers

Loading 2502.13923v1…