PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen3-VL Technical Report

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, Ke Zhu

arXiv:2511.21631v2cs.CVcs.AI

TL;DR

Vision-language models must advance multimodal reasoning without sacrificing language proficiency, while handling increasingly long interleaved inputs. Qwen3-VL addresses this with multimodal training and architectural upgrades, achieving strong performance across text, vision, and multimodal reasoning benchmarks with native 256K-token support.

  • Problem

    Vision-language models need stronger multimodal reasoning and long-context capabilities without sacrificing the linguistic proficiency of their underlying language models.

  • Method

    Qwen3-VL combines optimized multimodal training, 256K-token context adaptation, dense and MoE variants, and architectural upgrades for vision-language alignment and temporal grounding.

  • Results

    Qwen3-VL achieves superior performance across text, vision, and multimodal reasoning benchmarks while maintaining strong pure-text capabilities.

  • Takeaways & Limitations

    Qwen3-VL provides a flexible vision-language foundation for long-document, image-sequence, video, and real-world multimodal workflows.

  • Takeaways & Limitations

    Video benchmark comparisons may not be fully fair because resource and API limits constrained the number of input frames for competing models.

Abstract

from arXiv · show

We introduce Qwen3-VL, the most capable vision-language model in the Qwen series to date, achieving superior performance across a broad range of multimodal benchmarks. It natively supports interleaved contexts of up to 256K tokens, seamlessly integrating text, images, and video. The model family includes both dense (2B/4B/8B/32B) and mixture-of-experts (30B-A3B/235B-A22B) variants to accommodate diverse latency-quality trade-offs. Qwen3-VL delivers three core pillars: (i) markedly stronger pure-text understanding, surpassing comparable text-only backbones in several cases; (ii) robust long-context comprehension with a native 256K-token window for both text and interleaved multimodal inputs, enabling faithful retention, retrieval, and cross-referencing across long documents and videos; and (iii) advanced multimodal reasoning across single-image, multi-image, and video tasks, demonstrating leading performance on comprehensive evaluations such as MMMU and visual-math benchmarks (e.g., MathVista and MathVision). Architecturally, we introduce three key upgrades: (i) an enhanced interleaved-MRoPE for stronger spatial-temporal modeling across images and video; (ii) DeepStack integration, which effectively leverages multi-level ViT features to tighten vision-language alignment; and (iii) text-based time alignment for video, evolving from T-RoPE to explicit textual timestamp alignment for more precise temporal grounding. Under comparable token budgets and latency constraints, Qwen3-VL achieves superior performance in both dense and Mixture-of-Experts (MoE) architectures. We envision Qwen3-VL serving as a foundational engine for image-grounded reasoning, agentic decision-making, and multimodal code intelligence in real-world workflows.

1 Introduction

Qwen3-VL is presented as a multimodal foundation-model family designed to advance long-context understanding, multimodal reasoning, and real-world applications while preserving strong language proficiency. The report attributes these capabilities to expanded model variants, architectural upgrades, redesigned training data, and a staged training pipeline.

  • Motivation: Qwen3-VL targets downstream applications spanning long-context understanding, STEM reasoning, GUI comprehension and interaction, and agentic workflows.The introduction emphasizes that progress in vision–language models should not erode the underlying LLM’s linguistic capabilities.
  • Model Family: The family comprises four dense models (2B/4B/8B/32B) and two MoE models (30B-A3B / 235B-A22B), each supporting context windows up to 256K tokens.The models are built on the Qwen3 series and are intended to enable long-context understanding.
  • Architecture: Architectural improvements include interleaved MRoPE, DeepStack vision-feature integration, and text-based temporal alignment for more precise image–video modeling.Interleaved MRoPE distributes temporal, horizontal, and vertical dimensions across frequency ranges to address limitations in long-video understanding.
  • Training Data: Training data upgrades add enhanced captions, broader omni-recognition and OCR, normalized 3D/spatial grounding, code and long-document corpora, temporally grounded video, reasoning, and GUI-agent interactions.These additions are designed to connect perception, reasoning, and action across multimodal workflows.
  • Training Pipeline: The training pipeline combines pretraining and post-training, progressing from merger-only warm-up alignment through 8K, 32K, and 256K full-parameter training before long-chain-of-thought supervision and distillation.The staged process progressively increases context length and applies multiple post-training phases.
  • Positioning: Together, these innovations position Qwen3-VL as a platform integrating perception, reasoning, and action across diverse real-world multimodal applications.The report states that later sections evaluate its performance on text, vision, and multimodal reasoning benchmarks.

2 Model Architecture

Qwen3-VL uses a three-module vision encoder–MLP merger–LLM architecture, with dense and MoE Qwen3-based variants. Its upgrades include multi-level visual feature injection and textual timestamp encoding for more effective temporal video understanding.

  • Overall Architecture: Qwen3-VL comprises a vision encoder, an MLP-based vision–language merger, and a large language model.The architecture follows Qwen2.5-VL while processing multimodal inputs through these three modules.
  • Language Model: Qwen3-VL includes four dense variants—2B/4B/8B/32B—and two MoE variants—30B-A3B and 235B-A22B—built on Qwen3 backbones.The flagship Qwen3-VL-235B-A22B has 235B total parameters and 22B activated per token.
  • Vision Encoder: The vision encoder uses SigLIP-2 with dynamic resolutions, 2D-RoPE, and interpolated absolute position embeddings adapted to input size.SigLIP2-SO-400M is the default, while SigLIP2-Large (300M) serves small-scale LLMs.
  • Vision-Language Merger: A two-layer MLP compresses 2 × 2 vision-encoder features into one visual token aligned with the LLM hidden dimension.Specialized merger modules additionally support DeepStack.
  • DeepStack: DeepStack extracts features from three vision-encoder levels and adds projected visual tokens to the corresponding hidden states of the first three LLM layers.This preserves information spanning low- to high-level visual representations.
  • Temporal Encoding: Textual timestamps prefix video temporal patches, with seconds and HMS formats trained to improve temporal perception for video grounding and dense captioning.The approach modestly increases context length but addresses sparse temporal identifiers and sampling demands in long videos.

3 Pre-Training

Qwen3-VL uses a three-module architecture and four-stage pre-training curriculum that progresses from modality alignment to ultra-long-context adaptation. Its training data is refined and diversified to support multimodal grounding, world knowledge, coding, and video comprehension.

  • Pre-training architecture: Qwen3-VL comprises a SigLIP-2-based vision encoder, an MLP vision–language merger, and a Qwen3 LLM backbone trained through four progressively expanded stages.The stages build capabilities from basic alignment toward ultra-long-context understanding.
  • Stage 0: Vision-Language Alignment: Stage 0 trains only the MLP merger on approximately 67B tokens, while the vision encoder and LLM backbone remain frozen.The data contains image-caption pairs, visual knowledge collections, and OCR data.
  • Stage 1: Multimodal Pre-Training: Stage 1 unfreezes all components for joint training on approximately 1T tokens, using interleaved image-text documents, grounding, VQA, STEM, and video data at 8,192 tokens.The multimodal mixture introduces temporal understanding while preserving the LLM’s language capabilities.
  • Stage 2: Long-Context Pre-Training: Stage 2 quadruples sequence length to 32,768 with all parameters trainable, while increasing text-only data to strengthen long-form comprehension.Training uses approximately 1T tokens with an adjusted mixture for long-context tasks.
  • Stage 3: Ultra-Long-Context Adaptation: Stage 3 raises sequence length to 262,144 and trains on a focused 100B-token dataset emphasizing long-video and long-document understanding.This final adaptation targets the model’s operational context limits.
  • Data construction: The corpus combines refined multilingual image-caption pairs and interleaved text-image documents with entity-balanced knowledge data and dedicated grounding, coding, and video data.Processing includes recaptioning, semantic deduplication, visual-embedding clustering, multimodal parsing, and sequences of up to 256K tokens.

4 Post-Training

Qwen3-VL uses a three-stage post-training pipeline—supervised fine-tuning, strong-to-weak distillation, and reinforcement learning—to improve instruction following, reasoning, performance, and alignment. The process combines staged long-context training, curated multimodal reasoning data, text-only distillation, and adaptive tool-use training.

  • Post-training pipeline: Three post-training stages refine instruction following, reasoning abilities, model performance, and alignment: SFT, strong-to-weak distillation, and RL.RL is divided into Reasoning RL and General RL.
  • Supervised Fine-Tuning: SFT first trains at 32k context, then extends to 256k with long-document and long-video data, using standard or Chain-of-Thought formats for different model types.Thinking models use CoT formats that explicitly model reasoning.
  • Strong-to-Weak Distillation: Strong-to-weak distillation transfers teacher capabilities through text-only data to fine-tune the LLM backbone, improving reasoning across text-centric and multimodal tasks.The method uses a powerful teacher model to train student models.
  • Training data: The post-training data includes approximately 1,200,000 samples, with one-third text-only and two-thirds image-text or video-text pairs.The multimodal content targets interpretation of complex, real-world scenarios.
  • Data curation: A multi-dimensional filtering framework advances only data meeting stringent quality, reliability, and ethical criteria to SFT.Filtering combines query verification and revision with rule-based and model-based response checks.
  • Reasoning data: The thinking-model cold-start dataset maintains an approximate 1:1 vision-language/text-only ratio and yields challenging data for bootstrapping advanced multimodal reasoning.It emphasizes STEM and agentic workflows alongside VQA, OCR, grounding, and video analysis.
  • Agentic capabilities: A two-stage training paradigm adds agentic capabilities, while an explicit tool-calling reward promotes adaptive tool exploration aligned with task complexity.The reward addresses degeneration into a single tool call regardless of task demands.

5 Evaluation … 5.3 Alignment and Subjective Tasks

Qwen3-VL shows strong, scalable performance across general visual question answering, multimodal reasoning, and alignment evaluations. Its largest and medium-sized variants achieve leading benchmark results, while the flagship model substantially improves instruction following and hallucination reduction against closed-source systems.

  • 5.1 General Visual Question Answering: Across MMBench-V1.1, RealWorldQA, MMStar, and SimpleVQA, Qwen3-VL performs competitively across model sizes from 2B to 235B parameters.The evaluation covers general visual question answering across the Qwen3-VL family.
  • 5.1 General Visual Question Answering: 78.7 is Qwen3-VL-235B-A22B-Thinking’s highest score on MMStar, while Qwen3-VL-235B-A22B-Instruct leads MMBench at 89.3/88.9 and RealWorldQA at 79.2.The MMStar result is reported for thinking mode; the MMBench and RealWorldQA results are reported for non-reasoning mode.
  • 5.1 General Visual Question Answering: Scaling from Qwen3-VL-2B to Qwen3-VL-8B raises MMBench-EN thinking scores from 79.9 to 85.3 and MMStar thinking scores from 68.1 to 75.3.The 8B model achieves the highest performance across all five reported benchmarks.
  • 5.2 Multimodal Reasoning: Qwen3-VL-32B consistently outperforms Gemini-2.5-Flash and GPT-5-mini on multimodal reasoning, surpasses Qwen2.5-VL-72B, and is joined by competitive Qwen3-VL-30B-A3B MoE results.The reasoning evaluation focuses primarily on STEM-related tasks and visual puzzles, including MMMU, MathVision, MathVista, and related benchmarks.
  • 5.2 Multimodal Reasoning: Among small models, Qwen3-VL-8B maintains the clearest overall advantage, while Qwen3-VL-4B achieves the highest scores on DynaMath and VisuLogic.Even Qwen3-VL-2B demonstrates strong reasoning capabilities against GPT-5-Nano.
  • 5.3 Alignment and Subjective Tasks: The alignment evaluation tests complex instruction following and image-level hallucination reduction with MM-MT-Bench, HallusionBench, and MIA-Bench.MM-MT-Bench evaluates multimodal instruction-tuned models through multi-turn LLM-as-a-judge assessment.
  • 5.3 Alignment and Subjective Tasks: On HallusionBench, Qwen3-VL-235B-A22B-Thinking surpasses Gemini-2.5-pro, GPT-5, and Claude opus 4.1 by 3.0, 1.0, and 6.3 points, respectively, while achieving the overall best MIA-Bench score.These results are reported for the flagship model, which consistently outperforms other closed-source models.

5.4 Text Recognition and Document Understanding

Qwen3-VL establishes strong document understanding across OCR, VQA, chart reasoning, and long-document benchmarks, with flagship and smaller models outperforming comparable systems in many evaluations. It also expands multilingual support to 39 languages, exceeding 70% accuracy on 32 of them.

  • OCR and document benchmarks: Qwen3-VL-235B-A22B-Instruct establishes a new state of the art on OCR-focused and comprehensive OCR benchmarks, marginally outperforming its Thinking counterpart.The evaluated OCR benchmarks include CC-OCR, OmniDocBench, OCRBench, and OCRBench_v2.
  • OCR and document benchmarks: On the reasoning subset of CharXiv, Qwen3-VL-235B-A22B-Thinking surpasses Instruct and ranks second only to GPT5-thinking and Gemini-2.5-Pro-Thinking.The subset requires deep chart comprehension and multi-step reasoning.
  • Model-size comparisons: Qwen3-VL-30BA3B and Qwen3-VL-32B consistently outperform Gemini-2.5-Flash and GPT-5-mini across most evaluation metrics.Qwen3-VL-8B, Qwen3-VL-4B, and Qwen3-VL-2B remain competitive on OCR parsing, VQA, and comprehensive benchmark suites.
  • Long-document understanding: 57.0%/56.2% overall accuracy is achieved by Qwen3-VL-235B-A22B under instruct/thinking settings on MMLongBench-Doc, showcasing state-of-the-art long-document understanding.The comparison is among flagship models.
  • Multilingual support: 32 of 39 supported languages exceed 70% accuracy on Qwen3-VL’s in-house multilingual OCR test set, expanding support from Qwen2.5-VL’s 10 non-English/Chinese languages.The paper characterizes the 70% threshold as practical for real-world usability.

5.5 2D and 3D Grounding … 5.8 Embodied and Spatial Understanding

Across grounding, fine-grained perception, multi-image understanding, and embodied-spatial tasks, Qwen3-VL is evaluated on broad benchmark suites and reports strong results, including SOTA performance and competitive comparisons with leading models. The evaluations cover 2D/3D localization, high-resolution visual perception, cross-image reasoning, and spatial benchmarks.

  • 5.5 2D and 3D Grounding: Qwen3-VL is evaluated on RefCOCO/+/g, ODinW-13, and CountBench for referring comprehension, open-vocabulary detection, and counting.ODinW-13 uses mean Average Precision (mAP), with all dataset categories provided simultaneously in the prompt.
  • 5.5 2D and 3D Grounding: 48.6 mAP is achieved by Qwen3-VL-235B-A22B on the reported 2D grounding and counting evaluation.The passage identifies this flagship model as achieving state-of-the-art results across 2D grounding and counting benchmarks.
  • 5.5 2D and 3D Grounding: Qwen3-VL evaluates 3D object localization on Omni3D against comparable-scale models using mean Average Precision (mAP).Omni3D includes ARKitScenes, Hypersim, and SUN RGB-D, with each input formed as an image-text pair.
  • 5.6 Fine-grained Perception: With tools, Qwen3-VL-235B-A22B reaches 93.7 on V*, 85.3 on HRBench-4k, and 82.3 on HRBench-8k, achieving SOTA across all three benchmarks.The passage describes a substantial fine-grained visual-understanding improvement over Qwen2.5-VL-72B.
  • 5.7 Multi-Image Understanding: Qwen3-VL is trained and evaluated for multi-image referring grounding, visual correspondence, and multi-hop reasoning across diverse visual patterns.These cross-image pattern-learning techniques target higher-level contextual analysis and advanced recognition and reasoning.
  • 5.8 Embodied and Spatial Understanding: Qwen3-VL-235B-A22B is benchmarked on ERQA, VSIBench, EmbSpatial, RefSpatial, and RoboSpatialHome for embodied and spatial understanding.Across these challenging benchmarks, it rivals Gemini-2.5-Pro, GPT-5, and Claude-Opus-4.1.

5.9 Video Understanding · 5.10 Agent

Qwen3-VL substantially improves video understanding through architectural and data-scaling enhancements, with the 8B variant competitive with Qwen2.5-VL 72B. Its 235B-A22B model achieves state-of-the-art GUI grounding across multiple interactive-interface tasks.

  • 5.9 Video Understanding: Qwen3-VL 8B achieves performance competitive with the significantly larger Qwen2.5-VL 72B model on video understanding.The improvements combine interleaved MRoPE, textual timestamps, and scaling temporally dense video captions.
  • 5.9 Video Understanding: Video evaluation covers general understanding, temporal grounding, reasoning, and long-form understanding tasks.Benchmarks include VideoMME, MVBench, Charades-STA, VideoMMMU, MMVU, LVBench, and MLVU.
  • 5.9 Video Understanding: The study compares video understanding against state-of-the-art proprietary models, including Gemini 2.5 Pro and GPT.The supplied passage introduces this comparison but is truncated after naming GPT.
  • 5.9 Video Understanding: Video benchmarks cap each video at 2,048 frames and 224K total video tokens.Frame-token limits are 768 for VideoMMMU and MMVU, and 640 elsewhere; sampling uses 4 fps for Charades-STA and 2 fps otherwise.
  • 5.10 Agent: Qwen3-VL-235B-A22B achieves state-of-the-art GUI-grounding performance across multiple tasks.The evaluated tasks are ScreenSpot, ScreenSpot Pro, and OSWorldG.
  • 5.10 Agent: The GUI-grounding evaluation covers interactive interfaces on desktop, mobile, and PC.The agent evaluation also assesses decision-making through AndroidWorld and OSWorld.

5.11 Text-Centric Tasks

Qwen3-VL demonstrates strong text-centric performance across flagship, mid-sized, and edge-side models, often matching or surpassing text-only and leading model baselines. These results also support the effectiveness of Strong-to-Weak Distillation for building lightweight models at reduced cost and effort.

  • Qwen3-VL-235B-A22B: Qwen3-VL-235B-A22B-Instruct achieves competitive results against leading instruct and thinking models, particularly surpassing others on reasoning-demanding mathematics and coding tasks.Compared baselines include DeepSeek V3 0324, Claude-Opus-4, and Qwen3-235B-A22B-Instruct-2507.
  • Qwen3-VL-32B / 30B-A3B: Qwen3-VL-32B and Qwen3-VL-30B-A3B improve significantly over Qwen3-32B and Qwen3-30B-A3B across all instruct benchmarks.Qwen3-VL-30B-A3B is comparable to or better than Qwen3-30B-A3B-2507, particularly on AIME-25 and HMMT-25.
  • Qwen3-VL-32B / 30B-A3B: Qwen3-VL-32B and Qwen3-VL-30B-A3B surpass their thinking-model baselines on most benchmarks, with Qwen3-VL-30B-A3B remaining comparable to Qwen3-30B-A3B-2507.The comparisons use corresponding text-only counterparts and Qwen3-30B-A3B-2507.
  • Qwen3-VL-8B / 4B / 2B: The edge-side results demonstrate the efficacy of Strong-to-Weak Distillation for building lightweight models with remarkably reduced costs and efforts.The claim follows the reported performance of Qwen3-VL-2B, Qwen3-VL-4B, and Qwen3-VL-8B against their baselines.

5.12 Ablation Study

The ablations show that Qwen3-ViT improves holistic multimodal evaluation and that DeepStack strengthens fine-grained visual understanding. Long-context evaluation further demonstrates perfect retrieval accuracy through 30-minute, 256K-token videos and 99.5% accuracy when extended to 1M tokens.

  • Qwen3-ViT ablation: Qwen3-ViT remains competitive on standard benchmarks while achieving substantial gains on OmniBench versus SigLIP-2.The comparison uses the CLIP pretraining stage and the same 1.7B Qwen3 language model for downstream evaluation.
  • DeepStack ablation: DeepStack delivers overall gains across benchmarks, especially improving fine-grained visual understanding on InfoVQA and DocVQA.The ablation uses an internal 15B-A2B LLM pretrained on 200 billion tokens without post-training.
  • Long-context evaluation: 100% accuracy is achieved on videos up to 30 minutes, corresponding to a 256K-token context, in the video Needle-in-a-Haystack evaluation.The task inserts a salient needle frame at varying temporal positions and asks the model to locate it and answer the associated question.
  • Long-context evaluation: 99.5% accuracy is retained when YaRN-based positional extension extrapolates the model to sequences up to 1M tokens, approximately 2 hours of video.Videos are uniformly sampled at 1 FPS during evaluation.

6 Conclusion

Qwen3-VL advances multimodal understanding and generation through high-quality multimodal data iteration and architectural innovations, while retaining strong pure-text capabilities. The authors envision it as a foundation for embodied AI agents that connect digital and physical worlds through grounded, context-aware multimodal action.

  • Contributions: Qwen3-VL advances multimodal understanding and generation as a state-of-the-art vision-language foundation model series.It achieves strong performance across a broad spectrum of multimodal benchmarks while maintaining strong pure-text capabilities.
  • Contributions: Enhanced interleaved-MRoPE, DeepStack alignment, and text-based temporal grounding underpin Qwen3-VL’s multimodal advances.These architectural innovations are integrated with high-quality multimodal data iteration.
  • Future direction: Qwen3-VL is envisioned as a foundational engine for embodied AI agents bridging digital and physical worlds.Such agents would perceive and reason over rich multimodal inputs while executing decisive, context-aware actions in dynamic environments.
  • Future direction: The envisioned agents would interact with users, manipulate digital interfaces, and guide robotic systems through grounded, multimodal decision-making.These capabilities are framed as context-aware actions in dynamic environments.

7 Contributions and Acknowledgments

This section credits the Qwen3-VL contributors, listed alphabetically by last name, and acknowledges supporting teams led by seven individuals.

  • Contributions: All Qwen3-VL contributors are listed alphabetically by last name.The section separately identifies core contributors and contributors.
  • Contributions: The core contributor list includes 38 named individuals.The list begins with Shuai Bai and ends with Jun Tan.
  • Contributions: The contributor list includes 40 additional named individuals.The list begins with Yizhong Cao and ends with Siqi Zhang.
  • Acknowledgments: The paper acknowledges unwavering support from teams led by seven individuals.The acknowledged team leads are Zulong Chen, Bing Deng, Feiyu Gao, Guanjun Jiang, Yue Liu, Hangdi Xing, and Daijun Yu.

A Benchmarks

Qwen3-VL is evaluated across a broad benchmark suite covering multimodal reasoning, visual and document understanding, grounding, spatial and video capabilities, GUI agents, coding, and text-centric tasks. The evaluation spans specialized tests of perception, reasoning, instruction following, interactive control, and multimodal code generation.

  • Evaluation Scope: The evaluation covers multimodal reasoning, general visual question answering, subjective experience and instruction following, document understanding, grounding, spatial reasoning, video understanding, GUI agents, coding, and text-centric tasks.This benchmark suite is designed to assess distinct multimodal capabilities across diverse task families.
  • Multimodal Reasoning: 12 multimodal reasoning benchmarks span mathematics, STEM, visual reasoning, and puzzle-solving, including MMMU, MMMU-Pro, MathVision, MathVista, and LogicVista.The listed suite also includes MathVision-Wildphoto, We-Math, MathVerse, DynaMath, Math-VR, and VisualPuzzles.
  • Document Understanding: 10 OCR and document-understanding benchmarks assess capabilities through DocVQA, InfoVQA, AI2D, ChartQA, OCRBench, OCRBench_v2, CC-OCR, OmniDocBench, CharXiv, and MMLongBench-Doc.The document suite covers diverse OCR-related benchmarks alongside document comprehension tasks.
  • Grounding and Spatial Understanding: 11 benchmarks evaluate 2D grounding, 3D grounding, counting, and spatial understanding, including RefCOCO/+/g, ODinW-13, CountBench, ARKitScenes, Hypersim, and SUN RGB-D.Additional evaluations include ERQA, VSIBench, EmbSpatial, RefSpatial, and RoboSpatialHom.
  • Video, GUI, and Coding: Seven video benchmarks, five GUI-agent benchmarks, and three coding benchmarks assess video understanding, interface perception and decision-making, front-end reconstruction, and SVG generation.The video suite includes VideoMME and MVBench; GUI evaluation covers ScreenSpot, AndroidWorld, and OSWorld; coding uses Design2Code, ChartMimic, and UniSVG.

B Evaluation Prompts

This section provides the complete evaluation prompts, applied consistently during inference to support reproducibility, fairness, and comparability. It covers standard multiple-choice formats, iterative visual research with tools, and agent interaction protocols.

  • B Evaluation Prompts: The complete benchmark prompt set is provided and applied consistently during inference to ensure reproducibility, fairness, and comparability.The prompts are intended to facilitate future research.
  • B.1 STEM & Puzzle: STEM and puzzle evaluation uses a question-and-options format followed by an instruction to select the correct answer.The same question-and-options structure is also used for multi-image evaluation.
  • B.7 Multi-Image: Multi-image evaluation presents the question and options, then asks the model to select the correct answer from the options above.This prompt structure parallels the STEM and puzzle format.
  • B.9 Perception with Tool: Perception-with-tool prompts assign a visual research-assistant role and require close image inspection, information lookup, iterative review, and synthesized answers.The loop continues until research is complete, and the model may call one or more functions.
  • B.9 Perception with Tool: Perception-with-tool function calls use provided XML-wrapped signatures, including an image_zoom_in_tool for cropping image regions by bounding box and label.The required arguments include bbox_2d, label, and img_idx.
  • B.9 Perception with Tool: Each perception-with-tool call must return a JSON object containing the function name and arguments inside tool_call tags, followed by the image question and optionally its options.The appendix includes both question-only and question-with-options variants.
  • B.11 Agent: Agent prompts cover Screenspot, Screenspot-Pro, and OSWorld-G and permit computer_use functions for mouse movement and left-click interaction.The protocol specifies display resolution, centered cursor placement, and termination with failure when the task is infeasible.
Loading 2511.21631v2…