PaperlayerView exact revision on arXiv

Source-linked AI summary

Kimi-VL Technical Report

Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Mengfan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen, Zongyu Lin

arXiv:2504.07491v3cs.CV

TL;DR

Open-source VLM development has lagged language-only models in scalability, computational efficiency, and advanced reasoning. Kimi-VL combines an MoE language model with native-resolution vision, and reports competitive or superior performance across diverse multimodal tasks.

  • Problem

    Open-source VLMs have lagged language-only models in scalability, computational efficiency, and advanced reasoning capabilities.

    Source: §1 · p. 2

  • Method

    Kimi-VL pairs an MoE language model with a native-resolution MoonViT vision encoder, and Kimi-VL-Thinking adds long-CoT training and reinforcement learning.

    Source: §1 · pp. 2–3

  • Results

    Kimi-VL is competitive or superior across multimodal benchmarks, outperforming Qwen2.5-VL-7B on 19 of 24 benchmarks.

    Source: §1 · p. 3, §4.1 · p. 10

  • Takeaways & Limitations

    Kimi-VL demonstrates adaptability and efficiency across multimodal, long-context, and high-resolution tasks, with potential for future research and industrial applications.

    Source: §5 · p. 19

    Despite its 128K context window, limited attention-layer parameters leave long-context abilities insufficient for certain advanced applications involving extremely long sequences.

    Source: §5 · p. 19

Abstract

from arXiv · show

We present Kimi-VL, an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM) that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities - all while activating only 2.8B parameters in its language decoder (Kimi-VL-A3B). Kimi-VL demonstrates strong performance across challenging domains: as a general-purpose VLM, Kimi-VL excels in multi-turn agent tasks (e.g., OSWorld), matching flagship models. Furthermore, it exhibits remarkable capabilities across diverse challenging vision language tasks, including college-level image and video comprehension, OCR, mathematical reasoning, and multi-image understanding. In comparative evaluations, it effectively competes with cutting-edge efficient VLMs such as GPT-4o-mini, Qwen2.5-VL-7B, and Gemma-3-12B-IT, while surpassing GPT-4o in several key domains. Kimi-VL also advances in processing long contexts and perceiving clearly. With a 128K extended context window, Kimi-VL can process diverse long inputs, achieving impressive scores of 64.5 on LongVideoBench and 35.1 on MMLongBench-Doc. Its native-resolution vision encoder, MoonViT, further allows it to see and understand ultra-high-resolution visual inputs, achieving 83.2 on InfoVQA and 34.5 on ScreenSpot-Pro, while maintaining lower computational cost for common tasks. Building upon Kimi-VL, we introduce an advanced long-thinking variant: Kimi-VL-Thinking-2506. Developed through long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL), the latest model exhibits strong long-horizon reasoning capabilities (64.0 on MMMU, 46.3 on MMMU-Pro, 56.9 on MathVision, 80.1 on MathVista, 65.2 on VideoMMMU) while obtaining robust general abilities. Code and models are publicly accessible at https://github.com/MoonshotAI/Kimi-VL.

1 Introduction

Kimi-VL addresses open-source VLM gaps in scalability, efficiency, and advanced reasoning with a native-resolution vision encoder and a MoE language model. A long-thinking variant extends its multimodal reasoning capabilities.

Architecture

  • Kimi-VL pairs a 2.8B-activated (16B-total) Moonlight MoE decoder with a 400M native-resolution MoonViT encoder and MLP projector.The model handles diverse tasks across single-image, multi-image, video, and long-document inputs.Source: §1 · p. 2, Figure 3 · p. 3

Kimi-VL capabilities

  • Without long thinking, Kimi-VL has text ability comparable to efficient pure-text LLMs and is competitive on MMMU, MathVista, and OSWorld.Source: §1 · p. 3
  • Within its 128K context window, Kimi-VL handles multimodal long-context inputs and is far ahead of similar-scale competitors on long-video benchmarks and MMLongBench-Doc.Source: §1 · p. 3
  • Kimi-VL is all-round competitive with efficient dense and MoE VLMs across visual perception, visual world knowledge, OCR, and high-resolution OS screenshots.Source: §1 · p. 3

Kimi-VL-Thinking

  • Kimi-VL-Thinking-2506 uses long-CoT activation and RL to improve hard multimodal reasoning while retaining or improving common visual understanding.The report says it outperforms many larger VLMs on hard reasoning benchmarks, with the 2506 version further improving reasoning while retaining or improving performance in high-resolution perception, OS grounding, video, and long-document understanding.Source: §1 · p. 3

2 Approach

Kimi-VL combines native-resolution vision processing with an MoE language model, then develops multimodal, long-context, and reasoning capabilities through staged pre-training and post-training. Its approach also includes distributed optimization and training infrastructure.

2.1 Model Architecture

  • MoonViT packs patches from varying-resolution images into sequences and combines interpolated SigLIP positions with 2D RoPE for fine-grained spatial encoding.The Thinking-2506 model further trains MoonViT to encode up to 3.2 million pixels from one image, four times the original limit.Source: §2.1 · pp. 3–4

2.3 Pre-Training Stages

  • 4.4T tokens across four pre-training stages follow the text-only language-model checkpoint: standalone ViT training, then joint pre-training, cooldown, and long-context activation.The three stages that update the language model are joint training stages.Source: Figure 4 · p. 4, §2.3 · p. 4

2.1 Model Architecture

  • 2.8B activated parameters power the 16B-parameter Moonlight MoE language model, while a two-layer MLP projector uses 2×2 pixel-shuffle downsampling to connect vision features to LLM embeddings.The language model starts from a checkpoint trained on 5.2T text tokens and is continued on 2.3T tokens of joint multimodal and text-only data.Source: §2.1 · p. 4

2.2 Muon Optimizer

  • Enhanced Muon adds weight decay and adjusted parameter-update scales, with a ZeRO-1 distributed implementation used to optimize all model parameters.The implementation is designed for memory efficiency and reduced communication overhead while preserving the optimizer’s mathematical properties.Source: §2.2 · p. 4

2.3 Pre-Training Stages

  • 2T tokens train MoonViT with SigLIP contrastive and image-conditioned caption losses, followed by 0.1T-token alignment and 1.4T-token joint pre-training.Joint pre-training gradually increases multimodal data after language-only initial steps; cooldown adds curated and synthetic language and multimodal QA data to improve model abilities.Source: §2.3 · p. 5
  • 131,072-token context activation proceeds in two fourfold extensions, using 25% long data and 75% shorter-data replay; the model passes text and video needle-in-a-haystack evaluations.Long-context data includes long text, interleaved multimodal data, videos, and documents.Source: §2.3 · pp. 5–6, Table 2 · p. 5

2.4 Post-Training Stages

  • Long-CoT warmup SFT teaches planning, evaluation, reflection, and exploration; RL refines reasoning with correctness rewards, sampling strategies, and penalties for excessively long responses.Joint instruction SFT creates the interactive model, while large-scale RL develops error detection and iterative refinement using reasoning history, with standard autoregressive inference.Source: §2.4 · pp. 6–7

2.5 Infrastructure

  • 4D data, expert, pipeline, and context parallelism raise training throughput around 60% over a 7B dense VLM; ZeRO1 and selective activation checkpointing further reduce memory use.Context parallelism splits long sequences to reduce peak attention memory, and recomputation expands to more layers for extremely long sequences to prevent out-of-memory errors.Source: §2.5 · pp. 7–8

3 Data Construction

Kimi-VL’s data construction combines multimodal pre-training, instruction tuning, and long-chain-of-thought reasoning data. The corpus spans six multimodal categories and uses quality controls and tailored data pipelines to support alignment, generalization, and model capabilities.

3.1 Pre-Training Data

  • The pre-training corpus combines caption, interleaving, OCR, knowledge, video, and agent data, with filtering, synthesis, and deduplication pipelines for quality and cross-modal alignment.The multimodal strategy is designed to preserve language-model capabilities while aligning knowledge across modalities.Source: §3.1 · p. 8
  • Image-text interleaving data from open-source and in-house sources supports multi-image comprehension, detailed image knowledge, longer multimodal contexts, and language ability; reordering preserves image-text sequence.The team also synthesizes interleaving data to help retain text knowledge, while filtering and deduplication support data quality.Source: §3.1 · p. 9
  • OCR data spans clean and augmented single- and multi-page inputs, multilingual and dense layouts, handwriting, and optical image types; augmentation and document-scale data support robustness and long-document comprehension.The OCR pipeline handles figures, tables, geometry diagrams, mermaid plots, and natural-scene text, with augmentations including rotation, distortion, color adjustment, and noise.Source: §3.1 · p. 9
  • Agent data combines virtual-environment screenshots and actions, grounding and trajectory formats, desktop, mobile, and web action spaces, icon data, and human-annotated multi-step computer-use demonstrations.The demonstrations include synthesized chain-of-thought and target real-world desktop tasks on Ubuntu and Windows.Source: §3.1 · p. 9
  • Video data is used in pre-training, cooldown, and long-context activation to support long image-dominated sequences and fine-grained spatiotemporal understanding; long-video captions are dense, with synthetic descriptions limited to reduce hallucination risk.Sources include open-source and in-house web-scale videos of varying durations and scenes, covering tasks such as description and grounding.Source: §3.1 · pp. 9–10
  • Text pre-training uses Moonlight’s English, Chinese, code, mathematics and reasoning, and knowledge domains, with filtering, source validation, and empirically tuned sampling that upsamples valuable subsets while retaining diversity.Isolated evaluations identify subsets that contribute to knowledge acquisition, while other document types remain represented to support generalization.Source: §3.1 · p. 10

3.2 Instruction Data

  • Instruction data uses human-annotated seeds and ranked, refined model responses for non-reasoning tasks, rejection sampling for verifiable reasoning tasks, and an approximately 1:1 text-to-image token ratio.Covered tasks include chart interpretation, agent grounding, OCR, image-grounded conversation, visual coding and reasoning, and math and science problems.Source: §3.2 · p. 10

3.3 Reasoning Data

  • Long-CoT reasoning data is built from multi-step QA with ground-truth labels by sampling trajectories from Kimi k1.5 and filtering incorrect responses with a reward model and rule-based rewards.This pipeline supports multimodal reasoning during both long-CoT supervised fine-tuning and reinforcement learning.Source: §3.3 · p. 10

4 Evaluation · 4.1 Comparison to the State-of-the-Art Models

Kimi-VL is a compact vision-language model that compares favorably with larger and efficient models across academic, visual, mathematical, document, agent, and video benchmarks. Its strongest reported results span multi-image reasoning, OCR, interface interaction, long-form understanding, and fine-grained video perception.

4 Evaluation; 4.1 Comparison to the State-of-the-Art Models

  • Across Table 3’s benchmarks, Kimi-VL beats Qwen2.5-VL-7B on 19 of 24 and DeepSeek-VL2 on most, using fewer activated and total parameters.Kimi-VL uses 2.8B activated and 16B total parameters versus DeepSeek-VL2’s 4.5B activated and 28B total; it has 2.59× fewer activated parameters than Qwen2.5-VL-7B.Source: §4.1 · pp. 10–11, Table 3 · p. 11

4.1.1 College-level Academic Problems

  • On MMMU validation, Kimi-VL scores 57.0%, exceeding DeepSeek-VL2’s 51.1%; on video college-level problems, it outperforms Qwen2.5-VL-7B and DeepSeek-VL2.Its MMMU result is comparable to Qwen2.5-VL-7B (58.6%) and Gemma-3-12B-IT (59.6%); on video college-level problems, only Gemma-3-12B-IT, with over 10B parameters, scores higher.Source: §4.1.1 · p. 11

4.1.2 General Visual Ability

  • Kimi-VL scores 83.1% on MMBench-EN-v1.1, 84.9% on AI2D, and 57.3% on BLINK, outperforming all compared models on AI2D and BLINK.It performs on par with GPT-4o on MMBench-EN-v1.1. On MMVet it scores 66.7%, close to Qwen2.5-VL-7B (67.1%) and GPT-4o-mini (66.9%); its RealWorldQA score is 68.1%.Source: §4.1.2 · p. 12

4.1.3 Mathematical Reasoning

  • Kimi-VL scores 68.7% on MathVista, exceeding GPT-4o’s 63.8% and Qwen2.5-VL-7B’s 68.2%, but trails Qwen2.5-VL-7B and Gemma-12B-IT on MathVision.On MathVision, it outperforms DeepSeek-VL2 and Llama-3.2-11B-Inst.; the report says Kimi-VL-Thinking improves with RL and test-time scaling to be on par with 30B-level VLMs (Table 4).Source: §4.1.3 · pp. 12–13

4.1.4 Document Understanding and OCR

  • Kimi-VL achieves 83.2% on InfoVQA and 86.7% on OCRBench, surpassing GPT-4o on InfoVQA and all compared models on OCRBench.Its InfoVQA score also exceeds DeepSeek-VL2’s 78.1%; on OCRBench, it exceeds GPT-4o-mini’s 78.5% and DeepSeek-VL2’s 81.1%.Source: §4.1.4 · p. 13

4.1.5 Agent Grounding and Multi-turn Agent Interaction

  • Kimi-VL reaches 8.22% on OSWorld and 10.4% on WindowsAgentArena, outperforming GPT-4o on both multi-step agent benchmarks.For grounding, it scores 92.0% on ScreenSpot-V2 and 34.5% on ScreenSpot-Pro, which evaluates 4K screens.Source: §4.1.5 · p. 13

4.1.6 Long Document and Long Video Understanding

  • Kimi-VL scores 35.1% on MMLongBench-Doc and 64.5% on LongVideoBench, outperforming efficient-model competitors on long-form document and video understanding.It scores 74.2% on the MLVU MCQ subset, surpassing GPT-4o (64.6%) and Qwen2.5-VL-7B (70.2%); on Video-MME with subtitles it reaches 72.6%, and leads efficient VLMs in the without-subtitles setting.Source: §4.1.6 · p. 14

4.1.7 Egocentric and Fine-grained Video Perception

  • Kimi-VL scores 78.5% on EgoSchema, 37.4% on VSI-Bench, and 31.7% on TOMATO, exceeding the cited GPT-4o or Qwen2.5-VL-7B comparisons on each.The benchmarks assess egocentric understanding, spatial relationships and object correspondences in video, and fine-grained temporal perception, respectively.Source: §4.1.7 · p. 14

4.2 Kimi-VL-A3B-Thinking: A Reasoning Extension of Kimi-VL · 4.3 Kimi-VL-A3B-Thinking-2506: From Reasoning Extension to Integrated Thinking Model

Kimi-VL-Thinking adds long-CoT reasoning and reinforcement learning to Kimi-VL, while Thinking-2506 integrates broader perception, video, long-document, and OS-agent abilities. Thinking-2506 improves reasoning and general multimodal results while reducing token use, and extends performance across video, long-context, high-resolution, and OS-grounding tasks.

4.2 Kimi-VL-A3B-Thinking: A Reasoning Extension of Kimi-VL

  • Kimi-VL-Thinking extends Kimi-VL with long-CoT activation and reinforcement learning to enable chain-of-thought reasoning in a long-thinking model.The report validates its performance on image benchmarks in Table 4.Source: §4.2 · p. 16
  • 71.3% on MathVista beats GPT-4o’s 63.8%, 61.7% on MMMU beats GPT-4o-mini’s 60.0%, and 36.8% on MathVision exceeds GPT-4o’s 30.4%.Thinking also gains 2.6% on MathVista, 4.7% on MMMU, and 15.4% on MathVision over base Kimi-VL; it reaches 35.5% on MathVision versus Gemma-3-27B-IT’s 35.5% and QVQ-72B’s 35.9%, with 3B activated parameters.Source: §4.2 · p. 16, Table 4 · p. 17
  • 18.7% at 1k tokens to 36.8% at 16k tokens on MathVision shows test-time scaling, while MathVista reaches 70.9% at 4k tokens and then shows no significant gains through 16k.Accuracy also trends upward with longer thinking on MMMU, but the benefit of additional computation varies by benchmark.Source: §4.2 · pp. 16, 18, Figure 13 · p. 18

4.3 Kimi-VL-A3B-Thinking-2506: From Reasoning Extension to Integrated Thinking Model

  • 56.9% on MathVision, 80.1% on MathVista, 46.3% on MMMU-Pro, and 64.0% on MMMU are Thinking-2506’s reasoning results, with average output tokens reduced by around 20%.Output length falls from 2.9K to 2.4K on MMMU-val and from 5.8K to 4.4K on MathVision.Source: Table 4 · p. 17, §4.3 · p. 18
  • 84.4% on MMBench-EN-v1.1, 70.4% on MMStar, 70.0% on RealWorldQA, and 78.4% on MMVet show Thinking-2506’s competitive or superior general multimodal understanding.It uses an average of 180 tokens per MMBench answer, one-third the previous thinking model’s token use while improving accuracy by 8.4%.Source: Table 5 · p. 18, §4.3 · p. 19
  • 65.2% on VideoMMMU is a new open-source-model state of the art; 71.9% on Video-MME matches Instruct, and 42.1% on MMLongBench-Doc matches GPT-4o.The MMLongBench-Doc result is 10% above the previous thinking model and 7% above the previous instruct model; it is described as the first open-source model to match GPT-4o there.Source: Table 5 · p. 18, §4.3 · p. 19
  • 83.2% on V* Benchmark, 52.8% on ScreenSpot-Pro, and 52.5% on OSWorld-G show gains in high-resolution perception and OS grounding after continual MoonViT training.MoonViT training supports up to 3.2 million input pixels; V* Benchmark is reported without external tools, and OSWorld-G uses the full set with refusal samples. The model integrates perception, video, long-document, and OS-agent abilities.Source: §4.3 · pp. 18–19

5 Conclusion, Limitation, and Future Work

Kimi-VL combines a scalable MoE architecture, a 128K context window, native-resolution MoonViT, and long-chain reasoning to support efficient multimodal, long-context, and high-resolution tasks. The conclusion identifies remaining limits in model size, reasoning, and long-context capacity, and proposes scaling and training improvements.

5 Conclusion, Limitation, and Future Work

  • Kimi-VL combines balanced multimodal and text-only training with MoE efficiency, 128K context, native-resolution MoonViT, and long-chain reasoning for multimodal, long-context, and high-resolution tasks.The report describes this design as adaptable and efficient, with potential for future research and industrial applications.Source: §5 · p. 19
  • Kimi-VL remains limited for specialized or language-dependent problems, complex multi-step reasoning, and advanced long-context tasks involving extremely long sequences or high-volume information.The conclusion attributes the long-context limitation to attention layers whose parameter count is comparable to a 3B model, despite the 128K context window.Source: §5 · p. 19
  • Future work will scale model size, expand pre-training data, improve post-training algorithms, and refine test-time scaling for a stronger thinking model.Planned efforts include optimizing Kimi-VL and releasing larger versions.Source: §5 · p. 19

Appendix … B.1 Image Benchmark

This appendix describes the image benchmarks used to evaluate academic and general visual reasoning, spatial understanding, mathematics, multi-image cognition, infographic interpretation, and OCR.

B.1 Image Benchmark

  • MMMU contains 11.5K questions from college exams, quizzes, and textbooks spanning six academic fields (Yue, Ni, et al. 2024).Its fields include Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.Source: §B.1 · p. 23
  • MMBench-EN-v1.1 evaluates 20 ability dimensions with 2,974 multiple-choice questions, organized around perception and reasoning (Yuan Liu et al. 2023).Its taxonomy supports evaluation at different levels across the ability dimensions.Source: §B.1 · p. 23
  • MMStar comprises 1,500 human-selected challenge samples assessing six core capabilities and 18 detailed axes (Lin Chen et al. 2024).Its selection is designed to be balanced and purified, and to require vision for multimodal evaluation.Source: §B.1 · p. 23
  • MMVet tests 16 integrations derived from six core vision-language capabilities (W. Yu et al. 2024).It is based on the idea that complex task-solving can involve integrating different core capabilities.Source: §B.1 · p. 23
  • RealWorldQA assesses real-world spatial understanding using more than 700 images paired with questions and verifiable answers (x.ai 2024).AI2D contains over 5,000 grade-school science diagrams, over 150,000 rich annotations, ground-truth syntactic parses, and more than 15,000 multiple-choice questions (Kembhavi et al. 2016).Source: §B.1 · p. 23
  • MathVision contains 3,040 visual-context math problems from real competitions, across 16 disciplines and five difficulty levels (K. Wang et al. 2024).MathVista combines mathematical and visual tasks requiring fine-grained visual understanding and compositional reasoning (P. Lu et al. 2023).Source: §B.1 · p. 23
  • BLINK evaluates multi-image visual cognition, including depth relationships, feature matching, digital forensics, and spatiotemporal reasoning (X. Fu et al. 2024).It includes diverse multi-image perceptual-similarity tasks validated through standardized protocols.Source: §B.1 · p. 23
  • InfoVQA tests reasoning over complex infographics using ANLS on the test set, while OCRBench scores five OCR tasks out of 1,000 points (Yuliang Liu et al. 2023).OCRBench tasks include text recognition, scene-text VQA, document VQA, key-information extraction, and handwritten math-expression recognition.Source: §B.1 · p. 23

B.2 Video and Long Document Benchmark

This section describes video benchmarks for college-level knowledge acquisition, expert and long-form video understanding, and temporal and visual-spatial reasoning.

B.2 Video and Long Document Benchmark

  • VideoMMMU (K. Hu et al. 2025) evaluates college-level knowledge using 900 questions on 300 expert videos across six disciplines and three cognitive stages.The stages are Perception, Comprehension, and Adaptation.Source: §B.2 · p. 23
  • Video-MME (C. Fu et al. 2024) covers 900 videos and 2,700 QA pairs with and without subtitles; MMVU (Y. Zhao et al. 2025) tests expert video understanding across 1,529 videos and 27 subjects.Video-MME videos total 254 hours; MMVU contains 3,000 expert-annotated questions.Source: §B.2 · pp. 23–24
  • MLVU (J. Zhou et al. 2024) evaluates long-video comprehension across 1,730 videos and 3,102 question-answer pairs, including action order, counting, topic reasoning, and anomaly recognition.Its close-ended tasks also include plot QA, ego reasoning, and needle QA.Source: §B.2 · p. 24
  • LongVideoBench (H. Wu et al. 2024) assesses long-form multimodal perception and relations with 6,678 questions; EgoSchema (Mangalam et al. 2023) tests egocentric long-video understanding across over 250 hours.LongVideoBench includes 3,763 videos with subtitles; EgoSchema has over 5,031 multiple-choice question-answer pairs.Source: §B.2 · p. 24
  • TOMATO (Shangguan et al. 2025) targets temporal reasoning, while VSI-Bench (Yang et al. 2024) evaluates visual-spatial capability in indoor-scene videos.TOMATO includes 1,417 videos and 1,484 questions; VSI-Bench has around 290 videos and over 5,000 question-answer pairs.Source: §B.2 · p. 24

B.3 Agent Benchmark

The section describes benchmarks for GUI grounding and multimodal agents, spanning corrected multi-platform interfaces, professional high-resolution UIs, interactive tasks across operating systems, and realistic Windows environments.

ScreenSpot V2

  • ScreenSpot V2 [Zhiyong Wu et al. 2024] evaluates GUI grounding across web, desktop, and mobile interfaces, correcting annotation and bounding-box issues in the original benchmark.The update addresses incorrect or ambiguous annotations, spelling mistakes, and mislabeled bounding boxes.Source: §B.3 · p. 24

ScreenSpot Pro

  • ScreenSpot Pro [K. Li et al. 2025] evaluates GUI grounding in complex, high-resolution professional interfaces using 1,581 real-world images and expert-annotated tasks.Its domain-specific interface conventions challenge models beyond consumer applications.Source: §B.3 · p. 24

OSWorld

  • OSWorld [T. Xie et al. 2024] evaluates open-ended computer tasks in an interactive real-computer environment across Ubuntu, Windows, and macOS.It supports task setup, execution-based evaluation, and interactive learning across arbitrary applications.Source: §B.3 · p. 24

WindowsAgentArena

  • WindowsAgentArena [Bonatti et al. 2024] evaluates multimodal agents interacting with applications and web tools in realistic Windows environments built on the OSWorld framework.Evaluations can be completed in under 20 minutes on Azure.Source: §B.3 · p. 24

Related papers

Loading 2504.07491v3…