PaperlayerView exact revision on arXiv

Source-linked AI summary

Kimi-VL Technical Report

Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Mengfan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen, Zongyu Lin

arXiv:2504.07491v3cs.CV

TL;DR

Open-source VLMs have lagged in scalable efficiency and advanced reasoning. Kimi-VL addresses these gaps with an MoE architecture, native-resolution vision, long-context processing, and long-CoT training with reinforcement learning. It achieves competitive broad multimodal performance, strong long-context results, and improved complex reasoning in Kimi-VL-Thinking-2506.

  • Problem

    Open-source VLMs have lagged in scalability, computational efficiency, and advanced reasoning, particularly because many use dense architectures without long-CoT reasoning.

  • Method

    Kimi-VL combines an MoE language model, native-resolution MoonViT vision encoder, multimodal training, a 128K context window, and long-CoT supervised fine-tuning with reinforcement learning for its thinking variant.

  • Results

    Kimi-VL demonstrates competitive or superior performance across multimodal reasoning, agent, long-context, perception, OCR, and high-resolution tasks, while Kimi-VL-Thinking-2506 further improves complex reasoning.

  • Takeaways & Limitations

    Kimi-VL indicates that an efficient open-source VLM can combine multimodal capability, long-context understanding, high-resolution perception, and advanced reasoning across diverse tasks.

  • Takeaways & Limitations

    The model remains limited for highly specialized or language-dependent problems, intricate multi-step reasoning, and some extremely long-sequence applications despite its 128K context window.

Abstract

from arXiv · show

We present Kimi-VL, an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM) that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities - all while activating only 2.8B parameters in its language decoder (Kimi-VL-A3B). Kimi-VL demonstrates strong performance across challenging domains: as a general-purpose VLM, Kimi-VL excels in multi-turn agent tasks (e.g., OSWorld), matching flagship models. Furthermore, it exhibits remarkable capabilities across diverse challenging vision language tasks, including college-level image and video comprehension, OCR, mathematical reasoning, and multi-image understanding. In comparative evaluations, it effectively competes with cutting-edge efficient VLMs such as GPT-4o-mini, Qwen2.5-VL-7B, and Gemma-3-12B-IT, while surpassing GPT-4o in several key domains. Kimi-VL also advances in processing long contexts and perceiving clearly. With a 128K extended context window, Kimi-VL can process diverse long inputs, achieving impressive scores of 64.5 on LongVideoBench and 35.1 on MMLongBench-Doc. Its native-resolution vision encoder, MoonViT, further allows it to see and understand ultra-high-resolution visual inputs, achieving 83.2 on InfoVQA and 34.5 on ScreenSpot-Pro, while maintaining lower computational cost for common tasks. Building upon Kimi-VL, we introduce an advanced long-thinking variant: Kimi-VL-Thinking-2506. Developed through long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL), the latest model exhibits strong long-horizon reasoning capabilities (64.0 on MMMU, 46.3 on MMMU-Pro, 56.9 on MathVision, 80.1 on MathVista, 65.2 on VideoMMMU) while obtaining robust general abilities. Code and models are publicly accessible at https://github.com/MoonshotAI/Kimi-VL.

1 Introduction

Kimi-VL addresses open-source VLM gaps in scalability, efficiency, and advanced reasoning through an MoE-based architecture with native-resolution vision. It delivers competitive multimodal, long-context, high-resolution, and reasoning capabilities, while Kimi-VL-Thinking extends performance through long-CoT training and reinforcement learning.

  • Open-source VLMs have lagged in scalability, computational efficiency, and advanced reasoning, with many relying on dense architectures without long-CoT reasoning.
  • Kimi-VL combines a 2.8B-activated MoE language model with a 400M native-resolution MoonViT encoder for diverse multimodal inputs and tasks.
  • Kimi-VL is competitive in multimodal reasoning and multi-turn agent benchmarks, including MMMU, MathVista, and OSWorld, without long thinking.
  • Kimi-VL handles long-context multimodal understanding within a 128K context window and performs strongly on long-video and long-document benchmarks.
  • Kimi-VL provides competitive ability across visual perception, visual world knowledge, OCR, and high-resolution OS screenshot tasks.
  • Long-CoT activation and reinforcement learning produce Kimi-VL-Thinking, which substantially improves complex multimodal reasoning while retaining or improving general visual abilities.

2 Approach

Kimi-VL combines a native-resolution MoonViT encoder, an MLP projector, and a sparse MoE language model, then trains them through staged multimodal and long-context procedures. Post-training adds instruction tuning and long-CoT reinforcement learning to support dialogue, long-context processing, and extended reasoning.

  • Model Architecture: Kimi-VL uses MoonViT, an MLP projector, and a Moonlight MoE language model with 2.8B activated and 16B total parameters.The architecture connects native-resolution visual features to the language model through a two-layer MLP.
  • Model Architecture: MoonViT processes images at native varying resolutions, packs patch sequences, and uses 2D RoPE to improve fine-grained positional representation at high resolutions.Its preprocessing shares computation operators with language-model training for varying image resolutions.
  • Pre-Training Stages: 4.4T tokens span four pre-training stages: standalone ViT training followed by joint pre-training, cooldown, and long-context activation.The joint stages update language and multimodal capabilities while preserving text abilities through combined training.
  • Post-Training Stages: Long-CoT supervised fine-tuning and reinforcement learning produce Kimi-VL-Thinking, whose training develops planning, metareasoning, and error-detection abilities.Inference remains standard autoregressive generation rather than requiring specialized parallel planning algorithms.
  • Pre-Training Stages: 128K context is activated through two fourfold extensions using 25% long data and 75% replayed shorter data, covering long text, videos, and documents.The model passes needle-in-a-haystack evaluations on both long pure-text and long-video inputs after this activation.

3 Data Construction

Kimi-VL’s data construction combines multimodal pre-training sources with filtering, synthesis, deduplication, and domain-specific processing. The corpus covers alignment, interleaved multimodal context, OCR, knowledge, agent interaction, video, text, conversational instruction following, and long-CoT reasoning.

  • Corpus Overview: The multimodal pre-training corpus contains six categories: caption, interleaving, OCR, knowledge, video, and agent data.
  • Corpus Overview: Data processing pipelines use filtering, synthesis, and deduplication to preserve language capabilities and align knowledge across modalities.
  • Multimodal Data: Interleaving data supports multi-image comprehension, longer multimodal context learning, detailed image knowledge, and language-ability maintenance.The construction also applies reordering so images and text remain correctly aligned.
  • OCR Data: OCR data spans multilingual, dense-layout, web, handwritten, single-page, and multi-page inputs, enabling both image OCR and long-document comprehension.The model is trained on figures, tables, geometry diagrams, plots, and natural-scene text as well.
  • Agent and Video Data: Agent data combines screenshots and action trajectories collected from virtual-machine environments into dense grounding and continuous trajectory formats.
  • Agent and Video Data: Video data covers long-context sequences such as hour-long videos and fine-grained spatio-temporal correspondence in short clips.Sources include open datasets and in-house web-scale videos spanning varied durations, scenes, and tasks.
  • Post-Training Data: Instruction-tuning data enhances conversational and instruction-following abilities across non-reasoning domains, while long-CoT data supplies verified multi-step reasoning paths.

4 Evaluation

Kimi-VL delivers competitive or superior results across academic, visual, mathematical, OCR, agent, long-context, video, and multi-image benchmarks with a parameter-efficient architecture. Its long-thinking variants further improve complex multimodal reasoning, while test-time scaling and the 2506 update enhance performance and efficiency.

  • Overall Comparison: Kimi-VL outperforms Qwen2.5-VL-7B on 19 out of 24 benchmarks despite using fewer activated parameters.Kimi-VL activates 2.8B parameters versus 4.5B for DeepSeek-VL2 and 2.59× more activated parameters for Qwen2.5-VL-7B.
  • Visual Understanding: 83.1% on MMBench-EN-v1.1, 84.9% on AI2D, and 68.1% on RealWorldQA demonstrate strong general visual understanding.Kimi-VL outperforms all compared efficient VLMs on MMBench-EN-v1.1 and surpasses GPT-4o on AI2D.
  • OCR and Documents: 83.2% on InfoVQA and 86.7% on OCRBench show strong document understanding and optical character recognition.These results outperform GPT-4o on InfoVQA and all compared models on OCRBench.
  • Agent Capabilities: 8.22% on OSWorld and 10.4% on WindowsAgentArena exceed GPT-4o, while ScreenSpot-Pro reaches 34.5% for difficult 4K-screen grounding.ScreenSpot-V2 accuracy is 92.0%.
  • Long-Context Understanding: 35.1% on MMLongBench-Doc and 64.5% on LongVideoBench establish competitive long-form understanding across documents and videos.Kimi-VL uses a 128K context window and surpasses efficient competitors on these tasks.
  • Test-Time Scaling: Increasing maximum thinking length steadily improves MathVision accuracy from 18.7% at 1k tokens to 36.8% at 16k tokens, while MathVista saturates at 70.9% by 4k tokens.Longer reasoning chains therefore benefit benchmarks differently.
  • Thinking-2506: 56.9% on MathVision, 80.1% on MathVista, 46.3% on MMMU-Pro, and 64.0% on MMMU mark gains for Kimi-VL-Thinking-2506.The 2506 version also reduces average reasoning output length by around 20%.

5 Conclusion, Limitation, and Future Work

Kimi-VL combines MoE-based efficiency, a 128K context window, native-resolution visual processing, and long-chain reasoning across multimodal tasks. The authors identify limits in model scale, reasoning depth, and advanced long-context processing, and propose scaling data, models, and post-training.

  • Conclusion: Kimi-VL uses balanced multimodal and text-only training with an MoE architecture for scalable efficiency.Its 128K context window and MoonViT encoder support lengthy text, video, and high-resolution visual inputs.
  • Conclusion: Kimi-VL-Thinking supports long-chain reasoning for complex image and video inference.The conclusion presents this capability alongside long-context and high-resolution processing.
  • Limitations: The current model remains limited for highly specialized, language-dependent, or extremely complex problems.The authors attribute this boundary partly to the model’s limited size.
  • Limitations: Reasoning has not reached its theoretical upper bound for intricate tasks requiring multi-step inference or deeper contextual understanding.This is identified as a remaining challenge rather than a universal failure.
  • Limitations: Despite its 128K context window, limited attention-layer parameters constrain performance on extremely long or high-volume contextual inputs.The attention layers are described as comparable to a 3B model.
  • Future Work: Future work will scale model size and pretraining data while refining post-training and test-time scaling mechanisms.The authors also plan larger Kimi-VL versions and further optimization.

Core Contributors

The paper’s listed core contributors are provided as author names. The supplied passages contain no roles, affiliations, or contribution descriptions.

  • Authors: The contributor list includes Bohong Yin, Bowei Xing, Cheng Chen, Chu Wei, and Dehao Zhang.The passage continues with additional authors.
  • Authors: The contributor list also includes Zaida Zhou, Zhaowei Li, Zhiqi Huang, Zhilin Yang, and Ziwei Chen.These names appear in the continuation of the author list.

B.1 Image Benchmark

The image benchmarks cover general multimodal ability, visual perception, OCR, multi-image cognition, and mathematical reasoning across varied datasets and task designs.

  • General multimodal reasoning: MMMU evaluates 11.5K college-level multimodal questions across six academic fields.Its questions come from exams, quizzes, and textbooks.
  • General multimodal reasoning: MMBench-EN-v1.1 tests 20 ability dimensions through 2,974 multiple-choice questions spanning perception and reasoning.
  • General multimodal reasoning: MMStar uses 1,500 human-selected challenge samples to assess six core capabilities and 18 detailed axes.
  • Mathematical reasoning: MathVision contains 3,040 competition-sourced visual mathematics problems spanning 16 disciplines and five difficulty levels.
  • Multi-image understanding: BLINK evaluates multi-image visual cognition through depth, feature matching, digital forensics, and spatiotemporal reasoning tasks.
  • OCR and visual information extraction: InfoVQA assesses reasoning over infographics combining text, graphics, and visual elements, using ANLS on the test set.

B.2 Video and Long Document Benchmark

The video and long-document benchmarks span academic knowledge, expert-level understanding, temporal reasoning, spatial comprehension, and long-form video-language perception.

  • Academic video understanding: VideoMMMU evaluates college-level knowledge acquisition with 300 expert-level videos and 900 questions across six academic disciplines.Questions target perception, comprehension, and adaptation.
  • Long-form video understanding: Video-MME contains 900 videos totaling 254 hours and 2,700 question-answer pairs across six domains, evaluated with and without subtitles.
  • Long-form video understanding: MLVU evaluates long-video comprehension across 1,730 videos and 3,102 question-answer pairs from scenarios including sport, egocentric, life-record, and tutorial content.
  • Long-form video understanding: LongVideoBench measures long-form multimodal perception and relation capability using 3,763 videos, subtitles, and 6,678 questions across 17 categories.
  • Egocentric video understanding: EgoSchema measures long-form video understanding in egocentric scenarios using over 5,031 questions and more than 250 hours of real-world video.
  • Temporal reasoning: TOMATO focuses on temporal reasoning through 1,417 videos and 1,484 annotated questions covering counting, direction, rotation, trends, velocity, and visual cues.

B.3 Agent Benchmark

The agent benchmarks evaluate GUI grounding and interactive multimodal task execution across high-resolution professional interfaces and complete operating-system environments.

  • GUI grounding: ScreenSpot V2 evaluates GUI grounding across web, desktop, and mobile interfaces while addressing annotation and bounding-box problems in the original dataset.
  • GUI grounding: ScreenSpot Pro tests GUI grounding in high-resolution, complex professional interfaces using 1,581 real-world images and expert-annotated tasks.
  • Interactive computer agents: OSWorld evaluates open-ended multimodal agent tasks in realistic computer environments spanning Ubuntu, Windows, and macOS.The platform supports task setup, execution-based evaluation, and interactive learning across arbitrary applications.
  • Interactive computer agents: WindowsAgentArena assesses multimodal agents in realistic Windows environments with full application and web-tool interaction built on OSWorld.
Loading 2504.07491v3…