Source-linked AI summary
Kwai Keye-VL-2.0 Technical Report
Kwai Keye Team, Bin Wen, Changyi Liu, Chengru Song, Chongling Rao, Guowang Zhang, Han Li, Haonan Fan, Hengrui Ju, Jiankang Chen, Jiapeng Chen, Jiawei Yuan, Kaixuan Yang, Kaiyu Jiang, Kun Gai, Lingzhi Zhou, Na Nie, Sen Na, Tianke Zhang, Tingting Gao, Xuanyu Zheng, Yulong Chen, Fan Yang, Haixuan Gao, Lele Yang, Mingqiao Liu, Muxi Diao, Qi Zhang, Qile Su, Wei Chen, Wentao Hong, Xingyu Lu, Yancheng Long, Yankai Yang, Yingxin Li, Yiyang Fan, Yu Xia, Yuzhe Chen, Ziliang Lai, Chuan Yi, Haonan Jia, Tianming Liang, Weixin Xu, Xiaoxiao Ma, Yang Tian, Yufei Han, Feng Han, Hang Li, Jing Wang, Jinghui Jia, Junmin Chen, Junyu Shi, Ruilin Zhang
TL;DR
Hour-level video understanding faces prohibitive context costs and conflicts between multimodal agent capabilities and foundational reasoning. Keye-VL-2.0 combines sparse long-context modeling with staged multimodal post-training, achieving leading performance at its scale on long-video comprehension and fine-grained temporal localization.
Problem
Extending multimodal models from short videos to long-horizon agentic reasoning is limited by extreme context costs and catastrophic forgetting during heterogeneous task integration.
Method
Keye-VL-2.0 combines DSA in a GQA-based MoE multimodal backbone with cross-modal on-policy distillation and specialized reinforcement learning.
Results
Keye-VL-2.0 achieves leading performance at its scale on long-video comprehension and fine-grained temporal localization while remaining competitive across code, tool-use, OCR, and reasoning benchmarks.
Takeaways & Limitations
The model extends effective context modeling to 256K tokens and is designed to support deployable hour-level video applications under controllable training and inference cost.
Abstract
from arXiv · showhide
We introduce Kwai Keye-VL-2.0-30B-A3B, an open-source Mixture-of-Experts (MoE) multimodal foundation model designed to advance long-video understanding and agentic intelligence. To address the challenges of ultra-long contexts, information redundancy, and prohibitive computational costs inherent in hour-level videos, Keye-VL-2.0 is the first to adapt DeepSeek Sparse Attention (DSA) to GQA-based multimodal architectures, enabling lossless 256K context processing while capturing critical frames and long-range temporal dependencies. This architecture is underpinned by a highly optimized training and inference infrastructure, including scalable video I/O, heterogeneous ViT-LM parallelism, and custom DSA kernels that significantly maximize throughput and minimize computational overhead. Furthermore, to overcome the algorithmic dilemma of catastrophic forgetting during multi-task alignment, we introduce Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) paired with Context-RL and Video-RL. By distilling dense token-level teacher feedback from on-policy rollouts back into the MoE backbone, which activates only 3B parameters, Keye-VL-2.0 natively empowers advanced agent collaboration across Code, Tool, and Search scenarios with multimodal self-correction. Extensive evaluations across video understanding, temporal grounding, reasoning, STEM, and agent benchmarks demonstrate that Keye-VL-2.0-30B-A3B achieves state-of-the-art performance among models of similar scale, particularly excelling in fine-grained temporal localization on TimeLens and long-video comprehension on Video-MME-v2 and LongVideoBench. We release our model checkpoints to accelerate community progress toward scalable and robust multimodal agentic applications.
1 Introduction
Kwai Keye-VL-2.0-30B-A3B extends the Keye-VL line toward long-horizon agentic reasoning by addressing extreme video-context scaling and multimodal alignment conflicts. Its introduction highlights a multimodal DSA-based MoE architecture, Cross-Modal MOPD, and competitive temporal-localization performance.
- Motivation: Keye-VL-2.0-30B-A3B advances earlier Keye-VL models from short-video comprehension toward long-horizon agentic reasoning while confronting infrastructure and alignment roadblocks.The model builds on Keye-VL and Keye-VL-1.5, which established vision-language alignment and short-video comprehension.
- Extreme Context Scaling via Multimodal DSA: Standard dense attention at 256K contexts causes KV-cache expansion and computational walls, forcing aggressive frame subsampling that sacrifices temporal continuity.The introduction identifies extreme-context scaling as a central infrastructure bottleneck for multimodal video models.
- Extreme Context Scaling via Multimodal DSA: Keye-VL-2.0 addresses extreme-context scaling with a 30-billion-parameter Mixture-of-Experts foundation that activates only 3 billion parameters.The architecture introduces multimodal DeepSeek Sparse Attention to overcome latency and scalability limits.
- Resolving Modality Conflict via Cross-Modal MOPD: Directly adding video understanding and tool use can trigger catastrophic forgetting of foundational STEM, mathematical, and linguistic reasoning abilities.The paper frames this degradation as the “Multimodal Alignment Dilemma.”
- Resolving Modality Conflict via Cross-Modal MOPD: Cross-Modal Multi-Teacher On-Policy Distillation is introduced to address ambiguity and instability in end-to-end co-optimization during multimodal alignment.The method is presented as a response to the conflict between advanced multimodal capabilities and retained foundational reasoning.
- Results: Keye-VL-2.0-30B-A3B achieves top-tier fine-grained temporal-localization results across TimeLens benchmarks and substantially outperforms Gemini-3-Flash on several temporal-grounding settings.The cited TimeLens benchmarks include ActivityNet, QVHighlights, and Charades, with comparisons spanning open- and closed-source models.
2 Model Architecture
Keye-VL-2.0 combines a four-component multimodal backbone with native-resolution visual processing, unified image-video encoding, and GQA-compatible DSA sparse attention. These designs preserve fine-grained visual and long-range multimodal information while reducing the cost of 256K-context modeling.
- Core Components: The backbone comprises a ViT vision encoder, Qwen3-30B-A3B-Thinking-2507 language decoder, trained MLP projector, and GQA-compatible DSA sparse attention module.The ViT is inherited from Keye-VL-1.5-8B, while the projector is randomly initialized and trained in Stage 0.
- DSA Sparse Attention: DSA replaces quadratic full attention for 256K multimodal contexts with global MQA-style indexing and grouped GQA aggregation over selected tokens.With k = 2048, core attention complexity is reduced from O(L^2) to O(Lk), where k ≪ L.
- Native-Resolution Vision Encoder: Native-resolution visual encoding preserves aspect ratio, layout relations, object geometry, and fine-grained text signals for document, OCR, chart, and video understanding.The encoder uses adaptive position encoding, 2D RoPE, sequence packing, and distribution-aligned ViT pre-training.
- Unified Image-Video Encoding: Unified dynamic-resolution encoding allocates image tokens by original pixel size and encodes each sampled video frame as an independent high-resolution image.The same visual encoder handles images and videos while preserving temporal information through additional frame-level signaling.
- DSA Sparse Attention: The Lightning Indexer shares one key head across query heads, reducing computation and memory traffic while remaining efficient for sequences containing hundreds of thousands of tokens.Its efficiency is further supported by FP8 implementation and ReLU-based scoring.
- DSA Training: Two-stage DSA training first aligns the indexer with dense GQA attention, then trains the full model in sparse mode to rely on dynamically selected evidence while preserving dense-model capabilities.Dense warm-up uses approximately 2B multimodal tokens before sparse adaptation unfreezes all parameters.
3 Pre-Training
Keye-VL-2.0 is pre-trained through a four-stage curriculum that progresses from projector-only visual-language alignment to full-parameter 256K-context multimodal training. The curriculum expands capabilities across perception, reasoning, task-oriented supervision, and ultra-long multimodal inputs.
- Curriculum: The four-stage curriculum progresses from Projector initialization, through 32K and 64K full-parameter training, to 256K long-context multimodal training.Stage 0 trains only the Projector; Stages 1–3 continue full-parameter training while progressively extending context length.
- Stage 2: Stage 2 extends training to 64K and approximately 2T tokens, adding task-oriented supervision for OCR, STEM, GUI, grounding, counting, VQA, coding, tool use, and search.Pure-text math, STEM, code, tool-use, and search/RAG data preserve language-side reasoning and agentic capabilities during multimodal training.
- Stage 3: Stage 3 extends the maximum sequence length to 256K, mixing long- and short-context samples 1:1 across long videos, documents, code, and agent trajectories.The stage targets retrieval, aggregation, and cross-position reasoning while preserving performance on conventional inputs.
- Stage 0: Stage 0 aligns visual and language representations using caption and interleaved image-text data while freezing the ViT and LLM.Only the Projector is updated, providing a low-risk initialization before full-parameter multimodal training.
- Stage 1: Stage 1 trains all parameters at 32K on approximately 1T tokens for vision-language alignment, image and video understanding, OCR, and general language capability.Its data include captions, interleaved image-text and video-text sequences, pure-text QA, and OCR; videos are split into 15-second segments.
4 Post-Training
Post-training combines large-scale multimodal supervised fine-tuning with synthetic reasoning supervision and domain-specific reinforcement learning. Cross-Modal Multi-Teacher On-Policy Distillation consolidates heterogeneous capabilities while preserving modality-specific reasoning, perception, and agent behavior.
- Supervised Fine-Tuning: SFT uses approximately 500B tokens spanning text, video, perception, reasoning, agent, and long-context data to stabilize multimodal instruction following.About 40% is text-only, supporting instruction following, knowledge QA, and textual reasoning.
- Multimodal Instruction Mixture: Video, perception, reasoning, agent, and long-context data respectively train temporal localization, fine-grained extraction, cross-modal inference, task execution, and long-range information modeling.The mixture is designed for complementary capability development rather than simply increasing visual-data proportion.
- Reasoning Supervision: Synthetic CoT augments final-answer supervision with filtered teacher reasoning traces, while direct-answer data preserve concise outputs for simple perception tasks.Mathematical CoT samples receive Doubt2Clean second-pass review across 27 datasets.
- Synthetic-Data Reinforcement Learning: Procedurally generated image-difference tasks provide rule-verifiable rewards for changed-region localization and structured operation-set prediction without requiring a learned reward model.The framework uses controlled edits whose ground truth is known during generation.
- Agent Reinforcement Learning: Search RL trains multi-turn retrieval, reading, and answer-generation trajectories with final-answer correctness as the primary reward and lightweight intermediate verification when available.Tasks vary in query refinement, result selection, and content checking requirements.
- Multi-Teacher Distillation: Cross-Modal Multi-Teacher On-Policy Distillation addresses cross-domain interference by combining dynamic teacher routing, SPRR alignment, top-k overlap estimation, token-category scaling, and localized repetition penalties.These mechanisms consolidate heterogeneous RL teachers without forcing all domains into one response style.
5 Efficient Training and Inference Infrastructure
The infrastructure removes video I/O and heterogeneous-workload bottlenecks through scalable data services, specialized parallelism, and load balancing, while optimized DSA kernels improve training and inference efficiency. It also supports deterministic, memory-efficient RL training and ultra-long-video inference.
- Training Infrastructure: ExtraIO decouples video decoding and frame sampling from training asynchronously, while heterogeneous ViT–LM parallelism and two-level load balancing address variable-length workloads.ViT and LM modules use separate sharding strategies, with recompute-or-offload reducing ViT activation memory to nearly zero.
- Training Infrastructure: Approximately 20% higher end-to-end training throughput results from balancing multimodal tokens and LM samples across ViT and LM parallel dimensions.The scheme equalizes compute and memory across ViT DP and LM DP/PP, mitigating underutilization from variable visual-token ratios and sequence lengths.
- DSA Kernels: More than 2× speedup over an open-source-code baseline comes from optimizing DSA with FlashInfer and TileLang.Additional optimizations reduce score-storage waste under packing and accelerate short-sequence long-context SFT through causal KV iteration.
- RL Infrastructure: Deterministic Top-k computation preserves training–inference consistency while flashinfer.topk delivers a 2–3× speedup and chunked indexing reduces peak memory for variable-length RL batches.The RL implementation partitions score computation along the sequence dimension and aggregates chunk-wise Top-k results.
- Ultra-Long-Video Inference: Over 3× lower prefill cost and over 5× lower decode cost versus full attention are achieved for 128K-context inference through DSA-specific decode optimizations.The system also chunks ViT processing and deduplicates similar adjacent-query Top-k KV sets to reduce peak memory and effective KV tokens.
6 Comprehensive Evaluation
Keye-VL-2.0-30B-A3B performs strongly across video understanding, coding, tool use, and general multimodal reasoning benchmarks. It is particularly effective in long-video understanding, temporal grounding, hallucination resistance, and visual mathematical reasoning.
- Video understanding: Keye-VL-2.0 achieves the best result on LongVideoBench and Video-MME-v2 accuracy, while remaining competitive on MLVU and Video-MME.On Video-MME-v2, strong accuracy under both 64-frame and 512-frame settings indicates benefits from denser visual context without losing long-range aggregation ability.
- Temporal grounding: Keye-VL-2.0 achieves the best mIoU on all three TimeLens subsets, validating its fine-grained temporal alignment.TimeLens reports mIoU on ActivityNet-TimeLens, QVHighlights-TimeLens, and Charades-TimeLens using cleaned annotations.
- Coding: 64.2 on LiveCodeBench v6 and 71.5 on OJBench show strong algorithmic reasoning and execution-based self-correction, while 62.0 on SWE-bench Verified is competitive.The benchmarks cover contamination-resistant programming, online-judge-style correctness, and repository-level issue resolution.
- Tool use: Keye-VL-2.0 achieves the best results on τ2-Bench and VitaBench and ranks second on BFCL-V4.These results indicate strong tool selection, parameter filling, state tracking, and recovery behavior in multi-turn environments.
- General vision-language reasoning: Keye-VL-2.0-30B-A3B maintains strong general multimodal performance, particularly in hallucination resistance and visual mathematical reasoning.The evaluation spans perception, document understanding, grounding, counting, spatial understanding, visual mathematics, dynamic reasoning, and expert-level multimodal understanding.
7 Conclusion and Future Work
Keye-VL-2.0 is presented as an open-source 30B-class MoE multimodal model with 3B active parameters, 256K-token context modeling, and practical hour-level video understanding. Its post-training methods consolidate heterogeneous teachers and rewards without sacrificing core reasoning, while future work targets real-world business deployment.
- Contributions: Keye-VL-2.0 is an open-source 30B-class MoE multimodal model with only 3B active parameters and effective 256K-token context modeling.DeepSeek Sparse Attention is integrated into a GQA-based multimodal backbone to support hour-level video understanding under controllable training and inference cost.
- Post-training: Cross-Modal Multi-Teacher On-Policy Distillation, Context-RL, Video-RL, and domain-specific reinforcement learning consolidate heterogeneous teachers and rewards without sacrificing core reasoning ability.The pipeline addresses capability conflicts among perception, reasoning, long-context understanding, and agentic behaviors.
- Future work: Future work will integrate fine-grained long-video perception and image-text understanding into product pipelines for generative recommendation, content governance, and commercial targeting.These applications are intended to provide denser semantic signals for recommendation matching, content quality assessment, and fine-grained advertising labels.
A Case Study
The case study presents qualitative, user-facing examples of Keye-VL-2.0 across text, image, video, and agentic service tasks, while omitting internal reasoning traces.
- A Case Study: Qualitative examples illustrate Keye-VL-2.0’s final user-facing responses on representative text, image, video, and agentic service tasks.Because the model uses a thinking-oriented policy by default, the case study omits internal reasoning traces and separate thinking or answer tags.
Case I: Logical Constraint Solving · Prompt
The case presents a symbolic cybersecurity puzzle in which five unique letter-digit keys must be assigned to slots despite exactly one corrupted evidence node. Keye-VL-2.0 solves it by identifying the corrupted node, constructing a valid sequence, and verifying all statements.
- Case I: Logical Constraint Solving: Five unique Access Keys must fill five slots, each combining one letter from {A, B, C, D, E} with one digit from {1, 2, 3, 4, 5}.No letter or digit may repeat.
- Prompt: Exactly one of five intercepted evidence nodes is corrupted, making every statement in that node false while statements in intact nodes remain true.The solver must distinguish the corrupted node from the truthful nodes.
- Prompt: The puzzle requires satisfying multiple symbolic constraints simultaneously across key identities, slot positions, and node statements.The access-key structure couples letter uniqueness, digit uniqueness, and evidence-node consistency.
- Case I: Logical Constraint Solving: Keye-VL-2.0 identifies the single corrupted evidence node as part of solving the case.This step determines which node's statements should be interpreted as false.
- Case I: Logical Constraint Solving: Keye-VL-2.0 constructs a valid slot assignment for the five access keys.The assignment respects the puzzle's no-repetition requirements.
- Case I: Logical Constraint Solving: Keye-VL-2.0 verifies every statement against the final sequence after determining the assignment.Verification checks the completed sequence against both the corrupted and intact-node conditions.
Case II: Spatial Reasoning in an Indoor Layout · Prompt
This case presents a labeled top-down indoor scene for evaluating spatial reasoning. The prompt tests orientation, facing relationships, egocentric left-right relations, object positions, and movement direction.
- Prompt: The image depicts a top-down room view in which each furniture item is identified by a letter.The scene is introduced as an indoor layout for answering spatial questions.
- Prompt: The prompt asks for the facing directions of A, the sofa, and C, the TV, and whether they face each other.These questions evaluate object orientation and pairwise facing relationships.
- Prompt: The prompt asks which furniture lies on the left and right when entering through G, the door.This tests egocentric left-right reasoning from an entrance viewpoint.
- Prompt: The prompt asks for the spatial relationship between D, the floor lamp, and H, the window.This evaluates relative positioning among labeled room objects.
- Prompt: The prompt asks which direction B, the coffee table, should move to reach E, the dining table, and asks about a room corner.These questions test directional movement and location identification.
- Prompt: Figure 7 characterizes the case as identifying object orientations, egocentric left-right relations, furniture positions, and object-movement direction.The figure summarizes the spatial capabilities exercised by the labeled indoor scene.
- Prompt: The surrounding cases extend multimodal understanding to cardiac anatomy, including circulation, valve localization, pathway reconstruction, and error detection.The anatomical example uses a labeled heart diagram with blood-flow arrows and explanatory text.
- Prompt: The video examples cover scene-level understanding in a historical documentary and a daily vlog while preserving temporal boundaries and fine-grained details.The documentary tracks transitions across historical and expert-discussion scenes, whereas the vlog follows school preparation, activities, family interaction, and an evening routine.
Case VI: Multi-Domain Service Agent
Keye-VL-2.0 demonstrates multi-domain service orchestration by coordinating personalized workflows across hotel, delivery, and in-store services.
- Multi-Domain Service Orchestration: Keye-VL-2.0 coordinates personalization, search, geolocation, delivery-time estimation, booking, payment, recommendation, and order creation across hotel, delivery, and in-store domains.The case illustrates an end-to-end orchestration workflow spanning multiple service functions and domains.
B Contribution (Alphabetical order) · Core Contributors · Supporting Contributors
The contribution section lists core and supporting contributors, with the core-contributor roster substantially larger than the supporting-contributor roster.
- Core Contributors: Core Contributors include Bin Wen, Changyi Liu, Chengru Song, Chongling Rao, Guowang Zhang, Han Li, Haonan Fan, and Hengrui Ju.
- Core Contributors: Core Contributors also include Jiankang Chen, Jiapeng Chen, Jiawei Yuan, Kaixuan Yang, Kaiyu Jiang, Kun Gai, and Lingzhi Zhou.
- Core Contributors: The remaining Core Contributors are Na Nie, Sen Na, Tianke Zhang, Tingting Gao, Xuanyu Zheng, and Yulong Chen.
- Supporting Contributors: Supporting Contributors include Feng Han, Hang Li, Jing Wang, and Jinghui Jia.
- Supporting Contributors: Supporting Contributors also include Junmin Chen, Junyu Shi, Ruilin Zhang.