PaperlayerView exact revision on arXiv

Source-linked AI summary

Kwai Keye-VL 1.5 Technical Report

Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, Fan Yang, Guorui Zhou, Guowang Zhang, Han Shen, Hao Peng, Haojie Ding, Hao Wang, Haonan Fan, Hengrui Ju, Jiaming Huang, Jiangxia Cao, Jiankang Chen, Jingyun Hua, Kaibing Chen, Kaiyu Jiang, Kaiyu Tang, Kun Gai, Muhao Wei, Qiang Wang, Ruitao Wang, Sen Na, Shengnan Zhang, Siyang Mao, Sui Huang, Tianke Zhang, Tingting Gao, Wei Chen, Wei Yuan, Xiangyu Wu, Xiao Hu, Xingyu Lu, Yi-Fan Zhang, Yiping Yang, Yulong Chen, Zeyi Lu, Zhenhua Wu, Zhixin Ling, Zhuoran Yang, Ziming Li, Di Xu, Haixuan Gao, Hang Li, Jing Wang, Lejian Ren, Qigen Hu, Qianqian Wang, Shiyao Wang, Xinchen Luo, Yan Li, Yuhang Hu, Zixing Zhang

arXiv:2509.01563v3cs.CV

TL;DR

Video models face a trade-off between temporal coverage and spatial resolution. Keye-VL-1.5 combines adaptive Slow-Fast encoding, progressive context-length pre-training, and reasoning- and preference-focused post-training; evaluations report advances in video understanding while maintaining strong general vision-language performance.

  • Problem

    Video understanding requires models to handle temporal relationships while balancing temporal coverage against spatial resolution.

    Source: §1 · p. 4

  • Method

    Keye-VL-1.5 combines adaptive Slow-Fast video encoding, progressive context-length pre-training, and reasoning- and preference-focused post-training.

    Source: §1 · p. 4, §7 · p. 24

  • Results

    Keye-VL-1.5 advances video understanding while maintaining strong performance on general vision-language tasks.

    Source: §1 · p. 4, §7 · p. 24

  • Takeaways & Limitations

    The authors present Keye-VL-1.5 as a practical solution for next-generation multimodal models aimed at complex video understanding and reasoning.

    Source: §1 · p. 4

    Keye-VL-1.5 trails MiMoVL-7B-RL-2508 in fluency (+0.23), relevance (+0.08), and creativity (+0.15), indicating challenges in language-generation sophistication.

    Source: §6.5 · p. 21

Abstract

from arXiv · show

In recent years, the development of Large Language Models (LLMs) has significantly advanced, extending their capabilities to multimodal tasks through Multimodal Large Language Models (MLLMs). However, video understanding remains a challenging area due to the dynamic and information-dense nature of videos. Existing models struggle with the trade-off between spatial resolution and temporal coverage when processing video content. We present Keye-VL-1.5, which addresses fundamental challenges in video comprehension through three key innovations. First, we introduce a novel Slow-Fast video encoding strategy that dynamically allocates computational resources based on inter-frame similarity, processing key frames with significant visual changes at higher resolution (Slow pathway) while handling relatively static frames with increased temporal coverage at lower resolution (Fast pathway). Second, we implement a progressive four-stage pre-training methodology that systematically extends the model's context length from 8K to 128K tokens, enabling processing of longer videos and more complex visual content. Third, we develop a comprehensive post-training pipeline focusing on reasoning enhancement and human preference alignment, incorporating a 5-step chain-of-thought data construction process, iterative GSPO-based reinforcement learning with progressive prompt hinting for difficult cases, and alignment training. Through extensive evaluation on public benchmarks and rigorous internal human assessment, Keye-VL-1.5 demonstrates significant improvements over existing models, particularly excelling in video understanding tasks while maintaining competitive performance on general multimodal benchmarks.

1 Introduction

Keye-VL-1.5 is an 8-billion-parameter multimodal model designed to address video understanding’s trade-off between temporal coverage and spatial resolution. It combines adaptive video encoding, progressive long-context pre-training, and reasoning- and preference-focused post-training.

Motivation

  • Video understanding remains challenging because dynamic, information-dense content requires temporal reasoning while balancing temporal coverage against spatial resolution.Uniform frame sampling at fixed resolution can perform suboptimally when fine-grained visual detail and temporal consistency are needed.Source: §1 · p. 4

Results

  • Keye-VL-1.5 achieves state-of-the-art video-understanding performance and shows significant improvements over existing models, particularly on video tasks.The report also describes it as maintaining robust capabilities in general vision-language tasks and validates it on public benchmarks and through internal human assessment.Source: §1 · p. 4

Architecture and Slow-Fast Video Encoding

  • Slow-Fast encoding allocates higher resolution to visually changing key frames and greater temporal coverage at lower resolution to relatively static frames.The allocation is based on inter-frame similarity and guided by patch-based similarity functions, balancing spatial detail with temporal breadth.Source: §1 · p. 4

Progressive Pre-training with Long Context Extension

  • Four pre-training stages progressively extend context from 8K to 128K tokens, supporting longer videos and more complex visual content.The methodology begins with cross-modal alignment and multi-task learning; model fusion combines different data mixtures to improve robustness and reduce bias.Source: §1 · p. 4

Post-training for Reasoning and Human Preference Alignment

  • Post-training combines five-step chain-of-thought data construction, iterative GSPO reinforcement learning with progressive hints for difficult samples, and alignment training.The pipeline uses the RL model to generate improved SFT data for subsequent RL rounds, then aligns responses for instruction following, formatting, and human preferences.Source: §1 · p. 4

2 Model Architecture

Keye-VL-1.5 combines a native-resolution vision encoder with image- and video-specific visual encoding strategies, built around a ViT, MLP projector, and Qwen3-8B language decoder.

2 Model Architecture

  • The model uses SigLIP-400M-384-14 as its vision encoder, a randomly initialized MLP projector pretrained in Stage 1, and Qwen3-8B as its language decoder.The report describes this as a classic MLLM architecture with three components: a ViT, projector, and language decoder.Source: §2 · p. 5

2.1 Vision Encoder with Native-Resolution

  • The native-resolution ViT extends SigLIP’s learned position embeddings through interpolation, adds 2D RoPE, and trains across variable-resolution images using NaViT packing and FlashAttention.The ViT is initialized from SigLIP-400M-384-14; its native-resolution modifications are optimized with SigLIP loss using 500B tokens from open-source and in-house data.Source: §2.1 · pp. 5–6

2.2 Visual Encoding

  • Image encoding allocates 20,480 LLM-side tokens per image, intended to cover images exceeding tens of millions of pixels.The report says this budget is sufficient for the model to see image details.Source: §2.2 · p. 6
  • Slow-Fast video encoding assigns higher resolution to rapidly changing frames and more frames at lower resolution to relatively static content.The first frame is slow; a later frame is fast when its patch similarity to the latest slow frame exceeds 95%, and fast-frame tokens receive 30% of a slow frame’s budget. Binary search fits slow-frame tokens within the video budget, while special tokens and absolute timestamps mark frame boundaries and timing.Source: §2.2 · pp. 6–7, Figure 3 · p. 6

3 Pre-Training

The pre-training section describes a diverse multimodal corpus and a staged training recipe, including context extension from 8K to 128K tokens. Its data pipelines cover image captions, OCR and VQA, grounding and counting, interleaved content, and video understanding.

3.1 Data Pipeline

  • The corpus exceeds 1 trillion tokens across six data categories, with category-specific filtering, CLIP screening, high-quality MLLM selection, and image deduplication.The categories are image caption, OCR and VQA, grounding and counting, interleaved, video understanding, and pure text data.Source: §3.1 · p. 7

3.1.1 Image Caption Data

  • Caption data retains image-text pairs with CLIP scores above 0.9 and uses MLLM-generated recaptions to improve fine-grained image understanding and preserve instruction-following.Multiple caption/question-answer formats and counterfactual trap questions are used to discourage default captioning and encourage visual grounding.Source: §3.1.1 · p. 7

3.1.2 OCR & VQA Data

  • Chinese OCR and VQA data combines MLLM-generated examples, rendered text and documents, and instruction-following samples to cover diverse recognition and question-answering tasks.Rendered samples vary in backgrounds, layouts, fonts, sizes, resolutions, and text types; instruction-following OCR includes 13 reading and locating template classes.Source: §3.1.2 · p. 8

3.1.3 Grounding & Counting Data

  • Grounding data uses integer-normalized center points, boxes, and polygons, while temporal grounding synthesizes timestamp question-answer pairs from filtered video events.The temporal pipeline segments short videos, filters repetitive or low-quality clips, then generates timestamp questions to support temporal-causality reasoning.Source: §3.1.3 · p. 8

3.1.4 Interleaved Text-Image Data

  • Interleaved image-text data supports long multimodal context, vision-language alignment, general knowledge retention, and cross-modal generalization.The in-house pipeline renders academic PDFs and structured STEM content with images in their original positions, filtering garbled text, broken images, and mismatched text-image pairs.Source: §3.1.4 · pp. 8–9

3.1.5 Video Data

  • Video data pipelines interleave speech transcripts with frames, generate captions at varied frame rates, add frame-level OCR, and train temporal-order and cross-video matching tasks.The reasoning tasks include restoring shuffled frames to chronological order and selecting a contextually relevant video from candidates.Source: §3.1.5 · pp. 9–10

3.2 Training Recipe

  • Training progresses from vision-encoder pre-training and cross-modal alignment to end-to-end multimodal pre-training and high-quality-data annealing.The stages adapt a SigLIP-initialized ViT, align it with a Qwen3-8B language model, unfreeze all parameters for multi-task training, and refine on curated data.Source: §3.2 · p. 10
  • Context length increases from 8K in the first two stages to 128K during annealing, with long-context modalities and parallelism changes supporting the extension.At 128K, experiments allocate 24% of tokens to video, 50% to images, and 26% to text.Source: §3.2 · p. 10

4 Post-Training … 4.3 LongCoT Cold-Start

Post-training combines broad SFT and preference optimization with reward modeling and a quality-controlled LongCoT cold start. The pipeline generates and refines reasoning data, then uses domain experts to address identified weaknesses before reinforcement learning.

4.1 Non-Reasoning Stage: SFT + MPO

  • The SFT pool exceeds 7.5 million multimodal QA samples, spanning 70,000 TaskGalaxy task types and emphasizing challenging reasoning, grounding, counting, GUI, and long-context data.MLLM-generated reasoning paths help filter overly simple samples; human annotators craft image and video captions for reliability.Source: §4.1 · p. 11
  • Annealing at a lower learning rate in later training phases contributes approximately 1% performance improvement across open-source and internal benchmarks, while MPO optimizes paired preferences.MPO pairs high- and low-quality samples using reward-model scores and human annotations; its dataset includes 250k open-source samples, 150k text-only samples, and 26k human-annotated samples (Wang et al. (2024b); Zhang et al. (2025b)).Source: §4.1 · p. 11

4.2 Keye-Reward Model

  • Keye-Reward adapts Keye-VL-preview for data filtering and RL, using think/no_think formats, nine evaluation dimensions, preference-data SFT, and filtered RL signals.SFT uses R1-Reward (Zhang et al. (2025c)), MMPR (Wang et al. (2024b)), and manually labeled samples; annealing addresses preference for longer responses. RL filters excessive length differences and uses format and outcome rewards.Source: §4.2 · pp. 11–12

4.3 LongCoT Cold-Start

  • LongCoT cold-start training supplies high-quality reasoning data to strengthen long-chain reasoning before subsequent reinforcement learning.The authors frame it as the starting point for the later RL phase.Source: §4.3 · p. 12

4.3.1 Data Construction Pipeline

  • The five-step LongCoT pipeline combines multimodal QA collection, question rewriting and task merging, and multi-path reasoning generation with step-wise and holistic confidence scores.It targets domains including math, STEM, OCR, grounding, counting, GUI, and business applications, selecting diverse but correct trajectories and favoring higher-confidence chains.Source: §4.3.1 · pp. 12–13
  • Automated assessment checks answer correctness and reasoning validity, then assigns samples to high-, moderate-, or low-quality tiers, discarding incorrect answers or severely flawed reasoning.Answer evaluation accommodates task-specific fuzzy matching and equivalent expressions; each reasoning step is checked for logic, factual accuracy, and relevance.Source: §4.3.1 · p. 13
  • Human review refines moderate-quality and potentially redundant samples, while five-point quality scores guide adaptive use, repeating scores 4–5 more often during training.The scoring scale ranges from simple or ambiguous questions to highly multimodal tasks requiring advanced reasoning.Source: §4.3.1 · pp. 13–14

4.3.2 Model Merging with Domain Specific Experts

  • After identifying weaknesses in text processing, mathematics, and OCR, the authors train domain-specific experts, including a specialized OCR model, and merge experts with the LongCoT cold-start model.OCR work targets license plates, street signage, and official seals, using verified annotations to generate questions and ground-truth answers.Source: §4.3.2 · p. 14

4.4 Iterative General RL

The section strengthens reasoning through GSPO-based RLVR and progressive hints for hard cases, then iteratively co-improves the RL and cold-start models.

4.4 Iterative General RL

  • Starting from the cold-start model, General RL applies GSPO (Zheng et al., 2025) to RLVR training in a cycle that enhances both models.The approach targets improved reasoning ability.Source: §4.4 · p. 14

4.4.1 General RLVR Training

  • The RLVR set spans math, science and technology, reasoning and puzzles, code, chart QA, visual grounding, spatial relations, and counting, with domain proportions adjusted based on ablation findings.Each training example has a verifiable answer for rule-based reward calculation.Source: §4.4.1 · p. 14

4.4.2 Progressive Hint Sampling

  • For repeatedly failing hard cases with reliable references, progressive hints move through five levels from concept to complete solution, and the minimal successful hint guides policy updates.The hierarchy proceeds through strategy, tools or formulas, and a concrete operational step before the complete solution.Source: §4.4.2 · p. 15

4.4.3 Iterative General RL & Cold-Start Enhancement

  • Each round replaces cold-start answers only when reward-model-scored RL samples beat ground truth, retrains the cold-start model, then seeds RL and filters data with sampling accuracy between 0 and 1.The resulting filtered data is used for the next General RL round.Source: §4.4.3 · p. 15

4.5 Alignment RL

Alignment RL extends General RL toward real-world use with task-specific training and a reward framework for instruction following, format adherence, and preference alignment. Its data construction spans constrained instructions, reasoning, and retrieval-augmented answering.

4.5 Alignment RL

  • Alignment RL targets real-world use through instruction following, format adherence, and preference alignment for open-ended responses.The intended gains include meeting requirements for content, format, length, and structure, while improving response reliability, interactivity, and style.Source: §4.5 · p. 15

4.5.1 Reward System Design

  • The reward system combines rule-based structural checks, generative evaluation against references and safety criteria, and model-based human-preference scoring for tasks without ground truth.Rule-based rewards check reasoning and output formats; generative rewards assess reference alignment, reasoning consistency, key attributes, and safety; the system aims to guide accurate, ethical, contextually appropriate outputs.Source: §4.5.1 · p. 16

4.5.2 Data Construction

  • Data construction covers constrained instruction following, multi-step math and logic, and news-based retrieval-augmented answering.Instruction data includes 17k multimodal and 23k text queries with 2–6 constraints, using rule-based rewards for hard constraints and generative rewards for soft ones; reasoning uses 12k queries designed for 3–5 solution steps, and RAG rewards search, summaries, and final-answer consistency with GSPO optimization.Source: §4.5.2 · p. 16

5 Training Infrastructure

Keye-VL-1.5 optimizes training infrastructure to address architectural heterogeneity, load imbalance, and I/O bottlenecks. Its design uses specialized parallelism, sample-aware GPU allocation, and topology-aware data loading and recovery.

Heterogeneous Hybrid Parallel Strategy

  • ViT uses data parallelism, while the parameter- and memory-intensive LLM combines pipeline, tensor, and data parallelism to enable 128K-sequence training.This heterogeneous hybrid strategy matches each component’s computational characteristics and resource demands.Source: §5 · p. 16

Dynamic Load Balancing Mechanism

  • A greedy allocator distributes samples by pre-estimated complexity across GPUs, balancing step duration and improving hardware utilization.The mechanism addresses multimodal load imbalance, such as the greater computational cost of high-resolution video compared with static images.Source: §5 · p. 16

Flexible and Scalable Dataloader

  • The dataloader shards data across DP processes, assigns acquisition and preprocessing to PP0, and fetches and broadcasts data from one process per TP/CP group.An I/O server offloads CPU-intensive tasks such as video decoding, while instance-level perfect resume restarts at the last successfully processed sample after interruption.Source: §5 · pp. 16–17

6 Evaluation

Keye-VL-1.5 performs strongly on public multimodal benchmarks, especially video tasks, while internal evaluations assess its capabilities and training strategies. The results also identify trade-offs in language generation and model merging.

6.1 Zero-shot Image Classification of ViT

  • On six zero-shot image classification datasets, the 2D RoPE native-resolution ViT is competitive with base SigLIP, while the 1D interpolation variant performs slightly worse.The report suggests ambiguous spatial localization from interpolated 1D positions as a possible reason for the latter result.Source: §6.1 · p. 18

6.2 SlowFast Video Encoding Strategy Discussion

  • Keye-VL-1.5-Base peaks on overall VideoMME at 384 frames, later than Qwen-2.5-VL at 128, and its performance decline begins later across video categories.Its category inflection points for short, medium, and long videos are 192, 512, and 384 frames, versus Qwen-2.5-VL’s 128, 384, and 128. It uses more LLM-side visual tokens at low frame counts and fewer at high counts, with more stable results across FPS settings.Source: §6.2 · p. 18

6.3 Public Benchmarks

  • Keye-VL-1.5 scores 79.5% on OpenCompass, 71.4% on MMMUval, and 86.7% on AI2D, surpassing all other models on these benchmarks.It also achieves the best performance on MMBench and MMStar and significantly outperforms Qwen2.5-VL 8B and InternVL3-8B on mathematical reasoning, with results comparable to MiMo-VL 7B-RL.Source: Table 3 · p. 19, §6.3 · p. 19
  • Keye-VL-1.5 improves on other open-source models in public video benchmarks, including a 6.5% absolute gain on Video-MMMU.The report identifies accurate video understanding as a core advantage.Source: §6.3 · p. 19

6.4 Internal Benchmarks

  • The cited passages describe public benchmarks as limited in specialized-domain and temporal coverage, using overly simple questions and restrictive answer formats, while risking contamination and English-language bias.Source: §6.4 · pp. 19–20

6.5 Evaluation Results

  • Keye-VL-1.5-8B scores 3.53 overall, improving by +0.51 over Keye-VL-Preview and leading MiMoVL-7B-RL-2508 by +0.13.It gains +0.57 in correctness over Preview, but MiMoVL-7B-RL-2508 performs better in fluency (+0.23), relevance (+0.08), and creativity (+0.15).Source: Table 4 · p. 20, §6.5 · p. 21
  • Keye-VL-1.5-8B scores 3.81 in Reasoning Ability, 3.36 in Temporal Information Understanding, and 4.29 in Robustness, a +0.83 lead over MiMoVL-7B-RL-2508.It matches MiMoVL-7B-RL-2508 in Visual Element Recognition (3.49) and Creative Ability (3.66), and improves over Preview by +0.35 in visual recognition, +1.00 in reasoning, and +0.77 in temporal understanding.Source: Table 5 · p. 20, §6.5 · p. 21

6.4 Internal Benchmarks

  • The internal benchmark scores answers on a 1–5 scale using three annotators and also compares model outputs through Good-Same-Bad preferences.GPT-4o or Gemini 1.5 Pro can serve as baseline models; annotators assess answers against video content and reference guidelines.Source: §6.4 · p. 20

7 Conclusion and Discussion

Keye-VL-1.5 combines Slow-Fast video encoding, extended-context pre-training, and reasoning- and preference-focused post-training. The report says it advances video understanding while maintaining strong performance on general vision-language tasks.

7 Conclusion and Discussion

  • Keye-VL-1.5 advances video understanding while maintaining strong performance on general vision-language tasks.The conclusion attributes this evaluation outcome to the model as a whole.Source: §7 · p. 24
  • The Slow-Fast video encoding strategy balances temporal coverage and spatial resolution.The conclusion describes this as an efficient balance.Source: §7 · p. 24
  • Progressive pre-training with extended context enables the model to handle longer videos and complex visual content.Source: §7 · p. 24
  • Post-training focused on reasoning and human preference alignment improves instruction-following and reasoning abilities.Source: §7 · p. 24

A Case Study · Video Case I: Grounding

In the grounding case, Keye-VL-1.5 accurately identifies the brief time range when a handbag appears in a 26-second video.

Video Case I: Grounding

  • Keye-VL-1.5 pinpoints the handbag’s roughly two-second appearance near the video’s end with 0.1-second precision.The 26-second video mainly introduces a power bank’s performance, advantages, and features.Source: Figure 9 · p. 30

Video Case II: Content Understanding

Keye-VL-1.5 links the video action to the caption’s context to explain the big dog’s behavior. In the example, it identifies the ear-biting as a way to teach the little dog not to eat food from the table.

Video Case II: Content Understanding

  • Keye-VL-1.5 correlates the video with the caption, identifying the big dog’s ear-biting and holding the little dog down as ways to teach it not to eat from the table.The caption says the dog dad holds the little dog down to prevent it from eating food on the dining table.Source: Figure 10 · p. 31

Video Case III: Content Description

In a video of hail falling in a forest, Keye-VL-1.5 captures much of the scene but does not identify the falling objects as hail.

Video Case III: Content Description

  • The model mentions precipitation and correctly identifies the rest of the content, but mistakes the hail for another falling object.The passage notes that even human viewers without prior knowledge could mistake the hail for petals or raindrops.Source: Figure 11 · p. 32

Related papers

Loading 2509.01563v3…