Source-linked AI summary
Kwai Keye-VL 1.5 Technical Report
Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, Fan Yang, Guorui Zhou, Guowang Zhang, Han Shen, Hao Peng, Haojie Ding, Hao Wang, Haonan Fan, Hengrui Ju, Jiaming Huang, Jiangxia Cao, Jiankang Chen, Jingyun Hua, Kaibing Chen, Kaiyu Jiang, Kaiyu Tang, Kun Gai, Muhao Wei, Qiang Wang, Ruitao Wang, Sen Na, Shengnan Zhang, Siyang Mao, Sui Huang, Tianke Zhang, Tingting Gao, Wei Chen, Wei Yuan, Xiangyu Wu, Xiao Hu, Xingyu Lu, Yi-Fan Zhang, Yiping Yang, Yulong Chen, Zeyi Lu, Zhenhua Wu, Zhixin Ling, Zhuoran Yang, Ziming Li, Di Xu, Haixuan Gao, Hang Li, Jing Wang, Lejian Ren, Qigen Hu, Qianqian Wang, Shiyao Wang, Xinchen Luo, Yan Li, Yuhang Hu, Zixing Zhang
TL;DR
Video understanding is difficult because models must balance temporal coverage with spatial resolution. Keye-VL-1.5 addresses this with adaptive Slow-Fast encoding, progressive long-context pre-training, and reasoning- and preference-focused post-training, and reports stronger performance particularly on video tasks while maintaining general multimodal performance.
Problem
Video understanding requires temporal relationships and sequential information while balancing temporal coverage against spatial resolution.
Method
Keye-VL-1.5 combines inter-frame-similarity-guided Slow-Fast encoding, progressive pre-training, and post-training with chain-of-thought data, GSPO reinforcement learning, and preference alignment.
Results
Evaluations on public benchmarks and internal human assessment report significant improvements over existing models, particularly for video understanding, while maintaining strong general vision-language performance.
Takeaways & Limitations
The paper presents an 8-billion-parameter multimodal model that advances video understanding while retaining competitive general multimodal capabilities.
Takeaways & Limitations
Internal evaluation identifies language and cultural bias in many test sets, including bias toward English-language scenarios.
Abstract
from arXiv · showhide
In recent years, the development of Large Language Models (LLMs) has significantly advanced, extending their capabilities to multimodal tasks through Multimodal Large Language Models (MLLMs). However, video understanding remains a challenging area due to the dynamic and information-dense nature of videos. Existing models struggle with the trade-off between spatial resolution and temporal coverage when processing video content. We present Keye-VL-1.5, which addresses fundamental challenges in video comprehension through three key innovations. First, we introduce a novel Slow-Fast video encoding strategy that dynamically allocates computational resources based on inter-frame similarity, processing key frames with significant visual changes at higher resolution (Slow pathway) while handling relatively static frames with increased temporal coverage at lower resolution (Fast pathway). Second, we implement a progressive four-stage pre-training methodology that systematically extends the model's context length from 8K to 128K tokens, enabling processing of longer videos and more complex visual content. Third, we develop a comprehensive post-training pipeline focusing on reasoning enhancement and human preference alignment, incorporating a 5-step chain-of-thought data construction process, iterative GSPO-based reinforcement learning with progressive prompt hinting for difficult cases, and alignment training. Through extensive evaluation on public benchmarks and rigorous internal human assessment, Keye-VL-1.5 demonstrates significant improvements over existing models, particularly excelling in video understanding tasks while maintaining competitive performance on general multimodal benchmarks.
1 Introduction
Keye-VL-1.5 targets video understanding’s spatial-resolution and temporal-coverage trade-off with adaptive encoding, long-context pre-training, and reasoning- and preference-focused post-training. Evaluations report improvements over existing models, especially for video understanding, while retaining general multimodal capabilities.
- Motivation: Video understanding requires temporal modeling while balancing fine-grained spatial detail against broad temporal coverage.Uniform frame sampling at fixed resolution can underperform when both visual detail and temporal consistency matter.
- Contributions: Slow-Fast encoding allocates higher resolution to visually changing key frames and lower resolution with greater temporal coverage to relatively static frames.The allocation is guided by inter-frame similarity and patch-based similarity functions.
- Contributions: Four-stage progressive pre-training extends context length from 8K to 128K tokens for longer videos and more complex visual content.The stages progressively build multimodal capabilities and extend context during annealing.
- Contributions: Post-training combines a 5-step chain-of-thought data pipeline, GSPO reinforcement learning with progressive hints, and human-preference alignment.The pipeline addresses reasoning enhancement and alignment with human preferences.
- Results: Public benchmarks and internal human assessment show significant improvements over existing models, particularly on video understanding tasks.The paper positions Keye-VL-1.5 as an 8-billion-parameter multimodal foundation model with robust general vision-language capabilities.
2 Model Architecture
Keye-VL-1.5 combines a Qwen3-8B language decoder, SigLIP-initialized vision encoding, native-resolution processing, and Slow-Fast video encoding. Its video pathway allocates resolution and frame coverage according to visual change while preserving temporal information.
- Architecture: The architecture contains a Vision Transformer, an MLP projector, and a Qwen3-8B language decoder with a SigLIP-initialized vision encoder.The projector maps and merges visual tokens for the language decoder.
- Native-Resolution Vision: Native-resolution vision processing preserves image structure and avoids redundant splicing or splitting operations.Position-embedding interpolation and 2D RoPE support resolution-adaptive visual modeling.
- Slow-Fast Video Encoding: Slow-Fast encoding uses fewer high-resolution frames for rapid changes and more low-resolution frames for relatively static content.A patch-similarity rule identifies frames, and fast frames receive 30% of a slow frame’s token budget.
- Slow-Fast Video Encoding: The model adds special tokens and absolute timestamps to guide video learning.These signals accompany the Slow-Fast representation shown in the video-encoding demonstration.
3 Pre-Training
Keye-VL-1.5 uses a diverse multimodal pre-training corpus and a four-stage progressive training strategy. Data construction spans image, text, OCR, grounding, interleaved, and video understanding tasks with filtering and synthetic-data procedures.
- Data Construction: The pre-training corpus exceeds 1 trillion tokens across six categories, including image captioning, OCR and VQA, grounding and counting, interleaved data, video understanding, and pure text.The pipeline uses customized filtering mechanisms for overall data quality.
- Image Caption Data: Image-caption quality is controlled with similarity filtering, retaining high-similarity pairs such as those with CLIP score > 0.9 and re-captioning lower-quality data.The approach combines open-source and in-house image data.
- Image Caption Data: Instruction diversity is supported by caption-question-answer formats that require switching between describing images and answering follow-up questions.The reversed ordering helps counter a default tendency toward caption generation.
- Image Caption Data: Counterfactual trap questions encourage responses grounded in visual content rather than textual priors.The stated purpose is to improve robustness and faithfulness.
- Grounding Data: Grounding data uses center points, bounding boxes, and polygons with integer coordinates normalized to [0, 1000), while temporal grounding follows a coarse-to-fine synthesis pipeline.Temporal clips are filtered to remove repetitive or meaningless descriptions.
- Interleaved Data: Interleaved text-image data supports long multimodal context modeling, preserves general knowledge, improves vision-language alignment, and strengthens cross-modal generalization.The paper also describes an in-house interleaved-data generation pipeline for academic PDFs and structured knowledge.
4 Post-Training
Keye-VL-1.5’s post-training combines supervised fine-tuning, preference optimization, and reasoning-oriented data and reinforcement-learning stages. The recipe emphasizes data quality, progressive refinement, and measurable gains from annealing.
- Supervised Fine-Tuning: The supervised fine-tuning candidate pool contains over 7.5 million multimodal question-answer samples.TaskGalaxy organizes data across 70,000 multimodal task types, including grounding, counting, GUI, and dialogue.
- Supervised Fine-Tuning: Data complexity is assessed through multiple reasoning paths, while the training mix increases mathematical, logical, complex-task, and long-context examples.Human annotators also craft captions for training images and videos.
- Training Strategy: Annealing at a lower learning rate contributes approximately a 1% performance improvement across open-source and internal benchmarks.The training strategy uses a dynamic learning rate with annealing in later phases.
- Preference Optimization: After SFT, MPO constructs high- and low-quality sample pairs using reward-model scores and human annotations from open-source, text-only, and human-annotated datasets.The listed datasets contain 250k open-source, 150k text-only, and 26k human-annotated samples.
4.2 Keye-Reward Model
The Keye-Reward Model supports data filtering and reinforcement-learning training by evaluating response quality through structured reasoning and preference signals.
- The reward model adapts Keye-VL-preview through SFT and RL for data filtering and reinforcement-learning training.
- Inputs include a query, two candidate responses, and task instructions for comparative quality evaluation.
- Think mode evaluates each response across nine dimensions before producing a comprehensive assessment.
- SFT combines R1-Reward, MMPR, and manually labeled Keye-VL-preview samples, then anneals on cases where good responses are shorter than bad responses.
- RL uses preference data from Keye-VL-preview and MMPR, filters extreme length differences, and applies format and outcome rewards.
- The reward model scores Keye-VL sampling results to update training data and provide reward signals.
4.3 LongCoT Cold-Start
The LongCoT cold-start process constructs diverse reasoning data through multimodal enhancement, confidence-aware generation, multi-level quality assessment, human refinement, and adaptive utilization.
- LongCoT data is constructed to strengthen long-chain reasoning and initialize subsequent reinforcement learning.
- Data Construction Pipeline: The five-step pipeline combines automated generation, quality assessment, targeted human enhancement, and adaptive data utilization.
- Data Collection and Enhancement: Multimodal QA data spans mathematics, STEM, OCR, visual grounding, counting, GUI scenarios, and business applications.
- Data Collection and Enhancement: Question rewriting and task merging transform simple inputs into more challenging, coherent multistep or multitask instructions.
- Reasoning Generation: Multiple reasoning trajectories are generated, scored for step-wise and overall confidence, and selected for diverse yet reliable reasoning paths.
- Quality Assessment: Answer correctness and reasoning validity are assessed separately, producing Category A, B, and C samples for retention, refinement, or removal.
- Human Enhancement: Human review refines Category B and potentially redundant Category A samples by improving coherence, conciseness, and reasoning quality.
- Data Utilization: Samples scoring 4 or 5 are repeated more often during training, while lower-scoring samples are used sparingly.
4.4 Iterative General RL
Iterative General RL applies GSPO with verifiable rewards, progressive hints for persistent hard cases, and repeated updates linking the RL and cold-start models.
- General RL uses GSPO for reinforcement learning with verifiable rewards and cyclically improves both the RL and cold-start models.
- Training Data: The RLVR dataset covers mathematics, science, logic, code, charts, grounding, spatial relations, and counting, with verifiable answers for rule-based rewards.
- Training Algorithm: GSPO optimizes at the sequence level using sequence-based importance weights and group-based advantage estimation.
- Progressive Hint Sampling: Progressive hint sampling addresses difficult examples on which the model repeatedly fails, using selected cases with reliable answers and suitable difficulty.
- Progressive Hint Sampling: Hints follow a five-level hierarchy from concepts and strategies to tools, concrete steps, and complete solutions.
- Progressive Hint Sampling: The lowest hint level that enables a correct response is treated as the minimal required intervention and used to update the policy.
- Iterative Training: Each iteration performs General RL, reward-scored rejection sampling, cold-start data replacement, and retraining before the next RL round.
4.5 Alignment RL
Alignment RL targets instruction following, format adherence, preference alignment, and real-world response quality through diversified tasks and multiple reward types.
- Alignment RL follows General RL to improve performance in real-world application scenarios through diversified tasks and reward modeling.
- Alignment Targets: The alignment targets include content, format, length, structured output, response style, reliability, and interactivity.
- Reward System: Rule-based, generative, and model-based rewards evaluate formatting, reference alignment, reasoning consistency, safety, ethics, and human preferences.
- Reward System: The reward system guides outputs toward accurate, ethical, and contextually appropriate responses.
- Task Construction: Instruction-following training covers 17k multimodal and 23k pure-text queries with hard and soft constraints evaluated by different rewards.
- Task Construction: Reasoning training uses 12k mathematical and logical queries with prescribed steps, combining rule-based outcome rewards and generative process rewards.
- Task Construction: RAG tasks evaluate search behavior, summary quality, and final-answer consistency using generative rewards.
5 Training Infrastructure
The training infrastructure addresses architectural heterogeneity, multimodal load imbalance, and I/O bottlenecks with specialized parallelism, balancing, and data-loading mechanisms.
- Architectural heterogeneity: Heterogeneous hybrid parallelism assigns data parallelism to ViT and combines pipeline, tensor, and data parallelism for the LLM.This strategy addresses differing computational and resource demands across the two components.
- Training scale: The infrastructure strategy is presented as a prerequisite for training Keye-VL-1.5 with 128K-token sequences.The stated connection is specifically made for the heterogeneous hybrid parallel strategy.
- Load imbalance: Dynamic load balancing targets unequal processing times caused by variation in multimodal input complexity.High-resolution videos can require more computation than static images, leaving other GPUs waiting.
- I/O bottlenecks: A topology-aware dataloader distributes dataset shards across data-parallel processes and centralizes acquisition at the first pipeline stage.The design also coordinates data fetching within tensor-parallel groups.
- I/O bottlenecks: An I/O server offloads video decoding from training nodes, while instance-level resume supports continuation after interruption.These mechanisms target CPU bottlenecks and interrupted-task recovery.
6 Evaluation
Keye-VL-1.5 shows strong performance across video, general vision-language, reasoning, OCR, and internal human evaluations. Results also reveal trade-offs involving language-generation sophistication, expert-model merging, and benchmark coverage.
- Video Understanding: Keye-VL-1.5’s performance begins declining later than Qwen-2.5-VL across short, medium, and long videos, indicating stronger multi-frame information integration.The inflection points are 192/512/384 for Keye-VL-1.5-Base versus 128/384/128 for Qwen-2.5-VL.
- Video Understanding: Keye-VL-1.5-Base uses visual tokens more flexibly than Qwen-2.5-VL, generating more at low frame counts and fewer at high frame counts.It also shows more stable evaluation results across different FPS settings.
- Public Benchmarks: Keye-VL-1.5 achieves competitive or leading general vision-language results, scoring 79.5% on OpenCompass, 71.4% on MMMUval, and 86.7% on AI2D.It also outperforms Qwen2.5-VL 8B and InternVL3-8B on mathematical reasoning while matching MiMo-VL 7B-RL.
- Public Benchmarks: Video-MMMU shows a 6.5% absolute improvement, making video understanding a core advantage over other open-source models.The paper reports superior capabilities in video-centric scenarios and significant gains on public video benchmarks.
- Internal Human Assessment: 3.53 overall composite score represents a +0.51 improvement over Keye-VL-Preview, with especially large gains in correctness and completeness.The reported improvements are +0.57 in correctness, +0.25 in completeness, and +0.11 in relevance.
- Expert Models and Model Merging: 84.51% average OCR performance results from merging the base model with a specialized OCR expert, exceeding the expert-only score of 83.65%.The merged model also surpasses MiMo-VL on TextVQA and ChartQA, while merge quality depends on training duration, learning rate, and parameter divergence.
7 Conclusion and Discussion
Keye-VL-1.5 enhances video understanding and vision-language performance through Slow-Fast encoding, progressive long-context pre-training, and post-training for reasoning and human preference alignment.
- Keye-VL-1.5 advances video understanding while maintaining strong performance on general vision-language tasks.
A Case Study
The case studies illustrate Keye-VL-1.5’s ability to localize brief visual events and connect video content with captions, while also revealing a hail-identification failure.
- 0.1 seconds: Keye-VL-1.5 accurately pinpoints the handbag’s brief appearance near the video’s end.The handbag appears for about two seconds in the final portion of the 26-second video.
- Keye-VL-1.5 connects the caption and video to infer that ear-biting and holding-down both discourage the puppy from eating food.
- Keye-VL-1.5 fails to identify falling hail, although it recognizes precipitation and correctly identifies the remaining content.
B Authors (Alphabetical order)
The paper lists its core and additional contributors in alphabetical order.
- The core contributor list includes the researchers responsible for the paper’s main contribution.
- The additional contributor list names further contributors separately from the core contributors.