PaperlayerView exact revision on arXiv

Source-linked AI summary

Kimi k1.5: Scaling Reinforcement Learning with LLMs

Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Weixin Xu, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, Zonghan Yang, Zongyu Lin

arXiv:2501.12599v4cs.AIcs.LG

TL;DR

Pretraining-based scaling is limited by available high-quality data, motivating reinforcement learning as another route for improving LLMs. The paper presents Kimi k1.5’s long-context RL, policy optimization, multimodal training, and long2short methods, reporting state-of-the-art reasoning results across benchmarks and modalities while matching OpenAI’s o1. It also identifies infrastructure efficiency and future credit-assignment improvements as continuing challenges.

  • Problem

    Next-token pretraining is limited by the amount of available high-quality training data, motivating reinforcement learning as a possible new scaling axis for LLM improvement.

  • Method

    Kimi k1.5 combines long-context RL, improved policy optimization, multimodal training, partial rollouts, and long2short methods without using Monte Carlo tree search, value functions, or process reward models.

  • Results

    Kimi k1.5 achieves state-of-the-art reasoning performance across multiple benchmarks and modalities, including 77.5 on AIME, 96.2 on MATH 500, and 74.9 on MathVista, matching OpenAI’s o1.

  • Takeaways & Limitations

    Long-context scaling and improved policy optimization can produce strong RL performance without more complex techniques, while long2short methods improve short-CoT models and token efficiency.

  • Takeaways & Limitations

    Further improvement in the efficiency and scalability of long-context RL training remains an important open question because the training and inference infrastructure involves complex parallelism across systems.

Abstract

from arXiv · show

Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).

1 Introduction

Kimi k1.5 explores reinforcement learning as a new scaling axis for LLM improvement, centered on long-context scaling and improved policy optimization. The approach also uses long-CoT techniques to improve short-CoT models.

  • RL enables LLMs to explore with rewards rather than relying only on a fixed supply of pre-existing training data.
  • Long-context scaling to 128k and partial rollouts improve training efficiency by reusing large chunks of previous trajectories.
  • Improved policy optimization combines long-CoT RL with online mirror descent, sampling strategies, length penalties, and optimized data recipes.
  • The resulting framework is simplistic and avoids Monte Carlo tree search, value functions, and process reward models.
  • Long2short methods use long-CoT techniques to improve short-CoT models, including length penalties with long-CoT activations and model merging.
  • 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, and 74.9 on MathVista are reported for the long-CoT version across benchmarks and modalities.

2 Approach: Reinforcement Learning with LLMs

Kimi k1.5 combines curated multimodal RL data, long-context and long-CoT training, simplified policy optimization, and infrastructure changes to improve reasoning and efficiency.

  • RL data curation: The RL program uses diverse, balanced, objectively evaluable prompts spanning STEM, coding, general reasoning, and image-text tasks.Difficulty is estimated from an SFT model’s repeated-sampling pass rate, enabling filtering and curriculum design.
  • Long-CoT RL: Long-CoT RL trains autoregressive reasoning sequences that can include planning skills such as error identification, backtracking, and solution refinement.The approach seeks exploratory reasoning paths that improve problem-solving beyond training-set accuracy.
  • Policy and reward design: The system omits a value network to improve training efficiency and uses Chain-of-Thought reward modeling, which reached 98.5 accuracy versus 84.4 for Classic RM in spot checks.The reported reward-model comparison motivates using more correct feedback during RL.
  • Efficiency and long2short: Length rewards address overthinking by favoring shorter correct responses and penalizing long incorrect responses, while long2short methods transfer long-CoT benefits to short-CoT models.The goal is improved token efficiency under limited test-time budgets.
  • Long-context scaling: Partial rollouts scale long-context RL by capping each trajectory’s output budget, storing unfinished segments, and reusing prior segments across iterations.This prevents long trajectories from monopolizing resources while asynchronous workers continue processing shorter tasks.
  • RL infrastructure: Hybrid deployment colocates training and inference through Kubernetes Sidecar containers sharing GPUs, allowing independent image iteration and rapid phase switching.The Megatron-vLLM implementation takes less than one minute from training to inference and about ten seconds in the reverse direction.

3 Experiments

Experiments evaluate Kimi k1.5 across text, reasoning, and vision benchmarks, then examine scaling behavior, long2short efficiency, and training-strategy ablations. Results show strong multimodal reasoning, continued gains from longer context, improved token efficiency, and benefits from the proposed optimization and curriculum methods.

  • 3.1 Evaluation: Kimi k1.5 is evaluated across text, reasoning, and vision benchmarks spanning language understanding, coding, mathematics, and visual reasoning.The evaluation includes MMLU, IF-Eval, CLUEWSC, C-EVAL, HumanEval-Mul, LiveCodeBench, Codeforces, AIME 2024, MATH500, MMMU, MATH-Vision, and MathVista.
  • 3.2 Main Results: The long-CoT model achieves state-of-the-art results across multiple modalities after long-CoT supervised fine-tuning, vision-text joint RL, and test-time computation scaling.The passage reports enhanced long-term reasoning and performance across a range of modalities.
  • 3.2 Main Results: The short-CoT model delivers competitive or superior performance against leading open-source and proprietary models across text, vision, and reasoning tasks.Reported strengths include natural language understanding, mathematics, coding, and logical reasoning.
  • 3.3 Long Context Scaling: 128k context scaling produces continued improvement on hard reasoning benchmarks, while training accuracy and response length increase together during RL.More challenging benchmarks show steeper response-length growth, indicating more elaborate solutions for complex problems.
  • 3.4 Long2short: The proposed long2short RL algorithm achieves the highest token efficiency among compared methods, and all k1.5 variants outperform other models on this measure.Comparisons include DPO, shortest rejection sampling, and model merging; k1.5-short w/ rl reaches a reported Pass@1 score of 60.
  • 3.5 Ablation Studies: Longer CoTs allow the smaller model to reach comparable performance to the larger model, although the larger model generally remains more token-efficient.The larger model has a higher performance upper bound when context length is scaled, while smaller models may be viable under test-time compute budgets.
  • 3.5 Ablation Studies: Negative gradients improve sample complexity relative to ReST, while curriculum sampling significantly improves performance over uniform sampling.The curriculum first uses diverse problems for warm-up, then focuses on hard questions.

4 Conclusions

Kimi k1.5’s recipe combines long-context scaling, improved policy optimization, and long2short methods to obtain strong reasoning performance with a relatively simple RL framework. The authors identify efficiency and scalability of long-context RL as an open question.

  • 4 Conclusions: Long-context scaling and improved policy optimization support strong performance without Monte Carlo tree search, value functions, or process reward models.The approach uses infrastructure optimizations such as partial rollouts for efficient long-context RL training.
  • 4 Conclusions: Further improving the efficiency and scalability of long-context RL remains an important open question.
  • 4 Conclusions: The policy-optimization contribution combines long-CoT RL, a variant of online mirror descent, sampling strategies, length penalty, and data-recipe optimization.
  • 4 Conclusions: Long2short methods largely improve short-CoT models and can be combined iteratively with long-CoT RL to improve token efficiency within a fixed context-length budget.

B Pretraining

The pretraining discussion emphasizes that RL efficiency depends closely on the underlying base model, while limited transparency around open-source data pipelines complicates community understanding.

  • B Pretraining: RL efficiency is closely tied to the performance of the underlying base model.
  • B Pretraining: Many recent open-source models lack full transparency regarding their data processing pipelines and recipes.

B.1 Language Data

The language-data recipe spans English, Chinese, code, mathematics and reasoning, and knowledge, using layered filtering, scoring, deduplication, and empirical sampling to emphasize quality while preserving diversity.

  • B.1 Language Data: The pretraining corpus covers five domains: English, Chinese, Code, Mathematics & Reasoning, and Knowledge.
  • B.1 Language Data: English and Chinese data are filtered with rule-based heuristics, FastText classification, embedding similarity, and LLM-based quality assessment.
  • B.1 Language Data: Documents receive combined quality scores that determine dynamic sampling rates, with high-quality documents upsampled and low-quality documents downsampled.
  • B.1 Language Data: Code data is preprocessed with cleaning and strategic sampling to address language imbalance.
  • B.1 Language Data: Mathematical data undergoes specialized cleaning, including FastText-based initial filtering and language-model cleaning to produce high-quality data.
  • B.1 Language Data: Knowledge documents are annotated for OCR quality, educational value, and document type before filtering for OCR artifacts and prioritizing pedagogically relevant material.
  • B.1 Language Data: Sampling is empirically determined to upsample valuable document subsets while maintaining data diversity and model generalization.

B.2 Multimodal Data

The multimodal pretraining recipe combines captioning, interleaving, OCR, knowledge, and question-answering data with filtering, synthesis, deduplication, and modality-specific processing.

  • B.2 Multimodal Data: The multimodal corpus contains captioning, interleaving, OCR, knowledge, and general question-answering data.
  • B.2 Multimodal Data: Multimodal processing pipelines use filtering, synthesis, and deduplication to preserve language capabilities and align knowledge across modalities.
  • B.2 Multimodal Data: Image-text interleaving data supports multi-image comprehension, detailed image knowledge, longer multimodal context learning, and language-ability preservation.
  • B.2 Multimodal Data: Interleaving data is additionally reordered so images and text remain in the correct order after standard quality control.
  • B.2 Multimodal Data: OCR data covers multilingual text, dense layouts, web content, handwriting, figures, tables, geometry diagrams, mermaid plots, and natural-scene text.
  • B.2 Multimodal Data: The knowledge corpus uses a standardized taxonomy and multimodal processing to support diverse knowledge coverage and visual reasoning, including geometry data.
  • B.2 Multimodal Data: An additional pipeline captures purely textual information embedded in images because basic OCR alone may limit training effectiveness for internet-sourced knowledge.
  • B.2 Multimodal Data: General QA data includes grounding, table/chart question answering, web agents, general QA, and in-house datasets balanced for difficulty and diversity.

B.3 Model Architecture

Kimi k-series models use a multimodal Transformer decoder with architectural and optimization improvements for stable large-scale RL training and efficient inference. Scaling experiments emphasize pretraining-data quality and diversity as the main source of base-model performance.

  • B.3 Model Architecture: Kimi k-series models use a Transformer decoder integrating multimodal capabilities with architectural and optimization improvements.These changes support stable large-scale training and efficient inference for reinforcement learning.
  • B.3 Model Architecture: Most base-model performance comes from improving pretraining-data quality and diversity.Architecture scaling experiments are outside this report’s scope and are deferred to future publications.

B.4 Training Stages

Kimi k1.5 is trained through three stages that progressively establish language, multimodal, cooldown, and long-context capabilities. The model supports interleaved image-text input and uses large-scale RL to enhance reasoning.

  • B.4 Training Stages: Kimi k1.5 training comprises vision-language pretraining, vision-language cooldown, and long-context activation stages.Each stage targets a particular capability enhancement.
  • B.4 Training Stages: Kimi k1.5 accepts interleaved images and text as input while using large-scale reinforcement learning to enhance reasoning.
  • B.4 Training Stages: The pretraining stage begins with language-only training, then introduces interleaved vision-language data and increases vision-text data to 30%.The visual tower is first trained separately before language-model layers are unfrozen.
  • B.4 Training Stages: The cooldown stage uses high-quality language and vision-language datasets, with synthetic data improving mathematical reasoning, knowledge tasks, and code generation.
  • B.4 Training Stages: The long-context activation stage upsamples long-context cooldown data and uses 40% full attention with 60% partial attention.This stage enables extended-sequence processing and tasks requiring longer context.

C.1 Text Benchmark

The text benchmark suite evaluates broad knowledge, instruction following, Chinese coreference, and Chinese knowledge and reasoning across several domains and difficulty levels.

  • C.1 Text Benchmark: MMLU covers 57 STEM, humanities, and social-science subjects from elementary to advanced professional difficulty.It tests world knowledge and problem-solving ability.
  • C.1 Text Benchmark: IF-Eval evaluates verifiable instruction following with more than 500 prompts.Reported Table 3 results come from an intermediate model because of a version shift.
  • C.1 Text Benchmark: CLUEWSC tests Chinese pronoun–noun-phrase coreference using data from Chinese fiction books.
  • C.1 Text Benchmark: C-EVAL assesses advanced Chinese knowledge and reasoning through 13,948 multiple-choice questions across 52 disciplines and four difficulty levels.

C.2 Reasoning Benchmark

The reasoning benchmark suite spans multilingual code generation, contamination-controlled live coding, advanced mathematics, and long-CoT programming performance. Codeforces evaluation additionally uses majority voting and model-generated test cases.

  • C.2 Reasoning Benchmark: HumanEval-Mul evaluates translations of HumanEval in eight mainstream programming languages.The selected languages are Python, Java, Cpp, C#, JavaScript, TypeScript, PHP, and Bash.
  • C.2 Reasoning Benchmark: LiveCodeBench provides contamination-free coding evaluation with live updates, varied scenarios, high-quality tests, and balanced difficulty.Short-CoT uses release v4 questions from 2408-2411, while long-CoT uses release v5 questions from 2412-2502.
  • C.2 Reasoning Benchmark: AIME 2024 tests advanced mathematics and logical thinking through competition questions for top high school students.
  • C.2 Reasoning Benchmark: MATH-500 contains 500 problems spanning algebra, calculus, probability, and other topics, testing computation and mathematical reasoning.Higher scores indicate stronger mathematical problem-solving capabilities.
  • C.2 Reasoning Benchmark: Codeforces evaluation uses majority voting over k1.5 long-CoT code snippets and test cases generated by the same model.The reported Codeforces ELO percentile was extracted from an OpenAI Day12 talk.

C.3 Image Benchmark

This section describes multimodal benchmarks that test academic knowledge, mathematical reasoning with visual contexts, and compositional visual-mathematical understanding.

  • 11.5K multimodal questions span six academic fields, including Art & Design, Business, Science, and Tech & Engineering.
  • MATH-Vision contains 3,040 visual-context mathematics problems from real competitions across 16 disciplines and five difficulty levels.
  • MathVista combines mathematical and visual tasks requiring fine-grained visual understanding and compositional reasoning.
Loading 2501.12599v4…