PaperlayerView exact revision on arXiv

Source-linked AI summary

Kimi K2: Open Agentic Intelligence

Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Chenxiao Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Yuyao Ge, Shangyi Geng, Qizheng Gu, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Yunjia He, Chao Hong, Hao Hu, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang, Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yang Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Haoyu Lu, Lijun Lu, Yashuo Luo, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Zeyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Lin Sui, Xinjie Sun, Flood Sung, Yunpeng Tai, Heyi Tang, Jiawen Tao, Qifeng Teng, Chaoran Tian, Chensi Wang, Dinglu Wang, Feng Wang, Hailong Wang, Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang, Si Wang, Xinyuan Wang, Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang, Zhengtao Wang, Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Haoning Wu, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Jin Xie, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Jing Xu, Jing Xu, Junjie Yan, Yuzi Yan, Hao Yang, Xiaofei Yang, Yi Yang, Ying Yang, Zhen Yang, Zhilin Yang, Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Siyu Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yadong Zhang, Yangkun Zhang, Yichi Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Zijia Zhao, Huabin Zheng, Shaojie Zheng, Longguang Zhong, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Jinguo Zhu, Zhen Zhu, Weiyu Zhuang, Xinxing Zu

arXiv:2507.20534v2cs.LGcs.AIcs.CL

TL;DR

Kimi K2 addresses the challenge of building efficient, stable models with scalable agentic capabilities. It combines MuonClip-based pre-training with synthetic tool-use data and unified reinforcement learning, achieving strong results across agentic, coding, reasoning, and mathematics benchmarks. The released checkpoints support further research and deployment of agentic intelligence.

  • Problem

    Agentic capabilities such as multi-step reasoning, long-term planning, and tool use are rare in natural data and costly to scale, while pre-training faces limited high-quality data.

  • Method

    Kimi K2 uses a 1.04 trillion-parameter MoE architecture, MuonClip for token-efficient stable pre-training, large-scale synthetic tool-use data, and reinforcement learning with verifiable and self-critique rewards.

  • Results

    Kimi K2 achieves state-of-the-art performance among open-weight models across agentic and reasoning benchmarks, including 65.8 on SWE-bench Verified and 66.1 on Tau2-bench.

  • Takeaways & Limitations

    Kimi K2 establishes a highly capable open-weight model for agentic and reasoning tasks, with particular strength in software engineering and tool-use settings.

  • Takeaways & Limitations

    On hard reasoning tasks or unclear tool definitions, Kimi K2 may generate excessive tokens, truncated outputs, or incomplete tool calls; one-shot software-project prompting underperforms agentic coding frameworks.

Abstract

from arXiv · show

We introduce Kimi K2, a Mixture-of-Experts (MoE) large language model with 32 billion activated parameters and 1 trillion total parameters. We propose the MuonClip optimizer, which improves upon Muon with a novel QK-clip technique to address training instability while enjoying the advanced token efficiency of Muon. Based on MuonClip, K2 was pre-trained on 15.5 trillion tokens with zero loss spike. During post-training, K2 undergoes a multi-stage post-training process, highlighted by a large-scale agentic data synthesis pipeline and a joint reinforcement learning (RL) stage, where the model improves its capabilities through interactions with real and synthetic environments. Kimi K2 achieves state-of-the-art performance among open-source non-thinking models, with strengths in agentic capabilities. Notably, K2 obtains 66.1 on Tau2-Bench, 76.5 on ACEBench (En), 65.8 on SWE-Bench Verified, and 47.3 on SWE-Bench Multilingual -- surpassing most open and closed-sourced baselines in non-thinking settings. It also exhibits strong capabilities in coding, mathematics, and reasoning tasks, with a score of 53.7 on LiveCodeBench v6, 49.5 on AIME 2025, 75.1 on GPQA-Diamond, and 27.1 on OJBench, all without extended thinking. These results position Kimi K2 as one of the most capable open-source large language models to date, particularly in software engineering and agentic tasks. We release our base and post-trained model checkpoints to facilitate future research and applications of agentic intelligence.

1 Introduction

Kimi K2 targets agentic intelligence by combining token-efficient, stable pre-training with scalable agentic data synthesis and reinforcement learning. It reports strong performance across agentic, coding, mathematics, and reasoning benchmarks, while releasing checkpoints for further research and deployment.

  • Motivation: Agentic intelligence requires models to perceive, plan, reason, and act through interactions with complex environments rather than relying only on static imitation learning.The paper identifies multi-step reasoning, long-term planning, and tool use as capabilities that are rare in natural data and costly to scale.
  • Model: Kimi K2 is a 1.04 trillion-parameter Mixture-of-Experts model with 32 billion activated parameters designed to address agentic capability challenges.
  • Pre-training: MuonClip combines token-efficient Muon with QK-Clip, enabling pre-training on 15.5 trillion tokens without a single loss spike.This addresses the paper’s joint concerns about token efficiency and training stability.
  • Post-training: A large-scale synthesis pipeline generates high-fidelity, verifiably correct tool-use interactions across diverse tools, agents, tasks, and trajectories.The pipeline uses simulated and real-world environments to produce structured agentic demonstrations at scale.
  • Post-training: The reinforcement learning framework combines verifiable rewards with self-critique rubric rewards, allowing learning from external tasks and evaluation of the model’s own outputs.This extends alignment from static settings toward open-ended domains.
  • Results: 66.1 on Tau2-bench, 76.5 on ACEBench (en), 65.8 on SWE-bench Verified, and 47.3 on SWE-bench Multilingual demonstrate strong non-thinking benchmark performance.The paper also reports 53.7 on LiveCodeBench v6, 27.1 on OJBench, 49.5 on AIME 2025, and 75.1 on GPQA-Diamond.
  • Impact: Open-sourcing base and post-trained checkpoints enables the community to explore, refine, and deploy agentic intelligence at scale.

2 Pre-training

Kimi K2’s pre-training combines token-efficient optimization, QK-Clip stability control, synthetic rephrasing, and sparse MoE architecture design. These choices target efficient use of limited high-quality tokens while preventing attention-logit instability during large-scale training.

  • MuonClip: Stable Training with Weight Clipping: MuonClip integrates the token-efficient Muon optimizer with QK-Clip to address exploding attention logits during scaled training.QK-Clip rescales query and key projection weights after updates when the per-head maximum logit exceeds threshold τ, without altering current-step forward/backward computation.
  • Pre-training Data: Improving Token Utility with Rephrasing: Token-efficient pre-training uses synthetic rephrasing to increase the utility of high-quality tokens without significant overfitting.The data strategy includes domain-specialized rephrasing for knowledge and mathematics, addressing the trade-off between insufficient single-epoch exposure and repetitive multi-epoch training.
  • Model Architecture: Kimi K2 uses a 1.04 trillion-parameter sparse MoE architecture with 32 billion activated parameters, informed by scaling-law analyses.The model uses 384 experts, activates 8 per forward pass, and employs Multi-head Latent Attention; increasing sparsity at fixed activated parameters consistently lowered training and validation loss.

3 Post-Training

Kimi K2’s post-training combines large-scale synthetic and real-environment tool-use data with instruction tuning and reinforcement learning to develop agentic behavior. The pipeline emphasizes diverse, verifiable trajectories and extends alignment to both objective and subjective tasks.

  • Agentic Data Synthesis: Kimi K2’s tool-use synthesis pipeline generates tools, agents, tasks, and multi-turn trajectories from real and synthetic environments.It combines tool repositories, agent diversification, rubric-based tasks, user simulation, and trajectory generation.
  • Agentic Data Synthesis: Only trajectories meeting task-specific success criteria are retained, preserving high-quality data while allowing varied completion strategies.An LLM-based judge evaluates trajectories against explicit task rubrics.
  • Agentic Data Synthesis: Real execution sandboxes complement scalable simulation for coding and software-engineering scenarios requiring authentic feedback.These sandboxes execute actual code and provide objective signals such as test-suite pass rates.
  • Reinforcement Learning: Reinforcement learning is scaled across diverse tasks through an extensible Gym-like framework with verifiable rewards.The post-training program also includes math, STEM, logical tasks, and self-critic feedback for subjective objectives.
  • Reinforcement Learning: Self-critic feedback extends alignment toward helpfulness, creativity, reasoning depth, factuality, and safety beyond tasks with verifiable rewards.The paper reports improvements in user-intent understanding, creative writing, complex reasoning, and nuanced language comprehension.

4 Evaluations

Kimi-K2-Instruct is evaluated across agentic, coding, general capability, safety, and open-ended benchmarks under non-thinking settings. It shows strong performance across these domains, while safety evaluation includes subjectivity and limited relevance for some base-model tool-use tests.

  • Agentic and Competitive Coding: 65.8% on SWE-bench Verified, 47.3% on SWE-bench Multilingual, and 39.1% on SWE-Lancer show strong real-world software-engineering performance.Kimi-K2-Instruct also scores 53.7% on LiveCodeBench v6 and 27.1% on OJBench.
  • Agentic Tool Use: 66.1 Pass@1 on τ2-Bench and 76.5 on ACEBench demonstrate strong multi-turn tool-use performance.These benchmarks assess grounded, controlled, agent-driven tool orchestration across domains.
  • General Capabilities: 89.5% on MMLU, 92.7% on MMLU-Redux, and 89.8% on IFEval indicate strong general knowledge and instruction-following capabilities.The model also reports 75.1% on GPQA-Diamond, 93.5% on DROP, and 55.0% on MRCR.
  • Open-Ended Evaluation: Kimi-K2-Instruct ranks first among open-source models and fifth overall on the LMSYS Arena leaderboard based on over 3,000 user votes.The leaderboard result reflects blind, open-ended user preferences as of July 17, 2025.
  • Base Model Evaluation: Kimi-K2-Base achieves state-of-the-art results on 10 of 12 English language benchmarks and leads on three of four mathematical benchmarks.It also achieves state-of-the-art results across all listed Chinese language benchmarks.
  • Safety Evaluation: Safety results vary by attack strategy: Base64 transformations often approach 100% passing rates, whereas Crescendo generally lowers them.Complex attacks do not always outperform basic prompts because repeated transformations can make adversarial meaning less meaningful.

5 Limitations

Current Kimi K2 models have limitations in hard reasoning, tool use, and one-shot software-project completion. Excessive generation can cause truncated outputs or incomplete tool calls.

  • Model Limitations: Hard reasoning tasks and unclear tool definitions may trigger excessive tokens, truncated outputs, or incomplete tool calls.Unnecessarily enabling tool use can also reduce performance on certain tasks.
  • Model Limitations: One-shot prompting is less successful than using K2 within an agentic coding framework for complete software projects.The authors identify this as an internal-test limitation and plan future improvements.

6 Conclusions

The paper presents Kimi K2 as an open-weight MoE model for agentic intelligence, combining efficient stable pre-training with synthetic tool-use data and unified reinforcement learning. It reports state-of-the-art results across agentic and reasoning benchmarks.

  • Conclusion: Kimi K2 is a 1T-parameter open-weight MoE model built for agentic intelligence.The model uses 32 billion activated parameters, according to the paper context.
  • Conclusion: MuonClip and a 15.5T-token high-quality dataset provide stable, scalable pre-training.Post-training combines synthetic tool-use data with verifiable rewards and self-critic feedback.
  • Conclusion: Kimi K2 achieves state-of-the-art performance on agentic and reasoning benchmarks, which the authors characterize as making it the most capable open-weight LLM to date.

A Contributions

The appendix documents the tool-calling interface used by Kimi K2, including declarations, invocation messages, results, schemas, parallel calls, and constrained decoding.

  • Tool-Calling Interface: Tool calling uses three message components: a declaration, an assistant invocation section, and a tool result message.The declaration lists available tools and argument schemas; results carry the call id and execution output.
  • Tool-Calling Interface: Tool declarations can be expressed in TypeScript, which concisely represents parameter types and constraints alongside JSON-compatible definitions.The paper illustrates simple weather and calculator tools in both formats.
  • Tool-Calling Interface: Parallel tool calling places multiple tool calls in one response turn, with unique ids formatted as functions.{tool-name}:{counter}.The counter increases automatically from 0 across tool calls in the dialogue.
  • Tool-Calling Interface: The enforcer constrained-decoding module ensures tool-call tokens follow the predefined template and declared JSON schema.It activates when the tool-call section begins to reduce parsing format errors.

C Evaluation Details

Kimi-K2-Instruct is evaluated across coding, software engineering, tool use, mathematics, STEM, factuality, long-context, and open-ended benchmarks. It shows strong results across these settings, including high performance and user preference on English and Chinese tasks.

  • Coding and Software Engineering: 53.7% on LiveCodeBench and 27.1% on OJBench demonstrate strong competitive-coding performance.The evaluation covers medium- and hard-level coding challenges, as well as multilingual programming tasks.
  • Coding and Software Engineering: 71.6% on SWE-bench Verified, 47.3% on SWE-bench Multilingual, and 39.1% on SWE-lancer approach Claude 4 Sonnet’s results.The corresponding Claude 4 Sonnet scores are 80.2%, 51.0%, and 40.8%.
  • Agentic Evaluation: Kimi-K2-Instruct achieves strong tool-use performance on τ2-Bench and ACEBench, which evaluate multi-turn interaction and API-grounded bilingual tasks.τ2-Bench uses dual control over shared state, while ACEBench evaluates English and Chinese API use.
  • Math, STEM, and General Tasks: 75.1% on GPQA-Diamond and 69.6% on AIME 2024 show strong STEM and mathematical performance.The model’s average mathematics performance exceeds several listed baselines.
  • Factuality and Long Context: 88.5 on FACTS Grounding and 93.5% on DROP indicate strong factuality and long-context performance, while FRAMES and LongBench v2 remain slightly behind DeepSeek-V3-0324.On MRCR, Kimi-K2-Instruct scores 55.0% versus 50.8% for DeepSeek-V3-0324.
  • Open-Ended Evaluation: 54.5% on hard Arena-Hard prompts and 85.0% on creative writing establish state-of-the-art win rates among open-source models.The benchmark uses LLM-as-a-judge protocols for diverse open-ended English prompts.
  • Open-Ended Evaluation: Kimi-K2-Instruct wins against ChatGPT-4o-latest, Claude Sonnet 4, and DeepSeek-V3-0324 on the Chinese in-house benchmark by 65.4%, 64.6%, and 59.6%.Loss rates remain around 17% in each comparison.
  • Open-Ended Evaluation: Over 3,000 blind human votes placed Kimi-K2-Instruct first among open-source models and fifth overall on the LMSYS Arena leaderboard.The leaderboard provides a complementary perspective to LLM-as-a-judge evaluations.

D QK-Clip Does Not Impair Model Quality

QK-Clip constrains attention logits with minimal impact on model quality. It is transiently active during unstable training and deactivates after the model stabilizes.

  • Design: QK-Clip follows a minimal-intervention design that activates only when necessary and deactivates after training stabilizes.The authors report negligible impact on model quality.
  • Small-Scale Ablations: Negligible loss-curve changes at τ = 30 show that aggressive QK-Clip clipping does not impair convergence or training dynamics.The ablation compares small MoE models trained with vanilla Muon and MuonClip.
  • Self-Deactivation: After 70,000 steps, every attention head reduced Smax below 100 at some point, rendering QK-Clip inactive.Once training stabilizes, QK-Clip has no further effect.

E Why Muon is More Prone to Logit Explosion

The paper attributes Muon’s greater susceptibility to logit explosion to how its updates distribute singular values and amplify attention-weight spectral norms. This mechanism increases the risk of excessively large attention logits during training.

  • Attention-Specific Amplification: Logit explosion occurs when the largest pre-softmax attention score grows unboundedly during training.With RMS-Norm bounding input-vector norms, the growth is primarily attributed to the spectral norm of Wq or Wk.
  • Structural Difference in Updates: Muon updates have equal singular values and full effective rank, whereas Adam updates typically have a skewed spectrum dominated by a few large singular values.This structural difference is the proposed explanation for Muon’s greater susceptibility.
  • Empirical Verification: Weights trained with Muon show higher singular-value entropy and effective rank than weights trained with Adam in the 16B Moonlight model.The observation corroborates the theoretical intuition about update structure.
  • Attention-Specific Amplification: Attention logits use a bilinear form whose spectral-norm dependence compounds singular-value increases in Wq and Wk.Muon’s tendency to enlarge singular values therefore translates into higher logit-explosion risk.

F.3 Limitations

The evaluation framework may favor confident, decisive responses over appropriately cautious or multi-perspective answers in ambiguous and open-ended settings.

  • Evaluation Framework: The rubric prohibits self-qualification and favors direct, singular answers, which may reduce appropriately cautious responses in complex or open-ended scenarios.These constraints can favor confident and assertive response styles even when ambiguity or subjectivity is present.
  • Evaluation Framework: The model may occasionally overstate certainty where ambiguity, nuance, or epistemic modesty would be more appropriate.Future framework iterations may incorporate finer-grained handling of calibrated uncertainty.

G Engine Switching Pipeline for RL Training

The RL weight-update design organizes parameter movement through staged H2D transfer, broadcast, and reload operations. Because concurrent H2D and broadcast saturate PCIe on NVIDIA H800 clusters, the implementation adopts a two-stage pipeline while relying on sharding to reduce transfer overhead at scale.

  • Pipeline design: The checkpoint engine uses one H2D buffer and two IPC buffers per GPU to support staged parameter movement.The H2D buffer loads offloaded parameters, while IPC buffers support GPU-to-GPU broadcast and shared access by inference engines.
  • Pipeline design: The theoretical three-stage pipeline performs asynchronous H2D copying, broadcast, and simultaneous inference-engine reload.A weight shard moves from H2D to an IPC buffer after copying, while inference engines load from the other IPC buffer.
  • Implementation constraint: PCIe saturation collapses concurrent H2D and broadcast into sequential operations on NVIDIA H800 clusters.The observed constraint motivates replacing the theoretical three-stage arrangement with a simpler two-stage scheme.
  • Implementation constraint: The adopted two-stage scheme performs synchronous H2D transfer first, then overlaps broadcast with reload.All devices participate in one H2D transfer, after which broadcast and reload proceed in parallel.
  • Scaling assumption: At large scale, splitting the model into small shards can allow the full parameter set to fit into the H2D buffer in one transfer.This removes the overhead associated with multiple synchronous H2D copy operations.
  • Outcome: Overlapping H2D, broadcast, and reload enables high-bandwidth resharding from training engines to inference engines.
Loading 2507.20534v2…