PaperlayerView exact revision on arXiv

Source-linked AI summary

Kimi Linear: An Expressive, Efficient Attention Architecture

Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, Yulun Du

arXiv:2510.26692v2cs.CLcs.LG

TL;DR

Long-context and RL inference expose quadratic attention costs and linearly growing KV caches as bottlenecks. Kimi Linear combines channel-gated KDA with periodic full attention, matching or outperforming full-attention baselines while reducing memory and increasing decoding throughput.

  • Problem

    Quadratic self-attention time and linearly growing KV caches create computational and memory overheads that hinder long-context scaling, throughput, and real-time interactivity.

  • Method

    Kimi Linear interleaves channel-gated KDA with full attention layers at a fixed 3:1 ratio, using specialized DPLR transitions for efficient chunkwise computation.

  • Results

    Kimi Linear consistently matches or outperforms full-attention baselines across short-context, long-context, and RL-style tasks, reducing memory by up to 75% and increasing decoding throughput by up to 6.3×.

  • Takeaways & Limitations

    Kimi Linear is presented as a strong alternative to full-attention architectures for efficient large-language-model inference, including million-token contexts.

  • Takeaways & Limitations

    RL training carries a risk of degenerating general capabilities, so the recipe incorporates PTX loss and concurrent supervised fine-tuning to mitigate it.

Abstract

from arXiv · show

We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule. We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths. To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.

1 Introduction

Kimi Linear addresses the quality, memory, and throughput limitations of standard attention with a hybrid architecture centered on KDA. It combines fine-grained memory control, efficient chunkwise computation, and periodic full-attention layers to improve performance across contexts and RL-style evaluations.

  • Kimi Linear targets the quadratic computation and linearly growing KV cache that hinder long-horizon inference, context scaling, and real-time interaction.
  • KDA extends Gated DeltaNet with channel-wise forgetting rates, enabling more precise regulation of finite-state recurrent memory.
  • KDA uses specialized DPLR transition dynamics and a bespoke chunkwise-parallel algorithm that reduces computation relative to general DPLR formulations.
  • A uniform 3:1 KDA-to-full-attention ratio reduces memory and KV-cache usage by up to 75% while preserving global information flow.
  • Kimi Linear matches or outperforms strong full-attention baselines across short-context, long-context, and RL-style tasks, with up to 6× higher decoding throughput at 1M context.
  • The released KDA kernels, vLLM integration, and checkpoints are drop-in compatible with existing full-attention pipelines.

2 Preliminary

The preliminary section frames linear attention as recurrent associative memory and traces its progression from unforgetting accumulation through delta-rule correction to gated forgetting. These mechanisms address interference but retain different expressive and computational trade-offs.

  • Chunkwise notation divides the sequence into L/C chunks of length C, with each chunk’s initial state inherited from the previous chunk’s final state.
  • Linear attention maintains a matrix-valued recurrent state that accumulates key–value associations as an associative memory.
  • Without forgetting, continual reinforcement of recent key–value pairs makes the state grow unbounded and causes interference over long contexts.
  • The classical delta rule updates associative memory toward the mapping k_t 7→v_t using rank-1 transformations that support chunkwise parallelization.
  • Gated DeltaNet adds a scalar forget gate that acts as weight decay, controlling memory lifespan while preserving DeltaNet’s parallelizable structure.
  • Gated DeltaNet can also be interpreted as data-dependent, learnable multiplicative positional encoding that relaxes RoPE’s orthogonality constraint.

3 Kimi Delta Attention: Improving Delta Rule with Fine-grained Gating

KDA refines the gated delta rule with diagonal, fine-grained gating and packages recurrent updates into an efficient chunkwise computation. Its implementation combines compact rank-1 representations, reduced non-matmul work, and Tensor Core-friendly execution.

  • KDA replaces GDN’s scalar decay with a diagonal gate that independently controls memory decay across feature dimensions.
  • KDA compresses a series of rank-1 transformations into a dense chunkwise representation while maintaining stability under diagonal gating.
  • The WY representation packs rank-1 updates into a compact form and avoids an additional matrix inversion in subsequent computations.
  • The UT transform reduces non-matmul FLOPs to improve hardware utilization during training.
  • KDA uses inter-block recurrence and intra-block parallelism during output computation to increase matrix-multiplication throughput on Tensor Cores.
  • KDA matches generalized DPLR representational capacity while addressing precision and speed issues associated with fine-grained decay implementations.

4 The Kimi Linear Model Architecture

Kimi Linear uses a layerwise hybrid that alternates KDA with full MLA layers, combining efficient recurrent processing with global attention. The architecture also assigns positional encoding responsibilities to KDA and reports favorable kernel-speed scaling.

  • The model backbone builds on Moonlight and adds fine-grained gating plus components intended to improve Kimi Linear’s expressiveness.
  • KDA receives ShortConv- and Swish-processed q, k, and v representations, while q and k use L2 normalization for eigenvalue stability.
  • Kimi Linear’s blocks contain token mixing followed by a mixture-of-experts channel-mixing layer, with three KDA layers interleaved before one MLA layer.
  • The output stage applies head-wise RMSNorm and a low-rank data-dependent gate before output projection.
  • The architecture uses a layerwise 3:1 KDA-to-full-MLA ratio, selected for infrastructure simplicity and training stability.
  • KDA layers carry positional information and recency bias while MLA layers use NoPE, enabling efficient MQA conversion during inference.

5 Experiments

Experiments show Kimi Linear’s KDA and 3:1 KDA-to-MLA hybrid consistently outperform strong baselines across synthetic, pretraining, SFT, long-context, and RL evaluations, while improving efficiency at scale.

  • Synthetic tests: KDA achieves the highest accuracy as sequence length increases from 256 to 2,048 tokens across the synthetic tasks.On Palindrome and MQAR, KDA converges significantly faster than GDN, while Mamba2 fails on all tasks in the tested setting.
  • Ablations: The 3:1 KDA-to-MLA ratio yields the lowest training and validation losses among tested hybrid configurations.A 7:1 ratio worsens validation performance, whereas 1:1 increases inference overhead; the pure full-attention baseline performs poorly.
  • Scaling laws: 1.16× computational efficiency is achieved by Kimi Linear relative to MLA baselines under compute-optimal scaling-law training.The scaling-law comparison uses the best 3:1 hybrid ratio for Kimi Linear and matched MLA training configurations.
  • Pretraining: Kimi Linear consistently outperforms MLA and GDN-H across almost all evaluated pretraining categories using the same 1.4T-token recipe.It leads on general knowledge, math and code reasoning, and Chinese tasks, including BBH, MMLU, HellaSwag, GSM8K, CRUXEval, CEval, and CMMLU, with some exceptions such as EvalPlus.
  • Supervised fine-tuning: Kimi Linear consistently outperforms MLA and GDN-H after the same SFT recipe across general, math, and code benchmarks.It leads on difficult tasks including AIME 2025, HMMT 2025, PolyMath-en, and LiveCodeBench, despite minor exceptions on MATH500 and EvalPlus.
  • Long context and RL: 54.5 is Kimi Linear’s highest average score across 128k long-context benchmarks, with RULER 84.3 and RepoQA 68.5 among its strongest results.It outperforms MLA and GDN-H on most tasks, except LongBench V2 and Frames; at 1M sequences, decoding is 2.9× faster than MLA.
  • Long context and RL: Kimi Linear’s training and test accuracy improve faster than MLA during reasoning-intensive long-form generation under RL.The gap widens throughout Math RL training, including on MATH500 and AIME2025.
  • Overall findings: Kimi Linear remains the top performer overall across pretraining, SFT, long-context, and RL stages, although GDN-H falls behind MLA on long-context evaluations.This performance hierarchy shifts by evaluation regime rather than remaining fixed across all stages.

6 Discussions

The discussion frames KDA as a data-dependent positional mechanism and a constrained DPLR variant designed to retain expressiveness while improving hardware efficiency. Kimi Linear combines KDA with periodic full attention, using fixed-size recurrent state for efficient long-context decoding.

  • Kimi Delta Attention as learnable position embeddings: KDA can be viewed as multiplicative positional encoding with data-dependent transitions, potentially addressing RoPE’s fixed-frequency extrapolation limitations.The paper compares gated delta-rule linear attention with RoPE-style formulations and motivates channel-wise gating by RoPE’s fine-grained positional encoding.
  • Relation to DPLR: KDA uses a constrained DPLR form that shares α_t and sets a_t = β_tk_t, b_t = k_t ⊙ α_t for efficient fine-grained decay and delta-rule updates.This specialization combines multiplicative state decay with a Householder-style transformation while avoiding the full generality of DPLR.
  • Relation to DPLR: Fixing a = b = k removes two secondary chunking steps and reduces redundant operations caused by reciprocal cumulative-decay terms.The optimization addresses numerical-stability overhead in chunkwise DPLR computation.
  • Relation to DPLR: KDA eliminates roughly three matrix multiplications during inter-chunk and output computation, accelerating the kernel.The reduction targets both inter-chunk processing and output computation.
  • Relation to DPLR: KDA reaches nearly 2× the speed of DPLR for sequence lengths up to 64k.The kernel benchmark compares KDA directly with the general DPLR formulation.
  • Long-context decoding: Linear KDA maintains a fixed-sized state of d_k × d_v per head, while the hybrid recurrent-plus-parallel design approaches a 3:1 efficiency ratio as sequence length grows.The reported 1M-context result is a 2.3× speedup, reflecting the asymptotic hybrid efficiency trend.

7 Related Works

Related work contrasts full, linear, sparse, and hybrid attention for long-context modeling. The paper positions Kimi Linear as an inter-layer hybrid that combines structured linear memory with periodic global attention to address retrieval and efficiency limitations.

  • Efficient Subquadratic Attention: Quadratic self-attention remains a bottleneck for million-token contexts used in agentic tool use and repository-level code analysis.This motivates efficient attention mechanisms for long-context inference.
  • Efficient Subquadratic Attention: Linear attention replaces the explicit O(T^2) similarity matrix with kernelized feature interactions, enabling linear-time computation in sequence length.The reformulation computes attention through two associative matrix products.
  • Efficient Subquadratic Attention: Linear attention uses a fixed-size matrix-valued state and learnable gates to selectively retain and forget information instead of storing an expanding KV cache.The fast-weight view interprets this state as a low-capacity associative table updated online.
  • Efficient Subquadratic Attention: Pure linear attention still struggles with precise retrieval and exact copying, motivating hybrid designs and structured delta-rule updates.These weaknesses matter for robust recall beyond 1M tokens and extensive tool-use contexts.
  • Hybrid Model: Inter-layer hybrids stack full-attention and linear layers at a predefined ratio, simplifying distributed optimization and KV-cache management relative to intra-layer hybrids.Kimi Linear uses a fixed 3:1 interleaving and selects KDA for stronger retrieval and copying performance.
  • Hybrid Model: Kimi Linear combines channel-wise-gated KDA with global attention, reducing memory usage by up to 75% and achieving up to 6.3× higher decoding throughput than full-attention baselines.The paper presents this design as a scalable approach for agentic intelligence and test-time scaling.

A Contributions

The supplied passages list author information for the contributions section but do not describe specific technical contributions.

  • A Contributions: The supplied contribution-section passage contains author and affiliation markers rather than technical contribution statements.No method, experiment, or result is described in these passages.
  • A Contributions: The passage lists Zongyu Lin, Xingcheng Yao, Jiaxi Hu, and other contributors.The names are presented as part of the paper’s author list.
  • A Contributions: Additional contributor names continue across the supplied author-list passages.These passages provide bibliographic information without technical claims.

1 Soochow University, China

The supplied passages contain contributor names but do not provide substantive information about Soochow University or a technical section.

  • 1 Soochow University, China: The supplied passage lists additional contributor names associated with the paper.No institutional description or technical claim is included.
  • 1 Soochow University, China: The author list continues with further contributor names.The passage does not specify roles, methods, or results.
  • 1 Soochow University, China: The remaining supplied passages continue the paper’s author list.They provide no substantive discussion of Soochow University.

B Derivations for Chunkwise Parallelism of KDA

This section derives parallelizable forms for KDA by transforming recurrent state transitions into matrix representations. It uses cumulative generalized Householder products and the classic WY representation to support parallel computation.

  • KDA’s recurrent form is first recalled before deriving equivalent matrix formulations for chunkwise parallelism.
  • The cumulative product of generalized Householder matrices is optimized using the classic WY representation.
  • Proposition 1 expresses the matrix P_r[t] through a recurrence involving the auxiliary vector w_r.
  • The stated propositions are established by mathematical induction, with the inductive steps concluding that the recurrences hold.
  • The resulting recurrent quantities can also be expressed in parallelizable form for computation.
  • Proposition 2 expresses the matrix H_r[t] through a recurrence involving the auxiliary vector u_r.

C Pseudo Code for chunkwise KDA

The chunkwise KDA implementation reshapes inputs into chunked head-wise tensors, accumulates gating values, constructs masked transformations, and updates recurrent state across chunks.

  • The implementation defines chunk_kda with query, key, value, gate, beta, optional initial state, and a configurable chunk size.
  • Inputs are rearranged into batch, head, chunk, and within-chunk dimensions before computation in floating-point format.
  • The code scales queries by K^-0.5, computes cumulative gates, and constructs an upper-triangular mask for chunkwise operations.
  • Within each chunk, the implementation forms transformed keys and values, initializes the state S, and iterates over chunks to produce outputs.
  • Listing 1 presents pseudo PyTorch-style code for the chunked form of KDA.

D Kimi Linear@5.7T results

With extended 5.7T-token training, Kimi Linear consistently outperforms Moonlight across nearly all reported benchmarks, including a 1M-context RULER score of 94.8.

  • 5.7T tokens: Kimi Linear consistently outperforms Moonlight across nearly all benchmarks with 3× sparsity and a new attention architecture.
  • Moonlight-Instruct was not evaluated on tasks exceeding its 8K context limit.
  • 94.8: Kimi Linear@5.7T obtains this RULER score at 1M context length.
  • Tables 8 and 9 report comparisons across diverse tasks for base and instruction-tuned Kimi Linear and Moonlight models.
Loading 2510.26692v2…