PaperlayerView exact revision on arXiv

Source-linked AI summary

Kimi Linear: An Expressive, Efficient Attention Architecture

Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, Yulun Du

arXiv:2510.26692v2cs.CLcs.LG

TL;DR

Standard attention’s quadratic time complexity and growing KV cache impose computational and memory overheads in long-horizon and reinforcement-learning settings. Kimi Linear combines channel-wise-gated KDA with periodic full-attention layers, matching or outperforming full-attention baselines across evaluated scenarios while reaching up to 6× decoding throughput at 1M context.

  • Problem

    Standard attention’s quadratic time complexity and linearly growing KV cache impose computational and memory overheads in long-horizon and reinforcement-learning settings.

    Source: §1 · p. 2

  • Method

    Kimi Linear combines channel-wise-gated KDA with periodic full-attention layers in a fixed 3:1 inter-layer hybrid.

    Source: §1 · p. 2

  • Results

    Kimi Linear matches or outperforms strong full-attention baselines across short-context, long-context, and RL-style tasks, with up to 6× higher decoding throughput at 1M context.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The hybrid reduces memory and KV-cache usage by up to 75% during long-sequence generation, positioning Kimi Linear as an alternative to full-attention architectures.

    Source: §1 · p. 2, Conclusion · p. 18

    RL training risks degeneration of general capabilities, which the authors mitigate with PTX loss and concurrent supervised fine-tuning.

    Source: §5.4.2 · p. 10

Abstract

from arXiv · show

We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule. We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths. To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.

1 Introduction

Kimi Linear addresses the inference costs of long-horizon and RL workloads with a hybrid architecture that combines linear attention efficiency with periodic global attention. Its introduction presents KDA’s finer-grained memory gating and chunkwise computation as central to that design.

1 Introduction

  • Softmax attention’s quadratic time complexity and linearly growing KV cache burden long-horizon inference, while linear attention’s finite-state capacity makes long-sequence modeling and in-context retrieval challenging.Hybrid models are presented as a practical compromise, but prior work often used limited scale or lacked broad evaluation.Source: §1 · p. 2
  • KDA extends Gated DeltaNet [111] with channel-wise independent forgetting rates, while specialized DPLR transitions enable chunkwise-parallel computation that reduces work and remains consistent with the classical delta rule.The finer-grained gating is intended to regulate finite-state RNN memory more precisely.Source: §1 · p. 2, Contributions · p. 2

Contributions

  • After 1.4T-token training, Kimi Linear matches or outperforms strong full-attention baselines across short-context, long-context, and RL-style tasks, with up to 75% lower KV-cache usage and up to 6× higher decoding throughput at 1M context.The comparison uses matched-scale pretraining and evaluation.Source: §1 · p. 2, Contributions · p. 2
  • Kimi Linear interleaves KDA with periodic full-attention layers in a uniform 3:1 ratio, preserving global information flow while reducing memory and KV-cache use during long-sequence generation.The architecture’s 3:1 KDA-to-global-attention ratio is also identified as a contribution.Source: §1 · p. 2, Contributions · p. 2
  • The authors release KDA kernels with vLLM integration and pretrained and instruction-tuned checkpoints, compatible with existing full-attention pipelines without changes to caching or scheduling interfaces.The release is intended to support further research on hybrid architectures.Source: §1 · p. 2, Contributions · p. 2

2 Preliminary

The preliminaries define the paper’s notation and frame linear attention as recurrent associative memory, motivating delta-rule correction and gated forgetting.

2.1 Notation

  • The notation defines vector and matrix-valued memory symbols, triangular masks, and chunk indexing, with each chunk’s initial state equal to the previous chunk’s final state.A sequence is divided into chunks of length C, and the state matrices are re-indexed by chunk.Source: §2.1 · p. 3

2.2 Linear Attention and the Gated Delta Rule

  • Linear attention [48] stores key–value associations in a recurrent matrix state, but unbounded accumulation without forgetting causes interference over long contexts.The fast-weight view [84] [85] interprets this update as gradient descent on an unbounded correlation objective that offers no criterion for erasing memories.Source: §2.2 · p. 3
  • DeltaNet [84] updates associative memory by online gradient descent on reconstruction loss, using a rank-1 classical delta rule that supports hardware-efficient chunkwise parallelization [112].The rule continually corrects the memory toward the mapping from each key to its value.Source: §2.2 · p. 3
  • Gated DeltaNet [111] adds a scalar forget gate that decays fast weights, controlling memory lifespan and mitigating interference while preserving DeltaNet’s parallelizable structure.The gate acts like data-dependent L2 regularization [8] and improves stability and long-context generalization.Source: §2.2 · p. 3

3 Kimi Delta Attention: Improving Delta Rule with Fine-grained Gating

Kimi Delta Attention (KDA) refines scalar decay with a fine-grained diagonal gate, while its chunkwise algorithm supports efficient computation. KDA improves operator efficiency over the standard DPLR formulation.

3 Kimi Delta Attention: Improving Delta Rule with Fine-grained Gating

  • KDA replaces GDN’s scalar decay with a fine-grained diagonalized gate that controls memory decay and positional awareness.Its representational capacity aligns with the generalized DPLR formulation, with both exhibiting fine-grained decay behavior.Source: §3 · p. 4, §3.2 · p. 5

3.1 Hardware-Efficient Chunkwise Algorithm

  • The chunkwise algorithm packs rank-1 updates into a dense WY representation and uses a UT transform to reduce non-matmul FLOPs.It uses Comba’s formulation of P to avoid an additional matrix inversion and an inter-block recurrent, intra-block parallel output strategy to utilize Tensor Cores.Source: §3 · p. 4, §3.1 · pp. 4–5

3.2 Efficiency Analysis

  • KDA improves operator efficiency by roughly 100% compared to the DPLR formulation.Binding both variables a and b to k reduces second-level chunk matrix computations from four to two and eliminates three additional matrix multiplications, while retaining fine-grained decay behavior.Source: §3.2 · p. 5

4 The Kimi Linear Model Architecture

Kimi Linear builds on the Moonlight backbone, combining KDA and full MLA layers in a 3:1 layerwise hybrid. KDA provides positional awareness, while full MLA layers use NoPE.

Model architecture

  • Kimi Linear follows the Moonlight [62] backbone and adds fine-grained gating plus components to improve KDA expressiveness.Its blocks pair a token-mixing layer with a MoE channel-mixing layer.Source: §4 · p. 5, Figure 3 · p. 6

Neural Parameterization

  • ShortConv with Swish processes q, k, and v, while L2Norm on q and k is used to ensure eigenvalue stability.Key and value head dimensions are both 128 in all experiments.Source: §4 · p. 5
  • A low-rank output gate preserves parameter fairness, matches full-rank gating comparably in performance, and alleviates the Attention Sink [79].The output projection is preceded by head-wise RMSNorm.Source: §4 · p. 6

Hybrid model architecture

  • Interleaving three KDA layers with one full MLA layer gives the model its chosen quality–throughput trade-off.The architecture uses layerwise alternation rather than mixing KDA and MLA heads within layers, for infrastructure simplicity and training stability.Source: §4 · p. 6, Figure 3 · p. 6

No Position Encoding (NoPE) for MLA Layers

  • NoPE in all full MLA layers assigns positional information and recency bias to KDA and enables inference-time conversion to pure MQA.NoPE also simplifies long-context training by removing the need for RoPE frequency-base tuning or methods such as YaRN [72].Source: §4 · pp. 6–7

5 Experiments

Across pretraining, supervised fine-tuning, long-context evaluation, and reinforcement learning, Kimi Linear ranked as the strongest overall model among the tested systems, while also improving inference efficiency at long sequence lengths. Ablations and synthetic tasks further support the effectiveness of its hybrid design and fine-grained decay.

5.1 Synthetic tests

  • KDA achieved the highest accuracy as synthetic-task sequences grew from 256 to 2,048 tokens and converged faster than GDN on Palindrome and MQAR.Mamba2 failed on all three tasks in the reported settings; the authors attribute KDA’s results to fine-grained decay that selectively forgets irrelevant information while preserving important memories.Source: §5.1 · pp. 7–8

5.2 Ablation on Key Components of Kimi Linear

  • The 3:1 KDA-to-MLA ratio performed best; Sigmoid gating outperformed ungated and swish variants, and convolution remained useful in hybrid models.A 7:1 ratio had comparable training loss but worse validation performance; 1:1 had similar validation loss with greater inference overhead, while 0:1 performed poorly. Kimi Linear also had stronger long-context performance than its RoPE variant, which scored similarly on short-context tasks.Source: §5.2 · pp. 8–9

5.3 Scaling Law of Kimi Linear

  • With the 3:1 hybrid ratio, Kimi Linear achieved approximately 1.16× computational efficiency versus MLA in compute-optimal scaling-law experiments.The experiments used Moonlight-based MoE models, activating 8 of 64 experts; MLA model hyperparameters were tuned through grid search, while KDA otherwise followed MLA’s training configuration.Source: §5.3 · p. 9, Figure 5 · p. 9

5.4 Experimental Setup

  • Fair comparisons used matched model architecture, parameter count, and training setup, including a 48B-total, 3B-active model, and a shared 1.4T-token pretraining recipe.Models were evaluated against MLA and GDN-H, with a Kimi Linear RoPE variant; pretraining used a 4,096-token context, MuonClip, WSD scheduling, and a 32-million-token global batch.Source: §5.4 · p. 9, §5.4.1 · p. 10

5.4.2 Post-training recipe

  • Post-training used staged instruction tuning and math, code, and STEM RL prompts, with PTX loss and stability techniques to mitigate capability loss and training instability.PTX included concurrent SFT on diverse reasoning and general-purpose data; RL used truncated importance sampling plus dynamic KL penalties and mini-batch sizes.Source: §5.4.2 · p. 10

5.5.1 Kimi Linear@1.4T results

  • At 128k context, Kimi Linear scored 84.3 on RULER, 68.5 on RepoQA, and 54.5 average; it also led most SFT benchmarks and improved faster than MLA during RL.Long-context exceptions included LongBench V2 and Frames. After SFT, exceptions included MATH500 and EvalPlus; during RL, Kimi Linear improved faster and better than MLA on MATH500 and AIME 2025.Source: §5.5.1 · pp. 11–12, Table 4 · p. 11, Table 5 · p. 12
  • Kimi Linear ranked first overall across pretraining, SFT, long-context evaluation, and RL, outperforming full-attention MLA across the reported stages.At pretraining, the reported hierarchy was Kimi Linear over GDN-H over MLA; in long-context evaluation GDN-H fell behind MLA. Best-per-column results are shown in Table 3.Source: Table 3 · p. 11, §5.5.1 · p. 12

5.6 Efficiency Comparison

  • At 1M context, Kimi Linear decoded 6× faster than full-attention MLA and was 2.9× faster for sequence processing.For sequence processing, the reported advantage over MLA was 2.3× at 512k and 2.9× at 1M; prefilling was comparable at 4k–16k and faster from 128k onward.Source: Figure 7 · p. 13, §5.6 · p. 13

6 Discussions

The discussion presents KDA as a learnable positional mechanism with channel-wise gating, explains how its constrained DPLR form improves efficiency, and describes the hybrid inference strategy’s fixed-state long-context benefits.

6.1 Kimi Delta Attention as learnable position embeddings

  • KDA treats GDN’s data-dependent transition as a learnable positional encoding and adds channel-wise gates; Kimi Linear uses NoPE in its global MLA layers.The channel-wise gate addresses standard GDN’s per-head scalar decay, which lacks RoPE’s per-dimension frequency diversity.Source: §6.1 · pp. 13–14

6.2 Relation to DPLR

  • KDA constrains DPLR by setting a=b=k, avoiding secondary chunking and roughly three matrix multiplications; its kernel runs nearly 2× as fast as DPLR up to 64k.These changes reduce redundant operations and improve kernel-level efficiency.Source: §6.2 · p. 15

6.3 Complexity Analysis

  • At a 1M-token context, Kimi Linear achieves a 2.3× speedup; in long-context scenarios up to 1M tokens, memory efficiency yields a theoretical decoding speedup of up to 6.3×.For autoregressive generation, it switches to a recurrent kernel and maintains a fixed 128×128 state per head regardless of sequence length.Source: §6.3 · p. 16

7 Related Works

The section surveys linear and sparse approaches to efficient long-context attention, then motivates inter-layer hybrids and positions Kimi Linear as a practical combination of KDA and full attention.

7.1 Efficient Subquadratic Attention

  • Linear attention replaces the explicit quadratic similarity matrix with linear-time feature interactions and gated recurrent state updates, while sparse attention selects a subset of tokens.Sparse methods include static window patterns and dynamic clustering or routing; dynamic selection can add overhead that limits speedup without dedicated kernel acceleration.Source: §7.1 · pp. 16–17
  • Gated linear attention uses a fixed-size matrix state and learnable gates to retain or forget information, with variants differing in forget-gate parameterization.RetNet [92] uses data-independent scalar decay, Mamba2 [16] uses data-dependent scalar decay, and GLA [114] uses diagonal channel-wise gates.Source: §7.1 · p. 17
  • At a one-million-token context, KDA achieves over a 2× decoding speedup and competitive performance against the full-attention baseline.Linear attention uses a constant-size state, whereas sparse attention can retrieve fine-grained history but stores the full KV cache for selection; the approaches may be combined.Source: §7.1 · p. 17

7.2 Hybrid Model

  • Inter-layer hybrids stack distinct attention types to reduce the system complexity of intra-layer hybrids, and Kimi Linear interleaves KDA and full attention at a fixed 3:1 ratio.The repeating structure simplifies KV-cache management, and the authors report KDA performs better overall than Mamba2 [16], particularly on retrieval and copying.Source: §7.2 · p. 18
  • Hybrid models can be sensitive to RoPE base frequency when extending context, and the authors find that combining KDA layers with NoPE full-attention layers facilitates context-window extension.Related approaches include Falcon-H [126] with a high base frequency and SwanGPT [76] with interleaved RoPE- and NoPE-based full-attention layers.Source: §7.2 · p. 18

Conclusion

  • Kimi Linear reduces memory usage by up to 75% and achieves up to 6.3× higher decoding throughput while outperforming full-attention baselines.It interleaves KDA with global attention at a 3:1 ratio.Source: Conclusion · p. 18

A Contributions

The section lists the paper’s contributors and explains how contribution order and name markers should be interpreted.

  • Authors are listed by significance of contribution, with project leaders appearing last.Source: A Contributions · p. 24
  • The project was developed at Moonshot AI with external collaborators marked #; names marked * indicate people no longer on the team.Source: A Contributions · p. 24
  • The section names the paper’s contributors, including Zongyu Lin, Xingcheng Yao, and Bo Pang.Source: A Contributions · p. 24

2 The Hong Kong University of Science and Technology (Guangzhou) … B Derivations for Chunkwise Parallelism of KDA

This appendix derives parallelizable matrix forms for KDA’s recurrent computation. It uses a WY representation for cumulative generalized Householder matrices and gives recurrence-based expressions for two matrices.

B Derivations for Chunkwise Parallelism of KDA

C Pseudo Code for chunkwise KDA

The chunkwise KDA pseudocode splits the sequence into chunks, constructs within-chunk transformations, and combines chunk outputs with a recurrent state. It also uses secondary chunk processing for numerical stability.

C Pseudo Code for chunkwise KDA

  • Within each chunk, KDA builds a gated key-similarity matrix, applies forward substitution, and uses the result to transform keys and values.The code first accumulates g, applies beta to the matrix, and then uses the transformed tensors to form w and u.Source: C Pseudo Code for chunkwise KDA · p. 27
  • For each chunk, KDA computes outputs using queries, corrected values, and the current state, then decays and updates that state.An optional initial state is added before processing; the state update uses the final chunk gate and key-value contributions.Source: C Pseudo Code for chunkwise KDA · p. 27
  • The pseudocode uses secondary chunking within output computation for numerical stability.It loops over positions within the chunk when constructing the masked query-key matrix.Source: C Pseudo Code for chunkwise KDA · p. 27

D Kimi Linear@5.7T results

With 3× sparsity and a new attention design, Kimi Linear trained on 5.7T tokens outperforms Moonlight across nearly all benchmarks. It scores 94.8 on RULER at 1M context length.

D Kimi Linear@5.7T results

  • With 3× sparsity and a new attention design, Kimi Linear@5.7T consistently outperforms Moonlight across nearly all benchmarks.Results cover base and instruction-tuned models; Moonlight-Instruct was not evaluated on tasks exceeding its 8K context limit.Source: D Kimi Linear@5.7T results · p. 28, Table 8 · p. 28, Table 9 · p. 28
  • 94.8 is Kimi Linear@5.7T’s RULER score at 1M context length.The paper presents this long-context result as evidence that Kimi Linear is a promising alternative to full-attention architectures, with comparable or superior results and potentially more efficient resource use.Source: D Kimi Linear@5.7T results · p. 28

Related papers

Loading 2510.26692v2…