PaperlayerView exact revision on arXiv

Source-linked AI summary

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning

Ling Team, Bin Han, Caizhi Tang, Chen Liang, Donghao Zhang, Fan Yuan, Feng Zhu, Jie Gao, Jingyu Hu, Longfei Li, Meng Li, Mingyang Zhang, Peijie Jiang, Peng Jiao, Qian Zhao, Qingyuan Yang, Wenbo Shen, Xinxing Yang, Yalin Zhang, Yankun Ren, Yao Zhao, Yibo Cao, Yixuan Sun, Yue Zhang, Yuchen Fang, Zibin Lin, Zixuan Cheng, Jun Zhou

arXiv:2510.19338v2cs.LGcs.AIcs.CL

TL;DR

Long-context workloads face attention-related I/O and compute costs, while pure linear attention has practical limitations at industrial scale and common pre-training lengths. The report presents Ring-linear models combining linear and softmax attention, with reported training and inference efficiency gains. It also reports stable long-horizon RL training through training–inference alignment.

  • Problem

    Long-context support is increasingly needed, but attention overhead rises with sequence length and pure linear attention has limitations at industrial scale and common pre-training lengths.

    Source: §1 · p. 2

  • Method

    The Ring-linear series combines linear and softmax attention in a hybrid architecture and uses sparse MoE design.

    Source: §1 · p. 2, §2 · p. 3, §2.1 · p. 3

  • Results

    50% improvement in training performance versus Megatron’s native blockwise FP8 mixed-precision method and 90% increase in inference efficiency were reported.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Systematic training–inference alignment supported stable long-horizon reinforcement-learning training.

    Source: §1 · p. 3

Abstract

from arXiv · show

In this technical report, we present the Ring-linear model series, specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0. Ring-mini-linear-2.0 comprises 16B parameters and 957M activations, while Ring-flash-linear-2.0 contains 104B parameters and 6.1B activations. Both models adopt a hybrid architecture that effectively integrates linear attention and softmax attention, significantly reducing I/O and computational overhead in long-context inference scenarios. Compared to a 32 billion parameter dense model, this series reduces inference cost to 1/10, and compared to the original Ring series, the cost is also reduced by over 50%. Furthermore, through systematic exploration of the ratio between different attention mechanisms in the hybrid architecture, we have identified the currently optimal model structure. Additionally, by leveraging our self-developed high-performance FP8 operator library-linghe, overall training efficiency has been improved by 50%. Benefiting from the high alignment between the training and inference engine operators, the models can undergo long-term, stable, and highly efficient optimization during the reinforcement learning phase, consistently maintaining SOTA performance across multiple challenging complex reasoning benchmarks.

1 Introduction

Long-context demands make attention a resource bottleneck, while pure linear attention can underperform at industrial scale and offers limited pre-training gains at common context lengths. The paper introduces Ring-linear hybrid models and reports efficiency improvements, systematic architecture study, and stable long-horizon RL training.

Motivation

  • Softmax attention has quadratic sequence-length compute and linear output-length I/O overhead, constraining longer-context scaling.Source: §1 · p. 2

Background

  • Linear attention reduces computational complexity from O(n^2d) to O(nd^2) and uses constant-size state memory, improving long-text efficiency.Source: §1 · p. 2

Motivation

  • Pure linear attention’s efficiency gains emerge mainly beyond 8K tokens, while 4K–8K pre-training lengths and MoE computation constrain its pre-training advantages.Source: §1 · p. 2

Contributions

  • The Ring-linear series comprises Ring-mini-linear-2.0 and Ring-flash-linear-2.0, built on a hybrid architecture.Source: §1 · p. 2
  • 50% better training performance and 90% higher inference efficiency were achieved through infrastructure-level optimizations for hybrid linear models.The training comparison is against Megatron’s native blockwise FP8 mixed-precision training method.Source: §1 · p. 3
  • Stable long-horizon RL training was achieved through systematic training–inference alignment, improving model-training stability.Source: §1 · p. 3

2 Model Architecture

Ring-linear combines sparse MoE with linear attention and selected softmax attention, using architecture studies and design choices to balance efficiency with model performance. Its layer-group configurations, recurrent linear-attention state, and implementation choices target long-context training and decoding.

2.1 Basic Architecture

  • Ring-linear uses N layer groups, each with M linear-attention blocks and a GQA block, and a 1/32 activation-ratio MoE; its small-activation MoE models offer up to sevenfold efficiency improvements over dense counterparts.The first block uses a dense MLP rather than an MoE MLP.Source: §2.1 · pp. 3–4, Table 1 · p. 3

2.2 Hybrid Linear Attention

  • Softmax attention has quadratic sequence-length compute and KV-cache storage that grows linearly, limiting efficiency in long-text settings.Source: §2.2 · p. 4

2.2.1 Linear Attention

  • Linear attention uses fixed decay and recurrent prediction with O(nd^2) complexity; its KV-cache state requires constant storage throughout generation.The state contrasts with softmax attention’s linearly growing KV cache.Source: §2.2.1 · pp. 4–5

2.2.2 Hybrid Architecture

  • Hybrid attention consistently outperforms pure softmax in scaling-law curves; the models use M=7 for Ring-flash-linear-2.0 and M=4 for Ring-mini-linear-2.0.The figure compares hybrid M=1 with softmax and different layer-group sizes; M=7 performs well under high FLOP budgets.Source: Figure 3 · p. 5, §2.2.2 · p. 6

2.2.3 Key Architecture Design Choices

  • Adding RoPE to half the Q and K dimensions reduced training language-model loss by approximately 0.004; power-law rather than linear head-wise decay reduced it by approximately 0.04.The authors also report that the decay choice significantly affected downstream-task performance; grouped RMSNorm avoids tensor-parallel all-reduce operations between the linear-attention kernel and output projection.Source: §2.2.3 · p. 6

2.2.4 Cost Analysis for Decoding

  • KV-cache/state-memory access size is compared across sequence lengths and related methods because accelerator memory bandwidth makes it a key factor in decoding speed.Source: §2.2.4 · p. 6, Figure 4 · p. 7

3 Computation Optimization

Section 3 combines kernel fusion, FP8-specific recomputation, and inference optimizations to improve hybrid-model efficiency across training and inference.

3.1 GPU Kernel Optimization

  • Kernel fusion reduces computation latency and activation memory, enabling larger micro-batches that improved MoE training efficiency by over 20%.The reported gain followed increasing micro-batch size from 1 to 2 or from 2 to 4.Source: §3.1 · p. 7
  • Fused kernels combine linear-attention gating and QK normalization operations, while direct routing-map modification integrates padding and unpadding into permute operations.The approach also fuses MoE router casting with computation and reduces linear-attention prefill to one Triton kernel, with an optional additional kernel when Q/K are split.Source: §3.1 · pp. 7–8

3.2 FP8 training optimization

  • FP8 training optimization fuses quantization with adjacent operations and uses state-aware recomputation to avoid redundant quantized-tensor computation.The fused SiLU path reduces I/O volume from approximately 8MN to 4MN; backward operations such as SiLU backward reduce kernel time by nearly half.Source: §3.2 · pp. 8–9

3.3 Training efficiency

  • 77% higher training throughput was achieved for Ring-mini-linear-2.0 with fused kernels, MBS=4, and TP=1 versus its baseline.The optimized setup used GBS=4352; the baseline used GBS=4416, MBS=4, TP=2, and 32 H800 GPUs.Source: Figure 6 · p. 9, §3.3 · p. 9
  • 57% higher overall throughput was achieved for Ring-flash-linear-2.0 after increasing MBS from 1 to 2 and changing PP/VPP from 6/2 to 8/1.Fused kernels alone improved throughput by 25% over the baseline.Source: §3.3 · p. 9

3.4 Inference efficiency

  • Beyond 128K, prefill throughput exceeds Ring-2.0 by more than 2.5 times and baseline models by over 8 times; at 64K, decode throughput exceeds Ring-2.0 by more than twice and baselines by over tenfold.The evaluation measures prefill at batch size 1 and decode at batch size 64.Source: Figure 7 · p. 10, §3.4 · p. 10, Figure 8 · p. 11
  • Optimized fused linear-attention kernels integrated into SGLang and vLLM support more inference modes and improve system throughput.A tree-mask-compatible linear-attention kernel also enables speculative decoding in the Flood offline inference framework; porting to SGLang was underway.Source: §3.4 · pp. 10–11

4 Continued Pre-Training

The continued pre-training procedure initializes Ring-linear models from Ling base models, restores capabilities, and progressively extends context length. The resulting models recover more than 98% of original-model performance in most categories.

4 Continued Pre-Training

  • Ring-mini-linear and Ring-flash-linear models were initialized from corresponding Ling base models by converting each linear-attention QKV projection to MHA and randomly initializing gate-projection and RMSNorm parameters.A two-stage continued pre-training strategy was then used to restore the original base models’ capabilities.Source: §4 · p. 11
  • Continued training used the original corpus at 4K context for 600B tokens for Ring-mini-linear-base and 1T tokens for Ring-flash-linear-base, followed by context extension from 4K to 32K and then 128K.The mid-training stage progressively increased the share of high-quality reasoning data.Source: §4 · p. 11
  • More than 98% of the original models’ performance was restored in most categories, though minor deficiencies remained in reasoning and professional knowledge tasks.The authors suggest knowledge forgetting during continued pre-training may explain these deficiencies.Source: Figure 9 · p. 12, §4 · p. 12
  • The procedure replaced the WSD scheduler with WSM, whose merged mid-training checkpoints simulate learning-rate decay; Ring-linear-base models performed on par with Ling-base models trained on 20T tokens across benchmarks.Figure 9 reports normalized scores relative to Ling-base-2.0.Source: §4 · pp. 11–12, Figure 9 · p. 12

5 Post-Training

Post-training combines broad supervised fine-tuning with long-context reinforcement learning. Systematic training–inference alignment supports stable long-horizon RL, and aligned rollout probabilities improve later-stage rewards while reducing recomputation.

5.2 Reinforcement Learning

  • The authors report that systematic alignment stabilizes RL sufficiently for long-horizon training, without requiring additional algorithmic modifications.Figure 10 is described as showing improved efficiency and stability from aligning individual modules.Source: §5 · p. 12, Figure 10 · p. 13, §5.2 · p. 14

5.1 Supervised Fine-Tuning

  • SFT data spans difficult reasoning and general-purpose domains, including re-synthesized function-calling examples; an earlier-epoch checkpoint is selected for RL.SFT uses a 128K context window, and the earlier checkpoint is chosen to limit overfitting and leave room for RL.Source: §5.1 · pp. 12–13

5.2 Reinforcement Learning

  • RL trains across mathematics, coding, science, logic, and subjective tasks using screened samples and a sufficiently long context window such as 64K.The report says smaller windows such as 32K can increase truncation and lower the performance ceiling for high-difficulty data.Source: §5.2 · p. 13
  • Training–inference discrepancies accumulate across layers, undermining on-policy RL; MoE expert-selection differences and longer outputs exacerbate the mismatch.The discrepancy can produce token probabilities of 0 during training and 1 during inference.Source: §5.2 · p. 13

5.2.1 Systematic Training-Inference Alignment

  • Alignment compares matching inputs module-by-module and layer-by-layer, targeting implementation consistency, appropriate precision, and nondeterminism.The process covers training prefill versus inference prefill, training prefill versus inference decode, and different parallelization configurations.Source: §5.2.1 · p. 14
  • Linear-attention KV-cache accumulation needs higher precision: BF16 initialization can cause growing divergence, while a custom GEMM supports precise LM-head computation with BF16 inputs.The report also flags FP32 RMSNorm computation and implementation differences in RoPE as alignment concerns.Source: §5.2.1 · pp. 14–15

5.2.2 From the Algorithmic Perspective

  • After systematic alignment, rollout probabilities yield higher late-stage rewards, steadier training–inference disparity, and avoid recomputing training probabilities.Figure 12 compares rollout probabilities with recomputed training probabilities.Source: Figure 12 · p. 16, §5.2.2 · p. 16

5.2.3 Experimental Results of Reinforcement Learning

  • Reward and test scores both rise consistently during Ring-mini-linear-2.0’s RL process.The reported training curves show continued growth in both measures.Source: Figure 13 · p. 16, §5.2.3 · p. 16

6 Evaluation

The evaluation compares the Ring-linear models with similarly scaled counterparts across reasoning benchmarks. Both models are reported to perform comparably to their respective comparison groups across varied reasoning tasks.

6.1 Evaluation Setups

  • The evaluation covers 17 benchmarks across mathematical, agent-and-coding, and general reasoning dimensions, comparing each model with similarly scaled counterparts.The comparison groups include Ring, Qwen, GPT-OSS, Gemini, Seed-OSS, and Qwen3-Next models.Source: §6.1 · pp. 16–17

6.2 Main Results

  • Ring-mini-linear-2.0, with 1.6B activated parameters, performs comparably to counterpart models across various reasoning tasks.Table 2 summarizes its benchmark results.Source: Table 2 · p. 17, §6.2 · p. 17
  • Ring-flash-linear-2.0 delivers highly comparable performance to its counterparts across diverse reasoning dimensions.Table 3 summarizes its benchmark results.Source: §6.2 · p. 17, Table 3 · p. 18

7 Conclusion

The conclusion presents Ring-mini-linear-2.0 and Ring-flash-linear-2.0 as hybrid Ring-linear models, while identifying memory and computation costs that remain in the architecture.

7 Conclusion

  • The Ring-linear models combine hybrid architecture with LingHe FP8 fused kernels to improve training and inference efficiency.The report also attributes more stable RL training to a well-aligned RL system.Source: §7 · p. 18
  • The conclusion says aligned RL training contributes to greater training stability and a higher model performance ceiling.Source: §7 · p. 18
  • Maintaining equal Q, K, and V head counts in linear attention brings heavy memory overhead, while remaining softmax modules add computational bottlenecks.The report identifies both as limitations affecting the hybrid architecture’s efficiency.Source: §7 · p. 18

Related papers

Loading 2510.19338v2…