PaperlayerView exact revision on arXiv

Source-linked AI summary

Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model

Ling Team, Anqi Shen, Baihui Li, Bin Hu, Bin Jing, Cai Chen, Chao Huang, Chao Zhang, Chaokun Yang, Cheng Lin, Chengyao Wen, Congqi Li, Deng Zhao, Dingbo Yuan, Donghai You, Fagui Mao, Fanzhuang Meng, Feng Xu, Guojie Li, Guowei Wang, Hao Dai, Haonan Zheng, Hong Liu, Jia Guo, Jiaming Liu, Jian Liu, Jianhao Fu, Jiannan Shi, Jianwen Wang, Jianxin Lai, Jin Yang, Jun Mei, Jun Zhou, Junbo Zhao, Junping Zhao, Kuan Xu, Le Su, Lei Chen, Li Tang, Liang Jiang, Liangcheng Fu, Lianhao Xu, Linfeng Shi, Lisha Liao, Longfei Zheng, Meng Li, Mingchun Chen, Qi Zuo, Qiang Cheng, Qianggang Cao, Qitao Shi, Quanrui Guo, Senlin Zhu, Shaofei Wang, Shaomian Zheng, Shuaicheng Li, Shuwei Gu, Siba Chen, Tao Wu, Tao Zhang, Tianyu Zhang, Tianyu Zhou, Tiwei Bie, Tongkai Yang, Wang Hong, Wang Ren, Weihua Chen, Wenbo Yu, Wengang Zheng, Xiangchun Wang, Xiaodong Yan, Xiaopei Wan, Xin Zhao, Xinyu Kong, Xinyu Tang, Xudong Han, Xudong Wang, Xuemin Yang, Xueyu Hu, Yalin Zhang, Yan Sun, Yicheng Shan, Yilong Wang, Yingying Xu, Yongkang Liu, Yongzhen Guo, Yuanyuan Wang, Yuchen Yan, Yuefan Wang, Yuhong Guo, Zehuan Li, Zhankai Xu, Zhe Li, Zhenduo Zhang, Zhengke Gui, Zhenxuan Pan, Zhenyu Huang, Zhenzhong Lan, Zhiqiang Ding, Zhiqiang Zhang, Zhixun Li, Zhizhen Liu, Zihao Wang, Zujie Wen

arXiv:2510.18855v2cs.CLcs.AI

TL;DR

Trillion-parameter reasoning models face training instability, inefficient long-rollout processing, and RL-system bottlenecks. Ring-1T addresses these challenges with IcePop, C3PO++, and ASystem, achieving state-of-the-art results across demanding reasoning benchmarks while leaving some consistency and capability limitations.

  • Problem

    Trillion-parameter RL training is constrained by training-inference discrepancies, long-rollout inefficiency, and systemic bottlenecks, especially for large MoE models.

  • Method

    Ring-1T combines IcePop for discrepancy-based token correction, C3PO++ for budget-controlled rollout partitioning, and ASystem for large-scale asynchronous RL infrastructure.

  • Results

    Ring-1T achieves state-of-the-art performance across challenging reasoning benchmarks, including 93.40% on AIME 25 and 86.72% on HMMT 25, while leading all open-weights models.

  • Takeaways & Limitations

    The work demonstrates that interconnected algorithmic and infrastructure advances can support stable and efficient training of an open-weights trillion-parameter thinking model.

  • Takeaways & Limitations

    IcePop does not achieve perfect training-inference consistency, and Ring-1T’s training under-optimizes advanced agentic skills such as tool use.

Abstract

from arXiv · show

We present Ring-1T, the first open-source, state-of-the-art thinking model with a trillion-scale parameter. It features 1 trillion total parameters and activates approximately 50 billion per token. Training such models at a trillion-parameter scale introduces unprecedented challenges, including train-inference misalignment, inefficiencies in rollout processing, and bottlenecks in the RL system. To address these, we pioneer three interconnected innovations: (1) IcePop stabilizes RL training via token-level discrepancy masking and clipping, resolving instability from training-inference mismatches; (2) C3PO++ improves resource utilization for long rollouts under a token budget by dynamically partitioning them, thereby obtaining high time efficiency; and (3) ASystem, a high-performance RL framework designed to overcome the systemic bottlenecks that impede trillion-parameter model training. Ring-1T delivers breakthrough results across critical benchmarks: 93.4 on AIME-2025, 86.72 on HMMT-2025, 2088 on CodeForces, and 55.94 on ARC-AGI-1. Notably, it attains a silver medal-level result on the IMO-2025, underscoring its exceptional reasoning capabilities. By releasing the complete 1T parameter MoE model to the community, we provide the research community with direct access to cutting-edge reasoning capabilities. This contribution marks a significant milestone in democratizing large-scale reasoning intelligence and establishes a new baseline for open-source model performance.

1 Introduction

Ring-1T addresses the unexplored challenges of trillion-parameter open-source reasoning models with innovations for RL stability, rollout efficiency, and system scalability. It achieves leading performance across demanding reasoning benchmarks.

  • Ring-1T targets the uncharted frontier of trillion-parameter thinking models, where training instability and prohibitive computational costs remain central challenges.The model is presented as an open-source Mixture-of-Experts system scaled to one trillion total parameters.
  • 93.4 on AIME-2025, 86.72 on HMMT-2025, 2088 on CodeForces, and 55.94 on ARC-AGI-v1 establish Ring-1T’s state-of-the-art performance across challenging benchmarks.The model also achieved a silver medal-level IMO-2025 result by correctly solving four problems and partially proving a fifth without code generation or external symbolic solvers.
  • IcePop stabilizes RL by clipping tokens with excessive training-inference discrepancies while preserving efficient updates and inference speed.The method selectively removes unstable contributions rather than slowing inference.
  • C3PO++ improves rollout efficiency through budget-controlled scheduling, avoiding inefficient single-pass processing of oversized sequences and reusing them through batched continuation.Its design addresses rollout-stage bottlenecks under long-sequence processing.
  • ASystem provides asynchronous RL infrastructure with SingleController + SPMD operations, multi-phase masking acceleration, and efficient data packing and sharding.The framework is designed for large-scale asynchronous training.
  • The paper evaluates the model against leading open- and closed-weight systems after describing long-CoT SFT, large-scale RL, and the three core system contributions.The stated paper structure separates training methodology from benchmark evaluation.

2 Approach

Ring-1T combines a three-stage training pipeline with IcePop, C3PO++, and ASystem to address instability, rollout inefficiency, and infrastructure bottlenecks in trillion-parameter RL.

  • Training pipeline: Ring-1T trains through long-CoT SFT, reasoning-oriented RL with verifiable rewards, and general RL with human feedback.The stages target reasoning ability first, then broader alignment, instruction following, safety, and usability.
  • IcePop: IcePop suppresses unstable updates by calibrating gradients within an acceptable discrepancy region and discarding noisy updates beyond it.The method addresses compounding training-inference mismatch in MoE RL through double-sided masking calibration.
  • IcePop: On Ring-1T, IcePop largely mitigates probability discrepancy and gradient-norm instability between inference and training engines.The reported training dynamics compare original GRPO before and after applying IcePop.
  • C3PO++: C3PO++ manages long rollouts under a token budget by retaining unfinished rollouts across policy updates, yielding approximately 2.5× rollout and 1.5× end-to-end speedups per step.Its reward remains close to baseline and benchmark performance is reported as on par with baseline performance.
  • ASystem: ASystem supplies large-scale RL infrastructure through memory management, weight synchronization, hybrid runtime, and sandbox execution components.AMem reduces memory bottlenecks, AState synchronizes trillion-parameter models in under 10 seconds, and Hybrid Runtime unifies scalable training-inference execution.

3 Evaluation

Ring-1T is evaluated across eight domains against leading open- and closed-weight models under standardized context and hyperparameter settings. It achieves leading results in mathematics, coding, logical reasoning, alignment, and healthcare, including top or near-top benchmark scores.

  • Mathematical Reasoning: 93.40% on AIME 25 and 86.72% on HMMT 25 place Ring-1T second overall and first among open-weights models.It also scores 82.63% on Omni-MATH and 88.54% on CNMO 2024.
  • Mathematical Reasoning: On IMO 2025, Ring-1T solved Problems 1, 3, 4, and 5 on its first attempt, corresponding to silver-medal-level performance.It produced a nearly complete proof for Problem 2 on its third attempt but matched an incorrect answer on Problem 6.
  • Coding Capabilities: 2088 on CodeForces is the highest score among all compared models, while 78.30% on LiveCodeBench-v6 leads the reported coding results.Ring-1T exceeds DeepSeek-V3.1 by 2.97 points and Qwen3-235B-A22B-Thinking-2507 by 2.58 points on LiveCodeBench-v6.
  • Logical Reasoning: 55.94% on ARC-AGI-1 ranks second overall, behind GPT-5-Thinking at 65.70%, and exceeds DeepSeek-V3.1 by 15.32 points.The score also exceeds Qwen3-235B-A22B-Thinking-2507 by 7.82 points.
  • Human Alignment: 81.59% on ArenaHard v2 ranks second overall, while an Elo rating of 84.52 leads all models and Creative Writing v3 reaches 85.40%.The results are reported as evidence of balancing human-preference alignment with broad capabilities.
  • Healthcare Capabilities: 57.93% on HealthBench ranks second overall and leads the open-source models.The evaluation covers eight domains, including knowledge, coding, mathematics, reasoning, alignment, healthcare, multi-turn, and agent tasks.

4 Conclusion

Ring-1T is presented as the first open-weights trillion-parameter thinking model, enabled by three innovations addressing RL stability, rollout efficiency, and system scalability. The resulting model demonstrates state-of-the-art performance across mathematical reasoning, competitive programming, and general intelligence benchmarks.

  • 4 Conclusion: Ring-1T combines IcePop, C3PO++, and ASystem to address training-inference mismatch, long-trajectory rollout efficiency, and trillion-parameter RL scalability.Together, these innovations enabled stable and efficient training at the trillion-parameter scale.
  • 4 Conclusion: The reported benchmark results demonstrate breakthrough performance across mathematical reasoning, competitive programming, and general intelligence.The conclusion states that trillion-parameter reasoning models are feasible and capable.

5 Limitations & Future Work

Ring-1T’s limitations concern inference efficiency, imperfect training-inference consistency, and under-optimized agentic capabilities. The authors identify these constraints as directions for future work.

  • Future Work: The paper frames these architecture, consistency, and capability constraints as fruitful directions for future research.The limitation discussion explicitly presents the model and training systems as having several remaining limitations.
  • Model Architecture & Inference Efficiency: Inference remains costly because extensive internal thought generation makes GQA’s computational overhead non-trivial.The authors propose exploring MoBA and advanced linear-attention variants for higher throughput.
  • Training-Inference Consistency: IcePop mitigates major training-inference mismatch but does not achieve perfect consistency, leaving numerical operator discrepancies as a latent instability source.The authors identify resolving this systems challenge as important for stable future scaling.
  • Capability Deficiencies: Ring-1T’s training strategy prioritizes foundational natural-language reasoning, leaving advanced agentic skills such as tool use under-optimized.Future work will explore specialized data and agentic reinforcement learning.

6 Contributors

The contributor section lists the paper’s authors and identifies corresponding authors with a dagger marker. The authors are presented alphabetically by first name.

  • Contributors: Authors are listed alphabetically by first name.
  • Contributors: The paper provides an extensive contributor list spanning the Ling Team.The names are distributed across multiple consecutive entries.
  • Contributors: A dagger symbol identifies corresponding authors.

A.3 Reinforcement Learning Infrastructure

The paper motivates ASystem by identifying memory, synchronization, and orchestration bottlenecks in trillion-parameter RL infrastructure. ASystem addresses these challenges through live memory offloading and more efficient distributed state movement.

  • Infrastructure Challenges: Trillion-parameter RL infrastructure must address memory efficiency, state synchronization, and system orchestration and reproducibility.These three challenges motivate the design of ASystem.
  • Memory Efficiency: Existing frameworks retain model states and communication groups in GPU memory, causing static and inefficient memory usage.The cited systems include vLLM, SGLang, Megatron-LM, VeRL, and OpenRLHF.
  • State Synchronization: Distributed checkpoint sharing can suffer synchronization latencies of tens of minutes because of limited bandwidth and throughput.The paper contrasts early file-system-based sharing with more recent synchronization systems.
  • System Orchestration & Reproducibility: Existing tightly coupled frameworks make backend integration costly and slow, while distributed rollout behavior remains non-deterministic.The paper attributes non-determinism to fluctuating batch sizes and non-associative floating-point arithmetic.
  • ASystem: ASystem uses AMem for live memory offloading and AState for zero-redundancy peer-to-peer state transmission with hardware-aware communication.

B Theoretical Analysis for IcePop

The theoretical analysis introduces a theorem concerning compounding probability discrepancy between inference and training policies. It defines the two policies as π_infer(·; θ) and π_train(·; θ).

  • Theorem 2: Theorem 2 is titled “Compounding probability discrepancy.”
  • Policy Definitions: The theorem distinguishes the policy loaded by the inference engine from the policy loaded by the training engine.
  • Notation: The analysis denotes the inference and training policies as π_infer(·; θ) and π_train(·; θ).

Consider the update with a step size of µ

Under assumptions of smoothness, bias alignment, and bounded gradients, the update analysis derives a stepsize-dependent bound for the iterates.

  • Consider the update with a step size of µ: The analysis first assumes a neighborhood containing the iterates {θ_t} and then establishes the first claim from the resulting bound.The displayed derivation includes an intermediate result identified as giving the first claim.
  • Consider the update with a step size of µ: The result applies for any stepsize µ ∈(0, ¯µ] chosen so assumptions (A1)–(A4) hold throughout the trajectory.The assumptions include L-smoothness, bias alignment, and a local gradient bound.
  • Consider the update with a step size of µ: The proof combines the update inequalities with a quadratic-term bound using ∥g_t∥≤G and ¯µ.The quadratic contribution is bounded by −L 2 ¯µ µ G2 before the inequalities are combined.
  • Consider the update with a step size of µ: The compounding form is obtained by rewriting the update and applying the stated assumptions.The proof explicitly introduces a rewrite before using assumptions (A2)–(A3).

C Preliminary Analysis for IcePop on Ring-mini-2.0

IcePop is analyzed as a stabilization mechanism for RL training, probability mismatch, exploration, and ill-conditioned tokens, with masking-range sensitivity affecting the dynamics.

  • Training Stability: Both IcePop and TIS avoid the baseline’s rapid training crashes within 600 gradient steps, while maintaining stable reward and gradient dynamics.The baseline reward collapses after 180–200 steps and its gradient norm explodes; IcePop and TIS remain stable.
  • Probability Discrepancy: IcePop keeps training-inference probability discrepancy relatively low, decreasing within 400 steps while avoiding the baseline’s rapid divergence.The comparison concerns KL divergence and maximum probability difference during training.
  • Exploration Space: IcePop maintains lower token log probabilities than TIS, supporting broader exploration because low-probability tokens are more likely to be selected.The passage links lower log probabilities with avoiding overconfident predictions and increasing response diversity.
  • Ill-conditioned Tokens: The masking mechanism clips about 1–2‰ of training tokens, and clipped tokens have higher entropy than the full token population.The clipping ratio rises as training progresses, indicating increasingly subtle harmful updates.
  • Sensitivity Analysis: The default masking range [0.5, 5.0] stabilizes training and enriches sampling diversity, whereas [0.5, 2.0] destabilizes training.The wider range [0.4, 5.0] also stabilizes training but includes higher-log-probability tokens than the default.

D Training Data Analysis

The training-data analysis characterizes the diversity of SFT domains and the complexity distribution of RL data.

  • D Training Data Analysis: SFT data are analyzed by domain distribution to illustrate dataset diversity supporting the model’s broad knowledge base.The domain distribution is presented in Figure 14.
  • D Training Data Analysis: RL data are analyzed by difficulty distribution to characterize the complexity of the reinforcement-learning dataset.The complexity distribution is presented in Figure 15.

E IMO Case

The IMO case studies Ring-1T’s silver-medal-level result through a geometry problem whose feasible numbers of sunny lines are characterized as 0, 1, and 3.

  • E IMO Case: Ring-1T achieved a silver-medal performance at IMO 2025, and the case presents a detailed solution to the competition’s first problem.The solution is described as generated by Ring-1T.
  • E IMO Case: The geometry problem asks which nonnegative k allow n distinct lines covering a triangular grid with exactly k sunny lines.Sunny lines are those not parallel to the x-axis, y-axis, or x + y = 0.
  • E IMO Case: Even k ≥2 are impossible because the uncovered-point requirement exceeds the maximum coverage supplied by k sunny lines.The coverage deficit is stated as the reason for excluding even k ≥2.
  • E IMO Case: Odd k ≥5 are impossible because geometric constraints leave points that the remaining sunny lines cannot cover.For k = 2t −1 with t ≥3, the uncovered remainder exceeds the maximum attainable coverage.
  • E IMO Case: Explicit constructions establish k = 0, 1, and 3 using diagonal, horizontal-plus-sunny, and horizontal-plus-three-sunny-line configurations.The k = 3 construction covers a P3 subgrid with three sunny lines.
  • E IMO Case: The complete characterization is k = 0, k = 1, and k = 3; all other values are infeasible.Even k ≥2, odd k ≥5, and k > n are ruled out.
Loading 2510.18855v2…