PaperlayerView exact revision on arXiv

Source-linked AI summary

Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model

Ling Team, Anqi Shen, Baihui Li, Bin Hu, Bin Jing, Cai Chen, Chao Huang, Chao Zhang, Chaokun Yang, Cheng Lin, Chengyao Wen, Congqi Li, Deng Zhao, Dingbo Yuan, Donghai You, Fagui Mao, Fanzhuang Meng, Feng Xu, Guojie Li, Guowei Wang, Hao Dai, Haonan Zheng, Hong Liu, Jia Guo, Jiaming Liu, Jian Liu, Jianhao Fu, Jiannan Shi, Jianwen Wang, Jianxin Lai, Jin Yang, Jun Mei, Jun Zhou, Junbo Zhao, Junping Zhao, Kuan Xu, Le Su, Lei Chen, Li Tang, Liang Jiang, Liangcheng Fu, Lianhao Xu, Linfeng Shi, Lisha Liao, Longfei Zheng, Meng Li, Mingchun Chen, Qi Zuo, Qiang Cheng, Qianggang Cao, Qitao Shi, Quanrui Guo, Senlin Zhu, Shaofei Wang, Shaomian Zheng, Shuaicheng Li, Shuwei Gu, Siba Chen, Tao Wu, Tao Zhang, Tianyu Zhang, Tianyu Zhou, Tiwei Bie, Tongkai Yang, Wang Hong, Wang Ren, Weihua Chen, Wenbo Yu, Wengang Zheng, Xiangchun Wang, Xiaodong Yan, Xiaopei Wan, Xin Zhao, Xinyu Kong, Xinyu Tang, Xudong Han, Xudong Wang, Xuemin Yang, Xueyu Hu, Yalin Zhang, Yan Sun, Yicheng Shan, Yilong Wang, Yingying Xu, Yongkang Liu, Yongzhen Guo, Yuanyuan Wang, Yuchen Yan, Yuefan Wang, Yuhong Guo, Zehuan Li, Zhankai Xu, Zhe Li, Zhenduo Zhang, Zhengke Gui, Zhenxuan Pan, Zhenyu Huang, Zhenzhong Lan, Zhiqiang Ding, Zhiqiang Zhang, Zhixun Li, Zhizhen Liu, Zihao Wang, Zujie Wen

arXiv:2510.18855v2cs.CLcs.AI

TL;DR

Scaling reinforcement learning to trillion-parameter models presents stability and efficiency challenges. Ring-1T combines IcePop, C3PO++, and ASystem to train an open-source trillion-parameter thinking model, reporting state-of-the-art performance across challenging benchmarks and a silver medal-level result on IMO-2025.

  • Problem

    Scaling reinforcement learning to trillion-parameter models involves efficiency and stability challenges, including probability discrepancies that can destabilize MoE training, especially over long reasoning traces.

    Source: §2.3 · p. 4, §2.3.2 · p. 6

  • Method

    Ring-1T combines IcePop to stabilize mismatched RL updates, C3PO++ to schedule long rollouts under token budgets, and ASystem infrastructure for trillion-parameter RL.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    Ring-1T reports state-of-the-art performance across challenging benchmarks, including 93.4 on AIME-2025, and a silver medal-level result on IMO-2025.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work demonstrates stable and efficient RL training at trillion-parameter scale while presenting an open-weights thinking model.

    Source: §4 · p. 17

    IcePop mitigates major training-inference mismatch but does not achieve perfect consistency, leaving numerical operator discrepancies as a latent source of instability.

    Source: §5 · p. 17

Abstract

from arXiv · show

We present Ring-1T, the first open-source, state-of-the-art thinking model with a trillion-scale parameter. It features 1 trillion total parameters and activates approximately 50 billion per token. Training such models at a trillion-parameter scale introduces unprecedented challenges, including train-inference misalignment, inefficiencies in rollout processing, and bottlenecks in the RL system. To address these, we pioneer three interconnected innovations: (1) IcePop stabilizes RL training via token-level discrepancy masking and clipping, resolving instability from training-inference mismatches; (2) C3PO++ improves resource utilization for long rollouts under a token budget by dynamically partitioning them, thereby obtaining high time efficiency; and (3) ASystem, a high-performance RL framework designed to overcome the systemic bottlenecks that impede trillion-parameter model training. Ring-1T delivers breakthrough results across critical benchmarks: 93.4 on AIME-2025, 86.72 on HMMT-2025, 2088 on CodeForces, and 55.94 on ARC-AGI-1. Notably, it attains a silver medal-level result on the IMO-2025, underscoring its exceptional reasoning capabilities. By releasing the complete 1T parameter MoE model to the community, we provide the research community with direct access to cutting-edge reasoning capabilities. This contribution marks a significant milestone in democratizing large-scale reasoning intelligence and establishes a new baseline for open-source model performance.

1 Introduction

Ring-1T is an open-source, trillion-parameter MoE thinking model introduced alongside methods for addressing instability and efficiency challenges in trillion-scale RL training. The paper presents IcePop, C3PO++, and ASystem as interconnected training innovations.

  • With 1T total and approximately 50B activated parameters, Ring-1T scores 93.4 on AIME-2025, 86.72 on HMMT-2025, 2088 on CodeForces, 55.94 on ARC-AGI-v1, and silver medal-level on IMO-2025.The IMO-2025 result came from one submission without code generation or external symbolic solvers.Source: §1 · p. 2
  • IcePop stabilizes RL training by clipping tokens with excessive training-inference discrepancy, removing unstable contributions while preserving efficient updates without slowing inference.Source: §1 · p. 2
  • C3PO++ uses budget-controlled rollout scheduling to avoid single-pass processing of oversized sequences and enable efficient reuse through batched continuation.Source: §1 · p. 2
  • ASystem supports large-scale asynchronous RL training with SingleController + SPMD, fully asynchronous operations, multi-phase masking acceleration, and efficient data packing and sharding.Source: §1 · pp. 2–3

2 Approach

This section outlines Ring-1T’s training approach, from long-CoT SFT through reasoning-oriented and general RL, and describes the mechanisms and framework supporting it: IcePop, C3PO++, and ASystem. It covers the RL execution stack’s training-inference coordination, isolated task feedback, and algorithm development, as well as reported training stability and rollout-efficiency improvements with comparable benchmark performance.

2.1 Training Pipeline

  • The Ling-1T-base model, with 1 trillion total and 50 billion activated parameters, undergoes long-CoT SFT, reasoning RL, and general RL to develop reasoning and human-alignment capabilities.Long-CoT SFT uses multi-domain trajectories; reasoning RL applies verifiable rewards across math, code, science, and logic; general RL uses RLHF to enhance alignment, instruction following, creative writing, safety, and usability while preserving reasoning strength.Source: §2.1 · pp. 3–4

2.2 Long-CoT SFT

  • Long-CoT SFT builds coherent multi-step reasoning from a collected and filtered dataset spanning mathematics (46%), STEM (26%), code (20%), and others (8%).The data were sourced from open repositories, expert generation, and LLM synthesis, then deduplicated, filtered for harmful content and benchmark overlap, and screened for low-quality samples; training used 64k-token sequences for 3 epochs at a learning rate of 2 × 10^-4.Source: §2.2 · p. 4

2.3.1 RL Data

  • The RL dataset spans math, code, science, logic, and general tasks, with multi-stage curation, dual LLM and rule-based scoring, and fine-grained metadata for dynamic sampling and cross-domain blending.These methods are intended to improve training efficiency and performance on complex tasks.Source: §2.3.1 · p. 5

2.3.2 IcePop: Discard All Noisy Gradient Updates

  • IcePop addresses compounding train-inference probability discrepancies in MoE and long-CoT RL with double-sided token-level calibration and masking that discard updates outside an acceptable region.For reasoning RL, the recipe sets IcePop α = 0.5 and β = 5, with 480 prompts per step and 8 rollouts per prompt; general RL uses GRPO with 80 questions per step and 8 outputs each.Source: §2.3.2 · p. 6, §2.3.4 · pp. 8–9

2.3.3 C3PO++: Dynamically Partition Rollouts with Budget

  • C3PO++ uses parallel inference and training pools under a token budget, updates on completed trajectories, resumes unfinished rollouts across policy versions, and purges rollouts exceeding a retention threshold.Dynamic partitioning is designed to prevent resource idleness from individual long rollouts while stabilizing updates and improving rollout efficiency.Source: §2.3.3 · pp. 7–8, Figure 4 · p. 7

2.4.1 IcePop

  • IcePop improved AIME25 (Avg@64) over training, raising the base score (63%) by over 14% and widening its gap over TIS by relative 6%; on Ring-1T, it mitigated GRPO’s rising gradient norms and probability discrepancy.The preliminary comparison used Ring-mini-2.0; the Ring-1T experiment reported that applying IcePop largely mitigated mismatch and stabilized RL training.Source: §2.4.1 · p. 9, Figure 5 · p. 9, Figure 6 · p. 10

2.4.2 C3PO++

  • C3PO++ achieves approximately 2.5 times rollout-phase speedup per step and about 1.5 times end-to-end speedup per step, while maintaining comparable reward dynamics and on-par reasoning-benchmark performance against the baseline.Source: §2.4.2 · pp. 10–11

2.5 Large-Scale RL Infrastructure: ASystem

  • ASystem co-designs distributed RL infrastructure for Ring-1T, combining a hybrid runtime, GPU memory management, zero-redundancy peer-to-peer weight synchronization, and a serverless rollout-validation environment.Its SingleController + SPMD architecture separates control and data flow, supports plug-and-play training, inference, and reward-model backends, and includes fast-fail reporting and automatic recovery; AState synchronizes trillion-parameter models in under 10 seconds.Source: §2.5 · pp. 11–12

2.5.1 Hybrid Runtime: A Unified Training-Inference Execution Environment

  • Hybrid Runtime unifies resource scheduling, parallelism, and execution, while AState and AMem address weight synchronization and GPU memory efficiency.AState uses zero-redundancy shard transfer and hardware-aware RDMA, NCCL, and shared-memory selection for sub-second in-place updates. AMem provides state switching, multipath transfer Shen et al. (2025), and cross-GPU/node memory pooling without model changes or convergence impact.Source: §2.5.1 · pp. 12–13

2.5.2 ASandbox: An On-Demand Serverless Sandbox Engine

  • 100ms startup and 5,000 QPS/200ms throughput support ASandbox’s rapid, scalable execution of isolated RL tasks.The serverless engine offers specialized math, code, STEM, and terminal sandboxes, with kernel-level isolation and automatic node-failure detection and isolation.Source: §2.5.2 · p. 13

2.5.3 AReaL: A High-Performance RL Algorithm Framework

  • AReaL is an open-source RL framework (Fu et al., 2025) with single-controller and SPMD APIs and plugins for flexible algorithm development.Its asynchronous pipeline overlaps trajectory generation, reward computation, and training; packing and sharding reduce padding and rebalancing overhead, automated recovery handles failures, and control/data-plane separation avoids a single-controller bottleneck.Source: §2.5.3 · p. 13

3 Evaluation

Ring-1T is evaluated across eight benchmark domains under standardized conditions, with results highlighting strong performance in mathematics, coding, logical reasoning, human alignment, and healthcare. Its evaluation includes an IMO-2025 silver-medal-level result using natural-language reasoning.

3.1 Benchmarks

  • The evaluation covers eight domains: knowledge, coding, math, reasoning, alignment, healthcare, multi-turn, and agent tasks.The models compared include leading open-weights models and proprietary API models.Source: §3.1 · p. 14

3.2 Evaluation Settings

  • Benchmarks use 128K, 64K, or 32K context windows depending on the task, with YaRN extending context for models with insufficient native context; baseline settings follow official or vendor recommendations.For baselines with officially reported results, the higher score between the reproduction and official release is reported.Source: §3.2 · p. 14

3.3 Results

  • 93.40% on AIME 25 and 86.72% on HMMT 25 earn Ring-1T the second-highest overall rank and the lead among open-weights models; it also scores 82.63% on Omni-MATH and 88.54% on CNMO 2024.The paper attributes its competition-level math results to a stable, efficient RL training recipe and a diverse, high-quality math training dataset.Source: §3.3 · pp. 14, 16, Table 1 · p. 15
  • Ring-1T solved IMO-2025 Problems 1, 3, 4, and 5 on its first attempt, corresponding to silver-medal level, and produced a nearly complete geometric proof for Problem 2 on its third attempt.For Problem 6, it converged to the same incorrect answer, 4048, as Gemini 2.5 Pro; the correct answer is 2112.Source: §3.3 · p. 16
  • 78.30% on LiveCodeBench-v6 (2408-2505) exceeds DeepSeek-V3.1 by 2.97 points and Qwen3-235B-A22B-Thinking-2507 by 2.58 percentage points; Ring-1T scores 2088 on CodeForces, the highest score among all models.The paper attributes its programming performance to a carefully synthesized dataset.Source: §3.3 · p. 16
  • Using carefully selected logical games from multiple domains, Ring-1T scores 55.94% on ARC-AGI-1, ranking second overall behind GPT-5-Thinking (65.70%) and 15.32 percentage points above DeepSeek-V3.1 (40.62%).Source: §3.3 · p. 16
  • 81.59% on ArenaHard v2 ranks Ring-1T second overall, 1.32 percentage points behind GPT-5-Thinking; its 84.52 Elo rating leads all models, and its 85.40% Creative Writing v3 score is within 0.1 percentage points of the leading open-source model.The paper says a general RL training stage supports strong performance on general tasks alongside reasoning RL training.Source: §3.3 · p. 16
  • 57.93% on HealthBench ranks Ring-1T second overall and leads the open-source models.The paper interprets this result as indicating proficient clinical knowledge integration and potential viability for complex healthcare tasks.Source: §3.3 · p. 16

4 Conclusion

Ring-1T is presented as the first open-weights 1-trillion-parameter thinking model, enabled by innovations for training stability, efficient long-trajectory rollouts, and scalable RL. Its benchmark performance demonstrates the feasibility and capability of trillion-parameter reasoning models.

4 Conclusion

  • Ring-1T demonstrated state-of-the-art performance across benchmarks in mathematical reasoning, competitive programming, and general intelligence.The authors state that these results validate their approach and systems.Source: §4 · p. 17
  • The work combines IcePop for training-inference mismatches, C3PO++ for efficient long-trajectory rollouts, and ASystem for scalability bottlenecks and training stability.Together, the three innovations enabled stable and efficient training of Ring-1T.Source: §4 · p. 17
  • Ring-1T is presented as the first open-weights 1-trillion-parameter thinking model, showing that trillion-parameter reasoning models are feasible and can exhibit exceptional capability.The paper describes the project as addressing system and algorithmic challenges in scaling reinforcement learning to the trillion-parameter regime.Source: §4 · p. 17

5 Limitations & Future Work

Ring-1T’s limitations include costly inference for long thought generation, residual training–inference inconsistency, and under-optimized agentic skills and data quality; the paper identifies corresponding directions for future work.

Model Architecture & Inference Efficiency

  • GQA (Ainslie et al., 2023) balances performance and speed but remains costly for long thought generation; future work will explore MoBA (Lu et al., 2025) or advanced linear attention for higher inference throughput.Source: §5 · p. 17

Training-Inference Consistency

  • IcePop mitigates major training–inference mismatch but does not ensure perfect consistency, as numerical differences between training and inference operators remain a source of instability.Resolving this systems challenge is described as necessary for stable scaling of future models.Source: §5 · p. 17

Capability Deficiencies

  • Ring-1T’s training optimized foundational language reasoning, leaving agentic skills such as tool use under-optimized; future work will add specialized data and agentic RL and address identity confusion and code-switching through refined curation.The paper attributes the latter issues to data impurity and insufficient regularization.Source: §5 · p. 17

Appendix … A.2 Efficient Reinforcement Learning

The appendix contrasts approaches to RL stability under train–inference probability mismatch with methods for improving long-rollout efficiency. It presents C3PO++ as using token-budget-based dynamic cutoff and parallel inference to improve efficiency without sacrificing training performance.

A.1 Stable Reinforcement Learning

  • Different training and inference backends can misalign token-probability calculations and destabilize RL training.GSPO caches old-policy activated experts and replays their routing, but this adds computation and memory overhead.Source: §A.1 · p. 23
  • GSPO uses sequence-level importance ratios and clipping to avoid training crashes caused by high-variance gradients.IcePop’s techniques are independent of sequence-level optimization and can be incorporated into ongoing research efforts.Source: §A.1 · p. 23
  • TIS applies importance-sampling correction to training–inference probability divergence, whereas it moderates gradients for tokens outside IcePop’s masking range.The passage reports that these updates can amplify over training and ultimately lead to a benchmark-performance plateau.Source: §A.1 · p. 23

A.2 Efficient Reinforcement Learning

  • C3PO++ improves inference efficiency without sacrificing training performance, while using parallel high-throughput inference to alleviate rollout bottlenecks.Its design supports compute utilization during training.Source: §A.2 · p. 23
  • C3PO++ dynamically cuts off rollout generation based on a token budget to balance training-update stabilization with inference efficiency.This differs from prior work focused on inference-stage resource utilization.Source: §A.2 · p. 23
  • Long-sequence rollouts consume substantial training resources and time; TPPO (Fan et al., 2025) truncates rollouts while eliminating biased estimation from incomplete trajectories.Prior work (Fu et al., 2025) instead uses high-efficiency generation management for asynchronous RL systems with interruptible rollout workers.Source: §A.2 · p. 23

A.3 Reinforcement Learning Infrastructure

At trillion-parameter scale, reinforcement learning infrastructure faces memory, weight-synchronization, and orchestration and reproducibility challenges; ASystem addresses them with AMem, AState, and a Hybrid Runtime.

Memory Efficiency

  • AMem enables live memory offloading, addressing static GPU residency and Slime’s costly NCCL-group recreation, which can take several minutes at scale.Existing training and inference systems often retain model states and communication groups in GPU memory throughout execution.Source: §A.3 · p. 24

State Synchronization

  • AState uses zero-redundancy P2P transmission and hardware-aware multi-transport communication for sub-second weight synchronization and in-place updates.Prior synchronization approaches had redundant data movement and struggled with numerous small tensors, leaving end-to-end synchronization in the minute range.Source: §A.3 · p. 24

System Orchestration & Reproducibility

  • Hybrid Runtime unifies an RL-specific API across backends and uses Tracker, Analyzer, and Replayer precision alignment for deterministic, reproducible execution.Existing frameworks’ tight component coupling and nondeterministic rollout behavior complicate backend integration, ablations, and root-cause analysis.Source: §A.3 · p. 24

B Theoretical Analysis for IcePop

The section introduces Theorem 2 on compounding probability discrepancy between inference and training policies.

B Theoretical Analysis for IcePop

  • Theorem 2 defines inference and training policies, πinfer(·; θ) and πtrain(·; θ), and introduces their probability discrepancy.The supplied passage ends before stating the discrepancy or theorem result.Source: B Theoretical Analysis for IcePop · p. 24

Consider the update with a step size of µ

Under local assumptions (A1)–(A4), the section analyzes updates with step size µ and derives a one-step claim and a compounding form for sufficiently small steps.

Consider the update with a step size of µ

C Preliminary Analysis for IcePop on Ring-mini-2.0

IcePop’s preliminary analysis on Ring-mini-2.0 finds stable RL training and comparatively controlled training–inference probability discrepancies. It also examines exploration, clipped tokens, and sensitivity to the masking range.

Training Stability

Probability Discrepancy

  • IcePop keeps training–inference KL divergence reasonable, and its maximum probability discrepancy remains relatively low and decreases within 400 steps, unlike the faster-growing TIS and baseline discrepancies.The authors attribute TIS’s larger extreme discrepancies and faster growth probably to noisy policy updates during training.Source: C Preliminary Analysis for IcePop on… · p. 26, Figure 11 · p. 27

Exploration Space

Ill-conditioned Tokens

  • The masking mechanism clips around 1–2‰ of training tokens, with the ratio rising during training; clipped tokens have higher entropy than tokens overall.The analysis suggests increasingly subtle but harmful gradient updates motivate a higher clipping ratio as training progresses.Source: C Preliminary Analysis for IcePop on… · p. 27, Figure 12 · p. 27

Sensitivity Analysis

D Training Data Analysis

The section analyzes SFT domain composition and RL data complexity, using their distributions to characterize the datasets.

E IMO Case

For the IMO 2025 first problem, Ring-1T’s solution characterizes the feasible counts of sunny lines among the n distinct lines covering the specified triangular grid. It reports a silver-medal performance at IMO 2025.

E IMO Case

  • The constructions make k = 0 possible with n diagonal lines and k = 1 with n − 1 horizontal lines plus one sunny line.For k = 0, the diagonal lines x + y = 2, …, n + 1 cover the grid; for k = 1, a sunny line covers the sole point left uncovered by the horizontal lines.Source: E IMO Case · p. 30
  • For k = 3, n − 3 horizontal lines leave a subgrid isomorphic to P3, which three sunny lines partition.The three lines have slopes 1, −1/2, and −2 and cover the six points of P3.Source: E IMO Case · pp. 30–31
  • For even k ≥ 2, the uncovered points outnumber the maximum coverage by k sunny lines, so no configuration exists.The passage states the minimum uncovered count U(k) exceeds the maximum coverage by k sunny lines.Source: E IMO Case · pp. 29–30
  • For odd k ≥ 5, geometric constraints on the triangular grid leave too many points uncovered for the remaining sunny lines to cover.Only three disjoint t-point sunny lines exist for k = 2t − 1, t ≥ 3; the remaining 2t − 4 lines cannot cover the residual 2t(t − 2) points.Source: E IMO Case · pp. 29–30
  • Exactly k = 0, k = 1, and k = 3 are feasible; no other nonnegative values satisfy the problem’s conditions.The characterization applies for n ≥ 3.Source: E IMO Case · pp. 29, 31

Related papers

Loading 2510.18855v2…