PaperlayerView exact revision on arXiv

Source-linked AI summary

INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning

Prime Intellect Team, Sami Jaghouar, Justus Mattern, Jack Min Ong, Jannik Straube, Manveer Basra, Aaron Pazdera, Kushal Thaman, Matthew Di Ferrante, Felix Gabriel, Fares Obeid, Kemal Erdem, Michael Keiblinger, Johannes Hagemann

arXiv:2505.07291v1cs.LGcs.DC

TL;DR

RL reasoning training typically relies on centralized GPU clusters and fast interconnects, leaving globally distributed training as an alternative to explore. INTELLECT-2 trains a 32-billion-parameter model with RL across a permissionless contributor network. The authors report gains on mathematics and coding benchmarks, while noting a slight IFEval drop and broader hardware feasibility from shifting most workload to inference.

  • Problem

    RL training is typically centralized and requires large clusters of colocated GPUs and fast interconnects, motivating exploration of globally distributed compute.

    Source: §1 · p. 3

  • Method

    INTELLECT-2 collaboratively trains a 32-billion-parameter language model with RL across a permissionless, globally distributed contributor network.

    Source: §1 · p. 3

  • Results

    INTELLECT-2 improved QwQ-32B on mathematics and coding benchmarks with a slight IFEval drop, while shifting most workload to inference makes decentralized training feasible across broader hardware.

    Source: §4.2 · p. 18, §5 · p. 19

  • Takeaways & Limitations

    The authors open-source INTELLECT-2, its tasks and verifier environments, and PRIME-RL to support research into decentralized training.

    Source: §6 · p. 19

    The orchestrator and discovery service remain centralized, creating potential single points of failure, limiting horizontal scalability, and introducing trust assumptions that may not suit more distributed or permissionless settings.

    Source: §2.4.3 · p. 10

Abstract

from arXiv · show

We introduce INTELLECT-2, the first globally distributed reinforcement learning (RL) training run of a 32 billion parameter language model. Unlike traditional centralized training efforts, INTELLECT-2 trains a reasoning model using fully asynchronous RL across a dynamic, heterogeneous swarm of permissionless compute contributors. To enable a training run with this unique infrastructure, we built various components from scratch: we introduce PRIME-RL, our training framework purpose-built for distributed asynchronous reinforcement learning, based on top of novel components such as TOPLOC, which verifies rollouts from untrusted inference workers, and SHARDCAST, which efficiently broadcasts policy weights from training nodes to inference workers. Beyond infrastructure components, we propose modifications to the standard GRPO training recipe and data filtering techniques that were crucial to achieve training stability and ensure that our model successfully learned its training objective, thus improving upon QwQ-32B, the state of the art reasoning model in the 32B parameter range. We open-source INTELLECT-2 along with all of our code and data, hoping to encourage and enable more open research in the field of decentralized training.

1 Introduction

Although reasoning models benefit from test-time compute scaling, RL training is typically centralized and depends on colocated GPUs and fast interconnects. INTELLECT-2 demonstrates globally distributed RL training of a 32-billion-parameter model and releases the model and training resources openly.

1 Introduction

  • Test-time compute scaling allows language models to improve by spending more time reasoning, and RL approaches have learned reasoning capabilities in models including DeepSeek-R1, QwQ, and OpenAI’s o1.Source: §1 · p. 3
  • RL training is typically centralized, requiring large clusters of colocated GPUs and fast interconnects, while INTELLECT-2 presents a decentralized alternative.Source: §1 · p. 3
  • INTELLECT-2 is the first large-scale experiment to collaboratively train a 32-billion-parameter language model through RL across a permissionless, globally distributed contributor network.The paper frames RL as well suited to decentralized, globally distributed compute because it is inherently more asynchronous.Source: §1 · p. 3
  • The authors open-source the INTELLECT-2 model, tasks, verifier environments, and PRIME-RL framework for globally distributed RL training.Source: §1 · p. 3

2 Training Infrastructure

INTELLECT-2’s infrastructure combines asynchronous RL components to coordinate rollouts, verification, training, and policy distribution across decentralized compute. The setup supports heterogeneous contributors, overlaps communication with ongoing work, and uses consumer-grade GPUs for inference workers.

2 Training Infrastructure

  • PRIME-RL decouples rollout generation, model training, and weight broadcasting so asynchronous training can operate across heterogeneous, unreliable networks.Source: §2 · p. 3
  • SHARDCAST distributes large files through an HTTP-based tree-topology network, and Protocol Testnet aggregates and coordinates global compute resources.Source: §2 · p. 3
  • Inference rollout workers generate reasoning traces, TOPLOC validators verify them, and GRPO training workers aggregate verified data, update the policy, and distribute new weights via SHARDCAST.Source: §2 · p. 3, Figure 1 · p. 4
  • Policy-weight broadcasts overlap with ongoing inference and training, eliminating communication overhead, while contributors can generate rollouts at their own pace on varied hardware.Source: §2 · p. 3
  • Inference workers, which make up most of the compute in this setup, can run on consumer-grade GPUs, and TOPLOC validation is significantly faster than generation.Source: §2 · p. 3

2.1 PRIME-RL: A Framework for Distributed Asynchronous Reinforcement Learning

PRIME-RL separates training and inference into asynchronous processes, allowing trusted training nodes to update the model while decentralized nodes generate rollouts. Its training, inference, and verifier components support this distributed pipeline.

2.1 PRIME-RL: A Framework for Distributed Asynchronous Reinforcement Learning

  • PRIME-RL natively runs training and inference asynchronously, separating trusted centralized model updates from rollouts generated on trustless decentralized nodes.Its components communicate only when exchanging data and checkpoints, removing the need for a centralized orchestrator such as Ray.Source: §2.1 · p. 4

2.1.1 Training

  • Training shards model weights, gradients, and optimizer states across GPUs with PyTorch FSDP2, and computes log-probabilities using the policy at the start of each optimization step.PRIME-RL currently implements GRPO training with auxiliary KL and entropy losses.Source: §2.1.1 · p. 4

2.1.2 Inference

  • Inference workers generate rollouts with vLLM, poll for the step needing rollouts, and can dynamically join or leave without interrupting training.TOPLOC proof construction hashes final hidden states at 32-token intervals and runs asynchronously on the CPU, reducing tokens-per-second throughput by about 1%.Source: §2.1.2 · p. 5

2.1.3 Verifiers

  • PRIME-RL uses symbolic math verifiers and Python unit-test execution for coding tasks; current sandboxing and code sanitization suit simple algorithmic challenges.More complex coding tasks, such as those requiring filesystem access, will need stronger isolation mechanisms.Source: §2.1.3 · p. 5

2.2 SHARDCAST: Efficient Policy Weight Broadcasts

SHARDCAST distributes policy checkpoints from training servers to decentralized inference workers through relays. It pipelines sharded downloads, balances client requests across relays, and verifies assembled checkpoint integrity, while choosing centralized relays over trustless peer-to-peer transfers.

2.2 SHARDCAST: Efficient Policy Weight Broadcasts

  • SHARDCAST streams checkpoint shards through relay servers so inference workers can begin downloading before the full checkpoint is available.Relays retain only the latest five checkpoint versions; outdated-checkpoint rollouts are typically rejected or discarded.Source: §2.2 · pp. 5–6

2.2.1 Rate Limiting & Firewall

  • Relay servers use per-IP rate limits and firewall rules restricted to known active inference nodes to reduce attack exposure and blacklist misbehaving nodes.Source: §2.2.1 · p. 6

2.2.2 Maximizing Client Throughput & Load Balancing

  • Clients sample relays based on expected throughput rather than always choosing the fastest; probabilistic sampling can use multiple connections and outperform greedy selection even without contention.Clients initialize bandwidth and success estimates with dummy-file requests, then update them with an exponential moving average and a healing factor that encourages exploration.Source: §2.2.2 · p. 6

2.2.3 Assembled Model Weights Integrity Checks

  • Inference workers compare the assembled checkpoint’s SHA-256 checksum with the training-node reference, discard mismatches, and attempt another available checkpoint.The implementation uses trusted relays rather than trustless peer-to-peer shard transfers, citing complexity and security risks from exposing worker IP addresses to peers.Source: §2.2.3 · p. 6

2.3 TOPLOC: Enabling Trustless Inference

TOPLOC validates rollouts from trustless inference workers through computation, sampling, and data-sanity checks. The surrounding infrastructure coordinates contributors, schedules work, and processes validated rollouts, while retaining centralized services with stated scalability and deployment limits.

2.3 TOPLOC: Enabling Trustless Inference

  • TOPLOC validators check computation, sampling, and data sanity; the illustrated workflow audits inference commits and rewards valid batches while penalizing invalid ones.Randomly sampling batches can further speed verification, and providers do not know which generations will be checked.Source: §2.3 · p. 6, Figure 3 · p. 7

2.3.1 Computation checks

  • TOPLOC proofs commit to decoding hidden states, which validators reconstruct from prefill to check consistency with the submitted commitments.The proofs are described as robust to GPU nondeterminism and tensor-parallel configurations, and as detecting quantized or malicious model variants.Source: §2.3.1 · p. 7

2.3.2 Sampling checks

  • Validators check valid sequence termination and EOS probability, and inspect logit distributions for signs of improper token sampling.An EOS token must have probability above 0.1; using a smaller generator with the larger model only for prefill can produce a bimodal logit distribution.Source: §2.3.2 · pp. 7–8

2.3.3 Sanity checks

  • Sanity checks prevent sample cherry-picking through deterministic selection and verify reported scalar bounds and Parquet schema.The sampling seed combines the node address, training step, and submission count, and validators reproduce sampling from that seed.Source: §2.3.3 · p. 8

2.4.1 System Architecture

  • The system architecture uses a ledger to track training-run and contributor information, while discovery and orchestration services manage worker metadata and workloads.The discovery service limits worker IP visibility to the orchestrator, reducing denial-of-service risk.Source: §2.4.1 · pp. 8–9

2.4.2 Operational Flows

  • Workers register through discovery and the ledger, then heartbeats let the orchestrator track node health and remove unresponsive workers.The orchestrator can invite workers into the pool after registration; workers may re-register if they return online.Source: §2.4.2 · pp. 9–10
  • Tasks are distributed through a pull-based scheduling model, and validators accept or reject uploaded rollouts before training data loaders read them.Rejected files result in the originating node being slashed and evicted from the compute pool.Source: §2.4.2 · p. 10

2.4.3 Design Trade-offs & Limitations

  • Centralized orchestration and discovery create potential single points of failure and limit horizontal scalability; the worker also excludes Kubernetes deployments requiring Docker access.The authors plan a DHT-based peer-to-peer discovery design and are developing a containerized worker for orchestration platforms.Source: §2.4.3 · p. 10

3 Training Recipe

INTELLECT-2 combines task correctness and response-length rewards to train reasoning in mathematics and coding while allowing a prompted thinking budget. The recipe uses asynchronous RL; in a smaller-model ablation, reward trajectories matched the synchronous baseline through asynchrony levels up to four.

3 Training Recipe

  • The training objective combines mathematics and coding task rewards with a length reward tied to a thinking budget specified in the prompt.The stated goal is reasoning capability in mathematics and coding, with user control over the requested number of thinking tokens.Source: §3.1 · p. 11, §3 · p. 11

3.1 Training Data & Rewards

  • The 285k-task dataset contains 259k mathematics problems and 26k Python algorithmic coding challenges, selected for verifiable solutions.Math sources include NuminaMath-1.5 and Deepscaler; coding tasks were curated for SYNTHETIC-1.Source: §3.1.1 · p. 11

3.1.1 Task Rewards

  • Mathematics and coding responses receive binary task rewards: 1 for correct answers and 0 for incorrect answers, with no partial coding-test credit.The authors withhold partial coding rewards to discourage reward hacking such as memorizing public tests.Source: §3.1.1 · p. 11

3.1.2 Length Rewards

  • A length penalty weighted by α is combined with task reward according to the difference between target and actual response length.Target lengths are sampled from a small discrete set, such as 2,000, 4,000, or 6,000 tokens, rather than a continuous range.Source: §3.1.2 · p. 11

3.2 Asynchronous Reinforcement Learning

  • Decentralized asynchronous training uses rollouts from weights at least two RL steps old when weight broadcasts delay access to the updated policy.The delay depends on broadcast duration; centralized asynchronous training instead collects rollouts from the previous RL step.Source: §3.2 · p. 12, Figure 6 · p. 12
  • At asynchrony levels up to four, the model’s reward trajectory matched the synchronous baseline in the reported ablation.The comparison used a replicated synchronous training run and PRIME-RL asynchronous training on DeepSeek-R1-Distill-Qwen-1.5B.Source: §3.2 · p. 12

3.3 Offline & Online Data Filtering

Filtering training data by difficulty improved reward learning, and online filtering avoids training steps with no group-relative learning signal. The section describes offline and online filtering as complementary parts of the training recipe.

3.3 Offline & Online Data Filtering

  • The filtering recipe combines pre-run dataset filtering with selective choice of training samples from generated rollouts.The authors report that difficulty filtering had a significant impact on training performance in their ablations.Source: §3.3 · p. 12

3.3.1 Offline Data Filtering

  • On the original Deepscaler mathematics dataset, DeepSeek-R1-Distill-Qwen-7B rewards barely improved; filtering excluded problems with pass@8 above 50% or below 12.5%.Figure 8 contrasts the unfiltered trajectory with training on samples where the base model’s pass@8 was between 1 and 4.Source: §3.3.1 · p. 12, Figure 8 · p. 13
  • Filtering out problems that were too easy or too difficult significantly improved rewards and informed INTELLECT-2’s dataset prefiltering.The reported filtered set retained problems with base-model pass@8 between 1 and 4.Source: §3.3.1 · pp. 12–13, Figure 8 · p. 13

3.3.2 Online Data Filtering

  • Online filtering keeps sampling responses until a batch has non-zero advantages, avoiding training steps with no group-relative signal.With binary rewards, if every completion for a problem receives the same reward, all group-based advantages are zero; continued sampling also increases inference demand.Source: §3.3.2 · p. 13

3.4 Two-Sided GRPO Clipping for Increased Training Stability

One-sided probability-ratio clipping in GRPO can leave negative-advantage updates unbounded, contributing to training instability. The proposed two-sided clipping adds an upper bound for that case.

3.4 Two-Sided GRPO Clipping for Increased Training Stability

  • The authors identify loss and gradient-norm spikes, especially at larger model sizes, as instabilities leading to model collapse.They identify one-sided token probability-ratio clipping in GRPO and PPO-like objectives as a major cause.Source: §3.4 · p. 13
  • One-sided clipping can leave negative-advantage ratios unclipped, allowing huge loss values and gradient updates.The stated clipping parameter ε limits the token probability ratio to avoid excessively large losses and gradient updates.Source: §3.4 · p. 14
  • Adding an upper bound δ for negative-advantage token probability ratios significantly stabilized training.The authors set δ above 1+ε to retain large updates away from bad rollouts while avoiding ratios of a hundred or higher.Source: §3.4 · p. 14

3.5 Mitigating Training Instability at Scale

The authors report several training instabilities at larger scales, including growing gradient norms, clip ratios, and entropy loss. They describe clipping and implementation choices that mitigate or delay some failures, while noting remaining stability concerns.

3.5 Mitigating Training Instability at Scale

  • Gradient-norm growth often precedes more severe instability, and aggressive clipping at thresholds as low as 0.05–0.1 delays the growing-gradient phase and potential crashes.The intervention does not eliminate instability, but the authors report it does not significantly impede convergence and extends the viable training period.Source: §3.5 · p. 14
  • The token probability clip ratio rises during training and correlates with gradient-norm growth, tracking logit differences between consecutive optimizer steps.The authors describe this as an additional instability alongside gradient-norm and entropy instabilities.Source: §3.5 · p. 14
  • Entropy loss turns upward after initially decreasing, typically preceding collapse; increasing KL-penalty weight delays collapse but slows learning.The observed entropy-loss increase begins past approximately 150 steps in the cited figure.Source: §3.5 · p. 14, Figure 10 · p. 15
  • Gradient norms and clipping ratios rise across model scales; the 32B models show greater instability, with QwQ-32B diverging earlier than R1-Distill-32b.The cited figure reports escalating gradient norms and clipping ratios on MATH-trained models.Source: Figure 9 · p. 15
  • QwQ training was less stable than DeepSeek-R1-Distill-Qwen-32B training, which the authors hypothesize may relate to QwQ’s prior reinforcement learning with verifiable rewards.Both models are based on the same Qwen 2.5 pretrained model; the proposed explanation is a hypothesis, not a demonstrated cause.Source: §3.5 · p. 15
  • Using torch.compile led to catastrophic collapse regardless of model size, so the authors disabled it at a cost of slightly increased memory usage.They suggest a faulty generated kernel as a likely source of the issue.Source: §3.5 · p. 15

4 Experiments

INTELLECT-2 improves QwQ-32B on mathematics and coding benchmarks, with a slight IFEval drop, in experiments using asynchronous decentralized reinforcement learning. The runs improved task rewards and overlapped communication with computation, but did not achieve precise thinking-budget adherence within the reported timeframe.

4 Experiments

  • The two-week experimental program used a trusted training cluster and validator nodes alongside trustless, community-contributed heterogeneous inference workers.The main runs were TARGET-SHORT, with target lengths 1,000–4,000, and TARGET-LONG, with target lengths 2,000–10,000.Source: §4 · p. 15, §4.2 · p. 16
  • For DeepSeek-R1-Distill-Qwen-1.5B on Deepscaler mathematics, torch.compile led to early instability and reward collapse, while the no-compile baseline remained stable across 1,200 steps.Source: Figure 11 · p. 16

4.1 Experimental Setup

  • The main run used QwQ-32B with modified GRPO, token-level policy loss, and two-step asynchrony.Each rollout step generated 4,096 samples, followed by eight optimizer steps; the setup also used 32K maximum sequence length and separately computed token log probabilities because vLLM values were numerically unstable.Source: §4.1 · p. 16
  • Cross-sample sequence packing reduced padding waste while preserving complete RL samples, enabling scaling beyond 20K+ sequence lengths.The method adapts the attention mask and packs samples into the sequence dimension.Source: §4.1 · p. 16

4.2 Results

  • Task rewards rose during TARGET-SHORT and TARGET-LONG, while length penalties declined more slowly and did not converge during the runs.The runs were too short for precise adherence to the requested thinking budget; the cause and possible effect of stronger length-reward weighting remain open.Source: Figure 12 · p. 17, §4.2 · pp. 17–18
  • The training-to-inference FLOPs ratio averaged 4.5× in both experiments, with more compute spent on decentralized inference workers.The report identifies scaling inference computations, including longer reasoning chains, as an advantage of decentralized RL.Source: §4.2 · p. 17
  • Asynchronous RL overlapped communication with computation; training rollout steps took about 22 minutes in TARGET-SHORT and 21 minutes in TARGET-LONG.Broadcasts averaged 14 minutes, and verified samples sufficient for a batch took about 22 minutes in TARGET-SHORT versus 29 minutes in TARGET-LONG.Source: §4.2 · pp. 16–17
  • INTELLECT-2 increased QwQ-32B performance on mathematics and coding benchmarks, with a slight drop on IFEval.The comparison is reported in Table 1; the supplied caption does not provide individual benchmark values.Source: §4.2 · p. 18, Table 1 · p. 18

5 Discussion: Decentralized Training in the Test-Time-Compute Paradigm

The discussion argues that test-time-compute RL can suit decentralized training because it shifts compute toward inference and reduces coordination needs. It describes asynchronous overlap as a way to hide communication overhead, while noting open questions about larger delays and future scaling.

5 Discussion: Decentralized Training in the Test-Time-Compute Paradigm

  • Test-time-compute scaling shifts compute toward inference and reduces coordination requirements, which the authors argue makes it particularly suited to decentralized training.The shift can enable broader participation from heterogeneous devices.Source: §5 · p. 18
  • Communication remains a primary bottleneck in decentralized training as model sizes increase, even after techniques such as DiLoCo reduce pretraining communication overhead.Source: §5 · p. 18
  • Delaying rollouts can trade off some policy freshness for useful reasoning traces and positive-reward training signals, unlike delaying gradient application.The authors call for further investigation beyond two-step delays; they report that 4–5-step delays could hide multiple blocking stages and improve compute utilization.Source: §5 · pp. 18–19
  • INTELLECT-2 used about a 1:4 training-to-inference compute ratio, and the authors anticipate inference taking a still larger share as test-time reasoning scales.They connect this asymmetry and inference's lower memory requirements to the feasibility of decentralized training across broader hardware.Source: §5 · p. 19
  • As positive rewards become sparser on harder tasks, generating high-quality rollouts can dominate compute because only a small subset contains strong learning signals.The discussion identifies dataset filtering as a key driver of this shift toward inference compute.Source: §5 · p. 19

6 Conclusion & Future Work

The report presents INTELLECT-2 as the first globally distributed RL run of a 32-billion-parameter language model and releases the model and supporting infrastructure. It frames decentralized training as an initial step, with future work spanning inference-heavy RL, tool use, community-built environments, and model merging.

6 Conclusion & Future Work

  • INTELLECT-2 is presented as the first globally distributed reinforcement-learning run of a 32-billion-parameter language model, with the model, tasks, verifier environments, and infrastructure open-sourced.The release includes PRIME-RL.Source: §6 · p. 19
  • The report describes decentralized frontier reasoning training as a first step and identifies increasing inference relative to training compute as future work.It points to inference-heavy methods and online data filtering as areas to explore, if proven effective.Source: §6 · p. 19
  • The authors identify tool calls such as web browsing, code interpreters, and APIs as a direction for scaling open-source reasoning models.Source: §6 · p. 19
  • The report invites contributors to build diverse RL environments, framing their development as a parallelizable software-engineering task suited to community effort.Source: §6 · p. 19
  • Whether model merging extends to reasoning tasks remains open, but merging models trained on distinct reasoning domains could support scaling asynchronous RL across parallel compute resources.Source: §6 · p. 20
Loading 2505.07291v1…