PaperlayerView exact revision on arXiv

Source-linked AI summary

Skywork Open Reasoner 1 Technical Report

Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, Yahui Zhou

arXiv:2505.22312v2cs.LGcs.AIcs.CL

TL;DR

Skywork-OR1 addresses how to improve the reasoning abilities of long Chain-of-Thought models with reinforcement learning efficiently and scalably. It introduces an RL recipe built on DeepSeek-R1-Distill, achieving substantial benchmark gains while showing that mitigating premature entropy collapse is critical for better test performance.

  • Problem

    Long CoT models generate sequences averaging over 10K tokens, but efficient and scalable RL improvement for SFT-trained long CoT models remains unclear.

  • Method

    Skywork-OR1 is an efficient, scalable RL recipe for long CoT models, supported by open-source data, preprocessing, filtering, and adaptive entropy control.

  • Results

    Skywork-OR1 improves reasoning across mathematical and coding benchmarks; its 32B model scores 82.2 on AIME24, 73.3 on AIME25, and 63.0 on LiveCodeBench.

  • Takeaways & Limitations

    Mitigating premature entropy collapse is critical for improved test performance, while adaptive entropy control maintains exploration and learning plasticity during training.

  • Takeaways & Limitations

    Adaptive entropy control is not recommended when NSGD is large, although entropy dynamics remain acceptably stable when NSGD = 1 or 2.

Abstract

from arXiv · show

The success of DeepSeek-R1 underscores the significant role of reinforcement learning (RL) in enhancing the reasoning capabilities of large language models (LLMs). In this work, we present Skywork-OR1, an effective and scalable RL implementation for long Chain-of-Thought (CoT) models. Building on the DeepSeek-R1-Distill model series, our RL approach achieves notable performance gains, increasing average accuracy across AIME24, AIME25, and LiveCodeBench from 57.8% to 72.8% (+15.0%) for the 32B model and from 43.6% to 57.5% (+13.9%) for the 7B model. Our Skywork-OR1-32B model surpasses both DeepSeek-R1 and Qwen3-32B on the AIME24 and AIME25 benchmarks, while achieving comparable results on LiveCodeBench. The Skywork-OR1-7B and Skywork-OR1-Math-7B models demonstrate competitive reasoning capabilities among models of similar size. We perform comprehensive ablation studies on the core components of our training pipeline to validate their effectiveness. Additionally, we thoroughly investigate the phenomenon of entropy collapse, identify key factors affecting entropy dynamics, and demonstrate that mitigating premature entropy collapse is critical for improved test performance. To support community research, we fully open-source our model weights, training code, and training datasets.

1 Introduction

Skywork-OR1 addresses the open question of efficiently improving already-supervised long-CoT models with RL. Its recipe combines data and training choices with entropy-collapse mitigation, yielding stronger reasoning performance and open resources.

  • Motivation: Long-CoT models generate over 10K-token reasoning sequences, yet efficient and scalable RL methods for such SFT models remain unclear.Prior reproduction efforts largely applied RL to base models rather than long-CoT models that had already undergone SFT.
  • Contribution: Skywork-OR1 is an efficient and scalable RL recipe built on the DeepSeek-R1-Distill model series and open-source datasets.The report uses rigorous preprocessing and filtering and evaluates released 7B and 32B models.
  • Results: Skywork-OR1-32B scores 82.2 on AIME24, 73.3 on AIME25, and 63.0 on LiveCodeBench, outperforming DeepSeek-R1 and Qwen3-32B in mathematics.The 7B variants are reported as competitive with similarly sized models.
  • Training pipeline: The report validates data filtering and multi-stage training, finds that truncated-trajectory noise does not improve large-context scaling, and releases code, data, and weights.The authors also state that KL penalties hinder further test-performance improvements during multi-stage training and are omitted.
  • Entropy dynamics: Adaptive entropy control keeps entropy lower-bounded by a target, preserving exploration and learning plasticity while test performance steadily improves.The method adaptively adjusts the entropy-loss coefficient according to current and target entropy.
  • Entropy dynamics: Faster entropy collapse generally correlates with poorer test performance, while appropriate control can improve test outcomes.Higher sampling temperature, on-policy updates, and entropy-loss adjustments are identified as important factors in entropy dynamics.

2 Preliminaries

The preliminaries define policy-gradient RL objectives and the batch-level surrogate used for optimization. They then introduce PPO and GRPO, including trust-region clipping, group-normalized rewards, and response-length normalization.

  • Policy-gradient objective: Policy-gradient RL seeks a policy π that maximizes expected reward over prompts, responses, and a reward function.The reward is defined for responses sampled by the policy from a prompt distribution.
  • Policy-gradient objective: At each step, the surrogate objective samples N prompts and responses from the current policy under context length T and temperature τ.The resulting batch is optimized as a tractable approximation to the policy objective.
  • Vanilla policy gradient: Vanilla policy gradient uses gradient ascent on policy parameters and a first-order surrogate policy loss.The notation includes token sequences, token positions, and prefix contexts.
  • PPO: PPO performs multiple policy-loss updates while clipping the new policy to remain within a trust region of the previous policy.The clip hyperparameter ε controls the restriction, and PPO generally uses GAE in practice.
  • GRPO: GRPO samples M independent responses per prompt, normalizes rewards within each group for token-level advantages, and adds response-length normalization.The formulation also uses a reference-policy trust-region constraint and binary rule-based rewards.

3 MAGIC in Skywork-OR1

Skywork-OR1 uses MAGIC, a modified GRPO training pipeline designed around multi-stage training and adaptive entropy scheduling. The report introduces the recipe before analyzing its components.

  • MAGIC: The Skywork-OR1 training pipeline is built on a modified version of GRPO called Multi-stage Adaptive entropy scheduling for GRPO in Convergence (MAGIC).The subsequent analysis evaluates the effectiveness of MAGIC’s components.

3.1 MAGIC

MAGIC refines GRPO through staged training, selective data collection, and a token-level loss designed to preserve exploration and learning plasticity. Its recipe includes filtering easy or uninformative prompts, high-temperature sampling, on-policy updates, and omitting KL loss.

  • Framework components: MAGIC combines data collection, training-strategy refinements, and a token-level policy loss into its post-training framework.The loss removes response-length normalization and averages policy loss across all tokens in a training batch.
  • Data Collection: Filtering removes prompts that are always correct, always incorrect, or already solved by the actor, keeping training focused on challenging problems.Filtering occurs before training and at the beginning of each stage.
  • Data Collection: Rejection sampling excludes prompt groups with zero advantages because their responses can increase the relative influence of KL or entropy losses and destabilize training.Only groups with non-zero advantages are included in training batches.
  • Training Strategy: Multi-stage training progressively increases context length while reducing computational costs and preserving scalability.The training process is divided into multiple stages.
  • Training Strategy: High-temperature sampling uses τ = 1 to enhance exploration and learning plasticity, avoiding the rapid low-entropy states observed with τ = 0.6.The smaller temperature caused immediate low entropy on math data and rapid transitions on code data.
  • Training Strategy: On-policy updates significantly slow entropy collapse and lead to higher test performance, while the Math-7B setup was not strictly on-policy.Skywork-OR1-7B and Skywork-OR1-32B used on-policy training; Skywork-OR1-Math-7B used two gradient steps per training step.
  • Loss Function: Adaptive entropy control addresses the difficulty of selecting a fixed entropy-loss coefficient because its effects depend strongly on the coefficient and training data.The entropy loss is intended to preserve exploration, learning plasticity, and generalization.
  • Loss Function: MAGIC omits KL loss because the authors found it hindered performance gains, particularly during later stages of multi-stage training.This choice is part of the training recipe.

3.2 Effectiveness of MAGIC Components

The MAGIC ablations identify data quality, multi-stage scheduling, truncation handling, and entropy dynamics as important factors in RL training for long-CoT models. The results support rigorous filtering, multi-stage training, no advantage masking, adaptive entropy control, and removing KL loss in the released pipeline.

  • Data Mixture: Rigorous filtering and quality control accelerate learning, while looser data mixtures can cause performance to degrade sharply after initial gains.The proposed stringent mixture outperforms a baseline with looser thresholds; adding potentially noisy data produces slower early progress.
  • Multi-Stage Training: Multi-stage training reaches the same final AIME24 accuracy with approximately 100 fewer training hours over 1000 steps.An 8K Stage I context reduces response length and inference costs before later stages restore scaling.
  • Multi-Stage Training: Multi-stage training reduces average response length from approximately 12.5K to 5.4K tokens initially while preserving later performance scaling.Stages II and III increase response length alongside performance improvements.
  • Advantage Mask for Truncated Responses: Advantage masking does not improve large-context test-time scaling, whereas no masking maintains accuracy and improves token efficiency.At 32K context, masked strategies leave accuracy unchanged; the final pipeline therefore applies no advantage mask.
  • Entropy Dynamics: Low-temperature sampling can trigger entropy collapse within approximately 100 steps, initially slowing learning and ultimately causing stagnation.Higher-temperature sampling lowers early test accuracy but ultimately yields greater performance improvements.
  • Adaptive Entropy Control: Adaptive Entropy Control maintains entropy at a reasonable level and prevents premature collapse by adjusting the entropy-loss coefficient toward a target.The experiments use tgt-ent = 0.2 and ∆ = 0.005.
  • No KL Loss: Removing KL loss avoids pulling the actor back toward the reference policy, which otherwise drives KL divergence toward zero and limits AIME24 improvement.The released models therefore use β = 0 across training stages.

4 Empirical Studies on Mitigating Policy Entropy Collapse

The study examines premature policy entropy collapse in long-CoT RL and evaluates rollout diversity, policy-update choices, and entropy-control methods. It finds that delaying collapse generally improves test performance, while entropy dynamics depend strongly on update settings, training data, and control parameters.

  • Empirical Results of Our Entropy Collapse Study: Faster entropy collapse generally leads to worse test performance, while appropriate entropy control can improve results.The study links premature policy convergence with degraded test performance and evaluates controls that prevent it.
  • 4.3 The Impact of Rollout-Diversity-Related Hyperparameters: Increasing rollout batch size or group size has only minor effects on entropy dynamics, whereas higher sampling temperature significantly changes initial entropy.Across DR = 16, 32, 64 and gs = 4, 8, 16, on-policy configurations showed no significant entropy differences and no collapse.
  • 4.4 Policy Update: Increasing SGD steps per training step accelerates entropy collapse and generally degrades test performance, partly because it introduces off-policy data.Experiments with NSGD ∈ {2, 4} showed rapid entropy decay and inconsistent test improvement, unlike the gradual decline under an on-policy configuration.
  • 4.5 Preventing Premature Entropy Collapse: Entropy loss is highly sensitive to training data and its coefficient, so adaptive coefficient adjustment or clip-higher can stabilize entropy and improve test performance.Adaptive entropy control kept entropy above a target in Skywork-OR1-Math-7B, while higher-clip ratios such as 0.25 or 0.265 prevented collapse; 0.28 caused sharp entropy growth and poor performance.
  • 4.5 Preventing Premature Entropy Collapse: Entropy-control coefficients can produce opposite failures: larger αk values cause faster entropy increases and model collapse, while αk = 1e-4 yields persistent decay toward zero.The coefficient experiments tested αk values from 1e-4 through 1e-2.

5 Empirical Studies on Training Resource Allocation

The resource-allocation studies ask how to improve long-CoT RL efficiency under fixed resources and how to allocate additional resources for better test performance. Rollout generation dominates training time, making larger rollout budgets more attractive than simply adding SGD steps when resources increase.

  • 5 Empirical Studies on Training Resource Allocation: The study separates resource allocation into improving efficiency with fixed resources and allocating additional resources for test performance or efficiency.The synchronous training time is decomposed into rollout, policy-update, and other operations.
  • 5.1 Fixed Computational Resources: Rollout time tR primarily determines total training time ttotal for long-CoT models.Table 7 analyzes Skywork-OR1-32B over 1000 training steps and reports that rollout time dominates total time.
  • 5.1 Fixed Computational Resources: Increasing SGD steps improves training efficiency with similar total time when rollout batch size is fixed, but the associated experiments report worse performance.With fixed DR, NSGD ∈ {2, 4} raises policy-update time only modestly relative to total training time.
  • 5.2 Additional Computational Resources: As computational resources increase, the reduction in rollout time diminishes because generation becomes dominated by the longest response.This limits the efficiency benefit of further scaling rollout hardware once sufficient resources are available.
  • 5.2 Additional Computational Resources: Increasing rollout batch size or group size with additional resources improves test performance with similar total training time.Figures 25 and 26 report this pattern for DR and gs under matched resource allocations.

6 Dataset Preparation

The dataset pipeline curates verifiable, correct, challenging math and coding problems through deduplication, answer checking, difficulty estimation, and human-LLM quality assessment. The process produces approximately 105K math problems and 13.7K coding questions while addressing incomplete, duplicated, or unreliable examples.

  • Math Data: The math pipeline starts from NuminaMath-1.5 and applies URL or figure removal, cross-dataset deduplication, and AIME24/AIME25 decontamination.These steps yield approximately 105K math problems.
  • Selection Criteria: The pipeline selects verifiable, correct, and challenging problems for both math and coding RL data.Problems lacking verification, correct answers, or useful difficulty are excluded or filtered.
  • Coding Data: The coding pipeline removes corrupted or incomplete tests, verifies solutions against all tests, and deduplicates by embedding similarity.The final coding set contains 13.7K questions: 2.7K from LeetCode and 11K from TACO.
  • Difficulty Estimation: Difficulty estimation uses correct-solution percentages from N=16 math rollouts and N=8 coding rollouts to avoid zero-advantage GRPO groups.The rollouts use temperature 1.0 and a maximum token length of 32K.
  • Quality Assessment: Human and LLM inspection removes problems with unclear wording, incomplete information, poor formatting, or distracting content.Some invalid or incomplete problems passed model-based difficulty estimation because models solved them once or recognized similar examples.

7 Math & Code Verifiers

The paper evaluates math verifiers and a secure code sandbox for generating reliable RL rewards. It finds important verifier and sandbox failure modes, including parsing errors, false classifications, execution instability, and unsupported multiple-output tasks.

  • 7.1 Math Verifiers: The verifier study compares the original MATH, DeepScaleR, Qwen2.5, PRIME, and Math-Verify systems using rollout data and difficulty-level distributions.Figure 27 summarizes correct-rollout counts across difficulty levels 0–8 for the retained verifiers.
  • 7.1 Math Verifiers: Qwen2.5 can lose mathematical information during parsing and PRIME can stall, so both were excluded from further analysis.The observed parsing issue includes failure to retain exponents in expressions such as boxed{a^2}.
  • 7.1 Math Verifiers: The original MATH and DeepScaleR verifiers produced higher rates of false positives and false negatives than the evaluated alternatives.The comparison combines verifier outputs with human judgments.
  • 7.1 Math Verifiers: Wrapping the gold answer in boxed{} is crucial because directly parsing the gold answer can alter the mathematical expression.The final reward procedure extracts and parses the generated answer, then applies verification with the boxed gold answer.
  • 7.2 Code Sandboxes: The code sandbox validates syntax, monitors memory, and uses multiprocessing after asynchronous process pools caused crashes and incorrect test results.Processes exceeding 50GB are terminated to prevent resource exhaustion.
  • 7.2 Code Sandboxes: The sandbox more accurately evaluates solution correctness than PRIME but does not handle multiple valid outputs for the same input.This limitation affects nondeterministic or open-ended code-testing scenarios.

8 Experiments

The experiments fine-tune DeepSeek-R1-Distill-based models with curated math and code data, multi-stage training, and standardized benchmark evaluation. Skywork-OR1 models improve substantially over their base SFT models, with the 32B model outperforming strong contemporary models on key math benchmarks.

  • Training Configurations: Skywork-OR1-7B and Skywork-OR1-32B are fine-tuned from DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Qwen-32B, respectively.Training uses math and code problems collected from various sources with preprocessing, difficulty filtering, and quality control.
  • Benchmarks & Baselines: Evaluation covers AIME 2024, AIME 2025, and LiveCodeBench, using maximum generation length 32,768 tokens with avg@32 for AIME and avg@4 for LiveCodeBench.Responses are generated with temperature 1 and top-p 1.
  • Evaluation Results of Skywork-OR1 models: Skywork-OR1-32B outperforms DeepSeek-R1 and Qwen3-32B on AIME24 and AIME25, while Skywork-OR1-7B and Skywork-OR1-Math-7B remain competitive among similarly sized models.Skywork-OR1-7B scores 70.2 on AIME24, 54.6 on AIME25, and 47.6 on LiveCodeBench; Skywork-OR1-Math-7B scores 69.8, 52.3, and 43.6, respectively.

9 Conclusion

The conclusion presents Skywork-OR1 as a scalable RL implementation for long-CoT models and reports strong reasoning results across mathematical and coding benchmarks. It also identifies training components and entropy dynamics associated with performance, while releasing resources for community research.

  • Skywork-OR1 is an effective and scalable RL implementation for enhancing the reasoning capabilities of long-CoT models.
  • Skywork-OR1-32B outperforms DeepSeek-R1 and Qwen3-32B on AIME24 and AIME25 while achieving comparable LiveCodeBench results.The 7B models demonstrate competitive reasoning performance among similarly sized models.
  • Ablations validate data mixture and filtration, multi-stage training without advantage masking, high-temperature sampling, exclusion of KL loss, and adaptive entropy control.
  • Preventing premature entropy collapse is critical for achieving optimal test performance, with key entropy-dynamics factors identified through extensive investigation.
  • The authors release model weights, training code, and training datasets to support community research.
Loading 2505.22312v2…