PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang

arXiv:2501.12948v2cs.CLcs.AIcs.LG

TL;DR

The paper addresses limits from reliance on human-provided exemplars and develops reinforcement-learning-based training for LLM reasoning. It reports emergent self-verification and reflection, strong reasoning capabilities, and distilled smaller models with strong reasoning performance.

  • Problem

    Reliance on provided exemplars can prevent exploration of superior, non-human-like reasoning pathways.

  • Method

    The paper trains reasoning models with reinforcement learning, including a pure-RL stage using Group Relative Policy Optimization without supervised fine-tuning.

  • Results

    Sophisticated reasoning behaviors, including self-verification and reflection, emerged during reinforcement learning, while distilled smaller models showed strong reasoning capabilities.

  • Takeaways & Limitations

    The results indicate that hard reasoning questions, reliable verification, and sufficient computational resources can unlock substantial reasoning potential in pretrained checkpoints.

  • Takeaways & Limitations

    Scaling pure reinforcement learning remains an open challenge for complex tasks that lack reliable reward models.

Abstract

from arXiv · show

General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) and chain-of-thought prompting, have achieved considerable success on foundational reasoning tasks. However, this success is heavily contingent upon extensive human-annotated demonstrations, and models' capabilities are still insufficient for more complex problems. Here we show that the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories. The proposed RL framework facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation. Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and STEM fields, surpassing its counterparts trained via conventional supervised learning on human demonstrations. Moreover, the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models.

1. Introduction

The introduction motivates self-evolving reasoning in LLMs through reinforcement learning with minimal human labeling, while addressing limitations of human-provided reasoning exemplars and DeepSeek-R1-Zero. It also presents distilled smaller models as more accessible resources for studying and deploying long-chain-of-thought reasoning.

  • Human-provided reasoning exemplars can cap model performance by restricting exploration of superior, non-human-like reasoning pathways.
  • The proposed approach develops reasoning through self-evolution with reinforcement learning and minimal human labeling, using DeepSeek-V3-Base and GRPO.Rewards are based solely on final-prediction correctness against ground-truth answers, without constraining the reasoning process.
  • DeepSeek-R1-Zero shows excellent reasoning but suffers from poor readability, language mixing, and limited performance beyond reasoning tasks such as writing and open-domain question answering.The introduction presents DeepSeek-R1 as addressing these limitations.
  • Several distilled smaller models are publicly available to broaden access to powerful AI at lower energy cost.They exhibit strong reasoning capabilities and surpass their original instruction-tuned counterparts, while providing resources for studying long-chain-of-thought reasoning.

2. DeepSeek-R1-Zero

DeepSeek-R1-Zero is trained on DeepSeek-V3 Base using reinforcement learning alone, with GRPO and rule-based accuracy and format rewards instead of supervised fine-tuning. During training, it improves substantially on verifiable reasoning tasks while autonomously developing longer thinking and advanced strategies such as reflection and alternative-solution exploration.

  • 2. DeepSeek-R1-Zero: DeepSeek-R1-Zero is trained exclusively with reinforcement learning on DeepSeek-V3 Base, using GRPO (Shao et al., 2024) without supervised fine-tuning.The training template requires a reasoning process followed by a final answer while imposing no content-specific constraints, enabling observation of the model’s natural RL progression.
  • 2. DeepSeek-R1-Zero: GRPO samples a group of outputs for each question and optimizes the policy using group rewards, advantages, a reference policy, and hyperparameters.The method was designed to simplify PPO (Schulman et al., 2017) and reduce its resource consumption for large-scale RL.
  • 2. DeepSeek-R1-Zero: Rule-based rewards combine equally weighted accuracy and format signals, verifying correctness in mathematics and code while requiring reasoning inside <think> and </think> tags.The approach avoids neural reward models because they can be vulnerable to reward hacking and add computational and pipeline complexity.
  • 2. DeepSeek-R1-Zero: DeepSeek-R1-Zero’s thinking time steadily increases through intrinsic adaptation, with long chain-of-thought responses expanding to hundreds or thousands of tokens.This increased thinking time supports progressively refined reasoning and problem-solving strategies.
  • 2. DeepSeek-R1-Zero: Reinforcement learning autonomously induces reflective reasoning, systematic exploration of alternative solutions, and an “aha moment” during training.These behaviors significantly boost performance on verifiable tasks such as mathematics and coding without explicitly teaching solution procedures.

3. DeepSeek-R1

DeepSeek-R1 addresses DeepSeek-R1-Zero’s readability and language-mixing problems through a pipeline combining conversational cold-start data, reinforcement learning, rejection sampling, and supervised fine-tuning. Its training uses rule-based reasoning rewards, preference-based general rewards, and an added language-consistency reward to improve readability while retaining reasoning and writing capabilities.

  • Pipeline: DeepSeek-R1’s pipeline targets DeepSeek-R1-Zero’s poor readability and language mixing, especially between English and Chinese.These issues arise because DeepSeek-V3-Base is trained on multiple languages.
  • Pipeline: The pipeline begins with thousands of conversational, human-aligned cold-start examples, followed by RL, rejection sampling, and another SFT stage.The final SFT stage combines reasoning and non-reasoning datasets to support both reasoning performance and advanced writing capabilities.
  • Reward Design: The language-consistency reward is added directly to the final reward for both reasoning and non-reasoning data, improving readability despite a slight performance degradation in ablations.The reward is motivated by alignment with human preferences.
  • Reward Modeling: Reasoning training follows DeepSeek-R1-Zero with rule-based rewards for mathematics, coding, and logical reasoning, while general data uses reward models.The general reward is selected from either helpfulness or safety rewards according to each query’s dataset category.
  • RL Training: The second RL stage lowers the sampling temperature to 0.7 because higher temperatures produce incoherent generations, and adds general instruction data and preference-based rewards only during its final 400 steps.The stage lasts 1,700 training steps; more preference-reward training may cause reward hacking.

4. Experiment

The experiments evaluate DeepSeek-R1 across broad knowledge, instruction-following, reasoning, coding, mathematics, and STEM benchmarks. Results indicate that reasoning-oriented reinforcement learning substantially improves advanced reasoning while having limited effect on user-preference benchmarks, and that mixed datasets improve general language and coding performance.

  • Evaluation Setup: The evaluation spans MMLU, MMLU-Redux, MMLU-Pro, C-Eval, CMMLU, IFEval, FRAMES, GPQA Diamond, SimpleQA, C-SimpleQA, SWE-Bench Verified, Aider, LiveCodeBench, Codeforces, CNMO 2024, and the American Invitational Mathematics Examination.The listed benchmarks cover knowledge, instruction following, general reasoning, coding, mathematics, and STEM-related capabilities.
  • Developmental Stages: Across developmental stages, DeepSeek-R1 Dev1 improves instruction following over DeepSeek-R1-Zero but partially degrades reasoning, most notably on AIME, because the cold-start dataset is limited.Table 3 summarizes performance across the developmental stages shown in Figure 2.
  • Results: Reasoning-oriented RL substantially improves performance on code-generation, mathematical-problem-solving, and STEM benchmarks, while producing only marginal gains on general-purpose AlpacaEval 2.0.These results suggest that RL primarily enhances reasoning capabilities rather than user-preference-oriented performance.
  • Developmental Stages: DeepSeek-R1 Dev3 combines reasoning and non-reasoning datasets in SFT, improving general language generation and coding performance over Dev2 on AlpacaEval 2.0 and Aider-Polyglot.The gains are attributed to large-scale non-reasoning corpora and code-engineering datasets.
  • Supplementary Analyses: Supplementary analyses compare DeepSeek-R1 with other models, evaluate safety and fresh test sets, examine mathematical categories and test-time scaling, and assess transfer of reasoning to smaller models.These analyses extend the evaluation beyond the main developmental-stage experiments.

5. Ethics and Safety Statement

DeepSeek-R1’s enhanced reasoning capability introduces ethical risks, including more feasible dangerous content under jailbreak attacks and vulnerability to fine-tuning that compromises safety protections. Comprehensive evaluations find its inherent safety generally moderate, comparable to GPT-4o (2024-05-13).

  • DeepSeek-R1 can be jailbroken to generate dangerous content, including explosive manufacturing plans with improved operational feasibility and executability.
  • The public model is vulnerable to further fine-tuning that could compromise its inherent safety protections.
  • Across multilingual, open-source, in-house, and jailbreak safety evaluations, DeepSeek-R1’s inherent safety is generally moderate, comparable to GPT-4o (2024-05-13).

6. Conclusion, Limitation, and Future Work

DeepSeek-R1-Zero and DeepSeek-R1 use large-scale reinforcement learning to elicit reasoning behaviors, including organically emerging self-verification and reflection, without relying primarily on large-scale human annotation. Despite frontier reasoning results, they remain limited in structured output, tool use, token efficiency, multilingual handling, prompting, software engineering, and reliable reward construction.

  • Conclusion: DeepSeek-R1-Zero and DeepSeek-R1 use large-scale reinforcement learning to incentivize model reasoning, with self-verification and reflection emerging organically.The authors identify hard reasoning questions, reliable verification, and sufficient computational resources as key ingredients for unlocking pretrained checkpoints’ reasoning potential.
  • Capability limitations: DeepSeek-R1 remains suboptimal in structured output and cannot use tools, while also exhibiting inefficient excess reasoning and language mixing outside Chinese and English.The authors expect future RL environments to address structured output and tool use, while further optimization is needed for token efficiency and multilingual behavior.
  • Capability limitations: DeepSeek-R1 is sensitive to prompting, performs worse with few-shot prompts, and shows limited software-engineering gains because lengthy evaluations constrain large-scale reinforcement learning.The recommended setting is zero-shot prompting with a directly stated problem and output format; future work includes rejection sampling and asynchronous evaluation.
  • Methodological limitations: Pure reinforcement learning depends on reliable reward signals, but difficult-to-construct or model-generated reward models can be exploited through reward hacking.For tasks lacking reliable signals, the current method uses human-annotated supervised data and only hundreds of RL steps; the authors hope to develop robust reward models.
  • Future work: Future systems may extend verifier-based reasoning broadly and improve accuracy by integrating tools such as compilers, search engines, and biological or chemical reagents.The proposed tools could retrieve or compute information and validate results in the real world, although reliable reward modeling remains difficult for some tasks.

7. Author List … A.1. DeepSeek-V3

The author list is organized by contribution role, while the appendix describes DeepSeek-V3 and clarifies model naming and training-data distinctions. Core contributors established outcome-based RL, R1-Zero, and related training innovations.

  • 7. Author List: The author list is organized by contribution role, with alphabetical ordering by first name within each category; asterisks mark former team affiliations.The listed categories include core contributors and contributors.
  • 7. Author List: Core authors showed that outcome-based RL induces long Chain-of-Thought patterns and breakthrough reasoning capabilities in LLMs.They contributed equally to creating R1-Zero, whose work laid the foundation for R1.
  • 7. Author List: Junxiao Song proposed and implemented GRPO’s initial version and introduced rule-based rewards for mathematics; Peiyi Wang and Runxin Xu later refined GRPO.Zhibin Gou proposed a large PPO clipping strategy to enhance GRPO performance.
  • A.1. DeepSeek-V3: DeepSeek V3 is an advanced open-source LLM released in December 2024, built on a Mixture-of-Experts architecture with 671 billion total and 37 billion activated parameters per token.It is designed to rival leading models while maintaining cost efficiency and performance.
  • A.1. DeepSeek-V3: DeepSeek-V3-Base training uses plain web pages and e-books without intentionally adding OpenAI-generated synthetic data.Some web pages nevertheless contain many OpenAI-model-generated answers, potentially transferring knowledge indirectly; cooldown data were naturally occurring.
  • A.1. DeepSeek-V3: The paper distinguishes DeepSeek-V3-Base as the base model and DeepSeek-V3 as the instructed model.DeepSeek-R1 and DeepSeek-R1-Zero are trained on DeepSeek-V3-Base, while DeepSeek-R1 also uses non-reasoning data from DeepSeek-V3 SFT data; R1-Dev1 through R1-Dev3 are intermediate checkpoints.

A.2. Conventional Post-Training Paradigm … B.3.2. DeepSeek-R1 Cold Start

The paper contrasts conventional SFT-then-RL post-training with a reasoning-oriented pipeline that uses GRPO, scalable infrastructure, curated RL data, and cold-start reasoning traces. It argues that SFT can constrain exploration, motivating direct or lightly supervised development of reflection and verification strategies.

  • A.2. Conventional Post-Training Paradigm: The conventional post-training paradigm applies supervised fine-tuning before reinforcement learning, combining curated task alignment with reward-based refinement toward broader human-centric objectives.SFT trains on labeled input-output pairs, while RL optimizes outputs against a reward signal such as human preferences in RLHF.
  • A.2. Conventional Post-Training Paradigm: SFT may impede reasoning exploration because human target responses can omit reflection and verification, motivating DeepSeek-R1-Zero to discover reasoning patterns independently of human priors.This design directly addresses the limitation that supervised targets are not always optimal for learning effective reasoning strategies.
  • A.3. A Comparison of GRPO and PPO: GRPO offers a more practical alternative to PPO because it avoids the value model, while PPO can match performance only after additional hyperparameter tuning.GRPO estimates advantages from group scores, whereas PPO relies on a learned value model and GAE; the latter adds memory, computation, and tuning costs.
  • B. Training Details: The RL training system uses a decoupled, extensible architecture with modular rollout, training, inference, and reward components to improve efficiency and scalability for large models.Model instances can be offloaded from VRAM between phases, freeing memory for subsequent modules.
  • B.2. Reward Model Prompt: The framework’s reward-model prompt judges assistant answers against a reference answer for correctness, helpfulness, relevance, concision, and ambiguity handling.The evaluator first generates its own answer, corrects inaccuracies, and then compares the candidate responses.
  • B.3. Data Recipe: The RL data recipe combines mathematics, coding, STEM, and logic reasoning problems with general data intended to improve helpfulness and harmlessness.The mathematics collection contains 26k questions, while the logic collection contains 15k automatically evaluable problems.
  • B.3.1. RL Data: RL training covers mathematics, coding, STEM, and logic tasks, with automatic or binary evaluation used where applicable to provide verifiable rewards.The datasets include diverse mathematical domains and real-world or synthetic logic problems.
  • B.3.2. DeepSeek-R1 Cold Start: DeepSeek-R1’s cold start uses curated long-CoT examples emphasizing first-person, readable reasoning with reflection and verification, followed by filtering and human verification.Thousands of R1-Zero trajectories are sampled at temperature 1.0 and retained when final answers are correct and formatting is readable; code data uses validated adversarial test cases.

B.3.3. 800K Supervised Data … C. Self-Evolution of DeepSeek-R1-Zero

The paper constructs an approximately 800,000-sample supervised dataset combining reasoning and non-reasoning data, then specifies training procedures and documents reward-hacking and language-consistency effects. The supplied passages do not provide substantive evidence for the C. Self-Evolution subsection.

  • B.3.3. 800K Supervised Data: Reasoning data are generated by rejection-sampling trajectories from the first-stage RL checkpoint, with some additional examples judged by a generative reward model.DeepSeek-V3 receives ground-truth answers and model predictions for judgment on data not evaluable by rule-based rewards.
  • B.3.3. 800K Supervised Data: Non-reasoning data reuse the DeepSeek-V3 pipeline and add software-engineering examples, while artificial reasoning traces are human-verified to improve query interpretation and output structure.The traces are intended to clarify format constraints, user intentions, and required response structure.
  • B.3.3. 800K Supervised Data: Approximately 800,000 supervised samples span reasoning and non-reasoning domains, but the predominance of single-turn interactions may limit multi-turn conversational ability.The dataset includes Chinese- and English-language mathematics across varied topics and difficulty levels, alongside writing, factual QA, self-cognition, translation, software engineering, and other data.
  • B.4.1. Hyper-Parameters of DeepSeek-R1-Zero-Qwen-32B: DeepSeek-R1-Zero-Qwen-32B uses 16 sampled outputs per question, 32 unique questions per step, and a batch size of 512, with the reference model replaced every 400 steps.The rollout maximum length is 32,768, with learning rate 2e-6, KL coefficient 0.001, and sampling temperature 1.
  • B.4.2. Hyper-Parameters of SFT; B.4.3. Hyper-Parameters of Distillation: SFT and distillation use curated data for 2–3 epochs with cosine-decay learning rates, 32,768-token context, and batch sizes of 128 for SFT and 64 for distillation.SFT starts at 5 × 10^-5 and decreases to 5 × 10^-6; distillation decreases the initial rate to one-tenth.
  • B.4.4. Training Cost: Training DeepSeek-R1 and R1-Zero used 64*8 H800 GPUs, while the experiments scaled from a promising 30B-parameter model toward 660B models.The supplied passage states that DeepSeek-R1-Zero training required approximately 198 hours.
  • B.5. Reward Hacking: Helpful-reward models can induce reward hacking, in which the LLM achieves high reward while diverging from authentic human intent because of reward-model biases or inaccuracies.The supplied passage identifies this behavior as a limitation of the helpful reward model.
  • B.6. Ablation Study of Language Consistency Reward: Without the Language Consistency reward, language consistency deteriorates during RL; adding it maintains stable consistency, with comparable mathematics performance but slight coding degradation.The comparison is reported for DeepSeek-R1-Distill-Qwen-7B using the same cold-start data as DeepSeek-R1.

C.1. Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training

DeepSeek-R1-Zero rapidly masters simpler MATH problems, while its performance on difficult reasoning tasks improves substantially throughout training. Raw difficulty-level comparisons require caution because the dataset is unevenly distributed.

  • C.1. Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training: 0.55→0.90 accuracy on level-5 MATH problems and 0.78→0.95 on level-4 problems show the largest gains during training.Levels 1–3 quickly reach and maintain 0.90–0.95 accuracy, whereas levels 4–5 improve markedly over time.
  • C.1. Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training: Higher-level problems can slightly exceed level-1 accuracy because level 1 contains only 43 of 500 examples, making 95–97% accuracy reflect just 1–2 unsolved problems.Higher difficulty levels contain approximately 100 questions each, so their percentages are based on larger samples.
  • C.1. Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training: Complex reasoning capability on MATH levels 3–5 improves significantly over time, whereas simpler tasks are mastered early in training.This trend remains meaningful despite nuances in comparing raw accuracy percentages across difficulty levels.

C.2. Evolution of Advanced Reasoning Behaviors in DeepSeek-R1-Zero during Training · D. Evaluation of DeepSeek-R1 · D.1. Experiment Setup

During training, DeepSeek-R1-Zero increasingly exhibits reflective reasoning, with some strategies emerging at distinct development stages. DeepSeek-R1 evaluation spans knowledge, reasoning, mathematics, coding, and software-engineering benchmarks, using decontamination, standardized prompts, strong baselines, and sampling-based estimates.

  • C.2. Evolution of Advanced Reasoning Behaviors in DeepSeek-R1-Zero during Training: Reflective-word frequency rises 5- to 7-fold during training, suggesting reinforcement learning generates longer chains of intermediate reasoning tokens.The tracked words were selected and merged by three human experts.
  • D. Evaluation of DeepSeek-R1: The evaluation covers encyclopedic and long-tail knowledge, Ph.D.-level science, instruction following, mathematics, algorithmic contests, and real-world software engineering.Benchmarks include MMLU-family datasets, SimpleQA, GPQA Diamond, IFEval, AIME, MATH-500, CNMO 2024, LiveCodeBench, Codeforces, SWE-Verified, and Aider.
  • D.1. Experiment Setup: For distilled models, representative results are reported on AIME 2024, MATH-500, GPQA Diamond, Codeforces, and LiveCodeBench.These benchmarks sample mathematics, science, and coding capabilities for distilled-model evaluation.
  • D.1. Experiment Setup: Benchmark contamination is reduced by filtering matching 10-gram sequences from evaluation questions and solutions, but paraphrased test-set contamination may remain for benchmarks released before 2024.DeepSeek-V3’s July 2024 knowledge cutoff predates CNMO 2024, while the authors acknowledge limitations of n-gram filtering.
  • D.1. Experiment Setup: Evaluations compare DeepSeek-R1 with DeepSeek-V3, Claude-Sonnet-3.5-1022, GPT-4o-0513, OpenAI-o1-mini, and OpenAI-o1-1217, plus QwQ-32B-Preview for distilled models.OpenAI-o1-1217 results are taken from official reports because API access is challenging in mainland China.
  • D.1. Experiment Setup: To improve reliability for long-output reasoning, models use a 32,768-token limit and default to pass@k with temperature 0.6, top-p 0.95, and typically 4-64 sampled responses.The setup reports pass@1 under non-zero-temperature sampling rather than relying on greedy decoding, which produced repetition and checkpoint variability.

D.2. Main Results … E.3. Mathematical Capabilities Breakdown by Categories

DeepSeek-R1 achieves strong results across reasoning, knowledge, coding, writing, and human-preference evaluations, while its safety performance varies substantially with risk controls and attack conditions. Further analyses show gains over DeepSeek-V3 and GPT-4o in mathematical and competitive-programming tasks, with category-level strengths in number theory and algebra but weaker geometry and combinatorics.

  • D.2. Main Results: DeepSeek-R1 outperforms DeepSeek-V3 on MMLU, MMLU-Pro, GPQA Diamond, and FRAMES, with especially large reinforcement-learning gains on STEM questions, while instruction-following improvements reflect final-stage SFT and RL data.It also performs strongly on AlpacaEval2.0 and ArenaHard, indicating strengths in writing and open-domain question answering.
  • D.2. Main Results: DeepSeek-R1 matches OpenAI-o1-1217 on mathematics, surpasses other models by a large margin, and performs strongly on coding benchmarks, while OpenAI-o1-1217 leads on Aider but is comparable on SWE Verified.It also outperforms 96.3% of Codeforces participants and exceeds the mean human score on AIME.
  • D.2. Main Results: DeepSeek-R1 shares first place with OpenAI-o1 and Gemini-Exp-1206 on ChatbotArena’s style-control ranking as of January 24, 2025.ChatbotArena ranks models through user preferences from pairwise comparisons.
  • D.3.1. Risk Control System for DeepSeek-R1: The deployed risk-control system filters potentially risky queries and sends flagged dialogues to DeepSeek-V3 for model-based review, significantly improving service safety, particularly against jailbreak attacks.Developers are recommended to implement similar customizable risk controls when deploying DeepSeek-R1.
  • D.3. DeepSeek-R1 Safety Report; D.3.2. R1 Safety Evaluation on Standard Benchmarks: With risk controls, DeepSeek-R1’s safety is comparable to frontier models across standard benchmarks, and it remains strong across discrimination, bias, violence, extremism, and privacy categories.The reported comparisons distinguish pure-model results from results including the risk-control system.
  • D.3.5. Robustness against Jailbreaking: Jailbreak attacks substantially reduce safety for all tested models, while reasoning models show a distinct comparative pattern under attack, confirming that current frontier systems remain vulnerable.Claude-3.7-Sonnet’s safe-response proportion decreases by 33.8% under the reported attacks.
  • E. More Analysis; E.1. Performance Comparison with DeepSeek-V3; E.2. Generalization to Real-World Competitions; E.3. Mathematical Capabilities Breakdown by Categories: Post-training improves DeepSeek-R1 over DeepSeek-V3 in competitive programming, mathematical reasoning, and MMLU/MMLU-Pro domains, while DeepSeek-V3 retains an instruction-following advantage; R1 outperforms GPT-4o 0513 in olympiad tasks and is strongest in number theory and algebra but weaker in geometry and combinatorics.MMLU-Pro gains occur across all domains, especially mathematics and physics, whereas MMLU STEM gains are smaller; R1’s USAMO-related performance places it among top-tier high-school students.

E.4. An Analysis on CoT Length · E.5. Performance of Each Stage on Problems of Varying Difficulty · F. DeepSeek-R1 Distillation

DeepSeek-R1 adaptively scales reasoning computation with problem difficulty, while its distilled outputs transfer strong reasoning performance to much smaller models. Across training stages, improvements are concentrated on medium and hard problems, and distillation enables broader, more efficient access to these capabilities.

  • E.4. An Analysis on CoT Length: DeepSeek-R1 scales computation by generating longer reasoning chains to verify or correct steps, backtrack, and explore alternatives when initial approaches fail.The complexity of a problem directly correlates with the model’s adaptive chain-of-thought length.
  • E.4. An Analysis on CoT Length: 61.8% Pass@1 is achieved with an average of 8,793 thinking tokens per problem, while compute ranges from fewer than 7,000 tokens for simple problems to more than 18,000 for difficult ones.DeepSeek-R1 dynamically adjusts test-time computation on 2024 mathematics competition problems according to difficulty.
  • E.4. An Analysis on CoT Length: Reasoning models retain an advantage over non-reasoning models because self-reflection, backtracking, and alternative exploration enable more effective reasoning-scale computation than independent majority-vote samples.Non-reasoning models repeatedly sample potentially incorrect final solutions without building on prior attempts, making scaling token-inefficient.
  • E.4. An Analysis on CoT Length: DeepSeek-R1’s Pass@64 on AIME 2024 reaches 90.0%, compared with 79.8% Pass@1, showing that independent reasoning-chain sampling can complement extended reasoning.Extended chains can still become trapped in incorrect logic paths, motivating test-time methods such as majority voting or MCTS.
  • E.5. Performance of Each Stage on Problems of Varying Difficulty: Across every DeepSeek-R1 stage, simple LiveCodeBench problems are generally solved correctly, while the main improvements occur on medium and hard problems.The stage-wise results show that each stage significantly improves complex coding-reasoning performance.
  • F. DeepSeek-R1 Distillation: Distillation transfers DeepSeek-R1-generated reasoning to Qwen and LLaMA foundation models using a curated dataset of 800,000 samples.The approach addresses the substantial computational resources required for LLM training and deployment.
  • F. DeepSeek-R1 Distillation: A 1.5-billion-parameter DeepSeek-R1-Distill-Qwen model surpasses non-reasoning baselines on mathematical benchmarks through straightforward output distillation.The distilled models are evaluated on AIME, GPQA, Codeforces, MATH-500, and LiveCodeBench against GPT-4o and Claude-3.5-Sonnet.
  • F. DeepSeek-R1 Distillation: Distillation enables smaller models to achieve strong performance and outperforms reinforcement learning alone for smaller architectures, reducing computational requirements for broader access.The distilled models use supervised fine-tuning without an RL stage, although RL could further improve performance.

F.1. Distillation v.s. Reinforcement Learning · G. Discussion

Large-scale reinforcement learning can produce strong reasoning in smaller base models, but distillation from a more powerful model is more effective and computationally economical. The results also show that reinforcement learning can autonomously develop advanced reasoning strategies without reasoning-trajectory supervision.

  • F.1. Distillation v.s. Reinforcement Learning: Distilled DeepSeek-R1-Distill-Qwen-32B performs significantly better than RL-trained Qwen2.5-32B-Zero across the reported reasoning benchmarks.Qwen2.5-32B-Zero was trained on math, code, and STEM data for over 10K steps, while the comparison appears in Table 16.
  • F.1. Distillation v.s. Reinforcement Learning: Large-scale RL raised Qwen2.5-32B-Base to performance on par with QwQ-32B-Preview.The resulting model was named Qwen2.5-32B-Zero after more than 10K training steps.
  • F.1. Distillation v.s. Reinforcement Learning: Distillation into smaller models yields excellent results, whereas smaller models trained with large-scale RL require enormous computational power and may still underperform distillation.The comparison is presented as a conclusion from the Qwen2.5-32B experiments.
  • G. Discussion: Distillation is both economical and effective, while surpassing human-intelligence boundaries may require more powerful base models and larger-scale RL.This conclusion qualifies the relative efficiency of distillation and the scaling requirements for further progress.
  • F.1. Distillation v.s. Reinforcement Learning: The Qwen2-Math-7B experiment used a base model released before OpenAI-o1, avoiding exposure to reasoning-trajectory data.This experiment was conducted before the launch of the first reasoning model, OpenAI-o1.
  • G. Discussion: The Qwen2-Math-7B-Zero results demonstrate that large-scale RL can autonomously develop advanced reasoning strategies.This finding follows the model’s significant outperformance of non-reasoning baselines.

G.1. Key Findings … H.1. Chain-of-thought Reasoning

The paper finds that pure RL requires sufficiently capable base models and reliable verifiers, while an iterative SFT–RL pipeline enables reasoning trajectories beyond human demonstrations. It also reports limitations of PRM and MCTS approaches and situates the work within chain-of-thought and related prompting methods.

  • G.1. Key Findings: Pure RL produced substantial gains only with larger, more expressive base models, whereas 7B dense and 16B MoE models failed to improve meaningfully on AIME.The authors therefore recommend sufficiently large architectures when validating RL from scratch.
  • G.1. Key Findings: DeepSeek-R1-Zero depends strongly on reliable reward signals, with rule-based reward models and ground-truth-aware LLM evaluation identified as robust defenses against reward hacking.The LLM-based framework is particularly effective for tasks with well-defined correctness criteria.
  • G.1. Key Findings: The proposed iterative pipeline combines SFT and RL: RL explores reasoning trajectories, while SFT remains crucial for capabilities not fully realized through human-annotated traces alone.Without RL, long-chain reasoning patterns required by complex CoT prompting would remain largely unexplored.
  • G.2. Unsuccessful Attempts: Process Reward Models face three practical barriers: defining fine-grained reasoning steps, judging intermediate-step correctness, and automating reliable annotations.These limitations may hinder PRM’s ultimate success in general reasoning (Lightman et al., 2024; Uesato et al., 2022; Wang et al., 2023a).
  • G.2. Unsuccessful Attempts: MCTS training used tagged reasoning steps, value-model-guided answers, and iterative actor–value-model refinement, but scaling is constrained by the search space and value-model dependence.The value model directly influences generation quality because it guides each search step.
  • G.2. Unsuccessful Attempts: MCTS can improve inference with a pre-trained value model, but iteratively improving performance through self-search remains difficult.Token generation has an exponentially larger search space than chess, and node limits can trap the model in local optima.
  • H.1. Chain-of-thought Reasoning: Chain-of-thought reasoning (Wei et al., 2022b) improved arithmetic, commonsense, and symbolic reasoning by eliciting intermediate steps before final answers; its effectiveness scales with model size (Suzgun et al., 2023).Kojima et al. (2022) extended CoT to the zero-shot setting.
  • H.1. Chain-of-thought Reasoning: Subsequent prompting methods expanded CoT through self-consistency (Wang et al., 2023b), least-to-most decomposition (Zhou et al., 2023a), and tree-of-thoughts search (Yao et al., 2023a).These methods aggregate multiple paths, solve sequential subquestions, or explore multiple reasoning alternatives.

H.2. Scaling Inference-time Compute · H.3. Reinforcement Learning for Reasoning Enhancement

The sections frame inference-time compute scaling as a broad family of methods that trade additional test-time computation for performance, while positioning the work as combining increased RL and test-time compute to enhance in-context search. They contrast this framework with conventional reasoning-oriented RL pipelines that rely on human demonstrations, preference rewards, or process-based rewards.

  • H.2. Scaling Inference-time Compute: Inference-time compute scaling improves model performance by allocating more computation during inference, a strategy motivated by constraints on available human data.The section defines inference-time scaling broadly as methods that improve performance by increasing inference compute.
  • H.2. Scaling Inference-time Compute: Parallel generation scales inference by producing diverse reasoning chains and selecting an answer with a reranker, process-based reward model, or majority vote.These selection mechanisms trade additional computation for improved answer choice.
  • H.2. Scaling Inference-time Compute: Search methods such as Monte Carlo Tree Search and Beam Search guide exploration of the solution space more effectively than unguided generation.These methods are presented as alternatives to parallel generation within inference-time scaling.
  • H.2. Scaling Inference-time Compute: Other inference-time scaling methods include iterative self-correction with feedback, tool use for knowledge- or compute-intensive tasks, and test-time training.These approaches respectively refine outputs, augment testing with tools, or update the model during inference.
  • H.2. Scaling Inference-time Compute: The work combines additional RL compute with increased test-time compute, integrating test-time scaling into reinforcement learning that incentivizes enhanced in-context search abilities.The approach increases test-time computation through more tokens while also scaling RL compute.
  • H.3. Reinforcement Learning for Reasoning Enhancement: Traditional reasoning-oriented RL pipelines use supervised fine-tuning on human demonstrations, train a reward model on preferences, and then optimize the language model.The demonstrations provide strong initialization and help prevent mode collapse.
  • H.3. Reinforcement Learning for Reasoning Enhancement: STaR-style methods fine-tune on self-generated chains of thought leading to correct answers, while process-based rewards assess both final-answer correctness and reasoning soundness.Unlike these approaches, the work applies outcome-based RL.

I. Open Weights, Code, and Data · J. Evaluation Prompts and Settings

The paper releases model weights, inference code, usage guidance, and training data to support reproduction and ecosystem development. Evaluation spans knowledge, reasoning, coding, instruction-following, factuality, open-ended preference, multilingual, and mathematical benchmarks with task-specific settings.

  • I. Open Weights, Code, and Data: DeepSeek-R1, DeepSeek-R1-Zero, and six distilled Qwen- and Llama-based models are publicly available on HuggingFace.The released distilled models range from 1.5B to 70B parameters.
  • I. Open Weights, Code, and Data: SFT and RL data are also released publicly, with the review-stage data supplied as an attachment.This extends the release beyond model artifacts to training data used in supervised fine-tuning and reinforcement learning.
  • J. Evaluation Prompts and Settings: The evaluation suite covers broad knowledge and reasoning through MMLU, MMLU-Redux, and the more difficult MMLU-Pro, alongside graduate-level STEM questions in GPQA.MMLU spans 57 subjects, MMLU-Redux contains 5,700 re-annotated questions, and MMLU-Pro emphasizes deeper multistep reasoning.
  • J. Evaluation Prompts and Settings: Coding and mathematical reasoning are assessed with LiveCodeBench and math benchmarks covering AIME, MATH, and CNMO.LiveCodeBench collects evolving algorithmic problems from LeetCode, AtCoder, and CodeForces.
  • J. Evaluation Prompts and Settings: Other task-specific evaluations measure reading comprehension, instruction compliance, factuality, and source-grounded synthesis using DROP, IFEval, SimpleQA, C-SimpleQA, and FRAMES.FRAMES uses an Oracle Prompt containing ground-truth Wikipedia articles, isolating reasoning and synthesis over provided sources without external retrieval.
  • J. Evaluation Prompts and Settings: Open-ended capability and likely user preference are evaluated with Arena-Hard and AlpacaEval 2.0 using model-based judgment.Arena-Hard emphasizes challenging coding and mathematics prompts, whereas AlpacaEval 2.0 generally uses less difficult prompts and rarely requires reasoning.
  • J. Evaluation Prompts and Settings: The suite also includes Chinese commonsense, academic knowledge, and factuality evaluations through CLUEWSC, C-EVAL, and C-SimpleQA.C-EVAL covers 52 Chinese academic disciplines, while CLUEWSC targets contextual commonsense reasoning and C-SimpleQA tests precise Chinese fact retrieval.
Loading 2501.12948v2…