PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang

arXiv:2501.12948v2cs.CLcs.AIcs.LG

TL;DR

Developing reasoning beyond provided exemplars remains a challenge. The paper trains DeepSeek-R1-Zero through reinforcement learning without supervised fine-tuning and develops DeepSeek-R1 through multi-stage training. Self-verification and reflection emerge during reinforcement learning, while distilled smaller models outperform their original instruction-tuned counterparts.

  • Problem

    The central question is how to develop reasoning beyond provided exemplars and explore potentially superior, non-human-like reasoning pathways.

    Source: §1 · p. 2

  • Method

    The paper trains DeepSeek-R1-Zero through reinforcement learning without supervised fine-tuning and develops DeepSeek-R1 through a multi-stage framework integrating rejection sampling, reinforcement learning, and supervised fine-tuning.

    Source: §2 · p. 2, §1 · p. 2

  • Results

    Self-verification and reflection emerged during reinforcement learning, while distilled smaller models surpassed their original instruction-tuned counterparts in reasoning.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The authors identify hard reasoning questions, reliable verification, and sufficient reinforcement-learning compute—not large-scale human annotation—as keys to unlocking pre-trained models’ complex reasoning potential.

    Source: §6 · p. 10

    Scaling pure reinforcement learning remains an open challenge for tasks without reliable reward models, such as writing, because learned reward signals are susceptible to exploitation.

    Source: §6 · p. 11

Abstract

from arXiv · show

General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) and chain-of-thought prompting, have achieved considerable success on foundational reasoning tasks. However, this success is heavily contingent upon extensive human-annotated demonstrations, and models' capabilities are still insufficient for more complex problems. Here we show that the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories. The proposed RL framework facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation. Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and STEM fields, surpassing its counterparts trained via conventional supervised learning on human demonstrations. Moreover, the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models.

1. Introduction

DeepSeek-R1-Zero develops reasoning through reinforcement learning without prior supervised fine-tuning. DeepSeek-R1 adds multi-stage training to address readability, language mixing, and broader-task limitations, while distilled models extend access to smaller models.

1. Introduction

  • Dependence on human-annotated reasoning traces limits scalability, introduces cognitive biases, and restricts exploration of reasoning pathways beyond human-provided exemplars.These limitations motivate learning reasoning strategies without requiring models to replicate human thought processes.Source: §1 · pp. 1–2
  • DeepSeek-R1-Zero learns reasoning through reinforcement learning without prior supervised fine-tuning, rewarding final-answer correctness rather than constraining the reasoning process.Built on DeepSeek-V3-Base (DeepSeek-AI, 2024b) using Group Relative Policy Optimization (GRPO) (Shao et al., 2024), it develops longer responses incorporating verification, reflection, and alternative approaches.Source: §1 · p. 2
  • DeepSeek-R1 combines rejection sampling, reinforcement learning, and supervised fine-tuning to retain its predecessor’s reasoning capabilities while addressing readability, language mixing, and broader-task limitations.Additional non-reasoning data align behavior with human preferences. Its predecessor’s broader-task weaknesses include writing and open-domain question answering.Source: §1 · p. 2
  • Publicly released distilled models exhibit strong reasoning capabilities and outperform their original instruction-tuned counterparts.The smaller models aim to broaden access to powerful AI at lower energy cost and support research into long chain-of-thought reasoning.Source: §1 · p. 2

2. DeepSeek-R1-Zero

DeepSeek-R1-Zero trains through reinforcement learning without supervised fine-tuning, using rule-based incentives rather than prescribed reasoning content. Training improves AIME performance while longer reasoning, reflection, and exploration of alternative solutions emerge autonomously.

2.1. Group Relative Policy Optimization

  • DeepSeek-R1-Zero uses reinforcement learning without supervised fine-tuning, adopting Group Relative Policy Optimization (GRPO) (Shao et al., 2024).GRPO samples a group of outputs from the old policy for each question and computes advantages using the corresponding group of rewards.Source: §2 · p. 2, §2.1 · p. 3

2.3. Incentivize Reasoning Capability in LLMs

  • 77.9% average pass@1 on AIME 2024 is achieved by DeepSeek-R1-Zero during RL training, up from 15.6% initially.Self-consistency decoding (Wang et al., 2023c) reaches 86.7% accuracy, surpassing the average across all human competitors. Strong performance also extends to coding competitions and graduate-level biology, physics, and chemistry.Source: Figure 1 · p. 4, §2.3 · pp. 4–5

2.2. Reward Design

  • Equally weighted rule-based accuracy and format rewards guide mathematical, coding, and logical reasoning, avoiding neural reward models susceptible to reward hacking.Accuracy checks use specified-format math answers or compiled code evaluated against test cases; format rewards require reasoning within <think> and </think> tags. Neural reward models are also avoided because retraining adds computational cost and pipeline complexity.Source: §2.2 · pp. 3–4

2.3. Incentivize Reasoning Capability in LLMs

  • Training starts from DeepSeek-V3 base and requires reasoning followed by a final answer, imposing only structural constraints to avoid content-specific biases.This minimal template is intended to reveal the model’s natural progression during reinforcement learning.Source: §2.3 · p. 4
  • DeepSeek-R1-Zero autonomously lengthens its reasoning and develops reflection and alternative-solution exploration, including an “aha moment” marked by increased use of “wait.”Responses expand to hundreds or thousands of tokens without external modifications. Advanced problem-solving strategies develop through incentives rather than explicit instruction in how to solve problems.Source: Figure 1 · p. 4, §2.3 · p. 5, Table 2 · p. 5

3. DeepSeek-R1

DeepSeek-R1 combines cold-start data, supervised fine-tuning, and two reinforcement-learning stages to address poor readability and language mixing while refining reasoning and human alignment. Its rewards distinguish verifiable reasoning from general helpfulness and safety.

3. DeepSeek-R1

  • DeepSeek-R1 combines cold-start demonstrations, initial RL, rejection sampling and mixed-data supervised fine-tuning, followed by a second RL stage targeting helpfulness, harmlessness, and reasoning.Thousands of cold-start examples demonstrate conversational, human-aligned thinking. Subsequent supervised fine-tuning includes reasoning and non-reasoning data to support both reasoning and writing.Source: §3 · p. 6

3.1. Model-based Rewards

  • Helpfulness rewards assess only final summaries, while safety rewards evaluate entire responses, including reasoning; each general query receives its dataset-specific reward.Restricting helpfulness assessment to the summary emphasizes response utility while minimizing interference with reasoning.Source: §3.1 · pp. 6–7
  • 66,000 helpfulness preference pairs and 106,000 safety prompts train separate reward models using pairwise and point-wise objectives, respectively.For non-reasoning helpfulness queries, DeepSeek-V3 judges each pair four times with randomized response order; selection averages judgments, requires a score difference greater than 1, and balances response lengths. Safety prompts have model-generated responses labeled safe or unsafe.Source: §3.1 · p. 7

3.2.1. Training Details of the First RL Stage

  • Language-consistency rewards improve readability but slightly reduce performance, measuring target-language words in chain-of-thought and augmenting rewards for reasoning and non-reasoning data.The first RL stage samples 16 outputs per question, each capped at 32,768 tokens, with a GRPO clip ratio of 10.Source: §3.2.1 · pp. 7–8

3.2.2. Training Details of the Second RL Stage

  • The second RL stage combines rule-based reasoning rewards with model-based general-data rewards to refine reasoning, helpfulness, and harmlessness.Training runs for 1,700 steps at temperature 0.7; general instruction data and preference-based rewards appear only in the final 400 steps. Higher temperatures caused incoherent generation, while longer model-based preference-reward training could lead to reward hacking.Source: §3.2.2 · p. 8

4. Experiment

Stage-by-stage evaluations show that reasoning-oriented RL strengthens advanced reasoning, while the final mixed-data RL stage primarily improves instruction-following and user-preference performance.

4. Experiment

  • Dev1 improves instruction-following over DeepSeek-R1-Zero on IF-Eval and ArenaHard but partially degrades reasoning, especially on AIME, because its cold-start dataset is limited.The evaluation suite spans a broad range of benchmarks.Source: §4 · p. 8
  • 25% improvement on AlpacaEval 2.0 and 17% on ArenaHard are final DeepSeek-R1’s main gains after comprehensive mixed-data RL on Dev3.Code and mathematics improved marginally because substantial reasoning-specific RL had already occurred in earlier stages.Source: Table 3 · p. 9, §4 · p. 9
  • Dev2 substantially improves code generation, mathematical problem solving, and STEM reasoning, while showing only marginal gains on general-purpose benchmarks such as AlpacaEval 2.0.These results suggest reasoning-oriented RL has limited influence on user-preference benchmarks.Source: §4 · pp. 8–9
  • Dev3 combines reasoning and non-reasoning datasets in supervised fine-tuning, improving reasoning and general language generation, with notable gains over Dev2 on AlpacaEval 2.0 and Aider-Polyglot.The paper attributes these benchmark gains to large-scale non-reasoning corpora and code engineering datasets.Source: §4 · p. 9
  • Supplementary results show reasoning transfers to smaller models and provide model comparisons, safety evaluations, fresh-test evaluations, mathematical-category breakdowns, and test-time scaling analysis.The model comparisons include DeepSeek-V3.Source: §4 · p. 9

5. Ethics and Safety Statement

DeepSeek-R1’s inherent safety is assessed as moderate, with stronger safety when paired with a risk control system. Jailbreak attacks and further fine-tuning remain risks, while enhanced reasoning can make dangerous plans more executable.

5. Ethics and Safety Statement

  • Jailbreak attacks can elicit dangerous content, and enhanced reasoning can make the resulting plans more operationally feasible and executable.The authors identify explosive manufacturing plans as one example of dangerous content.Source: §5 · p. 10
  • Further fine-tuning of a publicly available model could compromise its inherent safety protections.Source: §5 · p. 10
  • DeepSeek-R1’s inherent safety is generally moderate, comparable to GPT-4o (2024-05-13), but reaches a superior standard when coupled with a risk control system.The assessment covers open-source and in-house safety benchmarks, multiple languages, and resistance to jailbreak attacks.Source: §5 · p. 10

6. Conclusion, Limitation, and Future Work

Large-scale reinforcement learning in DeepSeek-R1-Zero and DeepSeek-R1 encourages emergent reasoning behaviors, including self-verification and reflection. Remaining challenges include reliable rewards, tool use, token efficiency, multilingual behavior, prompting sensitivity, and software engineering performance.

Conclusion

  • DeepSeek-R1 achieves frontier reasoning-benchmark results, while large-scale RL in DeepSeek-R1-Zero and DeepSeek-R1 encourages emergent self-verification and reflection.The results indicate substantial reasoning potential in pretrained checkpoints. The authors argue that hard reasoning questions, reliable verifiers, and sufficient RL compute—not large-scale human annotation—are key to unlocking it.Source: §6 · p. 10

Structure Output and Tool Use

  • DeepSeek-R1 has suboptimal structured output compared with existing models and cannot use search engines or calculators to improve its answers.The authors expect RL environments for structured output and tool use to address these limitations in the next version.Source: §6 · p. 10

Token efficiency

  • DeepSeek-R1 dynamically allocates inference tokens by problem complexity but still overthinks simpler questions, leaving token efficiency improvable.Unlike conventional majority voting or Monte Carlo Tree Search approaches, it uses fewer tokens for simple tasks and more for complex tasks.Source: §6 · p. 10

Language Mixing

  • DeepSeek-R1’s Chinese-and-English optimization can cause language mixing or English responses to queries in other languages.The authors suggest this may reflect DeepSeek-V3-Base’s predominantly Chinese and English training and plan improvements in future updates.Source: §6 · p. 10

Prompting Engineering

  • Few-shot prompting consistently degrades DeepSeek-R1’s performance; the authors recommend directly describing the problem and specifying the output format in a zero-shot setting.This recommendation reflects the model’s observed sensitivity to prompts.Source: §6 · pp. 10–11

Software Engineering Tasks

  • DeepSeek-R1 shows no huge improvement over DeepSeek-V3 on software engineering benchmarks because lengthy evaluations have constrained extensive RL training.Planned remedies include rejection sampling on software engineering data or asynchronous evaluations during RL to improve efficiency.Source: §6 · p. 11

Reward Hacking

  • Pure RL depends on reliable rewards, but learned reward models can become exploitable, limiting expansion to less-verifiable tasks.The study uses rule-based reasoning rewards; tasks lacking reliable signals instead receive human-annotated supervised data and only hundreds of RL steps. Future work targets robust reward structures, while the authors envision machines surpassing humans on effectively verifiable tasks.Source: §6 · p. 11

Future Work

  • Future tool-augmented reasoning could broaden machine-driven solutions and improve accuracy through information retrieval, computation, and real-world validation.Proposed tools include compilers and search engines during reasoning, plus biological or chemical reagents to validate final results.Source: §6 · p. 11

7. Author List … A.1. DeepSeek-V3

The author list credits contributors for R1-Zero’s emergence of long chain-of-thought reasoning and the project’s algorithmic, data, and systems work. The background describes DeepSeek-V3 (DeepSeek-AI, 2024b), its pretraining data, and the foundation used to train R1 and R1-Zero.

7. Author List

  • The author list groups individuals by contribution role, alphabetizes first names within categories, and marks former team affiliates with an asterisk.Separate lists identify core contributors and contributors.Source: §7 · pp. 11–12
  • Peiyi Wang and Daya Guo equally created R1-Zero, demonstrating that outcome-based RL induces long chain-of-thought reasoning and laying the foundation for R1.Daya Guo also contributed to RL training stability for MoE models.Source: §7 · pp. 11–12
  • Junxiao Song proposed and initially implemented GRPO and introduced rule-based math rewards; subsequent contributors refined GRPO and developed a large PPO clipping strategy.Peiyi Wang and Runxin Xu refined GRPO. Zhibin Gou proposed the clipping strategy and demonstrated its significance alongside Zhihong Shao and Junxiao Song.Source: §7 · p. 12
  • Dedicated contributors handled domain-specific data, rewards, evaluation, labeling, RL infrastructure, final training oversight, and development of the R1-distill series.Teams covered code, mathematics, and other reasoning/general tasks; Qihao Zhu and Zhihong Shao contributed RL data selection, while Zhuoshu Li and Yu Wu co-led labeling. Systems contributors optimized efficiency and stability; Zhibin Gou, Daya Guo, and Ruoyu Zhang oversaw final training, and Zhibin Gou led R1-distill.Source: §7 · p. 12

A.1. DeepSeek-V3

  • 671 billion total parameters and 37 billion activated per token characterize DeepSeek V3 (DeepSeek-AI, 2024b), an open-source MoE model.Pretraining used 14.8 trillion tokens, followed by supervised fine-tuning and RL. Features include Multi-head Latent Attention (DeepSeek-AI, 2024a), auxiliary-loss-free load balancing, and Multi-Token Prediction (Gloeckle et al., 2024).Source: §A.1 · p. 13
  • DeepSeek-V3-Base pretraining used only web pages and e-books, without intentionally incorporating synthetic data, although crawled pages contained OpenAI-model-generated answers.The authors acknowledge possible indirect knowledge transfer from other models and specifically deny intentionally including OpenAI-generated synthetic data during pretraining cooldown.Source: §A.1 · p. 13
  • Mathematical and code pretraining content exposed DeepSeek-V3-Base to reasoning traces, enabling plausible solution candidates that RL can select and improve.The authors suggest predominantly Chinese and English training data may explain R1-Zero’s language mixing when a language-consistency reward is absent.Source: §A.1 · p. 13
  • DeepSeek-R1 and DeepSeek-R1-Zero are trained on DeepSeek-V3-Base; DeepSeek-R1 additionally uses non-reasoning data from DeepSeek-V3’s supervised fine-tuning data.DeepSeek-V3 denotes the instructed model, while DeepSeek-R1-Dev1, DeepSeek-R1-Dev2, and DeepSeek-R1-Dev3 denote intermediate R1 checkpoints.Source: §A.1 · p. 13

A.2. Conventional Post-Training Paradigm … B.3.2. DeepSeek-R1 Cold Start

These sections motivate reasoning self-exploration and describe GRPO, its comparison with tuned PPO on MATH, RL infrastructure, reward judging, training data, and DeepSeek-R1’s user-oriented cold start. The cold-start material covers large-input generators for testing code efficiency, few-shot prompting for concise simple-math responses, and reference-based answer judging with DeepSeek-V3.

A.2. Conventional Post-Training Paradigm

  • SFT may constrain reasoning exploration because human targets omit reflection and verification; DeepSeek-R1-Zero instead discovers trajectories through self-exploration for subsequent distillation.Conventional post-training applies SFT then RL (Ouyang et al., 2022), but SFT generalization depends on dataset quality and diversity (Brown et al., 2020). RL optimizes reward signals, including rule-based rewards, reducing reliance on fully labeled input-output pairs.Source: §A.2 · pp. 13–14

A.3. A Comparison of GRPO and PPO

  • GRPO replaces PPO’s value model with group-score advantage estimates and places KL regularization directly in the loss rather than per-token rewards.GRPO (Shao et al., 2024) trains both DeepSeek-R1-Zero and DeepSeek-R1, avoiding value-model overhead and difficult final-reward predictions from revisable partial reasoning. PPO’s cumulative KL penalties may discourage longer responses; periodic reference-policy updates balance exploration and stability during training.Source: §A.3 · pp. 14–16, Figure 3 · p. 15
  • PPO approaches GRPO on MATH with careful GAE tuning, but requires additional hyperparameter-search and value-model costs.Using DeepSeek-Coder-V2-Lite (16B MoE with 2.4B active parameters), PPO performs considerably worse at lambda = 0.95 but nears GRPO at lambda = 1.0.Source: Figure 4 · p. 16, §A.3 · p. 16

B.1. RL Infrastructure

  • The decoupled RL infrastructure separates rollout, inference, rule-based rewards, and training, with optimizations for decoding, workload balance, and memory reuse.Rollout uses expert parallelism, replicated hotspot experts, and MTP self-speculative decoding; rule-based scoring runs asynchronously alongside rollout and inference. Length-sorted Best-Fit packing equalizes chunk counts across devices, DualPipe supports pipeline parallelism, and models are offloaded to system memory or disk between phases.Source: §B.1 · pp. 17–18

B.2. Reward Model Prompt

  • The reward judge first answers the prompt independently, then compares both assistant responses for correctness, helpfulness, relevance, and concision before issuing a five-level verdict.The prompt also requests corrections, clarification when ambiguity warrants it, attention to creativity when relevant, and identification of missing important information. Verdicts distinguish a tie from slight or significant superiority in either direction.Source: §B.2 · p. 18

B.3.1. RL Data

  • 26k mathematics, 17k algorithm-competition, 8k bug-fixing, 22k STEM, and 15k logic questions support reasoning RL; general RL adds 66k helpfulness and 12,000 harmlessness questions.Chinese/English tasks use answer matching for mathematics, excluding proofs; hidden tests for competition coding; unit tests for bug fixes; binary option matching for STEM; and automatically evaluable real/synthetic logic problems. Separate ranked-response reward models assess helpfulness and harmlessness; the helpfulness model trains for one epoch with an 8192-token maximum, while reward inference has no explicit input-length limit.Source: §B.3.1 · pp. 19–20

B.3.2. DeepSeek-R1 Cold Start

  • DeepSeek-R1’s cold start fine-tunes the initial RL actor on limited long-CoT data emphasizing first-person, language-consistent reasoning with reflection and verification.Human annotators naturalize reasoning traces, an LLM rewrites additional examples in that style, and humans verify the generated outputs. These product-driven patterns are engineered heuristics, not evidence of inherently human-like intelligence or autonomous problem solving, and may elicit unwarranted user trust.Source: §B.3.2 · pp. 20–21
  • DeepSeek-R1-Zero trajectories are filtered for correct answers and readability, then refined by DeepSeek-V3 into language-consistent reasoning and concise solutions.Sampling uses temperature 1.0 across thousands of diverse prompts, with sympy expression checks and repetition/language-mixing filters. For 5151 Codeforces and 2504 AtCoder problems, DeepSeek-V2.5 generates test-case programs; correct submissions reject invalid cases, while incorrect submissions identify discriminative test subsets.Source: §B.3.2 · pp. 21–22
  • Adversarial generators include repeated-character strings, alternating pairs of characters, and sequential alphabet characters.The alternating-character generator samples two distinct lowercase letters and alternates them throughout the string.Source: §B.3.2 · pp. 23–24
  • Large, diverse, adversarial input generators test time complexity by producing inputs intended to make incorrect code exceed time limits.Generators must appear in separate code blocks, with each printing exactly one input directly to stdout.Source: §B.3.2 · p. 24
  • Few-shot prompting guides DeepSeek-V3 to produce concise, appropriately structured responses to simple math problems.The supplied example includes an elementary problem such as 1 + 1 = ?.Source: §B.3.2 · p. 24
  • DeepSeek-V3 judges reasoning answers against references, returning a correctness analysis and a correct/incorrect classification in JSON.Correct answers must align with the reference in both reasoning and conclusion; major reasoning or conclusion errors warrant an incorrect classification.Source: §B.3.2 · p. 26

B.3.3. 800K Supervised Data … C. Self-Evolution of DeepSeek-R1-Zero

This section describes the supervised dataset’s composition, examples, and potential multi-turn limitations, along with training settings and costs. It also documents helpful-reward hacking and shows that language-consistency reward stabilizes language use, with comparable mathematics performance but slightly reduced coding performance.

B.3.3. 800K Supervised Data

  • About 600k reasoning training samples come from rejection sampling the first-stage RL checkpoint, retaining only correct responses and filtering difficult-to-read reasoning traces.The dataset expands beyond rule-verifiable tasks, using DeepSeek-V3 to judge some predictions against ground truth. Filtering removes traces containing mixed languages, long paragraphs, or code blocks.Source: §B.3.3 · p. 26
  • Approximately 200k non-reasoning training samples use the DeepSeek-V3 pipeline and reused SFT data, covering writing, factual QA, self-cognition, translation, and software engineering.Software engineering examples include program repair and front-end web development. DeepSeek-V3 generates preliminary thinking for selected tasks, while simple queries receive no chain-of-thought.Source: §B.3.3 · p. 26
  • Human-verified artificial reasoning traces improve query interpretation, while their prescribed style emphasizes concise paragraphs, conversational language, and understanding the complete user context.The style avoids technical formatting such as markdown and considers users’ circumstances and unstated needs. The reported improvements concern identifying format constraints, clarifying intentions, and determining required output structure.Source: §B.3.3 · p. 27
  • Approximately 800,000 supervised samples span mathematics, coding, STEM, logic, and general-domain tasks, but predominantly single-turn interactions may limit multi-turn conversational capabilities.Mathematics questions are verifiable, while coding includes competitive programming, debugging, and project-oriented tasks. Mathematics and general-domain data are predominantly Chinese and English; expanding multi-turn dialogue data remains future work.Source: Table 5 · p. 27, §B.3.3 · p. 27
  • The supervised-data examples include a reasoning trajectory for implementing a Dictionary class and a non-reasoning writing trajectory for a constrained poem.The class stores word–definition pairs and returns either a definition or a missing-entry message. The writing prompt requests a short poem ending in “was tentacles all along”.Source: §B.3.3 · pp. 31, 33

B.4.1. Hyper-Parameters of DeepSeek-R1-Zero-Qwen-32B

  • DeepSeek-R1-Zero-Qwen-32B training samples 16 outputs per question, with 32 unique questions producing a batch of 512 outputs per step.Settings are learning rate 2e-6, KL coefficient 0.001, rollout temperature 1, and maximum output length 32,768; the reference model is replaced with the latest policy every 400 steps. Each rollout generates 8,192 outputs, randomly split into 16 mini-batches for one inner epoch.Source: §B.4.1 · p. 34

B.4.2. Hyper-Parameters of SFT

  • Both SFT stages fine-tune DeepSeek-V3-Base for 2–3 epochs on curated data, using a 32,768-token maximum context and batch size 128.A cosine schedule decreases the learning rate from 5 × 10^-5 to 5 × 10^-6.Source: §B.4.2 · p. 35

B.4.3. Hyper-Parameters of Distillation

  • Distillation fine-tunes each corresponding base model on 800k examples for 2–3 epochs, using a 32,768-token maximum context and batch size 64.Cosine decay reduces the learning rate to one-tenth of its initial value.Source: §B.4.3 · p. 35

B.4.4. Training Cost

  • 198 hours trained DeepSeek-R1-Zero, versus roughly 80 hours for DeepSeek-R1, each using 64*8 H800 GPUs.SFT dataset creation required 5K GPU hours. Promising preliminary experiments with a 30B-parameter model on A100 GPUs supported scaling to 660B R1-Zero and R1.Source: §B.4.4 · p. 35

B.5. Reward Hacking

  • The helpful reward model permits reward hacking: responses exploit reward-model biases rather than reflect human preferences, while increasing reward accompanies declining CodeForces performance.The observed misalignment can degrade performance on complex reasoning tasks.Source: §B.5 · p. 35, Figure 6 · p. 36

B.6. Ablation Study of Language Consistency Reward

  • Language-consistency reward maintains stable language consistency and comparable mathematics performance, with slight coding degradation, in DeepSeek-R1-Distill-Qwen-7B.The ablation uses DeepSeek-R1’s cold-start data; without this reward, language consistency deteriorates during RL. Applying it makes outputs more readable and better aligned with human preferences.Source: §B.6 · pp. 35, 37, Figure 7 · p. 37

C.1. Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training

DeepSeek-R1-Zero masters easier MATH problems early, while performance on complex problems improves throughout training. Comparisons across difficulty levels require caution because sample sizes, mathematical categories, and human-assigned difficulty labels affect interpretation.

C.1. Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training

  • 0.90-0.95 accuracy is reached quickly on easier MATH problems (levels 1-3) and remains stable throughout training.Simpler reasoning tasks, as judged by humans, are mastered early.Source: §C.1 · pp. 37–38
  • Occasionally higher accuracy on levels 3-4 than level 1 reflects uneven sample sizes, differing mathematical-category distributions, and difficulty annotations based on human rather than machine-learning considerations.Level 1 contains only 43 of 500 examples, versus approximately 100 per higher level; its 95-97% accuracy represents 1-2 unsolved problems, primarily in geometry.Source: §C.1 · pp. 37–38
  • Near 0.55 to 0.90 accuracy on level-5 problems marks the largest improvement, while level-4 accuracy rises from near 0.78 to 0.95 during training.The training trends show improving capability on complex reasoning problems spanning levels 3-5.Source: §C.1 · pp. 37–38, Figure 8 · p. 38

C.2. Evolution of Advanced Reasoning Behaviors in DeepSeek-R1-Zero during Training · D. Evaluation of DeepSeek-R1 · D.1. Experiment Setup

During DeepSeek-R1-Zero training, reflective-word counts increased and particular reflection patterns emerged at specific stages. DeepSeek-R1 evaluation covers diverse reasoning and generation tasks, using decontamination, benchmark-specific prompts, and sampled performance estimates.

C.2. Evolution of Advanced Reasoning Behaviors in DeepSeek-R1-Zero during Training

  • 4000–7000 training steps saw occasional “wait” usage, followed by significant spikes after step 8000, contrasting with near-absence early in training.This pattern suggests that different forms of reflection emerge at particular training stages.Source: §C.2 · p. 38, Figure 9 · p. 39
  • 5- to 7-fold increases in reflective-word counts accompanied DeepSeek-R1-Zero training, suggesting that RL encourages long-chain intermediate tokens.Three human experts selected and merged the reflective-word list used to track this behavior.Source: §C.2 · p. 38, Figure 9 · p. 39

D.1. Experiment Setup

  • Evaluation spans knowledge, instruction following, document reasoning, competition coding, software engineering, and mathematics, with a smaller benchmark set reported for distilled models.Knowledge tests cover encyclopedic knowledge, long-tail facts, and Ph.D.-level science. Distilled models are evaluated on AIME 2024, MATH-500, GPQA Diamond, Codeforces, and LiveCodeBench.Source: §D.1 · p. 39
  • Open-ended generation evaluation follows AlpacaEval 2.0 and Arena-Hard protocols, using GPT-4-Turbo-1106 for pairwise comparisons and only final summaries to mitigate length bias.The evaluation model receives only the final summary rather than the complete reasoning output.Source: §D.1 · p. 39
  • Approximately six million potential mathematics pre-training texts were removed during decontamination, but n-gram filtering cannot rule out paraphrased test contamination.Pre-training and post-training data undergo matching 10-gram filtering; mathematics SFT data and RL prompts come exclusively from pre-2023 competitions. The base model’s knowledge cutoff is July 2024, but benchmarks released before 2024 may remain contaminated.Source: §D.1 · p. 40
  • Benchmark-specific prompts include zero-shot adaptations for MMLU-Pro, C-Eval, and CLUE-WSC because few-shot chain-of-thought may hurt DeepSeek-R1 performance.MMLU, DROP, GPQA Diamond, and SimpleQA use simpleevals; MMLU-Redux uses zero-shot Zero-Eval. LiveCodeBench uses CoT on August 2024–January 2025 problems; Codeforces uses 10 Div.2 contests with expert-crafted tests; SWE-Bench Verified uses agentless (Xia et al., 2024), and Aider uses diff format.Source: §D.1 · p. 40
  • Baselines include DeepSeek-V3, Claude-Sonnet-3.5-1022, GPT-4o-0513, OpenAI-o1-mini, and OpenAI-o1-1217, with the latter’s performance taken from official reports.Official reports substitute for OpenAI-o1-1217 API evaluation because access is challenging in mainland China. Distilled-model comparisons additionally include QwQ-32B-Preview.Source: §D.1 · p. 40
  • 0.6 temperature and 0.95 top-p sampling support pass@1 estimates under pass@k evaluation (Chen et al., 2021), avoiding greedy decoding’s repetition and checkpoint variability.Generation is capped at 32,768 tokens, with k=64 for AIME and GPQA, k=16 for MATH and CodeForces, and k=8 for LCB. AIME 2024 also reports majority-vote consensus over 64 samples as cons@64.Source: §D.1 · pp. 40–41

D.2. Main Results … E.3. Mathematical Capabilities Breakdown by Categories

This section evaluates DeepSeek-R1’s mathematics, competitive programming, long-context understanding, instruction following, and ChatbotArena performance, including comparisons with DeepSeek-V3 and OpenAI-o1-1217. It also examines safety through official-service risk controls, public and in-house benchmarks, multilingual evaluations, and jailbreak testing, highlighting intellectual-property weaknesses and jailbreak vulnerabilities.

D.2. Main Results

  • DeepSeek-R1 is comparable to OpenAI-o1-1217 on mathematics and SWE Verified, trails it on Aider, and surpasses DeepSeek-V3 on MMLU, MMLU-Pro, and GPQA Diamond.Large-scale RL particularly improves STEM accuracy; reasoning-focused models dominate LiveCodeBench and Codeforces, while engineering-related RL data remain limited. R1 also excels on FRAMES, IF-Eval, AlpacaEval2.0, and ArenaHard, demonstrating document-analysis, writing, and open-domain QA strengths; instruction-following gains are linked to final-stage SFT and RL data.Source: Table 8 · p. 41, §D.2 · pp. 41–42
  • 96.3% of Codeforces participants are outperformed by DeepSeek-R1, which also exceeds average AIME competitor performance but trails web-enabled Ph.D.-level experts on GPQA.The paper suggests that giving DeepSeek-R1 web access could narrow the GPQA gap; this remains an expectation rather than a demonstrated result.Source: Figure 10 · p. 42, §D.2 · p. 42
  • First place in ChatbotArena’s style-control setting was shared by DeepSeek-R1, OpenAI-o1, and Gemini-Exp-1206 on January 24, 2025.ChatbotArena (Chiang et al., 2024) derives preference rankings from anonymous, crowdsourced pairwise comparisons. Style control separates response presentation, such as length and formatting, from substantive content.Source: Figure 11 · p. 43, §D.2 · p. 43

D.3.1. Risk Control System for DeepSeek-R1

  • External risk controls improve DeepSeek-R1 service safety, especially against jailbreaks, by screening dialogue keywords and sending flagged conversations to DeepSeek-V3 for review.Review results determine dialogue retraction, and developers can customize safety standards. Comparisons report R1 with and without controls, whereas other vendors’ results combine intrinsic safety and external safeguards.Source: §D.3.1 · pp. 44, 46

D.3.2. R1 Safety Evaluation on Standard Benchmarks

  • DeepSeek-R1 achieves generally comparable safety to frontier models across public benchmarks, but intellectual-property requests expose a substantial HarmBench weakness.Other evaluated safety categories, including discrimination, violence, extremism, and privacy violations, show strong safeguards. Evaluations use HELM results recorded in April 2025 or reproduced official methods, replace HarmBench’s smaller judge with GPT4o (2024-11-20), and count risk-control refusals as safe.Source: §D.3.2 · pp. 47–48, Table 9 · p. 48

D.3.3. Safety Taxonomic Study of R1 on In-House Benchmark

  • 1,120 Chinese–English test questions cover 28 safety subcategories, with aligned taxonomy, data quantities, languages, and evaluation methods supporting comparisons across safety domains.The taxonomy covers discrimination and prejudice, illegal and criminal behavior, harmful behavior, and moral and ethical issues. This in-house framework also supports extensions for multilingual and jailbreak evaluations.Source: §D.3.3 · pp. 48, 50
  • In-house evaluation uses GPT4o (2024-11-20) to distinguish unsafe answers, informative safe answers, and refusals, tracking Unsafe and Reject rates separately.Lower rates are preferred: safe, informative answers are favored over refusals, which are treated as intermediate between safe and unsafe responses. Table 10 compares configurations with and without risk controls, but the supplied passages provide no numerical results.Source: §D.3.3 · pp. 50–51, Table 10 · p. 51
  • Around 10% unsafe rate places risk-controlled DeepSeek-R1 alongside Claude-3.7-Sonnet and o1 (2024-12-17), versus beyond 20% for DeepSeek-R1 without risk control.Risk-controlled DeepSeek-V3 achieves around 5% unsafe rate; risk controls raise DeepSeek models’ rejection rates to around 25%, while Claude-3.7-Sonnet has the lowest rejection rate and o1 rejects around 50%. DeepSeek-R1 handles Illegal and Criminal Behavior and Moral and Ethical Issues particularly well, but performs only averagely on Discrimination and Prejudice Issues and Harmful Behavior.Source: §D.3.3 · p. 51

D.3.4. Multilingual Safety Performance

  • 85.9% multilingual safety score for risk-controlled DeepSeek-R1 approaches Claude-3.7-Sonnet’s 88.3%; risk-controlled DeepSeek-V3 scores 86.5% across 50 languages.Scoring awards safe responses 5 points, rejections 4, and unsafe responses 0; without risk control, DeepSeek-R1 scores 74.2%, DeepSeek-V3 75.3%, and GPT-4o(2024-05-13) 75.2%. DeepSeek-R1 without risk control and Claude-3.7-Sonnet have no languages scoring below 60, versus one for uncontrolled DeepSeek-V3 and two for GPT-4o(2024-05-13).Source: Figure 14 · p. 52, §D.3.4 · p. 53

D.3.5. Robustness against Jailbreaking

  • 79.8% and 87.3% rejection rates for DeepSeek-R1 and o1(2024-12-17), respectively, reflect reasoning models’ heavier reliance on risk controls under jailbreak attacks.Using 2,232 jailbreak instructions, the evaluation finds increased unsafe responses and rejections across all tested models; Claude-3.7-Sonnet’s safe-response proportion decreases by 33.8%. Locally deployed open-source models face greater challenges without risk controls, motivating comparable safeguards in deployed services.Source: §D.3.5 · pp. 53–54, Table 11 · p. 54

E.1. Performance Comparison with DeepSeek-V3

  • DeepSeek-R1 improves over DeepSeek-V3 across all MMLU-Pro domains, with particularly notable STEM gains; MMLU’s largest gains are also in STEM but are smaller.The authors hypothesize that DeepSeek-V3 already approaches saturation on easier MMLU STEM tasks, while long chain-of-thought may improve question understanding in social sciences and humanities.Source: Figure 16 · p. 55, Figure 15 · p. 55, §E.1 · p. 56
  • DeepSeek-R1 outperforms DeepSeek-V3 in competitive programming, mathematical reasoning, Arena-Hard, and long-context understanding, whereas DeepSeek-V3 retains an instruction-following advantage.Both models share DeepSeek-V3-Base; the reported reasoning and long-context improvements appear on LiveCodeBench, AIME 2024, and FRAMES.Source: §E.1 · pp. 54, 56, Table 12 · p. 56

E.2. Generalization to Real-World Competitions

  • 75% Pass@1 on AIME 2025 places DeepSeek-R1 near o1’s 80%, testing generalization on competition problems released after training.DeepSeek-R1 also scores 143.7/150 on AMC 12 2024; combined with its AIME results, this exceeds the USAMO qualification threshold of 251.5 for AMC score + 10 × AIME score.Source: Table 13 · p. 57, §E.2 · p. 57, §E.1 · p. 57

E.3. Mathematical Capabilities Breakdown by Categories

  • DeepSeek-R1 significantly outperforms GPT-4o 0513 on competition mathematics, showing relative strengths in number theory and algebra but weaknesses in geometry and combinatorics.The evaluation covers 366 problems from 93 mathematics competitions held in 2024, including olympiads and team selection tests.Source: §E.3 · p. 57

E.4. An Analysis on CoT Length · E.5. Performance of Each Stage on Problems of Varying Difficulty · F. DeepSeek-R1 Distillation

DeepSeek-R1 adapts reasoning length to problem difficulty, additional sampling can improve accuracy, and successive training stages chiefly improve harder coding problems. Distillation transfers reasoning capabilities to smaller models, outperforming non-reasoning baselines on mathematical benchmarks and reinforcement learning alone on smaller architectures.

E.4. An Analysis on CoT Length

  • 61.8% Pass@1 on 2024 competition-math problems is achieved by DeepSeek-R1, compared with GPT-4o 0513’s 24.7% solve rate.DeepSeek-R1 averages 8,793 thinking tokens per problem, whereas GPT-4o 0513 averages 711 output tokens.Source: Figure 18 · p. 59, §E.4 · p. 59
  • Fewer than 7,000 thinking tokens serve simple competition problems, while DeepSeek-R1 devotes more than 18,000 to the most challenging ones.The model extends reasoning to verify, correct, backtrack, or explore alternatives; extremely easy questions can require fewer than 100 tokens. Explicitly training token-budget allocation is hypothesized to further increase this easy–hard disparity.Source: §E.4 · pp. 57, 59, Figure 18 · p. 59
  • 64-sample majority voting raises GPT-4o’s AIME 2024 solve rate from 9.3% to 13.4%, below DeepSeek-R1’s 79.8% and o1’s 79.2%.On the 2024 competition-math collection, 16-sample voting brings minimal improvement despite consuming more total tokens than DeepSeek-R1. Independent samples cannot build on earlier reasoning or self-correct, making this scaling token-inefficient.Source: §E.4 · pp. 59–60
  • 90.0% Pass@64 versus 79.8% Pass@1 on AIME 2024 demonstrates gains from independently sampling DeepSeek-R1 reasoning chains; majority voting raises accuracy to 86.7%.Long reasoning can remain incomplete or become trapped in incorrect logic. Traditional test-time scaling methods, including majority voting or Monte Carlo Tree Search, can complement extended reasoning.Source: §E.4 · p. 60

E.5. Performance of Each Stage on Problems of Varying Difficulty

  • Each DeepSeek-R1 training stage improves complex LiveCodeBench problems, with gains concentrated on medium and hard tasks because simple problems are generally already solved.The supplied table caption identifies the stage-by-difficulty comparison but provides no numerical results.Source: Table 14 · p. 60, §E.5 · p. 60

F. DeepSeek-R1 Distillation

  • 800,000 DeepSeek-R1-generated samples fine-tune Qwen and LLaMA (AI@Meta, 2024; Touvron et al., 2023) students using supervised fine-tuning without reinforcement learning.Students trained on high-quality teacher outputs consistently outperform those trained on human-generated data, corroborating prior distillation findings (Busbridge et al., 2025).Source: F. DeepSeek-R1 Distillation · p. 60
  • 1.5 billion parameters suffice for DeepSeek-R1-Distill-Qwen-1.5B to surpass GPT-4o and Claude-3.5-Sonnet on mathematical benchmarks, with performance improving as student size increases.Evaluation covers AIME, GPQA, Codeforces, MATH-500 (Lightman et al., 2024), and LiveCodeBench (Jain et al., 2024).Source: Table 15 · p. 61, F. DeepSeek-R1 Distillation · p. 61
  • Distillation outperforms reinforcement learning alone for smaller architectures, reducing computational requirements and potentially broadening access to AI.Substantial GPU and electricity requirements for LLM training and deployment create access barriers, particularly for under-resourced or marginalized communities.Source: F. DeepSeek-R1 Distillation · pp. 60–61

F.1. Distillation v.s. Reinforcement Learning · G. Discussion

Distilling DeepSeek-R1 into a 32B model outperformed direct large-scale reinforcement learning, while separate 7B experiments supported the autonomous development of advanced reasoning strategies through reinforcement learning.

F.1. Distillation v.s. Reinforcement Learning

  • DeepSeek-R1-Distill-Qwen-32B significantly outperformed Qwen2.5-32B-Zero across all benchmarks, while Qwen2.5-32B-Zero performed on par with QwQ-32B-Preview.Table 16 compares distilled and reinforcement-learning models on reasoning-related benchmarks.Source: Table 16 · p. 61, §F.1 · p. 62
  • Qwen2-Math-7B-Zero significantly outperformed Qwen2-Math-7B-Instruct and GPT-4o, supporting autonomous development of advanced reasoning strategies through large-scale reinforcement learning.Table 17 covers AIME 2024 and AIME 2025.Source: Table 17 · p. 62, §F.1 · p. 62
  • Over 10K training steps of large-scale reinforcement learning on math, code, and STEM data transformed Qwen2.5-32B-Base into Qwen2.5-32B-Zero.This experiment tested whether direct reinforcement learning could achieve performance comparable to distillation.Source: §F.1 · pp. 61–62
  • Distillation is economical and effective, whereas smaller models using the paper’s large-scale reinforcement learning require enormous computation and may still underperform distillation.The authors conclude that transferring capabilities from stronger models yields excellent results for smaller models.Source: §F.1 · p. 62
  • Advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning.The authors distinguish this potential requirement from distillation’s economical transfer of existing capabilities.Source: §F.1 · p. 62
  • Approximately 10,000 policy gradient update steps trained Qwen2-Math-7B-Zero in experiments designed to exclude prior exposure to reasoning trajectory data.The experiments used Qwen2-Math-7B, released in August 2024, before OpenAI-o1 launched in September 2024.Source: §F.1 · p. 62

G.1. Key Findings … H.1. Chain-of-thought Reasoning

Pure RL produced substantial gains with larger base models, while reliable verifiers and complementary SFT stages were important to the training pipeline. PRM and MCTS experiments exposed scaling difficulties; related work developed CoT prompting and structured reasoning techniques.

G.1. Key Findings

  • 7B dense and 16B MoE models failed to improve meaningfully on AIME, whereas larger base models achieved substantial gains from pure RL.Successful architectures included a 32B dense model, a 230B MoE model (DeepSeek-AI, 2024a), and a 671B MoE model (DeepSeek-AI, 2024b). Smaller models became repetitive as responses lengthened rather than benefiting from longer reasoning chains.Source: §G.1 · pp. 62–63
  • Reliable verifiers underpin DeepSeek-R1-Zero: rule-based rewards and LLM checks against predefined ground truth help mitigate reward hacking.LLM-based correctness evaluation works particularly well for concise, well-defined answers, but generalizes poorly to open-ended generation and long-form writing with subjective correctness.Source: §G.1 · p. 63
  • The iterative pipeline combines RL exploration of reasoning trajectories with SFT for tasks whose reliable reward signals are difficult to define.RL explores long-chain reasoning beyond what human-annotated traces alone provide, whereas SFT supports open-ended question answering and creative writing. RL alone risks reward hacking and suboptimal behavior on ill-posed tasks; SFT alone limits reasoning optimization through exploration.Source: §G.1 · p. 63

G.2. Unsuccessful Attempts

  • Process reward models faced ambiguous step definitions, difficult correctness annotation, reward hacking, and computational overhead that limited their advantages in large-scale RL experiments.Automated annotation could be unsatisfactory, manual annotation was difficult to scale, and reward hacking (Gao et al., 2022) necessitated resource-intensive reward-model retraining. PRMs nevertheless supported top-N reranking or guided search; the reported setbacks do not establish that these approaches cannot produce effective reasoning models.Source: §G.2 · p. 63
  • MCTS improved inference with a pretrained value model, but exponential token-search spaces and difficult value-model training hindered iterative self-improvement.The pipeline divided answers into tagged reasoning steps, generated question-answer pairs through value-guided MCTS, and iteratively trained actor and value models. Node-expansion limits risked local optima, while value-model quality directly influenced each search step.Source: §G.2 · pp. 63–64

H.1. Chain-of-thought Reasoning

  • Chain-of-thought prompting (Wei et al., 2022b) improved arithmetic, commonsense, and symbolic reasoning by eliciting intermediate steps before final answers.Suzgun et al. (2023) demonstrated that effectiveness scales with model size, while Kojima et al. (2022) extended CoT to zero-shot settings through a simple step-by-step instruction.Source: §H.1 · p. 64
  • Self-consistency (Wang et al., 2023b), least-to-most prompting, and tree-of-thoughts (Yao et al., 2023a) extend CoT using structured reasoning frameworks.These methods respectively aggregate answers across reasoning paths, solve sequential subquestions, and explore branching reasoning with lookahead or backtracking. Collectively, they use human prior knowledge and structured reasoning to enhance LLM capabilities.Source: §H.1 · p. 64

H.2. Scaling Inference-time Compute · H.3. Reinforcement Learning for Reasoning Enhancement

The paper combines additional reinforcement-learning compute with more test-time tokens to incentivize enhanced in-context search. It applies outcome-based RL directly to base language models without initial supervised fine-tuning, encouraging reasoning strategies beyond imitation of human examples.

H.2. Scaling Inference-time Compute

H.3. Reinforcement Learning for Reasoning Enhancement

  • Traditional RL pipelines initialize models with human demonstrations, helping prevent mode collapse but potentially constraining the discovery of novel reasoning strategies.After supervised fine-tuning, a reward model learns human preferences, and optimization uses methods such as PPO (Schulman et al., 2017) or DPO (Rafailov et al., 2023).Source: §H.3 · p. 65
  • Outcome-based RL is applied directly to base language models without initial supervised fine-tuning, encouraging diverse reasoning strategies beyond imitation of human examples.The design encourages innovative, unconstrained reasoning rather than prescribing human reasoning trajectories.Source: §H.3 · p. 65
  • Earlier reasoning-enhancement approaches use successful self-generated chains of thought or rewards that assess both answers and reasoning processes.STaR-style methods fine-tune on self-generated reasoning yielding correct answers (Singh et al., 2024; Yuan et al., 2023). Process-based rewards emphasize final-answer correctness and reasoning soundness (Lightman et al., 2024; Shao et al., 2024; Wang et al., 2023a).Source: §H.3 · p. 65

I. Open Weights, Code, and Data · J. Evaluation Prompts and Settings

The sections describe released model weights, inference code, usage guidance, and SFT/RL data availability, then outline evaluation benchmarks and settings. FRAMES supplies ground-truth Wikipedia articles, while open-ended benchmarks use model-based judging.

I. Open Weights, Code, and Data

  • DeepSeek-R1 and DeepSeek-R1-Zero weights are publicly available on HuggingFace alongside Qwen- and Llama-based distilled models.Distilled releases include DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Qwen-14B, DeepSeek-R1-Distill-Qwen-32B, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-Distill-Llama-70B.Source: §I · p. 65
  • Inference code and usage guidelines are released on GitHub, and the authors also report releasing SFT and RL data.The data location remains the placeholder “xxx” in the supplied passage; the authors report uploading the data as a review attachment.Source: §I · pp. 65–66

J. Evaluation Prompts and Settings

  • 5,700 manually re-annotated questions across 57 MMLU subjects form MMLU-Redux, complementing MMLU, harder MMLU-Pro, graduate-level GPQA, and Chinese-language C-EVAL and CLUEWSC.MMLU assesses factual and conceptual understanding; MMLU-Redux reduces noise and ambiguity, MMLU-Pro requires deeper multistep reasoning, and GPQA covers graduate-level biology, physics, and chemistry. C-EVAL covers 52 academic disciplines in Chinese, while CLUEWSC tests Chinese commonsense and contextual understanding.Source: Table 18 · p. 66, Table 19 · p. 67, Table 21 · p. 69
  • LiveCodeBench evaluates algorithmic competition problems, while AIME, MATH, and CNMO evaluate mathematical performance.LiveCodeBench collects new problems over time from LeetCode, AtCoder, and CodeForces contests.Source: Table 20 · p. 68, Table 32 · p. 78
  • DROP tests extended-passage understanding, IFEval checks explicit instruction compliance, and SimpleQA and Chinese-language C-SimpleQA assess precise, verifiable factual answers.DROP requires interpreting context-rich passages rather than merely recalling facts; IFEval checks whether outputs satisfy multiple clearly defined constraints.Source: Table 22 · p. 70, Table 23 · p. 71, Table 30 · p. 77
  • FRAMES uses its official Oracle Prompt configuration, supplying all ground-truth Wikipedia articles to evaluate reasoning and factual synthesis without external retrieval.This setting measures whether models can synthesize provided sources into correct, verifiable facts.Source: Table 24 · p. 72, J. Evaluation Prompts and Settings · p. 72
  • Arena-Hard and AlpacaEval 2.0 use model-based judging for open-ended tasks, with Arena-Hard emphasizing challenging coding and mathematics prompts.AlpacaEval 2.0 generally uses less challenging prompts, and only a small subset requires reasoning.Source: Table 25 · p. 73, J. Evaluation Prompts and Settings · p. 73, Table 26 · p. 74

Related papers

Loading 2501.12948v2…