PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo

arXiv:2402.03300v3cs.CLcs.AIcs.LG

TL;DR

Open-source language models lag behind closed models in mathematical reasoning, motivating a stronger publicly available model. DeepSeekMath combines web-mined mathematical pretraining data with GRPO, achieving competitive benchmark performance and improving MATH results to 51.7%.

  • Problem

    Open-source language models considerably trail closed models in mathematical reasoning performance, despite advances in mathematical reasoning benchmarks.

  • Method

    DeepSeekMath uses a 120B-token Common Crawl math corpus and Group Relative Policy Optimization, which omits the critic model to reduce training resources.

  • Results

    DeepSeekMath surpasses all evaluated open-source base models on English benchmarks and reaches 51.7% on MATH after reinforcement learning.

  • Takeaways & Limitations

    The study shows that web pages can provide high-quality mathematical data and that GRPO can improve reasoning with lower memory consumption.

  • Takeaways & Limitations

    DeepSeekMath is relatively weaker than closed models on geometry and theorem-proving tasks and performs worse than GPT-4 in few-shot evaluation.

Abstract

from arXiv · show

Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.

1. Introduction

DeepSeekMath is introduced as a domain-specific model built through scalable mathematical pre-training on a 120B-token Common Crawl corpus and reinforcement-learning exploration. Its approach combines mathematical instruction tuning with GRPO, while evaluations cover English and Chinese mathematical reasoning benchmarks and broader reasoning capabilities.

  • Scalable mathematical pre-training: 120B math tokens form the DeepSeekMath Corpus, extracted from Common Crawl through a fastText-based classifier.The corpus is presented as a large-scale, high-quality pre-training resource filtered for mathematical content.
  • Model development: DeepSeekMath-Base is initialized from DeepSeek-Coder-Base-v1.5 7B, and mathematical training also improves MMLU and BBH performance.The paper reports that code-model initialization is preferable to starting from a general language model.
  • Instruction tuning: Mathematical instruction tuning uses chain-of-thought, program-of-thought, and tool-integrated reasoning data; DeepSeekMath-Instruct 7B beats all 7B counterparts.The resulting model is also described as comparable with 70B open-source instruction-tuned models.
  • Reinforcement learning: GRPO removes the critic model by estimating the baseline from group scores, significantly reducing training resources relative to PPO.The method is introduced as a variant of Proximal Policy Optimization and improves the instruction-tuned model using instruction-tuning data.
  • Evaluation: The evaluation covers English and Chinese mathematical benchmarks spanning grade-school through college-level problems, including MATH, GSM8K, CMATH, and Gaokao tasks.The models are evaluated both on self-contained text solutions without tool use and on their ability to use tools.

2. Math Pre-Training

DeepSeekMath constructs a large mathematical corpus from Common Crawl through an iterative, quality-focused selection pipeline and shows strong downstream reasoning performance. The resulting corpus is high-quality, multilingual, and larger than existing alternatives, while DeepSeekMath-Base 7B leads open-source base models across eight benchmarks.

  • Corpus construction: An iterative pipeline systematically collects mathematical web pages from Common Crawl, beginning with a small, high-quality seed corpus.The approach is also applicable to other domains, including coding.
  • Corpus construction: OpenWebMath seeds a fastText retriever using 500,000 positive and 500,000 negative examples to recall similar mathematical web pages.The pipeline also deduplicates Common Crawl and ranks recalled pages by fastText scores to filter low-quality content.
  • Corpus construction: Additional mathematical web sources enrich the seed corpus after the first iteration because limited seed diversity leaves many pages uncollected.This enrichment is used to optimize the fastText model through domain-level analysis of Common Crawl.
  • Corpus evaluation: The DeepSeekMath Corpus is high-quality, multilingual, and the largest among the compared mathematical corpora.It leads on eight mathematical benchmarks, improves reasoning in both English and Chinese, and shows higher average quality than Proof-Pile-2 at 50B tokens.
  • DeepSeekMath-Base 7B: DeepSeekMath-Base 7B leads open-source base models across all eight benchmarks and surpasses them by over 10% absolute on MATH.It also outperforms Minerva 540B on the competition-level MATH dataset.

3. Supervised Fine-Tuning

DeepSeekMath-Instruct 7B is instruction-tuned on a 776K-example mathematical dataset spanning English and Chinese problems, diverse fields, complexity levels, and reasoning formats. Without tools, it achieves strong step-by-step reasoning on MATH; with natural-language and program-based tools, it approaches 60% accuracy on MATH and surpasses existing open-source models.

  • 3. Supervised Fine-Tuning: 776K training examples cover English and Chinese mathematical problems from different fields and complexity levels.The dataset pairs problems with solutions in chain-of-thought, program-of-thought, and tool-integrated reasoning formats.
  • 3. Supervised Fine-Tuning: The English collection includes GSM8K, MATH, MathInstruct, and Lila-OOD data using tool-integrated solutions, CoT, or PoT.It spans algebra, probability, number theory, calculus, and geometry.
  • 3. Supervised Fine-Tuning: 500 training steps use batch size 256, constant learning rate 5e-5, and a maximum context length of 4K tokens.Training examples are randomly concatenated until reaching the context limit.
  • 3. Supervised Fine-Tuning: At least 9% absolute improvement on MATH separates DeepSeekMath-Instruct 7B from the majority of proprietary and all open-source models when tools are disallowed.This advantage holds even against substantially larger or math-focused reinforcement-trained models.
  • 3. Supervised Fine-Tuning: Approaching 60% accuracy on MATH with natural-language reasoning and program-based tool use, the model surpasses all existing open-source models.On other benchmarks, it is competitive with DeepSeek-LLM-Chat 67B, which is 10 times larger.

4. Reinforcement Learning

This section introduces Group Relative Policy Optimization (GRPO), an efficient RL algorithm for improving mathematical reasoning after supervised fine-tuning. GRPO removes PPO’s value model by estimating advantages from relative rewards within output groups, reducing training resources while supporting outcome and process supervision.

  • Motivation and contribution: RL further improves LLM mathematical reasoning after supervised fine-tuning, motivating the introduction of GRPO.GRPO is presented as an efficient and effective RL algorithm.
  • GRPO versus PPO: GRPO foregoes PPO’s value model and estimates the baseline from group scores, significantly reducing training resources.PPO’s value function adds substantial memory and computational burden because it is typically another model comparable in size to the policy model.
  • Group-relative advantage estimation: For each question, GRPO samples a group of outputs, scores them, normalizes rewards relative to the group, and uses those relative rewards to calculate token advantages.With outcome supervision, every token in an output receives its normalized final reward as the advantage.
  • Outcome and process supervision: Process supervision assigns rewards at the ends of reasoning steps and computes each token’s advantage from normalized rewards of subsequent steps.This addresses the limitation that outcome supervision provides reward only at the end of an output, which may be insufficient for complex mathematical tasks.
  • Results: 88.2% and 51.7% are DeepSeekMath-RL 7B’s GSM8K and MATH accuracies with chain-of-thought reasoning, surpassing all open-source models from 7B to 70B and most closed-source models.The RL training uses chain-of-thought-format GSM8K and MATH questions from the SFT data, totaling around 144K questions.

5. Discussion

The discussion finds that code training improves mathematical reasoning with and without tool use, whereas arXiv-only training is ineffective in the tested settings. It also shows that online and iterative reinforcement-learning methods improve performance, while data, algorithm, and reward design remain important limitations.

  • Code training: Code training improves mathematical reasoning both with and without tool use.Under two-stage training, code training enhances reasoning and subsequent math training yields further gains; mixing code and math tokens helps with tool use but compromises reasoning without tools.
  • ArXiv data: ArXiv-only training produces no notable mathematical improvements and can cause deterioration across tested benchmarks.This finding was observed for DeepSeek-LLM 1.3B trained for 150B tokens and DeepSeek-Coder-Base-v1.5 7B trained for 40B tokens on separate arXiv corpora.
  • Reinforcement learning: Online RFT significantly outperforms RFT on two benchmarks, with its advantage emerging later in training.The methods are comparable early, when actor and supervised fine-tuning models resemble each other more closely.
  • Reinforcement learning: GRPO surpasses online RFT, while GRPO+PS outperforms GRPO+OS and two rounds of iterative RL further improve performance.The discussion attributes these gains to altered positive and negative gradient coefficients and fine-grained, step-aware coefficients.
  • Reinforcement learning: RL improves Maj@K accuracy but not Pass@K, indicating that it enhances overall output quality rather than the set of sampled solutions.The comparison is between instruction-tuned and RL models on two benchmarks.
  • Limitations: RL remains constrained by limited question and output sampling, unreliable reward signals, and reward models’ need for stronger out-of-distribution generalization.The paper identifies data-source, algorithm, and reward-function design as important directions for improvement.

6. Conclusion, Limitation, and Future Work

DeepSeekMath outperforms open-source models on competition-level MATH and approaches closed-model performance through continued training with substantial Common Crawl mathematics data. Its limitations include weaker geometry and theorem-proving performance and inferior few-shot capability compared with GPT-4.

  • Conclusion: DeepSeekMath outperforms all open-source models on the competition-level MATH benchmark and approaches closed-model performance.
  • Conclusion: 120B math tokens from Common Crawl form a significant component of DeepSeekMath’s 500B-token continual-training corpus.The model is initialized with DeepSeek-Coder-v1.5 7B.
  • Conclusion: The ablation study finds that web pages offer substantial potential for high-quality mathematical data, whereas arXiv may be less beneficial than expected.
  • Limitation: DeepSeekMath is relatively weaker than closed models on geometry and theorem-proof tasks.The model cannot handle some triangle- and ellipse-related problems, possibly indicating data-selection bias in pre-training and fine-tuning.
  • Limitation: DeepSeekMath performs worse than GPT-4 on few-shot capability because of its model scale.GPT-4 could improve its performance with few-shot input.

A. Appendix · A.1. Analysis of Reinforcement Learning

The appendix analyzes reinforcement-learning methods by deriving their data sources, gradient coefficients, algorithms, and reward functions. It covers SFT, RFT, Online RFT, DPO, PPO, and GRPO.

  • A. Appendix: The appendix provides detailed derivations for reinforcement-learning data sources and gradient coefficients.It frames these derivations as part of the algorithm and reward-function analysis.
  • A.1. Analysis of Reinforcement Learning: The analysis covers algorithms and reward functions across SFT, RFT, Online RFT, DPO, PPO, and GRPO.These methods are included in the comparative reinforcement-learning analysis.

A.1.1. Supervised Fine-tuning

Supervised fine-tuning is formulated as maximizing an objective, with its gradient specified explicitly. The method uses a dataset for SFT, treats the reward function as human selection, and fixes the gradient coefficient at 1.

  • Supervised Fine-tuning: Supervised fine-tuning maximizes a specified objective.The objective is denoted J_SFT(θ) in the supplied passage.
  • Supervised Fine-tuning: The gradient of J_SFT(θ) is defined explicitly.The supplied passage introduces the gradient formulation without including its full expression.
  • Supervised Fine-tuning: The SFT dataset provides the training data, human selection serves as the reward function, and the gradient coefficient is fixed at 1.The gradient coefficient is always set to 1.

A.1.2. Rejection Sampling Fine-tuning

Rejection Sampling Fine-tuning samples multiple outputs from a supervised fine-tuned LLM for each question, then trains on sampled outputs paired with correct answers. It uses answer correctness as a rule-based reward, assigning different gradient coefficients to correct and incorrect outputs.

  • Method: Rejection Sampling Fine-tuning samples multiple outputs from the supervised fine-tuned LLM for each question and trains on outputs with the correct answer.Its objective is to maximize the stated RFT objective.
  • Method: The method uses a rule-based reward function that determines whether each sampled answer is correct.The data source is questions from the SFT dataset with outputs sampled from the SFT model.
  • Method: Correct answers receive gradient coefficient 1, whereas incorrect answers receive coefficient 0.These coefficients are specified for the sampled output o.

A.1.3. Online Rejection Sampling Fine-tuning

Online RFT differs from RFT by sampling outputs from the real-time policy model π_θ rather than the SFT model π_θ^sft, which determines its gradient.

  • Online sampling: Online RFT samples outputs from the real-time policy model π_θ.This is the defining sampling choice for Online RFT.
  • Online sampling: RFT samples outputs from the SFT model π_θ^sft, unlike Online RFT.The passage identifies this as the only difference between the two procedures.
  • Gradient: The sampling-policy distinction determines the gradient of online RFT.The passage introduces the online RFT gradient immediately after specifying the sampling difference.

A.1.4. Direct Preference Optimization (DPO) · A.1.5. Proximal Policy Optimization (PPO) · A.1.6. Group Relative Policy Optimization (GRPO)

The section presents DPO, PPO, and GRPO through their objectives, gradients, data and reward sources, and gradient coefficients. It also specifies PPO’s advantage estimation and GRPO’s group-based advantage computation.

  • A.1.4. Direct Preference Optimization (DPO): DPO defines an optimization objective for preference-based training.The passage introduces the DPO objective without displaying its equation.
  • A.1.4. Direct Preference Optimization (DPO): DPO’s gradient is explicitly derived from its objective.The passage introduces the gradient of J_DPO(θ).
  • A.1.4. Direct Preference Optimization (DPO): DPO samples outputs from the SFT model and uses human preference, or a mathematical-task rule, as its reward.Its gradient coefficient is also specified in the section.
  • A.1.5. Proximal Policy Optimization (PPO): PPO defines a clipped policy-optimization objective for its training procedure.The passage introduces the PPO objective without displaying its equation.
  • A.1.5. Proximal Policy Optimization (PPO): Under a single-update assumption where π_θold = π_θ, PPO’s min and clip operations can be removed for analysis.This simplification applies after each exploration stage.
  • A.1.5. Proximal Policy Optimization (PPO): PPO samples policy-model outputs, uses a reward model, and derives its gradient from the objective.The section also specifies a gradient coefficient.
  • A.1.5. Proximal Policy Optimization (PPO): PPO computes the advantage with Generalized Advantage Estimation using future rewards and a learned value function V_ψ.The rewards are denoted {r≥t}.
  • A.1.6. Group Relative Policy Optimization (GRPO): GRPO defines an objective and gradient using policy-model outputs, a reward model, and advantages computed from group reward scores.Its simplified objective assumes π_θold = π_θ, and the section specifies a gradient coefficient.
Loading 2402.03300v3…