Source-linked AI summary
DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu
TL;DR
Existing training data constrains RL progress because it is often insufficiently difficult, contaminated with benchmarks, and lacking verifiable answers. DeepMath-103K addresses these gaps with a curated mathematical dataset containing challenging, decontaminated problems, verifiable answers, and diverse solutions; models trained on it achieve new state-of-the-art results on many mathematical benchmarks.
Problem
Existing mathematical training datasets often lack sufficient difficulty, benchmark decontamination, and verifiable answers needed for advancing RL-based reasoning.
Method
DeepMath-103K curates diverse mathematical sources into structured problems, adding verifiable final answers and three distinct R1 solutions for RLVR and SFT.
Results
Models trained on DeepMath-103K achieve new state-of-the-art results on many mathematical benchmarks, including 64.0 pass@1 on AIME24 for DeepMath-Omn-1.5B and 57.3 on AIME25.
Takeaways & Limitations
DeepMath-103K provides a large-scale resource for training reasoning models through RLVR and supports diverse training paradigms with verifiable answers and multiple solutions.
Takeaways & Limitations
Some open-ended questions lack easily verifiable final answers, while complex answers can be difficult or infeasible to verify automatically.
Abstract
from arXiv · showhide
Reinforcement learning (RL) with large language models shows promise in complex reasoning. However, its progress is hindered by the lack of large-scale training data that is sufficiently challenging, contamination-free and verifiable. To this end, we introduce DeepMath-103K, a large-scale mathematical dataset designed with high difficulty (primarily levels 5-9), rigorous decontamination against numerous benchmarks, and verifiable answers for rule-based RL reward. It further includes three distinct R1 solutions adaptable for diverse training paradigms such as supervised fine-tuning (SFT). Spanning a wide range of mathematical topics, DeepMath-103K fosters the development of generalizable and advancing reasoning. Notably, models trained on DeepMath-103K achieve state-of-the-art results on challenging mathematical benchmarks and demonstrate generalization beyond math such as biology, physics and chemistry, underscoring its broad efficacy. Data: https://huggingface.co/datasets/zwhe99/DeepMath-103K.
1 Introduction
DeepMath-103K addresses limitations in existing RL training data by combining challenging, decontaminated mathematical problems with verifiable answers and diverse solutions. Models trained on the dataset achieve strong mathematical benchmark results and generalize to biology, physics, and chemistry.
- Motivation: Existing RL training datasets often lack sufficient difficulty, benchmark decontamination, and verifiable answers for advanced reasoning.These limitations constrain progress in training reasoning models with RL and verifiable rewards.
- Dataset design: DeepMath-103K targets RLVR with a high concentration of challenging problems, especially at difficulty levels ≥ 5.Its difficulty distribution is skewed toward higher levels compared with existing open resources.
- Dataset design: The dataset applies rigorous decontamination against a comprehensive suite of standard benchmarks to support trustworthy evaluation.This addresses potential overlap between training data and evaluation benchmarks.
- Dataset design: Every problem includes a rule-verifiable final answer and three distinct R1 solutions, supporting RLVR and training paradigms such as SFT.The answers are validated for extraction and rule-based verification.
- Data curation: DeepMath-103K transforms informal discussions from sources such as Math StackExchange into structured QA data with greater problem variety and diversity.This acquisition strategy differs from simply recombining common, well-formatted benchmark problems.
- Results: DeepMath-Zero-(Math)-7B improves pass@1 by +12.7 (+23.0) on AIME24 and +12.1 (+19.1) on AIME25, establishing new SOTA performance.The model starts from Qwen-2.5-(Math)-7B without RL training.
- Results: DeepMath-Omn-1.5B reaches pass@1 accuracies of 64.0 on AIME24 and 57.3 on AIME25, surpassing o1-mini on AIME24.The comparison reports o1-mini at 63.6 on AIME24.
- Results: DeepMath models achieve best GPQA-Diamond scores on biology, physics, and chemistry relative to the baselines.The reported generalization extends beyond mathematical evaluation.
2 Overview of DeepMath-103K
DeepMath-103K is curated from diverse mathematical sources into a challenging, decontaminated dataset with broad topic coverage, unique problems, and verifiable answers for RLVR.
- Topical diversity: DeepMath-103K covers mathematical topics from Prealgebra and Plane Geometry to Abstract Algebra, including Group Theory and Field Theory.
- Data novelty and uniqueness: 82.81K problems in DeepMath-103K are not found in other datasets under the embedding-based uniqueness analysis.Samples with embedding similarity greater than 0.98 were treated as the same.
- Curation pipeline: The curation pipeline starts with 2,869K raw questions, retains 95K after decontamination, difficulty filtering at level ≥5, and answer verification, then adds 8K SimpleRL problems.
- Curation cost: The pipeline required $138,000 in GPT-4o API fees and 127,000 H20 GPU hours.
- Decontamination: Semantic decontamination compares each candidate with five retrieved benchmark examples and uses an LLM judge to remove duplicates or paraphrases.The approach targets near-duplicates and paraphrased questions beyond exact lexical overlap.
- Answer verification: Every retained problem has a robustly verifiable final answer produced through question standardization and consistency checking across three generated solutions and an original solution when available.All extracted final answers must be identical for retention.
4 DeepMath Series Models
DeepMath-103K supports both zero-RL training from base models and RLVR fine-tuning of instruction-tuned models, with gains across mathematics and broader STEM reasoning.
- Zero RL training: DeepMath-Zero-7B and DeepMath-Zero-Math-7B achieve new SOTA results on all evaluated mathematical benchmarks after zero-RL training from base models.
- RL on instruction-tuned models: DeepMath-Omn-1.5B attains new SOTA results among 1.5B-scale models on all evaluated benchmarks and surpasses o1-mini and o3-mini at low effort.
- RL on instruction-tuned models: DeepMath-1.5B reaches 82.3% on AMC23 and 61.8% on OlympiadBench after RLVR fine-tuning.
- Beyond mathematics: DeepMath series models achieve superior performance on GPQA-Diamond, covering biology, physics, and chemistry, indicating reasoning generalization beyond mathematics.
- Training dynamics: Figure 9 tracks rollout response length and four cognitive behaviors during DeepMath-Zero-7B training, both of which increase over training.
5 Related Work
Mathematical reasoning datasets support different post-training stages, including continued pre-training, supervised fine-tuning, and reinforcement learning.
- Post-training stages: Existing mathematical reasoning datasets span CPT, SFT, and RL, with CPT injecting mathematical knowledge and SFT providing problems with step-by-step solutions.
6 Conclusion
DeepMath-103K is introduced as a challenging, decontaminated, and verifiable dataset for RLVR-based reasoning research. Models trained on it achieve new state-of-the-art results on many math benchmarks and generalize beyond mathematics.
- DeepMath-103K combines challenging problems, rigorous decontamination, verifiable final answers, and multiple diverse solutions.
- Its curation pipeline uses less structured mathematical forums to produce greater problem novelty and diversity than existing resources.
- Models trained on DeepMath-103K achieve new state-of-the-art results on many mathematical benchmarks.
- The trained models exhibit remarkable generalization to domains beyond mathematics.
- The released dataset, code, and model weights provide a platform for further research on advanced reasoning.
A Contamination Analysis of Existing Datasets
The contamination analysis examines overlap between several training datasets and the commonly used MATH500 benchmark using normalized indel similarity.
- The analysis compares ORZ-129K, DSR-Preview, DAPO-17K, Open-RS, Open-R1, and DeepMath-103K for potential contamination.
- MATH500 serves as the benchmark for detecting possible contamination in the compared datasets.
- The study uses string-based comparison based on normalized indel similarity between problems.
B SFT Results
SFT on DeepMath-103K improves Qwen-2.5-7B performance across all evaluated benchmarks, with further gains from using multiple solutions, although it remains behind RL.
- SFT on DeepMath-103K with one R1 solution per problem significantly improves base-model performance across all benchmarks.
- Using all three R1 solutions yields further gains over using one solution per problem.
- SFT still lags behind RL in the reported comparison.
- Table 5 reports math reasoning performance after fine-tuning Qwen-2.5-7B via SFT.
- DeepMath-Zero-7B is included as an RL counterpart for reference.
C Training Details
The paper uses verl as its training framework and reports the DeepMath series training configurations in Table 6.
- verl is used as the training framework for the DeepMath series models.
- The training configurations are presented as implementation details for the DeepMath series.
- Table 6 lists the configurations used to train the DeepMath series models.
D Licenses for Existing Assets
Table 7 lists the licenses for existing assets.
- Table 7 presents licenses for existing assets.
E Limitations and Broader Impacts
DeepMath-103K has documented limitations alongside broader research benefits. Its difficulty assessment may be biased, topical diversity may be unbalanced, and creation was computationally intensive, while public release may support RL reasoning research and more generalizable AI.
- Difficulty assessment relies on LLM evaluations, potentially introducing bias.
- Topical diversity may not be perfectly balanced, and dataset creation was computationally intensive.
- Public release may lower barriers for RL reasoning research, improve benchmark reliability, and foster more generalizable AI.