DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving

1 Introduction

• Mathematical reasoning is a major challenge for state-of-the-art LLMs, motivating the paper's focus on mathematical problem-solving.The authors situate instruction tuning as a cost-effective approach to improving pretrained models' mathematical reasoning.Source: §1 · p. 1
• 21.2% to 46.6% on MATH and 51.0% to 82.5% on GSM8K for Llama3-8B accompany DART-Math's gains over VRT and leading public datasets across six benchmarks.DART-Math uses 590k examples, the same training-example count as VRT; the compared public datasets use 395k and 2.3 million examples.Source: Figure 1 · p. 2, §1 · p. 2
• Existing rejection-sampled datasets favor easy queries; MetaMathQA generated no new responses for 51.1% of the hardest MATH training queries.The authors hypothesize that this bias may hinder learning because difficult examples are considered more crucial for training.Source: §1 · p. 2
• DART allocates more sampling trials to challenging queries, using Uniform to collect equal correct-response counts per query and Prop2Diff to bias samples toward difficult queries.The datasets use about 590k examples and are synthesized with DeepSeekMath-7B-RL rather than a proprietary model.Source: §1 · p. 2

2.1 Background: Rejection Sampling and Rejection Tuning

• Rejection sampling draws multiple responses per query and filters incorrect answers against the ground truth; fine-tuning on the retained responses is rejection tuning.Queries are often kept constant or altered in a controlled manner so ground-truth answers remain available for filtering.Source: §2.1 · p. 3

2.2 On the Imbalance of Rejection-Based Data Synthesis

• The analysis examines MetaMathQA-MATH-AnsAug, a rejection-sampled subset of MetaMathQA built from original MATH training queries.MATH is used because it is challenging and has human-annotated difficulty levels.Source: §2.2 · pp. 3–4
• On MATH, MetaMathQA's Level 5 query share decreases by 51.1%, and those queries account for only 10.5% of synthetic responses.The authors report that this bias is less pronounced on relatively simple datasets such as GSM8K.Source: §2.2 · p. 4

3.1 Open-Weight Models Are Able to Generate Good Responses

• 90% pass@k accuracy on MATH500 is achieved by DeepSeekMath-7B-RL when sampling over 100 responses per query.The authors use this result to support the potential of strategic sampling to address insufficient hard-query coverage.Source: §3.1 · p. 4

3.2 DARS — Difficulty-Aware Rejection Sampling

• DARS-Uniform samples until each query has a preset number of correct responses, while DARS-Prop2Diff makes that number proportional to difficulty.The most challenging queries receive a preset target, and sampling stops at a per-query cap on raw samples.Source: §3.2 · p. 4
• Prop2Diff guarantees at least one synthetic response per query, while DARS explicitly controls the dataset distribution and targets higher coverage of the hardest queries.The authors contrast this with ToRA and MARIO, which focus on coverage without explicitly managing the distribution.Source: §3.2 · p. 5
• DART measures query difficulty by fail rate, the proportion of incorrect responses among a fixed number of samples.The authors use DeepSeekMath-7B-RL to assess difficulty and note that matching the difficulty-evaluation model to the generative model may further improve performance.Source: §3.2 · p. 5

3.3 The DART-Math Datasets

• DART-Math-Uniform and DART-Math-Hard each contain about 590k samples, built from GSM8K and MATH training queries using DeepSeekMath-7B-RL.The queries are kept fixed, the data uses natural-language reasoning without external tools, and the authors describe the datasets as generally smaller than most previous datasets.Source: §3.3 · p. 5
• DART typically needs more sampling trials than vanilla rejection sampling for a comparable-size dataset, but its overall training cost does not exceed vanilla instruction tuning.The authors characterize synthesis as a one-time cost that can be amortized across training runs and users of the public dataset.Source: §3.3 · pp. 5–6

4.1 General Setup

• Synthetic MATH and GSM8K responses from DeepSeekMath-7B-RL were instruction-tuned on four base models and evaluated across six in- and out-of-domain benchmarks without external tools.The evaluation used greedy decoding.Source: §4.1 · p. 6

4.2 Main Results

• Across six benchmarks, Llama3-8B Prop2Diff averages +4.5 points over VRT, with nearly +7 on MATH for both Mistral-7B and Llama3-8B.DART-Math based on all four base models outperforms VRT on most benchmarks; gains are smaller for DeepSeekMath-7B and Llama3-70B.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Analysis

• MATH performance improves steadily with larger training sets; DART scales better than VRT on Mistral-7B and Llama3-8B, but approaches are similar on DeepSeekMath-7B.Scaling DeepSeekMath-7B to one million samples yields only a 3-point gain, compared with gains above 20 points for Mistral-7B and Llama3-8B.Source: Figure 3 · p. 8, §4.3 · p. 8

4.2 Main Results

• DART-Math beats MetaMath in all reported cases and reaches 7–8B state of the art on MATH, OlympiadBench-Math, and TheoremQA.DART-Math-Mistral-7B Prop2Diff surpasses Mistral-7B-MMIQC by 4.6 points using a quarter of its training samples.Source: §4.2 · p. 8

4.3 Analysis

• At small data sizes, one-response coverage adds about 8 points on GSM8K, while adding coverage to VRT does not significantly improve MATH.As training data grows, the difference between Prop2Diff with and without the coverage constraint diminishes.Source: Figure 4 · p. 9, §4.3 · p. 9
• For 585k examples, reaching a 90% query target costs about 5M raw samples for DARS-Uniform (nmax=2048) versus over 15M for DARS-Prop2Diff (nmax≥8K).At over 95% query coverage, synthesis used about 150M samples and 160 NVIDIA A100 GPU days; the authors describe this as a one-time cost that can be amortized.Source: §4.3 · p. 9

• The paper identifies vanilla rejection tuning’s easy-query bias and proposes DART as a remedy for mathematical instruction tuning.Source: §5 · p. 9
• DART datasets and models are open-source and do not rely on proprietary models; experiments across base models and benchmarks demonstrate the approach’s effectiveness.Source: §5 · pp. 9–10
• Fail rate may be a suboptimal difficulty metric, and DART-Math is limited to natural-language reasoning, although code-generation integration is proposed for future exploration.Source: §5 · p. 10

• ToRA and MARIO are outside the direct comparison because they use code alongside natural language and have not released their datasets.Source: A Comparison to Methods Based on… · p. 15
• The paper implements natural-language versions of their data-synthesis strategies for comparison with DARS.Source: A Comparison to Methods Based on… · p. 15
• The comparison is limited to those natural-language implementations, which the authors describe as comparable with DARS.Source: A Comparison to Methods Based on… · p. 15

A.1 DARS produces distributions not biased towards easy queries

• DARS distributes responses uniformly or toward harder queries, while ToRA/MARIO focus on coverage without explicitly managing difficulty distribution.ToRA/MARIO datasets may therefore remain biased toward easy queries.Source: §A.1 · p. 15
• ToRA/MARIO produce fewer responses for difficult problems than easy ones, whereas DART-Math-Hard produces 10x more for MATH Level 5 than GSM8K queries.Absolute response counts are not directly comparable across methods; the passage compares their distributions.Source: §A.1 · p. 15
• High coverage alone does not guarantee superior performance.Table 3 compares datasets synthesized by non-vanilla rejection-sampling methods; RPQ means average responses per query.Source: §A.1 · p. 15, Table 3 · p. 15

A.2 DARS achieves high coverage even on the hardest queries

• MetaMathQA-MATH-AnsAug reaches 82.8% overall MATH training-set coverage yet still misses a significant portion of hard queries and skews toward easy ones.The dataset used evenly allocated budgets.Source: §A.2 · p. 15
• Overall MATH training-set coverage does not establish that hard queries are well covered.The reported coverage comparison is broken out by difficulty level.Source: §A.2 · p. 15
• DART-Math datasets cover 99.6% of Level 5 MATH queries, compared with 68% for ToRA-Corpus-16k.Table 4 reports coverage rates across difficulty levels; DART-Math-* datasets share coverage because of the Cover operation.Source: §A.2 · p. 15

A.3 Details of Re-implementing Data Synthesis Strategies of ToRA and MARIO

• The ToRA-Corpus-16k replication samples 10 trials for problems not answered correctly by greedy decoding and keeps up to four correct responses per problem.The replication uses GPT-4o mini for these sampling steps.Source: §A.3 · p. 16
• For the ToRA-style pipeline, DeepSeekMath-7B-RL replaces trained CodeLlama models to avoid additional training and align with DART-Math.The adaptation samples 733k distinct correct responses, then re-samples up to 64 trials per problem for incorrect responses, yielding 225k correct samples.Source: §A.3 · p. 16
• The adapted ToRA pipeline forms a final dataset of 72k responses, following a merge step that forms 69k responses in the described method.The original final dataset combines ToRA-Corpus-16k with data from the preceding step.Source: §A.3 · p. 16
• The replication substitutes GPT-4o mini for GPT-4 in the early greedy and rejection-sampling steps.It also omits MetaMath’s new-problem step because it could skew the response distribution toward easy queries.Source: §A.3 · p. 16
• The adaptation of ToRA’s high-trial step achieves 91.3% MATH coverage, versus 93.8% in the described method.The described step samples 100 trials and keeps up to four correct responses for remaining unanswered MATH queries.Source: §A.3 · p. 16

B.1 Training Setup

• Training uses sequence packing, 4096-token computation sequences, and batch size 64, with gradient accumulation enabled when memory requires it.Llama3-8B training on eight A100 GPUs uses two accumulation steps under the stated setting.Source: §B.1 · p. 17
• Training uses Adam with zero weight decay, linear warmup and cosine scheduling; default training is three epochs, but Llama3 models train for one.Learning rates vary by base model, and Llama3’s one-epoch choice follows preliminary experiments in which it consistently outperformed three epochs.Source: §B.1 · p. 17

B.2 Synthesis Setup

• Synthesis uses top-p sampling at p = 0.95 and temperature t = 1.6, selected after searching temperatures from 0.3 to 1.8.Higher temperatures produced more diverse responses, while temperatures at or above 1.7 caused nonsense outputs.Source: §B.2 · pp. 17–18
• An answer-extraction and grading pipeline uses regular expressions and SymPy to judge raw-response correctness across varied mathematical objects and special-text answers.The authors say it can correctly process most such objects and texts.Source: §B.2 · p. 18
• For efficiency, the authors combine DARS-Uniform synthesis with fail-rate calculation, using ku = 192 to build the data pool.Fail rates are calculated from the sampled responses; the appendix notes that the more accurate calculation is not necessary for general algorithm implementations.Source: §B.2 · p. 18

B.3 Evaluation Setup

• SFT and RL models are evaluated zero-shot, while base models use benchmark-specific few-shot prompts.The base-model prompts use 4-shot examples for MATH, GSM8K, CollegeMath, DeepMind Mathematics, and OlympiadBench-Math, and 5-shot for TheoremQA.Source: §B.3 · p. 18
• Evaluation uses greedy decoding but runs each evaluation with at least three random seeds because vLLM may remain stochastic.The setup uses vLLM with top-p sampling at p = 0.95; evaluation sets temperature t = 0.Source: §B.3 · p. 18
• OlympiadBench grading uses its official correctness component but retains the paper’s answer extraction because the official extractor misses a non-negligible share of answers.The issue is noted especially for base-model in-context-learning outputs.Source: §B.3 · p. 18

C.1 Domain-wise Performance on MATH

• DART improves domain-wise and domain-macro-average MATH performance for rejection-tuned Mistral-7B and Llama3-8B models.Table 6 reports results across seven domains and gives macro averages with equal domain weights and micro averages with equal query weights.Source: §C.1 · p. 18, Table 6 · p. 19

C.2 DART achieves comparable performance with RL

• Sole SFT with DART achieves performance comparable to DeepSeekMath-7B-RL, although DART is an SFT method and the RL method is not usually directly comparable.Table 7 presents the comparison on DeepSeekMath-7B.Source: §C.2 · p. 19, Table 7 · p. 19

D Related Work

• DART addresses the easy-query bias of rejection-sampling synthesis by adaptively allocating candidate samples to control the final training distribution.Prior work often samples the same number of candidates per query regardless of difficulty.Source: D Related Work · p. 19
• Unlike data-selection approaches that do not consider the final training distribution, DART focuses on difficulty and controls that distribution.The related-work discussion frames this as a perspective for data selection and construction.Source: D Related Work · p. 19

NeurIPS Paper Checklist

• The authors state that experimental details are disclosed, and that datasets, models, and code are publicly available.The checklist points to Section 3 and Appendix B for experimental details and identifies the assets as publicly available.Source: NeurIPS Paper Checklist · pp. 21–22
• The checklist reports no error bars because repeated LLM experiments are too expensive, while describing the work as mathematical-data synthesis with little societal impact.The authors also state that limitations are discussed and that the work uses common public mathematical datasets.Source: NeurIPS Paper Checklist · pp. 20, 22, 23