Common 7B Language Models Already Possess Strong Math Capabilities

1 Introduction

• 97.7% GSM8K and 72.0% MATH accuracy with best-of-256 generations contrast with single-generation accuracy of 49.5% and 7.9%, respectively.The authors interpret the gap as instability in eliciting correct answers, rather than a lack of underlying capability.Source: Figure 1 · p. 1, §1 · p. 2
• Scaling SFT data improves accuracy, but the supply of publicly accessible real math questions constrains continued scaling.The introduction says accuracy improves nearly linearly or super-linearly with exponentially increased SFT data and has not plateaued on available data.Source: §1 · p. 2
• GPT-4 Turbo generates new questions from preference questions, then a GPT-4 Turbo-based verifier checks them; synthetic questions perform nearly on par with real ones.The authors present this “brand-new” generation strategy as effective for scaling SFT data.Source: §1 · p. 2
• 82.6% GSM8K and 40.6% MATH accuracy make the synthetic-SFT-scaled LLaMA-2 7B the first to exceed 80% and 40% on these benchmarks.The paper reports these results after scaling SFT data, including synthetic questions.Source: §1 · p. 2
• As SFT data grows, accuracy tends to plateau with 256 attempts but increases markedly for one response, pointing to greater answer-generation stability.The paper also reports power-law accuracy scaling with chain-of-thought steps, improved reliability at each reasoning step with more SFT data, and further gains on hard questions from resampling longer-CoT examples; calculation errors are more readily mitigated than reasoning errors.Source: §1 · p. 2

• DeepSeek-MATH-7B also surpasses 80% accuracy, but uses a stronger math-pre-trained base model and a sophisticated RL algorithm; the authors describe the results as complementary.Source: §2 · p. 2
• Pass@N measures whether any of N random generations is correct, while PassRatio@N measures the share that are correct.The generation temperature is 0.73 to increase diversity.Source: §2 · p. 3
• 97.7% on GSM8K and 72.0% on MATH are the LLaMA-2 7B model’s Pass@256 accuracies.Pass@256 counts a question as solved when at least one of 256 random generations is correct.Source: §2 · p. 3
• 48.2% on GSM8K and 7.9% on MATH are its PassRatio@256 accuracies, indicating that correct answers are not consistently extracted.PassRatio@N measures the percentage of correct answers among N generations.Source: §2 · p. 3
• MATH evaluation uses 500 test samples for efficiency, and the models are evaluated in instruction-following settings using SFT versions trained on 7.5K examples.The authors cite prior work stating that SFT does not enhance capabilities and may reduce them through “alignment taxes”.Source: §2 · p. 3

3 Scaling SFT Data using Synthetic Math Questions

• 26.7% to 50.2% on GSM8K and 4.2% to 8.4% on MATH are the accuracy gains from scaling real SFT data, with no signs of saturation.The authors turn to synthetic data after exhausting the available real data.Source: §3 · p. 3
• The GPT-4 Turbo generation process creates a new question from a reference, verifies it through attempted solutions, and generates a chain-of-thought answer.Combining question solving and verification in a prompt consistently improves question validity across benchmarks.Source: §3 · pp. 3–4
• Synthetic math questions are nearly as effective as real training questions, supporting their use for further SFT-data scaling.Source: §3 · pp. 3–4
• Other explored synthetic methods are effective but marginally less so than the authors’ approach.The cited alternatives are methods proposed by Xu et al. (2023), Yu et al. (2023), and An et al. (2023).Source: §3 · p. 4

Scaling to about a Million SFT Math Data

• 960K GSM8K and 480K MATH synthetic SFT examples yield state-of-the-art accuracy with the straightforward scaling strategy.The authors describe the scaling behavior as nearly perfect.Source: §3 · p. 3, Scaling to about a Million SFT Math Data · p. 4
• The authors report that accuracy has not yet reached its peak, leaving the effects of additional scaling for future research.Source: Scaling to about a Million SFT Math Data · p. 4

4.1 Datasets and Evaluations

• Evaluation covers GSM8K and MATH in-domain, plus SVAMP, ASDiv, and the Hungarian National High School Exam for cross-dataset or out-of-domain assessment.The Hungarian exam has 33 challenging problems with human-annotated answers; the other benchmarks use automatic scripts.Source: §4.1 · p. 4

4.2 Implementation Details

• Synthetic questions are generated with GPT-4 Turbo, and SFT uses Adam for three epochs with a cosine schedule and a maximum sequence length of 2048.The experiments use eight Nvidia H100 GPUs; the most resource-intensive 70B run with 960K data points takes 1900 H100 GPU hours.Source: §4.2 · p. 4

4.3 Main Results and Comparison with State-of-the-art Models

• On LLaMA-2-7B, the approach beats prior same-base-model methods by +14.2 on GSM8K and +20.8 on MATH.It also exceeds WizardMath-70B on GSM8K, 82.6 versus 81.6.Source: Table 2 · p. 5, §4.3 · p. 5
• The mixed GSM8K–MATH model ranks second on the Hungarian exam, behind GPT-4, and the authors report no significant benchmark overfitting.The mix contains 240K synthetic MATH samples and 240K synthetic GSM8K samples.Source: §4.3 · p. 5

4.4 What Happens behind Performance Improvements?

• With more synthetic data, PassRatio@256 rises significantly while Pass@256 changes only slightly, pointing to greater stability rather than stronger answer capability.PassRatio@256 reflects stability in generating a correct answer, and its growth resembles Pass@1.Source: §4.4 · p. 6
• Increasing the share of complex training problems leaves simpler-problem accuracy virtually unchanged while improving complex-problem accuracy; PassRatio@256 rises from 71.1 to 72.8.The latter three accuracy points fall below a curve fitted to the first four, which the authors suggest may relate to fewer complex training problems.Source: §4.4 · p. 7
• Calculation errors decrease faster than reasoning errors as synthetic data grows on GSM8K.Reasoning errors include lost conditions and concept confusion; calculation errors include quantitative-relation analysis and numerical computation mistakes.Source: §4.4 · p. 7

4.5 Ablations on the Data Synthetic Schema

• As synthetic-data scale grows, the choice of generation strategy matters more: the proposed method and adding constraints perform more strongly than alternatives.At 7.5K and 30K samples, the methods have negligible performance gaps; question verification improves MATH but has no significant impact on GSM8K.Source: §4.5 · pp. 7–8

Large Language Models

• Open-source models such as LLaMA, LLaMA-2, and Mixtral have become competitive with closed-source models on some tasks, and this work fine-tunes open models on synthetic math data.Source: §5 · p. 8

Reasoning Framework for Improving Mathematical Capability

• Prompting and inference-strategy work improves reasoning without fine-tuning, whereas this paper fine-tunes the model itself; the approaches are described as complementary.Source: §5 · p. 8

Fine-tuned LLM for Math

• Synthetic-data methods include MetaMath’s question bootstrapping, LeMA’s GPT-4 mistake correction, and MuggleMath’s GSM8K augmentation with predefined operations.The paper characterizes its own synthetic-data method as simpler and more scalable because it introduces fewer prior constraints.Source: §5 · p. 8

SFT Data Scaling

• Prior instruction-tuning work reports that 1,000 high-quality examples can help general tasks, while mathematical and coding performance has also been reported to scale with data size.A later study used 1.5 million instruction-tuning examples for top performance; the intrinsic reasons behind scaling had not been thoroughly investigated.Source: §5 · pp. 8–9

6 Conclusion

• The study concludes that common 7B models already possess strong mathematical capabilities, while scaled SFT data improves the stability of their problem solving.The analysis attributes improvements primarily to single-step reasoning accuracy, finds larger reductions in calculation than logical-reasoning errors, and reports that resampling can improve harder questions.Source: §6 · p. 9
• The conclusion reports that Xwin-Math reaches performance comparable to or above larger counterparts, with extra resampling of training data improving accuracy on harder questions.Source: §6 · p. 9

A Synthetic Prompt on GSM8K

• The verified generation prompt adds step-by-step solving and revision against quality principles, retaining an already compliant question without modification.Source: A Synthetic Prompt on GSM8K · p. 12
• The answer-generation prompt requests a detailed step-by-step solution and a final answer expressed only as an integer or decimal number.Source: A Synthetic Prompt on GSM8K · p. 12
• The unverified question-generation prompt asks for a similar new problem, with one answerable numerical target and enough restated information to solve it independently.Source: A Synthetic Prompt on GSM8K · p. 12

B Additional Results

• Xwin-Math’s aggregate performance on GSM8K and MATH is second only to GPT-4, according to Figure 7.Source: Figure 7 · p. 13
• The leakage check compares losses on training and regenerated or reference test sets; the authors report no synthesis leakage because Δ1 is close to zero and Δ2 is significantly greater than zero on both benchmarks.The comparison includes a 256-sample training subset, regenerated answers for training and test questions, and a reference test set generated from test questions.Source: Table 6 · p. 13

C.1 GSM8K

• The original seed asks for the total trees from 60 mango trees and five fewer than half as many coconut trees.Source: §C.1 · p. 14
• The unverified example changes the seed to 80 apple trees, making the stated one-third-minus-four cherry-tree count fractional.The accompanying verification notes that 80 is not divisible by 3 and changes it to 81 to obtain a whole-number result.Source: §C.1 · p. 14
• The verified example changes the apple-tree count to 81, yielding 23 cherry trees and 104 trees altogether.Source: §C.1 · p. 14

C.2 MATH

• Another displayed question asks for the sum of the consecutive integers surrounding log10 28471, but its solution is not shown.Source: §C.2 · p. 15
• The verification step finds that log2 16384 is an integer, so the question is changed to use 10000, which is not a power of 2.The revised logarithm lies between 13 and 14.Source: §C.2 · p. 15
• The revised question’s answer is 27, since 2^13 < 10000 < 2^14 and the two consecutive integers sum to 13 + 14.The solution identifies the bounds before adding them.Source: §C.2 · p. 15

D.1 GSM8K

• The 7.5k greedy response reaches the correct counts for rubies and emeralds but makes a wrong final addition.It adds 175 + 140 + 280 as 605 rather than the ground-truth total of 595.Source: §D.1 · p. 16
• For the treasure chest with 175 diamonds, 35 fewer rubies, and twice as many emeralds as rubies, the correct total is 595 gems; 605 is marked incorrect.Source: §D.1 · p. 16
• With 960k data, Xwin-Math-7B greedy decode gives the correct total of 595 gems.Its response calculates 140 rubies and 280 emeralds before summing.Source: §D.1 · p. 16

D.2 MATH

• A 7.5k-data greedy response gives 19/3 and is marked incorrect for the function-value sum.The passage labels the result a wrong calculation.Source: §D.2 · p. 17
• The temperature=0.7 sample begins evaluating f(-2) and sets up the requested sum from the function values.Source: §D.2 · p. 17
• With 480k data, Xwin-Math-7B greedy decode gives the correct answer, 14/3, for the function-value sum.The response evaluates the inputs and combines the fractions.Source: §D.2 · p. 17