TinyGSM: achieving >80% on GSM8k with small language models

1 Introduction

• The motivating gap is that small-model mathematical reasoning remains underexplored, while models exceeding 80% on GSM8K have commonly been at least 34B parameters.The authors aim to cross the 80% threshold while keeping model size friendly.Source: §1 · p. 1
• The verifier scores multiple candidate answers from the student model and selects the highest-scoring one, using labeled GSM8K training data.The introduction presents verifier selection as bridging the student–teacher performance gap.Source: §1 · pp. 1–2
• 81.5% GSM8K accuracy is reported for the 1.3B model with a verifier, rivaling GPT-3.5’s 77.4% and surpassing much larger open-source models.The result is shown in Figure 1; the introduction describes it as a new state of the art for billion-parameter-scale models.Source: Figure 1 · p. 2, §1 · p. 2
• TinyGSM contains 12.3M GPT-3.5-generated GSM8K-style problems paired with Python solutions.Finetuning Phi-1.5 1.3B raises its GSM8K test accuracy from 44.6% to 68.2%; the 125M model reaches 63.1%.Source: §1 · p. 2
• Scaling the verifier from 125M to 1.3B yields a 7.2% performance boost, compared with 5.1% from scaling the generator.The introduction identifies data diversity as important for verifier success.Source: §1 · p. 2

2 Related works

• Synthetic-data work addresses limited training data, while TinyGSM is described as much larger than MetaMath’s 40,000 questions.The discussion also notes concerns about diversity and distribution drift when training on synthetic data.Source: §2 · pp. 2, 4
• Verifier selection builds on prior work selecting among 100 candidates, which reported a 20% accuracy boost; this paper studies verifier design choices more thoroughly.Self-consistency selects the most frequent candidate, but the authors say it was not particularly helpful in their setup.Source: §2 · p. 4
• Prior work finds process supervision helpful for complex math problems, while this paper evaluates final accuracy but trains on full programs.The cited discussion notes a cost-efficacy tradeoff for process supervision on general problems.Source: §2 · p. 4
• Small-model self-improvement methods are described as undiscovered, motivating a separate verifier that is initialized from the generator and fully finetuned.The passage contrasts this with self-refine and self-verify approaches that ask a model to critique or verify its own outputs.Source: §2 · p. 4
• Prompt-based methods such as in-context learning and Chain-of-Thought are characterized as targeting large models rather than small language models.The cited discussion describes Chain-of-Thought as generating intermediate steps.Source: §2 · p. 4

3 The TinyGSM dataset

• GSM8K’s 7,473 training problems are considered too few to train a reasonably sized language model, motivating synthetic augmentation.The authors focus on grade-school problems that combine elementary math with natural-language comprehension.Source: §3 · p. 4
• GPT-3.5-turbo generates question variants from sampled GSM8K training questions, with each variant paired with a Python solution.In the examples, the question is a function docstring and the solution is the function body; executable code lets the authors use a Python interpreter for calculations.Source: §3 · pp. 4–5, Figure 2 · p. 5
• About one-third of TinyGSM consists of synthetic problems containing irrelevant information, generated to enhance robustness.These examples augment GSM-IC, which introduces irrelevant context into GSM8K questions.Source: §3 · p. 5
• TinyGSM contains 12.3M question-solution pairs generated from GSM8K training questions and their irrelevant-context variants.Temperature sampling and prompts encouraging grammatical diversity and multiple reasoning steps are used to encourage variety.Source: §3 · p. 5
• Filtering removes problems that are too short or lack numbers and code solutions that are not executable, but does not check question or solution correctness.The authors avoid majority-vote consistency filtering because GPT-3.5-turbo is consistent mainly on easy problems, which could remove challenging examples.Source: §3 · p. 5

4 Solving grade school math with small language models

• The 1.3B phi-GSM model achieves 81.5% accuracy on GSM8K.The reported result uses synthetic data and a verifier.Source: §4 · p. 5

4.1 Learning from synthetic data

• After TinyGSM finetuning, the 1.3B model reaches 68.2% accuracy on GSM8K.The training setup uses Adam, FP16, and up to 20k finetuning steps.Source: §4.1 · p. 6

4.2 Improving small models with a verifier

• The verifier learns from labeled GSM8K questions and selects among sampled candidate solutions using its score for each generation.Training labels reflect final-answer correctness, not intermediate-step correctness; at test time, the last-token score selects among 48 candidates.Source: §4.2 · p. 7, Figure 4 · p. 7
• With verifier selection, 125M generation plus a 1.3B verifier reaches 76.1%, while 1.3B generation plus a 125M verifier reaches 71.7%.The figure evaluates Pass@1 on GSM8K after selecting the highest-scoring answer from 48 candidates.Source: Figure 4 · p. 7, §4.2 · p. 8
• Training verifier data from multiple checkpoints improves performance from 78.9% to 81.5%.The data uses checkpoints at 1k, 6k, and 12k steps, with generations at temperatures 0.5 and 0.7.Source: §4.2 · p. 7

4 Solving grade school math with small language models

• Generated problems can violate common sense, and temperature-1 generations can yield accuracy more than 10% below greedy decoding.The paper notes that some generated quantities may not be integers.Source: §4 · p. 5, §4.2 · p. 7

5.1 Contamination test

• 22 of 11.0M unique synthetic questions had a nonzero 13-gram test-set match, while 38k questions had a nonzero 8-gram match.The 8-gram matches were around 0.35% of the full set; punctuation and numbers were removed before matching.Source: §5.1 · p. 8
• N-gram matching cannot guard against contamination such as paraphrasing, and checking beyond exact matches remains an open problem.The authors found embedding similarity unsuitable because CodeGen 350M clusters reflected semantic topics more than structural or functional similarity in math questions.Source: §5.1 · p. 8

5.2 Evaluation on SVAMP

• 75.6% accuracy on SVAMP was achieved by the 1.3B model without further finetuning.Figure 6 presents SVAMP test accuracies.Source: §5.2 · p. 9, Figure 6 · p. 9

6 Discussions

• A 1.3B generation model achieved 81.5% on GSM8K, setting a new state of the art for small language models.The approach combines GPT-3.5-generated TinyGSM with a verifier whose quality is boosted by diverse generations.Source: §6 · p. 9
• Code solutions can help avoid execution or calculation errors but tend to perform worse on equation-solving questions, likely because Python syntax does not naturally support equations.The paper says combining code and natural-language formats has the potential to further boost performance.Source: §6 · p. 9
• Given a model-size budget, scaling the verifier may use parameters more efficiently than scaling the generation model.The authors note that this contrasts with their intuition that verification is easier than generation.Source: §6 · p. 9

A.1 Other prompts

• Most TinyGSM data was generated by prompting GPT-3.5-turbo for both a question and its solution; the remaining data used a two-step process.In that process, the model first generated question variants and then Python solutions for those variants.Source: §A.1 · p. 14
• Figure 7 gives the question-variant prompt for GSM8K, and Figure 8 gives the corresponding prompt for GSM8K-IC.The paper identifies Figures 7–9 as the exact prompts for the two-step generation process.Source: Figure 7 · p. 14, Figure 8 · p. 14, §A.1 · p. 14

A.2 Contamination check: 13-gram collisions

• 22 questions out of 11.0M unique synthetic questions had 13-gram collisions with test-set questions.Examples are shown in Figure 10.Source: §A.2 · p. 14

• Table 2 compares performance with and without pretraining.Source: Table 2 · p. 14
• Figure 9 presents a prompt template for generating a code solution from a question.Source: Figure 9 · p. 15
• Figure 10 presents a prompt template for generating a code solution from a question.Source: Figure 10 · p. 15