Source-linked AI summary
Orca-Math: Unlocking the potential of SLMs in Grade School Math
Arindam Mitra, Hamed Khanpour, Corby Rosset, Ahmed Awadallah
TL;DR
Small language models’ reasoning capabilities remain an open scale question, and ensemble-based routes can require many model calls. Orca-Math combines synthetic math data with iterative learning from teacher feedback; its 7B model trained on 200K data reports 86.81% accuracy on GSM8k.
Problem
How much scale is needed for SLM reasoning remains an open question, while ensemble-based approaches can use many model calls and substantial data.
Source: §1 · p. 2Method
The paper pairs a 200K synthetic math-problem dataset with iterative learning in which an SLM practices and receives teacher feedback on its solutions.
Source: §1 · p. 2Results
86.81% accuracy on GSM8k was achieved by the 7B model trained with 200K data, surpassing the previously perceived 80% barrier.
Source: §7 · p. 11Takeaways & Limitations
Agent-based dataset generation is presented as valuable for creating more diverse grade-school math problems, alongside innovative learning strategies for SLMs.
Source: §7 · p. 11The GSM8K test split is excluded from training.
Source: §5.3 · p. 10
Abstract
from arXiv · showhide
Mathematical word problem-solving has long been recognized as a complex task for small language models (SLMs). A recent study hypothesized that the smallest model size, needed to achieve over 80% accuracy on the GSM8K benchmark, is 34 billion parameters. To reach this level of performance with smaller models, researcher often train SLMs to generate Python code or use tools to help avoid calculation errors. Additionally, they employ ensembling, where outputs of up to 100 model runs are combined to arrive at a more accurate result. Result selection is done using consensus, majority vote or a separate a verifier model used in conjunction with the SLM. Ensembling provides a substantial boost in accuracy but at a significant cost increase with multiple calls to the model (e.g., Phi-GSM uses top-48 to boost the performance from 68.2 to 81.5). In this work, we present Orca-Math, a 7-billion-parameter SLM based on the Mistral-7B, which achieves 86.81% on GSM8k without the need for multiple model calls or the use of verifiers, code execution or any other external tools. Our approach has the following key elements: (1) A high quality synthetic dataset of 200K math problems created using a multi-agent setup where agents collaborate to create the data, (2) An iterative learning techniques that enables the SLM to practice solving problems, receive feedback on its solutions and learn from preference pairs incorporating the SLM solutions and the feedback. When trained with Supervised Fine-Tuning alone, Orca-Math achieves 81.50% on GSM8k pass@1 metric. With iterative preference learning, Orca-Math achieves 86.81% pass@1. Orca-Math surpasses the performance of significantly larger models such as LLAMA-2-70B, WizardMath-70B, Gemini-Pro, ChatGPT-3.5. It also significantly outperforms other smaller models while using much smaller data (hundreds of thousands vs. millions of problems).
1 Problem Setup
The paper asks how much scale is needed for SLM reasoning and explores grade-school math through a teacher-student approach. It introduces synthetic data and iterative learning, reporting 86.81% pass@1 without external tools.
1 Problem Setup
- Whether scale is needed to achieve reasoning capabilities in SLMs remains an open research question, while ensembling raises model-call costs.Ensembling may combine up to 100 runs using consensus, majority vote, or a verifier; generating and verifying 100 solutions can require 200 model calls.Source: §1 · p. 2
- The approach extends teacher-student training with a synthetic dataset of 200K math problems and an iterative procedure that lets the SLM practice and receive teacher feedback.The feedback evaluates student solutions or provides a teacher solution.Source: §1 · p. 2
- 81.50% on GSM8k pass@1 is achieved with supervised fine-tuning alone.Source: §1 · p. 2
- 86.81% pass@1 exceeds LLAMA-2-70B (56.8%), WizardMath-70B (81.6%), Gemini Pro (86.5% with 32 trials), and GPT-3.5 (77.4%).The paper reports this result without multiple model calls, verifiers, code execution, or other external tools.Source: §1 · p. 2
2 Dataset Construction: Agent-Instruct
Agent-Instruct builds a grade-school math dataset by expanding seed problems into varied and more challenging examples, with generated solutions and additional filtered data sources.
Dataset Construction
- 36,217 problems from open-source datasets form the seed set, supplemented with problems from the train and validation splits of Lila.Source: §2 · p. 4
- The Ask Me Anything agent creates multiple word problems from each seed; the process yields 120,445 new problems with similar narratives and GPT4-Turbo solutions.The agent first converts a seed question into a statement, then creates new problems from numbers in that statement.Source: §2 · pp. 4–5
- Suggester and Editor agents collaboratively increase problem difficulty, with the Suggester proposing changes and the Editor producing revised problems over multiple rounds.The multi-agent workflow uses AutoGen [37].Source: §2 · p. 6
- After two iterations, problems whose GPT4-Turbo answers exceed 1,800 characters are filtered, leaving 37,157 problems from this process.Source: §2 · p. 7
- The dataset also includes 6,216 DMath problems for which GPT4-Turbo solutions align with the precise gold-standard answers.Source: §2 · p. 7
3 Training
Training begins with supervised fine-tuning of Mistral-7B, then builds preference data from student responses and teacher judgments for iterative preference learning.
3.1 Supervised Fine-Tuning Experiment (Iteration #1)
- Mistral-7B is fine-tuned on Orca-Math-200K for one epoch, with loss computed only on answer tokens.The instruction format pairs each question with its answer.Source: §3.1 · p. 7
3.2 Iterative Learning from both Positive and Negative Signals
- For each problem, four SFT-model responses and a GPT4-Turbo solution are assessed against the teacher answer to build positive and negative preference sets.The teacher solution is treated as positive; the negative set may be empty when all four student responses align with the teacher.Source: §3.2 · p. 7
- The paper evaluates DPO and KTO for learning from positive and negative feedback, then uses the KTO-trained M2 model to generate responses for iteration #3.KTO distinguishes itself by requiring a binary yes-or-no quality judgment.Source: §3.2 · pp. 7–8
4 Evaluation
Evaluation uses GPT-4 to extract a model answer and compare it with the gold answer. The accompanying example illustrates the teacher-solution check and its final correctness verdict.
4 Evaluation
- GPT4-based-Exact-Match prompts GPT-4 to extract the generated answer’s final short answer and match it against the gold short answer.Source: §4 · p. 8
SYSTEM
- The evaluation prompt asks GPT-4 to compare a student’s answer with the problem setter’s solution, allowing different solution steps when the final answer matches.Source: SYSTEM · p. 8
ASSISTANT
- In the example, the date interval is counted as 50 days, 20% off leaves 40 working days, and the student’s 240-person answer is judged correct.Source: SYSTEM · p. 8, ASSISTANT · p. 9
5 Results
The results compare iterative training procedures, ablations, performance on other word-problem benchmarks, and a text-contamination check. Iteration-2 KTO performs best in its training group, while the reported ablations favor student-generated positives and retaining all-positive-response questions.
5 Results
- Iteration-2 KTO performs better than the other procedures in its group on the 1,319-question GSM8K test set.Source: §5 · p. 9, Table 2 · p. 9
5.1 Ablation Studies
- Removing student-generated positives causes a significant performance drop with either DPO or KTO.Source: §5.1 · p. 9, Table 3 · p. 10
- Including questions whose sampled responses are all positive is beneficial; omitting them removes about 80,000 iteration-2 questions and 104,000 iteration-3 questions.Source: §5.1 · pp. 9–10, Table 4 · p. 10
5.2 Math Benchmarks beyond GSM8k
- For six other structured word-problem benchmarks, evaluation uses single-number answers, GPT-4-based exact match, and greedy decoding.Source: §5.2 · p. 10, Table 5 · p. 10
5.3 Contamination Check
- The contamination check found eight test questions above the 0.5 unigram-overlap threshold; with only seed problems in training, it found seven, and for n ≥2 it found none.Source: §5.3 · pp. 10–11
- The check compares test questions with top-k analogous training questions selected by TF-IDF cosine similarity, then measures n-gram overlap using Jaccard similarity.Source: §5.3 · p. 10
6 Related Works
Related work spans synthetic datasets across general and specialized domains, alongside research on learning from preference signals. This work analyzes agent-based data generation and iterative preference learning for grade-school math.
6 Related Works
- Math-related datasets [40] [43] are among the synthetic-data work closely related to this study.The broader dataset literature also covers specialized and generic domains [28] [23] [9] [45] [6] [36].Source: §6 · p. 11
- Prior studies [31] [10] [22] [3] [5] [41] have demonstrated the usefulness of preference learning.This study examines iterative preference learning in grade-school math and reports KTO robustness over DPO and benefits from model-generated positives.Source: §6 · p. 11
- The authors frame their analysis as a preliminary step toward iterative learning and self-improvement of small language models in challenging domains.Their analysis focuses on agent-based synthetic data generation and iterative preference learning in grade-school math.Source: §6 · p. 11
7 Conclusions
The study reports that a 7B small language model trained with 200K data achieved 86.81% accuracy, surpassing the previously perceived 80% barrier on GSM8k. It also identifies agent-based dataset generation as valuable for producing more diverse and interesting datasets.
7 Conclusions
- 86.81% accuracy was achieved by the 7B model trained with 200K data, surpassing the previously perceived 80% barrier on GSM8k.The study attributes substantial enhancement of SLM mathematical reasoning capabilities to iterative learning with positive and negative signals.Source: §7 · p. 11
- Agent-based dataset generation proved valuable for creating more diverse and interesting datasets.The study highlights this alongside innovative learning strategies as relevant to advancing SLM performance.Source: §7 · p. 11
- The findings highlight the potential for substantial improvements in SLM mathematical reasoning capabilities.The study reports achieving this through iterative learning techniques that leverage positive and negative signals.Source: §7 · p. 11