PaperlayerView exact revision on arXiv

Source-linked AI summary

WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct

Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, Dongmei Zhang

arXiv:2308.09583v3cs.CLcs.AIcs.LG

TL;DR

Open-source LLMs lack math-specific optimization despite strong general capabilities, motivating WizardMath. The paper applies RLEIF, combining evolved math instructions with process supervision and reinforcement learning, and reports state-of-the-art results on GSM8k and MATH, including strong comparisons with proprietary models.

  • Problem

    Most existing open-source LLMs are pretrained on internet data without math-related optimization, limiting their specialized mathematical reasoning capabilities.

  • Method

    WizardMath applies Reinforcement Learning from Evol-Instruct Feedback (RLEIF), combining Math Evol-Instruct, Process Supervision, and reinforcement learning for mathematical reasoning.

  • Results

    WizardMath achieves state-of-the-art performance over open-source LLMs on GSM8k and MATH, while WizardMath 70B surpasses several proprietary LLMs.

  • Takeaways & Limitations

    Instruction evolution and process supervision are highlighted as pivotal to WizardMath’s exceptional mathematical performance.

Abstract

from arXiv · show

Large language models (LLMs), such as GPT-4, have shown remarkable performance in natural language processing (NLP) tasks, including challenging mathematical reasoning. However, most existing open-source models are only pre-trained on large-scale internet data and without math-related optimization. In this paper, we present WizardMath, which enhances the mathematical CoT reasoning abilities of LLMs without using external python tools, by applying our proposed Reinforcement Learning from Evol-Instruct Feedback (RLEIF) method to the domain of math. Through extensive experiments on two mathematical reasoning benchmarks, namely GSM8k and MATH, we reveal the extraordinary capabilities of our model. Remarkably, WizardMath-Mistral 7B surpasses top-tier open-source LLMs by a substantial margin with higher data efficiency. Furthermore, WizardMath 70B even outperforms GPT-3.5-Turbo, Claude 2, Gemini Pro and GPT-4-early-version. Additionally, our preliminary exploration highlights the pivotal role of instruction evolution and process supervision in achieving exceptional math performance. For more details refer to https://github.com/nlpxucan/WizardLM

1 INTRODUCTION

WizardMath targets mathematical reasoning in open-source LLMs through RLEIF, combining evolved math instructions with process supervision and reinforcement learning. Experiments on GSM8k and MATH report strong performance across model sizes and against open-source and proprietary systems.

  • WizardMath outperforms other open-source LLMs at the same model size on GSM8k and MATH, achieving state-of-the-art performance.
  • WizardMath-Mistral 7B improves pass@1 over MetaMath-Mistral 7B by 12.8 points on GSM8k and 26.8 points on MATH.The reported scores are 90.7 vs. 77.9 on GSM8k and 55.4 vs. 28.6 on MATH.
  • WizardMath 70B significantly surpasses several proprietary LLMs, including GPT-3.5-Turbo, Claude 2, Gemini Pro, and an early GPT-4 version.
  • WizardMath enhances mathematical reasoning across problems ranging from grade-school to high-school levels.
  • WizardMath applies Reinforcement Learning from Evol-Instruct Feedback (RLEIF) to improve LLM mathematical reasoning.The method combines Math Evol-Instruct with Process Supervision and reinforcement learning.
  • RLEIF is presented as a fully AI-powered automatic reinforcement learning method alongside Math Evol-Instruct and Process Supervision.

2 RELATED WORK

Related work improves mathematical reasoning through prompting, math-focused training, data synthesis, hybrid reasoning, and reinforcement-learning supervision. Process-supervised reward models are contrasted with outcome-supervised approaches for handling incorrect reasoning paths.

  • Process-supervised reward models address discrepancies from incorrect reasoning paths that nevertheless lead to correct outcomes, contrasting with outcome-based supervision.

3 METHOD

WizardMath’s RLEIF method combines math instruction evolution with instruction- and process-level reward models, then uses PPO to optimize mathematical reasoning. It also aggregates multiple reasoning paths using majority voting and PRM verification.

  • Math Evol-Instruct: RLEIF creates math instructions with varied complexity through downward and upward evolution.Downward evolution makes questions easier, while upward evolution adds constraints, concretizes questions, or increases reasoning demands.
  • Math Evol-Instruct: GPT-4 evolves each instruction for 5 rounds, comprising 2 downward and 3 upward rounds, with each round building on the previous one.
  • Reward models: The Instruction Reward Model ranks evolved instructions by difficulty and definition quality.GPT-4 supplies pairwise rankings, favoring instructions with higher difficulty and clearer definitions.
  • Reward models: The Process-supervised Reward Model scores the correctness of each reasoning step in a generated answer.GPT-4 provides step-level correctness labels, with yi = 1 for a correct step and yi = 0 otherwise.
  • Reinforcement learning: PPO rewards both the instruction and reasoning steps, using the instruction reward and the minimum step reward as the answer reward.The final reward for an instruction-answer pair is formed by multiplying these two reward components.
  • Verification: Majority voting and a PRM verifier aggregate predictions from different reasoning paths.The PRM assigns scores to individual reasoning paths before aggregation.

4 EXPERIMENT

WizardMath is evaluated on GSM8k and MATH across model sizes, backbones, reward-model variants, verification strategies, and out-of-domain tasks. Results show strong benchmark performance, while ablations support contributions from Evol-Instruct, process supervision, and instruction-quality scoring.

  • Main Results: WizardMath-Llama 70B surpasses earlier GPT-4 versions, Claude-2, and Gemini Pro, while performing on par with GPT-4-0314.It exceeds GPT-3.5-Turbo by 11.2% on GSM8k and 15.5% on MATH.
  • Main Results: WizardMath outperforms prior open-source models at matched parameter sizes on GSM8k and MATH, with higher data efficiency across 0.1B–70B parameters.WizardMath-Llama 70B improves over MetaMath-Llama 70B by 10.5% on GSM8K and 32.0% on MATH.
  • Main Results: WizardMath-Mistral 7B reaches 90.7 on GSM8k and 55.4 on MATH, exceeding MetaMath-Mistral 7B at 77.9 and 28.6 respectively.It also outperforms Baichuan 3 on GSM8k and GPT-4-0314 on MATH.
  • Reward Models and Verification: Combining PRM with IRM yields a 6%–8% overall improvement over SFT, while adding IRM to PRM contributes an additional 2.5%–4% gain.The results support using both process supervision and instruction-quality scoring during PPO training.
  • Reward Models and Verification: PRM improves reasoning performance over ORM, and PRM-based generators and verifiers outperform SFT, ORM, and majority-voting alternatives.PRM superiority becomes more evident as the number of candidate solutions increases, and PRM performs strongly on the more challenging MATH benchmark.
  • Ablation and Generalization: Math Evol-Instruct improves performance across benchmark and out-of-domain evaluations, with WizardMath achieving 5%–10% average gains across seven out-of-domain tasks.Two downward evolution rounds improve GSM8k by 14.8% and MATH by 19.6% over the original data; upward and combined evolution also improve results.

5 CONCLUSION

WizardMath fine-tunes mathematical reasoning with RLEIF and achieves state-of-the-art performance on GSM8k and MATH across grade- to high-school problems. The results also indicate that instruction evolution and process supervision are pivotal to its performance.

  • WizardMath is a mathematics model fine-tuned with Reinforcement Learning from Evol-Instruct Feedback.
  • WizardMath achieves state-of-the-art performance over existing open-source LLMs on GSM8k and MATH across grade- to high-school problems.
  • WizardMath 70B outperforms several proprietary LLMs, including ChatGPT-3.5, Claude Instant, PaLM-2, and Gemini Pro.
  • Instruction evolution and process supervision are identified as pivotal factors in achieving exceptional performance.

A.1 MATH EVOLUTION PROMPTS

The Math Evol-Instruct prompts systematically modify mathematical instructions to increase or decrease difficulty while preserving coherence and controlling ambiguity. The appendix illustrates these procedures through structured prompts and GSM8k and MATH examples.

  • Upward Evolution: Upward evolution increases instruction complexity by identifying modifiable components, planning coherent changes, rewriting the instruction, and reviewing the result.The process requires modifying or expanding at least three components and avoiding unjustified variables or conditions.
  • Prompt Structure: The prompts require strict four-step output formatting: identify elements, formulate a plan, rewrite the instruction, and provide the final rewritten instruction.
  • Downward Evolution: Downward evolution decreases difficulty while preserving the core scenario and avoiding ambiguity or confusion.The prompt similarly requires planning changes across multiple interconnected components.
  • GSM8k Examples: GSM8k evolution examples transform routine word problems by changing participants, conditions, quantities, or settings while retaining mathematical structure.
  • MATH Examples: MATH evolution examples add conditions, divisibility requirements, variables, or requested verification steps to increase problem complexity.

A.2 IRM PROMPT

The IRM prompt ranks mathematical instructions by completeness and difficulty, using criteria spanning statement quality, conceptual difficulty, computational complexity, and contextual relevance. GPT-4-generated rankings provide the training case shown in Table 10.

  • Evaluation Criteria: The IRM evaluates problem-statement completeness, conceptual difficulty, computational complexity, and contextualisation.
  • Evaluation Criteria: Problem-statement evaluation checks clarity, accuracy, sufficient conditions, interpretive ambiguity, and mathematical notation.
  • Evaluation Criteria: Conceptual and computational evaluation considers underlying principles, required mathematical knowledge, critical thinking, calculations, and algorithmic sophistication.
  • Evaluation Criteria: The ranking prompt instructs evaluators to avoid position, length, and theory biases and to remain objective.
  • Ranking Objective: The instruction-ranking task orders problems from most to least complete and clear, while also ranking them by mathematical difficulty.
  • GPT-4 Scoring: Table 10 presents a GPT-4 case study scoring evolved instructions on difficulty and definition.

A.3 PRM PROMPT

The PRM prompt judges whether a solution is correct and scores each reasoning step using explicit correctness criteria. Its structured output supports process-level supervision of mathematical solutions.

  • Task and Output: The PRM task first produces a True-or-False judgment of the complete solution, then assigns a score to every solving step.
  • Scoring: Step scores are 1 for correct, 0 for ambiguous, meaningless, subtly misleading, or unhelpful, and -1 for incorrect.
  • Evaluation Criteria: The prompt evaluates problem-solving thoughts, calculation accuracy, logical coherence, and use of basic mathematical theories and principles.
  • Task and Output: The required JSON output records each step index, its score, and a reason explaining the rating.
  • Case Study: A GSM8k case study compares PRM and ORM scoring, with PRM detecting wrong reasoning steps that ORM failed to detect.

A.4 DATA CONTAMINATION CHECK

The study checks whether instruction evolution causes training–test contamination and finds no evidence of increased similarity across evolution rounds.

  • Similarity scores remained relatively low across all evolution rounds.The authors used embedding retrieval and GPT-4 similarity judgments after removing the two most similar retrieved instructions.
  • The evolution process did not yield higher similarity scores between evolved training data and GSM8k or MATH test instructions.The authors attribute performance gains primarily to more complex and comprehensive data from downward and upward instruction evolution.

A.5 SIMILARITY CHECKING AND DATA FILTERING

The contamination check retrieves potentially similar training samples for each test problem, scores their similarity with GPT-4, and filters the closest matches.

  • Similarity scores range from 1 for completely different problems to 10 for identical problems.The scoring prompt evaluates overlap in variables, conditions, participants, actions, topics, content, concepts, and structure.
  • gte-large retrieves the top five training samples for each GSM8k and MATH test sample used as a query.All test samples serve as queries against the evolved training data.
  • The two most similar instructions are removed as an additional filtering step.This procedure is intended to prevent leakage from the test datasets into training data.
  • GPT-4 evaluates similarity between each test problem and its retrieved training samples.The retrieved samples are then assessed for possible overlap with the test set.

A.6 DETAILED EXPLANATION OF OUR METHOD FLOW.

RLEIF combines Math Evol-Instruct, instruction and process reward models, and PPO reinforcement learning to improve mathematical reasoning without external Python tools.

  • The method integrates instruction evolution with reinforcement learning to enhance mathematical reasoning capabilities.The pipeline includes supervised fine-tuning, reward modeling, and reinforcement learning operations.
  • Instruction Evolution and SFT: RLEIF evolves original math instructions upward and downward before applying them to supervised fine-tuning.The pipeline generates both grade-school and challenging high-school instruction data.
  • Reward Model Training: The method trains an Instruction Quality Scoring Reward Model and a Process-Supervised Reward Model.IRM assesses instruction quality from GPT-4 rankings, while PRM learns from GPT-4-labeled step-by-step solutions.
  • Reinforcement Learning with PPO: PPO combines IRM instruction scores and PRM step scores into a final reward for reinforcement learning.The policy model generates responses, which are scored and used for PPO training.
  • Evaluation: WizardMath-SFT improves over DART-Math by 2%–7% on GSM8k and approximately 5%–10% on MATH for several listed base models.The comparisons cover Llama3.2 1B, Llama3.2 3B, Llama3-8B, Llama3.1-8B, and Llama2-7B.
  • Evaluation: WizardMath-SFT is evaluated against state-of-the-art models on GSM8k and MATH using CoT pass@1 without external Python tools.Tables 13–15 compare models across base-model scales and data-synthesis methods.

A.8 THE PERFORMANCE OF WIZARDMATH ON THE OTHER DIFFERENT BASE MODELS

Across diverse base models and SFT backbones, WizardMath combines effective Math Evol-Instruct data synthesis with IRM- and PRM-based PPO improvements on GSM8k and MATH.

  • Base-model results: 16.7% on GSM8k and 14.5% on MATH: WizardMath-Mathstral improves over Mathstral-7B-v0.1-Base.The reported scores are 93.8 vs. 77.1 on GSM8k and 70.9 vs. 56.6 on MATH.
  • Base-model results: RLEIF improves math-optimized base models by 15%–25% on MATH and 8%–16% on GSM8k, with an exception for Qwen2.5-Base.The comparison includes Mathstral-7B-v0.1-Base and Qwen2.5-Math-Base models.
  • Reward-model impact: 6.4%–6.9% on GSM8k and 5.2%–6.4% on MATH: IRM and PRM improve DART-Math models based on Llama2-7B and Mistral-7B-v0.1.The gains are reported after PPO reinforcement learning.
  • Reward-model impact: PPO with IRM and PRM improves MetaMath models by 8%–9% on GSM8k and 5%–8% on MATH.Xwin-Math-Llama2-7B also improves by 6%–8% on both benchmarks.
  • Limitations: High GPT-4 distillation cost limits large-scale data generation, while evolved instructions lack correct answers needed by Math-Shepherd.Manual annotation is also described as expertise-intensive, time-consuming, and costly.
  • Instruction evolution: Two rounds of downward evolution improve GSM8k by 14.8% and MATH by 19.6% over the original instructions.The reported scores are 74.5 vs. 59.7 on GSM8k and 34.7 vs. 15.1 on MATH.
  • Method contributions: A fully AI-automated pipeline synthesizes SFT, PRM, and IRM datasets and uses AI to annotate step-level PRM data.The authors state that this design ensures scalability.
Loading 2308.09583v3…