PaperlayerView exact revision on arXiv

Source-linked AI summary

System-2 Mathematical Reasoning via Enriched Instruction Tuning

Huanqia Cai, Yijun Yang, Zhifeng Li

arXiv:2412.16964v2cs.AIcs.CL

TL;DR

LLMs struggle with the deliberate reasoning complex mathematical problems require, and high-quality multi-step reasoning data is difficult to obtain. EIT enriches human-annotated answers with AI-generated plans and reasoning steps, then fine-tunes open-source LLMs; it reports strong results on MATH and GSM8K, including performance competitive with tool-augmented methods.

  • Problem

    LLMs’ next-token prediction can fall short of the deliberate system-2 reasoning needed for complex mathematics.

    Source: §1 · pp. 1–2

  • Method

    EIT uses AI to add plans and omitted reasoning context to human-annotated mathematical responses, then fine-tunes open-source LLMs on the enriched data.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    EIT achieves 84.1% accuracy on GSM8K and 32.5% on MATH, with competitive performance against tool-augmented methods on the two benchmarks.

    Source: §1 · p. 3, §7 · pp. 8–9

  • Takeaways & Limitations

    The paper reports that finer-grained reasoning trajectories consistently lead to higher testing accuracy, suggesting dataset granularity and size are both important for training math LLMs.

    Source: §7 · p. 9

    EIT strictly follows the original response and does not add mathematical or symbolic calculations.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Solving complex mathematical problems via system-2 reasoning is a natural human skill, yet it remains a significant challenge for current large language models (LLMs). We identify the scarcity of deliberate multi-step reasoning data as a primary limiting factor. To this end, we introduce Enriched Instruction Tuning (EIT), a method that enriches existing human-annotated mathematical datasets by synergizing human and AI feedback to create fine-grained reasoning trajectories. These datasets are then used to fine-tune open-source LLMs, enhancing their mathematical reasoning abilities without reliance on any symbolic verification program. Concretely, EIT is composed of two critical steps: Enriching with Reasoning Plan (ERP) and Enriching with Reasoning Step (ERS). The former generates a high-level plan that breaks down complex instructions into a sequence of simpler objectives, while ERS fills in reasoning contexts often overlooked by human annotators, creating a smoother reasoning trajectory for LLM fine-tuning. Unlike existing CoT prompting methods that generate reasoning chains only depending on LLM's internal knowledge, our method leverages human-annotated initial answers as ``meta-knowledge'' to help LLMs generate more detailed and precise reasoning processes, leading to a more trustworthy LLM expert for complex mathematical problems. In experiments, EIT achieves an accuracy of 84.1% on GSM8K and 32.5% on MATH, surpassing state-of-the-art fine-tuning and prompting methods, and even matching the performance of tool-augmented methods.

1 Introduction

LLMs struggle to apply deliberate system-2 reasoning to complex mathematics, while existing human- and AI-feedback approaches have important limitations. EIT combines human-annotated answers with AI-generated plans and reasoning steps; it reports strong results on MATH and GSM8K.

Motivation

  • LLMs’ reliance on next-token prediction can fall short of the deliberate system-2 reasoning needed for complex mathematical problems.The passage describes this as a barrier to using LLMs in applications requiring mathematical reasoning.Source: §1 · pp. 1–2

Challenges

  • Human-feedback datasets are expensive and difficult to assess reliably, while AI feedback risks relying on LLM-generated reasoning that may itself be weak.Prompting methods can also suffer factual hallucinations and error propagation over long reasoning trajectories.Source: §1 · p. 2

Approach

  • EIT uses human-annotated initial answers as guidance for AI-generated plans and reasoning contexts, then fine-tunes open-source LLMs on the enriched trajectories.ERP decomposes an instruction into lower-level objectives; ERS fills in omitted reasoning contexts.Source: §1 · pp. 2–3

Results

  • EIT achieves 32.5% accuracy on MATH and 84.1% on GSM8K, improving MATH by 2.7% over MetaMath and matching tool-using methods on GSM8K.The paper also reports that EIT outperforms MathCoder on GSM8K.Source: §1 · p. 3

2 Related Work

Prior work has explored prompting, human-feedback fine-tuning, and AI-feedback data generation for mathematical reasoning. The passages describe limitations involving hallucinations, annotation costs, assessment reliability, and AI models’ reasoning ability.

Prompt-based methods

Human-feedback fine-tuning

  • Large-scale human reasoning datasets are costly and impractical to collect because professional annotators are scarce and correctness assessments can be biased.Source: §2 · p. 3

AI feedback

3 Enriched Instruction Tuning

EIT combines human-annotated mathematical answers with AI-generated reasoning enrichment, then uses the enriched dataset to fine-tune an open-source LLM. Its ERP and ERS stages add plans and omitted reasoning context while retaining the original response as guidance.

3 Enriched Instruction Tuning

  • EIT combines human-annotated responses with AI-generated reasoning trajectories and uses the resulting enriched dataset to fine-tune an open-source LLM.The paper frames this as combining strengths of human and AI feedback.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Generation of Enriched Instruction Dataset

  • Human mathematical responses can be accurate yet sparse, omitting implicit reasoning steps, and asking annotators to write complete trajectories is described as impractical.Source: §3.1 · p. 4
  • Off-the-shelf LLMs are prompted to enrich existing mathematical datasets with missing reasoning contexts.The prompts use few-shot examples.Source: §3.1 · p. 4
  • EIT’s enrichment strictly follows the original response and adds no mathematical or symbolic calculations.The passage says this preserves human annotations’ accuracy while mitigating hallucinations in long-horizon reasoning.Source: §3.1 · p. 4
  • ERP decomposes an instruction into lower-level objectives, and ERS uses that plan to add omitted context and reasoning steps for a more coherent trajectory.Source: §3.1 · p. 4
  • Table 1 is identified as a comparison of EIT’s augmentation strategies with other state-of-the-art methods.The supplied caption gives the comparison’s scope but no results.Source: Table 1 · p. 5

3.2 Finetuning LLMs on Enriched Instruction Dataset

  • The training setup uses token-level supervised fine-tuning, predicting the response for an input instruction with a next-token objective.The target-response likelihood is optimized using maximum likelihood estimation with mini-batch stochastic gradient descent.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate fine-tuned LLaMA-2 models on MATH and GSM8K using EITMath, a dataset of 15k enriched problems split evenly between the benchmarks.

4 Experiments

  • The evaluation fine-tunes open-source LLaMA-2 models at 13B and 70B scales and tests them on MATH and GSM8K.Source: §4 · p. 5, §4.2 · p. 5

4.1 Datasets

  • EITMath contains 15k problems, with 7.5k each from MATH and GSM8K, and corresponding answers enriched using ERP and ERS.Table 2 presents the dataset composition.Source: §4.1 · p. 5, Table 2 · p. 5

4.2 Experimental Setup

  • GPT-4-1106-preview generates enriched responses; training uses AdamW for 3 epochs at a 2e-5 learning rate, with 32 A100 GPUs.Batch sizes are 32 for the 70B model and 128 for the 13B model.Source: §4.2 · p. 5
  • For ablations, EIT uses 7.5k examples per benchmark, and a 70k version samples GPT-4 responses at varied temperatures, slightly improving performance.Table 3 covers ERP, ERS, and MetaMath ablations on MATH and GSM8K.Source: §4.2 · p. 5, Table 3 · p. 6

5 Ablation Study

The ablations report gains from ERP and ERS individually and the highest accuracies when combined; adding EIT to MetaMath yields further improvements.

Effects of ERP

  • ERP raises accuracy from 14.9% to 17.2% on MATH and from 67.3% to 72.0% on GSM8K.The reported improvements are 2.3% and 4.7%, respectively.Source: §5 · p. 6

Effects of ERS

  • ERS raises accuracy from 14.9% to 19.5% on MATH and from 67.3% to 76.3% on GSM8K.The reported GSM8K improvement is greater than the MATH improvement, with stronger gains on the easier benchmark.Source: §5 · p. 6

Effects of Combining ERP and ERS

  • Combining ERP and ERS yields peak accuracies of 21.3% on MATH and 78.8% on GSM8K.Source: §5 · p. 6

Effects of Combining EIT and MetaMath

  • Adding EIT to MetaMath improves accuracy by 1.3% on MATH and 2.3% on GSM8K over response enrichment alone.Compared with question augmentation alone, the gains are 3.6% on MATH and 6.9% on GSM8K.Source: §5 · pp. 6–7

6 Comparison with State of the Art

EIT performs strongly on MATH and GSM8K, and the analyses relate performance to dataset scale, reasoning-step granularity, and response perplexity. Its use of human-annotated answers as guidance also distinguishes it from CoT prompting.

6.1 Results on MATH and GSM8K

  • EIT-70B records 32.5% accuracy on MATH and 84.1% accuracy on GSM8K, outperforming same-type open-source baselines and matching tool-augmented methods on GSM8K.On MATH, EIT-70B improves 2.7% over MetaMath-70B; on GSM8K, it improves 2.5% over WizardMath and 1.8% over MetaMath with the same parameters.Source: §6.1 · p. 7

6.2 Analysis from a Scaling Law Perspective

  • Adding EITMath improved MATH performance as dataset scale increased, whereas adding RFT consistently reduced performance across the tested scales.The experiments combine different-sized MetaMathQA subsets with either EITMath or RFT for LLaMA-2-70B fine-tuning.Source: §6.2 · p. 7, Figure 3 · p. 8
  • MATH performance showed a positive correlation with the average response-token count in fine-tuning data, used as a measure of reasoning-step granularity.The datasets were derived from MATH using the same strategy as EITMath, with different average response lengths.Source: §6.2 · pp. 7–8, Figure 3 · p. 8

6.3 Analysis from a Perplexity Perspective

  • EITMath responses had lower perplexity than other mathematical datasets, while models fine-tuned on EITMath achieved the highest GSM8K accuracy.Perplexity was calculated with LLaMA-2-7B without additional fine-tuning; GSM8K accuracy was measured on fine-tuned LLaMA-2-70B models.Source: §6.3 · p. 8, Figure 3 · p. 8

6.4 Comparison between EIT and CoT

  • EIT uses human-annotated initial answers as “meta-knowledge” to guide fuller reasoning, and outperforms prompting with ERP and ERS as well as standard CoT prompting and fine-tuning.The comparison reports nearly 25.3% and 51.2% improvements over ERP-and-ERS prompting, and 22.2% and 13.1% over standard CoT prompting and fine-tuning, respectively.Source: §6.4 · p. 8

7 Conclusion

EIT combines human and AI feedback to create fine-grained reasoning trajectories for fine-tuning open-source LLMs. The paper reports competitive results on MATH and GSM8K and identifies reasoning-trajectory granularity and dataset size as potentially important training factors.

7 Conclusion

  • EIT fine-tunes open-source LLMs on human-AI-enriched instruction datasets to support system-2 mathematical reasoning without external tools.The paper reports competitive performance on MATH and GSM8K and says finer-grained reasoning trajectories consistently lead to higher test accuracy.Source: §7 · pp. 8–9

A Detailed Related Work

  • Unlike prior work focused on either human or AI feedback, EIT combines both to construct fine-grained reasoning trajectories for fine-tuning.The paper describes large-scale human-feedback collection as expensive and impractical, while AI-generated trajectories alone do not directly provide the needed system-2 mathematical reasoning capability.Source: A Detailed Related Work · p. 12

B Quality Analysis of Our Proposed EITMath Dataset

The quality analysis assesses EITMath with response perplexity and final-answer accuracy. It reports high accuracy for generated answers and lower perplexity than the original answers.

B Quality Analysis of Our Proposed EITMath Dataset

  • EITMath quality analysis uses perplexity and final-answer accuracy to assess generated responses.The paper also compares generated final answers with their original answers.Source: B Quality Analysis of Our Proposed… · p. 12
  • 99.7% accuracy is reported for EIT-generated answers, compared with 71.5% for GPT-4 final answers without EIT prompts.The comparison concerns the accuracy of final answers against the original answers.Source: B Quality Analysis of Our Proposed… · p. 12
  • EIT-generated data had significantly lower perplexity than the original answers.The paper interprets its analysis as indicating an inverse relationship between perplexity and accuracy.Source: B Quality Analysis of Our Proposed… · p. 12

C Discussion on Prompting and Fine-tuning Methods

EIT achieves higher reported accuracy than the compared prompting and standard CoT methods. The discussion contrasts prompting demands with fine-tuning results and attributes fine-tuning’s advantage to avoiding instability from complex prompts.

C Discussion on Prompting and Fine-tuning Methods

  • ERP-and-ERS few-shot prompting trails standard CoT prompting, while standard CoT fine-tuning beats CoT prompting by 11.5% on GSM8K.The paper attributes the fine-tuning improvement to stimulating CoT capabilities while avoiding instability from complex prompts.Source: C Discussion on Prompting and… · p. 12
  • EIT reports 32.5% accuracy on MATH and 84.1% on GSM8K, improving over ERP-and-ERS prompting and standard CoT fine-tuning.The reported improvements are nearly 25.3% and 51.2% over prompting with ERP and ERS, and over 15% over standard CoT fine-tuning.Source: C Discussion on Prompting and… · p. 12
  • Human-annotated initial answers serve as “meta-knowledge” for generating smoother reasoning trajectories and mitigating hallucination accumulation as inference length increases.The discussion says this helps ensure the accuracy of the constructed data.Source: C Discussion on Prompting and… · p. 12

D More Results on Self-Consistency

With self-consistency decoding, EIT reaches higher GSM8K accuracy than ToRA using fewer samples; the discussion also notes that answer-only datasets are harder to enrich.

D More Results on Self-Consistency

E Discussion on Datasets Only with Final Answers

  • Enriching datasets that contain only final answers is more challenging, and the authors leave an extension to this setting for future work.Their proposed extension would first generate sparse responses, then progressively enrich correct-answer trajectories.Source: E Discussion on Datasets Only with… · p. 13

F Datasets Details

The paper uses MATH and GSM8K, which differ in problem level and provide training and testing splits with step-by-step or fine-grained solutions.

F Datasets Details

  • MATH contains 12,500 competition-level problems, split into 7,500 for training and 5,000 for testing.Each problem has a step-by-step solution and a distinct final answer.Source: F Datasets Details · p. 13
  • MATH spans seven subject categories, from Prealgebra and Algebra to Geometry and Precalculus.The other listed categories are Number Theory, Counting and Probability, and Intermediate Algebra.Source: F Datasets Details · p. 13
  • GSM8K is a grade-school word-problem dataset with 7,473 training problems and 1,319 testing problems.Its solutions provide more fine-grained steps involving basic arithmetic operations; it is generally considered less challenging than MATH.Source: F Datasets Details · p. 13

G Analysis on Case Study

The case study contrasts EIT’s guided, detailed reasoning with errors in SFT and MetaMath; the accompanying prompts illustrate how EIT generates plans and enriches answers.

G Analysis on Case Study

  • EIT’s plan guides the yards-to-inches solution through an improper fraction and conversion factor, avoiding SFT’s calculation error and MetaMath’s missed relationship.EIT returns the correct answer after multiplying 13 by 36 and dividing by 4.Source: G Analysis on Case Study · pp. 13–14

H.1 Prompt for Enriching GSM8K (ERP and ERS)

  • The GSM8K enrichment prompt asks for a plan followed by a solution that preserves the supplied arithmetic and final answer.Its example plans separate calculations for large and small pizzas before adding the totals.Source: §H.4 · p. 17

H.2 Prompt for Enriching MATH (ERP and ERS)

  • The MATH ERP-and-ERS prompt asks the answer enricher to fill logical gaps without changing the original pathway, formulas, or results.Source: §H.4 · p. 18

H.3 Prompt for Enriching MATH (ERP)

  • The MATH ERP prompt requests a high-level plan based on the question and answer to guide the detailed response.Source: §H.4 · p. 18

H.4 Prompt for Enriching MATH (ERS)

  • The ERS prompt asks for missing contexts and logical gaps while preserving pathways and formulas; its worked example converts 3 1/4 yards to 117 inches.The example uses 36 inches per yard and multiplies the improper fraction 13/4 by 36.Source: §H.4 · pp. 15, 19

Related papers

Loading 2412.16964v2…