Learning From Mistakes Makes LLM Better Reasoner

Shengnan An; Zexiong Ma; Zeqi Lin; Nanning Zheng; Jian-Guang Lou; Weizhu Chen

Source-linked AI summary

Learning From Mistakes Makes LLM Better Reasoner

Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, Weizhu Chen

arXiv:2310.20689v4Oct 31, 2023cs.CLcs.AI

TL;DR

LEMA asks whether LLM reasoning can improve by learning from mistakes, in addition to learning from correct reasoning. It generates corrections for inaccurate paths and fine-tunes with both correction and CoT data; results show consistent gains over CoT-only fine-tuning across models and tasks.

Problem
The paper investigates whether LLM reasoning can improve beyond fine-tuning on correct question-rationale examples by learning from mistakes.
Source: §1 · p. 2
Method
LEMA uses a corrector to identify and explain errors in inaccurate reasoning paths, generate corrected solutions, and combines these correction pairs with CoT data for fine-tuning.
Source: §1 · p. 2
Results
LEMA consistently improves over CoT-only fine-tuning across five open-source LLMs and five reasoning tasks; LLaMA-2-70B reaches 83.5% on GSM8K versus 81.4% with CoT alone.
Source: §1 · p. 2
Takeaways & Limitations
Controlled ablations find that CoT and correction data have non-homogeneous effectiveness, and moderately difficult questions are more suitable for expanding correction data.
Source: §1 · p. 3
The use of pre-trained language models may expose the work to risks such as social bias and privacy breaches.
Source: Ethics Statement · p. 10

Abstract

from arXiv · show

Large language models (LLMs) recently exhibited remarkable reasoning capabilities on solving math problems. To further improve their reasoning capabilities, this work explores whether LLMs can LEarn from MistAkes (LEMA), akin to the human learning process. Consider a human student who failed to solve a math problem, he will learn from what mistake he has made and how to correct it. Mimicking this error-driven learning process, LEMA incorporates mistake-correction data pairs during fine-tuning LLMs. Specifically, we first collect inaccurate reasoning paths from various LLMs, and then employ GPT-4 as a ''corrector'' to identify the mistake step, explain the reason for the mistake, correct the mistake and generate the final answer. In addition, we apply a correction-centric evolution strategy that effectively expands the question set for generating correction data. Experiments across various LLMs and reasoning tasks show that LEMA effectively improves CoT-alone fine-tuning. Our further ablations shed light on the non-homogeneous effectiveness between CoT data and correction data. These results suggest a significant potential for LLMs to improve through learning from their mistakes. Our code, models and prompts are publicly available at https://github.com/microsoft/LEMA.

1 Introduction

The paper asks whether LLMs can improve mathematical reasoning by learning from their mistakes, beyond fine-tuning on correct reasoning alone. LEMA combines correction data with CoT data; experiments report gains across models and tasks.

1 Introduction

The study examines whether learning from mistakes can further improve reasoning beyond fine-tuning on question-rationale CoT data.The motivation is that learners can improve by identifying and correcting mistakes after failed attempts.Source: §1 · p. 2
LEMA uses GPT-4 to correct inaccurate reasoning paths, then combines correction data with CoT data for fine-tuning.Corrections identify the incorrect step, explain the mistake, and revise the solution; moderately difficult questions are prioritized when expanding the question set.Source: §1 · p. 2
83.5% on GSM8K and 25.0% on MATH with LLaMA-2-70B, versus 81.4% and 23.6% with CoT-only fine-tuning.Correction-centric evolution further raises LLaMA-2-70B on MATH from 25.0% to 29.3%.Source: §1 · p. 2
With training data sizes and tokens controlled, mixing CoT and correction data outperforms either single source, and correction-centric evolution beats random seed selection.The authors report that moderately difficult questions are more suitable for expanding correction data.Source: §1 · p. 3

2 Methodology

LEMA generates correction pairs from inaccurate reasoning paths, expands the question set with correction-centric evolution, and fine-tunes on both correction and CoT data. Its correction pipeline filters for corrected answers matching the gold answer.

2 Methodology

LEMA has three stages: generating correction data, correction-centric evolution, and fine-tuning.Correction pairs are blended with CoT data for fine-tuning.Source: §2 · p. 3, §2.3 · p. 5

2.1 Correction Data Generation

The pipeline retains incorrect reasoning paths and uses a corrector to produce a checked correction identifying the error, explaining it, and giving a corrected solution.The correction is retained when its final answer matches the gold answer; GPT-4-generated corrections were judged sufficient for fine-tuning, with 35 excellent, 11 good, and 4 poor among 50 evaluated.Source: §2.1 · pp. 3–4

2.2 Correction-Centric Evolution

Correction-centric evolution selects seed questions already present in correction data, focusing expansion on moderately difficult questions rather than sampling seeds randomly.The strategy targets questions that reasoning models often fail but whose inaccurate paths the corrector can handle; very easy or very challenging questions are less valuable for collecting corrections.Source: §2.2 · pp. 4–5

2.3 Fine-Tuning LLMs

The fine-tuning comparison is between CoT-only data and LEMA, which trains on CoT data plus generated mistake-correction data.The main results table covers four mathematical tasks—GSM8K, MATH, SVAMP, and ASDiv—and commonsense task CSQA.Source: §2.3 · p. 5, Table 1 · p. 5

3 Experimental Setup

Experiments cover five reasoning tasks, with GSM8K training data also used to evaluate transfer to SVAMP and ASDiv. The study constructs CoT and correction data, then fine-tunes and evaluates open-source LLMs.

3.1 Tasks

The evaluation spans four math tasks—GSM8K, MATH, SVAMP, and ASDiv—and the commonsense reasoning task CSQA.SVAMP and ASDiv use GSM8K training data; CSQA receives four manually annotated CoT examples before its training set is used for augmentation and correction generation.Source: §3.1 · pp. 5–6

3.2 Data Construction

Correction data pairs inaccurate paths collected from multiple LLMs with GPT-4-generated corrections; CoT data uses GPT-4-generated reasoning paths filtered for incorrect answers or format.Correction-centric evolution samples 10K questions from correction data and uses GPT-4 to evolve them, retaining questions whose three sampled answers do not disagree.Source: §3.2 · p. 6

3.3 Fine-Tuning and Evaluation

The experiments fine-tune LLaMA-family and specialized math models, defaulting to QLoRA, and report the best checkpoint's accuracy under greedy decoding.Evaluation uses a shared CoT instruction without demonstrations, and LEMA models do not generate corrections at inference.Source: §3.3 · pp. 6–7

4 Results and Analysis

LEMA improves CoT-alone fine-tuning across the evaluated backbones and tasks, including controlled-data comparisons and specialized models. Correction-centric evolution further improves results, while its scaling benefit is greater with full fine-tuning than with QLoRA.

4.1 Can LLMs Learn From Mistakes?

LEMA improves CoT-alone fine-tuning across all five backbone LLMs and five tasks, with gains on SVAMP and ASDiv indicating some out-of-distribution generalizability.These tasks use GSM8K training data.Source: §4.1 · p. 7
With equal data sizes, LEMA gains persist for four of five backbone LLMs; LLaMA-2-7B is the exception.The result supports the authors' conclusion that stronger backbones can learn more effectively from mistakes.Source: §4.1 · p. 7
With 5.8M training tokens, LEMA still outperforms CoT-alone fine-tuning, even though the matched-token comparison adds CoT samples.The authors also report 82.2% accuracy for LLaMA-2-70B on GSM8K with CoT-6.8M, noting that adding more paths does not continuously improve performance.Source: §4.1 · p. 8
LLaMA-2-70B gains over 1% accuracy on all five tasks, while gains for the four less powerful models are occasionally less significant.Together with LLaMA-2-7B's exception under controlled data sizes, this suggests backbone strength influences learning from mistakes.Source: §4.1 · pp. 7–8
LEMA further improves specialized math models WizardMath and MetaMath on GSM8K relative to results reported in their original papers.The comparison uses the original papers' reported results because these models were already trained on substantial math CoT data.Source: §4.1 · p. 8

4.2 How Beneficial Is Correction-Centric Evolution?

Correction-centric evolution outperforms general evolution, and the authors report that moderately difficult questions are more suitable for expanding correction data.The comparison uses the same 10K seed questions for both strategies.Source: §4.2 · pp. 8–9
As the question set expands, LEMA improves significantly with full fine-tuning but only slightly with QLoRA.The authors interpret this trend as indicating that parameter-efficient fine-tuning can digest only a limited scale of correction data.Source: Figure 5 · p. 8, §4.2 · p. 9

5 Related Work

Prior work improves mathematical CoT through methods such as verification, tools, pretraining, and data augmentation. This work extends the data-augmentation perspective by examining mistake-correction data as auxiliary supervision.

Enhancing CoT reasoning for solving mathematical problems

Prior approaches to math reasoning include sampling or verification, executable programs and tools, math-related pretraining, and dataset augmentation.The paper places its approach within data augmentation.Source: §5 · p. 9

Data augmentation for mathematical tasks

LLM-assisted augmentation has generated CoT through rejection sampling, question evolution, query or response augmentation, and self-verification or FOBAR.The cited approaches include Yuan et al. (2023), Xu et al. (2023), Li et al. (2023a), and Yu et al. (2023).Source: §5 · p. 9
LEMA examines mistake-correction data as auxiliary supervision, informing models about errors in CoT reasoning and how to correct them.The paper frames this as a further step beyond existing auxiliary data such as code, reranking, and verification data.Source: §5 · pp. 9–10

6 Conclusion

The work explores whether learning from mistakes can further improve LLM reasoning, and reports experimental evidence for its effectiveness and potential.

LEMA explores whether learning from mistakes can further improve LLM reasoning, with experiments and in-depth analysis demonstrating its effectiveness and potential.Source: §6 · p. 10

Ethics Statement

The paper notes potential ethical risks associated with using pre-trained language models and discusses LLMs’ difficulty with self-correction. It outlines directions for more reliable self-correction.

Ethics Statement

Using pre-trained language models may expose this work to ethical risks such as social bias and privacy breaches.Source: Ethics Statement · p. 10

A.1 LLMs for Self-Correction

Advanced LLMs still struggle with self-correction; proposed directions include external supervision, process-based verification, and honest capability signaling.Source: §A.1 · p. 16

A.2 Training with Feedback

LEMA frames its approach as feedback-based instruction fine-tuning and as using GPT-4 as a “world model” to teach smaller models logic and rules.

A.2 Training with Feedback

LEMA injects GPT-4 feedback into smaller models through instruction-based fine-tuning rather than reinforcement-learning methods.The authors say this can save effort otherwise spent designing specialized learning algorithms.Source: §A.2 · p. 16

A.3 Learning From The World Model

LEMA uses GPT-4 as a “world model” to teach smaller models to follow logic and rules, rather than merely imitate step-by-step reasoning.Source: §A.3 · p. 16

B.1 Examples in Human Evaluation

The appendices show examples and describe data formats, question evolution, dataset statistics, and a numerical-answer filter for ASDiv evaluation.

B.1 Examples in Human Evaluation

Figure 13 presents three example corrections at different quality levels.Source: §B.1 · p. 16

B.2 Input-Output Formats for Fine-Tuning

For CoT and correction data, the input serves as the prompt and only output-part loss contributes to back-propagation.Source: §B.2 · p. 17, Figure 6 · p. 17

B.3 Evolution Prompt

The evolution prompt is used to generate new questions from a given seed question.Source: §B.3 · p. 17, Figure 7 · p. 17

B.1 Examples in Human Evaluation

Figure 8 provides an annotated CoT example for CSQA.Source: Figure 8 · p. 18

C.1 Data Statistics

Table 4 reports task and data-size statistics for the five experimental tasks without question evolution.Source: §C.1 · p. 18, Table 4 · p. 18

C.2 Evaluation on ASDiv

ASDiv evaluation retains 2,084 of 2,305 questions by filtering out answers that cannot be parsed as numbers by float(·).The authors note that the models were trained on GSM8K data with numerical answers.Source: §C.2 · p. 18

C.3 Data Construction For CSQA

Across the supplied sections, LEMA is compared with stronger math models, tested across fine-tuning checkpoints and training curves, and analyzed for its data construction and correction components.

C.3 Data Construction For CSQA

CSQA training examples provide only final answers, so the authors generate rationales containing candidate explanations, a predicted answer, and its selection rationale.GPT-4 generates the other rationales, and rationales without correct final answers are filtered.Source: §C.3 · pp. 18–19

C.4 Full Fine-Tuning Setting

Full fine-tuning uses a learning rate of 1e-5 and batch size 128; LLaMA-2-70B trains for three epochs and Llemma-34B for two.Results are reported on the final checkpoints.Source: §C.4 · p. 19

C.5 Another Round of Correction-Centric Evolution

A second correction-centric evolution round uses the same 10K seed questions but collects inaccurate paths from first-round fine-tuned models instead of vanilla models.Source: §C.5 · p. 19

D.1 Performances of Best Three Checkpoints

The best-three-checkpoint results and their average show that the main results are not caused by random training disturbances.Table 5 reports the best three checkpoints and their average; no specific values are supplied here.Source: §D.1 · p. 19, Table 5 · p. 19

D.2 Training Curves

Adding correction data produces clear improvements during LLaMA-2-70B’s 2,000-step fine-tuning, robust to random training disturbances.Figure 9 shows the performance curves over 2,000 steps.Source: Figure 9 · p. 19, §D.2 · p. 20

D.3 Comparison with SOTA Models

With about 70K CoT plus correction examples, LLaMA-2-70B + LEMA is comparable to MuggleMath-70B and MetaMath-70B, which use about 220K and 400K CoT examples, respectively.The comparison supports non-homogeneous effectiveness between CoT and correction data.Source: §D.3 · p. 20

D.4 Ablations of Correction Information

Omitting the corrected solution or mistake explanation significantly reduces LEMA performance, while omitting the explicit mistake step has less influence.The authors suggest the corrected solution and explanation may implicitly identify the incorrect step.Source: §D.4 · pp. 20–21

D.5 Additional Analysis to LEMA

LEMA outperforms CoT-alone fine-tuning even when the question distributions are matched. The analysis also reports a growing gap between full fine-tuning and QLoRA as correction data expands, and higher ∆PPL under LEMA.

D.5 Additional Analysis to LEMA

LEMA outperforms CoT-alone fine-tuning when both use the same question distribution.The comparison uses LEMA-45K and CoT-45K, whose questions are exactly the same.Source: §D.5 · p. 21
As correction data expands, the performance gap between full fine-tuning and QLoRA fine-tuning increases.The authors suggest correction data may be harder for parameter-efficient methods to incorporate because it is auxiliary to in-task training.Source: §D.5 · p. 21
LEMA consistently produces a higher ∆PPL than CoT-alone fine-tuning on the evaluated fine-tuned models.The comparison covers LLaMA-2-70B and LLaMA-65B using correction data for GSM8K and MATH.Source: §D.5 · p. 22

D.6 Further Analysis on Corrector

GPT-4 is used as the corrector because nearly half of 20 GPT-3.5-Turbo corrections were rated poor quality. GPT-4 corrects its own MATH mistakes at a low rate, and correction success declines as problem difficulty increases.

D.6 Further Analysis on Corrector

Nearly half of 20 GPT-3.5-Turbo-generated corrections were rated poor, so the authors use the more expensive GPT-4 corrector.They identify generating high-quality corrections without GPT-4 as a research direction.Source: §D.6 · p. 22
GPT-4 achieves an 8.0% self-correction success rate on its MATH training-set mistakes, versus 37.5% for LLaMA-2-70B mistakes and 26.9% for GPT-3.5-Turbo mistakes.The reported GPT-4 rate is based on 217 correct-final-answer corrections from 2,696 inaccurate reasoning paths.Source: §D.6 · p. 22
On MATH, correction success rates significantly decline as difficulty rises from Level 1 to Level 5.Collected inaccurate paths increase with difficulty, while correct corrections first increase and then decrease.Source: §D.6 · pp. 22–23