PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs cannot find reasoning errors, but can correct them given the error location

Gladys Tyen, Hassan Mansoor, Victor Cărbune, Peter Chen, Tony Mak

arXiv:2311.08516v3cs.AIcs.CLcs.LG

TL;DR

Self-correction often fails on reasoning errors because LLMs cannot reliably identify the mistakes they need to fix. The paper benchmarks mistake finding, tests correction with supplied locations, and trains an out-of-domain classifier; supplied locations improve downstream performance, while the classifier outperforms prompting a large model on mistake finding.

  • Problem

    Existing self-correction methods struggle with reasoning errors, motivating separate evaluation of LLM mistake finding and output correction.

  • Method

    The paper benchmarks five LLMs, evaluates backtracking with ground-truth mistake locations, and trains a small mistake-finding classifier on out-of-domain data.

  • Results

    Supplied mistake locations improve downstream performance across five reasoning tasks, and an out-of-domain classifier outperforms few-shot prompting for mistake finding.

  • Takeaways & Limitations

    Mistake identification, rather than correction, is the main bottleneck in self-correction for reasoning tasks within the paper’s evaluated settings.

  • Takeaways & Limitations

    The dataset uses artificial, unrealistic tasks, and backtracking is evaluated only on a limited, skewed sample rather than the original BIG-Bench datasets.

Abstract

from arXiv · show

While self-correction has shown promise in improving LLM outputs in terms of style and quality (e.g. Chen et al., 2023b; Madaan et al., 2023), recent attempts to self-correct logical or reasoning errors often cause correct answers to become incorrect, resulting in worse performances overall (Huang et al., 2023). In this paper, we show that poor self-correction performance stems from LLMs' inability to find logical mistakes, rather than their ability to correct a known mistake. Firstly, we benchmark several state-of-the-art LLMs on their mistake-finding ability and demonstrate that they generally struggle with the task, even in highly objective, unambiguous cases. Secondly, we test the correction abilities of LLMs -- separately from mistake finding -- using a backtracking setup that feeds ground truth mistake location information to the model. We show that this boosts downstream task performance across our 5 reasoning tasks, indicating that LLMs' correction abilities are robust. Finally, we show that it is possible to obtain mistake location information without ground truth labels or in-domain training data. We train a small classifier with out-of-domain data, which exhibits stronger mistake-finding performance than prompting a large model. We release our dataset of LLM-generated logical mistakes, BIG-Bench Mistake, to enable further research into locating LLM reasoning mistakes.

1 Introduction

The paper separates self-correction into mistake finding and output correction, arguing that LLMs struggle to locate reasoning errors even when cases are objective and unambiguous. It tests whether correction remains effective when mistake locations are supplied and whether classifiers can provide those locations.

  • Dataset: BIG-Bench Mistake is a released dataset of CoT-style traces annotated by the location of the first logical mistake.The dataset is intended to support research on locating LLM reasoning mistakes.
  • Motivation: LLMs struggle to find logical mistakes reliably, even in objective, unambiguous cases.This inability is hypothesized to contribute substantially to failures of self-correction on reasoning tasks.
  • Approach: Separating mistake finding from output correction shows that models can correct outputs when given mistake location information.The paper uses a backtracking setup that supplies ground-truth or oracle locations and reports robust correction with minimal effect on originally correct outputs.
  • Classifier-based mistake finding: A small classifier trained on out-of-domain data performs better at mistake finding than prompting a large model, without in-domain training data.The authors present this as a proof of concept and leave more sophisticated methods for future work.

2 BIG-Bench Mistake

BIG-Bench Mistake is a dataset of 2,186 PaLM 2 Unicorn CoT traces across five BIG-bench tasks, annotated with the first logical error or no mistake. Human and automatic procedures provide task-appropriate annotations, with subsequent errors ignored because they may depend on the first.

  • Dataset: BIG-Bench Mistake contains 2,186 CoT trace sets generated with PaLM 2 Unicorn and annotated for the first logical error.A mistake location is the step containing the first error, while traces without mistakes receive a no-mistake label.
  • Release: The released resource includes the BIG-Bench Mistake dataset and annotation materials for further study of LLM reasoning errors.The dataset is publicly released, and annotation guidelines and code are provided online.
  • Dataset: The dataset spans word sorting, shuffled-object tracking, logical deduction, multi-step arithmetic, and Dyck languages.These tasks were chosen because PaLM 2 performs poorly on them, their mistakes are relatively unambiguous, and annotation does not require specialist expertise.
  • Annotation: Each trace is annotated only at its first logical error, because later errors may depend on the original mistake.For four tasks, at least three annotators label steps and disagreements are resolved by majority vote; Dyck traces are mostly annotated automatically.
  • Annotation: The annotation set deliberately includes both incorrect and correct final answers, since a trace can contain a logical mistake yet still reach the correct answer.The initial sample uses 255 incorrect-answer and 45 correct-answer traces per task to expose more potentially erroneous steps while retaining correct-answer cases.

3 Can LLMs find reasoning mistakes in CoT traces?

The study evaluates whether LLMs can locate the first mistake in CoT traces using trace-level and step-level prompting. Across five tasks, the models struggle substantially: GPT-4 reaches only 52.87 overall accuracy with direct step-level prompting, while more elaborate prompting trades off performance on no-mistake traces.

  • 3.1 Discussion: The benchmark compares direct trace-level, direct step-level, and CoT step-level prompting, with increasing reasoning output and computational cost.Trace-level prompting predicts one location directly; step-level methods inspect each step, while CoT step-level prompting generates a reasoning sequence for every step.
  • 3.1 Discussion: 52.87 overall accuracy is GPT-4’s best reported mistake-location result, achieved with direct step-level prompting across the benchmark.The evaluation requires exact prediction of the first erroneous step or a correct no-mistake prediction; all five models struggle overall.
  • 3.1 Discussion: State-of-the-art LLMs struggle to find reasoning mistakes even in objective, unambiguous cases, supporting mistake finding as a bottleneck for self-correction.The paper hypothesizes that inability to identify errors contributes to failures in self-correcting reasoning errors.
  • 3.2 Comparison of prompting methods: Figure 1 shows that accuracy on traces with no mistakes decreases from direct trace-level prompting to CoT step-level prompting.The authors attribute this trade-off to the increasing number of generation calls, which raises the chance of identifying at least one mistake.
  • 3.3 Few-shot prompting for mistake location as a proxy for correctness: Mistake prompting is a poor proxy for final-answer correctness, with weighted F1 scores compared against a baseline of 78 from always predicting incorrectans.The result aligns with the finding that models struggle to locate mistakes and with prior evidence that some self-correction gains rely on oracle correctness information.

4 Can LLMs correct reasoning mistakes in CoT traces?

The backtracking experiment separates mistake correction from mistake finding by supplying mistake locations, showing that LLMs can improve reasoning outputs when the error location is known.

  • 4.1 Method: The method generates an initial CoT trace, identifies a mistake location, and regenerates the erroneous step using the preceding partial trace.If no mistake is found, the original trace is retained; regenerated continuations use temperature 0.
  • 4.1 Method: The same model is used to correct its original traces, with random-location backtracking providing a baseline against oracle-location backtracking.The setup is deliberately simple and avoids specialized correction prompts or many alternative generations.
  • 4.2 Discussion: The dataset contains 15% originally correct traces and 85% originally incorrect traces, so overall-accuracy changes depend on the starting accuracy.With only 45 correct traces per task, the authors defer a more comprehensive assessment of backtracking.
  • 4.2 Results: For word sorting, 11.11% of originally correct traces became incorrect, while 23.53% of originally incorrect traces became correct.Across the reported results, gains on incorrect traces exceed losses on originally correct traces, and oracle locations outperform random locations.
  • 4.2 Discussion: Using oracle mistake locations, backtracking improves overall downstream performance, indicating that LLMs can correct their own outputs.The authors identify mistake finding, rather than correction, as the main bottleneck in self-correction.

5 Obtaining mistake location information with a trained classifier

The paper replaces unavailable oracle mistake locations with trained classifiers, testing both the accuracy needed for effective backtracking and transfer from out-of-domain data.

  • 5 Obtaining mistake locations: The classifier approach is motivated by lower resource requirements, compatibility with any generator LLM, and more interpretable error localization.Independent updates are particularly useful for API-based models that cannot be fine-tuned.
  • 5 Obtaining mistake locations: The section asks whether imperfect mistake-finding can support backtracking and whether mistake-finding transfers across tasks without in-domain training data.These questions frame the accuracy simulation and out-of-domain classifier experiments.
  • 5.1 Required mistake-finding accuracy: The study simulates classifier accuracies by using gold locations at the target accuracy rate and random, distribution-matched locations otherwise.The sampled incorrect locations are constrained not to match the true location.
  • 5.1 Required mistake-finding accuracy: Backtracking remains effective for most tasks at roughly 60–70% mistake-finding accuracy, although higher accuracy produces better results.Losses on originally correct traces begin to plateau at 65%.
  • 5.2 Training a classifier on out-of-domain data: A small PaLM 2 Otter classifier trained on out-of-domain BIG-Bench Mistake data improves over few-shot prompting on 4 of 5 tasks.The classifier is substantially smaller than the inference model and can identify out-of-distribution mistakes.
  • 5.2 Training a classifier on out-of-domain data: The trained classifiers do not reach the accuracy threshold required for effective backtracking.The authors attribute the gap partly to the need for more data and leave model-size versus classifier-performance trade-offs for future work.

6 Related work

Related work situates the paper within self-correction research, distinguishing feedback sources and the stage at which correction is applied.

  • Related work: BIG-Bench Mistake broadens publicly available mistake annotations beyond PRM800K by covering five reasoning tasks and API-based LLM prompting.PRM800K focuses on Olympiad-level mathematics and a math-specialized generator.
  • Related work: Self-correction methods vary by feedback source, including task-generated signals, human feedback, trained feedback models, and prompting alone.Prompt-only approaches generally require external feedback for reasoning tasks.
  • Related work: Methods also differ in when correction occurs: during training, during generation, or after an output has already been generated.The paper positions its approach among methods that correct completed outputs.

7 Conclusion

The paper concludes that LLMs struggle to locate reasoning mistakes but can correct outputs when locations are supplied, making mistake finding a likely self-correction bottleneck.

  • 7 Conclusion: LLMs generally struggle to find mistakes but can correct outputs with mistake location information, boosting downstream performance.The conclusion frames mistake finding as an important bottleneck for reasoning self-correction.
  • 7 Conclusion: A small classifier trained on out-of-domain data improves mistake-finding accuracy over few-shot prompting, although it remains below the threshold needed for effective backtracking.BIG-Bench Mistake is released to support further research on locating reasoning mistakes.

Limitations

The dataset uses artificial, unrealistic tasks to reduce ambiguity during mistake finding, limiting conclusions about backtracking in realistic settings. The evaluation also excludes full original BIG-Bench datasets and instead uses a skewed sample, leaving broader validation for future work.

  • Limitations: Artificial, unrealistic tasks reduce ambiguity and subjectivity during mistake finding but limit evidence about backtracking in realistic settings.The authors identify this as a main dataset limitation and call for further work in more realistic settings.
  • Limitations: The study evaluates backtracking only on a limited, skewed sample rather than the original BIG-Bench datasets.The sampling was designed to maximise the value of human annotators’ time.
  • Limitations: Full evaluation on the original BIG-Bench datasets remains future work because this paper is positioned as a proof of concept for the importance of mistake finding.

A Dataset details

BIG-Bench Mistake contains English-language traces generated with PaLM 2 and annotated at precise Chain-of-Thought steps. The dataset and annotation resources use structured prompting and an interface designed to support consistent mistake localization.

  • Dataset construction: BIG-Bench Mistake uses PaLM 2-generated traces from five English-language BIG-Bench-based tasks.The dataset and underlying BIG-Bench tasks are released under the Apache License 2.0.
  • Prompt and trace format: Prompt examples are split into smaller steps and labeled “Thought N” so annotators and models can refer to precise mistake locations.The newline stop token produces one step per generation call, while the step labels support systematic splitting and location references.
  • Task adaptations: The dataset modifies logical-deduction prompts to replace ambiguous question-mark notation with textual object descriptions.This change reduces uncertainty about whether a question mark itself constitutes a mistake.
  • Mistake-location prompting: Mistake location is prompted using direct trace-level, direct step-level, or Chain-of-Thought step-level methods with 3-shot examples.The prompts follow OpenAI’s chat completion format and use temperature 0 without stop tokens for the reported results.
  • Release: The authors release BIG-Bench Mistake, its annotation guidelines, prompts, and examples for further research.The dataset and resources are available through the project repository.
  • Annotation interface: The annotation interface displays each question, target answer, and trace while enabling word highlighting and disabling later labels after a negative step.These controls were particularly helpful for word sorting and tracking shuffled objects.

C Training mistake-finding classifiers

The authors train task-held-out binary classifiers to judge whether each Chain-of-Thought step is correct given the task and preceding steps. Classifier training uses fixed optimization settings and selects checkpoints by validation performance.

  • Classifier training: Each mistake-finding classifier is trained on four tasks and evaluated on the fifth, repeating this holdout procedure across all five tasks.This trains classifiers with out-of-domain task data for each held-out evaluation task.
  • Optimization: Training runs for 20k steps with batch size 32, learning rate 1e−5, linear ramp, cosine decay, and validation-based checkpoint selection.The number of training steps for each classifier is listed in Table 8.
  • Classifier objective: The classifiers predict whether a Chain-of-Thought step is correct given the task and previous steps.Steps after the first annotated mistake are labeled incorrect during training because limited data motivates their inclusion.

E Benchmark scores

Benchmark scores are reported separately for traces without mistakes and traces containing mistakes, with combined scores forming the overall mistake-finding results. The accompanying table covers accuracy across five tasks.

  • Benchmark scores: The benchmark therefore separates performance on identifying mistake-free traces from performance on traces containing mistakes.
  • Benchmark scores: Table 9 reports mistake-finding accuracy across five tasks separately for correctmis and incorrectmis traces.The two subtables distinguish traces that do not contain mistakes from traces that do contain mistakes.
  • Benchmark scores: The combined scores from the two trace categories make up the overall results reported in Table 4.
Loading 2311.08516v3…