PaperlayerView exact revision on arXiv

Source-linked AI summary

Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems

Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, Bo Du

arXiv:2404.14963v5cs.CLcs.AI

TL;DR

CoT prompting improves LLM reasoning, but complex math word problems remain difficult, with semantic misunderstanding identified as a major error that prior methods neglect. DUP prompts models to extract the core question and relevant problem-solving information before answering. It reports consistent gains across reasoning benchmarks and new SOTA accuracy of 97.1% on GSM8K in the zero-shot setting.

  • Problem

    Prior prompting methods address calculation and step-missing errors, but semantic misunderstanding remains a major, under-explored limit on LLM reasoning for complex math word problems.

    Source: §1 · p. 1

  • Method

    DUP prompts LLMs to reveal the core question, extract relevant problem-solving information, and generate and extract the answer using both.

    Source: §1 · p. 2

  • Results

    DUP consistently outperforms other counterparts across reasoning datasets and achieves new SOTA accuracy of 97.1% on GSM8K and 94.2% on SVAMP.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that deeply understanding the whole problem is crucial for complex reasoning tasks.

    Source: §5 · p. 9

Abstract

from arXiv · show

Chain-of-Thought (CoT) prompting has enhanced the performance of Large Language Models (LLMs) across various reasoning tasks. However, CoT still falls short in dealing with complex math word problems, as it usually suffers from three pitfalls: semantic misunderstanding errors, calculation errors, and step-missing errors. Prior studies involve addressing the calculation errors and step-missing errors, but neglect the semantic misunderstanding errors, which is the major factor limiting the reasoning performance of LLMs. To this end, we propose a simple-yet-effective method, namely Deeply Understanding the Problems (DUP), to improve the LLMs' math problem-solving ability by addressing semantic misunderstanding errors. The core of our method is to encourage the LLMs to deeply understand the problems and extract the key problem-solving information used for better reasoning. Extensive experiments on 10 diverse reasoning benchmarks show that our DUP method consistently outperforms the other counterparts by a large margin. More encouragingly, DUP achieves a new SOTA result on the GSM8K benchmark, with an accuracy of 97.1% under the zero-shot setting.

1 Introduction

The introduction identifies semantic misunderstanding as the main CoT error limiting reasoning performance and proposes DUP to make LLMs extract the core question and relevant problem-solving information. DUP outperforms other counterparts and reaches reported state-of-the-art accuracy on GSM8K and SVAMP.

1 Introduction

  • Semantic misunderstanding is identified as CoT’s main reasoning limitation, while prior prompting work mainly reduces calculation and step-missing errors.The introduction frames reducing semantic misunderstanding as the central question for improving reasoning.Source: §1 · pp. 1–2
  • DUP reduces semantic misunderstanding, calculation, and step-missing errors in an analysis of 300 sampled GSM8K problems.The analysis assigns each incorrect answer to one of the three error types.Source: Figure 1 · p. 2
  • Complex math word problems can contain irrelevant content that may distract LLMs from the core question and relevant information.The authors connect this motivation to prior findings that irrelevant information can reduce some children’s and adults’ problem-solving accuracy.Source: §1 · p. 2
  • DUP has three stages: reveal the core question, extract relevant problem-solving information, then generate and extract the answer using both.The method is intended to filter irrelevant information and support better math problem solving.Source: §1 · p. 2
  • DUP achieves new SOTA results on GSM8K (97.1%) and SVAMP (94.2%), outperforming other counterparts across datasets and few-shot methods on most reasoning datasets.The authors also report that zero-shot DUP outperforms zero-shot counterparts by a large margin.Source: §1 · p. 2

2 Related Works

Prior work improves LLM reasoning through training and prompting, but complex reasoning remains challenging. Existing prompting methods address calculation and step-missing errors more than semantic misunderstanding, which this paper identifies as under-explored.

2.1 Reasoning with Large Language Models

  • LLMs continue to struggle with complex reasoning despite progress from reasoning-augmented models trained on mathematical data.The related-work section notes that such training-based approaches can require substantial data and computation and may reduce universal ability.Source: §2.1 · p. 2

2.2 Prompting Methods

  • Prompting methods strengthen reasoning without additional training, but remain limited on complex tasks such as math word problems.The review traces this line from few-shot CoT to later prompting strategies.Source: §2.2 · p. 3
  • Prior prompting work mainly targets calculation and step-missing errors, leaving semantic misunderstanding comparatively under-explored.The authors identify semantic misunderstanding as the major error type motivating their work.Source: §2.2 · p. 3
  • DUP applies human-inspired problem understanding by extracting core and relevant information, and is described as usable across LLMs without training cost.The authors present the approach as easy to apply to various LLMs.Source: §2.2 · p. 3

3 DUP Prompting

DUP addresses semantic misunderstanding by having LLMs identify a problem’s core question and the information needed to answer it before reasoning. Its three stages clarify the question, extract relevant information, and generate and extract a final answer.

3.1 Overview

  • DUP targets semantic misunderstanding, which the authors identify as a major but under-studied limit on LLM reasoning.The method is introduced as a zero-shot CoT prompting approach that encourages fuller problem understanding.Source: §3.1 · p. 3

3.2 Stage 1: Reveal the Core Question

  • Stage 1 prompts the LLM to extract the most comprehensive and detailed core question from the original problem.The resulting question is shorter and clearer, helping later steps focus on the input’s goal.Source: §3.2 · p. 3

3.3 Stage 2: Extract the Problem-solving Information

  • Stage 1’s extracted question guides Stage 2 to select only the most useful problem-solving information and list it item by item.The output is information useful for reasoning about the core question.Source: §3.3 · p. 3

3.4 Stage 3: Generate and Extract the Answer

  • The illustrated pipeline proceeds from revealing the core question, to extracting relevant information, to generating and extracting the answer.The final response uses both the core question and problem-solving information.Source: Figure 2 · p. 4, §3.4 · p. 4
  • Stage 3 places the extracted hint and core question into the original input, then asks the LLM to solve step by step.The combined prompt explicitly points to the goal and information needed to solve the problem.Source: §3.4 · p. 4
  • The final numerical answer is extracted from the generated reasoning text using an LLM, which the authors describe as more robust and accurate in practice than rule-based matching.The extraction follows prior work.Source: §3.4 · p. 4

4 Experiments

Across arithmetic, commonsense, and symbolic reasoning tasks, DUP outperforms zero-shot counterparts; with GPT-4, it reaches 97.1% on GSM8K and 94.2% on SVAMP.

4.1 Experimental Setup •

  • Experiments span six arithmetic benchmarks, two commonsense benchmarks, and two symbolic benchmarks, with zero-shot methods as the main comparison and few-shot baselines for reference.The Last Letter task skips DUP’s Stage 2 because it does not rely on problem-solving information; decoding uses temperature 0.Source: Table 1 · p. 4, §4.1 · pp. 4–5

4.2 Main Results •

  • 4% average accuracy gain over Zero-shot CoT on arithmetic benchmarks accompanies GPT-4 DUP results of 97.1% on GSM8K and 94.2% on SVAMP.The GPT-4 scores are reported as new state-of-the-art results.Source: Table 3 · p. 5, §4.2 · pp. 5–6
  • On commonsense tasks, DUP beats all zero-shot counterparts and matches or exceeds few-shot methods; symbolic results show zero-shot gains with some few-shot exceptions.On Last Letters, DUP scores 81.2% versus Zero-shot Least-to-Most at 83.2%; on Coin Flip, DUP scores 97.6% versus few-shot scores of 98.2% and 98.6%.Source: Table 4 · p. 6, Table 5 · p. 6, §4.2 · p. 6

4.3 Ablation Study

  • Removing any DUP stage lowers performance on GSM8K and AQuA, while the full three-stage combination performs best on both.Stage 1 extracts core questions, Stage 2 extracts problem-solving information, and Stage 3 solves step by step.Source: Table 6 · p. 6, §4.3 · p. 6

4.4 Discussion •

  • With self-consistency, DUP beats Zero-shot CoT on GSM8K and SVAMP; GPT-4 extraction outperforms GPT-3.5 extraction, while LLaMA2 extraction performs worse.The self-consistency comparison uses temperature 0.7 and 10 sampled paths.Source: Figure 3 · p. 6, §4.4 · p. 7, Figure 4 · p. 7
  • DUP-s matches DUP overall and performs better on AddSub and SingleEq; open-source LLaMA-2 and CodeLLaMA models using DUP beat Zero-shot CoT on GSM8K and AddSub.DUP-s merges the three-stage prompts into one prompt.Source: Figure 5 · p. 7, §4.4 · pp. 7–8, Table 7 · p. 8

4.5 Error Analysis

  • On randomly selected samples, DUP reduces semantic misunderstanding, calculation, and step-missing errors compared with Zero-shot CoT.The analysis samples 300 examples per dataset, except AQuA, which has 254.Source: Figure 6 · p. 8, §4.5 · p. 8
  • Manual error labels correlate significantly with LLM-based analysis, and the SVAMP case study shows DUP correcting a Zero-shot CoT failure.The authors report the correlation as validation of the reliability of their LLM-based error analysis.Source: Table 9 · p. 9, §4.5 · p. 9, Figure 7 · p. 9

5 Conclusion

The conclusion argues that deeply understanding the whole problem is crucial for complex reasoning and reports consistent performance gains from DUP across reasoning tasks.

5 Conclusion

  • The conclusion identifies deeply understanding the whole problem as crucial for tackling complex reasoning tasks.Source: §5 · p. 9
  • DUP brings consistent and significant performance gains across arithmetic, commonsense, and symbolic reasoning tasks and LLMs.It also outperforms zero-shot counterparts by a large margin and achieves new SOTA results on GSM8K and SVAMP.Source: §5 · p. 9
  • The paper reports that further discussions and systematic analyses reveal when and where DUP works well.Source: §5 · p. 9
  • The authors plan to explore applying DUP to non-reasoning tasks in future work.Source: §5 · p. 9

Related papers

Loading 2404.14963v5…