PaperlayerView exact revision on arXiv

Source-linked AI summary

DotaMath: Decomposition of Thought with Code Assistance and Self-correction for Mathematical Reasoning

Chengpeng Li, Guanting Dong, Mingfeng Xue, Ru Peng, Xiang Wang, Dayiheng Liu

arXiv:2407.04078v3cs.CLcs.AIcs.LG

TL;DR

Open-source LLMs still struggle with complex mathematical reasoning, including the decomposition and tool-feedback needs identified by the paper. DotaMath trains models on decomposed, code-assisted trajectories with self-correction, and reports performance across in-domain and out-of-domain benchmarks. DotaMath-deepseek-7B reaches 64.8% on MATH.

  • Problem

    Open-source LLMs struggle with complex mathematical reasoning, and the paper identifies task decomposition and more tool feedback as needs.

    Source: §1 · p. 2

  • Method

    DotaMath trains LLMs to decompose math problems into subtasks, use code and interpreter feedback, and self-correct using DotaMathQA.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    DotaMath outperforms open-source models across reported in-domain and out-of-domain benchmarks; DotaMath-deepseek-7B scores 64.8% on MATH.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper reports that DotaMath enhances open-source LLM capabilities on complex mathematical tasks.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have made impressive progress in handling simple math problems, yet they still struggle with more challenging and complex mathematical tasks. In this paper, we introduce a series of LLMs that employs the Decomposition of thought with code assistance and self-correction for mathematical reasoning, dubbed as DotaMath. DotaMath models tackle complex mathematical tasks by decomposing them into simpler logical subtasks, leveraging code to solve these subtasks, obtaining fine-grained feedback from the code interpreter, and engaging in self-reflection and correction. By annotating diverse interactive tool-use trajectories and employing query evolution on GSM8K and MATH datasets, we generate an instruction fine-tuning dataset called DotaMathQA with 574K query-response pairs. We train a series of base LLMs using imitation learning on DotaMathQA, resulting in DotaMath models that achieve remarkable performance compared to open-source LLMs across various in-domain and out-of-domain benchmarks. Notably, DotaMath-deepseek-7B showcases an outstanding performance of 64.8% on the competitive MATH dataset and 86.7% on GSM8K. Besides, DotaMath-deepseek-7B maintains strong competitiveness on a series of in-domain and out-of-domain benchmarks (Avg. 80.1%). Looking forward, we anticipate that the DotaMath paradigm will open new pathways for addressing intricate mathematical problems. Our code is publicly available at https://github.com/ChengpengLi1003/DotaMath.

1 Introduction

Open-source LLMs remain challenged by complex mathematical reasoning, with task decomposition and fine-grained tool feedback identified as unmet needs. DotaMath combines decomposition, code assistance, and self-correction, and its models outperform open-source models across the reported benchmarks.

1 Introduction

  • Open-source LLMs struggle with complex mathematical tasks, and the paper identifies missing task decomposition and fine-grained tool feedback as gaps.The introduction contrasts this challenge with stronger performance on simpler math problems.Source: §1 · pp. 1–2
  • DotaMath decomposes problems into logical subtasks, uses code to solve them, displays intermediate results, and trains self-correction through two types of instruction data.The interpreter provides finer-grained feedback, while displaying subtask results also makes responses more readable.Source: §1 · p. 2
  • DotaMathQA combines single-turn and self-correction multi-turn data, using query evolution to augment GSM8K and MATH questions.The dataset is based on the human-curated GSM8K and MATH datasets.Source: §1 · p. 2
  • 64.8% on MATH, 86.7% on GSM8K, and 80.1% average across two in-domain and four out-of-domain benchmarks are reported for DotaMath-deepseek-7B.The paper also reports that DotaMath outperforms open-source models across scales on the in-domain datasets and four out-of-domain benchmarks.Source: §1 · p. 2

2 Related Work

Prior work has explored intermediate reasoning steps, code tools, and synthetic mathematical training data. DotaMath combines logical-task decomposition with code and contributes a dataset with single-turn and self-correction examples.

2 Related Work

  • Earlier approaches used computational expressions and intermediate steps, while later work integrated code tools and combined Chain-of-Thought with coding.The cited paradigms include few-shot computational examples, Chain-of-Thought, and program-of-thought approaches.Source: §2 · p. 2
  • DotaMath distinguishes its problem-solving paradigm by decomposing mathematical problems into logical subtasks and using code to address them.The paper reports superior performance across two in-domain and four out-of-domain math datasets.Source: §2 · p. 2
  • DotaMathQA contributes both single-turn and self-correction multi-turn synthetic data, with effectiveness and generalizability demonstrated across backbones and benchmarks.The paper situates this dataset among work using advanced models to generate supervised fine-tuning data.Source: §2 · p. 3

3 Method

The method combines interpreter-based reasoning with a four-part instruction dataset and supervised fine-tuning. Its interaction procedure supports iterative correction, while the dataset includes single-turn and multi-turn trajectories.

3 Method

  • The method section covers interpreter-based mathematical reasoning, GPT-4 annotation for DotaMathQA, and supervised fine-tuning of foundational LLMs.These are presented respectively as the inference procedure, dataset construction, and training process.Source: §3 · p. 3

3.1 Inference Procedure

  • DotaMath decomposes a problem into subtasks, generates code, and feeds interpreter results back for analysis; it explains errors and repeats the process until solved or the invocation limit.The interaction trajectory records decompositions, code, results, error explanations, and the answer.Source: §3.1 · pp. 3–4

3.2 DotaMathQA Dataset Construction

  • DotaMathQA includes seed and augmented single-turn data plus automatic and rule-based multi-turn data; seed solutions are checked against references, while augmented responses are filtered for execution bugs.Seed data are sampled and re-annotated when needed; the four parts originate from GSM8K and MATH, with augmented queries added to the seed queries.Source: §3.2 · pp. 4–5
  • Automatic multi-turn data concatenate incorrect and corrected solutions, while rule-based data pair an incorrect solution and its error explanation with a correct solution; the four parts contain 80K, 2K, 10K, and 482K instances, respectively.The reported sizes are for Dseed-single, Dauto-multi, Drule-multi, and Daug-single.Source: §3.2 · p. 5

3.3 Supervised Fine-tuning

  • The training set represents each example as a math question and an interaction trajectory, then fine-tunes Llama2-7B, LLaMA3-8B, Llemma-7B, and DeepSeekMath-Base-7B.The objective maximizes the trajectory's log likelihood conditioned on the question equipped with a system prompt.Source: §3.3 · pp. 5–6

4 Experiments

Experiments compare DotaMath with open-source and proprietary models on in-domain and out-of-domain mathematics benchmarks, then examine its data formats, augmentation, and self-correction designs.

4.1 Experimental Setup

  • Evaluation covers GSM8K and MATH in-domain plus Mathematics, SVAMP, TabMWP, and ASDiv out-of-domain, using answer accuracy as the metric.The datasets span grade-school word problems through competition mathematics and include tabular, free-form, and multiple-choice questions.Source: §4.1 · p. 6

4.2 Baselines

  • Baselines include proprietary Claude-3, GPT-3.5, and GPT-4, alongside open-source models such as WizardMATH, MetaMATH, MuggleMATH, RFT, MATHCoder, ToRA, and MARIO.Only part of the results are listed in Table 2; additional results are reported in Table 10.Source: §4.2 · p. 6

4.3 Main Results

  • DotaMath performs best on untrained out-of-domain datasets; its 7B model surpasses most 70B open-source models on GSM8K and exceeds Claude-3 Opus on MATH.Separately, on average, DotaMath-deepseek outperforms the previous best open-source SFT model, DeepSeek-MATH-Instruct, by 4.4 points.Source: §4.3 · p. 6

4.4 Ablation Study

  • Removing decomposition, intermediate-process display, or multi-turn self-correction reduced GSM8K/MATH accuracy by 3.0%/6.0%, 1.1%/2.7%, and 1.3%/3.9%, respectively.These format ablations use unaugmented GSM8K and MATH training data because augmented data is larger and its accuracy is not guaranteed.Source: §4.4 · pp. 7–8, Table 3 · p. 7
  • Augmentation increased accuracy from 82.4 to 86.7 on GSM8K and from 59.0 to 64.8 on MATH.The reported improvements are 4.7% on GSM8K and 9.8% on MATH.Source: Figure 3 · p. 7, §4.4 · p. 8

4.5 Analysis

  • At MATH Level 5, the hardest level, augmentation raised performance from 35.5 to 41.9, an 18.7% improvement.Decomposition, intermediate-process display, and self-correction also showed their most significant changes on Level 5 questions.Source: §4.5 · p. 8, Table 4 · p. 8
  • Automatic multi-turn QA contains 1,934 instances versus 10,150 for rule-based QA, yet at equal size its performance is worse; combining both types performs better than either alone.Rule-based multi-turn data requires minimal GPT-4 annotation and avoids annotation failures.Source: §4.5 · pp. 8–9, Table 5 · p. 8
  • Self-annotation decreases GSM8K performance but increases MATH performance, while GPT-4 annotations continue to yield substantial improvements.The authors relate self-annotation's effectiveness to the model's current performance on each dataset.Source: Table 6 · p. 8, §4.5 · p. 9

5 Conclusion

DotaMath combines decomposition, code assistance, intermediate-process display, and self-correction, trained using a 574K-pair dataset. It performs best or near-best among open-source models across six mathematical benchmarks and improves on competition-level MATH; its design also supports program-result simulation without tools.

5 Conclusion

  • Across two in-domain and four out-of-domain mathematical benchmarks, DotaMath achieves the best or near-best performance among open-source models and significantly improves performance on the competition-level MATH dataset.Source: §5 · p. 9
  • DotaMath combines decomposition of thought, code assistance, intermediate-process display, and self-correction to address complex mathematical problems.The models are fine-tuned on DotaMathQA, an instruction dataset with 574K query-response pairs.Source: §5 · p. 9
  • DotaMath’s design improves its ability to simulate program results, supporting strong performance even without invoking tools.The conclusion reports that the designed modules provide greater assistance with difficult MATH problems.Source: §5 · p. 9

A Implementation Details

The implementation fine-tunes four 7B–8B-class backbone models for three epochs with shared training settings, while using model-specific context-length exceptions.

A Implementation Details

  • Training uses a 5e-5 learning rate, global batch size 256, cosine scheduler with 3% warm-up, and three epochs.The maximum context length is 4,096 tokens except for LLaMA2, which uses 2,048.Source: A Implementation Details · p. 13
  • Responses use greedy decoding, a maximum sequence length of 4,096 tokens except 2,048 for LLaMA2, and at most three tool uses.Checkpoints are not selected with early stopping.Source: A Implementation Details · p. 13
  • The fine-tuned backbones are LLaMA-2-7B, LLaMA-3-7B, Llemma-7B, and DeepSeekMATH-Base-7B, trained on 32 NVIDIA A100 GPUs.Source: A Implementation Details · p. 13

B.1 Analysis on Simulating Code Execution

DotaMath can infer interpreter-like results without tool calls, though its tool-free accuracy loss is substantially greater on MATH than on GSM8K.

B.1 Analysis on Simulating Code Execution

  • Without tools, DotaMath-deepseek loses 8.5% accuracy on GSM8K, compared with a 41.0% loss on MATH.On GSM8K, ToRA and ToRA-code drop by over 80%, and DeepSeekMATH-Instruct loses 52.3%.Source: §B.1 · p. 13
  • DotaMath’s output mechanism prints problem subtasks in a chain-of-thought manner and enables the model to infer results typically produced by the Python interpreter.The authors attribute the larger tool-free difficulty on MATH in part to frequent use of Python libraries such as sympy2 for complex calculations.Source: §B.1 · p. 13
  • The authors note that the tool-free MATH decline highlights the necessity of tools for solving complex mathematical problems.Source: §B.1 · p. 13

B.2 The Effectiveness of Data Augmentation and Format in MATH Sub-topics

In MATH subtopics, augmentation, decomposition, intermediate-process display, and self-correction have differing effects across subjects, with the largest reported format-related impact on Pre-calculus.

B.2 The Effectiveness of Data Augmentation and Format in MATH Sub-topics

  • Data augmentation improves Counting & Prob., Geometry, and Int. Algebra by over 16%, while Algebra sees the smallest gain at 3.4%.Without augmented data, Algebra already reaches 78.4.Source: §B.2 · p. 13
  • Removing decomposition affects all reported subtopics except Prealgebra.Source: §B.2 · p. 13
  • Removing intermediate-process display and self-correction has the greatest impact on Pre-calculus.The passage relates this pattern to harder problems being more prone to errors and therefore more in need of self-correction.Source: §B.2 · p. 13

B.3 The Effectiveness of Bug Filter

Without the Python bug filter, DotaMath-deepseek’s accuracy is lower on both GSM8K and MATH.

B.3 The Effectiveness of Bug Filter

  • 86.7 to 85.5 on GSM8K and 64.8 to 63.6 on MATH when the Python bug filter is removed.The passage reports the filtered and unfiltered accuracy values for each benchmark.Source: §B.3 · p. 13

C Additional results

The additional results section identifies benchmark-comparison tables, prompt templates, and single-turn successful cases. The examples show DotaMath breaking problems into subtasks and producing final answers.

C Additional results

  • Four additional tables cover benchmark comparisons, performance without the Python interpreter, MATH subtopics, and data filtering.Table 10 compares DotaMath with open- and proprietary-source LLMs on in-domain and out-of-domain benchmarks.Source: Table 10 · p. 14, Table 7 · p. 14, Table 8 · p. 14

D Prompt Templates

  • The prompt frames math solutions in PLAN-CODE format, integrating step-by-step reasoning and Python code; it directs solvers to break problems into subtasks, print every subtask’s results, and put the numeric answer last.Source: D Prompt Templates · p. 17

E.1 Singl-turn Successful cases

  • In the egg-sales example, the model calculates $18 in daily earnings from the eggs Janet sells.The output lists 9 eggs available for sale and $18 in daily earnings.Source: §E.1 · pp. 19–20
  • For the quadratic example, the model returns 12, 10, and 6 as the positive integer values of c that yield real and rational roots.The solution uses the condition that the discriminant is a perfect square.Source: §E.1 · p. 20

E.2 Multi-turn Successful cases

The examples use symbolic equations and code to solve decomposed math subtasks, then revise solutions when an error or omitted factor is identified. The corrected lattice-point example finds four solutions, while the states example totals 79.

Multi-turn Successful cases

  • In the states example, an unassigned India variable causes a runtime error; the correction defines it as a symbol before forming the relationship.The revised code sets states_india as a symbol and uses it in the equation.Source: §E.2 · pp. 21–22
  • 79 is the total when the USA has 50 states and India has 4 more than half as many.The worked example calculates 29 for India and adds the two counts.Source: §E.2 · p. 22
  • The initial lattice-point count of 2 misses solutions from negative factor pairs, so the corrected approach includes those pairs and returns 4 points.The correction enumerates factor pairs and solves a system for each pair.Source: §E.2 · pp. 23–24
  • 4 lattice points satisfy x^2 − y^2 = 17 after solving systems for all positive and negative factor pairs.The solutions are (9, −8), (−9, 8), (9, 8), and (−9, −8).Source: §E.2 · p. 24
Loading 2407.04078v3…