Source-linked AI summary
Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, Hongsheng Li
TL;DR
The paper examines how code generation, execution, and self-debugging affect GPT4-Code’s mathematical reasoning, addressing the need for systematic analysis and stronger verification. It introduces explicit code-based self-verification (CSV), which verifies answers and revises flawed solutions, and reports 84.32% accuracy on MATH.
Problem
The paper asks whether GPT4-Code can use code generation and self-debugging to automatically verify and correct solutions without assistance from other models or users.
Method
CSV prompts GPT4-Code to generate additional code that verifies answers and adjusts reasoning when flaws are detected, while verification states support weighted majority voting.
Results
84.32% accuracy was achieved on the MATH dataset, with the framework evaluated on MATH, GSM8K, and MMLU-Math.
Takeaways & Limitations
Code-based verification can let GPT4-Code revise flawed solutions and use verification states to improve majority voting.
Takeaways & Limitations
The analysis and improvements focus on GPT4-Code, and applying the methods to other LLMs remains future work.
Abstract
from arXiv · showhide
Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has brought significant advancements in addressing math reasoning problems. In particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter, shows remarkable performance on challenging math datasets. In this paper, we explore the effect of code on enhancing LLMs' reasoning capability by introducing different constraints on the \textit{Code Usage Frequency} of GPT-4 Code Interpreter. We found that its success can be largely attributed to its powerful skills in generating and executing code, evaluating the output of code execution, and rectifying its solution when receiving unreasonable outputs. Based on this insight, we propose a novel and effective prompting method, explicit \uline{c}ode-based \uline{s}elf-\uline{v}erification~(CSV), to further boost the mathematical reasoning potential of GPT-4 Code Interpreter. This method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to use code to self-verify its answers. In instances where the verification state registers as ``False'', the model shall automatically amend its solution, analogous to our approach of rectifying errors during a mathematics examination. Furthermore, we recognize that the states of the verification result indicate the confidence of a solution, which can improve the effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we achieve an impressive zero-shot accuracy on MATH dataset \textbf{(53.9\% $\to$ 84.3\%)}.
1 INTRODUCTION
The paper analyzes how GPT4-Code’s code generation, execution, and self-debugging support mathematical problem solving, then introduces explicit code-based self-verification (CSV) to verify and revise solutions. CSV improves performance across mathematical benchmarks, including 84.32% accuracy on MATH.
- Motivation: LLMs remain unreliable on mathematical reasoning, especially complex calculations, despite progress in general language and code-generation tasks.The paper situates its work against inaccurate or nonsensical outputs and limitations in complex calculation.
- Motivation: GPT4-Code’s initial zero-shot accuracy on MATH was 69.7%, compared with GPT-4’s 42.2%.This represents a reported improvement of 27.5% over GPT-4.
- Motivation: GPT4-Code’s performance reflects incremental code generation, execution, and solution refinement based on execution feedback.The paper calls this adaptive process self-debugging and analyzes it through Code Usage Frequency.
- Research gap: Self-debugging checks generated code steps but does not verify the reasoning process and final answer, motivating additional verification.The paper frames final-answer and reasoning verification as important to mathematical problem solving.
- Method: CSV prompts GPT4-Code to generate code that verifies answers and revises reasoning when verification detects a flaw.Unlike approaches using external verifiers, CSV uses GPT4-Code’s own code-generation and execution capabilities.
- Results: 84.32% accuracy on MATH was achieved by the proposed framework across evaluations on MATH, GSM8K, and MMLU-Math.The paper reports that this substantially outperformed base GPT4-Code and previous state-of-the-art methods.
2 RELATED WORK
Prior work improves mathematical reasoning through intermediate reasoning traces, code-assisted calculation, and verification strategies. The paper builds on these lines by regulating GPT4-Code’s code use and extending self-verification within the model.
- Chain-of-Thought Reasoning: Chain-of-Thought prompting elicits intermediate reasoning steps, while later work improves self-consistency through majority voting and expanded reasoning structures.Examples include few-shot reasoning, problem decomposition, and Tree-of-Thoughts.
- Connection to this paper: The paper regulates GPT4-Code’s code use with step-by-step prompts while combining code generation with self-verification.This connects CoT-style prompting with code-based mathematical reasoning.
- Solving Math Problems with Code: Code-assisted approaches address LLM weaknesses in arithmetic calculations by delegating or supporting computation with executable code.The related work discusses datasets and methods that use calculators or code for mathematical operations.
- Self-Verification: Self-verification research uses external verifiers, intermediate-step checking, multiple-answer ranking, and iterative refinement to assess solution correctness.The paper positions its approach within this broader movement toward verification and refinement.
3 METHOD
The method analyzes how code-use frequency relates to GPT4-Code’s mathematical accuracy, then introduces CSV to verify answers and revise solutions when verification fails. Verification states are also incorporated into weighted majority voting.
- Pilot experiments: 69.7% accuracy on MATH surpassed the previous state-of-the-art performance of 53.9% in the pilot experiment.The study uses GPT4-Code to investigate the code mechanisms underlying its strong mathematical performance.
- Pilot experiments: Code-constrained prompts compare no code, one code use, and unrestricted code use to examine how code affects solving.The basic prompt permits unrestricted code usage, while Prompt 1 prohibits code and Prompt 2 allows one code block.
- Pilot experiments: Higher Code Usage Frequency was positively correlated with accuracy, especially for more complicated MATH problems.The Basic Prompt used code multiple times and achieved notably enhanced accuracy; brief, frequent code segments were associated with improved performance.
- Pilot experiments: 6.9 percent accuracy gain over Prompt 1 was achieved by Prompt 2, suggesting symbolic Python chains improve computational capability over natural-language chains.Prompt 1 produced almost negligible code usage, whereas Prompt 2 produced approximately one code use.
- Explicit code-based self-verification: CSV adds an explicit code-based verification stage whose result is True, False, or Uncertain, with False prompting GPT4-Code to revise its solution.The method uses a zero-shot prompt and can autonomously verify and rectify solutions without external models or users.
- Explicit code-based self-verification: CSV accuracy consistently surpassed the Basic Prompt across all designated difficulty levels while Code Usage Frequency increased.The comparison was conducted with GPT4-Code on MATH, and the figure reports results across five difficulty levels.
- Verification-guided weighted majority voting: Verification-guided weighted majority voting samples k paths, associates each answer with a verification state, and weights candidate answers by confidence.True, Uncertain, and False states receive corresponding weights, with verified-true answers typically weighted most heavily and verified-false answers least.
4 EXPERIMENTS
Experiments evaluate code-based self-verification and verification-guided weighted voting across MATH and other reasoning datasets. Results show substantial gains over GPT4-Code, upward accuracy trends with code usage, and robustness to suitable voting weights, while improvements vary across subjects.
- 4.1 PERFORMANCE ON MATH: 84.32% accuracy on MATH follows 69.69% for GPT4-Code, 73.54% with CSV alone, and 84.32% with CSV plus verification-guided weighted voting.The final result uses 16 sampled paths.
- 4.1 PERFORMANCE ON MATH: Verification-guided weighted voting assigns different weights to True, Uncertain, and False verification states before voting over candidate answers.The framework generates multiple solutions, classifies their verification states, and uses the classified results to score possible answers.
- 4.1 PERFORMANCE ON MATH: Subject-level gains from code-based self-verification range from 7.6% to 0.6%, with Geometry improving by only 0.6%.The paper attributes the limited Geometry improvement to the multimodal requirements of geometry problems, which are beyond its scope.
- 4.2 PERFORMANCE ON OTHER DATASETS: The method achieves state-of-the-art results across GSM8K, MMLU-Math, and MMLU-STEM when integrated with GPT4-Code.The paper also reports comparisons with previous methods and open-source models.
- 4.2 PERFORMANCE ON OTHER DATASETS: Accuracy rises as Code Usage Frequency increases across prompts, difficulty levels, and subjects, with more frequent code use especially associated with harder math problems.Figure 5 evaluates prompt variants across levels and subjects.
- 4.4 ABLATION STUDY AND DISCUSSION: Code-based verification improves accuracy across all seven subtopics, while natural-language self-verification performs slightly below the Basic Prompt.The comparison isolates the contribution of code in the self-verification stage.
- 4.4 ABLATION STUDY AND DISCUSSION: When wT > wU ≥ wF, weighted voting consistently surpasses naive majority voting, whereas wT = 0.5, wU = 0.5, wF = 1 performs worse.The ablation describes the method as easy to tune and robust under the favorable ordering.
5 CONCLUSION AND LIMITATION
The paper attributes GPT4-Code’s mathematical performance to code generation, execution, and solution correction, then introduces self-verification and weighted voting to enhance it. Its scope remains restricted to GPT4-Code, with broader model applications left for future work.
- GPT4-Code’s mathematical performance is largely attributed to generating and executing code and rectifying solutions after implausible outputs.
- The paper introduces explicit code-based self-verification and verification-guided weighted majority voting to enhance GPT4-Code’s mathematical capabilities.
- The analysis and proposed improvements currently focus on GPT4-Code, limiting the demonstrated scope of the work.
- Applying the methods to other LLMs and using them to create code-validated datasets for open-source models remain future work.
A.1.1 CONFUSION MATRIX
The verification process is evaluated as binary classification using a 2x2 confusion matrix, with precision and recall connecting verification outcomes to actual answer correctness. Verification-guided voting is most effective when verification precision is high.
- The verification process uses a 2x2 confusion matrix whose rows represent predicted verification classes and columns represent actual answer correctness.
- True positives are verified TRUE answers that are actually correct, while true negatives are verified FALSE answers that are actually wrong.
- False positives are incorrect answers verified TRUE, while false negatives are correct answers verified FALSE.
- Precision measures the proportion of verified TRUE answers that are correct, whereas recall measures the proportion of actual correct answers verified TRUE.
- Verification-guided voting is bounded to be effective when verification precision is high.
A.1.2 PYTHON PACKAGE USAGE ANALYSIS
The experiments use multiple Python packages, with SymPy reported as the most frequently used package on the MATH dataset.
- SymPy is the most frequently used Python package in the MATH dataset experiments.
A.2 DETAILED EXPERIMENT RESULT ON MMLU DATASET
The appendix reports GPT4-Code’s MMLU performance across subsubcategory, category, and subcategory levels, while illustrating code-based solution steps for a repeating-decimal problem. Some domains show notably weak performance.
- GPT4-Code performs relatively poorly in engineering and humanities domains, with virology scoring below 60%.
- Figures 7 and 8 organize MMLU performance by subsubcategory, category, and subcategory.
- For 0.54 as a fraction, the example treats 0.54 as a repeating decimal and sets up equations before executing Python code.
- The example defines equations in Python using a long decimal approximation of the repeating value.
RESULT
The model corrects a failed equation-subtraction attempt by separately subtracting both sides, obtaining 99x = 54 and converting the resulting decimal solution to 6/11. The examples also show self-debugging after execution errors, solution adjustment from execution results, and mixed outcomes under code-use restrictions and verification states.
- A TypeError prompts the model to replace direct equation subtraction with separate subtraction of the left- and right-hand sides.The correction explicitly identifies the failed operation and introduces a new equation.
- The corrected subtraction produces the equation 99x = 54.
- Solving the equation gives approximately 0.545454545454545, which the model converts to the fraction 6/11.The decimal-to-fraction conversion uses symbolic simplification.
- GPT4-Code self-debugs when code execution emits an error and adjusts its solution based on the execution result.
- The bus-stop and least-common-multiple prompts provide additional math-problem examples, while two tables report success for True and failure for Uncertain and False.
- Examples report that unrestricted code use can succeed while using no code or code only once fails.This pattern appears in two example tables.