MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning

Introduction

• Open-source math reasoning lags closed-source systems: Llama-2-70B scores 56.8% on GSM8K and 13.5% on MATH, versus GPT-4 Code Interpreter’s 97% and 69.7%, respectively.The paper attributes GPT-4 Code Interpreter’s performance to generating and assessing execution results across code interleaved with natural-language reasoning (Zhou et al., 2023a).Source: §1 · pp. 1–2
• MathCodeInstruct contains 80k math problems with solutions interweaving natural-language reasoning, executable code, and execution results.The dataset includes existing GSM8K and MATH problems and newly formulated problems.Source: §1 · pp. 2–3
• The dataset pipeline starts with GPT-4 Code Interpreter-style GSM8K and MATH solutions, then uses problem interpolation to generate intermediate-difficulty questions and MathCoder-Initial solutions.The added problems are intended to bridge grade-school GSM8K and competition-level MATH; the final combined data is used to fine-tune base Llama-2 models.Source: §1 · p. 2
• Special tokens mark text, code, and execution segments, while inference runs detected code in Jupyter Notebooks and feeds its output back into autoregressive generation.The tokens are <|text|>, <|code|>, and <|execution|>.Source: §1 · p. 2
• MathCoder integrates natural-language reasoning, code generation, and execution feedback into open-source pretrained LLMs to improve mathematical reasoning.The paper describes this as the first systematic study to explicitly combine these elements in open-source models.Source: §1 · pp. 2–3
• Fine-tuning Llama-2 and CodeLlama with MathCodeInstruct and the proposed framework improves performance on mathematical reasoning benchmarks and supports out-of-domain Mathematics and SimulEq evaluation.The introduction reports high accuracy on GSM8K and MATH and good performance on the out-of-domain datasets, without giving scores here.Source: §1 · p. 3

2.1 MATHCODEINSTRUCT DATASET

• Seed data uses GPT-4 solutions to GSM8K and MATH training questions, filters for agreement with ground-truth answers, and fine-tunes CodeLlama-34B into MathCoder-Initial.The seed and generated data are combined as MathCodeInstruct, which is used to fine-tune base Llama-2 and CodeLlama into MathCoder-L and MathCoder-CL.Source: §2.1 · pp. 3–5
• LCE solutions interleave natural-language reasoning, Python code for complex computations, and execution outputs that inform subsequent reasoning.The solution sequence repeats language, code, and execution components, with each component influencing the next.Source: §2.1 · p. 4, Figure 3 · p. 4
• 83.2% of interpolated problems are harder than GSM8K, while 95.6% are easier than MATH.Problem interpolation prompts GPT-4 with a simple GSM8K problem and a difficult MATH problem to generate an intermediate-difficulty problem.Source: §2.1 · pp. 4–5, Figure 4 · p. 5
• New problems receive three LCE solutions from MathCoder-Initial, and only problems whose three answers match are retained for self-distillation.MathCoder-Initial scores 77.3% on GSM8K and 44.0% on MATH; using it for distillation reduces costs relative to GPT-4.Source: §2.1 · p. 5

2.2 SUPERVISED FINE-TUNING AND INFERENCE

• Special tokens mark language, code, execution, and block endings; training concatenates instances with cross-question masking and computes cross-entropy loss only on language and code tokens.Execution-result tokens are excluded from the loss because the model does not need to predict them.Source: §2.2 · p. 5
• At inference, code blocks run in a Jupyter Notebook with variables preserved across blocks, and execution outputs feed autoregressive reasoning until the final answer.The process uses special tokens to identify each component and resembles the GPT-4 Code Interpreter workflow.Source: §2.2 · p. 5

3.1 DATASETS AND IMPLEMENTATION DETAILS

• Evaluation covers GSM8K and MATH as fine-tuning in-domain datasets and SVAMP, Mathematics, and SimulEq as out-of-domain tests.The tested bases include Llama-2 and CodeLlama at multiple sizes; baselines receive CoT/few-shot prompting, while MathCoder is evaluated zero-shot without extra prompting.Source: §3.1 · p. 6

3.2 MAIN RESULTS

• CodeLlama-based MathCoder improves accuracy over same-size Llama-2 MathCoder by 4.1% at 7B and 3.0% at 13B.The authors attribute CodeLlama’s coding and reasoning capacity to additional code training, while noting Llama-2-70B can outperform CodeLlama-34B in MathCoder.Source: §3.2 · pp. 6–7
• MathCoder leads open-source competitors across all five datasets, though GPT-4 Code Interpreter retains a substantial gap; L-7B beats WizardMath-70B on three datasets with a +4.5% average-score gain.The comparison also reports MathCoder-L-2-70B at 73.1% versus MathCoder-CL-34B at 70.2%.Source: §3.2 · p. 6, Table 2 · p. 7
• MathCoder scores higher on algebra and prealgebra but struggles with harder geometry problems, suggesting code helps more on computationally intensive questions.Figure 5 indicates CodeLlama-based gains over Llama-2 are concentrated in calculation-heavy subjects such as algebra and number theory.Source: §3.2 · p. 7, Figure 5 · p. 8

3.3 ABLATION STUDY

• Problem interpolation significantly improves CodeLlama-34B performance across all five datasets by adding intermediate-difficulty problems and increasing training-set diversity.The experiment varies the portion of MathCodeInstruct questions created through the proposed interpolation method.Source: §3.3 · pp. 7–8, Table 4 · p. 8
• Model-predicted execution results (#1) outperform interpreter-returned results (#2) across five datasets, with a 34.0% average-accuracy improvement.The authors nevertheless conclude that actual execution can improve complex computations, noting that the model often fails to predict correct execution results; the reported comparison and conclusion point in different directions.Source: §3.3 · p. 8, Table 5 · p. 9
• 70.2% versus 69.1% average accuracy: omitting execution-result loss improves accuracy on four of five datasets by letting the model focus on natural language and code.This comparison is MathCoder-CL-34B trained without execution-result loss versus the model trained to predict execution results.Source: §3.3 · p. 8, Table 5 · p. 9

Instruction Tuning

• Unlike general instruction tuning, MathCoder tunes on model-generated high-quality math solutions; unlike Luo et al. (2023), its approach uses code to solve math problems.Earlier instruction-tuning work focused on general instruction following, while recent studies used synthetic instructions generated by powerful models.Source: §4 · p. 9

Mathematical Reasoning

• MathCoder combines natural language and code, drawing on Chain-of-Thought methods for multistep reasoning and code generation for complex computations.The paper also distinguishes math-related pre-training as another approach to improving general reasoning.Source: §4 · p. 9

Distillation

• MathCoder is framed as self-distillation, echoing prior demonstrations (Li et al., 2023b; Wang et al., 2022a; Allen-Zhu & Li, 2020): MathCoder-Initial solutions fine-tune CodeLlama-34B into MathCoder-CL-34B.MathCoder-Initial is built on CodeLlama-34B.Source: §4 · p. 9

• 83.9% on GSM8K and 45.2% on MATH; MathCoder leads open-source LLMs on five math datasets, exceeds ChatGPT-3.5 and PaLM-2 on GSM8K and MATH, and beats GPT-4 on MATH.Source: §5 · pp. 9–10
• MathCoder uses GSM8K and MATH seed data for GPT-4-generated reasoning, code, and execution problems, creates intermediate-level problems through interpretation, and applies training loss only to natural language and code.Source: §5 · p. 9
• GPT-4-dependent data generation constrains MathCoder to GPT-4’s capabilities and leaves it unable to solve theorem-proving problems; its unimodal design also struggles with complex geometry.Source: §5 · p. 10

A DATASET EXAMPLES

• Two examples illustrate the process of creating MathCodeInstruct.Source: A DATASET EXAMPLES · p. 14
• Figure 6 shows an example with only one LCE block.Source: A DATASET EXAMPLES · p. 14
• Figure 7 shows an example with three LCE blocks.Source: A DATASET EXAMPLES · p. 14

B EXAMPLES OF DIFFICULTY COMPARISON

• 83.2% and 95.6% correspond to the two most common cases: interpolation problems are harder than original GSM8K problems, and an original MATH problem is harder than an interpolation problem.The section presents these as examples of GPT-4 evaluating problem complexity in MathCodeInstruct.Source: B EXAMPLES OF DIFFICULTY COMPARISON · p. 14
• 15.3% of all problems are ties between a newly generated interpolation problem and its original GSM8K problem, according to the example shown.Source: B EXAMPLES OF DIFFICULTY COMPARISON · p. 14
• Less than 3% of all problems show the uncommon case in which GPT-4 judges the original GSM8K problem slightly more difficult than the interpolation problem.Source: B EXAMPLES OF DIFFICULTY COMPARISON · p. 14

C CREATING PROBLEMS USING ONLY GSM8K

• Five examples use GSM8K training problems to generate new problems that are more difficult than their originals.The examples appear in Figures 13–17.Source: C CREATING PROBLEMS USING ONLY GSM8K · p. 14
• The GSM8K-only generated problems resemble raw GSM8K problems and are only slightly more complicated, if not equally difficult.Some examples only change variable names or scale values; they remain much simpler than MATH problems.Source: C CREATING PROBLEMS USING ONLY GSM8K · p. 14
• Introducing MATH problems into interpolation gives GPT-4 a route to generate more challenging problems and can narrow the difficulty gap between GSM8K and MATH.The resulting interpolation problems resemble problems from both datasets.Source: C CREATING PROBLEMS USING ONLY GSM8K · p. 14

D MORE EXPERIMENT RESULTS

• Across MathCoder-L (7B, 13B, 70B) and MathCoder-CL (7B, 13B, 34B), CL’s MATH accuracy gains occur primarily in calculation-precise subjects: algebra, number theory, and counting and probability.Figure 18 compares the models by MATH problem level and subject.Source: D MORE EXPERIMENT RESULTS · p. 14

E CASE STUDY WITH COT, POT AND LCE

• LCE succeeded on the illustrated number-theory and algebra problems, while both CoT and PoT failed.The examples are MATH test problems 1191 and 2477; LCE intertwines natural language, code, and execution results.Source: E CASE STUDY WITH COT, POT AND LCE · p. 14, Figure 19 · p. 28, Figure 20 · p. 29
• On the geometry problem, CoT found coordinates but failed to calculate area, PoT could not interpret the conditions, and LCE used them correctly without calculation errors.This comparison concerns MATH test geometry problem 500.Source: E CASE STUDY WITH COT, POT AND LCE · p. 14, Figure 21 · p. 30, Figure 22 · p. 31