MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline

1 Introduction

• The data-generation approach adds human verification to correct easily rectifiable errors, manually fixes errors in the original GSM8K training set, and uses self-training and knowledge distillation for MATH.The authors describe MATH as more challenging and use additional sampling to select correct solutions.Source: §1 · p. 2
• The authors propose a tentative, easily replicable math-LLM fine-tuning protocol, beginning from Llemma [4] and applying supervised fine-tuning to an annotated dataset.They describe the supervised-fine-tuned model as a baseline.Source: §1 · pp. 2–3
• An outcome-supervised model trained on correct and incorrect solutions is proposed as an auxiliary way to compare and select candidate solutions.The authors say LoRA fine-tuning lets this model retain solution-generation capability while conserving computational resources.Source: §1 · p. 3
• The proposed format combines text analyses with code snippets to support logical reasoning and precise computation.The authors also report that both their SFT and Value LLMs can generate solutions, with the Value LLM serving as an outcome value model.Source: §1 · p. 3
• The authors report significant performance enhancement on mathematical reasoning tasks and compare their approach with PoT-style methods in Figure 1.The introduction describes PoT and PAL as code-centric approaches with little accompanying textual analysis.Source: §1 · pp. 1, 3, Figure 1 · p. 2

2 Dataset

• 98.3% of GSM8K training questions receive at least one correct solution after up to five snippets from GPT-3.5 and GPT-4, followed by two additional GPT-4 attempts for initially unsolved questions.The initial attempts use temperature 0; the follow-up attempts use temperature 0.6.Source: §2 · p. 3
• The authors manually review roughly 100 GSM8K answer discrepancies and correct inaccuracies in either the generated or original answers.They state that this leaves every GSM8K training question with at least one correct solution in the required format.Source: §2 · p. 3
• Temperature 0 does not guarantee deterministic outputs from gpt-3.5-turbo-0613 or gpt-4-0613.The authors note this qualification despite expecting temperature 0 to result in greedy decoding.Source: §2 · p. 3
• 71.8% of MATH training questions receive at least one correct solution from up to eight code-snippet attempts using GPT-4, while human review corrects easily rectifiable errors.The authors forgo manual solutions for 1,208 remaining questions because of the specialized expertise and budget required.Source: §2 · p. 4
• 93.8% MATH training-question coverage is reached by sampling up to 100 teacher-model solutions per unsolved question until at most four correct solutions are identified.The 26.9K unified dataset combines MATH with the GSM8K dataset; the teacher is fine-tuned on the 26.9K collection.Source: §2 · p. 4
• The authors add about 240K MetaMath questions transformed from GSM8K and MATH problems, then sample positive and negative solutions for training.They anticipate this augmentation will boost performance and provide training signals for the outcome value model.Source: §2 · p. 4
• The HTML-like format uses paragraph tags for text analyses and code tags for snippets, and its initial-iteration training loss is about 20% lower on average than the REACT-template format.The authors hypothesize that this format may bridge the pre-training and fine-tuning corpora.Source: §2 · p. 4

3 Fine-Tuning

• Llemma [4] is chosen because it extends Llama-2 [31] pre-training into mathematics and coding, matching the paper’s requirements.The authors also cite evidence that neither model excessively overfits GSM8K or MATH [35].Source: §3 · p. 4

3.1 Supervised Fine-Tuning with Full Parameters

• Full-parameter supervised fine-tuning minimizes cross-entropy on correct solutions while masking question tokens.The stage uses the specially curated dataset.Source: §3.1 · p. 5

3.2 Multi-Task Fine-Tuning with LoRA

• LoRA multi-task fine-tuning adds a sigmoid binary classifier to the token-generation model and trains with roughly equal positive and negative examples.The generation loss applies to correct solutions, while classification loss is calculated for tokens in both correct and incorrect solutions; labels follow solution correctness.Source: §3.2 · p. 5

3.3 Inference

• The Value LLM retains solution-generation ability while evaluating outcomes, so one LLM can serve both roles during inference.For K=20, the outlier-free selection rule excludes answers with frequency at or below δK=1, then chooses the eligible answer with the highest predicted outcome value; if all answers are unique, it selects the highest-value solution.Source: §3.2 · p. 5, §3.3 · pp. 5–6

4.1 Dataset Recap

• The positive SFT seed corpus contains 26.9K GPT- and human-annotated solutions; MetaMath augmentation adds about 55K sampled question-correct solution pairs from 240K new questions.The full collection contains 300K positive and negative examples with approximately balanced labels.Source: §4.1 · p. 6

4.2 Implementation Details

• The experiments fine-tune Llemma models on the curated corpus and cap inference at five code snippets for GSM8K and eight for MATH and out-of-domain datasets.The reported training setup generally uses a 5e-5 learning rate, reduced to 4e-5 for the 7B and 34B models.Source: §4.2 · pp. 6, 8

4.3 Main Results

• MARIO-34B trails MathCoder and ToRA on grade-school math but outperforms others on MATH and challenging out-of-domain problems, likely through code-centric text analysis.The authors report a similar pattern for their 7B model trained on an 82K dataset.Source: Table 2 · p. 7, §4.3 · p. 8
• Outlier-free OVM selection significantly improves results over majority voting; OVM generates solutions comparably to the SFT model but is slightly less effective on out-of-domain datasets.The authors attribute the out-of-domain weakness to continued fine-tuning on MetaMath questions originating from GSM8K and MATH.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 Ablation Study

• Incrementally adding data yields more pronounced MATH advances; the authors cite greater multi-step solution extraction and dataset-specific differences in GPT synthesis coverage and answer filtering.GPT synthesis covers 98.3% of GSM8K questions versus 71.8% of MATH questions.Source: §4.4 · pp. 8–9, Table 4 · p. 9
• With DeepSeek-MATH-7B, the hybrid format is more data-efficient despite using a dataset only 1/30 the size, while a stronger continual-pretraining model enhances math reasoning.The ablation varies the foundational math model and SFT data format.Source: Table 5 · p. 9, §4.4 · p. 9

4.5 Why GSM-Hard is not a good testset for MATH LLM?

• GSM-Hard’s number substitutions can produce physically implausible problems, and the authors advise against using it unless those errors are fixed.Their text-analysis approach checks whether code results fit real-world constraints, which can lower its measured accuracy on this dataset.Source: Table 6 · p. 9, §4.5 · p. 9

4.6 Why re-formatting to HTML?

• HTML-formatted data yields 20% lower fine-tuning loss than original REACT data, but only a marginal 1% MATH test-set improvement after three SFT epochs.The authors hypothesize that hyperparameter tuning could improve HTML-format performance and leave that investigation for future work.Source: Figure 4 · p. 10, §4.6 · p. 10

5 Related Works

• Research on mathematical reasoning explores calculators, code interpreters, and methods combining tool use with textual reasoning to address exact-calculation and symbolic-manipulation challenges.The cited tool and reasoning works are [8] [28], [12], and [43], respectively.Source: §5 · p. 10
• MARIO’s MATH corpus construction uses knowledge distillation, with more samples from a 34B SFT LLM.The passage describes teacher-model samples as a way to transfer knowledge to student models for fine-tuning [11, 17].Source: §5 · p. 10
• MARIO applies multi-task fine-tuning similar to a simple outcome-supervision model, against a backdrop of work on self-verification and trained verifiers.Prior approaches include prompt-based self-verification [36] and verifier intervention during decoding [8] [19] [40].Source: §5 · p. 10

6 Conclusion

• MARIO introduces a reproducible pipeline for constructing a math-specific dataset and fine-tuning an LLM with a Python code interpreter.The paper also describes its three-stage fine-tuning process and makes model checkpoints available as an open-source resource.Source: §6 · p. 10
• The authors report that integrating text analyses with code snippets enhances common-sense reasoning and precise computation in mathematical reasoning tasks.The conclusion also attributes improved model performance to incorporating a verifier model.Source: §6 · pp. 10–11
• The authors report a new state-of-the-art benchmark for roughly 7B-parameter LLMs on MATH and notable generalization to challenging out-of-domain math datasets.The fine-tuning method incorporates a verifier model requiring only a negligible number of additional parameters.Source: §6 · pp. 10–11

7.1.1 Reproducing Data Generation

• A provided command demonstrates reproducing data generation with gpt_react.py on the MATH dataset using GPT-4.The example question asks for the roots of x^4 + 4 = 0.Source: §7.1.1 · p. 16

7.1.2 Fine-tuning

• Training is performed mostly on the LLaMA-Factory code base.Source: §7.1.2 · p. 16

7.1.3 Inference

• The repository shows single-question and batch inference commands; batch question files use JSONL with a question key on each line.Batch inference is shown with vLLM.Source: §7.1.3 · p. 16

7.1.4 Evaluation Toolkit

• The evaluation toolkit takes a JSONL question file whose lines include pred and answer keys for predictions and ground truth.The section provides an eval.py command for running evaluation.Source: §7.1.4 · p. 16

7.1.5 Prompts

• The REACT prompt structures solutions as repeated Thought, Action, Action Input, and Observation steps, followed by a concise final answer.It directs the model to execute Python when a step requires accurate calculation, including symbolic simplification or solving equations.Source: §7.1.5 · pp. 16–17

7.2 Human Review for MATH

• Reviewers accept equivalent generated and dataset expressions even when Python-oriented output uses SymPy notation rather than LaTeX.The passage also gives equivalent matrix representations as an example.Source: §7.2 · p. 17
• These answer-format differences motivate the authors’ math evaluation toolkit.Source: §7.2 · p. 17

• Figure 5 combines text analyses and code snippets in a restructured display resembling HTML from the original REACT format.Source: Figure 5 · p. 18

7.4.1 Incorrect or unreasonable answer

• PoT omits 780786 and ToRA ignores that Pokemon counts must be integers; the model detects its fractional solution and returns 'None'.Source: §7.4.1 · p. 18

7.4.2 Incorrect or unreasonable intermediate step

• PoT produces an infeasible negative cows count, while the model detects that animal counts cannot be negative and returns 'None' after bounded self-verification.ToRA reaches the same answer as GSM-Hard despite generating an incorrect program output.Source: §7.4.2 · p. 20

7.4.3 Ambiguous question

• The model computes 377712.375 hours but rounds up to 377713 because the question's phrasing is ambiguous and asks how many hours are needed.Source: §7.4.3 · pp. 21–22