Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks

• Supervised fine-tuning alone enables near-perfect zero-shot large-number addition and subtraction, which the authors attribute mainly to LLaMA’s consistent number tokenization.The introduction says comparable performance is almost impossible with prior models including Bloom, OPT, and GPT-NeoX.Source: §1 · p. 2
• For difficult large-number multiplication and division, Goat decomposes unlearnable tasks into learnable steps using basic arithmetic principles and interpretable intermediate supervision.The model generates the proposed chain of thought before its final answer.Source: §1 · p. 2
• Experiments examine the decomposition in a synthetic setting and suggest Goat learns patterns that generalize to unseen data rather than merely memorizing computations.The setup is intended to study the effects of chain of thought and intermediate supervision while mitigating hard-to-control natural-language factors.Source: §1 · p. 2
• Goat-7B can be trained with LoRA on a 24GB VRAM GPU, and the authors release the model, dataset, and dataset-generation script.The pipeline is also described as potentially integrable with existing instruction-tuned models for math word problems.Source: §1 · p. 2
• Goat achieves state-of-the-art performance on positive-integer arithmetic, with potential to exceed GPT-4 and large-margin gains over its long multiplication and division methods.The reported GPT-4 comparison is specifically for long multiplication and long division.Source: §1 · p. 2

2.1 Instruction Tuning

• Instruction tuning aligns pretrained language models with human instructions; this study generates arithmetic instruction input-output pairs using a simple Python program.Source: §2.1 · pp. 2–3

2.2 Arithmetic Reasoning

• Language models can solve math word problems with chain of thought but may still make arithmetic errors despite correct reasoning, making accurate computation a distinct challenge.Source: §2.2 · p. 3

2.3 Arithmetic Computation

• Prior approaches use calculators or executable programs for computation; Goat instead targets in-model arithmetic, using fine-tuning for addition and subtraction while large multiplication and division remain challenging.The section notes that earlier chain-of-thought methods are unnecessary for some operations but challenging tasks have been difficult or inefficient for prior studies.Source: §2.3 · p. 3

3.1 Language Model

• LLaMA represents each digit as an individual token, and the authors attribute its arithmetic ability to this consistent number tokenization.They report that Bloom, OPT, GPT-NeoX, and Pythia fine-tuned on the same dataset do not match LLaMA’s arithmetic ability.Source: §3.1 · p. 3

3.2 Learnability of Arithmetic Tasks

• Learnability means generating sufficiently accurate direct answers within a predefined training period; the authors classify tasks experimentally and identify multi-digit multiplication and division as mainly unlearnable.They report that 2-digit-by-2-digit multiplication reaches around 90% accuracy after nearly 10 epochs by direct-answer training, versus comparable accuracy in 1 epoch with their chain-of-thought.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Addition and Subtraction

• Addition and subtraction are learnable without chain-of-thought, with near-perfect accuracy on an unseen test set despite limited training data.The paper contrasts this with earlier studies that used chain-of-thought for these operations.Source: §3.3 · pp. 4–5

3.4 Multiplication

• For multi-digit multiplication, the proposed chain-of-thought breaks the problem into extraction, splitting, expansion, one-digit products, and term-by-term addition.The example orders the larger number first to reduce the number of summation terms and thus chain-of-thought length.Source: §3.4 · p. 5

3.5 Division

• Multi-digit division is treated as unlearnable and decomposed with a modified slow-division recurrence that subtracts divisor multiples until the remainder is smaller than the divisor.The steps use one-digit multiplication, comparison, copying, subtraction, and quotient assembly.Source: §3.5 · p. 5

3.6 Settings

• The settings cover addition and subtraction of positive integers up to 16 digits, multiplication with products under 12 digits, and division with quotients under 6 digits and dividends under 12 digits.For non-divisible cases, the target is the least positive remainder; the authors report limited extrapolation beyond the training distribution, with larger-number use potentially increasing training cost.Source: §3.6 · pp. 5–6

3.7 Dataset

• The synthetic dataset contains around 1 million question-answer pairs with randomly generated numbers sampled from log space.Answers include the proposed chain-of-thought and final numerical output; small numbers may recur.Source: §3.7 · p. 6

3.8 Fine-tuning

• Fine-tuning uses randomly selected instruction templates and varied number-expression formatting; LoRA training for 8-digit addition with 100K instances takes about 1.5 hours on an A10 GPU.The authors report that Goat-7B can be fine-tuned using LoRA on a 24GB VRAM GPU.Source: §3.8 · p. 6

4.1 Metric

• Digit match supplements exact string match by reflecting per-token error rate when incorrect answers still share most digits with the target.The authors note that GPT-4’s exact-string-match accuracy is almost zero on large-number tasks.Source: §4.1 · p. 6

4 Experiments

• In zero-shot evaluation, Goat-7B achieves comparable or higher accuracy on BIG-bench arithmetic than few-shot PaLM-540B.Table 3 reports GPT-4 and Goat-7B results on BIG-bench and selected arithmetic tasks, in percentages.Source: §4 · p. 6, Table 3 · p. 7

4.2 Comparison

• Adding “Solve it step by step” gives GPT-4 only marginal accuracy improvement on multiplication and division, with common errors in digit alignment, copying, and one-digit multiplication.The authors report that some intermediate steps are incorrect even when the final answer is correct.Source: §4.2 · p. 7
• GPT-4 performs reasonably on 8D+8D and 16D+16D but fails on most 16D+8D tasks; the authors suggest inconsistent number tokenization as one possible factor.They state that this inconsistency may make aligning corresponding digits difficult.Source: §4.2 · p. 7

5.1 Ablation study

• Exact string-match accuracy without CoT stayed at zero for 4D × 4D multiplication and 6D ÷ 3D division.For multiplication, adding term by term was crucial, while split and expansion had minimal impact and were retained for interpretability.Source: Figure 2 · p. 7, Figure 3 · p. 8, §5.1 · p. 8

5.2 Extrapolation

• Addition accuracy gradually dropped as test numbers moved farther beyond the training distribution, which extended only to 16D+16D.The authors identify limited extrapolation as a limitation and note that training on more digits may increase training cost.Source: Figure 4 · p. 8, §5.2 · p. 8

5.3 Comparison with Other LLMs

• Bloom, OPT, GPT-J, GPT-NeoX, and Pythia struggled with arithmetic under the same dataset and training hyperparameters used for Goat.Even on multi-digit addition, which was learnable for LLaMA, these models had significantly higher fine-tuning loss; the authors connect this pattern to number tokenization.Source: §5.3 · p. 8

5.4 Few-shot Prompting with GPT-4

• GPT-4 generated correct answers more frequently with the proposed decomposition than with its default long multiplication and division methods.The paper presents this as support for the decomposition approach.Source: §5.4 · p. 8

6 Limitations

• LLMs may produce unreasonable answers when inputs deviate significantly from training data, and the study evaluates only integer operations.The authors also note that their straightforward arithmetic-rule CoT may not be the most efficient design for final answer generation; other algorithms may be more suitable for the model to learn.Source: §6 · pp. 8–9

7 Conclusion

• Goat demonstrates the feasibility of high-accuracy basic arithmetic through supervised fine-tuning alone and achieves state-of-the-art performance on elementary arithmetic tasks with the proposed CoT.The authors describe the work as a platform for studying working memory and intermediate supervision, and say the method may integrate with other instruction-tuned LLMs.Source: §7 · p. 9

A Hyperparameters

• Table 4 lists hyperparameters for fine-tuning LLaMA-7B.The passage does not provide the parameter values.Source: Table 4 · p. 11

B Tokenization

• Inconsistent tokenization can make number representations difficult to learn when tokens represent varying numbers of digits.Nogueira et al. (2021) report zero accuracy for addition of numbers with five or more digits in their evaluated setting.Source: B Tokenization · p. 11
• Nogueira et al. (2021) report that models with inconsistent number tokenization completely fail to learn addition of five or more digits, reaching zero accuracy.Their account attributes the difficulty to irregular mappings between token embeddings and numbers.Source: B Tokenization · p. 11

C Metric

• Exact string match is averaged per task, while digit match accuracy is 1 − CER to assess answers when exact match may be zero.CER is the percentage of incorrectly predicted characters, computed with Python torchmetrics.Source: C Metric · p. 11
• ChatGLM also splits numbers into individual digit tokens, but evaluating its arithmetic abilities is left for future work.Table 5 compares number tokenization across language models.Source: Table 5 · p. 12
• Natural-language fine-tuning templates generated by ChatGPT replace the {arithmetic} placeholder with randomly generated expressions.An example expression is 3425 ∗5823.Source: Table 6 · p. 13

D Simplified Synthetic Environment

• The simplified synthetic environment uses structured prompts and direct completions to study sub-task learnability and proposed chain-of-thought effectiveness.It avoids hard-to-control aspects of natural language.Source: D Simplified Synthetic Environment · p. 13

E Special Cases

• For multi-digit multiplication and division, the method omits chain-of-thought on special cases considered learnable and answers directly.These cases include multiplication when one operand has only one non-zero digit, such as 857483 × 400 = 342993200.Source: E Special Cases · p. 13
• Division is answered directly when the dividend equals the divisor, yielding quotient one, or is smaller, yielding quotient zero and the dividend as remainder.The examples are 358 ÷ 358 = 1 and 423 ÷ 968 = 0 R 423.Source: E Special Cases · p. 13

F Dataset

• N-digit-by-1-digit multiplication and division may require fewer samples to reach the same accuracy as other tasks during task-specific fine-tuning in the simplified environment.The authors note that optimal task proportions are difficult to determine and depend on problem settings.Source: F Dataset · p. 13
• The simplified synthetic environment formats examples with spaces between numbers and operators to reduce natural-language influence.Its operators represent addition, subtraction, multiplication, and division.Source: Table 7 · p. 14
• Figure 5 presents the composition of tasks in the dataset.The caption identifies the figure's scope but does not state the task proportions.Source: Figure 5 · p. 14
• Training loss shows no significant decrease after about one million samples, so the authors recommend a larger dataset and stopping when loss no longer decreases.Convergence also depends on hyperparameters such as batch size and learning rate.Source: F Dataset · p. 14

G Ablation Study

• The multiplication ablation removes one step at a time, but excludes extraction and product because the authors consider them crucial for multi-digit multiplication.The named steps include extraction, split, expansion, product, and adding terms term by term.Source: G Ablation Study · p. 14

Multiplication

• The multiplication and division passages include example questions, but do not state an ablation result or a comparison outcome.The examples ask for 397 × 4429 and 8914 / 64.Source: Multiplication · p. 14, Division · p. 14

Division

• The division ablation removes the middle step that computes the product for all iterations while keeping the other steps unchanged.The passage does not report the ablation's measured outcome.Source: Multiplication · p. 14

H Few-shot Prompting with GPT-4

• The multiplication examples show three-shot GPT-4 prompts that provide question-solution pairs and ask the model to solve a final multiplication problem in the same format.The examples include decomposing a multiplier into place-value terms.Source: H Few-shot Prompting with GPT-4 · p. 15
• The division examples likewise use three-shot GPT-4 prompts with question-solution pairs and solve problems by subtracting multiples of the divisor.One example prompt asks for 440331 / 22.Source: H Few-shot Prompting with GPT-4 · p. 15