Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

Zheng Yuan; Hongyi Yuan; Chengpeng Li; Guanting Dong; Keming Lu; Chuanqi Tan; Chang Zhou; Jingren Zhou

Source-linked AI summary

Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, Jingren Zhou

arXiv:2308.01825v2Aug 3, 2023cs.CL

TL;DR

Mathematical reasoning remains challenging to scale predictably in LLMs, particularly regarding pre-training quality and supervised or augmented data. The paper analyzes these relationships and introduces rejection sampling fine-tuning, finding that better pre-training and more distinct reasoning paths improve performance, with multi-model samples raising LLaMA-7B to 49.3% on GSM8K versus 35.9% for SFT.

Problem
The paper addresses the under-explored scaling relationship between LLM capacity, pre-training loss, supervised data, augmented data, and mathematical reasoning performance.
Method
The paper analyzes SFT and ICL across pre-training losses and supervised-data amounts, then uses rejection sampling to collect correct, deduplicated reasoning paths for RFT.
Results
49.3% GSM8K accuracy for LLaMA-7B with multi-model rejection samples exceeds its 35.9% SFT accuracy, while distinct reasoning paths and lower pre-training loss relate to better performance.
Takeaways & Limitations
Pre-training a better language model is more important than enlarging SFT or RFT data for already better models, while diverse reasoning paths benefit less performant models more.
Takeaways & Limitations
The paper does not regress scaling laws because many values and the pre-training losses, ICL prompts, and SFT settings are not aligned.

Abstract

from arXiv · show

Mathematical reasoning is a challenging task for large language models (LLMs), while the scaling relationship of it with respect to LLM capacity is under-explored. In this paper, we investigate how the pre-training loss, supervised data amount, and augmented data amount influence the reasoning performances of a supervised LLM. We find that pre-training loss is a better indicator of the model's performance than the model's parameter count. We apply supervised fine-tuning (SFT) with different amounts of supervised data and empirically find a log-linear relation between data amount and model performance, and we find better models improve less with enlarged supervised datasets. To augment more data samples for improving model performances without any human effort, we propose to apply Rejection sampling Fine-Tuning (RFT). RFT uses supervised models to generate and collect correct reasoning paths as augmented fine-tuning datasets. We find with augmented samples containing more distinct reasoning paths, RFT improves mathematical reasoning performance more for LLMs. We also find RFT brings more improvement for less performant LLMs. Furthermore, we combine rejection samples from multiple models which push LLaMA-7B to an accuracy of 49.3\% on GSM8K which outperforms the supervised fine-tuning (SFT) accuracy of 35.9\% significantly.

1 INTRODUCTION

The paper studies how pre-training loss, supervised-data volume, and augmented reasoning paths scale mathematical reasoning in supervised LLMs, focusing on single-inference deployment. It finds predictable performance relationships and shows rejection-sampling augmentation can substantially improve results.

The study focuses on supervised LLM mathematical reasoning with one inference, avoiding the computational expense of prompting and repeated inference.This setting is intended to be closer to online deployment scenarios.
Pre-training loss is approximately negatively linearly correlated with SFT and ICL accuracy within a range, making it a better indicator than model size or token count.
SFT performance improves log-linearly with supervised-data amount, but the benefit diminishes for better pre-trained models.
RFT improves as the number of distinct reasoning paths increases, and combining rejection samples from multiple models further enhances performance.The paper attributes this to more diverse reasoning paths supporting better reasoning generalization.
49.3 accuracy on GSM8K for LLaMA-7B exceeds its SFT accuracy of 35.9% after combining rejection samples from multiple models.

2 RELATED WORKS

Prior work establishes scaling laws for language-model loss and studies mathematical reasoning through prompting, fine-tuning, inference-time aggregation, and rejection-sampling augmentation. These approaches motivate examining how supervised models and data scale for mathematical reasoning.

Research on mathematical reasoning evaluates LLM methods across math word-problem benchmarks with different difficulty levels.
A central strategy is aggregating sampled reasoning paths during fine-tuning or inference to improve mathematical reasoning.
Inference-time methods include verifiers and majority voting over multiple sampled reasoning paths, while rejection sampling augments fine-tuning data.
Scaling-law research seeks predictable relationships between model parameters, data size, computation, and language-model loss.

3 THE FACTORS OF MATH REASONING ABILITY IN SUPERVISED LLM

The paper examines how pre-training loss, supervised-data amount, and augmented-data diversity relate to mathematical reasoning performance in supervised LLMs. It finds distinct scaling patterns: lower pre-training loss predicts stronger performance, supervised-data gains are log-linear but diminish for better models, and RFT benefits from diverse reasoning paths and multi-model aggregation.

Model Accuracy vs. Pre-training Loss: Pre-training loss is approximately negatively linearly correlated with SFT and ICL accuracy within the studied interval, making it a better indicator than parameter count.The comparison spans GPT-3, LLaMA, LLaMA2, and GPT-4, whose architectures, parameter counts, and tokenization differ.
Model Accuracy vs. Pre-training Loss: SFT consistently outperforms ICL, but its improvement diminishes as pre-training loss decreases.The authors caution that the apparent linear relation may only hold within the observed loss interval.
Model Accuracy vs. Supervised Data Count: SFT performance has a log-linear relation with supervised-data amount, while doubling the data yields smaller benefits for better pre-trained models.The relation is reported as stable across 1, 1/2, 1/4, and 1/8 of the GSM8K training set.
Model Accuracy vs. Augmented Data Count: RFT generates correct reasoning paths through rejection sampling and uses distinct equation lists as augmented fine-tuning data.For each question, candidate paths are filtered by answer correctness and Python-based calculation checks before deduplication.
Model Accuracy vs. Augmented Data Count: RFT gains depend more on distinct reasoning-path amount than raw sampling count, and increasing the sampling count produces diminishing improvements.Obtaining additional paths does not introduce new questions, so doubling paths improves less than doubling training samples.
Model Accuracy vs. Augmented Data Count: Aggregating rejection samples from multiple models improves performance across model sizes, with LLaMA-7B reaching 49.3 accuracy versus 35.9 for SFT.The combined dataset reduces performance gaps among same-size models; larger models contribute less unique reasoning diversity, with LLaMA-33B-SFT contributing only 6.5% exclusive paths.

4 DISCUSSION

The paper examines whether diverse reasoning paths and computational scaling can improve mathematical reasoning beyond standard fine-tuning. It finds that diverse augmented paths help, while further gains can require substantially more sampling or pre-training computation.

Diverse reasoning paths: RFT-U13B models generate more questions with larger numbers of unique calculation processes than RFT k=100 and SFT models.SFT models often produce only one calculation process and rarely exceed eight distinct processes per question.
Diverse reasoning paths: Diverse calculation paths in RFT training data equip LLMs to find diverse reasoning logic for solving math problems.
Discussion: Given a fixed amount of human-annotated data, lower pre-training loss and rejection-sampling augmentation are identified as two factors improving mathematical reasoning.
Computational trade-offs: SFT and RFT require negligible cost times relative to pre-training, but increasing distinct reasoning paths requires exponentially more sampling and faces an upper bound.The reported cost times are approximately 1 × 10^-5 for SFT and 1 × 10^-4 for RFT.
Computational trade-offs: RFT, SFT, and ICL follow the performance ordering RFT>SFT>ICL, while their improvement speeds follow RFT<SFT<ICL.The paper presents this ordering as an assumption based on its findings.

5 CONCLUSIONS

The paper studies how pre-training loss, supervised data, and distinct reasoning paths relate to mathematical reasoning performance. It concludes that better pre-training is especially important because stronger models benefit less from SFT and RFT.

5 CONCLUSIONS: Pre-training loss, supervised data amount, and distinct reasoning paths are related to mathematical reasoning performance.
5 CONCLUSIONS: Better language models benefit less from SFT and RFT, making lower pre-training loss the paper’s most important conclusion for improving math reasoning.

7 LIMITATIONS

The paper identifies several important omissions: larger-model RFT, math-focused pre-training, and formal scaling-law regression are left for future work.

RFT is not evaluated on 65B and 70B LLaMA models.
Math-focused pre-training is not studied, although prior work indicates it is useful.The paper also notes that losses from math-related pre-training may not align with losses from general-domain pre-training.
The paper does not regress scaling laws because many values and experimental settings are estimated or misaligned across models.The stated sources of misalignment include pre-training losses, ICL prompts, and SFT settings.

A DETAILED EXPERIMENT SETTING

The experiments fine-tune GSM8K under specified training settings and generate rejection-sampling data from SFT models, selecting diverse reasoning paths for augmentation.

GSM8K is fine-tuned for 3 epochs with batch size 128, peak learning rate 2e-5, and 3% warmup.GPU counts vary by model size, from 8 GPUs for 7B and 13B models to 32 GPUs for 65B and 70B models.
The study compares training-epoch choices for downsampled GSM8K subsets and reports the better test result from the two settings.The compared settings are 3 epochs and a fraction-dependent epoch choice.
An SFT model samples 100 reasoning paths per training example at temperature 0.7 for rejection-sampling augmentation.Equation lists are extracted, whitespace-normalized, and joined into strings for deduplication.
The selection algorithm chooses reasoning paths using Levenshtein distances between extracted equation representations.The objective is to find dissimilar paths because the authors seek diverse reasoning paths for better generalization.

B DETAILED RESULTS OF SFT AND RFT

The detailed-results section presents numerical SFT and RFT results, rejection-sampling cases, and examples of diverse correct reasoning paths that can provide additional supervision.

The paper reports detailed SFT and RFT results in Tables 5 and 6 using maj1@1 and maj1@100, respectively.
Rejection-sampling cases include reasoning paths with different reasoning complexity.The calculations in the examples are highlighted in red.
RFT considers reasoning paths with different calculation processes, including alternative equation forms or operation orders, when they yield correct answers.
Filtered reasoning paths sampled from LLMs are described as similar in quality to human-annotated reasoning demonstrations.The paper suggests rejection sampling can provide additional supervision signals for mathematical reasoning.

D.1 SELF QUERY AUGMENTATION

Self-query augmentation reformulates GSM8K to generate queries from reasoning results, but the resulting augmentation does not improve mathematical reasoning performance in the reported LLaMA 7B experiments.

Some fine-tuned LLM errors are reasonable reasoning chains containing interpretation or arithmetic mistakes.One example misinterprets 150% as 15% while retaining a reasonable calculation structure.
Wrong predictions may become correct under modified queries, motivating query generation for predicted reasoning chains.The paper relates this idea to hindsight experience replay, which changes objectives for failed samples to obtain positive rewards.
A reversed GSM8K task predicts queries from ground-truth reasoning results, and a LLaMA model generates queries for reasoning chains produced by a normally fine-tuned model.The authors evaluate mixtures of original and generated samples, as well as generated samples alone, on LLaMA 7B.
Self-query augmentation produces the worst results, while mixing generated and original data remains below the performance of original data alone.The paper concludes that naive self-query augmentation does not benefit fine-tuned mathematical reasoning performance.

D.2 SELF REVISING AUGMENTATION

The paper evaluates self-revising augmentation for mathematical reasoning by training models on sampled reasoning paths paired with ground-truth paths. Performance depends on selecting lexically diverse paths, while naive sampling can degrade results.

Revising augmentation: Self-revising augmentation samples K reasoning paths, concatenates one with the query, and pairs it with the ground-truth reasoning path for training.The revising dataset uses a sampling temperature of 0.7.
Initial results: 36.09% versus 35.90%: K = 1 yields only a marginal accuracy improvement over vanilla fine-tuning on GSM8K.Increasing K causes performance to degrade under the initial revising procedure.
Initial results: Training samples can suffer distribution discrepancy because training-set paths may be lexically more similar to ground-truth paths than test-set paths.The authors introduce two procedures to alleviate this discrepancy.
Revising augmentation: Selecting the sampled path with the largest Levenshtein distance produces uniformly better performance than the fine-tuning baseline across different K values.This procedure chooses the most lexically distant path relative to the ground-truth path.
Revising augmentation: The results indicate that lexical diversity of reasoning paths matters when constructing revising-training samples.The alternative procedure splits the training set into N folds, fine-tunes on N − 1 folds, and samples paths on the held-out fold.