How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition

Guanting Dong; Hongyi Yuan; Keming Lu; Chengpeng Li; Mingfeng Xue; Dayiheng Liu; Wei Wang; Zheng Yuan; Chang Zhou; Jingren Zhou

Source-linked AI summary

How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition

Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, Jingren Zhou

arXiv:2310.05492v4Oct 9, 2023cs.CLcs.AIcs.LG

TL;DR

The paper addresses limited evidence on how composite SFT data affects multiple LLM abilities. It evaluates data composition across model scales and training strategies, finding distinct scaling patterns, low-resource benefits but high-resource conflicts from mixing, and a proposed DMT strategy for balancing abilities.

Problem
Existing research mostly studies mathematical reasoning, coding, and human alignment separately, leaving composite SFT data composition insufficiently understood.
Method
The study evaluates three abilities across data amounts, composition ratios, LLaMA model scales, and four SFT strategies using benchmarked experiments.
Results
Abilities scale differently; mixed data helps in low-resource settings but conflicts in high-resource settings, while sequential training risks catastrophic forgetting.
Takeaways & Limitations
DMT is presented as a promising strategy for learning multiple abilities with different scaling patterns while alleviating conflicts and forgetting.
Takeaways & Limitations
The study focuses primarily on mathematical reasoning, code generation, and general human-aligned ability, and MT-Bench uses GPT-4 for scoring.

Abstract

from arXiv · show

Large language models (LLMs) with enormous pre-training tokens and parameters emerge diverse abilities, including math reasoning, code generation, and instruction following. These abilities are further enhanced by supervised fine-tuning (SFT). While the open-source community has explored ad-hoc SFT for enhancing individual capabilities, proprietary LLMs exhibit versatility across various skills. Therefore, understanding the facilitation of multiple abilities via SFT is paramount. In this study, we specifically focuses on the interplay of data composition between mathematical reasoning, code generation, and general human-aligning abilities during SFT. We propose four intriguing research questions to explore the association between model performance and various factors including data amount, composition ratio, model size and SFT strategies. Our experiments reveal that distinct capabilities scale differently and larger models generally show superior performance with same amount of data. Mathematical reasoning and code generation consistently improve with increasing data amount, whereas general abilities plateau after roughly a thousand samples. Moreover, we observe data composition appears to enhance various abilities under limited data conditions, yet can lead to performance conflicts when data is plentiful. Our findings also suggest the amount of composition data influences performance more than the composition ratio. In analysis of SFT strategies, we find that sequentially learning multiple skills risks catastrophic forgetting. Our proposed Dual-stage Mixed Fine-tuning (DMT) strategy offers a promising solution to learn multiple abilities with different scaling patterns.

1 Introduction

The paper investigates how supervised fine-tuning data composition affects mathematical reasoning, code generation, and general human-aligning abilities. It examines data amount, composition, model scale, and training strategy to understand multi-ability learning.

Existing work largely studies reasoning, coding, and human alignment separately, despite proprietary LLMs demonstrating versatility across these abilities.
The study focuses on how data amount, composition ratio, model scale, and SFT strategy relate to performance across the three abilities.
Experiments use GSM8K, HumanEval, and MT-Bench to evaluate mathematical reasoning, coding, and general human alignment, respectively.
Different abilities exhibit distinct scaling patterns, while larger models generally perform better with the same amount of SFT data.
Multi-task learning improves performance in low-resource settings but declines in high-resource settings, with larger models gaining more for math and general abilities.
Data amount directly influences abilities more than data ratio, while DMT is proposed to alleviate multi-task conflicts and catastrophic forgetting.

2 Related Works

Related work covers SFT for general instruction following and specialized reasoning or coding, alongside scaling-law research. Multi-task effects on in-domain performance remain comparatively under-explored.

Supervised Fine-Tuning of Large Language Models: SFT is broadly used to improve task solving, human-instruction alignment, instruction following, and domain-specific optimization.
Supervised Fine-Tuning of Large Language Models: Multi-task instruction fine-tuning has been used to improve zero-shot performance across numerous downstream NLP tasks through datasets such as FLAN.
Supervised Fine-Tuning of Large Language Models: The effects of multi-task training on in-domain performance remain under-explored, motivating closer study of composite SFT data.
Supervised Fine-Tuning of Large Language Models: SFT research separately targets human alignment, mathematical reasoning, and code generation, including datasets generated from user logs or model-assisted processes.
Scaling Laws in Large Language Models: Scaling-law studies examine how performance changes with model size, data, and computation across pre-training, transfer learning, preference modeling, and mathematical reasoning.

3 Experiments

Experiments evaluate individual and mixed SFT data across model sizes, data amounts, and composition ratios. They find ability-specific scaling, low-resource gains from mixing, high-resource conflicts, and limited effects from composition ratio.

Experimental Setup: The experimental framework represents each ability with an SFT dataset and evaluates corresponding in-domain metrics across dataset compositions, strategies, and LLM sizes.
Experimental Setup: GSM8K RFT, Code Alpaca, and ShareGPT represent mathematical reasoning, coding, and general human-aligning data, evaluated with GSM8K, HumanEval, and MT-Bench.
RQ1. Individual Ability Performance vs. Data Amount: General alignment emerges with around 1k samples and then improves slowly, while specialized abilities require more data; code scaling is irregular for 7B and 13B but approximately log-linear for 33B.
RQ2. Performance Difference vs. Mixed Data Amount: Mixed-source SFT improves abilities in low-resource settings but creates performance conflicts in high-resource settings compared with individual-source training.
RQ2. Performance Difference vs. Mixed Data Amount: As model size increases, low-resource gains from mixed data increase for mathematical reasoning and general abilities, with LLaMA-7B showing a code-specific outlier.
RQ3. Performance Difference vs. Data Ratio: The ratio experiments vary specialized and general data while controlling one source, testing whether composition ratio explains observed performance conflicts.
RQ3. Performance Difference vs. Data Ratio: Additional experiments examine generalization beyond the three primary abilities using World Knowledge, Language Understanding, and Translation.

3. Fixed 1/64 general data, scaling specialized

With general data fixed at 1/64, the study examines how specialized-data ratios and training strategies affect mathematical reasoning, coding, and alignment. Data ratio has limited effects overall, while DMT preserves specialized abilities better than sequential training.

Data-ratio effects: Data-ratio changes have minimal impact on mathematical reasoning when general or specialized data is scaled against fixed data.Math performance remains similar to the corresponding fixed-specialized baseline across the tested ratio range.
Data-ratio effects: HumanEval shows noticeable fluctuations when general data increases against fixed specialized data, unlike the more stable math results.The study attributes this difference to greater similarity between some task formats and data distributions.
Data-ratio effects: MT-Bench shows no significant fluctuations as specialized-data proportions vary, even when general data is abundant or scarce.This result is reported for both k = 1 and k = 1/64 settings.
Training strategies: Multi-task learning preserves specialized abilities but hurts general ability most, whereas sequential and mixed sequential training preserve general ability while losing specialized abilities.Multistage training’s specialized-ability loss is linked to catastrophic forgetting of prior knowledge.
Training strategies: DMT combines full specialized-data training first with a small specialized-data mixture during the general-ability stage to reduce conflicts and forgetting.This design follows the observed need for large specialized-data amounts and protection against forgetting during later general fine-tuning.
Training strategies: 32.6 to 41.92 and 15.24 to 17.68 are the LLaMA-7B DMT gains in mathematical reasoning and code generation over mixed sequential training.DMT also shows a slight MT-Bench improvement, indicating preservation of general capability.
Training strategies: 40.48 to 46.47 and 44.24 to 56.36 are DMT’s mathematical-reasoning gains for LLaMA-13B and 33B, respectively, over mixed sequential training.Code-generation scores also rise from 18.3 to 19.5 and 24.4 to 25.5, while MT-Bench increases from 5.93 to 6.03 and 6.43 to 6.69.

4 Discussion

The discussion analyzes representation distributions and specialized-domain ablations to explain composition effects. Removing code and math samples from ShareGPT reduces high-resource conflicts while retaining low-resource gains.

DMT analysis: Figure 5 pairs LLaMA-13B t-SNE plots with LLaMA-7B and 13B DMT performance scaling under different k values.As k rises from 1/256 to 1/4, general and specialized abilities show a linear inverse trend; higher k can reduce general ability.
Ablation rationale: Mixed data sources improve abilities under low-resource conditions but diminish them under high-resource conditions relative to single sources.The analysis tests whether ShareGPT’s embedded coding and mathematical samples explain the low-resource gain.
Ablation results: Removing code and math from ShareGPT mitigates high-resource performance conflicts while maintaining stable low-resource gains.The result is presented in Figure 6’s scaling-curve experiment.
Ablation results: The authors attribute the conflicts to distribution differences among ShareGPT, CodeAlpaca, and GSM8K RFT, which introduce extra SFT noise.They report that removing this gap enables better generalization of coding and mathematical abilities.

5 Conclusion

The conclusion reports distinct scaling patterns across mathematical reasoning, code generation, and general alignment. It finds that data amount matters more than ratio for conflicts and presents DMT as a solution for balancing abilities.

Different abilities scale differently, and larger models generally perform better with the same amount of SFT data.
Mixing data sources improves performance in low-resource settings but diminishes performance in high-resource settings, with stronger low-resource gains for larger models.
Data amount directly influences performance conflicts, whereas data ratio has insignificant impact within the experiments.
DMT effectively alleviates performance conflicts and offers a promising solution for activating multiple abilities.

Limitations

The paper’s scope and applicability are constrained by computational demands, GPT-4-based MT-Bench scoring, and focus on three SFT capabilities.

Using LLaMA-33B requires extensive training and inference resources, which may limit applicability.The evaluation also relies on GPT-4 scoring for MT-Bench, which may vary and diverge from human judgment.
The study primarily covers mathematical reasoning, code generation, and general human-aligned ability, while other SFT capabilities remain unexplored.

A SFT Datasets

The study uses separate SFT datasets for mathematical reasoning, code generation, and general human alignment, then evaluates them with task-specific benchmarks and standardized training settings.

SFT datasets: The SFT sources represent math reasoning, coding, and general human-aligning abilities through GSM8K RFT, Code Alpaca, and ShareGPT.GSM8K RFT contains 7.5K questions and 110K responses; Code Alpaca contains 20K examples; ShareGPT contains 86K human queries and responses.
Evaluation benchmarks: The experiments evaluate math reasoning with GSM8K, coding with HumanEval, and general human alignment with MT-Bench.GSM8K uses greedy-decoding accuracy maj@1, while HumanEval measures functional code correctness and MT-Bench evaluates multi-turn dialogues.
Evaluation benchmarks: The study supplements its primary evaluations with MATH and MBPP as out-of-domain benchmarks for mathematical reasoning and code generation.MATH contains challenging high-school problems, and MBPP contains around 1,000 crowd-sourced Python programming problems.
Training setup: SFT uses three epochs, batch size 128, peak learning rate 2e-5, and a 3% warmup, with final-epoch evaluation.Training uses NVIDIA A100 GPUs and greedy decoding for Pass@1 and maj@1; MT-Bench scores are averaged over three experiments.
Compute estimation: The paper estimates SFT computation from question and response lengths, non-embedding parameters, and sample counts.Each input has length nctx = nQ + nR, where nQ and nR denote question and generated-answer lengths.

E Validation Experiments in More SFT Abilities

Validation experiments extend the composition analysis to world knowledge, language understanding, and translation. Mixed-domain SFT improves performance in low-resource settings but reduces it in high-resource settings relative to individual-domain training.

Experimental design: The validation covers WorldQuestionsSP, CoNLL 2003, and IWSLT14 to assess world knowledge, language understanding, and translation.The study compares individual-domain and mixed-domain SFT on LLaMA models of different sizes.
Experimental design: The validation uses data proportions of 1, 1/2, 1/4, and 1/8 because the three datasets contain relatively small amounts of data.The scaling range therefore runs from the full data volume to one-eighth of the data volume.
Individual-domain results: Individual-domain language-understanding performance generally scales positively with data volume, while translation shows irregular behavior at smaller data ratios.World knowledge also shows positive scaling for F1 and Hits@1, but translation fluctuates notably when the ratio decreases from 1/4 to 1/8.
Mixed-domain results: Mixed-domain training improves abilities in low-resource settings and decreases them in high-resource settings compared with individual-domain training.This pattern holds across world knowledge, language understanding, and translation.

F Results on OOD Benchmarks in Math and Code

OOD evaluations on MATH and MBPP reproduce the paper’s main composition patterns: performance scales with individual-domain data, mixed data trades high-resource conflicts for low-resource gains, and DMT preserves specialized performance while prioritizing general abilities.

OOD results: Individual-domain performance on MATH and MBPP increases positively with data volume.This matches the paper’s earlier findings for individual-domain scaling.
OOD results: LLaMA-7B shows low-resource gains but high-resource performance conflicts under mixed-domain training on both MATH and MBPP.The comparison is between individual-domain and mixed-domain settings.
Training strategies: DMT maintains competitive MATH and MBPP results while prioritizing general abilities.The authors present this as further validation of DMT’s effectiveness.
Representation analysis: Representation visualizations show mathematical samples separating in middle and ending layers, while code and general samples retain some overlap.The pattern is observed for both the base model and the model trained with DMT.

H Equal Data Amount VS. Equal Data Proportion

The study compares mixing datasets with equal total data amounts against equal subset proportions. The two settings produce no significant difference across the three benchmarks, leaving the main conclusions unchanged.

Experimental comparison: The experiments compare equal-data-amount and equal-subset-proportion mixtures of different SFT abilities.Both settings use GSM8K RFT as the baseline and examine proportions 1/16, 1/64, and 1/256.
Experimental comparison: The equal-data-amount setting matches GSM8K RFT’s data amount while sampling corresponding amounts from Code Alpaca and ShareGPT.The comparison uses mixed samples from the three ability datasets.
Experimental comparison: The equal-subset-proportion setting samples each dataset according to its subset proportion before mixing them.This follows the setup used in RQ2.
Results: No significant difference appears across the three benchmark tests between the two mixture settings.The authors conclude that this comparison does not significantly affect the paper’s main experimental conclusions.

I Comparison Experiment of Different Training Sequences

The sequential-training experiments examine how training order affects retention across SFT abilities. Results indicate that final-stage abilities are retained relatively well, while some orders produce code-performance decline associated with catastrophic forgetting; DMT remains superior across random sampling seeds.

Training-order experiments: Six training orders were evaluated to investigate how sequential SFT affects different abilities.The results and analysis are reported in Table 7.
Training-order results: The ability trained in the final stage tends to retain relatively good performance.
Training-order results: Training general and code abilities in the first two stages noticeably decreases code capability, while math capability shows no significant impact.
Training-order results: Similar data distributions between code and general abilities may produce more severe catastrophic forgetting during continuous fine-tuning.
Robustness across subsets: DMT maintains its superiority under three different random-seed settings, and different sampled subsets do not affect the overall trend.