PaperlayerView exact revision on arXiv

Source-linked AI summary

AlpaGasus: Training A Better Alpaca with Fewer Data

Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, Hongxia Jin

arXiv:2307.08701v5cs.CL

TL;DR

Instruction-finetuning datasets can contain misleading low-quality responses, motivating automatic quality selection. AlpaGasus uses a strong LLM to filter examples before fine-tuning and outperforms Alpaca with substantially less data and training time.

  • Problem

    Instruction-finetuning data can contain incorrect or irrelevant responses, while human-based quality filtering is costly and difficult to perform accurately.

  • Method

    AlpaGasus prompts a strong LLM to score instruction, input, and response triplets, filters low-scoring examples, and fine-tunes on the retained subset.

  • Results

    AlpaGasus significantly outperforms Alpaca across four test sets using 9k filtered examples instead of 52k original examples.

  • Takeaways & Limitations

    The results support prioritizing instruction-finetuning data quality, with the approach generalizing across datasets, base models, and LLM filters.

  • Takeaways & Limitations

    The study evaluates only 7B and 13B models and leaves larger model sizes for future work.

Abstract

from arXiv · show

Large language models (LLMs) strengthen instruction-following capability through instruction-finetuning (IFT) on supervised instruction/response data. However, widely used IFT datasets (e.g., Alpaca's 52k data) surprisingly contain many low-quality instances with incorrect or irrelevant responses, which are misleading and detrimental to IFT. In this paper, we propose a simple and effective data selection strategy that automatically identifies and filters out low-quality data using a strong LLM (e.g., ChatGPT). To this end, we introduce AlpaGasus, which is finetuned on only 9k high-quality data filtered from the 52k Alpaca data. AlpaGasus significantly outperforms the original Alpaca as evaluated by GPT-4 on multiple test sets and the controlled human evaluation. Its 13B variant matches $>90\%$ performance of its teacher LLM (i.e., Text-Davinci-003 generating the 52k data) on test tasks. It also provides 5.7x faster training, reducing the training time for a 7B variant from 80 minutes (for Alpaca) to 14 minutes. Moreover, the experiments prove the efficacy of our method across diverse datasets, base models, and LLM filters. Overall, AlpaGasus demonstrates a novel data-centric IFT paradigm that can be generally applied to instruction-tuning data, leading to faster training and better instruction-following models. Our project page is available at: https://lichang-chen.github.io/AlpaGasus/

1 INTRODUCTION

The paper argues that instruction-finetuning data often contains misleading low-quality responses and introduces automatic LLM-based filtering to prioritize data quality over quantity. AlpaGasus uses a smaller filtered subset and achieves stronger instruction-following performance with lower training cost.

  • Motivation: Instruction-finetuning datasets can contain incorrect or irrelevant responses that mislead training and increase unnecessary costs.Human quality assessment is expensive and may miss eloquent but incorrect responses.
  • Approach: The proposed strategy uses a powerful LLM to evaluate training examples and filter low-quality data automatically.The method targets each instruction, input, and response tuple.
  • Results: 9k filtered examples produce a better AlpaGasus model than the original Alpaca trained on 52k examples under the same training configuration.The paper reports that the method retains only a small subset while improving model quality.
  • Results: 7B training time falls from 80 minutes to 14 minutes on 4× NVIDIA A100 GPUs.The comparison uses the original Alpaca training configuration.
  • Scope: The approach generalizes across datasets, base models, and LLM filters, supporting a broader data-centric IFT paradigm.Reported settings include Dolly, Alpaca, GPT4LLM, LLaMA-1, LLaMA-2, ChatGPT, and Claude-2.
  • Evaluation: The experiments evaluate instruction-following across four test sets, with ALPAGASUS significantly outperforming ALPACA for both 7B and 13B models.The evaluations use GPT-4 judging together with human-oriented evaluation designs.

2 METHODOLOGY

AlpaGasus rates each instruction-finetuning triplet with a strong LLM, keeps examples above a threshold, and fine-tunes a LLaMA-series model using an existing IFT pipeline. Accuracy is used as the rating dimension, with τ = 4.5 yielding 9,229 selected examples from Alpaca’s 52,002.

  • Overview: The method replaces costly human curation with a strong LLM auto-grader for selecting instruction-finetuning data.This design is intended to make filtering efficient, automatic, and accurate.
  • Data Rating: ChatGPT assigns a quality score to each (instruction, input, response) triplet.The rating prompt specifies a user-preferred dimension such as helpfulness or accuracy.
  • Data Filtering: Only triplets scoring above threshold τ are retained for fine-tuning.The selected subset S is obtained from dataset V by applying the score threshold.
  • Fine-Tuning: The selected data are used to fine-tune the base model with an existing instruction-finetuning framework.The training script follows the established Alpaca pipeline.
  • Additional Evaluators: Claude-2 is also used as a response-quality evaluator in an additional experiment.The paper places this evaluation in Appendix A.2.
  • Threshold Selection: Accuracy is the rating dimension, and τ = 4.5 selects 9,229 samples from Alpaca’s 52,002 examples.The threshold is chosen according to the score histogram.

3 EXPERIMENTAL SETUP

The experiments use diverse instruction test sets, baseline language models, and automated evaluation procedures to assess AlpaGasus. GPT-4 compares model responses while repeated response ordering reduces position bias in judging.

  • Test Sets: Four test sets from Self-Instruct, Vicuna, WizardLM, and Koala provide broader instruction coverage and reduce evaluation bias.The paper motivates this multi-source design because a single test set may not represent diverse instructions.
  • Baseline Models: The baseline suite includes Alpaca, Text-Davinci-003, ChatGPT, and Claude.Alpaca is trained on 52,002 teacher-generated instruction, input, and response samples.
  • Evaluation: GPT-4 serves as an automated judge comparing responses from the filtered and baseline models.The judge scores the two responses for each instruction drawn from a test set.
  • Evaluation: The judge rates responses on a 1-to-10 scale after receiving both model outputs.The evaluation addresses the difficulty of reproducing human judgments when multiple responses may be acceptable.
  • Evaluation Metrics: Win-Tie-Lose results combine judgments from both response orders to mitigate position bias.ALPAGASUS wins only when it wins twice or wins once and draws once; analogous rules define ties and losses.

4 EXPERIMENTAL RESULTS

Across experiments, AlpaGasus shows that filtering for data quality can outperform using more unfiltered data, while retaining performance with fewer examples and diverse evaluation settings.

  • AlpaGasus-9k significantly outperforms Alpaca-52k across the Vicuna, Koala, Self-Instruct, and WizardLM test sets.
  • AlpaGasus-9k significantly outperforms Alpaca-9k-random at both 7B and 13B model sizes.The models use the same initial model and number of training samples, differing only in data-selection criteria.
  • 9k selected examples produce the best performance among models trained on 3k, 6k, and 9k high-quality data across all four test sets.
  • 6k high-quality examples suffice to finetune LLaMA to performance similar to the original Alpaca.
  • Human evaluation records 63 wins, 64 ties, and 33 losses for AlpaGasus-13B versus Alpaca-13B over 160 prompts.
  • AlpaGasus-13B achieves at least 90% of the capacity of its teacher, text-Davinci-003.

5 HUMAN-WRITTEN INSTRUCTION SET FILTERING

The filtering strategy also applies to human-written instruction data, where a much smaller filtered subset outperforms training on the full Dolly dataset.

  • The method is applied to Databricks-Dolly-15k, a human-written dataset containing 15,000 prompt/response pairs.
  • 2,996 filtered Dolly examples yield superior performance to the full Dolly-15k dataset for 7B and 13B LLaMA models.The filtered subset uses a threshold of 4.5.
  • The 13B filtered-3k model consistently outperforms random-3k and full-15k baselines on all four benchmark datasets.
  • The results suggest that ChatGPT-based filtering can also be applied to GPT-4 responses.

6 CASE STUDY & ANALYSIS

Case studies and skill-level analyses examine where filtered training data improves model behavior and where imbalanced selection can weaken specific capabilities.

  • Case studies: In a math case study, AlpaGasus-13B produces a correct answer and receives a GPT-4 score of 10.0, versus 2.0 for Alpaca-9k-random.
  • Skill analysis: AlpaGasus-7B performs better or equally well than Alpaca on 22 of 29 WizardLM skills.
  • Skill analysis: Coding-related data are filtered at a higher ratio, leaving coding skill weaker than other skills.The reported filtering ratio for coding-related data is 82.25%.
  • Skill analysis: The coding-skill results indicate that instruction-tuning data should remain diverse and balanced across categories.

7 COST SAVING

Using fewer selected training examples substantially reduces instruction-finetuning time and estimated AWS computation costs for both 7B and 13B models.

  • 7B training time falls from 80 minutes to 14 minutes, while 13B training time falls from 5.5 hours to 1 hour.
  • Estimated AWS cost decreases from $27.31 to $4.78 for 7B and from $225.28 to $40.969 for 13B models.
  • The authors state that cost savings become progressively more pronounced as model size increases.

8 RELATED WORK

Prior work develops instruction-following models through diverse data-generation strategies and emphasizes data quality, automated cleaning, and broader evaluation. These studies motivate assessing instruction-tuned models with human instruction sets rather than relying only on conventional benchmarks.

  • Open-sourced Instruction-following models: Instruction-tuning data comes from crowdsourced human responses, machine-generated teacher-LM distillation, realistic user dialogues, and evolved instruction sets.Examples include Dolly, Alpaca, Vicuna, Koala, UltraChat, and WizardLM-related datasets.
  • Data-centric AI: Data-centric AI treats data quality as important as algorithms and advocates automated, adaptable cleaning across diverse domains.The literature frames this shift as increasing emphasis on data quality in the AI/ML lifecycle.
  • Evaluation of LLMs: Open-ended instruction-following evaluation has been underemphasized relative to factuality and reasoning benchmarks for pretrained models.Recent work supplies human instruction sets, instruction categories, and leaderboards to address this gap.

9 CONCLUSION

The paper concludes that automatically filtering instruction-tuning data with strong LLM evaluators can make instruction fine-tuning more effective and computationally efficient. Results remain favorable across alternative evaluation prompts, filters, base models, and datasets, although some experiments expose limitations in rating granularity.

  • 9 CONCLUSION: The method shows benefits in scalability, automation, performance, and computation savings, with evidence of generalization across datasets and LLMs.The conclusion presents this as a data-centric direction for practical LLM deployment.
  • 9 CONCLUSION: AlpaGasus consistently outperforms across all test sets when evaluated with an alternative prompt from Zheng et al. (2023).The same Win-Tie-Lose rules are applied under the alternate evaluation prompt.
  • 9 CONCLUSION: Using Claude2 as the filter, a 7B model trained on 8k selected examples beats the 52k Alpaca model on 3/4 test sets and exceeds the 8k random-data model.This supports transfer of the filtering approach beyond the ChatGPT-based filter.
  • 9 CONCLUSION: Claude2 assigns scores at intervals of 1 rather than 0.5, so the present 5-point scale does not provide fine-grained ratings.The paper leaves exploration of alternative rating scales to future work.
  • 9 CONCLUSION: On GPT4LLM data, filtering selects 13,721 examples and achieves comparable outcomes to the original dataset using merely 25% of its data.The filtered model outperforms randomly selected counterparts on the Vicuna and WizardLM test sets.

A.5 RESULTS ON OTHER RATING DIMENSIONS, E.G., HELPFULNESS?

Additional experiments show that filtering remains effective when helpfulness replaces accuracy as the rating dimension and when applied to the Dolly dataset. A smaller filtered subset can outperform models trained on all Alpaca data, but the appendix also documents the associated evaluation and training settings.

  • Results on other rating dimensions: 2k helpfulness-filtered examples suffice to surpass a model trained with 52k Alpaca examples.The score distributions for this rating dimension are shown in Figure 18.
  • Results on other rating dimensions: Filtered Alpaca models outperform randomly selected models across all instruction test sets and beat the complete Alpaca model on 3/4 test sets.The authors highlight this result as evidence that 2k data can outperform the original 52k dataset.
  • Dolly dataset: The Dolly experiments evaluate score distributions and benchmark results using the provided code and specified few-shot settings.The appendix reports MMLU, BBH, DROP, and HumanEval evaluations for Dolly and its filtered set.
  • Dolly dataset: Dolly training uses 10 epochs for 7B models and 15 epochs for 13B models, rather than Alpaca’s 3 and 5 epochs.The stated change is intended to avoid under-training.
  • Dolly dataset: The filtered Dolly dataset produces stronger instruction-following capacity than the original Dolly dataset in the reported 13B results.The comparison is shown for instruction-tuned LLaMA models across four tests.

H.1 ANALYSIS ON WIZARDLM TEST SET

Fine-grained analyses show that AlpaGasus improves across most WizardLM and Vicuna skill categories, while coding remains a weaker area because filtering disproportionately removes coding-related data. The 13B model reaches at least 91% of its teacher’s capacity, but larger model sizes remain untested.

  • H.1 ANALYSIS ON WIZARDLM TEST SET: AlpaGasus-7B performs better or equally well than Alpaca-7B on 22/29 WizardLM skills.The remaining seven skills include coding-related categories such as code generation.
  • H.1 ANALYSIS ON WIZARDLM TEST SET: Coding-related data has a filtering ratio of 718−85/52002 = 82.25%, leaving coding weaker than other skills.The authors identify this imbalance as evidence that training data should remain diverse and balanced across categories.
  • H.1 ANALYSIS ON WIZARDLM TEST SET: AlpaGasus outperforms Alpaca in most Vicuna categories, including Counterfactual, Roleplay, Knowledge, and Generic, and the conclusion persists as the base model scales.The detailed comparison is presented in Figure 23.
  • H.1 ANALYSIS ON WIZARDLM TEST SET: AlpaGasus-13B achieves ≥91% of the capacity of its Text-Davinci-003 teacher across the evaluated skills.The comparisons also include ChatGPT and Claude, with stronger relative performance in writing, roleplay, toxicity, and art than in coding and math.
  • Limitations: The study evaluates 7B and 13B models but leaves 33B, 65B, and 175B analyses for future work.Whether the same conclusion holds at those larger sizes is not established in this study.
Loading 2307.08701v5…