Source-linked AI summary
Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, Tianyi Zhou
TL;DR
Instruction-data filtering improves tuning efficiency and performance but can itself be costly when it relies on large language models. Superfiltering uses a smaller model to score and select data for a stronger model, and experiments report comparable or improved tuning performance with substantially lower filtering cost. The approach is supported by observed consistency in instruction-difficulty judgments across weak and strong models.
Problem
Instruction-data filtering can require expensive large-model inference, motivating whether a weaker model can select data for training a stronger model.
Method
Superfiltering uses a small model such as GPT-2 as a proxy to evaluate instruction difficulty and select data for instruction tuning a larger model.
Results
Superfiltering achieves comparable or superior instruction-tuning performance using 5% of the original data, while substantially accelerating filtering.
Takeaways & Limitations
Weak models can serve as efficient filters for stronger-model instruction tuning because their instruction-difficulty judgments are consistent enough for data selection.
Takeaways & Limitations
The study covers only three datasets and two LLaMA2 base models, and its selection primarily uses IFD scores rather than broader dimensions such as data diversity.
Abstract
from arXiv · showhide
Instruction tuning is critical to improve LLMs but usually suffers from low-quality and redundant data. Data filtering for instruction tuning has proved important in improving both the efficiency and performance of the tuning process. But it also leads to extra cost and computation due to the involvement of LLMs in this process. To reduce the filtering cost, we study Superfiltering: Can we use a smaller and weaker model to select data for finetuning a larger and stronger model? Despite the performance gap between weak and strong language models, we find their highly consistent capability to perceive instruction difficulty and data selection results. This enables us to use a much smaller and more efficient model to filter the instruction data used to train a larger language model. Not only does it largely speed up the data filtering, but the filtered-data-finetuned LLM achieves even better performance on standard benchmarks. Extensive experiments validate the efficacy and efficiency of our approach.
1 Introduction
Superfiltering addresses the cost of LLM-based instruction-data filtering by using a much smaller model to select data for training a stronger LLM. Experiments indicate that 5% selected data can match or exceed full-data tuning while substantially accelerating filtering.
- GPT-2 filtering speeds data selection by approximately 20 times, while 5%-filtered models outperform a 100%-data baseline in the illustrated comparison.
- LLM-based filtering can add expensive inference cost and latency when only a small fraction of a large dataset must be selected.
- Superfiltering uses a much smaller model, such as GPT-2, to select instruction-tuning data for a stronger student LLM.
- 5% of the original data can produce LLMs comparable or superior to models trained on the full dataset.
- The study evaluates Superfiltering across widely used instruction datasets and compares selected-data models with full-data models using pairwise evaluation.
- Weak and strong LLMs show strong consistency in perceiving and evaluating instruction difficulty.
2 Problem Formulation
The paper formulates instruction-tuning examples as instruction-response pairs and uses perplexity-based difficulty measures to identify informative data. It hypothesizes and empirically examines whether weak models can provide scores consistent with stronger models for filtering.
- 2.1 Preliminaries: Instruction-tuning data consists of triplets containing an instruction, optional input, and response, represented as instruction-response pairs after mapping inputs to instructions.
- 2.1 Preliminaries: Perplexity measures the difficulty of generating a response conditioned on its instruction.
- 2.1 Preliminaries: Perplexity is computed from the response length and token-level conditional probabilities.
- 2.1 Preliminaries: The IFD score compares response perplexity with and without the instruction; higher scores indicate less instructional help and greater difficulty.
- 2.2 Formulation and Motivations: Superfiltering seeks a low-cost score that identifies high-quality, informative data and remains consistent between weak and strong language models.
- 2.2 Formulation and Motivations: The central hypothesis is that weak and strong models may differ in intrinsic ability but perceive instruction difficulty similarly.
- 2.2 Formulation and Motivations: Experiments compare perplexity scales and sample orderings across models, finding differing scales but consistent orderings; IFD scales and orderings are also consistent.
- 2.2 Formulation and Motivations: GPT-4-judged win-tie-lose comparisons show Superfiltering-trained models outperform full-data baselines across base models, datasets, and selection ratios.
3 Weak-to-Strong Consistency
Weak and strong language models produce substantially different perplexity scales but preserve similar sample rankings, while IFD scores show consistency in both scale and ordering. This consistency supports using small models to filter data for larger-model instruction tuning.
- Weak-to-Strong Perplexity Consistency: Across datasets, stronger models have lower perplexity, but perplexity scales vary substantially between models.The study evaluates GPT-2 variants, GPT-NEO, and LLaMA2-7B on Alpaca, Alpaca-GPT4, and WizardLM 70k.
- Weak-to-Strong Perplexity Consistency: Spearman’s rank correlation measures whether different models assign similar orderings to the same instruction-tuning samples.The samples are sorted by model-computed perplexity and compared against the LLaMA2-7B ordering.
- Weak-to-Strong Perplexity Consistency: The lowest perplexity-ranking coefficient exceeds 0.7 for GPT-2 and LLaMA2-7B, while the highest exceeds 0.85 for GPT-NEO and LLaMA2-7B.These values indicate substantial consistency in how models rank instruction-tuning samples by perplexity.
- Weak-to-Strong IFD Consistency: Unlike perplexity, IFD scores remain similar in scale across models and retain high rank consistency across instruction-tuning datasets.IFD compares response-generation losses with and without instructional context; higher scores indicate less instructional help and greater difficulty.
- Weak-to-Strong IFD Consistency: Selecting the top 5%, 10%, or 15% by IFD produces substantial and increasing overlap between weak-model and LLaMA2-7B selections.The overlap is not complete, but it grows as the selection threshold increases.
- Superfiltering: Superfiltering uses GPT-2 as a low-resource substitute for larger filter models when selecting data for modern LLM instruction tuning.The approach is motivated by the observed weak-to-strong consistency in IFD-based difficulty evaluation.
4 Experimental Setup
The experiments apply the method to Alpaca and Alpaca-GPT4 and use standard automated evaluations alongside a human comparison study. Training follows common configurations with specified optimizers, learning rates, batch size, epochs, and sequence length.
- Datasets: The experiments use Alpaca, a 52,000-sample self-instruct dataset, and the higher-quality Alpaca-GPT4 dataset.Alpaca was generated with text-davinci-003, while Alpaca-GPT4 provides an additional dataset for validation.
- Training Configuration: Training uses Vicuna prompts and code, flash attention, Adam optimization, batch size 128, three epochs, and maximum length 2048.The learning rate is 2×10−5 for LLaMA2-7B and 1 × 10−5 for LLaMA2-13B, with a 0.03 warmup rate.
- Evaluation: Automated evaluation includes pair-wise comparison, Open LLM Leaderboard, and Alpaca Eval.
- Human Evaluation: A human study compares model responses to 100 WizardLM test instructions using helpfulness, relevance, accuracy, and level of detail.Three human participants perform the comparisons using the same criteria as the pair-wise evaluation.
5 Experimental Result
Experiments show that Superfiltering can select small, high-quality training subsets while reducing filtering cost and matching or exceeding full-data training. Across ablations and comparisons, weak-model filtering remains effective and efficient.
- Main results: Superfiltering models using 5%, 10%, or 15% data outperformed full-data models on the Open LLM Leaderboard and AlpacaEval for LLaMA2-7B and LLaMA-13B.The comparison used pairwise winning scores against corresponding full-data models.
- Main results: Human evaluation found 5% Superfiltering data won 50 of 100 Alpaca comparisons and 49 of 100 Alpaca-GPT4 comparisons against full-data training.For Alpaca, the model tied 18 and lost 32; for Alpaca-GPT4, it tied 5 and lost 46.
- Comparison with Other Methods: Superfiltering outperformed ChatGPT and reward-model selection in winning scores and efficiency, while direct IFD selection performed better but was 20× slower.Reward-model selection was reported as 175× slower than Superfiltering, and Superfiltering was the only method faster than subsequent training.
- Ablation Study: Across data-selection strategies, Superfiltering consistently outperformed full-data training, and IFD scores transferred across language models.LLaMA2-7B achieved the highest performance among tested IFD-scoring models, consistent with the model being trained.
6 Further Discussion
Further discussion argues that Superfiltering works without training proxy models or using extra hold-out sets. The authors connect this efficiency to weak-to-strong consistency in IFD-related metrics and propose dataset-level assessment using difficulty distributions.
- Methodological advantages: Superfiltering does not require training even weak language models or using extra hold-out sets.This contrasts with proxy-based approaches that train weak models or require hold-out data.
- Methodological advantages: Pre-trained weak models can act as proxies for strong models when IFD is used for data selection, without dataset-specific fine-tuning.The authors associate this with reduced out-of-distribution risk and a simpler selection process.
- Weak-to-strong consistency: Consistency of perplexity-based metrics across weak and strong models suggests small language models may serve as proxy models beyond data selection.The authors also describe the metric as an efficient way to assess instruction data across a dataset.
- Dataset assessment: Violin plots of Instruction Followed Difficulty scores provide a preliminary assessment of dataset quality and variation.The Alpaca dataset shows substantial quality variance, while WizardLM combines complex instructions with generation noise.
- Weak-to-strong consistency: The authors hypothesize that weak-to-strong consistency can be explained through perplexity and string-pattern perspectives.These explanations are presented as hypotheses and are discussed in an appendix.
7 Related Work
Related work covers instruction-tuning data selection and proxy models for larger systems. Existing selection methods often rely on proprietary LLMs, reward models, or the original pretrained model, while proxy research motivates resource-efficient alternatives.
- Instruction Tuning Data Selection: Instruction-tuning data selection methods use ChatGPT or Claude2 for quality and diversity assessment, reward models for scoring, or pretrained models for IFD calculation.These approaches aim to select data more efficiently but can depend on additional or large language models.
- Instruction Tuning Data Selection: Prior data-selection methods are described as overly reliant on large language models and potentially time-consuming.The cited approaches include proprietary-model tagging, reward-model assessment, and self-guided IFD scoring.
- Small Model Proxies for Large Models: Proxy-model research uses lightweight models to evaluate rationales, classify with expert embeddings, simplify complex systems, or provide weak supervision.Examples span free-text evaluation, nearest-neighbor classification, image classification, information extraction, and weak supervision.
8 Conclusion
The paper identifies consistency between weak and strong models in perceiving instruction difficulty and introduces Superfiltering as an efficient filtering approach. Experiments report lower computational overhead while maintaining or improving instruction-tuning performance.
- Conclusion: Superfiltering uses weaker models as proxies to evaluate instruction data for instruction tuning.The approach is based on the reported consistency between weak and strong language models in perceiving instruction difficulty.
- Conclusion: The method considerably reduces computational overhead while maintaining or improving instruction-tuning performance.The conclusion characterizes Superfiltering as scalable, resource-efficient, and effective.
Limitations
The paper identifies scope and design boundaries for Superfiltering and extends it with diversity-aware selection. Preliminary results suggest that using 2% of the data can match or exceed full-data performance, but broader validation remains needed.
- Superfiltering has been evaluated on only 3 datasets and 2 LLaMA2 base models, motivating testing across broader model and dataset collections.
- The method primarily uses IFD scores, so future improvements should incorporate additional selection dimensions such as data diversity.
- Superfiltering with Diversity: Superfiltering.D first selects a relatively high-quality subset with Superfiltering, then applies the Facility Location Function to compress it while balancing diversity and cluster representation.
- Superfiltering with Diversity: Using an approximately 80M-parameter sentence-transformers/all-MiniLM-L6-v2 encoder, preliminary experiments selected 20% and then 2% of Alpaca and Alpaca-GPT4 data.
- Superfiltering with Diversity: The preliminary 2%-data models achieved performance comparable to or better than models trained on full data.
B Evaluation Metric
The evaluation combines detailed response ratings, pairwise comparisons, and public leaderboards. These measures assess instruction-following quality, while AlpacaEval has narrower task coverage and was run only in the 5% setting because of budget constraints.
- The WizardLM and Vicuna test sets contain 218 and 80 diverse human-curated instructions, respectively.
- Responses are rated along dimensions including accuracy and relevance, following prior evaluation strategies for language-model outputs.
- AlpacaEval: AlpacaEval compares model outputs with Davinci003 on generic user instructions and is described as automated, efficient, and aligned with human expert annotations.
- AlpacaEval: AlpacaEval emphasizes simpler instructions, omits safety and complex-task evaluations, and may correlate win rates with response lengths; evaluation was limited to the 5% setting.
- Open LLM Leaderboard: The Open LLM Leaderboard evaluates ARC, HellaSwag, MMLU, and TruthfulQA, covering reasoning, common-sense understanding, and factual accuracy.
D Why Weak-to-Strong Consistent?
The paper attributes weak-to-strong consistency to shared structure in perplexity and instruction patterns. Although model scales differ, their sample rankings and qualitative preferences can remain aligned.
- Perplexity: The proposed explanation begins with consistency in perplexity because IFD is computed as a ratio of two perplexities.
- Perplexity: Similar training corpora may expose weak and strong models to the same underlying language distribution, supporting consistent perplexity rankings.
- String Pattern: On Alpaca, GPT2 and LLaMA2-7B showed strong consistency in the verb-noun pairs of samples selected by IFD scores.
- String Pattern: Both models assigned higher IFD scores to data requiring creativity, thinking skills, and deep understanding, while assigning lower scores to other samples.