PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Train Data-Efficient LLMs

Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley, Derek Zhiyuan Cheng

arXiv:2402.09668v1cs.LGcs.AIcs.CL

TL;DR

Large language model pre-training is resource-intensive, while scaling yields diminishing returns. This paper studies data-selection approaches for improving the tradeoff between data quantity and model quality, finding that LLM-based quality filtering offers a Pareto-optimal efficiency tradeoff and can outperform full-data training even after substantial data removal.

  • Problem

    LLM pre-training is highly data- and compute-intensive, and empirical scaling laws predict diminishing returns from linear increases in model or data size.

    Source: §1 · p. 1

  • Method

    The paper compares sampling algorithms that select high-quality data through capable proxies and maximize coverage through embedding similarity, including Density sampling in latent space.

    Source: §3.2 · p. 4, §6 · p. 8

  • Results

    Experiments find LLM-based quality filtering yields a Pareto-optimal efficiency tradeoff, and Ask-LLM can outperform full-data training after removing up to 90% of training samples.

    Source: §1.1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Coverage sampling can recover full-data performance, while Ask-LLM quality filtering can often exceed it, making LLM-based quality raters a worthwhile approach for pre-training.

    Source: §1.1 · p. 2

    Ask-LLM and perplexity scorers require one inference call per training sample, creating computation, carbon-emissions, and cost concerns.

    Source: §5 · p. 8

Abstract

from arXiv · show

The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.

1. Introduction

LLM pre-training is costly, and scaling data or model size faces diminishing returns. This paper compares quality- and coverage-oriented data selection, finding that ASK-LLM can outperform full-data training while using substantially less data.

1. Introduction

  • The study asks whether cheap coverage heuristics suffice for LLM pre-training or costly, example-level quality scoring offers benefits.The motivation is to improve the data-quantity/model-quality frontier amid diminishing returns from scaling data or model size.Source: §1 · p. 1
  • On T5-Large, ASK-LLM trained with 60% of the original dataset performs better and trains 70% faster than training with 100%.The scorer is Flan-T5-XL.Source: Figure 1 · p. 2

1.1. Contributions

  • Large-scale comparisons of data-pruning strategies are rare because they require costly pre-training across model sizes, sampling rates, and strategies, plus downstream fine-tuning.The paper benchmarks 19 strategies on T5-Large and T5-Small, using 524B tokens and 111 downstream tasks.Source: §1 · p. 2, §1.1 · p. 2
  • ASK-LLM and DENSITY isolate quality and coverage: a proxy LLM rates informative examples, while DENSITY diversifies selection across latent topics.The paper studies their roles alongside sampling cost, convergence, sampler correlations, and downstream performance.Source: §1.1 · p. 2
  • With 90% of training samples removed, ASK-LLM can still train better models than full-data training and consistently beat established curation routines.Even a 60M-parameter proxy can outperform most baselines.Source: §1.1 · p. 2
  • Coverage sampling can recover full-data performance, while ASK-LLM quality filtering can often exceed it.The authors conclude that LLM-based quality raters are effective for pre-training.Source: §1.1 · p. 2

2. Related Work

Related work organizes data selection around coverage and quality scoring. Coverage methods seek broad representation in feature space, while quality methods score examples and prioritize those with higher scores.

2. Related Work

  • Data selection has established foundations in coresets, sketching, importance sampling, filtering, and denoising.The review focuses on how such principles have been applied to machine learning, especially language-model training.Source: §2 · p. 2

2.1. Coverage Sampling

  • Coverage sampling selects examples spread across the input domain, often by embedding them and choosing points far from one another.Its motivation for language models is to preserve breadth across genres, topics, and languages.Source: §2.1 · p. 2
  • Cluster sampling groups inputs by embedding similarity and selects representatives; related approaches use submodular coverage, discrepancy minimization, or deduplication.These methods are described as popular, scalable, interpretable, and supported by theory.Source: §2.1 · pp. 2–3

2.2. Quality-score Sampling

  • Quality-score sampling rates each example and preferentially selects high-scoring points, extending score-and-sample methods into practical heuristics.Examples include validation-loss and uncertainty scores, EL2N, ensemble disagreement, and forgetting or memorization measures.Source: §2.2 · p. 3
  • LLM quality-score methods include low-perplexity filtering and selecting data close to curated sources such as Wikipedia or books.Other work applies these ideas to data-mixture weights, and concurrent work studies LLM-based paraphrasing and datamodel-based quality evaluation.Source: §2.2 · p. 3

3. Methods

The paper contrasts ASK-LLM, which scores example quality using an instruction-tuned proxy, with DENSITY, which targets coverage through latent-space density estimates. Different score-based sampling rules determine which examples are selected.

3. Methods

  • The two samplers target different goals and costs: ASK-LLM makes contextual quality judgments with an LLM call per example, while DENSITY is a cheaper diversified routine.DENSITY uses embeddings and kernel sums to estimate density; high-density regions are treated as redundant and under-represented regions as coverage opportunities.Source: §3 · p. 3, §3.2 · p. 4

3.1. ASK-LLM Sampling

  • ASK-LLM scores each example with an instruction-tuned proxy LLM, asking whether it should be used for training and treating the “yes” probability as its quality score.The approach is motivated by common perplexity-filtering errors involving missing context, repetitive nonsense, and informative niche examples.Source: §3.1 · p. 4

3.2. DENSITY Sampling

  • DENSITY estimates local density from embeddings with kernel sums, then uses a two-pass approximation that reduces computation from O(N^2) to O(N log N).Its density estimation operates in the model's latent space rather than using Jaccard distances on n-grams.Source: §3.2 · p. 4

3.3. Sampling Techniques

  • DENSITY uses inverse propensity sampling to maximize coverage, while ASK-LLM uses top-k selection because the “yes” probability is treated as a quality measure.Inverse propensity sampling reweights inverse scores to uniformize selected inputs; top- and bottom-K instead select score extremes.Source: §3.3 · p. 4

3.4. Relationships Between Methods

  • DENSITY measures latent-space data density, whereas perplexity measures likelihood under a scoring model; ASK-LLM's contextual quality judgment can avoid perplexity's in-distribution bias.DENSITY is a nonparametric model and cannot make the same fine-grained distinctions between texts on one topic as a perplexity filter.Source: §3.4 · p. 5

4. Experiments

Across T5-Small and T5-Large pre-training, the experiments compare sampling methods at varied data fractions under a fixed-token training budget. ASK-LLM and coverage methods show different tradeoffs: ASK-LLM performs strongly across the quantity-quality frontier, while DENSITY performs well in the middata regime.

4.1. Models

  • The experiments use T5-Small (60M) and T5-Large (800M), with quality samplers scored by T5-sized proxy models; training runs use batches of 1,024 sequences of length 512 for 1M steps.ASK-LLM uses instruction-tuned FLAN-T5 proxies.Source: §4.1 · p. 5

4.2. Datasets

  • Sampling uses 10%, 20%, 40%, 60%, or 80% of C4, and every model is trained for 524B tokens in an iso-compute setting.Smaller samples therefore receive more epochs, which the authors intend to give each curation method an equal chance to maximize performance.Source: §4.2 · p. 5

4.3. Evaluation

  • Models are evaluated on 111 downstream tasks, with over-scaling measuring average relative improvement against the next-largest model size.Figure 4 also compares in-distribution C4 perplexity and higher-quality-set perplexity, for which lower values are better.Source: §4.3 · p. 5, Figure 4 · p. 6

4.4. Does reasoning improve data efficiency?

  • ASK-LLM closes up to 33% of the performance gap to the next-largest model and outperforms full-data training, perplexity filtering, and coverage baselines; it also converges faster than perplexity filters.The comparisons cover T5-Small and T5-Large; Figure 4 reports tradeoffs, and Figure 5 averages intermediate training checkpoints across proxy sizes and sampling ratios.Source: §4.4 · p. 6, Figure 4 · p. 6, Figure 5 · p. 7

4.5. When are expensive quality scores justified?

  • Coverage scores, especially DENSITY, perform well at roughly 25% to 50% sampling, while ASK-LLM is Pareto optimal across the quantity-quality tradeoff.The authors identify full-data performance and the low-data regime as settings where costly quality scoring is most justified.Source: §4.5 · p. 7

4.6. Effect of quality-scoring model capacity

  • Larger ASK-LLM scoring models are increasingly beneficial as the target model grows, while perplexity filters show no comparable scaling trend.Even small ASK-LLM models perform compellingly, and ASK-LLM (Small) outperforms perplexity filtering with any scoring-model in Figure 4f.Source: §4.6 · p. 7

4.7. Do samplers prioritize different examples?

  • Perplexity and ASK-LLM select for almost uncorrelated criteria; T5-Large perplexity scores also become more nuanced from 20k to 700k training steps.DENSITY prioritizes coverage and maintains in-distribution test perplexity better than other strategies.Source: §4.7 · p. 8

5. Discussion

The discussion weighs the per-example cost of ASK-LLM and perplexity scoring against their potential value across pre-training runs. It also connects ASK-LLM’s distinct scoring behavior to questions about LLMs selecting data for future training.

5. Discussion

  • ASK-LLM and perplexity scoring each require an LLM inference call per training sample, creating computation, cost, and carbon-emissions concerns.The authors argue that these costs can be amortized across multiple pre-training runs, which together cost significantly more than the scoring calls.Source: §5 · p. 8
  • ASK-LLM scores show no correlation with perplexity scores, which the authors attribute to the information prompting adds to the sampler.They conclude that reasoning and context are crucial ingredients, and expect chain-of-thought prompting to further drive performance.Source: Figure 7 · p. 8
  • ASK-LLM’s results raise the question of whether LLM-based filters can intervene in the self-consumption loop and allow LLMs to self-improve.The discussion frames this as an open question prompted by concerns about degradation from recursively training on model-generated data.Source: §5 · p. 8

6. Conclusion

The conclusion compares quality filtering through capable proxies with coverage sampling through embedding similarity, finding that LLM-based quality filtering yields a Pareto-optimal tradeoff between data quantity and model quality.

6. Conclusion

  • LLM-based quality filtering yields a Pareto-optimal efficiency tradeoff between data quantity and model quality.The conclusion links this finding to training cost, self-improvement, and LLM training-data curation.Source: §6 · p. 8
  • The paper studies selecting high-quality data through capable proxies and maximizing coverage through embedding similarity.Source: §6 · p. 8
  • The authors identify implications for training cost, self-improvement, and LLM training-data curation.Source: §6 · p. 8

Impact Statement

The impact statement expects data-efficient pre-training to reduce amortized carbon emissions and pre-training cost while improving quality.

Impact Statement

Appendices

The appendix describes ASK-LLM sampling as scoring every training example with a model and selecting the highest-scoring examples. It discusses the scoring cost and several ways that cost can be reduced or justified.

A.1. ASK-LLM Sampling

  • ASK-LLM scores each training sample with a model and selects the k examples with the highest scores, without replacement.The algorithm uses the model’s score for “yes” given a prompt made from each sample.Source: §A.1 · p. 14
  • ASK-LLM scoring requires one inference call per training sample, but the authors argue its cost can be amortized across pre-training runs.Source: §A.1 · p. 14
  • The authors say ASK-LLM requires only forward passes over the dataset, unlike model training’s repeated forward and backward passes.They also contrast it with gradient-based curation techniques that require backward passes.Source: §A.1 · p. 14
  • ASK-LLM scoring can use memory-efficient quantized LLM inference setups, and advances in efficient inference can reduce its amortized cost.Source: §A.1 · p. 14
  • Quality scoring can be parallelized across small independent inference resources, and the authors say it uses less compute than teacher-student distillation.ASK-LLM requires a score for “yes” given a prompt, whereas the described distillation setup requires a larger teacher’s softmax predictions for each training token.Source: §A.1 · p. 14

A.2. DENSITY Sampling

DENSITY estimates local density in the model’s latent space and uses inverse-propensity sampling to diversify selections. Its two-pass procedure has stronger theoretical guarantees than the prior cluster-based approach, while avoiding clustering overhead.

A.2. DENSITY Sampling

  • DENSITY approximates local density with kernel sums and selects examples with probability proportional to their inverse scores.The described IPS procedure estimates scores with a KDE sketch built from embeddings.Source: §A.2 · pp. 14–15
  • DENSITY estimates density in the model’s latent space rather than using Jaccard distances over n-grams, which performed no better than random sampling for text.The method uses a positive-definite LSH kernel and a sketch-based consistent estimator of the kernel sum.Source: §A.2 · pp. 14–15
  • The two-pass procedure changes the algorithm’s sampling behavior and permits stronger theoretical guarantees than the earlier method’s cluster-population convergence.The earlier guarantee required strong assumptions about cluster structure and yielded only weak convergence for some diversity measures.Source: §A.2 · pp. 14–15
  • Theorem A.2 shows that inverse-propensity sampling from DENSITY scores converges to a uniform distribution over the input space in latent LLM representation space.The result follows from a consistent estimate of the kernel sum and inverse-density weighting.Source: §A.2 · p. 15
  • Scoring 364M embeddings required 80MB of memory and two linear dataset passes, while KDE uses two hyperparameters and showed little variation across tested choices.The compared clustering methods require choices including cluster count, optimizer, objective, and per-cluster sampling or deduplication settings.Source: §A.2 · p. 16

B. Data-curation Techniques

The appendix describes random, density-based, clustering-based, and quality-filtering approaches used in the data-curation comparisons. The methods include embedding-based coverage samplers and model-scored quality filters.

B.1. Random Sampling

  • Random sampling selects training examples uniformly at random and is a standard baseline with prior strong results in data-curation applications.Its stated rationale includes unbiased sampling.Source: §B.1 · p. 16

B.2. DENSITY Sampling

  • DENSITY sampling uses Sentence-T5-Base embeddings, PStable hashing, and a [1,000 × 20,000] sketch matrix.The embedding model’s contrastive training is cited as supporting distances among its 768-dimensional embeddings.Source: §B.2 · p. 16

B.3. SemDeDup

  • SemDeDup clusters Sentence-T5-Base embeddings into 10,000 groups and performs semantic deduplication within clusters to maximize coverage.It selects among nearby examples in the original dataset’s embedding space.Source: §B.3 · p. 16

B.4. SSL Prototypes

  • SSL Prototypes removes points closest to their assigned cluster centroids, using the embeddings and clustering shared with SemDeDup.The method is presented as removing prototypical points.Source: §B.4 · p. 17

B.5. Perplexity Filtering

  • Perplexity filtering prioritizes examples with lower proxy-language-model perplexity and was tested with T5 models and intermediate T5-Large checkpoints.The tested scoring models included T5-Small, Base, Large, XL, XXL, and T5-Large checkpoints from 20k to 700k steps.Source: §B.5 · p. 17

B.6. ASK-LLM Sampling

  • ASK-LLM uses instruction-tuned Flan-T5 models to score examples through the reasoning capabilities of instruction-tuned models.The scoring models ranged from Flan-T5-Small through Flan-T5-XXL.Source: §B.6 · p. 17

C. Downstream Evaluation Tasks

The appendix specifies the evaluation measures and benchmarks used to assess data curation. It also describes how quality-score distributions are constructed for several samplers.

C.1. Perplexity

  • Perplexity is the exponentiated average negative log-likelihood per sequence, measured on C4’s random validation sample, which may be lower quality and less representative of true model quality.The authors therefore describe this validation-set metric as a less reliable indicator of true model quality.Source: §C.1 · p. 17

C.2. HQ Perplexity

  • The HQ Perplexity validation set combines English wiki40b, realnews and webtext subsets of C4, and LM1B news commentary.It is constructed from non-web-scrape sources as an inexpensive metric intended to align better with model quality than C4 validation perplexity.Source: §C.2 · p. 17

C.3. GLUE

  • GLUE reports the average score across eleven tasks after fine-tuning on the concatenated training sets.The passage identifies GLUE as a natural-language-understanding meta-benchmark.Source: §C.3 · p. 17

C.4. SuperGLUE

  • SuperGLUE reports the average score across its tasks after fine-tuning on their concatenated training set.The passage describes it as a harder natural-language-understanding meta-benchmark than GLUE.Source: §C.4 · p. 17

C.5. CNN/DM

  • CNN/DM evaluates abstractive summarization using ROUGE-2 after fine-tuning on its training set.The evaluation follows the original T5 setting.Source: §C.5 · p. 17

C.6. SQuAD

  • SQuAD evaluates fine-tuned question answering with exact match as the metric.The passage describes SQuAD as a dataset used to evaluate language-model question answering.Source: §C.6 · p. 17

C.7. FLAN Instruction Tuning

  • FLAN instruction tuning is evaluated on MMLU, BBH, reasoning, and question-answering suites, with a macro-average across the four suites.The evaluation uses a selected checkpoint based on average performance across all tasks, then reports individual task performance on that checkpoint.Source: §C.7 · p. 18

D.1. (Figure 9) Quality-score Distribution for Different Samplers

  • Figure 9 plots normalized data-quality scores, with higher scores representing better quality, for DENSITY, Prototypes, SemDeDup, perplexity filtering, and ASK-LLM.The plotted values respectively represent KDE likelihood, negated centroid cosine similarity, negated within-cluster maximum cosine similarity, negated perplexity, and the log probability of “yes.”Source: §D.1 · p. 18

E.2. Low-quality Samples Identified by ASK-LLM

The section examines examples that ASK-LLM scoring models judge poorly and reports that the sampler flags harmful and non-contextual material without an apparent length bias.

E.2. Low-quality Samples Identified by ASK-LLM

  • ASK-LLM does not seem to have a length bias when identifying bad examples.Source: §E.2 · p. 34
  • ASK-LLM filters hateful or toxic examples that might hurt LLM training.Source: §E.2 · p. 34
  • ASK-LLM rejects non-contextual samples, including questions without answers and repeated nonsensical content, which perplexity filtering handles poorly because such examples can have low perplexity.Source: §E.2 · p. 34

E.3. Increasing-quality Samples Identified by ASK-LLM

This section presents qualitative examples of increasing- and decreasing-quality samples identified by ASK-LLM, with comparisons across scoring-model sizes and against perplexity filtering. The excerpts include promotional, technical, and product-copy text.

E.3. Increasing-quality Samples Identified by ASK-LLM

  • Larger ASK-LLM scoring models identify samples containing tail-end knowledge, such as rare world-events and named entities.This conclusion comes from samples Flan-T5-Small judged low quality but Flan-T5-XXL did not.Source: §E.3 · p. 37
  • The increasing-quality trend from Flan-T5-Small to Flan-T5-XXL is not correlated with quality-scoring model size in perplexity filtering.The passage contrasts ASK-LLM scoring-model disagreement with perplexity filtering.Source: §E.3 · p. 37

E.4. Decreasing-quality Samples Identified by ASK-LLM

  • Smaller ASK-LLM quality-scoring models sometimes mislabel non-informative samples, including content that is non-informative or repeated.This conclusion comes from samples Flan-T5-XXL judged low quality but Flan-T5-Small did not.Source: §E.4 · p. 39
  • The decreasing-quality trend from Flan-T5-Small to Flan-T5-XXL is not correlated with quality-scoring model size in perplexity filtering.The passage contrasts ASK-LLM scoring-model disagreement with perplexity filtering.Source: §E.4 · p. 39
  • The excerpts include plumbing advertisements that repeatedly target “Plumber Leopold” and hot-water repair searches.The copy repeats phrases and includes awkward wording such as “know hot to diagnose problems.”Source: §E.4 · p. 43
  • Example 30 labels estimated data quality as a percentile, with higher values better, and names ASK-LLM and perplexity filtering.The accompanying labels show model sizes Small, Base, Large, XL, and XXL, but provide no readable comparison values.Source: §E.4 · p. 43
  • Another excerpt consists of technical release-note text about TLS settings, capture points, supported databases, and an upgrade.The fragments include configuration instructions and a release-specific statement that no special upgrade procedure is required.Source: §E.4 · p. 43
  • Product-copy excerpts combine repeated display-stand keywords with awkward, garbled statements about quality control and customer service.The text also shifts incongruously to wigs, returns, and repair services.Source: §E.4 · p. 44

Related papers

Loading 2402.09668v1…