PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Data Selection for Language Models

Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, William Yang Wang

arXiv:2402.16827v3cs.CLcs.LG

TL;DR

Large language models use massive datasets across training stages, but data-selection goals vary with each stage’s objectives. This survey unifies existing methods under a probabilistic framework, develops a taxonomy, and identifies current practices and future research directions, while noting that filtering can introduce bias and data mixing creates domain trade-offs.

  • Problem

    Data selection is important across language-model training stages because each stage has different objectives and therefore different selection goals.

  • Method

    The survey organizes data-selection methods through a probabilistic framework centered on each method’s utility function and selection mechanism.

  • Results

    The survey provides a unified framework and taxonomy for comparing diverse data-selection methods and describes current best practices and research directions.

  • Takeaways & Limitations

    The review offers researchers a consolidated account of data-selection practices and highlights opportunities for further innovation and improvement.

  • Takeaways & Limitations

    Data mixing has an inherent trade-off: increasing one domain’s proportion decreases the relative representation of others, potentially harming underrepresented domains.

Abstract

from arXiv · show

A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may not be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount of training required. Data selection methods aim to determine which candidate data points to include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection research. Consequently, knowledge of effective data selection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies. To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related research areas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing an entry point for new and established researchers. Additionally, throughout this review we draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research.

1 Introduction

Data selection has become increasingly important for language models because training spans multiple stages with different objectives, while the literature remains difficult to access and synthesize. This survey organizes existing methods and provides a representative resource for understanding current practices.

  • Motivation: Language models use data selection across pretraining, instruction-tuning, alignment, in-context learning, and task-specific fine-tuning, whose objectives differ.The survey focuses primarily on pretraining while covering other training regimes more concisely.
  • Survey contribution: The survey unifies data selection methods through a conceptual framework and taxonomy centered on utility functions and selection mechanisms.The framework supports comparison under a probabilistic viewpoint, with emphasis on model pretraining.
  • Survey contribution: The survey provides a collected resource describing current practices and considerations for selecting language-model training data.It also identifies future research directions and cites methods not covered in depth.
  • Motivation: Data selection designs datasets from raw data to optimize objectives such as model performance, efficiency, evaluation integrity, bias, and toxicity.The pipeline may clean, filter, and mix data before training or evaluation.

2 A Taxonomy for Data Selection

The taxonomy treats data selection as constructing datasets from candidate points under explicit objectives, using utilities and selection mechanisms to control what data enters training or evaluation. It organizes methods by their goals, outputs, representations, and training-stage context.

  • Dataset units: A data point is an ordered collection of tokens used as one training or evaluation sample, and long documents may be split into multiple points.Data-point characteristics include simple statistics and distributed representations.
  • Foundations: Data selection creates a dataset from candidate points by filtering, cleaning, and selecting data to optimize an objective for a trained model.Selection methods may be composed sequentially, for example by filtering, cleaning, and mixing.
  • Unified framework: A utility function assigns each data point a real-valued score, while a selection mechanism uses that output to decide inclusion or repetition.Mechanisms can be deterministic indicators or probability-based selection rules.
  • Taxonomy dimensions: Distribution matching selects data resembling a target distribution, whereas diversification prioritizes heterogeneity and removes redundancies while preserving coverage.Target distributions may represent quality, language, or a domain; diversification operates in a representation space where similarity is measured.
  • Training stage: Pretraining receives greater emphasis because its candidate pools are larger and information about its data selection methods is more limited.Instruction-tuning can generally afford more expensive selection methods than pretraining.

3 Data Selection for Pretraining

Pretraining data selection uses pipelined filters to balance model performance, evaluation integrity, selection efficiency, and dataset scale. Language filtering commonly relies on fast classifiers or URL signals, while heuristic filtering remains difficult to validate systematically.

  • Overview: Pretraining requires massive text datasets, so pipelines commonly apply multiple filters to remove undesirable data before training.Different works may use different filters and ordering.
  • Language filtering: Language-filter thresholds trade off retained data quantity against lower-quality inclusions and possible bias against minority dialects.More accurate recurrent- or transformer-based detectors exist but require higher computational cost.
  • Language filtering: Language filtering is an early selection step, with classifier-based methods common for English-only and multilingual datasets.URL-based methods can be more reliable for very low-resource languages.
  • Language filtering: Code-language filtering is often performed by matching approved filename extensions, which works well for datasets containing file names.This approach has been used to filter data from 24 programming languages.
  • Open directions: Future work could identify code embedded in natural-language documents and develop direct measures of heuristic-filter quality to accelerate iteration.Stochastic selection may also let more data pass early filters for later filtering stages.
  • Heuristic approaches: Heuristic filters are expensive to validate because effectiveness requires manual inspection or model training and evaluation.The literature also lacks systematic comparisons establishing whether current heuristic best practices are optimal.

3.3 Data Quality

Data quality filtering selects documents resembling reference corpora through classifier- or perplexity-based utility scores, but its benefits and fairness remain unsettled. Reference-corpus bias, classifier bias, and computational cost complicate decisions about when quality filtering is appropriate.

  • Methods: Quality filtering selects data resembling a known high-quality reference corpus, using classifier-based or perplexity-based utility functions.Classifier-based filtering is more commonly used.
  • Methods: Classifier-based filtering compares candidate documents with high-quality corpora, whereas perplexity-based filtering scores candidates using a language model trained on those corpora.Both approaches require utility functions that are relatively cheap to compute compared with broader fuzzy matching.
  • Selection mechanisms: Stochastic selection can retain some documents outside the reference-corpus distribution while keeping the final corpus mostly high-scoring.This addresses the possibility that reference corpora omit some high-quality data.
  • Challenges: Quality filters can encode demographic and dialect biases because reference corpora and quality judgments may not represent all communities.A filter trained on Wikipedia, books, and newswire preferred high-school articles from wealthier, more-educated, and urban areas.
  • Open questions: Whether quality filtering is beneficial remains undecided, since performant models have been trained without quality filters.Some recent datasets instead allocate more compute to stringent deduplication.

3.4 Domain-Specific Selection

Domain-specific selection finds auxiliary data resembling a desired domain, commonly using Moore-Lewis-style utility estimates. The survey highlights computational costs, distributional skew, and perplexity-based alternatives.

  • Domain-specific selection aims to find auxiliary data whose distribution resembles a specific in-domain target such as medicine or law.
  • Moore-Lewis methods estimate utility from the probability ratio between in-domain and general-purpose language models, expressed in the log domain.
  • Modern language-model scale makes naive utility estimation costly, so efficient n-gram models are typically used instead.
  • Moore-Lewis-derived methods remain the gold standard, while datamodels and other attribution methods may require higher computational resources.
  • Domain-specific methods can skew distributions toward common modes in the in-domain data, motivating safeguards such as deduplication.
  • Perplexity-based methods require only an in-domain model and could support repeated selection rounds, although their performance may be lower.

3.5 Data Deduplication

Data deduplication removes repeated or near-repeated content through increasingly expensive matching methods. It is important for pretraining, but each approach involves accuracy, coherence, computational, or evidence limitations.

  • Internet-derived corpora contain duplicate and near-duplicate documents, so deduplication is generally preferred for pretraining when the evaluation distribution is unknown.
  • Deduplication methods use URLs, hashing, string metrics, or learned representations to identify exact or approximate matches.
  • URL matching is a cheap first-stage filter, while hash-based methods find exact matches but may accidentally remove nonmatching documents because of collisions.
  • Approximate matching commonly follows exact deduplication and includes MinHash-family methods, which compare shingled document representations using similarity thresholds.
  • Model-based matching captures semantic similarity but is costly, using pretrained representations to cluster similar documents.
  • Hashing-based approximate deduplication can produce more false positives for long documents because shingle order is ignored.
  • Model-based deduplication has shown good results for data efficiency, but its benefits for performance-focused settings require further study.

3.7 Specialized Selection for Multilingual Models

Multilingual selection reuses general filtering methods but requires language-specific parameters, native-speaker oversight, and script-aware handling. Data mixing further determines domain composition, with no general-purpose weighting method established.

  • Specialized Selection for Multilingual Models: Many filtering methods transfer across languages, but parameters such as length cutoffs and language-identification thresholds must be adapted per language.
  • Specialized Selection for Multilingual Models: Chinese may require a smaller minimum-length cutoff than English because its characters carry higher information density.
  • Specialized Selection for Multilingual Models: Low-resource multilingual data often requires manual inspection and native-speaker involvement to verify language identification and usefulness.
  • Data Mixing: Data mixing assigns weights to domains, and online methods update those weights during training.
  • Data Mixing: Optimal domain weights may depend on the tokenizer, dataset, architecture, optimizer, and other training hyperparameters.
  • Data Mixing: Upweighting one domain necessarily downweights others, potentially harming performance on less represented domains.
  • Data Mixing: No manual, heuristic, empirical, or principled weighting approach has yet proven generally best across evaluations.

3.9 Current Best Practices and Landscape For Data Selection

The survey describes current data-selection practices across filtering, mixing, deduplication, toxicity removal, multilingual handling, and commonly used research datasets. It emphasizes that best practices remain application-dependent and that automated data mixing lacks clear standards.

  • Best practices vary by evaluation protocol and use case, so no single selection strategy is best for all language-model applications.
  • Filtering: Filtering pipelines commonly begin with language identification and fast heuristics based on document length or sentence count.
  • Filtering: Quality filtering is more useful for general web data than for narrowly sourced domains, and stochastic filtering can reduce quality-filter bias.
  • Filtering: Deduplication benefits every pretraining selection pipeline, ranging from URL-based removal to hash-based approximate or exact matching.
  • Filtering: Multilingual selection generally reuses other filtering methods but should adjust hyperparameters by language and involve native speakers in quality decisions.
  • Research datasets: C4, The Pile, and RedPajama are prominent research datasets, differing in size, source composition, and suitability for benchmarking or data-mixing studies.

4 Data Selection for Instruction-Tuning and Multitask Training

Instruction-tuning and multitask selection address the mismatch between pretraining and downstream use by broadening and balancing supervised data. Recent methods increasingly target quality, diversity, and efficiency, while governance and legal concerns also shape selection.

  • Instruction-tuning and multitask training use supervised examples to broaden downstream capabilities beyond the pretrained data distribution.
  • Selection objectives include model performance and evaluation integrity, while synthetic-data workflows also remove system-specific disclaimers and refusals.
  • Diversification by scaling tasks and datasets: Task and dataset scaling, along with varied prompt templates and augmentation, increase training diversity and can improve generalization.
  • Manual and heuristic-based diversification: Capping examples from large sub-datasets prevents imbalance and supports greater diversity, whereas manual mixture searches can be costly.
  • Manual and heuristic-based diversification: No available data mixture is best across all evaluations, motivating principled selection methods for instruction-tuning.
  • Data efficiency: Model-based filtering and utility functions can preserve similar quality with less computation, including one approach using 2/3 the compute on a reduced FLAN dataset.
  • Data governance increasingly matters: 61% of 2023 instruction-tuning datasets had non-commercial restrictions, compared with no more than 30% in the prior decade.

5 Data Selection for Preference Fine-tuning: Alignment

Preference fine-tuning selects data using manual filters, model-based evaluations, and reward-based weighting to shape helpfulness and harmlessness. The field remains nascent because preference-training goals are still underspecified.

  • Alignment data selection prioritizes reducing bias and toxicity alongside model performance.
  • Preference fine-tuning commonly uses prompt–chosen-response–rejected-response trios, although some methods require only one preferred or rejected continuation.
  • RLHF and RLAIF independently train reward models that assign scalar utility for selecting or weighting samples, unlike DPO methods.
  • Current PreFT selection primarily uses manual filtering, model-based evaluation, and reward-model re-weighting such as rejection sampling.
  • Reward model re-weighting: Reward models can select the best of n generated candidates, and rejection sampling can continue instruction-tuning on the highest-ranked subset.
  • Reward model re-weighting: Synthetic-data selection requires diversity to reduce mode-collapse risk, while self-rewarded models may be biased toward their own rewards.

6 Data Selection for In-Context Learning

In-context learning selection chooses and orders demonstrations under prompt-space constraints, using retrieval, sampling, graph-based methods, or selective annotation. The central objectives are data efficiency and model performance.

  • In-context learning is sensitive to both the choice and ordering of demonstrations, making example selection important without fine-tuning.
  • Demonstration reordering: Demonstration reordering can use predicted-label entropy as a utility function because ordering significantly affects in-context performance.
  • Demonstration selection: Retrieval-based methods tailor demonstrations to each test input using similarity, predicted labels, one-shot utility, or cross-entropy differences.
  • Demonstration selection: Sampling-based methods select demonstrations through uniform sampling, validation-performance prediction, influence estimates, or informativeness and diversity scores.
  • Selective annotation: Selective annotation chooses informative unlabeled examples under a fixed budget, reducing human annotation effort before selecting demonstrations.
  • Graph-based approaches quantify example influence or combine diversity and uncertainty, with IDEAL reported to outperform vote-k at lower time consumption.

7 Data Selection for Task-specific Fine-tuning

Task-specific fine-tuning shares many data-selection methods with other training settings but is easier to evaluate because its target distribution and success criteria are narrower. Methods target distribution matching, auxiliary-data selection, data efficiency, diversity, and robustness.

  • Task-specific fine-tuning is easier to evaluate because its target distribution, use cases, and success definition are narrower.
  • Data selection can match a narrow target distribution, which is especially useful in data-limited settings such as few-shot learning.
  • Selecting from auxiliary data to improve generalization: Auxiliary-data methods select source tasks or domains to improve generalization to the target distribution, using task similarity or learned utility functions.
  • Selecting from auxiliary data to improve generalization: Similarity between auxiliary and target label spaces can serve as a utility heuristic, but only when the target task structure is sufficiently clear.
  • Considerations for task-specific fine-tuning: Task-specific selection also seeks greater data efficiency, sample diversity, and reduced subgroup bias, with expensive methods more plausible on small datasets.
  • Improving task-specific data efficiency: Selecting difficult examples can improve model performance while reducing dataset size by up to 60%.

8 Data Selection for Other Domains

Data selection methods extend beyond language models to vision-language and other domains, where they address filtering, mixing, efficiency, fairness, and robustness. Evidence includes substantial filtering, dataset-specific robustness, and limitations of some adaptive methods.

  • Data selection is active in computer vision, vision-language, and broader machine learning, with methods often serving data-efficiency, performance, and robustness goals.
  • Data selection for pretraining: Vision-language datasets apply language-style filtering ideas alongside modality-specific procedures such as image deduplication and resolution or aspect-ratio filtering.
  • Data selection for pretraining: World-model video datasets combine heuristic keyword filters with quality classifiers trained from 10,000 hand-labeled videos.
  • Data selection for pretraining: Heuristic filters are difficult to validate and require slow iteration, motivating trained filtering networks as an alternative.
  • Data mixing: Dynamic difficulty sampling weights vision-language tasks according to their losses in an online data-mixing procedure.
  • Adaptive batch selection: Adaptive batch selection emphasizes uncertain or nonredundant examples, but RHO-Loss has limited NLP validation and has been shown to be computationally inefficient.
  • Bias, fairness, and robustness: Progressive Data Expansion improves worst-group robustness, while combining datasets does not necessarily improve overall robustness and can dilute component-specific robustness.

9 Related Topics

Related areas overlap with data selection but pursue distinct objectives, including cleaning, augmentation, curation, curriculum learning, active learning, distillation, coresets, and data valuation. These methods differ in whether they remove, synthesize, organize, schedule, or assess data.

  • Data cleaning: Data cleaning removes undesirable content within data points, whereas data selection can remove entire points from a dataset.
  • Dataset distillation and coreset selection: Dataset distillation synthesizes a small dataset intended to match full-data performance, while coreset selection chooses a subset from the original dataset.
  • Data attribution and valuation: Data valuation and attribution estimate how individual training points affect predictions, supporting error identification, influence measurement, and contamination analysis.
  • Data augmentation: Data augmentation generates modified data to improve coverage, but semantic divergence from the desired distribution can reduce performance.
  • Data curation: Data curation includes discovery, organization, integration, annotation, cleaning, storage, and maintenance, overlapping with but exceeding data selection.
  • Curriculum learning: Curriculum learning schedules when examples are trained according to increasing complexity, unlike data selection’s choice of training or evaluation points.

10 Discussion

The discussion emphasizes evaluation integrity, memorization tradeoffs, filtering bias, and practical auditing. Decontamination methods protect benchmark validity, while filtering must balance removal of undesirable content against harmful distributional effects.

  • 10.1 Test set decontamination: Web-scale training raises concerns that evaluation benchmarks may appear in training data, making automated decontamination important for evaluation integrity.
  • 10.1 Test set decontamination: Decontamination methods include n-gram overlap detection, searchable training-data indexes, membership testing, and canary strings embedded in benchmarks.
  • 10.1 Test set decontamination: Models trained on undisclosed data can be difficult to verify for test-set contamination, motivating statistical and probability-based detection methods.
  • 10.2 Tradeoffs between memorization and generalization: Memorization can preserve desirable facts and code syntax but can also expose personally identifiable information and other sensitive training content.
  • 10.2 Tradeoffs between memorization and generalization: Model capacity and sample repetition significantly increase memorization, providing controllable factors for model developers.
  • 10.3 There’s no free lunch for filtering: Blocklist and toxicity filtering can disproportionately remove minority or marginalized text and may worsen evaluation loss.
  • 10.3 There’s no free lunch for filtering: Effective filtering requires improved precision and recall because removing undesirable content can also remove desirable data.
  • Practical selection pipelines should begin with dataset exploration and auditing, then compose filtering, cleaning, mixing, and other mechanisms as needed.

11 Future Directions: Challenges and Opportunities

The survey identifies four directions for accelerating data-selection research: scaling down experiments, directly evaluating data, creating shared benchmarks, and open-sourcing tools and practices. It also highlights clearer objectives and richer selection methods as opportunities for improving research and training data.

  • Scaling down: Scaling down datasets and models could broaden participation and speed iteration, but whether small-model results transfer to larger models remains uncertain.Training with 745x less data was reported to lose 1.9% on GLUE and 2.5% on Super-Natural Instructions; another study distilled benchmarks to 100 data points within 2% accuracy.
  • Metrics that directly evaluate data: Metrics that directly evaluate selected data could reduce reliance on costly model training, shortening development time and lowering barriers to entry.Such metrics could increase exploratory work and enable lower-resourced organizations and individuals to contribute to large-model development.
  • Data-centric benchmarks and challenges: Data-centric benchmarks and challenges could make methods comparable across settings, lower entry barriers, and encourage coordinated progress.Current studies often vary in dataset size, model architecture, and model size, limiting direct comparisons; appropriate toy settings remain unsettled.
  • Open-sourcing tools and best practices: Open-sourcing tools alongside emerging best practices could reduce startup overhead and let researchers focus on specific pipeline components.The survey presents open tools as a way to accelerate research progress as the field develops in the open.
  • Clarifying goals: Clearly specifying the target goal helps determine which selection methods are appropriate, with success less specific for preference finetuning than task-specific fine-tuning.The framework ties method suitability to the properties of the desired target distribution.
  • Beyond heuristic filtering: Improving pretraining data may require applying more expensive model-based, reward-based, synthetic-data, or additive selection techniques as compute becomes cheaper.The survey contrasts these approaches with the predominantly removal-oriented methods currently used for pretraining.

12 Conclusion

The survey reviews data selection research in depth and introduces a unified framework for comparing its methods. It uses that framework to describe current practices and identify future research directions, while emphasizing that substantial room for innovation remains.

  • 12 Conclusion: The paper compiles an in-depth review and unified framework for considering and comparing diverse data-selection methods.The framework supports descriptions of current best practices and considerations for selecting language-model training data.
  • 12 Conclusion: The framework also helps identify potentially fruitful future research directions, while the survey concludes that data selection still has substantial room for improvement.These directions are highlighted throughout the paper rather than presented as a completed solution.

A Heuristic Filtering Details

The appendix catalogs heuristic filters used in major language-model corpora, covering length, character composition, punctuation, boilerplate, language identification, and document-quality signals. It also records cases where filtering rules or thresholds are unspecified.

  • RefinedWeb and MassiveText: RefinedWeb removes lines matching patterns such as short examples, while MassiveText applies document-level length, word-length, symbol, formatting, alphabetic-content, and stop-word criteria.MassiveText requires documents to satisfy all listed conditions.
  • C4: C4 removes lines lacking terminal punctuation, containing fewer than five words or three sentences, or matching markers for JavaScript, profanity, placeholders, code, and policy boilerplate.These heuristics target short, non-natural-language, or recurring boilerplate content.
  • mC4: mC4 adapts filtering to multilingual data using line-length requirements, language-confidence thresholds, and additional numerical or content checks.It removes pages with fewer than three lines containing at least 200 characters and pages whose primary-language confidence is below 70%.
  • ROOTS Corpus: ROOTS applies source-specific code and text filters, including file-size, alphabetic-content, line-length, repetition, special-character, closed-class-word, and language-identification measures.The survey notes that ROOTS describes eight filters but does not specify their selection thresholds.
  • Other corpora: Llama, Pile, and related corpora combine domain-specific extraction and cleanup, but several heuristics lack reproducible details or are explicitly reported as unspecified.Examples include missing thresholds for uppercase-line and counter rules, unspecified Pile heuristics, and undocumented administrative-boilerplate removal.
Loading 2402.16827v3…