A Survey on Data Selection for Language Models

• The probabilistic view treats an optimal dataset as one matching the distribution on which the model will be evaluated, while selection also targets costs, evaluation integrity, bias, and toxicity.These goals include reducing dataset size, removing suspected evaluation data, and reducing undesirable behaviors.Source: §1 · p. 4
• Data selection matters across pretraining, instruction-tuning, alignment, and other language-model stages, but its objectives vary with each training stage.Pretraining commonly filters massive corpora for high-quality data, whereas task fine-tuning may select auxiliary samples that provide useful learning signals.Source: §1 · p. 4, Figure 1 · p. 5
• The survey’s framework represents every data selection method with a utility function and a selection mechanism, enabling a taxonomy and comparison of existing approaches.The survey focuses on model pretraining while also covering other training regimes and domains to identify future research directions.Source: §1 · p. 4
• Because exhaustive coverage would be prohibitively long, the survey discusses representative methods in depth and cites additional literature for methods not covered fully.Its stated goals are to collect current best practices and define a unified view that supports promising future research directions.Source: §1 · p. 4
• The language-model data pipeline cleans, filters, and mixes raw data into a final dataset for training or evaluation, with objectives spanning performance, efficiency, evaluation integrity, and bias or toxicity reduction.The relative importance of these objectives depends on the learning stage; pretraining may prioritize data and selection efficiency, while later stages have different concerns.Source: Figure 1 · p. 5

2.1 Background and Motivation of Data Selection

• A data point is an ordered token collection used for training or evaluation, while its characteristics describe its position among possible data points.Long documents may be split into multiple data points, and characteristics can include simple statistics or distributed representations.Source: §2.1 · pp. 5–6

2.2 A Unified Conceptual Framework for Data Selection

• Data selection filters, cleans, and selects candidate data points to create a dataset that optimizes an objective for a model trained on it.Selection functions may compose filtering, cleaning, and mixing components to adjust individual data-point characteristics and dataset distributions toward desirable regions.Source: §2.2 · p. 6
• Selection methods use utility functions to score data points and selection mechanisms to include, repeat, or exclude them, with sensitivity controlling threshold-based decisions.Utilities map data points to real numbers, while mechanisms may be indicators or probability functions.Source: §2.2 · p. 6

2.3.1 Distribution Matching vs. Diversification

• Distribution matching selects data resembling a target distribution, whereas diversification removes redundancies to broaden coverage and can improve efficiency, memorization, bias, and robustness.Matching targets may include high-quality data, languages, or domains; diversification measures similarity in a representation space.Source: §2.3.1 · p. 7, Figure 2 · p. 7

2.3.2 Altering the Dataset vs. Data Point

• Dataset-altering methods change data-point frequencies, whereas data-point-altering methods modify content within examples to better match desirable token distributions.Dataset alteration can filter, maintain, or oversample examples; data-point alteration can remove lines or text chunks such as HTML tags.Source: §2.3.2 · p. 8

2.3.3 Output Space: Binary vs. Natural Number Selection

• Binary-output methods are filtering, while natural-number-output methods are data mixing; both can be combined by filtering first and then mixing.Filtering usually removes undesirable points, whereas mixing prioritizes high-utility subsets by changing their sampling frequency.Source: §2.3.3 · p. 8

2.3.4 Training Stage

• Data selection applies across pretraining, instruction-tuning, alignment, in-context learning, and task-specific fine-tuning, whose goals and candidate-set sizes require different mechanisms.Pretraining has more candidates and less-defined targets than instruction-tuning, motivating efficient methods; information on pretraining selection is the most limited.Source: §2.3.4 · p. 8, Table 1 · p. 9

3 Data Selection for Pretraining

• Pretraining pipelines commonly apply multiple filters to massive text corpora, with ordering varying by dataset and use case while prioritizing performance, evaluation integrity, and selection efficiency.The survey presents language filtering and heuristic methods as components of broader pretraining data-selection pipelines.Source: §3 · p. 9, Figure 3 · p. 9, Figure 4 · p. 10

3.1 Language Filtering

• Language filtering selects documents in desired natural or coding languages, while multilingual and code-focused datasets also track the quantity of data from each language.Classifier-based methods are the norm, though URL-based selection can be more reliable for some low-resource languages.Source: §3.1 · pp. 9, 11
• FastText and other character n-gram classifiers provide fast language utilities, but thresholds balance retained quantity against non-target text, detector bias, and computational cost.FastText can process 1,000 documents per second on one CPU core, while multilingual filtering may require language-specific thresholds and remains weak for extremely low-resource languages.Source: §3.1 · pp. 9–11
• Code-language filtering is usually simple filename-extension matching, but current methods do not identify code embedded within natural-language documents.This leaves potential interleaved code and natural-language data underexploited in domains such as Stack Exchange.Source: §3.1 · p. 11

3.2 Heuristic Approaches

• Heuristic filters efficiently remove undesirable web text using item counts, repetition counts, existence, ratios, or statistics, but rely on surface properties rather than document content.They may remove desirable data, and their effectiveness is expensive to validate through manual inspection or model training and evaluation.Source: §3.2 · pp. 12–14
• Heuristics should reflect the target domain: code filters can constrain file length, alphabetic content, and line structure, whereas those rules may remove desirable natural-language data.Heuristics can also remove individual lines or HTML to reshape document characteristics instead of discarding entire data points.Source: §3.2 · pp. 13–14
• Deterministic thresholds dominate heuristic filtering, but stochastic selection could preserve some low-utility data and better accommodate long-tailed evaluation characteristics.The appropriate stochastic settings remain an open research question, particularly because later filters might remove retained undesirable data.Source: §3.2 · pp. 13–14
• Common heuristic practices have changed little, and no systematic study has compared the many possible heuristics, leaving progress and optimality uncertain.Data auditing and direct quality measurement could accelerate filter design and validation, which currently requires slow manual or model-based evaluation.Source: §3.2 · p. 14

Utility functions

• Classifier-based methods train lightweight models to score documents by similarity to high-quality reference corpora, while perplexity-based methods retain text with lower reference-model perplexity.Modern language models could model reference distributions better but would substantially increase computation; GPT-4-based ratings can also be costly and inherit system biases.Source: §3.3 · p. 15

Selection mechanisms

• Stochastic selection can retain mostly high-scoring documents while preserving some data outside the reference distribution.When utility is represented as an importance weight, examples can instead be sampled according to those weights, efficiently using the Gumbel top-k trick.Source: §3.3 · pp. 15–16

Potential challenges

• Quality filters can encode demographic, dialectal, and socioeconomic preferences rather than neutral judgments, so they must be designed to cover the desired populations.A Wikipedia-, books-, and newswire-based filter preferred high-school articles from wealthier, more-educated, and urban areas (Gururangan et al., 2022).Source: §3.3 · p. 16

Open questions

• Whether quality filtering improves pretraining remains undecided, since performant models have been trained without it and the conditions where it helps are not understood.MassiveText (Rae et al., 2021), RefinedWeb, and Dolma (Soldaini et al., 2024) are examples of works that forego quality filtering; RefinedWeb instead emphasizes stringent deduplication.Source: §3.3 · p. 16

Methods

• Classifier- and perplexity-based filtering are the main approaches, with classifier-based filtering being more popular.Both approaches compare candidate data with a known high-quality reference distribution, while classifier scoring commonly uses lightweight n-gram or hashed models.Source: §3.3 · pp. 15–16

Domain-Specific Selection

• Domain-specific selection finds auxiliary data whose distribution most resembles available in-domain data, such as medicine or law.These methods generally assume access to both in-domain data and auxiliary sources.Source: §3.4 · pp. 17–18

Common utility functions

• Moore-Lewis selection estimates domain relevance with the log probability ratio log(P(x(i)|I)) −log(P(x(i)|N)), using cross-entropy losses from in-domain and general-purpose models.The method derives this utility from estimates of P(x(i)|I) and P(x(i)|N).Source: §3.4 · p. 17

Selection mechanisms and thresholds

• Selection can be deterministic or stochastic: thresholds use held-out in-domain data, whereas importance-weight resampling can preserve minority target subpopulations.DSIR exemplifies stochastic selection through importance-weighted resampling.Source: §3.4 · p. 18

Comparison to data quality filters

• Domain-specific filtering generalizes quality filtering, but unlike proxy-based quality filtering, it can directly measure distributional match when sufficient in-domain data exists.Quality filtering therefore requires stochasticity because its reference corpus may not represent the desired distribution.Source: §3.4 · p. 18

Challenges for domain-specific selection

• Using only in-domain data can overrepresent common modes, whereas Moore-Lewis methods mitigate this bias; datamodels may improve high-ratio filtering but require substantially more compute.Engstrom et al. (2024) report improved performance over DSIR below 25% of the original dataset size, with significantly higher compute.Source: §3.4 · pp. 17–18

Common utility functions

• Moore-Lewis-style methods remain the most popular approach for domain-specific filtering, requiring one in-domain model and one general-purpose model.Source: §3.4 · p. 18

Domain-Specific Selection Summary & Ideas

• Perplexity-based selection using only an in-domain model could improve efficiency and support repeated bootstrapping rounds, though it may underperform Moore-Lewis methods.Data-attribution and valuation methods are also promising but generally have higher computational requirements.Source: §3.4 · pp. 17–18

3.5 Data Deduplication

• Deduplication is generally preferred for pretraining because it increases distributional coverage, reduces redundancy and training time, and can limit memorization, especially as model scale grows.Keeping near duplicates can still benefit high-value regions, while filtering duplicates sometimes improves downstream accuracy.Source: §3.5 · p. 19
• Approximate hashing groups document shingles and compares aggregated features, with Rae et al. (2021) reporting better model performance from more aggressive deduplication than Gao et al. (2020).Rae et al. (2021) used 13-grams, 450 hash functions, and a Jaccard threshold of 0.8 for MassiveText.Source: §3.5 · p. 20
• Model-based deduplication can improve data efficiency by removing semantic and prototypical similarity, but evidence is limited to noisy CommonCrawl data and may benefit from stronger embedding models.D4 combines MinHash, SemDeDup, K-means clustering, and SSL prototypes; the review calls for evaluation on cleaner datasets and with clustering-specific embeddings.Source: §3.5 · pp. 20–21
• Deduplication must account for false positives and span-handling choices, because approximate hashing can overmatch long documents and removing duplicated spans may affect document coherence.Laurençon et al. (2022) avoid removing near-duplicate documents longer than 6000 characters, while Penedo et al. (2023) find span removal can outperform dropping entire documents.Source: §3.5 · p. 20
• Deduplication pipelines commonly combine URL, hashing, string-metric, and model-representation methods, often applying cheap exact filters before more expensive approximate matching.Exact methods include URL matching, hashes, and Bloom filters; approximate methods include MinHashLSH, SimHash, edit similarity, and semantic embeddings.Source: §3.5 · pp. 19–21

3.6 Filtering Toxic and Explicit Content

• Longpre et al. (2023c) find that Jigsaw’s Perspective API toxicity filters reduce toxic text generation but also reduce models’ ability to detect toxic content.Because bias and fairness filtering lacks transparency and can reflect conflicting ideologies and implications, filter selection should depend on the end application.Source: §3.6 · p. 21
• Toxicity and explicit-content filtering uses URL blacklists, lexicons, classifiers, perplexity models, and PII detection or obfuscation, but heuristic thresholds and requirements are often unclear.PII handling commonly detects phone numbers, email addresses, IP addresses, and secret keys with regular expressions or classifiers, then replaces them with special tokens.Source: §3.6 · pp. 21–22
• Limited evidence on filtering effects motivates research into distributional and behavioral impacts, mitigation strategies, alignment, and decoding-time alternatives.The review specifically identifies reducing negative filtering effects and controlling toxic or explicit behavior at later development or evaluation stages as useful directions.Source: §3.6 · p. 22

3.7 Specialized Selection for Multilingual Models

• Multilingual filtering methods often transfer across languages, but their parameters must be adapted and frequently tuned for each language.Length thresholds may need to be lower for information-dense languages such as Chinese, and language-identification removal thresholds should be language-specific and determined by native speakers.Source: §3.7 · p. 23
• Low-resource multilingual data requires manual quality inspection and native-speaker involvement because noisy language classifications correlate with language rarity.Ignoring this issue has produced unusable corpora, while translation services and anomaly checks can help non-speakers assess quality.Source: §3.7 · p. 23

3.8 Data Mixing

• Data mixing samples domains hierarchically by first drawing a domain according to weight α_i and then sampling uniformly within that domain.Each point in domain D_i receives utility α_i/|D_i|, so methods generally use mutually disjoint domains.Source: §3.8 · p. 24
• Data-mixing weights may be tuned against downstream performance, but this can require pretraining many models or searching across numerous weight combinations.GLaM uses smaller-model performance and domain sizes, while Gopher selected among seven weight combinations.Source: §3.8 · p. 24
• Offline methods learn static domain weights before training, whereas online methods update sampling weights during training using losses, interactions, scaling laws, or information-gain signals.Examples include DoReMi and DoGE offline, and Skill-it, ShearedLLaMA, ODM, and relative-loss updates online.Source: §3.8 · pp. 24–25
• Upweighting one domain necessarily downweights others, creating a trade-off that can reduce performance on less represented domains.Future methods could model domain interactions to identify domains that can be reduced without harming performance.Source: §3.8 · pp. 25–26
• Learned mixing weights may fail to transfer across tokenizers or training budgets, potentially causing excessive repetition and overfitting on smaller domains.Weights learned at 50B tokens could expose domains to 20 epochs when applied to 1T-token training.Source: §3.8 · p. 25
• No data-mixing strategy has proven generally best: weights may be manual, heuristic, empirically tuned, or principled, each succeeding under specific evaluations.Principled approaches have achieved state-of-the-art mixing but can be brittle across tokenizers and longer training.Source: §3.8 · p. 26

3.9.1 Best Practices

• Filtering pipelines generally begin with efficient, broad filters and progress toward specialized, more costly methods.Language identification commonly uses fastText, followed by fast heuristics such as word- and sentence-count thresholds from C4 and MassiveText.Source: §3.9.1 · p. 26
• Quality filtering is useful for large web corpora but less so for specific domains, while stochastic filters can reduce quality-filter bias.Quality classifiers distinguish random web data from known high-quality domains; no commonly used open-source models are identified, and prior methods are described by Brown et al. (2020b).Source: §3.9.1 · pp. 26–27
• Deduplication benefits every pretraining pipeline, progressing from URL and hash-based matching to more expensive semantic methods such as SemDeDup (Abbas et al., 2023).Toxic, explicit, and personally identifiable content is commonly removed using URL, domain, word-list, and regex filters such as Dolma (Soldaini et al., 2024). Multilingual selection may require language-specific hyperparameters and native-speaker review.Source: §3.9.1 · p. 27

3.9.2 Current Landscape of Datasets For Data Selection Research

• RedPajama-2 offers 30T tokens, precomputed quality signals, and scale from billions to trillions of tokens, making it a strong resource for controlled selection experiments.C4 contains 750Gb of web-crawled data and generally yields lower benchmark scores; The Pile contains 800Gb across varied domains and suits data-mixing research, while automated data mixing remains immature.Source: §3.9.2 · p. 27, §3.9.1 · p. 27

Scope and objectives

• Instruction-tuning and multitask training assume pretrained language capabilities and use diverse supervised data to support broad downstream classification and generation.Instruction-tuning formats examples as instruction–output pairs to encourage more controllable and helpful outputs.Source: §4 · p. 28

Diversification by scaling tasks and datasets

• Scaling tasks, datasets, and prompt templates improves generalization, but early selection largely followed repository availability and pressure to scale rather than principled criteria.Translation, input inversion, negative examples, and varied prompt formats further expand task and template diversity; easier tasks may contribute less.Source: §4 · p. 28

Manual and heuristic-based diversification

• Manual mixing and balancing limit overrepresented sub-datasets to preserve diversity, with Iyer et al. (2022) finding at most 512 examples per sub-dataset performed best across held-out tests.Manual grid searches over mixture proportions can improve performance but are costly, and no mixture is consistently best across evaluations.Source: §4 · pp. 28–29

Model-based diversification

• Model-generated conversation data, iterative task creation, and evol-instruct methods diversify prompts toward real-world domains and topics absent from existing collections.Synthetic datasets tend to emphasize longer responses and brainstorming, planning, explanation, and creativity, while filtering may remove system-specific refusals and disclaimers.Source: §4 · p. 29

Data efficiency

• Data-efficient selection increasingly prioritizes quality, diversity, difficulty, and complexity, with LIMA achieving strong results from only 1000 highly curated training instances.Automated and model-based methods score, filter, or actively select examples; one FLAN filtering approach reached similar quality with 2/3 the compute.Source: §4 · p. 29

Data governance and other concerns

• Instruction-tuning selection increasingly incorporates licensing, copyright, multilinguality, task, and domain constraints because dataset suitability is not determined by performance alone.Non-commercial restrictions rose to 61% of 2023 datasets, and provenance tools support filtering by licenses, terms, and dataset characteristics.Source: §4 · pp. 29–30

Alignment objectives

• Selection for alignment primarily targets model performance while reducing bias and toxicity.These objectives are summarized in the alignment data-selection overview.Source: Figure 6 · p. 31

Preference fine-tuning setup

• Preference fine-tuning commonly uses prompt–chosen/rejected-response data to incorporate qualitative human preferences beyond next-token or accuracy-based objectives.KTO is an exception that requires only a chosen or rejected continuation, rather than both responses.Source: §5 · p. 31

Manual and heuristic selection

• Manual and heuristic pipelines filter preference data using engagement, multi-aspect quality criteria, accuracy checks, stratified sampling, length filtering, decontamination, and model-based diversity.These practices often involve undocumented external labeling-company processes and are therefore difficult to reproduce.Source: §5 · pp. 31–32

Model-based selection

• Model-based selection uses language models and critics to filter, rank, critique, or remove erroneous responses, increasingly supporting RLAIF reward-model cycles.Examples include GPT-4 ranking, critic models identifying diverse errors, and removal of edits that fail to address critiques.Source: §5 · p. 32

Reward model re-weighting

• Reward-model re-weighting selects or gates candidates through rejection sampling and Best-of-N, while newer feedback-loop methods iteratively improve rewards or policies with synthetic preference data.Rejection sampling is used for instruction data and reasoning paths; West-of-N and self-rewarding methods blur the boundary between synthetic-data generation and selection, with diversity needed to reduce mode collapse.Source: §5 · p. 32

Overview

• ICL demonstration choice and ordering affect performance, motivating selection methods that improve results without training the potentially large language model.Approaches select an ordering, choose from labeled data, or strategically annotate a small unlabeled set.Source: §6 · p. 33

Demonstration selection

• Retrieval-based methods tailor demonstrations to each test input using similarity, predicted labels, utility scores, cross-entropy differences, set-level scoring, or structural diversity.Fixed-set methods instead optimize average performance across target-task inputs using reinforcement learning or latent-token reconstruction.Source: §6 · p. 33
• Sampling-based methods select demonstrations without training a retriever, using uniform sampling, validation-performance prediction, influence, informativeness, diversity, coverage, or misconfidence.Several methods iteratively refine sets rather than selecting demonstrations independently.Source: §6 · pp. 33–34

Selective annotation

• Selective annotation uses a fixed budget to label informative unlabeled examples, reducing human annotation while supplying demonstrations for test-specific ICL selection.Annotation-set methods commonly combine difficulty or uncertainty with diversity, after which sentence-embedding similarity retrieves demonstrations for each test input.Source: §6 · p. 34
• IDEAL achieves better performance with lower time consumption than vote-k by using k-NN graphs, diffusion-based subset influence, and greedy selection.Other annotation methods combine diversity and uncertainty through max coverage or iteratively sample task-specific patterns for human correction.Source: §6 · p. 34

Overview

• Task-specific fine-tuning selects data either to match a narrow target distribution, especially in few-shot settings, or to diversify data for efficiency and robustness.These objectives center on model performance, data efficiency, and evaluation integrity.Source: §7 · p. 35, Figure 8 · p. 35

Selecting from auxiliary data to improve generalization

• Target-distribution matching is particularly useful when target data are scarce, but label-space heuristics require clear, distinctive task structure and may require human judgment.FETA (Albalak et al., 2022b) uses label-space similarity, while DEFT-Few (Ivison et al., 2023a) selects 500 nearest auxiliary points and Skill-it (Chen et al., 2023c) models skill dependencies.Source: §7 · pp. 35–36
• Auxiliary-data selection improves transfer by choosing data similar to the target, using methods such as domain classifiers, nearest neighbors, skill dependencies, and gradient-based utilities.Importance sampling, contrastive selection, and influence functions select very similar points; newer methods address scalability through online bandits, LoRA gradients, and random projections.Source: §7 · pp. 35–36

Improving task-specific data efficiency

• Data-efficiency methods diversify or filter target subsets using graph geometry, expected difficulty, neighbor-label agreement, and self-influence to remove ambiguous, noisy, or outlier examples.Azeemi et al. (2023) found that selecting difficult-to-learn examples improved language-task performance while reducing dataset size by up to 60%.Source: §7 · pp. 36–37

Selecting data for robustness

• Robustness methods prioritize underperforming subgroups or minimize worst-case subgroup error through group DRO and R-MDL, reducing bias toward subpopulations and spurious correlations.Sagawa et al. (2020) uses the generalization gap as utility, while Haghtalab et al. (2022) also seeks data efficiency across subgroups.Source: §7 · pp. 35, 37

Considerations for task-specific fine-tuning

• Because fine-tuning datasets are small, computationally expensive selection is plausible, although methods requiring full-dataset training or per-step subgroup validation may be prohibitive at web scale.Larger-scale adaptations could reduce update frequency or use smaller proxy models.Source: §7 · pp. 35, 37

Data selection for pretraining

• The cited preprocessing reduces an original corpus from around 50 billion images to 6 billion and filters interleaved images by deduplication, dimensions, aspect ratio, similarity, and face removal.Zhu et al. (2024) interleave only images with cosine similarity scores of at least 0.15, discard images smaller than 150 pixels or with aspect ratios above 2 or below 0.5, and remove images containing faces.Source: §8 · p. 38
• Non-language pretraining also uses heuristic and quality filtering, trained filtering networks, and online task mixing, but heuristic validation is difficult and filtering networks face a circular dependence on data quality.Bruce et al. (2024) trained a classifier from 10,000 hand-labeled videos; Fang et al. (2024) found that good filtering networks require high-quality training data.Source: §8 · p. 38

Data selection for task-specific fine-tuning

• Task-specific fine-tuning methods across domains generally build utility functions from model outputs, while synthetic-image selection through distribution matching supports strong image-classification performance.Yuan et al. (2024a) used maximum mean discrepancy to match synthetic data to target data, achieving good performance with synthetic data alone and when mixing real and synthetic data.Source: §8 · p. 39

Adaptive batch selection

• Adaptive batch selection emphasizes uncertain, inconsistent, or non-redundant examples, but adds computational overhead, has limited validation beyond small NLP tasks, and has shown minimal success for LLM pretraining.RHO-Loss was validated on SST2 and CoLA, but Kaddour et al. (2023) found it computationally inefficient; success in other domains motivates adapting such methods to language.Source: §8 · p. 39

Bias, fairness, and robustness

• Data selection supports fairness and robustness through balancing, reweighting, adversarial representations, and progressive expansion, but combining datasets can dilute robustness rather than improve it.Deng et al. (2023) reported better worst-group performance with Progressive Data Expansion, while Nguyen et al. (2022) found no single dataset dominated robustness and combined datasets reduced component robustness.Source: §8 · pp. 39–40

Increasing the rate of data selection research

• DataComp and DataPerf lower barriers to data-selection research through shared datasets and evaluations, finding benefits from selective datasets and automated methods while revealing domain- and task-dependent transfer limits.DataComp evaluated 21 submissions and DataPerf 54; DataComp does not assess generative capabilities, and language-domain benchmarking remains underused.Source: §8 · p. 40

Data cleaning

• Data cleaning removes text chunks within data points and also includes detecting mislabeled data, imputing missing values, and resolving inconsistencies.Some selection utility functions can support cleaning, but additional cleaning methods are outside this review’s scope.Source: §9 · p. 41

Dataset distillation and coreset selection

• Dataset distillation synthesizes a small dataset intended to match full-data training, whereas coreset selection chooses a subset whose trained model achieves optimal performance.Distillation methods have evolved from computationally costly bilevel optimization toward gradient, feature-distribution, trajectory, and kernel matching; coreset methods use ranking, geometry, uncertainty, loss, and gradient-matching criteria, including for LLM fine-tuning.Source: §9 · p. 41

Data attribution and valuation

• Data valuation and attribution quantify how training points affect predictions, supporting detection of mislabeled data, brittle predictions, train-test contamination, and influential examples.Data Shapley can provide a selection utility, while datamodels and TRAK support counterfactual or impact analysis (Park et al., 2023).Source: §9 · pp. 41–42

Data augmentation

• Data augmentation generates modified data to shift training coverage, but semantic divergence from the desired language distribution can cause undesirable distribution shift and reduced performance.Selection can complement augmentation, including for limited-data domains, so augmentations should remain close to the intended distribution.Source: §9 · p. 42

Data curation

• Data curation encompasses discovering, organizing, integrating, annotating, cleaning, storing, and maintaining data, generally before data selection creates a training dataset for a desired purpose.The practices overlap in areas such as cleaning and filtering, but selection also includes methods such as heuristic filters, data mixing, and toxic-content filtering.Source: §9 · p. 42

Curriculum learning and active learning

• Curriculum learning chooses when to train on examples according to model capability, while active learning allocates a fixed annotation budget for supervised training; both differ from data selection’s broader training-data choices.Curriculum learning has seen limited natural-language and LLM adoption, whereas active learning explicitly targets supervised settings (Graves et al., 2017; Xu et al., 2020; Fan & Jaggi, 2023).Source: §9 · p. 42

10.1 Test set decontamination

• Test-set decontamination is essential for trustworthy evaluation, using overlap detection, canaries, and methods for identifying models trained on benchmark data.Approaches include n-gram or indexed matching, canary auditing, min-k% probability, and statistical tests for contamination; code data additionally benefits from temporal splits or exact-match removal.Source: §10.1 · pp. 43–44

10.2 Tradeoffs between memorization and generalization

• Data selection should reduce harmful memorization without eliminating useful memorization of stable facts, APIs, and programming syntax.Memorization increases with model capacity and sample repetition, while deduplication can reduce undesirable memorization but must be applied selectively.Source: §10.2 · p. 44

10.3 There’s no free lunch for filtering

• Filtering can introduce bias and worsen evaluation loss, motivating methods with higher precision and recall when removing undesirable text.Blocklist and toxicity filtering disproportionately remove text by or about minority and marginalized groups.Source: §10.3 · pp. 44–45

10.4 Tools for data selection

• Data-selection workflows are supported by tools for dataset exploration, auditing, cleaning, deduplication, contamination detection, signal computation, and efficient filtering.Examples include corpus search engines, n-gram and overlap analysis, CCNet, Infini-gram, open-source quality and toxicity filters, and pip-installable deduplication or selection packages.Source: §10.4 · p. 45

10.5 Considerations when applying data selection to your setting

• Applying data selection depends on coverage goals, dataset size, test-distribution knowledge, compute constraints, and filtering cost.Use coverage-oriented selection for general pretraining or small task datasets, redundancy reduction for large task datasets, appropriate utilities or heuristics, inexpensive filters first, and smaller proxy models when possible.Source: §10.5 · pp. 45–46

11.1 Accelerating research on data selection

• Scaling down datasets and models could accelerate iteration and broaden participation, but whether small-model findings transfer to larger models remains uncertain.Training with 745x less data has been shown to lose only 1.9% on GLUE and 2.5% on super-natural instructions, while some benchmarks can be distilled to 100 data points within 2% of true accuracy.Source: §11.1 · p. 47
• Direct metrics for evaluating selected data could reduce costly model training, speed method development, and enable broader participation in the field.Existing intrinsic measures such as distance, density, diversity, tendencies, and association have not yet been connected to downstream model performance.Source: §11.1 · p. 47
• Data-centric benchmarks and challenges are needed to make selection methods comparable across settings, lower entry barriers, and encourage open progress.Language-model research spans many training settings and goals, so multiple benchmarks and challenges may be beneficial; appropriate toy settings remain unsettled.Source: §11.1 · pp. 47–48
• Open-sourcing tools that implement emerging data-selection best practices can reduce startup overhead and let researchers focus on specific pipeline components.The paper presents open development of practices and tools as mutually reinforcing ways to accelerate research progress.Source: §11.1 · p. 48

11.2 Better understanding the properties of the target distribution

• Clearer goals and target-distribution properties can guide method choice, especially for preference finetuning, where success is less specifically defined.For preference finetuning, diversification, auxiliary-data methods, and robustness-oriented selection are suggested as potentially useful directions.Source: §11.2 · p. 48

11.3 Shifting compute time from model training to data processing

• Future research may shift compute from model training to data processing through model-based filtering, synthetic or robust data generation, cheap utility approximations, and additive data selection.Additive selection could use language models to infill information, such as explanations, within existing data points.Source: §11.3 · p. 48

12 Conclusion

• The work compiles an in-depth review of data selection, provides a unified framework for comparing methods, and describes best practices for selecting language-model training data.The framework also supports identifying future research directions, while substantial room remains for innovation and improvement.Source: §12 · p. 49

RefinedWeb

• RefinedWeb removes lines composed only of numerical characters, counters, or a single word, and edits lines with fewer than 11 words matching specified patterns.Source: A Heuristic Filtering Details · p. 77

Limitations

• The review repeatedly notes that heuristic methods are difficult to reproduce because thresholds, exact patterns, and some selection mechanisms are omitted or unspecified.RefinedWeb leaves “mainly” uppercase and counter criteria undefined, while ROOTS reports eight filters without selection thresholds.Source: A Heuristic Filtering Details · pp. 77, 79

MassiveText

• MassiveText (Rae et al., 2021) removes documents failing any of five heuristics: word count, mean word length, symbol-to-word ratio, bullet-line proportion, or ellipsis-ending proportion.The thresholds are 50–100,000 words, mean word length 3–10 characters, symbol-to-word ratio less than 0.1, fewer than 90% of lines starting with bullets, and fewer than 30% ending with “. . . ”.Source: A Heuristic Filtering Details · p. 77

MADLAD-400

• MADLAD-400 (Kudugunta et al., 2023) removes a document when more than 20% of its sentences have a questionable characteristic, including a language-ID mismatch.Source: A Heuristic Filtering Details · p. 78

Other corpora

• Other corpora apply domain-specific preprocessing: ROOTS uses code statistics and several undocumented filters, Llama cleans each source separately, and Pile uses source-specific extraction and boilerplate removal.HTLM retains simplified HTML only when its text-to-HTML ratio exceeds 0.46, while HumanEval and AlphaCode filter oversized, long-line, auto-generated, or low-alphanumeric code files.Source: A Heuristic Filtering Details · pp. 79–80