PaperlayerView exact revision on arXiv

Source-linked AI summary

DsDm: Model-Aware Dataset Selection with Datamodels

Logan Engstrom, Axel Feldmann, Aleksander Madry

arXiv:2401.12926v1cs.LGstat.ML

TL;DR

Human-designed “high quality” filtering can fail to improve language-model performance relative to random selection. The paper instead formulates dataset selection as performance optimization and uses datamodels to estimate which training subset best serves target tasks. DsDm improves performance on specified and unseen tasks, with selected data matching randomly selected data trained using 2× the compute budget.

  • Problem

    Similarity-based filtering for “high quality” data may not improve—and can even hurt—model performance compared with random selection.

    Source: §1 · p. 1

  • Method

    DsDm models how a learning algorithm uses candidate training subsets to predict target-task performance, then selects the subset maximizing the estimated outcome.

    Source: §1 · pp. 1–2

  • Results

    DsDm-selected datasets yield models performing as well as models trained with 2× the compute budget on randomly selected data across diverse held-out benchmarks.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Choosing target tasks similar to expected deployment tasks can improve model performance on previously unseen tasks.

    Source: §1 · p. 2, §7 · p. 10

    The paper hypothesizes that fixed-source similarity selection can hurt by trading off data diversity for qualitatively cleaner data.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

When selecting data for training large-scale models, standard practice is to filter for examples that match human notions of data quality. Such filtering yields qualitatively clean datapoints that intuitively should improve model behavior. However, in practice the opposite can often happen: we find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data. To develop better methods for selecting data, we start by framing dataset selection as an optimization problem that we can directly solve for: given target tasks, a learning algorithm, and candidate data, select the subset that maximizes model performance. This framework thus avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks. Our resulting method greatly improves language model (LM) performance on both pre-specified tasks and previously unseen tasks. Specifically, choosing target tasks representative of standard LM problems and evaluating on diverse held-out benchmarks, our selected datasets provide a 2x compute multiplier over baseline methods.

1. Introduction

The paper argues that intuitive “high quality” data filtering can fail, and frames dataset selection as directly optimizing model performance instead. DsDm models how training subsets affect target-task performance and improves both specified-task results and generalization to unseen tasks.

1. Introduction

  • Similarity-based selection from “high quality” sources may not improve performance and can even underperform random selection.The paper evaluates standard methods used for models such as PaLM and GPT-3.Source: §1 · p. 1
  • Dataset selection is formulated as choosing a subset that maximizes performance for given target tasks, a learning algorithm, and candidate data.Finding the exact optimum is generally infeasible because it would require evaluating every possible subset.Source: §1 · p. 1
  • DsDm approximates this optimization by modeling how the learning algorithm uses training data to predict on target tasks, then selecting the subset with the best estimated performance.The method uses datamodels to approximate the mapping between training subsets and model performance.Source: §1 · p. 2
  • DsDm-selected datasets match models trained with 2× the compute budget on randomly selected data across diverse held-out benchmarks.The experiments train models up to 1.8B parameters, while no baseline outperforms random selection at the same compute budget.Source: §1 · p. 2

2. Estimating the optimal dataset selection with DsDm

This section defines dataset selection as minimizing target-task loss, then uses datamodels to estimate that loss efficiently. DsDm optimizes the estimated loss over candidate subsets, with linear datamodels selecting examples associated with the smallest estimated loss contributions.

2.1. Task-optimal dataset selection

  • Dataset selection seeks the size-k subset whose trained model minimizes population loss on a target distribution, accounting for target-data and learning-algorithm randomness.The learning algorithm maps a training set to a trained model, and the target loss measures that model on the target distribution.Source: §2.1 · p. 2
  • Exact optimization is computationally infeasible because evaluating every possible subset would require training and evaluating a model for each one.The paper therefore trades optimality for a tractable approximation.Source: §2.1 · p. 2, §2.2 · p. 2

2.2. Estimating model loss efficiently with datamodels

  • Datamodels approximate the mapping from a training-subset choice to the resulting trained model loss on each target sample.They use a parameterized function trained to predict loss from subset characteristic vectors rather than examining each subset directly.Source: §2.2 · pp. 2–3
  • The paper instantiates datamodels as linear functions of subset characteristic vectors, assigning each training example a fixed effect on predicted loss.This linear assumption ignores interactions in which an example’s effect changes with the other selected examples.Source: §2.2 · p. 3

2.3. DsDm: Dataset Selection with Datamodels

  • DsDm estimates target-distribution loss by averaging datamodel loss estimates over target samples, then selects the size-k subset minimizing that approximation.With linear datamodels, selection reduces to choosing examples corresponding to the smallest summed indices; other datamodel parameterizations are also possible.Source: §2.3 · p. 3

3. Evaluating DsDm

DsDm is evaluated against similarity-based and random selection on four language-model tasks using C4 data and fixed-token 125M-model training. It consistently improves target-task performance, including matching a 10×-compute larger model, while revealing that useful data need not resemble the target task and that excluding harmful examples matters.

3.1. Setup

  • The evaluation uses LAMBADA, CS-Algorithms, SQuAD, and Jeopardy as target tasks, with English C4 as the candidate dataset and 125M-parameter GPT-2-style models trained on 6 billion tokens.Source: §3.1 · p. 4

3.2. Results

  • DsDm most improves target-task performance across all four tasks and even outperforms a larger model trained with 10× the compute on randomly selected data.Performance is measured by mean label log-probability with fixed total tokens; smaller selections require more epochs.Source: §3.2 · p. 4, Figure 1 · p. 5
  • DsDm’s performance declines as the selected fraction grows, indicating that its highest-ranked samples are more useful on average than lower-ranked samples.At full selection, all methods choose the same subset; smaller selections contain more data predicted to improve performance.Source: §3.2 · p. 4, Figure 1 · p. 5
  • Classifier and DSIR often fail to outperform random selection, including on SQuAD and CS-Algorithms, showing that textual similarity alone does not reliably identify useful training data.The baselines select examples by classifier-based or n-gram distributional similarity to target-task samples.Source: §3.2 · p. 4, §3.1 · p. 4
  • DsDm-selected examples can be useful without matching target-task format: SQuAD selections often contain non-format-matching question-answering text, while CS-Algorithms and Jeopardy selections can appear nonsensical.Qualitative cleanliness or similarity therefore does not identify all data that improves task performance.Source: §3.2 · pp. 4–5
  • DsDm also excludes examples that resemble harmful or “mislabeled” data, whose inclusion can make training worse than random selection.For SQuAD, examples least likely to be selected by DsDm often contain QA text but are incoherent or malformed and hurt performance when used for training.Source: §3.2 · p. 5

4. Selecting data for broad model capabilities

The paper selects training data by optimizing model performance on representative target tasks rather than matching intuitively high-quality sources. Across diverse held-out benchmarks, DsDm improves related capabilities, matches 2× random-data compute, and requires diverse targets to avoid harming unrelated tasks.

4. Selecting data for broad model capabilities

  • DsDm models target-task performance to select training subsets, using LAMBADA, SQuAD, and Jeopardy as broad proxy tasks for held-out capabilities.The method estimates the optimal selection for these tasks together and compares it with similarity-based, deduplication, and random selection baselines.Source: §4 · pp. 5–6

4.1. Results

  • Random selection is a strong baseline: no baseline method outperforms random selection across the evaluated compute budgets.This pattern holds for the similarity-based baselines and is part of the held-out benchmark comparison.Source: §4.1 · pp. 6, 8, §4 · p. 6
  • 2× compute: DsDm models match randomly selected-data models trained with twice the compute and outperform all selection baselines across compute budgets.The comparison includes 1.3B DsDm models against 1.8B Chinchilla-optimal models trained on randomly selected data.Source: §4 · p. 6, §4.1 · p. 7, Figure 3 · p. 7
  • DsDm improves reading-comprehension and world-knowledge benchmarks while not reducing performance on unrelated categories on average.These gains are qualitatively related to the SQuAD and Jeopardy target tasks, whereas DSIR performs well on language understanding but poorly on some other categories.Source: §4.1 · p. 7, Table 1 · p. 8
  • Target-task choice shapes related benchmark performance: SQuAD most improves reading comprehension, Jeopardy most improves world knowledge, and all targets together maximize overall accuracy.Targeting LAMBADA can reduce world-knowledge accuracy, so targets must be diverse enough to cover downstream problems.Source: §4.1 · p. 7
  • Using the same LAMBADA, Jeopardy, and SQuAD targets, Classifier and DSIR still fail to outperform random selection, indicating that DsDm’s selection procedure matters beyond target choice.The head-to-head comparison targets the same tasks rather than the baselines’ usual high-quality sources.Source: §4.1 · pp. 7–8

5. Discussion

DsDm can steer model behavior beyond benchmark performance by targeting desired properties, while proxy-model selection and compute costs shape its practical use. Target-task choice generally improves related benchmark categories but can reduce performance on unrelated ones.

Broader opportunities

  • Selecting data with DsDm can control downstream model behavior beyond performance on a single benchmark.The framework is presented as a way to intervene on data curation within the training pipeline.Source: §5 · p. 8

Applications and broader opportunities

  • DsDm’s target-task framework could support alignment, fairness, and domain-specific performance improvements.The paper names low-resource languages and programming as example domains, and contrasts pretraining-time alignment with methods that typically operate after training.Source: §5 · p. 8

Training stronger models with weaker proxy models

  • Smaller proxy models can select data that substantially improves larger-model performance despite being much weaker on benchmarks.The authors describe proxy selection as the simplest scaling approach and suggest less naïve scaling could improve results further.Source: §5 · p. 8

Target-task effects

  • Choosing target tasks can improve related benchmark categories while reducing performance on unrelated tasks.SQuAD, Jeopardy, and LAMBADA improve reading comprehension, world knowledge, and language understanding respectively, while LAMBADA targeting can reduce world-knowledge performance.Source: Figure 4 · p. 9

Computational efficiency

  • DsDm’s computational overhead is concentrated in per-example gradients from four 125M-parameter proxy models and can be amortized through caching.The paper argues this overhead can become worthwhile when compared with the cost of computing gradients for standard-sized models.Source: Computational cost · p. 9

6. Related Work

Prior dataset-selection work commonly matches training examples to intuitively high-quality references or uses proxies such as loss, gradients, uncertainty, and deduplication. Related methods also target task performance, coresets, and instruction tuning across other domains.

LM pretraining selection

  • Standard LM pretraining selection often matches candidate data to a high-quality reference corpus such as Wikipedia.DSIR and Classifier methods are identified as examples of this two-step framework.Source: §6 · p. 9
  • Other LM selection methods use activation-space deduplication, loss differences between candidate and reference models, or simple document filters.Examples include removing documents that are too short or contain too many special characters.Source: §6 · p. 9

Selection beyond LM pretraining

  • Beyond language-model pretraining, prior work selects data to improve task performance, identify coresets, or instruction-tune language models.These methods span multiple application areas and dataset-selection objectives.Source: §6 · pp. 9–10

Computational context

  • Computing gradients for 3B–175B models is roughly 24×–1400× more expensive than for 125M models.This comparison contextualizes the computational scale of proxy-based selection.Source: §6 · p. 9

Selection beyond LM pretraining

  • Related selection approaches rely on prompting, uncertainty proxies such as loss or gradient norm, gradient matching, or deduplication in model-output space.The cited methods broadly differ in the proxy used to identify useful training examples.Source: §6 · p. 10

7. Conclusion

The paper formulates dataset selection as optimizing target-task performance and approximates that objective with datamodels. It reports reliable gains on target tasks and broader generalization to unseen tasks when target tasks resemble expected deployment tasks, while positioning dataset selection as a tool for fine-grained behavior control.

7. Conclusion

  • DsDm casts dataset selection as optimizing performance for target tasks, a learning algorithm, and a candidate training dataset.The method models how the learning algorithm uses training data to predict on target tasks.Source: §7 · p. 10
  • Choosing target tasks similar to expected deployment tasks can greatly improve performance on previously unseen tasks.This extends the framework beyond performance on explicitly specified target tasks.Source: §7 · p. 10
  • Dataset selection can serve as a fine-grained control point for downstream model behavior, including hypothesized fairness, alignment, and domain capabilities.The paper identifies better datamodeling as a route to stronger manipulation of behavior through training data.Source: §7 · p. 10

Appendices

The experiments use the English C4 training corpus, tokenized into fixed-length 1024-token examples. The resulting candidate dataset contains 216,948,746 examples from a diverse public web crawl.

A. Experimental Setup

  • The experimental setup uses the full English en.noblocklist training split of C4.The subset was prepared by AllenAI and did not filter curse words.Source: §A.1 · p. 16

A.1. Candidate dataset

  • C4 documents are concatenated with end-of-text tokens and sliced into 1024-token training examples.Each example generally contains 3,000–6,000 characters, or roughly a thousand words.Source: §A.1 · p. 16
  • The final candidate dataset contains 216,948,746 examples tokenized with the Pythia tokenizer.The examples are formed from the tokenized C4 corpus.Source: §A.1 · p. 16
  • As a public internet crawl, C4 contains diverse text, illustrated through random sample excerpts.The paper uses Figure 24 to contextualize this diversity.Source: §A.1 · p. 16

A.2. Target tasks

This appendix defines the target tasks and their target–holdout splits, then describes leakage checks and similarity-based or untargeted dataset-selection baselines.

A.2. Target tasks

  • SQuAD uses Wikipedia reading-comprehension questions, with 23,107 target examples and 10,557 validation holdout examples.Source: §A.2 · p. 16
  • Jeopardy is split into 876 target and 876 holdout trivia examples, while CS-Algorithms uses 660 target and 660 holdout algorithmic examples.Source: §A.2 · p. 16
  • LAMBADA tests predicting a passage’s final word from its broader context, using 2,577 target and 2,570 holdout samples after removing six overlaps.The task is designed so the final sentence alone is insufficient for crowdworkers, whereas the full passage enables prediction.Source: §A.2 · p. 16
  • Evaluation uses 0-shot prompting for LAMBADA and CS-Algorithms, 3-shot prompting for SQuAD and Jeopardy, and standard task-specific prompts.Source: §A.2 · p. 17

A.2.1. Mitigating train-test leakage

  • Leakage is conservatively defined by separately matching lowercased, whitespace-stripped contexts and continuations within C4 examples.Six LAMBADA test examples overlapped with C4 and were removed; no leakage was found for SQuAD, Jeopardy, or CS-Algorithms.Source: §A.2.1 · p. 19

A.3. Data selection baselines

  • Classifier and DSIR select textually similar data, whereas Random and SemDeDup provide untargeted baselines; SemDeDup retains about 20% using 11,000 clusters.Classifier uses FastText logistic regression, while DSIR matches hashed n-gram distributions to target data.Source: §A.3.1 · p. 19, §A.3.2 · p. 20

A.4. LM training details

The experiments train GPT-2-family models with specified optimization and compute choices, and evaluate them using log-probability or benchmark-specific accuracy procedures.

A.4. LM training details

  • Models use ADAM, 1,024-token sequences, batch size 1,024, cosine learning-rate decay, gradient clipping, and BF16 or FP8 hardware precision.Source: §A.4 · p. 20
  • Hyperparameters are generally selected by minimizing held-out 125M-model perplexity on C4, with increased weight decay for larger Section 4 models to avoid divergence.Source: §A.4 · p. 20
  • Chinchilla-optimal scaling uses roughly 20 training tokens per model parameter in the study’s compute regime.Source: §A.4 · p. 20

A.5. Evaluation metrics

  • Section 3 compares target-task performance with mean log-probability, while Section 4 evaluates unseen benchmarks with accuracy.Source: §A.5 · p. 21

A.5.1. Log-probability

  • Mean log-probability averages the model’s probability of generating the correct continuation tokens given each sample’s context.Source: §A.5.1 · p. 21

A.5.2. Accuracy

  • Accuracy is computed through multiple-choice selection, exact text match, or normalized fuzzy matching depending on the benchmark.Source: §A.5.2 · pp. 21–22

B. Datamodel estimation

  • Datamodel estimation is organized around datamodels, the TRAK estimator, language-model instantiation, and computational-cost analysis.Source: B. Datamodel estimation · p. 23

B.1. Datamodels refresher

Datamodels approximate how a training-subset choice affects a model’s output on a fixed example by learning a supervised mapping from subset indicators to loss.

B.1. Datamodels refresher

  • A datamodel predicts a fixed sample’s model output from a candidate training dataset, learning algorithm, and model-output function over a distribution of training subsets.Source: §B.1 · p. 23
  • The regression operates on each subset’s binary characteristic vector rather than on the subset directly.Source: §B.1 · p. 23
  • Linear datamodels assign each training datapoint a constant estimated effect on the loss when that datapoint is included.Source: §B.1 · p. 23

B.1.1. Estimating datamodels with data regression

This section presents methods for efficiently estimating datamodels, contrasting data regression with TRAK and specializing the approach to logistic regression, language modeling, and the paper’s practical LM setup. It also describes the computational cost of the estimation procedure and the task-optimal dataset-selection evaluation context.

B.1.1. Estimating datamodels with data regression

  • Data regression samples train subsets, trains a model on each, records its output, and fits linear regression, which asymptotically recovers the true datamodel parameters.A standard subset distribution is uniformly random fixed-size subsets.Source: §B.1.1 · p. 23

B.2. Estimating datamodels with TRAK

  • TRAK replaces data regression’s costly model-collection process with an efficient approximation that transforms the original learning problem into a convex one, trading approximation error for computation.Data regression requires training at least thousands of models in practice, whereas TRAK computes datamodels in the convex regime and transfers them back to the original algorithm.Source: §B.2 · p. 24, §B.1.1 · p. 24

B.2.1. Datamodels for logistic regression

  • For logistic regression, TRAK uses influence functions to approximate leave-one-out effects efficiently and thereby construct linear datamodels that can estimate outputs for arbitrary training subsets.The approach uses a single shared inverse term and requires only a few inner products and scalar multiplications per influence estimate.Source: §B.2.1 · pp. 24–25

B.2.2. Transforming learning algorithms to linear regression

  • TRAK linearizes the model output around the full-data parameters, defines a logistic-regression surrogate algorithm, and estimates the original datamodel through that surrogate.The surrogate preserves the mapping from training-set choice to model output rather than merely approximating parameter-to-output behavior.Source: §B.2.2 · pp. 25–26

B.2.3. TRAK estimator

  • TRAK makes its estimator practical by projecting parameter-space features with Johnson–Lindenstrauss matrices and averaging estimates across independently trained models to address dimensionality and randomness.Without dimensionality reduction, a 125M-parameter model would require storing and inverting a 500TB float32 matrix.Source: §B.2.3 · pp. 26–27

B.3. Datamodels for language modeling

  • For language modeling, datamodels use a mean multi-class margin that averages correct next-token probabilities across the sequence’s continuation tokens.This extends classifier-oriented output functions to the token-level cross-entropy setting.Source: §B.3 · p. 27

B.4. TRAK setup

  • The LM implementation trains four 125M models on roughly 82 million C4 samples each, uses projection dimension d = 16384, collects projected gradients, and averages per-model terms.Each model is trained for 80,000 batches, corresponding to approximately 38% of C4.Source: §B.4 · p. 28

B.5. Computational cost

  • 1.2×10^9 FBPs are required overall, dominated by projected gradients across four models (~73% of computation), while training each model contributes ~7%.The reported setup uses m = 4, nmodel = 82×10^6, |S| ≈217×10^6, and ndm ≈30000; the total is (m · nmodel)+(m · |S|)+(ndm) FBPs.Source: §B.5 · p. 29

C. Evaluating task-optimal dataset selection

  • The appendix evaluates task-optimal dataset selection across varying target tasks by describing the tasks, showing samples, detailing training, and presenting omitted figures.This section provides additional context for Section 3’s measurements of DsDm at optimal dataset selection.Source: C. Evaluating task-optimal dataset… · p. 29

C.1. Experimental setup

The evaluation uses four language-model target tasks spanning language understanding, algorithmic problem solving, reading comprehension, and world knowledge, with both zero-shot and three-shot settings. Additional analyses compare accuracy, computation, and counterfactual worst-sample selection.

C.1. Experimental setup

  • The evaluation covers LAMBADA, CS-Algorithms, SQuAD, and Jeopardy across zero-shot and three-shot settings.LAMBADA and CS-Algorithms use zero-shot evaluation, while SQuAD and Jeopardy use three-shot evaluation.Source: §C.1 · p. 29
  • The experiment trains models using the Section A.4 procedure and hyperparameters specified in the Section 3 rows of Table 2.Source: §C.1 · p. 29

C.2. Omitted figures

  • Accuracy reproduces the relative performance ordering from Figure 1 but is noisier and discontinuous compared with log-probability as the selected C4 fraction changes.Accuracy uses exact match for LAMBADA and CS-Algorithms and fuzzy matching for SQuAD and Jeopardy.Source: §C.2 · p. 29
  • The projected-gradient computation requires m · |S| forward-backward passes, while projection itself accounts for less than 1% of projected-gradient time.Source: §C.2 · p. 29
  • DsDm-selected data performs much worse when restricted to its least likely samples than randomly selected data, despite containing question-answering text.Source: §C.2 · p. 30

WARNING: SAMPLES MAY INCLUDE OFFENSIVE TEXT

This section provides qualitative visualizations of the highest- and lowest-ranked training samples selected by Random, DsDm, DSIR, and Classifier across target tasks, using sampled text slices. It also includes examples for specific tasks and supplementary details on evaluating data selections across broad model performance.

WARNING: SAMPLES MAY INCLUDE OFFENSIVE TEXT

  • The appendix visualizes the top and bottom 0.01% of samples for each selection method and target dataset using randomly chosen 500-character excerpts, with offensive or unrenderable samples replaced.The examples are shown across Random, DsDm, DSIR, and Classifier selections, with additional side-by-side SQuAD comparisons.Source: WARNING: SAMPLES MAY INCLUDE OFFENSIVE… · p. 30
  • Each method ranks C4 candidates differently: Random samples randomly, DsDm uses linear datamodel weights, DSIR uses inclusion log-probability, and Classifier uses margin toward the target-dataset class.The figures show these rankings for SQuAD, Jeopardy, LAMBADA, and CS-Algorithms.Source: WARNING: SAMPLES MAY INCLUDE OFFENSIVE… · p. 30
  • 1.3B Random-model performance is consistently matched by 125M DsDm models trained on selected data, including when baselines are near random selection.Each model trains on 25.6 million samples, equivalent to 12% of C4; the 1.3B Random model uses more than 10x the compute budget.Source: Figure 9 · p. 31
  • DsDm’s worst-ranked training examples produce models that perform much worse than models trained on randomly selected data, despite containing QA-related text.This reverses the effect of selecting examples predicted to most increase performance.Source: Figure 10 · p. 31
  • For Jeopardy, the section shows ranked examples produced by DSIR and Classifier.The displayed DSIR and Classifier examples include heterogeneous web-like text, lists, and structured data fragments.Source: WARNING: SAMPLES MAY INCLUDE OFFENSIVE… · p. 34, Figure 17 · p. 34, Figure 16
  • Figures 16–22 compare task-specific best and worst training examples selected by DsDm, DSIR, or Classifier across Jeopardy, LAMBADA, and CS-Algorithms.The examples are ranked separately for each task and selection method.Source: Figure 17 · p. 34, Figure 18 · p. 35, Figure 20 · p. 36
  • Each visual randomly samples from the top or bottom 0.01% of training examples according to the corresponding method and shows a random 512-character slice.The figures note that newline characters are represented by \n.Source: Figure 17 · p. 34, Figure 18 · p. 35, Figure 20 · p. 36
  • For LAMBADA, the section shows ranked examples produced by DsDm, DSIR, and Classifier, including narrative, sports, timestamped, bibliographic, and technical fragments.These examples illustrate the varied text types appearing among task-ranked samples.Source: WARNING: SAMPLES MAY INCLUDE OFFENSIVE… · p. 35, Figure 18 · p. 35, Figure 19
  • For CS-Algorithms, the section shows ranked examples produced by DsDm and DSIR, including product listings, repair text, vehicle specifications, prose, code-related content, and noisy strings.The examples span both readable and highly corrupted or fragmented text.Source: WARNING: SAMPLES MAY INCLUDE OFFENSIVE… · pp. 36–37, Figure 21, Figure 22
  • The classifier’s highest- and lowest-ranked CS-Algorithms training examples are shown as 512-character slices, contrasting selected samples with random training examples.Figure 23 samples examples from the top and bottom 0.01% of training data ranked by the classifier; Figure 24 provides random training slices for comparison.Source: Figure 23 · p. 37, Figure 24 · p. 38

D. Evaluating data selections for broad model performance

  • The appendix section covers model training procedures, dataset-selection baselines, evaluation procedures, and omitted figures for the broad-performance results.The provided passage identifies these topics but does not supply their specific procedures or findings.Source: D. Evaluating data selections for… · p. 39

D.1. Experimental setup

The experiments train GPT-2-style language models under multiple compute budgets and dataset-selection methods, using fifteen few-shot benchmarks for evaluation.

Model training

  • Models use 125M–1.3B parameters with approximately Chinchilla-optimal token-to-parameter ratios, plus a 1.8B randomly trained model for context.Selection methods are evaluated under varying training compute budgets.Source: §D.1 · p. 39
  • Targeted methods select data for four epochs, whereas Random and SemDeDup select data for one epoch without tuning the epoch count.The setup also avoids hyperparameter tuning over target tasks.Source: §D.1 · p. 39

Classifier and DSIR target task

  • Classifier and DSIR target an equally weighted mixture of Wikipedia, Books1, and OpenWebText as a replication of their intended high-quality distribution.These methods select data according to similarity with a target distribution.Source: §D.1 · p. 39

SemDeDup hyperparameters

  • SemDeDup reduces C4 to approximately 20% using ϵ = 0.3 and 11,000 clusters, following its originally described configuration.The cited configuration was originally reported to maximize downstream model accuracy.Source: §D.1 · p. 39

Evaluation details

  • Evaluation covers fifteen benchmarks with specified few-shot counts and accuracy metrics, following category and few-shot choices from the Mosaic Eval Gauntlet.Metrics include fuzzy string matching for open-ended baselines.Source: §D.1 · p. 39

D.2. Omitted figures

Targeting the DsDm language-model tasks, existing baselines do not beat random selection, while the results identify DsDm as necessary to improve performance on these tasks.

D.2. Omitted figures

  • DSIR and Classifier do not outperform randomly selecting data when targeting the relevant language-model tasks.The comparison concerns both targeted baselines evaluated against random selection.Source: §D.2 · p. 39, Figure 25 · p. 40
  • DsDm is necessary to improve model performance on the considered target tasks.The result is contrasted with targeted baseline methods and random selection.Source: §D.2 · p. 40
  • The evaluation covers benchmarks described by their category and corresponding accuracy procedure.Table 3 provides the benchmark descriptions and evaluation procedures used for the comparison.Source: Table 3 · p. 41
Loading 2401.12926v1…