PaperlayerView exact revision on arXiv

Source-linked AI summary

QuRating: Selecting High-Quality Data for Training Language Models

Alexander Wettig, Aatmik Gupta, Saumya Malik, Danqi Chen

arXiv:2402.09739v3cs.CLcs.LG

TL;DR

Choosing high-quality pre-training data remains difficult because its ideal properties are hard to characterize. QuRating uses LLM pairwise judgments and learned quality ratings to select data, finding that educational-value sampling improves in-context learning across tasks while balancing quality and diversity.

  • Problem

    The ideal properties of pre-training data remain hard to characterize, limiting evidence about how to select data effectively under resource constraints.

    Source: §1 · p. 1

  • Method

    QuRating elicits pairwise LLM judgments across four text qualities, converts them into scalar ratings, and samples documents using quality-weighted logits that balance quality and diversity.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    Educational-value sampling improves in-context learning on every task by an average of 1.8% versus uniform selection, similarly to training for 50% more steps.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Effective pre-training data selection requires balancing quality and diversity rather than choosing only the highest-rated documents.

    Source: §1 · p. 2

    QuRating is sensitive to LLM biases and limitations, while the robustness of automatic annotations requires large-scale human quality judgments.

    Source: §7 · p. 9

Abstract

from arXiv · show

Selecting high-quality pre-training data is important for creating capable language models, but existing methods rely on simple heuristics. We introduce QuRating, a method for selecting pre-training data that can capture human intuitions about data quality. In this paper, we investigate four qualities - writing style, required expertise, facts & trivia, and educational value - and find that LLMs are able to discern these qualities, especially when making pairwise judgments of texts. We train a QuRater model to learn scalar ratings from pairwise judgments, and use it to annotate a 260B training corpus with quality ratings for each of the four criteria. In our experiments, we select 30B tokens according to the different quality ratings and train 1.3B-parameter language models on the selected data. We find that it is important to balance quality and diversity. When we sample using quality ratings as logits over documents, our models obtain lower perplexity and stronger in-context learning performance than baselines. Our best model is based on educational value and performs similarly to a model trained with uniform sampling for 50% more steps. Beyond data selection, we use the quality ratings to construct a training curriculum which improves performance without changing the training dataset. We extensively analyze the quality ratings and discuss their characteristics, biases, and wider implications.

1. Introduction

QuRating captures human-perceived text quality through pairwise LLM judgments, scalar ratings, and quality-guided pre-training data selection. It improves language-model performance when quality and diversity are balanced, and also enables curricula and broader analyses of data selection.

Method

  • QuRating compares text pairs, trains a scalar-rating model, selects pre-training data, and evaluates which abstract qualities improve language-model performance.The four criteria are writing style, facts & trivia, educational value, and required expertise.Source: §1 · p. 1
  • LLMs provide more stable and accurate judgments in pairwise comparisons, while Bradley-Terry ratings become document-selection logits with temperature τ controlling quality–diversity trade-offs.τ →0 approaches top-k selection, whereas τ →∞ approaches uniform random sampling; the formulation connects ratings to RLHF rewards.Source: §1 · pp. 1–2, Figure 1 · p. 1

Results

  • 1.8% average ICL improvement across every task over uniform selection comes from educational-value sampling, matching a baseline trained for 50% more steps.Using τ = 2.0 gives more consistent task results and improves validation perplexity, while writing-style selection yields the best perplexity without substantial downstream gains.Source: §1 · p. 2
  • Selecting only the highest-rated documents excels on particular tasks but underperforms on others, demonstrating that quality selection must preserve diversity.Experiments select 30B of 260B tokens and compare against uniform sampling, perplexity filtering, and importance resampling using in-context learning on 10 diverse tasks.Source: §1 · p. 2

Curriculum

  • Curricula ordered by required expertise outperform random ordering when training on the same data.The curriculum uses the quality ratings without changing the training dataset.Source: §1 · p. 2

Analysis and resources

  • The paper analyzes rating distributions, examines high- and low-ranking documents, studies social impacts, and releases code, judgments, QuRater, and QuRatedPajama.QuRatedPajama contains ratings over 260B SlimPajama tokens, supporting scalable exploration of human notions of data quality.Source: §1 · p. 2

2. Background

Existing data-selection pipelines use hand-crafted or model-based proxies, but these can be too coarse or favor easily predictable text. QuRating instead uses pairwise LLM judgments of multiple quality criteria, alongside standard deduplication and sparse distillation perspectives.

Rule-based heuristics

  • Rule-based filters remove noisy artifacts using thresholds such as word length, stop-word fraction, and repetition, but lack precise quality measurement.Examples include C4 filters (Raffel et al., 2020) and Gopher rules (Rae et al., 2021).Source: §2 · p. 2

Model-based heuristics

  • Model-based selection uses target-domain similarity, generative-model resampling, or perplexity, but domains such as Wikipedia are imprecise proxies and likelihood favors simple repetitive text.These approaches include heuristic classification (Brown et al., 2020), DSIR (Xie et al., 2023b), and perplexity filtering (Wenzek et al., 2020; Muennighoff et al., 2023; Marion et al., 2023).Source: §2 · p. 2

LLM quality signals

  • QuRating queries LLMs across four quality criteria and uses pairwise comparisons for more stable rankings, selecting data to teach strong skills with fewer samples.This contrasts with Gunasekar et al. (2023), who filter for coding-related educational value and augment web data with synthetic LLM-generated data.Source: §2 · p. 2

Deduplication

  • Deduplication improves models by removing repeated data and increasing diversity, but should precede quality-based selection because it cannot conveniently sample a subset of a specified size.Deduplication is commonly fuzzy or semantic (Jiang et al., 2023; Abbas et al., 2023; Tirumala et al., 2023).Source: §2 · p. 2

Distillation

  • QuRating resembles sparse distillation by providing one quality signal per sequence, while models ultimately learn from selected raw documents rather than teacher feedback at every token.Knowledge distillation traditionally supplies richer token-level teacher feedback (Hinton et al., 2015; Kim & Rush, 2016; Sanh et al., 2019; Agarwal et al., 2024).Source: §2 · p. 3

3. Quantifying Qualitative Aspects of Text

QuRating captures human judgments of text quality through pairwise LLM comparisons and learned scalar ratings, avoiding hand-crafted heuristics. The method evaluates four complementary criteria, validates prompt judgments, and trains a QuRater model on large-scale comparisons.

3.1. Overview of the Method

  • QuRating converts pairwise quality judgments into scalar ratings with a Bradley-Terry model and a QuRater trained using binary cross-entropy.GPT-3.5-turbo compares text pairs and records confidence scores, enabling scalable quality signals without heuristic rules or proxy domains.Source: §3.1 · p. 3, §3 · p. 3

3.2. Choice of Criteria and Prompts

  • The four criteria target polished writing style, long-tail facts and trivia, educational value, and required expertise or prerequisite knowledge.The criteria are chosen to apply broadly, require content understanding, yield fine-grained rankings, and complement one another.Source: §3.2 · pp. 3–4
  • 97% prompt agreement was achieved on clear-cut cases for every criterion except facts and trivia, which reached 92%.GPT-3.5-turbo was evaluated against human preferences on 1,600 document pairs.Source: §3.2 · p. 4

3.3. Why Use Pairwise Comparisons?

  • 0.79 ± 0.01 Kendall tau agreement for pairwise judgments exceeded 0.61 ± 0.06 for individual ratings in a 10-document writing-style case study.Pairwise comparisons also better capture fine quality differences and are easier for annotators to verify.Source: §3.3 · p. 4

3.4. Training the QuRater Model

  • The QuRater fine-tunes a 1.3B-parameter Sheared-Llama model with four linear heads on large-scale pairwise judgments, achieving over 93% held-out accuracy.Judgments cover 250K text pairs per criterion, use both text orders and multiple continuations to reduce positional bias, and draw from 500K SlimPajama documents.Source: §3.4 · p. 4

4. Selecting Data By Quality Rating

QuRating selects corpus documents by sampling without replacement according to softmax probabilities from QuRater scores, with temperature controlling the quality–diversity tradeoff. The same scheme connects quality-weighted sampling to RLHF-style rewards and induces a curriculum when sampling order is preserved.

Quality-based sampling

  • Documents are sampled without replacement from corpus D using softmax probabilities based on QuRater ratings s_i.Sampling without replacement increases diversity and enables efficient Gumbel top-k sampling.Source: §4 · p. 4
  • As temperature τ approaches zero, sampling becomes top-k selection; as τ approaches infinity, it recovers uniform sampling.The temperature controls sample diversity and implicitly changes the language-modeling objective to reward-weighted regression.Source: §4 · p. 4

Quality ratings as rewards

  • Quality-weighted sampling approximates pre-training on a random subset followed by RLHF steering toward higher-rated documents, with τ acting as the KL-divergence weight.Unlike typical RLHF rewards, these ratings are not conditioned on user input and target data supporting generalizable skills and useful world knowledge.Source: §4 · p. 5

Curriculum learning with quality ratings

  • Preserving the sampling order creates a curriculum in which lower-rated examples appear earlier and higher-rated examples later, especially when training uses the entire corpus.This reverses the usual post-selection shuffling procedure.Source: §4 · p. 5

5. Experiments

Experiments evaluate QuRating through quality-based data selection and curriculum learning on 1.3B-parameter language models trained from scratch. Educational-value sampling gives the strongest overall gains, while balanced sampling and rating-based curricula improve performance beyond uniform training.

5.1. Setup

  • QuRatedPajama contains 260B annotated tokens scored on four criteria, with document ratings averaged from length-weighted segments of up to 512 tokens at an annotation cost of 520 NVIDIA H100 hours.The corpus uses 1024-token Llama-tokenizer sequences and supports data selection, curriculum training, and data discovery.Source: §5.1 · p. 5
  • The experiments select 30B-token subsets while preserving overall domain proportions, train randomly initialized 1.3B-parameter Transformers for one shuffled epoch, and evaluate perplexity, 10-task ICL, and instruction following.Instruction following uses 10,000 ShareGPT instruction-response pairs for supervised fine-tuning and 1,000 held-out instructions judged with GPT-4-0314.Source: §5.1 · p. 5, §5.2 · p. 5

5.2. Data Selection Methods

  • Selection compares uniform sampling, quality-based sampling at τ ∈ {0.0, 1.0, 2.0}, inverse sampling, criteria mixtures, DSIR (Xie et al., 2023b), and perplexity filtering.Uniform selection also includes a 45B-token baseline requiring 50% more compute, while DSIR targets English Wikipedia or Book-like data and perplexity filtering uses ShearedLlama-2.7B scores.Source: §5.2 · pp. 5–6

5.3. Results

  • Educational-value sampling at τ = 2.0 improves uniform sampling on all 10 ICL tasks, matches a uniform model trained with 50% more data and compute, and reaches a 57.3% instruction-tuning win rate.The result is reported alongside perplexity and ICL comparisons in Table 1 and instruction-following comparisons in Figure 3.Source: Table 1 · p. 6, §5.3 · p. 7, Figure 3 · p. 7
  • DSIR (Xie et al., 2023b) and perplexity filtering perform worse than uniform sampling, suggesting that their training-data bias does not translate into better ICL results.Perplexity is also an unreliable capability proxy: writing-style selection gives the lowest perplexity but only minor ICL improvements, with no clear cross-task trend.Source: §5.3 · p. 6
  • Sampling quality-rated documents with τ = 2.0 improves perplexity over uniform selection, whereas top-k selection substantially worsens perplexity and produces less balanced ICL performance across tasks.Top-k can achieve strong gains on individual tasks, but always underperforms uniform selection on at least one task; temperature-based sampling achieves better or equal average performance.Source: §5.3 · p. 6, Table 1 · p. 6
  • Facts-and-trivia and required-expertise selection improve average ICL but weaken commonsense reasoning, while low-rated-document controls provide no overall gain and can benefit all three commonsense tasks.Criteria mixing improves over baselines but does not beat educational-value or facts-and-trivia selection, possibly because writing style contributes 25% of the mixture.Source: §5.3 · p. 7

5.4. Curriculum Learning

  • Using required-expertise ratings to order the same uniformly selected examples improves average ICL by 0.6% for low-to-high expertise and 0.5% for the reverse, but only increasing expertise improves held-out perplexity.Thus, quality ratings improve curriculum performance without changing the training dataset.Source: §5.4 · p. 7

6. Analysis of Quality Ratings

The analysis finds that quality ratings vary widely within domains and clusters, capture distinctions beyond perplexity, and produce interpretable but imperfect preferences across document types. It also documents sampling-dependent social, topical, and geographical biases, motivating continued scrutiny of filtered pre-training data.

6.1. Distribution of Quality Ratings

  • Quality ratings span wide ranges within domains and clusters, so selecting entire domains is suboptimal; domains nonetheless show expected strengths across criteria.Books score highly for writing style, some Wikipedia documents for facts & trivia, and ArXiv for required expertise; all domains contribute to the top 5% for educational value.Source: §6.1 · pp. 7–8, Figure 4 · p. 8
  • QuRating differs meaningfully from perplexity filtering: rating–likelihood Spearman correlations range from 0.50 for writing style to -0.02 for required expertise.Documents with low quality ratings exhibit a wide range of likelihoods under Llama-2-7b (Touvron et al., 2023b).Source: §6.1 · p. 8

6.2. Data Inspection

  • Quality ratings correlate with interpretable code and text traits, favoring documented code for writing style and facts & trivia and ranking embedded-system code above CSS for required expertise.Higher-ranked StackExchange examples mix code and natural language, whereas lower-ranked examples include convoluted traces, logs, XML, HTML, and CSS.Source: §6.2 · p. 8
  • The educational-value criterion can take an educational shortcut, highly rating documents about schools or universities even when they are not inherently educational.The authors suggest a better prompt or stronger judge such as GPT-4 may remedy this limitation.Source: §6.2 · p. 8

6.3. Documenting Social Bias

  • At temperature τ = 2, QuRating produces slightly more balanced retention rates than prior methods, but top-k selection greatly exacerbates amplification and suppression across categories.Shopping-related topics are consistently suppressed, while amplified topics and roles generally reflect the selected criterion; writing-style sampling mildly favors European websites.Source: §6.3 · pp. 8–9, Table 2 · p. 9
  • The authors emphasize that filtering may cause representational and allocative harms, including stereotyping, erasure, or reduced performance on overlooked tasks, while fairness for web-scale pre-training remains unresolved.They note that web-scraped datasets are already socially and geographically skewed and that defining a fair pre-training distribution is difficult (Lucy et al., 2024; Dodge et al., 2021).Source: §6.3 · p. 9

7. Conclusion

QuRating is presented as a promising framework for improving language models and enabling exploration of alternative notions of data quality, while its annotations and conclusions remain subject to important limitations.

  • QuRating is a promising avenue for improving language models, and the authors release its judgments, QuRater model, language-model checkpoints, and annotated QuRatedPajama.Source: §7 · p. 9

Limitations

  • QuRating depends on LLMs’ ability to discern text qualities, making annotations sensitive to poorly understood model biases and to differences between pairwise judgments and individual-text scoring.Source: §7 · p. 9
  • The work calls for large-scale human judgments to assess annotation robustness and subjectivity, further study of social and linguistic biases, and experiments testing transfer beyond 1.3B-parameter models and across quality criteria.Source: §7 · p. 9

Impact Statement

QuRating’s data selection may reduce training costs and help organizations with fewer resources train stronger models, while clarifying links between pre-training data and model capabilities. Because web data and selection methods can encode harmful biases, QuRating should be combined with curation and bias evaluation, and its ratings should not be treated as measures of social or literary value.

Potential benefits

  • Data selection may reduce computational costs and training’s environmental footprint (Strubell et al., 2019; Lacoste et al., 2019; Patterson et al., 2021), while enabling resource-constrained organizations to train stronger models.The reported experiments remain expensive to reproduce, with each training run requiring 200 NVIDIA H100 GPU hours.Source: Impact Statement · p. 9

Scientific contribution

  • Studying intuitive data qualities provides insight into the relationship between pre-training data and model capabilities.Source: Impact Statement · p. 10

Risks and limitations

  • QuRating can encode social, linguistic, and geographical biases, so it should be paired with manual curation and careful bias evaluation before wider deployment.The authors note documented harms in web-trained language models, including social biases (Nadeem et al., 2021; Abid et al., 2021) and toxic generations (Gehman et al., 2020), and caution that quality ratings do not measure a text’s social or literary value or support textual or demographic studies.Source: Impact Statement · p. 10

Y. Logiqa: A challenge dataset for machine reading · Y. Black is to criminal as Caucasian is to police: De-

This section consists of related-work bibliography entries spanning machine-reading comprehension, logical reasoning, bias detection and removal in word embeddings, sentence completion, instruction-following evaluation, and character-level text classification.

Y. Logiqa: A challenge dataset for machine reading

Y. Black is to criminal as Caucasian is to police: De-

A. Full Prompts … Required Expertise

QuRating uses short pairwise prompts tailored to four quality criteria, validated against handpicked documents but still subject to model biases. The templates emphasize relative choices, concise outputs, and instructions covering each criterion’s intended notion of quality.

A. Full Prompts

  • The generic system response “You are a helpful assistant.” performed better than assigning an expert personality across all subjects.The authors identify personality effects on subjective judgments as a future-work question.Source: A. Full Prompts · p. 17
  • 97.6% writing-style, 91.9% facts-and-trivia, 98.2% educational-value, and 98.5% required-expertise agreement were achieved on 40 handpicked validation documents per criterion.The documents represented examples expected to be highly or poorly rated and were not used for prompt refinement.Source: A. Full Prompts · p. 17

Prompt Template

  • The common template asks the model to compare two text excerpts and choose the one that better satisfies a substituted quality criterion.The prompts were developed heuristically and iteratively using examples from SlimPajama, with GPT-3.5-turbo chosen for scalable annotation cost.Source: Prompt Template · p. 17, A. Full Prompts · p. 17

Writing Style

  • The writing-style prompt selects the text with a more polished and beautiful writing style while instructing the model to ignore language, length, and presentation order.The final dataset compares texts with equal token counts, but GPT-3.5-turbo still exhibits positional bias despite these instructions.Source: Writing Style · p. 17, A. Full Prompts · p. 17

Facts & Trivia

  • The facts-and-trivia prompt prefers specific facts and obscure trivia, presents excerpts as Option A and Option B, and requires a single-word A-or-B response.Because excerpts are cut off, the model must infer context and make a relative choice even when quality appears similar.Source: Facts & Trivia · p. 17

Educational Value

  • The educational-value prompt favors clear explanations, step-by-step reasoning, or questions and answers.These features define the intended educational-value criterion in the pairwise judgment.Source: Educational Value · p. 17

Required Expertise

  • The required-expertise prompt favors text requiring greater expertise and prerequisite knowledge, with validation sources curated across strong and weak examples.Table 4 additionally describes a 10-document ranking designed to test nuanced decreases in writing style.Source: Required Expertise · p. 17, Table 3 · p. 18, Table 4 · p. 18

B. QuRater Model · B.1. Judgment Dataset

The judgment dataset uses GPT-3.5-turbo pairwise choices across four criteria, with repeated generations approximating confidence and short snippets mitigating positional bias. It contains 250K queried pairs from English Wikipedia, with imperfectly correlated criteria and confident-label filtering for QuRater evaluation.

B.1. Judgment Dataset

  • GPT-3.5-turbo generates 20 “A”/“B” predictions per criterion and document order to reconstruct confidence, while criteria are queried separately.Performance deteriorated when all criteria were queried together.Source: §B.1 · p. 19
  • To reduce positional bias on long documents, the dataset ranks randomly extracted snippets of 256–512 tokens in half of cases and exactly 512 tokens otherwise.Snippet lengths are sampled using the Llama tokenizer (Touvron et al., 2023a).Source: §B.1 · p. 19
  • The dataset queries 250K pairs from the English Wikipedia subset, omits a small filter-blocked fraction, and costs $2820 to create.Labels are not collected for all Wikipedia languages represented in RedPajama (TogetherAI, 2023).Source: §B.1 · p. 19, Table 5 · p. 19
  • The four GPT-3.5-turbo criteria are positively but only moderately correlated, typically at Pearson coefficients of 0.45–0.55 and below 0.6 overall.Required expertise and writing style are least correlated, at 0.29.Source: §B.1 · p. 19, Figure 5 · p. 19

B. QuRater Model

  • QuRater uses separate prediction heads for each criterion, and the study compares multitask training with separate models and initialization from Sheared-Llama-1.3B against training from scratch.The initialization baseline is Sheared-Llama-1.3B (Xia et al., 2024).Source: §B.1 · p. 20, Table 6 · p. 20

B.1. Judgment Dataset

  • Evaluation is restricted to pairs with confident judgment labels, using confidence margins greater than 50% or 80%, where the margin is |pA − pB| = |2pB≻A − 1|.This confidence margin is also defined for the qualitative-choice statistics.Source: Table 5 · p. 19, §B.1 · p. 20

B. QuRater Model

  • The broader corpus-selection statistics count sequences from a 260B-token SlimPajama subset processed into exactly 1024-token sequences, so domain proportions differ from the raw corpus.The corpus statistics are reported in Table 7.Source: Table 7 · p. 20

B.2. QuRater Training

QuRater uses multi-task fine-tuning with confidence-filtered pairwise judgments to predict four quality criteria efficiently. Validation and domain-specific tests show that confident judgments improve performance, while multi-task and pre-trained initialization are generally advantageous.

Model and data

  • Four linear regression heads predict all quality criteria from the final sequence token in one forward pass.The model is initialized from Sheared-Llama-1.3B, a pruned version of Llama-2-7B.Source: §B.2 · p. 21
  • Training and evaluation retain only judgments with confidence margins of at least 50%, using a random 10% validation split for early stopping and hyperparameter selection.Early stopping uses validation accuracy averaged across criteria because non-confident predictions contain little quality signal and may reflect positional bias.Source: §B.2 · p. 21

Hyperparameters

  • The selected QuRater configuration uses a learning rate of 5 × 10−5 and 2 training epochs from the tested hyperparameter grid.The search varied learning rates of 2 × 10−5 and 5 × 10−5 and training durations of 2 or 4 epochs, with batch size fixed at 512.Source: §B.2 · p. 21

Evaluation

  • Multi-task fine-tuning usually matches or outperforms separate models, while pre-trained initialization substantially improves QuRater performance; educational value is the hardest criterion to predict.These findings hold on validation and specially procured domain-specific test sets, using confident GPT-3.5-turbo judgments.Source: §B.2 · p. 21

C. Connection of Exponential Sampling to RLHF · Z pD (y) exp

The paper connects exponential quality-based sampling to RLHF by treating the QuRater score as an input-independent reward and a pretrained language model as the reference policy. Under large-data assumptions, maximum-likelihood training on exponentially resampled documents approximates the RLHF-optimal policy, making the sampling strategy equivalent to quality-guided generation.

C. Connection of Exponential Sampling to RLHF

Z pD (y) exp

  • Maximum-likelihood training on documents resampled with probability proportional to exp of their quality ratings yields a weighted cross-entropy objective.Source: Z pD (y) exp · p. 21
  • The resampling procedure is implemented through Monte Carlo sampling and importance resampling, whose optimal model is the data distribution multiplied by the exponential quality weighting.Source: Z pD (y) exp · p. 21
  • Assuming a large corpus and that p_D(y) sufficiently approximates the underlying distribution ˆp_D(y), maximum likelihood on the resampled data approximates the RLHF-optimal policy p*(y).Source: Z pD (y) exp · p. 21
  • Thus, exponential sampling is equivalent to first training on the full dataset and then using RLHF to steer generation toward documents with higher quality ratings.Source: Z pD (y) exp · p. 21

D. Experimental Details

The experiments select 30B tokens from a 260B QuRatedPajama corpus while preserving RedPajama domain proportions, then train standardized 1.3B-parameter models and evaluate perplexity and in-context learning. Results are reported across domains and tasks, with perplexity not necessarily predicting task performance when training distributions vary.

Data and training setup

  • Each selection method preserves the original RedPajama domain proportions while selecting 30B tokens from the 260B-token QuRatedPajama corpus.QuRatedPajama is a curated subset of SlimPajama, itself a subset of RedPajama.Source: D. Experimental Details · p. 22
  • Models use a global batch size of 2048 sequences, a 5 × 10−4 learning rate with cosine decay, 5% warmup, Adam, weight decay 0.1, and RoPE/SwiGLU architecture.Each 1.3B-parameter model is trained on 8 NVIDIA H100 GPUs for 200 GPU hours over 30B tokens.Source: D. Experimental Details · p. 22

In-context learning settings

  • In-context learning uses task-specific few-shot counts that fit within 1024 tokens, reporting accuracy except EM for NQ and normalized accuracy when available.Demonstrations range from zero for BoolQ to 15 for ARC-easy, ARC-challenge, and WinoGrande.Source: D. Experimental Details · p. 22

Detailed results

  • Perplexity and in-context learning results are provided across domains and tasks in Tables 8 and 9, with curriculum models shown first and best results highlighted per column.Table 8 covers held-out per-token perplexity, while Table 9 covers task performance; inverse sampling denotes selecting low-rated documents.Source: D. Experimental Details · p. 22, Table 8 · p. 23, Table 9 · p. 24
  • Varying the training distribution breaks the usual relationship between perplexity and downstream task performance observed when varying model size or training-token count.Figure 6 plots perplexity against in-context learning performance across the models in Tables 8 and 9.Source: Figure 6 · p. 22, D. Experimental Details · p. 22

E. Further Analysis of Quality Ratings

The analysis examines quality ratings on 1M sequences from the 260B QuRatedPajama dataset, including domain composition and rating distributions. It finds that perplexity is not a good approximation for any quality criterion and evaluates how top-rated webpages alter category retention.

Dataset analysis

Rating analysis

  • Perplexity scores are not good approximations for any of the quality criteria.Figure 7 compares quality ratings with Llama-2-7B negative log-likelihoods over 1M training sequences.Source: Figure 7 · p. 25
  • Normalized quality-rating distributions are examined across clusters of 760K CommonCrawl and C4 training sequences.Figure 8 summarizes the rating distributions by cluster.Source: Figure 8 · p. 26

Selection analysis

  • Top-10% webpage selection by quality criterion is used to identify amplified and suppressed AboutMe categories and their retention rates.Table 10 reports category retention rates in percent for each criterion.Source: Table 10 · p. 27

F. Inspecting Raw Documents and Ratings

The section provides an unfiltered view of raw training documents across multiple corpora, sampled at four quality-rating percentiles.

Raw documents and ratings

  • Raw-document snippets span SlimPajama’s Wikipedia, Books, Stack Exchange, Github, and ArXiv subsets, plus C4 and CommonCrawl clusters, at the 5th, 30th, 70th, and 95th quality-rating percentiles.These documents are presented without filtering to expose the training data directly; the SlimPajama and C4/CommonCrawl selections correspond to Figures 4 and 8, respectively.Source: F. Inspecting Raw Documents and Ratings · p. 27

A small number of documents contain potentially sensitive content.

This section presents percentile-selected, quality-rated training examples from diverse CommonCrawl+C4 clusters and topics. It highlights that a small number of documents contain potentially sensitive material across several subject areas.

Examples by quality percentile

  • Raw CommonCrawl+C4 training examples from Clusters 4–9 are shown at the 5th, 30th, 70th, and 95th percentiles for each quality criterion.Ratings are normalized to zero mean and unit variance across the corpus, with cluster shares ranging from 4.6% to 6.7%.Source: Table 19 · p. 36, Table 20 · p. 37, Table 21 · p. 38

Domain coverage

  • The sampled clusters cover diverse domains, including business, software, travel, sports, conversational writing, environmental science, and healthcare.Representative passages include business reporting, hotel descriptions, sports commentary, ecological material, and medical advice or research content.Source: Table 19 · p. 36, Table 20 · p. 37, Table 21 · p. 38

Potentially sensitive content

  • Some examples contain potentially sensitive or contested material, including medical guidance, disease-related content, and speculative claims about environmental or cosmic hazards.The examples include advice for people with rheumatoid arthritis, oncology-related material, a historical account of cacao’s medical uses, and a critique of a doomsday interpretation of interstellar magnetic clouds.Source: A small number of documents contain… · pp. 41–42

Raw training examples

  • Raw examples are shown at the 5th, 30th, 70th, and 95th percentiles for CommonCrawl+C4 clusters covering war and military, research and science, administrative text, books, politics, entertainment, and education.The displayed cluster shares range from 2.9% to 3.9%, and the tables state that ratings are normalized to zero mean and unit variance across the corpus.Source: Table 26 · p. 43, Table 27 · p. 44, Table 28 · p. 45

Potentially sensitive content

  • The sampled content includes military reporting, political commentary, claims about government and conflict, and other material that may contain sensitive or controversial subject matter.Examples include military exercises, intelligence and political claims, conflict reporting, election coverage, and discussions of Israel and Palestine.Source: A small number of documents contain… · pp. 43, 47

Domain coverage

  • Other sampled clusters contain book reviews and literary or creative writing, scientific and educational material, legal or regulatory text, financial advice, and entertainment content.The examples range from book and movie discussions to housing policies, chemical permits, law courses, foreign-transaction fees, and classroom activities.Source: A small number of documents contain… · pp. 45, 46, 48
  • Tables 33–38 select raw training examples at the 5th, 30th, 70th, and 95th quality-rating percentiles within individual CommonCrawl+C4 corpus clusters.The identified clusters cover religious, legal, film, biological, academic, and music vocabulary, with shares from 1.8% to 2.8%.Source: Table 33 · p. 50, Table 34 · p. 51, Table 35 · p. 52
  • Potentially sensitive documents cover religion, sexuality, historical violence, migration, and health-related crises.Examples include contentious Catholic commentary, sexualized dialogue, discussion of Taiwan’s massacre, and reporting on migrant-camp fires and COVID-19 cases.Source: A small number of documents contain… · pp. 50, 52, 54
  • The material also includes legal and criminal-justice content, including court decisions, expungement guidance, cybercrime training, and patent litigation.These examples span judicial reasoning, criminal-record relief, cybersecurity and counter-terrorism curricula, and pharmaceutical patent law.Source: Table 34 · p. 51, A small number of documents contain… · p. 51
  • Sensational or potentially harmful media appears in entertainment and online-news documents, including violent film criticism, celebrity clickbait, and explicit or demeaning discussions.The examples include commentary on nuclear annihilation, celebrity sexual allegations, and sexualized acting interactions.Source: A small number of documents contain… · pp. 52, 56

Quality-rated examples

  • Table 40 samples documents at the 5th, 30th, 70th, and 95th quality-rating percentiles from CommonCrawl+C4 Cluster No. 25, a 1.7% music-related cluster.The cluster is associated with music, musical, band, festival, jazz, songs, and theatre; ratings are normalized to zero mean and unit variance across the corpus.Source: Table 40 · p. 57

Related papers

Loading 2402.09739v3…