PaperlayerView exact revision on arXiv

Source-linked AI summary

QuRating: Selecting High-Quality Data for Training Language Models

Alexander Wettig, Aatmik Gupta, Saumya Malik, Danqi Chen

arXiv:2402.09739v3cs.CLcs.LG

TL;DR

QuRating addresses the challenge of identifying high-quality pre-training data beyond simple heuristics by learning scalar ratings from pairwise LLM judgments of four text qualities. Applied to a 260B-token corpus, quality-aware sampling improves perplexity and in-context learning when quality and diversity are balanced, while the ratings also support a performance-improving curriculum. The approach remains sensitive to LLM biases and subjective judgments, whose robustness and downstream effects are not yet well understood.

  • Problem

    Ideal training-data properties remain difficult to characterize, although better understanding could support stronger models under resource constraints.

  • Method

    QuRating compares text pairs on four qualities, trains a QuRater model to produce scalar ratings, and uses those ratings to select data and construct curricula.

  • Results

    Educational-value selection improves in-context learning on every task by an average of 1.8% over uniform selection, comparable to training for 50% more steps.

  • Takeaways & Limitations

    Balancing quality and diversity makes human notions of data quality effective signals for scalable data selection and curriculum design.

  • Takeaways & Limitations

    QuRating depends on LLMs to discern text qualities, remains sensitive to incompletely understood biases, and requires more human judgments to assess annotation robustness.

Abstract

from arXiv · show

Selecting high-quality pre-training data is important for creating capable language models, but existing methods rely on simple heuristics. We introduce QuRating, a method for selecting pre-training data that can capture human intuitions about data quality. In this paper, we investigate four qualities - writing style, required expertise, facts & trivia, and educational value - and find that LLMs are able to discern these qualities, especially when making pairwise judgments of texts. We train a QuRater model to learn scalar ratings from pairwise judgments, and use it to annotate a 260B training corpus with quality ratings for each of the four criteria. In our experiments, we select 30B tokens according to the different quality ratings and train 1.3B-parameter language models on the selected data. We find that it is important to balance quality and diversity. When we sample using quality ratings as logits over documents, our models obtain lower perplexity and stronger in-context learning performance than baselines. Our best model is based on educational value and performs similarly to a model trained with uniform sampling for 50% more steps. Beyond data selection, we use the quality ratings to construct a training curriculum which improves performance without changing the training dataset. We extensively analyze the quality ratings and discuss their characteristics, biases, and wider implications.

1. Introduction

QuRating addresses the difficulty of characterizing ideal training data by using LLM judgments to model human-perceived text qualities and select pre-training data. Experiments show that balancing quality with diversity improves model performance, with educational-value selection improving in-context learning and quality-based curricula also helping.

  • QuRating: QuRating compares text pairs, learns scalar quality ratings, selects pre-training data, and evaluates which qualities benefit language-model training.The method targets human-perceived qualities through a sequence of pairwise judgments, rating prediction, data selection, and model evaluation.
  • Quality criteria: The study focuses on writing style, facts & trivia, educational value, and required expertise as data-selection criteria.These qualities are treated as subjective, and GPT-3.5-turbo is evaluated for discerning them in clear-cut cases.
  • Results: Sampling only the highest-rated documents improves some tasks but harms others, whereas temperature τ = 2.0 yields more consistent results and better validation perplexity.The temperature controls the trade-off between quality and diversity; writing-style selection gives the best perplexity but little downstream-task improvement.
  • Results: Educational-value selection improves in-context learning on every task by an average of 1.8% over uniform selection, comparable to training for 50% more steps.This is the strongest reported selection result in the introduction.
  • Curriculum: A curriculum ordered by required expertise outperforms random ordering on the same training data.The curriculum changes presentation order rather than the selected dataset.
  • Analysis and resources: The authors analyze rating distributions, inspect high- and low-ranking documents, and discuss social impacts, including effects on AboutMe web pages.The work also releases code, model outputs, the QuRater model, and the QuRatedPajama dataset.

2. Background

Prior data-selection methods use rules, proxy domains, language-model likelihood, or LLM judgments, but each provides an incomplete account of data quality. QuRating instead targets multiple fine-grained qualities through pairwise comparisons and sparse quality signals.

  • Rule-based heuristics: Rule-based pipelines remove noisy artifacts with thresholds on word length, stop-word fraction, and repetition, but do not precisely identify desirable examples.Binary filters are useful for exclusion but limited for ranking higher-quality data.
  • Model-based heuristics: Model-based heuristics select data resembling a target domain or having high language-model likelihood, but domain similarity is an imprecise proxy and likelihood favors simple repetitive content.These approaches include discriminator-based selection, generative resampling, and perplexity filtering.
  • LLM quality signals: LLM-based selection has previously used GPT-4 to identify educational coding data, whereas QuRating studies four criteria and uses pairwise comparisons for more stable rankings.The comparison distinguishes QuRating’s broader criteria and judgment format from prior LLM filtering.
  • Deduplication: Deduplication improves diversity by removing repeated data but is not designed to sample a subset of a specific size.It is therefore complementary to subset-selection methods.
  • Distillation: QuRating can be viewed as sparse distillation because it supplies one quality signal per sequence rather than teacher feedback at every token.The analogy contrasts sequence-level selection guidance with token-level knowledge distillation.

3. Quantifying Qualitative Aspects of Text

QuRating converts pairwise LLM judgments about abstract text qualities into scalar ratings that can guide scalable data selection. Pairwise judgments are used because they are more reliable for fine-grained comparisons, and the resulting QuRater model predicts all four criteria efficiently.

  • Pairwise judgments: QuRating asks which of two texts better exhibits a criterion, records the judge’s confidence, and builds a large judgment dataset from document pairs.GPT-3.5-turbo supplies judgments over sampled text snippets from a web-scale corpus.
  • Rating model: The Bradley-Terry model translates binary judgments into scalar quality ratings estimated by maximum likelihood.The QuRater model parameterizes the ratings and is trained with binary cross-entropy.
  • Criteria: The four criteria are chosen to be broadly applicable, content-sensitive, finely ranked, and complementary: style, facts and trivia, educational value, and required expertise.The prompts operationalize these criteria through polished writing, long-tail facts, explanations or reasoning, and prerequisite knowledge.
  • Prompting strategy: Pairwise prompting produces more reliable judgments and better discrimination of fine quality differences than judging individual texts.Pairwise judgments are also easier for human annotators to verify than precise individual scores.
  • Scale and training: The study queries GPT-3.5-turbo on 250K text pairs per criterion, reverses pair order, averages multiple continuations, and fine-tunes a 1.3B-parameter QuRater model.Reversing order and averaging continuations counteracts positional bias; the model uses four linear heads for the four criteria.

4. Selecting Data By Quality Rating

The selection framework samples documents according to QuRater scores converted into softmax probabilities, with temperature controlling the trade-off between quality concentration and diversity. Its limiting cases recover top-k selection and uniform sampling, while the sampling order can also define a curriculum.

  • Quality-weighted sampling: Each corpus document receives a QuRater score, and documents are sampled without replacement from softmax probabilities normalized over the corpus.The framework uses the scores as quality-sensitive sampling weights.
  • Quality-diversity trade-off: Temperature τ controls diversity: τ → 0 approaches top-k selection, while τ → ∞ becomes uniform sampling.This provides a continuous interpolation between concentrated quality selection and broad random sampling.
  • Interpretation: Quality-weighted sampling implicitly changes the language-model objective toward reward-weighted regression.The paper connects this formulation to reward-based views of language-model training.
  • Interpretation: For sufficiently large samples, the strategy approximates pre-training on a random subset followed by RLHF steering toward documents with higher quality ratings.Under this interpretation, temperature determines the strength of the quality preference relative to the KL term.
  • Curriculum learning: Sampling without replacement naturally creates a curriculum, and reversing the sampling order places lower-rated examples earlier and higher-rated examples later.This ordering differs from standard training, which randomly shuffles examples after selection.

5. Experiments

The experiments evaluate QuRating-based data selection and curricula using 1.3B-parameter models trained on 30B-token subsets, comparing quality-based sampling with uniform and other baselines. Sampling with temperature improves the quality–diversity trade-off, while educational-value selection gives the strongest ICL results and expertise-based curricula improve performance without changing the training examples.

  • Experimental Setup: The experiments train 1.3B-parameter language models on 30B-token subsets selected from QuRatedPajama and evaluate perplexity, ICL, and instruction following.The evaluation includes held-out validation perplexity, 10 ICL tasks, and instruction following after fine-tuning on ShareGPT examples.
  • Data Selection Methods: The study compares uniform sampling, QuRating-based sampling at τ ∈ {0.0, 1.0, 2.0}, inverse sampling, criteria mixing, DSIR, and perplexity filtering.The comparisons retain the overall dataset’s domain proportions, while uniform sampling also has a 45B-token baseline requiring 50% more compute.
  • Results: DSIR and perplexity filtering underperform uniform sampling, with perplexity results suggesting that these baselines introduce substantial training-data bias without improving ICL.This result indicates that lower perplexity or resemblance to proxy corpora does not reliably produce stronger model capabilities in these experiments.
  • Results: τ = 2.0 sampling improves perplexity over uniform selection, whereas top-k selection has worse perplexity and less balanced performance across ICL tasks.Top-k can produce strong gains on individual tasks, but sampling avoids a task-level failure relative to uniform selection and achieves better or equal average performance.
  • Results: Perplexity does not reliably predict ICL performance: writing-style selection yields the lowest perplexity but only minor ICL improvements, with no clear overall trend.Facts & trivia and required-expertise selection improve ICL on average but perform worse on commonsense reasoning, while criteria mixing does not surpass the strongest single criteria.
  • Results: Educational-value selection improves ICL performance over uniform sampling on all 10 tasks and performs comparably to uniform training with 50% more data and compute.It is also the only model reported to achieve a clear 57.3% win rate against the uniform model after instruction tuning.
  • Curriculum Learning: Expertise-based curricula improve performance even when the training examples remain fixed, with low-to-high and high-to-low ordering improving average ICL by 0.6% and 0.5%, respectively.Only the increasing-expertise curriculum improves held-out perplexity.

6. Analysis of Quality Ratings

The analysis shows that quality ratings vary substantially within and across domains, while also revealing meaningful behavioral patterns and social biases. These findings support document-level selection but highlight limitations in the ratings and the importance of monitoring representation.

  • 6.1. Distribution of Quality Ratings: Quality ratings span wide distributions within domains, so selecting entire domains would miss high-quality documents across the corpus.Book scores highly in writing style, parts of Wikipedia in facts & trivia, and ArXiv in required expertise; all domains contribute to the top 5% for educational value.
  • 6.1. Distribution of Quality Ratings: Ratings align with expected content characteristics across unsupervised clusters, yet nearly all clusters contain a wide range of quality scores.Clusters about cells, protein, genes, energy, climate, and species score highly on expertise, educational value, and facts & trivia, while book-author clusters score highly in writing style.
  • 6.1. Distribution of Quality Ratings: QuRating captures signals distinct from perplexity filtering, with Spearman correlations ranging from 0.50 for writing style to -0.02 for required expertise.Low-quality documents exhibit a wide range of likelihoods, indicating that likelihood and quality ratings do not coincide consistently.
  • 6.2. Data Inspection: On code, writing-style and facts-and-trivia ratings favor comments and documentation, while required-expertise ratings distinguish CSS stylesheets from embedded-system code.Higher-ranked StackExchange documents mix code and natural language, whereas lower-ranked examples include stack traces, logs, XML, HTML, and CSS.
  • 6.2. Data Inspection: Educational-value ratings sometimes reward education-related topics that are not inherently educational, suggesting a criterion-specific shortcut.The paper suggests that a better prompt or stronger judge such as GPT-4 might remedy this behavior.
  • 6.3. Documenting Social Bias: QuRating produces somewhat more balanced retention rates than prior methods, but top-k selection greatly exacerbates amplification and suppression across social categories.Writing-style sampling mildly favors European websites, while several social roles and shopping-related topics are suppressed across criteria.

7. Conclusion

The paper frames QuRating as a promising approach to data selection under resource constraints, while noting important unresolved limitations. It also releases key artifacts to support further research.

  • QuRating is presented as a promising avenue for improving language models under resource constraints.
  • The authors release pairwise judgments, the QuRater model, language-model checkpoints, and the annotated QuRatedPajama dataset.
  • The method depends on LLMs discerning text qualities, so its ratings remain sensitive to poorly understood LLM biases and limitations.
  • The experiments use relatively small 1.3B-parameter models, leaving transfer to larger models uncertain.

Impact Statement

The paper connects data selection with the cost and social risks of language-model training. It argues that quality-based selection can illuminate links between pre-training data and model capabilities, but requires bias-aware deployment practices.

  • Data selection may reduce computational costs and the environmental footprint of model training under resource constraints.
  • Studying intuitive data qualities sheds light on the relationship between pre-training data and model capabilities.
  • QuRating should be combined with manual curation and careful bias evaluation before wider deployment.
  • The ratings do not measure the social or literary value of text and should not be used for textual or demographic studies.

Y. Logiqa: A challenge dataset for machine reading

The supplied passages consist primarily of bibliographic reference fragments rather than substantive content describing the Logiqa dataset or its machine-reading challenge.

  • The passages identify cited works and publication details across language modeling, data selection, ranking, bias, and machine-reading research.
  • The supplied material does not state the Logiqa dataset’s construction, task definition, or reported results.
  • Several entries are split across adjacent passages, with author names, titles, venues, and URLs distributed between fragments.

A. Full Prompts

The appendix describes pairwise prompts for judging four text qualities and reports prompt validation, while noting persistent positional and personality-related biases. It also includes validation tables for strong or weak examples and nuanced writing-style rankings.

  • Prompt Validation: Prompt validation uses handpicked documents with clear quality differences to assess agreement with author preferences.
  • Prompt Validation: The prompts achieve 97.6% agreement for writing style, 91.9% for facts and trivia, 98.2% for educational value, and 98.5% for required expertise.
  • Prompt Validation: GPT-3.5-turbo still exhibits positional bias despite instructions intended to prevent order effects.
  • Prompt Template: The prompts ask an LLM to compare two text excerpts and choose which better exhibits a specified quality.
  • Criteria: The four criteria are writing style, facts and trivia, educational value, and required expertise.
  • Prompt Template: The prompts instruct judges to ignore language, text length, and presentation order when making comparisons.
  • Writing-Style Validation: A separate table ranks ten documents by progressively decreasing writing style to test nuanced LLM judgments.

B.1. Judgment Dataset

The judgment dataset uses GPT-3.5-turbo pairwise comparisons of short text snippets to assess four quality criteria, with confidence filtering and analyses of judgment correlations and coverage.

  • Pairwise judgments: GPT-3.5-turbo generates 20 ordered A/B predictions for each criterion and document pair to reconstruct judgment confidence.The prompts query each criterion separately because joint querying degraded performance.
  • Pairwise judgments: Long-document comparisons trigger positional bias, so the dataset ranks snippets of at most 512 Llama tokens.Snippet lengths are sampled from Uniform[256, 512] in half of cases and set to 512 otherwise.
  • Dataset statistics: 250K queried pairs yield judgment statistics reported in Table 5, with a small subset missing because of content filters and coverage limited to English Wikipedia.The dataset creation cost was $2820.
  • Judgment relationships: The criteria predictions are positively correlated but remain distinct, with typical Pearson correlations of 0.45-0.55 and 0.29 between required expertise and writing style.The correlations are all below 0.6, indicating differences across many documents.

B.2. QuRater Training

QuRater is fine-tuned to predict scalar quality ratings from confident pairwise judgments, using a multi-task architecture with separate heads for the four criteria.

  • Model training: QuRater fine-tuning uses Sheared-Llama-1.3B with four linear regression heads predicting ratings for the four criteria.The multi-task setup predicts all criteria in one forward pass.
  • Confidence filtering: Training and evaluation retain only judgments with confidence margins of at least 50%, excluding predictions containing little quality signal or positional bias.A 10% random held-out split supports early stopping and hyperparameter selection.
  • Optimization: The selected hyperparameters are a learning rate of 5 × 10^-5 and 2 training epochs with batch size 512.The search considered learning rates of 2 × 10^-5 and 5 × 10^-5 and 2 or 4 epochs.
  • Evaluation: Validation and domain-specific test results compare multi-task QuRater models with separate models and random initialization, with multi-task fine-tuning usually comparable or better.Educational value is the hardest category to predict, while performance increases when evaluation uses confident judgments.

C. Connection of Exponential Sampling to RLHF

The paper connects exponential quality-based document sampling to RLHF by treating quality ratings as rewards and the pre-trained language model as a reference distribution.

  • RLHF formulation: RLHF fine-tunes p(y|x) to maximize reward r(x,y) while remaining near a reference model p_ref(y|x) under a relaxed KL constraint.The rewards in the cited setup encourage helpful and harmless behavior.
  • QuRating connection: With QuRater score s(y) as an input-independent reward and p_D(y) as the reference model, the optimal policy defines a quality-weighted document distribution.This setting removes user-input conditioning from the reward model.
  • Sampling equivalence: Maximum-likelihood training on documents resampled proportional to exponential quality ratings approximates the same optimal policy under a large-data assumption.The approximation relies on p_D(y) sufficiently matching the underlying corpus distribution.
  • Interpretation: The sampling strategy is equivalent to first training on the full dataset and then using RLHF to guide generation toward documents with higher quality ratings.The equivalence is stated as a two-stage interpretation of exponential resampling.

D. Experimental Details

The experiments train 1.3B-parameter models on 30B-token subsets selected from a 260B-token corpus, evaluating perplexity and in-context learning across ten tasks.

  • Training data: Each selection method preserves the original RedPajama domain proportions while selecting 30B tokens from the 260B-token QuRatedPajama corpus.QuRatedPajama is curated from SlimPajama, itself a subset of RedPajama.
  • Training setup: The trained models use a global batch size of 2048 sequences, cosine learning-rate decay from 5 × 10^-4 to 5 × 10^-5, and 5% warmup.Each 30B-token run uses 8 NVIDIA H100 GPUs and costs 200 GPU hours.
  • Evaluation: The ten in-context learning tasks use task-specific few-shot counts constrained by a 1024-token context window, reporting accuracy except EM for NQ.The task suite includes ARC, SciQA, LogiQA, BoolQ, HellaSwag, PIQA, WinoGrande, NQ, and MMLU.
  • Evaluation: Tables 8 and 9 report held-out per-token perplexity and in-context learning performance for models trained with different selection methods.Figure 6 relates perplexity to ICL performance across models.

E. Further Analysis of Quality Ratings

The paper examines how quality ratings are distributed across corpus samples and whether conventional perplexity scores track those ratings. It finds that perplexity is not a good approximation for the investigated quality criteria.

  • Analysis setup: A random subset of 1M sequences from the 260B-token QuRatedPajama dataset is analyzed for rating distributions and domain composition.The analysis also studies distributions across unsupervised clusters when domains are too coarse.
  • Rating validity: Perplexity scores are not good approximations for any of the quality criteria.Figure 7 compares quality ratings with Llama-2-7B negative log-likelihood scores averaged over sequence tokens.
  • Cluster distributions: Quality ratings are normalized across CommonCrawl and C4 clusters to show how rating values are distributed across 760K training sequences.The normalized distributions are summarized in Figure 8.
  • Bias analysis: The AboutMe analysis selects the top 10% of webpages by each quality criterion and reports which categories are amplified or suppressed.Retention rates are reported for the selected data.

F. Inspecting Raw Documents and Ratings

The paper inspects unfiltered training documents at several points in the quality-rating distributions. This provides raw examples corresponding to low, middle, and high-rated data.

  • Raw-document inspection: Raw documents are shown at the 5th, 30th, 70th, and 95th percentiles of quality ratings.The examples are drawn from domain-level and cluster-level distributions.
  • Raw-document inspection: The documents are presented without filtering to provide an unfiltered view of the training data.This inspection covers Wikipedia, Books, StackExchange, Github, ArXiv, CommonCrawl, and C4 clusters.

A small number of documents contain potentially sensitive content.

The inspected corpus contains documents spanning technical, literary, historical, scientific, religious, and entertainment content. A small number of examples also include potentially sensitive or contentious material.

  • Potentially sensitive content: The collection also includes passages with violent, discriminatory, or otherwise disturbing subject matter.Examples include historical and fictional passages containing hostile or graphic language.
  • Potentially sensitive content: Some examples contain politically or socially charged material, including discussion of civilian casualties and contentious public claims.The excerpts vary in tone and evidentiary context.
  • Potentially sensitive content: Some documents discuss religion and religious practice in conversational or opinionated terms.One excerpt concerns First Communion and reactions to churchgoers’ behavior.
  • Content diversity: The examples include technical research prose, literary passages, historical discussion, and music-related content.These excerpts illustrate the broad topical range of the inspected training data.
Loading 2402.09739v3…