Source-linked AI summary
Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, Luca Soldaini
TL;DR
WebOrganizer addresses the difficulty of curating unstructured, web-crawled pre-training data by organizing pages into complementary topic and format domains and scaling annotations with distilled classifiers. It enables task-specific domain mixing, improves quality-filtered training data, and shows that quality filters themselves implicitly alter domain proportions.
Problem
Web-crawled pre-training corpora contain trillions of largely unstructured tokens, making their contents difficult to understand and systematic domain curation difficult.
Method
WebOrganizer builds topic and format taxonomies, distills large-language-model annotations into efficient classifiers, and optimizes domain mixtures for downstream tasks.
Results
Domain mixing improves downstream performance, raises FineWeb-Edu average accuracy from 54.2% to 56.2% over a 51.6% baseline, and complements quality filtering.
Takeaways & Limitations
Constructing and mixing domains provides a practical complement to document-level quality filtering and makes the composition of web-crawled training data more interpretable.
Takeaways & Limitations
WebOrganizer reduces diverse online content to discrete domains that may obscure important phenomena and introduce errors, biases, or misrepresentations; its taxonomy is not definitive ground truth.
Abstract
from arXiv · showhide
Modern language models are trained on large, unstructured datasets consisting of trillions of tokens and obtained by crawling the web. The unstructured nature makes it difficult to reason about their contents and develop systematic approaches to data curation. In this paper, we unpack monolithic web corpora by developing taxonomies of their contents and organizing them into domains. We introduce WebOrganizer, a framework for organizing web pages in terms of both their topic and format. Using these two complementary notions of domains, we automatically annotate pre-training data by distilling annotations from a large language model into efficient classifiers. This allows us to study how data from different domains should be mixed to improve models on downstream tasks, and we show that we can combine insights about effective topics and formats to further boost performance. We demonstrate that our domain mixing also improves existing methods that select data based on quality. Furthermore, we study and compare how quality-based methods will implicitly change the domain mixture. Overall, our work demonstrates that constructing and mixing domains provides a valuable complement to quality-based data curation methods, opening new avenues for effective and insightful pre-training data curation.
1. Introduction
WebOrganizer addresses the difficulty of curating monolithic web corpora by organizing pages into topic and format domains. These domains support task-specific mixture optimization, improve downstream performance, and complement quality filtering.
- Motivation: Web-crawled pre-training data has become difficult to understand because it consists largely of trillions of tokens without an informative domain summary.Recent curation increasingly relies on cleaning rules and quality filters instead of explicit domain organization.
- Approach: WebOrganizer constructs topic and format taxonomies and distills Llama-3.1-405B-Instruct annotations into efficient 140M-parameter classifiers.The two taxonomies classify pages by subject matter and style, respectively.
- Domain structure: K-means clusters mostly align with topics and do not reveal different formats.This comparison motivates complementary, human-interpretable domain taxonomies.
- Domain mixing: Task-specific mixture optimization favors Science & Technology for MMLU and Tutorial formats for HellaSwag.The framework adapts RegMix to predict which domains should be upsampled for downstream tasks.
- Results: FineWeb-Edu average accuracy increases from 54.2% to 56.2% with domain mixing, compared with a 51.6% baseline.Combining topic and format mixtures improves quality-filtered data and nearly doubles FineWeb-Edu's gain over baseline.
- Quality filters: Quality filters implicitly alter domain proportions, with FineWeb-Edu mixtures retaining up to 84% of quality-filtering gains.DCLM-fasttext's implicit mixtures perform considerably worse, indicating that quality filters can exploit information beyond broad domain effects.
2. Constructing Domains for Web-Scale Data
WebOrganizer creates compact, human-interpretable topic and format taxonomies for unstructured web corpora, refines them with model annotations and human oversight, and scales labeling through distillation. The resulting domains expose complementary structure that differs from embedding-based clustering.
- Motivation: CommonCrawl web corpora contain trillions of tokens without an inherent conceptual structure, limiting transparency about their internal composition.Small curated sources such as Wikipedia or StackOverflow do not explain most CommonCrawl content.
- Desiderata: WebOrganizer seeks compact domains aligned with meaningful human categories and sufficiently prevalent to summarize high-level corpus trends.The design rejects overly granular partitions such as URL domains.
- Taxonomy design: The framework uses complementary taxonomies: topic captures subject matter, while format captures style, intent, and venue.Each taxonomy ultimately contains 24 categories.
- Human-in-the-loop design: The taxonomies are iteratively refined from existing web directories and model annotations, with human oversight and domain expertise.The approach is flexible but requires human effort.
- Scalable annotation: Web-scale annotation is made practical by distilling Llama-3.1-405B-Instruct annotations into small classifiers.The classifiers imitate the large model using soft knowledge distillation rather than labeling the entire corpus directly with the large model.
- Domain statistics: Topic-format associations are generally weak: normalized mutual information is approximately 0.10, although pairs such as Documentation and Software Development co-occur more strongly.A value of 0 suggests independence, while 1 implies complete co-occurrence.
- Comparison to clustering: K-means clusters reflect topics more strongly than formats, with NMI(C; T) ≈0.46 versus NMI(C; F) ≈0.13.The pattern remains similar with 576 clusters, while clustering lacks inherent natural-language descriptions.
3. Optimizing Domain Mixtures for Downstream Tasks
The paper uses RegMix to learn domain mixtures tailored to downstream tasks, showing that different tasks favor distinct topic and format distributions.
- Mixture prediction: RegMix is adapted to optimize topic, format, and k-means domain mixtures for downstream-task performance.The authors train 512 small models for each domain set and fit a gradient-boosted tree regression model.
- Target tasks: The optimization directly targets MMLU, HellaSwag, and their average using normalized next-token loss on correct responses.MMLU emphasizes world knowledge and problem solving, while HellaSwag measures commonsense reasoning.
- Predicted mixtures: MMLU strongly upsamples Science & Technology, History, and Health topics, while promoting Academic Writing and Q&A Forums formats.The figure compares corpus proportions with RegMix-predicted mixtures for MMLU, HellaSwag, and both tasks.
- Predicted mixtures: HellaSwag favors smoother mixtures, notably amplifying Home & Hobbies, Fashion & Beauty, and Tutorials.Mixtures targeting both tasks tend to combine prominent components from the individual task mixtures.
4. Evaluating Pre-Training Data Curation with WebOrganizer
The evaluation tests WebOrganizer mixtures in a controlled pre-training setup and compares them with quality filtering. Domain reweighting improves performance broadly, combines effectively across topic and format, and strengthens quality filters.
- Topic × format selection: Topic-format selection assumes the two axes can be selected independently, using separate RegMix predictions for topics and formats.When a predicted pair exceeds available data, all available documents are taken and other pairs are upsampled to compensate.
- Quality filters: The method composes quality filtering with domain mixtures by setting token targets per domain and applying the quality filter separately within each subset.This effectively varies the quality threshold across domains according to the desired mixture.
- Domain mixing: Reweighting topic, format, and k-means domains improves downstream performance, with format rebalancing transferring to 6 of 7 additional tasks.The topic mixture achieves a 2.1% absolute gain over random sampling, while k-means reweighting improves overall performance by 1.6 percentage points.
- Topic and format mixtures: The combined topic-format mixture improves 8 of 9 tasks and yields a 3.0% absolute overall gain, narrowly exceeding FineWeb-Edu.It also reaches 33.2% MMLU when optimized for that task alone.
- Quality filters: Domain mixtures improve average FineWeb-Edu and DCLM-fasttext performance by 2.0% and 1.0% points, respectively.On HellaSwag, tailored mixing converts FineWeb-Edu's 1.5-point deficit versus random sampling into a 5% absolute gain.
5. Quality Filters as Implicit Domain Mixers
The paper reconstructs how quality filters implicitly alter topic and format proportions, then tests how much their gains can be reproduced through domain mixing alone. Domain mixtures approximate FineWeb-Edu better than DCLM-fasttext, while document-level selection remains stronger.
- Quality filtering can be interpreted as implicit domain mixing, because filters upsample certain domains before selecting documents within them.The paper reconstructs these mixtures from domain statistics of quality-filtered datasets.
- FineWeb-Edu deviates more strongly from the corpus in topics, whereas DCLM-fasttext amplifies more format categories.Both mixtures overlap with RegMix preferences for several domains, but their exact proportions diverge substantially.
- 73% of FineWeb-Edu’s MMLU gain and 84% of its average gain are recovered by its implicit Topic × Format mixture.The approximation replaces local document selection within domains with random sampling.
- A substantial gap remains for approximating DCLM-fasttext, indicating that its quality classifier depends more on choosing particular documents within domains.
- Held-out perplexity is substantially lower for domain mixing than for quality filtering and remains close to the baseline corpus.This supports the paper’s characterization of document-level quality filtering as a stronger intervention than domain rebalancing.
- The paper argues that data quality is task-sensitive and that WebOrganizer does not capture document-level properties such as misspellings or factual errors.These omissions may explain why DCLM-fasttext is poorly approximated by domain effects.
6. Related Work
Related work spans heuristic and quality-based data selection, curated domains, mixture optimization, and large-scale analyses of web-corpus composition. The paper positions WebOrganizer within these complementary approaches.
- Data selection: Common data-selection pipelines remove noisy websites with heuristic rules and typically follow filtering with deduplication.
- Data selection: Other selection methods use n-gram similarity, language-model perplexity, or large language models rating factuality and educational value.
- Data curation with domains: Prior work adds curated domains, but CommonCrawl often remains the majority source and can outperform domain curation.
- Data curation with domains: Related domain-selection studies combine topic labels with quality buckets or use fine-grained k-means clusters for influence-based selection.
- Data mixture optimization: Data-mixture methods tune domain proportions using validation losses, while other work partitions CommonCrawl by perplexity or infers mixtures from tokenization rules.
- Analysis of pre-training data: Other analyses examine web-corpus toxicity, source composition, age, licensing, duplicates, and personally identifiable information.
7. Conclusions
The paper introduces WebOrganizer to organize web corpora into topic and format domains, document their composition, and support domain-aware curation. It combines domain selection with quality filtering to target both domain ratios and document quality.
- WebOrganizer organizes unstructured web corpora into topic and format domains and annotates a 200B-token pre-training corpus.
- Rebalancing these annotated subsets increases downstream-task performance.
- Topic and format selection can be combined, while domain mixing can be integrated with quality filtering.
- The paper presents transparent data curation as an avenue for better documenting pre-training data and informing understanding of model strengths and weaknesses.
- The two taxonomies support analysis of web-corpus composition and its changes after quality filtering, while hierarchical or multilabel taxonomies remain future directions.
Impact Statement
The impact statement frames WebOrganizer as a transparency tool while emphasizing that taxonomy design and domain-rebalancing predictions have important limitations. The accompanying materials define topic and format categories for annotation.
- Taxonomizing web data may help researchers and the broader public understand available pre-training data for language models.
- Reducing diverse online content to discrete domains can obscure phenomena and introduce errors, biases, or misrepresentations.
- The proposed taxonomies are not presented as definitive ground truth, because multiple valid ways to define web taxonomies exist.
- Domain-rebalancing predictions are sensitive to noise from relatively few small model runs, and their transfer across model scales is uncertain.
- Domain descriptions: Topic definitions list common subtopics and ambiguous cases to produce sharper domain boundaries.
- Domain descriptions: Format definitions describe typical textual features that help models infer formats without HTML access.
- Annotation procedure: The classification prompt includes domain descriptions, page text, and URL, with randomly permuted domain choices to avoid positional bias.
B. Training Domain Classifiers
WebOrganizer trains efficient domain classifiers to reproduce large-language-model annotations, organizing web pages by topic and format. The analysis compares these taxonomies with URL-based corpus structure and k-means clusters.
- Annotation and classifier training: Llama-3.1-405B-Instruct supplies web-page annotations, including soft probabilities over category labels, for training domain classifiers.Annotations are generated with randomized category-description and few-shot-example orderings.
- Annotation and classifier training: The classifiers use a 140M-parameter embedding model fine-tuned to process long documents and predict domain labels.The embedding model has an 8192-token context window.
- Classifier evaluation: On high-confidence annotations covering 86% of topics and 79% of formats, classifier evaluation reports both average and worst-group accuracy.The evaluation uses 10K validation web pages and labels selected with at least 75% large-model confidence.
- Classifier evaluation: Two-stage training improves worst-group accuracy, while URL input provides a smaller benefit; classifier performance remains below the 405B-parameter annotator's ceiling.The annotator itself shows only 98% and 97% agreement across random seeds for topic and format annotations, respectively.
- Domain analysis: K-means document-embedding clusters primarily align with topic information rather than revealing distinct format structure.The paper analyzes topic-format relationships across the annotated 200B-token corpus.
C. RegMix Implementation
RegMix searches for effective domain mixtures by sampling candidate proportions, fitting regression models, and constraining the predicted mixture toward the corpus distribution. The implementation replaces an unstable averaging procedure with an adaptive KL-regularized search.
- Mixture sampling: Training mixtures are sampled hierarchically from a temperature-softened corpus prior using Dirichlet distributions.The procedure generates 512 random mixtures, with log α sampled uniformly from log 0.1 to log 10 and temperature τ = 2.
- Mixture evaluation: The implementation trains small 50M-parameter models on 1B-token samples from each mixture to provide downstream performance measurements.The 512 runs require approximately 360 NVIDIA H100 hours.
- Adaptive search: RegMix predicts downstream loss with boosted-tree regression and selects mixtures using a soft KL constraint rather than averaging a fixed number of lowest-loss candidates.The constraint encourages predictions to remain close to the corpus distribution and avoids sensitivity to simulation hyperparameters.
- Adaptive search: Adaptive search initializes its best mixture and search prior at the corpus prior, samples Dirichlet candidates, and limits each domain to 6.5 times its corpus proportion.The algorithm then performs a line search, smooths the search prior, and retains the best regularized mixture.
- Prediction analysis: The analysis trains regression models on all 512 mixtures and evaluates predictions with held-out mixtures using Spearman correlation.The study also compares the approach with Data Mixing Laws, which produces lower and less stable correlations.
- Domain motivation: URL-domain frequencies exhibit Zipfian long-tail behavior, motivating coarse-grained domain constructions beyond raw URL domains.The frequency statistics remove subdomains before ranking domain names.
D. Predicted Mixtures
The predicted mixtures compare topic, format, and k-means domains while examining their relationships and implicit amplification or suppression. RegMix searches are capped at 650% upsampling relative to corpus proportions.
- Predicted and implicit mixtures: The mixture experiments include predictions for a broader set of downstream tasks beyond the two primary optimization tasks.Table 9 reports the numerical mixture results, while Figure 8 visualizes additional task-specific mixtures.
- Topic-format relationships: Topic and format domains are compared through an NPMI matrix, where 0 denotes independence and 1 denotes full co-occurrence.The matrix places topics on the y-axis and formats on the x-axis.
- Topic-format relationships: K-means assignments are evaluated against topic and format annotations using NPMI matrices, with clustering primarily aligning to topics.The comparison tests whether embedding clusters capture information beyond the taxonomy dimensions.
- Predicted and implicit mixtures: Table 9 reports corpus proportions, RegMix-predicted weights, and quality-filter implicit mixtures, including amplification or suppression relative to the corpus.Parenthetical values indicate the relative change from the original domain distribution.
- Predicted and implicit mixtures: RegMix visualizations show predicted upsampling for topic and format domains as proportions of corpus distributions, capped at 650%.The cap constrains the maximum relative upsampling during the search.
E. Experimental Details
The experiments use standardized data pools, model-training settings, and multi-task evaluations to compare domain mixtures. The authors note evaluation-proxy and reproducibility limits that affect interpretation of the results.
- Data and training setup: The data pool contains 1.64T raw tokens from CommonCrawl and is preprocessed with RefinedWeb heuristic filters.The experiments use the 1b-1x DataComps-LM pool for comparisons with prior work.
- Data and training setup: Each training run selects approximately 30B tokens from a 200B-token base corpus after reserving about 1B tokens for validation.The larger selection target accounts for tokens later dropped during tokenization and packing.
- Data and training setup: Reference models contain 1,439,795,200 parameters and train on 28,795,904,000 tokens with batch size 256 and sequence length 2048.Torch compilation reduces a single training run to 183 NVIDIA H100 hours.
- Evaluation: Evaluation uses OLMES with five-shot prompts across nine tasks, including MMLU, HellaSwag, PIQA, WinoGrande, ARC, and OpenBookQA.The task suite also includes CommonSenseQA and Social IQa.
- Evaluation limitations: The DCLM Core score averages 22 tasks, but MMLU and HellaSwag may be poor proxies for identifying the best domain mixture at the 1b-1x scale.Some Core tasks are near random performance, and OLMES often produces higher accuracies on several common tasks.
- Evaluation limitations: The authors could not reproduce the exact Baseline and DCLM-fasttext performance, likely because of small data-preprocessing differences.This limits direct numerical comparison with those prior results.