Source-linked AI summary
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
TL;DR
Existing Common Crawl datasets improve benchmark accuracy by aggressively filtering data, but their reduced quantity and high duplication constrain long-horizon training. Nemotron-CC combines classifier ensembling, synthetic rephrasing, and fewer heuristic filters to improve the accuracy–quantity trade-off. Its long-horizon model outperforms Llama 3.1 8B, while the dataset provides substantially more unique real tokens than DCLM.
Problem
Aggressive filtering removes around 90% of Common Crawl data, while existing datasets contain roughly 80% near-duplicates, limiting unique-token availability for long-horizon training.
Method
The paper combines classifier ensembling, synthetic data generation, and reduced reliance on heuristic filters to create a 6.3T-token Common Crawl dataset.
Results
An 8B model trained for 15T tokens achieves higher overall accuracy than Llama 3.1 8B, including MMLU 70.3 versus 65.3; the full dataset has four times more unique real tokens than DCLM.
Takeaways & Limitations
Nemotron-CC supports stronger long-horizon pretraining while improving the trade-off between benchmark accuracy and the quantity of unique real tokens.
Takeaways & Limitations
The rephrased data was not verified for factual accuracy or fidelity to the original contents, leaving hallucination and content-diversity risks unresolved.
Abstract
from arXiv · showhide
Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show how to achieve better trade-offs between accuracy and data quantity by a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters. When training 8B parameter models for 1T tokens, using a high-quality subset of our data improves MMLU by 5.6 over DCLM, demonstrating the efficacy of our methods for boosting accuracies over a relatively short token horizon. Furthermore, our full 6.3T token dataset matches DCLM on MMLU, but contains four times more unique real tokens than DCLM. This unlocks state-of-the-art training over a long token horizon: an 8B parameter model trained for 15T tokens, of which 7.2T came from our dataset, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks. The dataset is available at https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html
1 Introduction
Nemotron-CC targets the tension between benchmark quality and token diversity by combining learned filtering, synthetic rephrasing, and fewer heuristic removals. Its 6.3T-token dataset improves short-horizon accuracy and supports stronger long-horizon training than comparable approaches.
- 1 Introduction: Aggressive filtering in recent datasets removes around 90% of Common Crawl data and leaves roughly 80% near-duplicates, limiting unique-token availability for long training horizons.The paper motivates retaining more diverse tokens because repeated exposure yields diminishing returns after four epochs compared with more unique tokens.
- 1 Introduction: Nemotron-CC transforms English Common Crawl into 6.3T tokens by combining classifier ensembling, synthetic data generation, and reduced reliance on heuristic filters.The released dataset contains 4.4T globally deduplicated original tokens and 1.9T synthetically generated tokens.
- 1 Introduction: Classifier ensembling selects a larger, more diverse set of high-quality tokens, while rephrasing reduces noise and creates fresh variants from high-quality data.The ablations also report that disabling traditional non-learned heuristics can increase high-quality token yield without hurting accuracy.
- 1 Introduction: The paper frames the pipeline as a learned flywheel in which improved data and language models iteratively support better synthetic data and quality classification.This is presented as the guiding principle for moving beyond a static heuristic pipeline.
2 Methods
Nemotron-CC increases usable Common Crawl volume by preserving more high-quality tokens, combining model-based quality labeling with synthetic generation and limited heuristic filtering. These choices produce a 6.3T-token dataset with substantially more unique tokens than comparable datasets.
- 2.1 HTML-to-text Extractor & Filter: +28.6% more high-quality tokens are yielded by Justext than Trafilatura, with 127B versus 104B high-quality tokens.The comparison uses FineWeb-Edu classifier scores and prioritizes absolute high-quality-token counts.
- 2.1 HTML-to-text Extractor & Filter: 18.1% of FineWeb-Edu-classified high-quality tokens are removed by the conventional heuristic filtering pipeline.
- 2.1 HTML-to-text Extractor & Filter: The pipeline therefore applies heuristic filters only to low-quality splits while retaining model-identified high-quality tokens.
- 2.2 Model-based Quality Labeling: Three classifiers assign ranked quality buckets, which are regrouped into downstream-quality categories using continued-pretraining evaluations.Bucket quality is measured by mixing 34% evaluated data with 66% default training data and averaging downstream-task performance.
- 2.3 Synthetic Data Generation: Synthetic generation rephrases low-quality documents to reduce noise and errors while preserving useful information, and produces diverse variants from high-quality data.Post-processing removes incomplete, malformed, or under-length outputs before incorporation into the dataset.
- 2.4 Putting It All Together: 6.3T tokens comprise 4.4T globally deduplicated original tokens and 1.9T synthetically derived tokens in Nemotron-CC.
3 Experiments
The experiments train standard 8B-parameter transformer language models using the open-source Megatron-LM library.
- 3 Experiments: Experiments use the open-source Megatron-LM library to train standard 8B-parameter transformer language models.
- 3 Experiments: Megatron-LM provides the open-source training framework for these transformer experiments.
- 3 Experiments: The model scale evaluated in this setup is 8B parameters.
Dataset ARC-E ARC-C H W RACE PIQA SIQA CSQA OBQA MMLU Avg
Experiments compare Nemotron-CC variants with DCLM, FineWeb-Edu, and Llama 3.1 across ten benchmarks, while ablations isolate extraction, filtering, classifier ensembling, and synthetic-data effects.
- Short Token Horizon (1T): 5.6 MMLU and 3.1 average-score gains over DCLM were obtained with Nemotron-CC-HQ in 1T-token training.Nemotron-CC-HQ improves over DCLM and FineWeb-Edu on every listed task except RACE.
- Short Token Horizon (1T): 4× more unique real tokens let the full 6.3T-token Nemotron-CC dataset match DCLM on MMLU and average accuracy.The authors expect this higher token diversity to help in data-constrained 15T-token training.
- Long Token Horizon (15T): MMLU 70.3 vs. Llama 3.1’s 65.3 after both models were trained for 15T tokens, with Nemotron-CC contributing 7.2T tokens.The Nemotron-CC model also achieves higher average accuracy than Llama 3.1 8B.
- Extractor & Filter Comparison: +2% MMLU resulted from removing filters only from high-quality data, while combined extractor and filter changes preserved downstream accuracy and increased token yield.The authors therefore retain filtering for low-quality data but remove it for high-quality data.
- Classifiers Comparison: 25% of tokens were labeled high-quality by the classifier ensemble, versus 9% for individual classifier selection, while average task performance remained highest.The ensemble combines classifiers with different quality standards and domain preferences.
- Evaluating Synthetic Data: Replacing four of eight high-quality-data epochs with synthetic data improved accuracy on most benchmarks, though some tasks showed slight declines.The authors suggest fresh tokens and styles may improve learning, while noting possible misinformation from synthesis.
4 Related Work
The paper builds on model-based filtering and synthetic-data research while targeting longer token horizons and larger quantities of English Common Crawl data.
- 4 Related Work: FineWeb-Edu and DCLM are the paper’s primary comparisons because they use model-based filtering to improve Common Crawl data quality.The paper seeks to improve their balance between accuracy and data quantity.
- 4 Related Work: Unlike concurrent Zyda-2, which combines existing datasets, this work focuses on creating a new English Common Crawl dataset.The comparison concerns dataset-creation strategy rather than only downstream filtering or deduplication.
- 4 Related Work: Prior synthetic-data work shows that rephrased or generated text can support pretraining quality, and this paper specializes prompting for low- and high-quality web data.The paper adopts the idea that compact models can rephrase web data into diverse, high-quality synthetic data.
5 Conclusion
The paper presents Nemotron-CC as a public English Common Crawl dataset designed to improve the trade-off between benchmark accuracy and unique-token quantity for long-horizon pretraining.
- 5 Conclusion: The method combines classifier ensembling, low- and high-quality document rephrasing, and reduced reliance on non-learned heuristics.The dataset is split by quality level and by original versus synthetic data type.
- 5 Conclusion: The released dataset enables further study of quality-versus-diversity trade-offs and short- versus long-horizon curricula.The paper releases the data publicly for community experimentation.
6 Limitations
The study’s main limitations concern limited exploration of pipeline choices, unverified fidelity of rephrased data, English-only scope, and the absence of dataset decontamination.
- 6 Limitations: Only one classifier-ensembling and quality-bucketing strategy was tested, and not all pipeline components received ablations.The authors identify possible improvements in high-quality sensitivity, language identification, and medium-quality-data rephrasing.
- 6 Limitations: Rephrased data was not checked for factual accuracy or fidelity, leaving hallucination and content-diversity risks unresolved.The paper specifically calls for mitigation of these risks in future work.
- 6 Limitations: The methods were evaluated only on English text, so adaptation to other languages remains untested.The authors state that more work is needed for multilingual application.
- 6 Limitations: The dataset was not decontaminated because consensus on the procedure and its impact remains unsettled for large models and long horizons.The comparison datasets and Llama 3.1 were also trained on contaminated data.
C Bucket Comparison
The bucket ablations test how individual data buckets affect benchmark accuracy while preserving the broader pretraining distribution.
- Each ablation continues a 900B-token checkpoint for 50B tokens, with 34% from the tested bucket and 66% from the original distribution.This design limits distribution shift while measuring bucket-specific effects.
- The study evaluates the benchmark accuracy of each of the 20 data buckets.
- Figure 4 reports the resulting bucket ablation accuracies.
D Training Details: Ablations
The ablation experiments train 8B-parameter transformer models for 1T tokens using a specified Megatron-LM configuration and controlled data blends.
- Training setup: The experiments train 8B-parameter transformer LLMs for 1T tokens using the open-source Megatron-LM library.
- Training setup: The model uses 32 layers, hidden dimension 4096, 32 attention heads, SwiGLU activations, and grouped-query attention with 8 query groups.
- Training setup: A single training run takes about 40 hours on 1024 NVIDIA H100 GPUs.
- Data blends: The data-blend experiments vary only the 73% English Common Crawl portion.
E Long-Horizon Curriculum Details
The 15T-token run uses a two-phase curriculum that draws 7.17T tokens from English Common Crawl across progressively stricter quality levels.
- Curriculum: 7.17T of the 15T training tokens come from English Common Crawl, representing 47.8% of the total.
- Curriculum: The first 9T-token phase uses 5.31T English Common Crawl tokens, while the second 6T-token phase uses 1.86T.
- Curriculum: The first phase includes medium, medium-high, and high-quality real and synthetic data, whereas the second uses only high-quality real and synthetic data.
- Dataset coverage: The main datasets use 99 Common Crawl snapshots spanning CC-MAIN-2013-20 through CC-MAIN-2024-30.
G Extractor & Filter Ablation
The extractor and filter ablation examines benchmark task coverage and synthetic-data prompt variants, followed by post-processing that removes malformed or insufficient outputs.
- Evaluation: The evaluation average covers ARC-Easy, ARC-Challenge, Hellaswag, Winogrande, RACE, PIQA, Commonsense QA, and Openbook QA.
- Classifier usage: The ablation and analysis use FineWeb-Edu classifier labels, but the final dataset preparation does not use that classifier.
- Prompt templates: The question-generation template requests up to eight questions spanning factual, comparison, comprehension, and problem-solving formats.
- Prompt templates: The prompt suite generates paraphrases, distilled factual lists, knowledge passages, extractive knowledge, and diverse question-answer pairs.