Source-linked AI summary
OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, Jimmy Ba
TL;DR
Open-source models and datasets lag in quantitative reasoning, while existing web preprocessing does not faithfully preserve mathematical notation. OpenWebMath addresses this gap with a publicly released, carefully processed mathematical web corpus and evaluations showing strong performance relative to much larger general-domain training data.
Problem
Minerva and its training datasets were not publicly released, while open-source models lag in quantitative reasoning and existing preprocessing fails to preserve mathematical notation.
Method
The authors extract Common Crawl documents while preserving LaTeX, apply language, quality, and mathematical-content filters, deduplicate them, and release the resulting dataset and reproducible pipeline.
Results
Models trained on OpenWebMath outperform models trained on 20x more general-domain tokens in mathematics.
Takeaways & Limitations
OpenWebMath provides an open dataset and documented pipeline for studying mathematical language-model pretraining and reasoning.
Takeaways & Limitations
The pipeline was run only once, so many design choices lack empirical justification and no ablation study is provided.
Abstract
from arXiv · showhide
There is growing evidence that pretraining on high quality, carefully thought-out tokens such as code or mathematics plays an important role in improving the reasoning abilities of large language models. For example, Minerva, a PaLM model finetuned on billions of tokens of mathematical documents from arXiv and the web, reported dramatically improved performance on problems that require quantitative reasoning. However, because all known open source web datasets employ preprocessing that does not faithfully preserve mathematical notation, the benefits of large scale training on quantitive web documents are unavailable to the research community. We introduce OpenWebMath, an open dataset inspired by these works containing 14.7B tokens of mathematical webpages from Common Crawl. We describe in detail our method for extracting text and LaTeX content and removing boilerplate from HTML documents, as well as our methods for quality filtering and deduplication. Additionally, we run small-scale experiments by training 1.4B parameter language models on OpenWebMath, showing that models trained on 14.7B tokens of our dataset surpass the performance of models trained on over 20x the amount of general language data. We hope that our dataset, openly released on the Hugging Face Hub, will help spur advances in the reasoning abilities of large language models.
1 INTRODUCTION
The paper introduces OpenWebMath as an open alternative to privately released mathematical pretraining data, combining preserved mathematical notation, quality filtering, deduplication, and publicly documented resources.
- OpenWebMath provides an open alternative to the Math Web Pages dataset used to train Minerva.
- The pipeline extracts text while preserving mathematical content as LaTeX, filters for high-quality English mathematical documents, and deduplicates the result.
- 14.7B tokens of high-quality mathematical web text are publicly released on the Hugging Face Hub.
- The authors extensively document the pipeline and open-source the code needed to reproduce their results.
- Models trained on OpenWebMath are evaluated for dataset quality and per-token effectiveness against existing mathematical pretraining datasets and mixtures.
2 RELATED WORK
Related work covers mathematical datasets and benchmarks alongside web-data pipelines for extraction, filtering, and deduplication. These prior resources establish both the available mathematical corpora and the processing challenges OpenWebMath addresses.
- Mathematics datasets: Existing mathematical datasets include informal problems, formal proofs, textbook-derived material, and large web-based collections.
- Mathematics benchmarks: Mathematics benchmarks assess language models on informal problems, proofs, automatically verifiable solutions, and STEM knowledge.
- Web data processing pipelines: Common Crawl-based datasets require extensive filtering and deduplication because the crawl contains substantial low-quality and duplicate content.
- Web data processing pipelines: Plain-text extraction from HTML can introduce boilerplate or omit mathematical content such as LaTeX equations.
- Web data processing pipelines: Web-data pipelines use language identification, perplexity, classifier, and rule-based methods to filter documents for quality.
3 BUILDING OPENWEBMATH
OpenWebMath builds a high-recall Common Crawl pipeline that preserves mathematical content, especially LaTeX, while filtering and processing diverse web documents. The pipeline combines staged prefiltering, custom extraction, boilerplate removal, quality filtering, and deduplication.
- Objectives: OpenWebMath targets broad mathematical coverage from Common Crawl while preserving formatting such as LaTeX equations.The dataset includes core mathematical documents and interdisciplinary pages containing mathematical formulas.
- Pipeline: The five-stage pipeline prefilters HTML, extracts text and equations, applies language and quality filters, and removes duplicates.Prefiltering reduces the number of documents requiring expensive processing.
- Evaluation: 1.4B-parameter models trained for 14.7B tokens on OpenWebMath or a 50/50 ProofPile–OpenWebMath mixture outperform Pythia 1.4B trained on 300B Pile tokens in perplexity evaluations.The reported evaluation also covers accuracy on different mathematics benchmarks.
- Text extraction: The custom extraction pipeline combines LaTeX extraction, plain-text extraction, DOM processing, and line processing to remove boilerplate while retaining mathematical content.DOM processing removes invisible elements, buttons, link clusters, and problematic elements; line processing removes common boilerplate phrases.
- LaTeX extraction: LaTeX extraction handles MathJax delimiters, equation environments, special element attributes, image URLs, and MathML or annotation blocks.Common Crawl HTML lacks JavaScript-rendered changes, so the pipeline detects delimiters and supports multiple encoding schemes.
- Filtering: MathScore predicts whether documents contain mathematical content from surrounding words after LaTeX is removed, retaining technical pages without extractable equations.Manual inspection found documents scoring below 0.2 very unlikely to contain useful mathematical content.
4 DATASET ANALYSIS
OpenWebMath spans diverse mathematical and technical web sources, and models trained on it show strong downstream performance relative to general and other mathematical datasets.
- Dataset scale: 14.7B tokens make OpenWebMath slightly smaller than Minerva’s 17.5B-token Math Web Pages and comparable to ProofPile’s 14.2B LLaMA tokens.The datasets have very little overlap, so OpenWebMath adds mathematical tokens unavailable to the open-source community.
- Data composition: OpenWebMath covers 131,206 domains, with 46% of characters concentrated in the top 100 domains.Its most common sources include discussion forums, blog posts, and scientific papers.
- Data composition: Most sampled documents concern mathematics, while other documents span physics, computer science, statistics, chemistry, and economics; 12% fit none of these categories.The analysis used embeddings, k-Means clustering, and GPT-based cluster classification.
- Data composition: Forum pages are the most prevalent website type, followed by substantial educational and reference content.GPT classified sampled documents and produced similar results when classifying the top 100 domains.
- Downstream performance: Models trained on OpenWebMath lead on perplexity benchmarks, with the OpenWebMath–ProofPile mixture performing best.OpenWebMath models also significantly outperform models without OpenWebMath on MATH-Algebra-Easy and LILA-multiarith.
5 CONCLUSION
OpenWebMath is an open 14.7B-token dataset of high-quality mathematical web documents with a documented, reproducible processing pipeline. Models trained on it outperform models trained on 20x more general-domain tokens in mathematics.
- Conclusion: OpenWebMath provides 14.7B high-quality mathematical web documents and documents its extraction, boilerplate-reduction, and filtering pipeline.The dataset includes Q&A forums, educational documents, blogs, and technical domains such as mathematics, physics, and computer science.
- Conclusion: Models trained on OpenWebMath outperform models trained on 20x more general-domain tokens in mathematics.The authors hope the dataset contributes to language models with improved mathematical reasoning capabilities.
Method Runtime (s) Source Code Link
Resiliparse was selected as the HTML text extraction tool after outperforming alternatives in efficiency on a 1k-document evaluation.
- Method Runtime (s) Source Code Link: Resiliparse was by far the most efficient tool among those evaluated on 1k HTML documents.This result motivated its use in the OpenWebMath processing pipeline.
A LIMITATIONS AND FUTURE WORK
OpenWebMath’s construction reflects substantial cost, subjectivity, and scope constraints. The pipeline was run only once, may filter too aggressively, and is English-only.
- Limitations: The pipeline was run only once because extracting data from all Common Crawl shards was expensive, leaving design choices without empirical justification or ablation studies.The authors also identify subjective choices in defining mathematical and high-quality documents and setting near-deduplication thresholds.
- Limitations: The prefilter reduces processed HTML documents to under 1% of Common Crawl, which may be too aggressive.Filtering and extraction choices involve practical trade-offs between cost and accuracy.
- Limitations: OpenWebMath is English-only, limiting its applications for researchers and users who speak other languages.The dataset also contains only web mathematics text rather than associated non-text content.
B TEXT EXTRACTION
The extraction pipeline was selected to preserve mathematical notation while removing boilerplate, and it iteratively expanded support for LaTeX formats found in Common Crawl.
- Common Crawl WET files were rejected because they mishandle LaTeX and provide no customization, while jusText removed boilerplate too aggressively.
- The extraction process iteratively refined rules by filtering documents containing \frac and investigating pages with no extractable LaTeX.
- MathJax was the most common LaTeX format in OpenWebMath, using user-defined delimiters for mathematical equations.
Math Keywords
The keyword and MathJax-handling strategy combines script detection, LaTeX-command detection, and a mathematical-content classifier, but edge cases remain.
- The pipeline considers MathJax scripts, common LaTeX commands, and MathScore classification when deciding whether dollar signs delimit equations.
- MathJax script detection can fail when scripts are nested or differently named, while the other methods can miss equations whose surrounding text appears nonmathematical.
- Extraction and filtering interact because formatting errors or boilerplate can lower classification and perplexity scores, excluding otherwise high-quality mathematical documents.The paper recommends tuning extraction on diverse documents before filtering to reduce this bias.
E DATASHEET
The datasheet records OpenWebMath’s collection period and preprocessing documentation, while noting that raw data is retained for reference.
- The paper provides a datasheet for OpenWebMath following the framework of Gebru et al.
- OpenWebMath uses Common Crawl shards gathered between 2013 and 2023.
- The dataset’s raw data was saved alongside the preprocessed data, with preprocessing details provided in section 3.5.
DISTRIBUTION
OpenWebMath is intended for distribution beyond its originating entity through the Hugging Face Hub.
- The dataset will be available on the Hugging Face Hub for NLP practitioners.