MathPile: A Billion-Token-Scale Pretraining Corpus for Math

• Existing open corpora generally target other domains, while some math-specific training corpora are not open-sourced and OpenWebMath draws only from web pages.Source: §1 · pp. 1–2
• MATHPILE combines textbooks, lecture notes, mathematics arXiv papers, and selected web and community sources, extending beyond web pages and formal proving.Source: §1 · p. 2
• The corpus applies source-specific preprocessing and prefiltering before global language identification, filtering, cleaning, and deduplication, and documents quality annotations and contamination checks.Source: §1 · p. 2
• Up to 5% average improvement was observed across mathematical reasoning benchmarks in continual pre-training experiments with MATHPILE.Source: §1 · p. 2

• Textbook collection combines 38 K-12 and 369 college-level textbooks with 467 course handouts or lecture notes, producing 874 documents after parsing and cleaning.Source: §2 · p. 3
• OpenPhi synthetic textbooks were filtered using $$ alongside common mathematical-expression markers, yielding 3,889 documents from 124,493.Source: §2 · p. 3
• The corpus includes 347,945 cleaned arXiv LaTeX papers, about 8.5 billion tokens, selected from 50 sub-subjects across six fields.Source: §2 · p. 3
• Other collected sources include 106,881 English mathematical Wikipedia entries, ProofWiki definitions and theorem-proof pairs, and quality-filtered StackExchange discussions.The reported collections include about 0.8 billion Wikipedia tokens, 7.6 million ProofWiki tokens, and 254 million StackExchange tokens.Source: §2 · p. 4
• Rule-based extraction yielded 4,307 SlimPajama-C4 and 72,137 SlimPajama-CommonCrawl documents, totaling about 633 million tokens.Source: §2 · p. 5

3 Global Data Processing

• The global workflow follows source-specific preprocessing with language identification, filtering, and deduplication across the corpus.Source: §3 · p. 5

3.1 Language Identification

• Custom language-identification thresholds addressed false non-English classifications caused by mathematical symbols and removed about 8,400 documents, totaling 231 million tokens.Thresholds were 0.1 for Wikipedia and StackExchange, 0.3 for arXiv, and 0.5 for Common Crawl; none were used for manually verified ProofWiki and Textbooks.Source: §3.1 · p. 5

3.2 Data Cleaning and Filtering

• Domain-specific cleaning and filtering assigned quality annotations to Wikipedia and Common Crawl documents and removed about 1,100 documents, or 17 million tokens.Source: §3.2 · p. 6

3.3 Data Deduplication

• About 714 million tokens were eliminated by deduplication; ProofWiki near-duplicates were retained when they represented distinct lemmas, proofs, or definitions.Source: §3.3 · p. 6

3.4 Data Contamination Detection

• The benchmark scan found 23 MATH and 2 MMLU-STEM questions, no contamination in other benchmarks, and removed all detected exact matches.Questions were checked through line-level exact matching; benchmark answers were included except for multiple-choice tests where only questions were used.Source: §3.4 · pp. 6–7

4.1 Statistics

• 9.5 billion tokens in 29GB of corpus were obtained from 2.2TB of raw data; arXiv is the largest component and Textbooks the smallest, though exceptionally high-quality.Table 3 presents component statistics including document and token counts.Source: §4.1 · p. 7, Table 3 · p. 7
• Document-length distributions are relatively smooth across sources; arXiv and Textbooks documents tend to be longer, while ProofWiki and StackExchange documents tend to be shorter.ProofWiki has many similar-length, short documents because its definitions, lemmas, and proofs follow a fixed format.Source: §4.1 · p. 7, Figure 3 · p. 7

4.2 Continual Pre-training Experiments

• Continual pre-training used Mistral-7B-v0.1 for three epochs with 4,096-token chunks and evaluated six math benchmarks using the same few-shot prompting and greedy decoding.The experiments used a global batch size of 1,024 and a maximum learning rate of 1e-5.Source: §4.2 · pp. 7–8
• MathPile subsets generally improved diverse math benchmarks, but StackExchange did not improve GSM8K and arXiv slightly improved GSM8K and MMLU-MATH while degrading MATH, SAT-MATH, and MathQA.Textbooks, Wikipedia, StackExchange, and CC improved GSM8K and MATH, but not the other benchmarks.Source: §4.2 · p. 8
• AMPS and DM-Mathematics pre-training degraded performance, while sampled OpenWebMath improved less than same- or smaller-scale Textbooks and Wikipedia subsets.The authors attribute the synthetic-set result to narrow, monotonous problem sets and suggest such datasets generally work better combined with broader corpora; they suggest OpenWebMath may need more tokens for substantial gains.Source: §4.2 · p. 8, Table 4 · p. 8
• A 10-billion-token, one-epoch pre-training run requires approximately 1,760 NVIDIA A100 GPU-hours, constraining further experiments.The authors cite limited computational resources as the reason they did not extensively test the entire dataset or data combinations.Source: §4.2 · p. 8
• Skipping the processing pipeline reduced Mistral’s mathematical reasoning abilities even at 2.18B training size, and correct LaTeX displays were vital for enhancing reasoning.The Wikipedia testbed compared raw and cleaned data and problematic versus correct LaTeX displays.Source: §4.2 · pp. 8–9, Table 5 · p. 8

Pre-training Corpora for Language Models

• Language-model corpora include broad collections such as Pile, web-focused datasets such as C4, and specialized multilingual or code datasets such as ROOTS and The Stack.Some large corpora and data used by recent models are closed-source, while RedPajama and SlimPajama release corpora.Source: §5 · p. 9

Pre-training Benchmarks and Corpora for Mathematical Reasoning

• Mathematical resources span arithmetic-to-competition benchmarks and corpora targeting problem-solving or theorem proving, while MATHPILE aims to fill gaps with a high-quality, diverse mathematical corpus.The related work contrasts public resources such as AMPS, ProofPile, and OpenWebMath with Minerva and MathMix, which are not public.Source: §5 · p. 9

6 Conclusion and Limitations

• MATHPILE combines diverse mathematical sources with preprocessing, prefiltering, language identification, cleaning, filtering, deduplication, and benchmark-test contamination detection.The authors aim for the corpus to enhance mathematical reasoning in language models, alone or with other datasets.Source: §6 · p. 9
• Creation decisions were made empirically and may not be optimal; the dataset is too small for extra-large-model training and is primarily English.The authors identify expanding common-crawl subsets, building high-quality datasets for other languages, data mixing, and model-based corpus refinement as future directions.Source: §6 · p. 9

Checklist

• The checklist says the paper discussed limitations and potential negative societal impacts were not discussed.Source: Checklist · p. 18
• The checklist reports that code, data, reproduction instructions, training details, and compute resources were provided, but error bars were not reported.Source: Checklist · p. 18
• The checklist marks the estimated participant wage and total compensation as not applicable.Source: Checklist · p. 18

MOTIVATION

• MATHPILE aims to enrich the open-source community with a math-centric pretraining dataset.Source: MOTIVATION · p. 20
• The dataset is intended to enhance mathematical language modeling.Source: MOTIVATION · p. 20
• MATHPILE can be used alone or together with general-domain corpora such as books and GitHub code.Source: MOTIVATION · p. 20

COMPOSITION

• Instances are text-only math-focused documents, including papers, textbooks, lecture notes, definitions, theorems, proofs, articles, Q&A, and webpages.Source: COMPOSITION · p. 20
• About 903,000 documents and 9.5 billion tokens comprise MATHPILE.Source: COMPOSITION · p. 20

PREPROCESSING

• Some documents, particularly web-sourced ones, may still fall short of the dataset’s quality standards.Source: COMPOSITION · p. 21

COLLECTION

• Textbooks were manually collected with quality checks, while data also came primarily from arXiv and the Internet Archive; Common Crawl data came from SlimPajama.Source: COLLECTION · p. 21

PREPROCESSING

• Processing included language identification, cleaning, filtering, deduplication, benchmark leakage detection, and removal of identified contaminated examples.Source: PREPROCESSING · p. 22
• MATHPILE is tailored for English, limiting its applicability in multilingual contexts, and imperfect cleaning may exclude valuable documents.Source: PREPROCESSING · p. 22

DISTRIBUTION

• MATHPILE has been made available through the HuggingFace Hub.Source: DISTRIBUTION · p. 22

MAINTENANCE

• The datasheet for MATHPILE follows Gebru et al. [21].Source: Table 6 · p. 23

B Ethics Statement

• MATHPILE follows source licenses when they are more restrictive than CC BY-NC-SA 4.0; otherwise, the released dataset uses CC BY-NC-SA 4.0.Source: B Ethics Statement · p. 24
• The authors did not obtain explicit consent from every author for some collected materials, including publicly available open-source textbooks, and acknowledge potential copyright issues.Source: B Ethics Statement · p. 24
• To mitigate risks, the authors prioritize clearly open-licensed or public-domain sources, invoke fair-use principles for some copyrighted or non-commercial material, and accept removal requests.Source: B Ethics Statement · p. 24
• The authors state that they bear full responsibility for potential violations of rights or licensing issues arising from the dataset.Source: B Ethics Statement · p. 24

C Examples of MATHPILE

• Examples span Common Crawl, Wikipedia, textbooks, ProofWiki, arXiv, and StackExchange documents.The examples include ProofWiki theorems, proofs, and definitions, as well as StackExchange questions with responses.Source: C Examples of MATHPILE · p. 24, Figure 4 · p. 25, Figure 5 · p. 26

D Details for Corpus Collection and Processing

• ArXiv processing removes comments, figure environments, acknowledgements, and references, restores many macros, and retains only main-body content.The appendices also list arXiv subjects, StackExchange sites, and human involvement and automation in collection and global processing.Source: D Details for Corpus Collection and… · p. 31, Table 7 · p. 31, Table 8 · p. 31

E Example for Quality Annotation

• A cleaned document is shown with quality annotations.The appendix also presents an example of HTML-to-text processing using html2text compared with Resiliparse plus DOM parsing.Source: Figure 11 · p. 32, E Example for Quality Annotation · p. 33, Figure 12 · p. 33

F Examples of Duplicates Encountered in the Deduplication Process

• Deduplication examples cover near-duplicate matches across sources, while contamination examples show benchmark test-set matches in the corpus and OpenWebMath.The examples include exact matches from MATH and MMLU-STEM test sets; the Common Crawl examples show repeated questions but no duplicate answers.Source: F Examples of Duplicates Encountered… · p. 33, Table 11 · p. 34, Table 12 · p. 35

G Evaluation of Continal Pre-trained Models on General Langauge Benchmarks

• The evaluation reports accuracy (acc norm) on PIQA, ARC-Easy, ARC-Challenge, SciQ, and HellaSwag.The experiments used EleutherAI’s lm-evaluation-harness.Source: G Evaluation of Continal Pre-trained… · p. 43
• General-language abilities showed no significant degradation after continual pre-training on MathPile subsets, with some benchmark improvements and slight metric declines.The authors note that these experiments used math-specific data exclusively and did not necessarily expect general-benchmark improvements.Source: G Evaluation of Continal Pre-trained… · p. 43, Table 20 · p. 43