PaperlayerView exact revision on arXiv

Source-linked AI summary

MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources

Huu Nguyen, Victor May, Harsh Raj, Marianna Nezhurina, Yishan Wang, Yanqi Luo, Minh Chien Vu, Taishi Nakamura, Ken Tsui, Van Khue Nguyen, David Salinas, Aleksandra Krasnodębska, Christoph Schuhmann, Mats Leon Richter, Xuan-Son, Vu, Jenia Jitsev

arXiv:2509.25531v5cs.CLcs.AIcs.LG

TL;DR

MixtureVitae addresses the shortage of high-performance pretraining data that mitigates legal risk. It combines permissive-first, risk-mitigated sources with synthetic instruction and reasoning data, and its models outperform permissive baselines while approaching leading non-permissive ones. In controlled experiments, its 1.7B base model outperforms SmolLM2-1.7B-Instruct on GSM8K, HumanEval, and MBPP despite training on about 36 times fewer tokens.

  • Problem

    High-performing pretraining datasets often rely on mixed-license or opaque web data, while strictly permissive corpora tend to lag on reasoning-heavy benchmarks.

    Source: §1 · p. 2

  • Method

    MixtureVitae combines permissively licensed, public-domain, and risk-mitigated sources with targeted synthetic instruction and reasoning data.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    In controlled 300B-token experiments, the 1.7B MixtureVitae base model outperforms SmolLM2-1.7B-Instruct on GSM8K, HumanEval, and MBPP despite SmolLM2's ≈11T training tokens.

    Source: §5 · p. 13

  • Takeaways & Limitations

    The three-tier licensing scheme and shard-level annotations offer a template for risk-mitigated mixtures and a reusable blueprint for compliant pretraining.

    Source: §5 · p. 13

    Model checkpoints and training logs are not included in the initial submission due to size and anonymity constraints, though the authors plan to release them upon publication.

    Source: §6.3 · p. 14

Abstract

from arXiv · show

We present MixtureVitae, an open-access pretraining corpus built to minimize legal risk while providing strong downstream performance. MixtureVitae follows a permissive-first, risk-mitigated sourcing strategy that combines public-domain and permissively licensed text (e.g., CC-BY/Apache) with carefully justified low-risk additions (e.g., government works and EU TDM-eligible sources). MixtureVitae adopts a simple, single-stage pretraining recipe that integrates a large proportion of permissive synthetic instruction and reasoning data-signals typically introduced during post-training and generally scarce in permissive web corpora. We categorize all sources into a three-tier scheme that reflects varying risk levels and provide shard-level provenance metadata to enable risk-aware usage. In controlled experiments using the open-sci-ref training protocol (fixed architectures and hyperparameters; 50B and 300B token budgets across 130M-1.7B parameters), models trained on MixtureVitae consistently outperform other permissive datasets across a suite of standard benchmarks, and at the 1.7B-parameters/300B-tokens setting, they surpass FineWeb-Edu and approach DCLM late in training. Performance is particularly strong on MMLU and on math and code benchmarks: a 1.7B model pretrained on 300B MixtureVitae tokens matches or exceeds a strong 1.7B instruction-tuned baseline on GSM8K, HumanEval, and MBPP, despite using over 36 times fewer tokens (300B vs. ~11T). Supported by a thorough decontamination analysis, these results show that permissive-first data with high instruction and reasoning density, tiered by licensing and provenance-related risk, can provide a practical and risk-mitigated foundation for training capable LLMs, reducing reliance on broad web scrapes without sacrificing competitiveness. Code: https://github.com/ontocord/mixturevitae

1 INTRODUCTION

The paper addresses the lack of high-performance pretraining datasets that mitigate legal risk, a gap that leaves practitioners choosing between performance and compliance. It introduces MixtureVitae, a permissive-first, risk-mitigated corpus that performs strongly against permissive baselines and competitively against non-permissive datasets.

1 INTRODUCTION

  • MixtureVitae combines permissively licensed, public-domain, and risk-mitigated text with targeted synthetic data to address the scarcity of organic reasoning and conversational dialogue in permissive sources.The synthetic data is derived from permissive models and sources.Source: §1 · p. 2
  • On math and code benchmarks, the 1.7B MixtureVitae base model matches or exceeds a strong 1.7B instruction-tuned baseline despite using over 36× fewer tokens.The comparison is against SmolLM2, trained on roughly 11T tokens.Source: §1 · pp. 2–3
  • MixtureVitae significantly outperforms other permissively licensed baselines, with its performance gap widening as model scale increases.The evaluation trains models from 130M to 1.7B parameters and compares them with prominent open datasets.Source: §1 · p. 2
  • The corpus uses auditable provenance tiers and a positive-inclusion pipeline rather than retroactive filtering.The paper also reports a large-scale 13-gram decontamination analysis and releases shard-level provenance metadata and curation code.Source: §1 · pp. 2–3
  • The paper frames its contribution as reducing reliance on indiscriminately scraped, high-risk copyrighted data while retaining capable-model performance.Its controlled comparisons use the open-sci-ref protocol across 130M–1.7B parameter scales.Source: §1 · p. 2

2 DATASET

MixtureVitae combines curated expertise, diverse web text, and instruction and reasoning data under a permissive-first, risk-mitigated sourcing strategy. Its curation pipeline selects sources by provenance and licensing, applies content and redundancy filters, and prepares domain-aware training examples.

2 DATASET

  • The corpus combines curated domain expertise, diverse web data, and instruction-following and reasoning datasets to support reasoning and task completion.The major categories are shown in Figure 1, with token counts, license distribution, and synthetic-data usage detailed in other figures.Source: §2 · p. 3, Figure 1 · p. 3

2.1 DATA SOURCES

  • Source selection prioritizes clear permissive licensing or public-domain status, then favors high-quality data and dense instruction and reasoning content.The stated capability targets include GSM8K and MMLU.Source: §2.1 · pp. 3–4

2.1.1 WEB-SCALE CORPORA

  • The web subset includes Nemotron-CC, MGACorpus, and FineFineWeb, plus synthetic rephrasings of Nemotron-CC and MGACorpus text.The paper also describes curated financial, multilingual, scientific, medical, code, patent, mathematics, video-transcript, news, and law sources.Source: §2.1.1 · p. 4, §2.1.2 · p. 4

2.1.3 INSTRUCTION AND REASONING DATASETS

  • Instruction and reasoning data includes general instruction and preference collections, reasoning corpora, domain-specific question-answer datasets, and math and coding instruction sets.The synthetic and web-rewrite data is generated from permissive or public-domain seeds using permissively licensed models.Source: §2.1.3 · p. 4

2.1.4 LICENSING TIERS AND RISK PROFILES

  • The three licensing tiers distinguish explicit open licenses and public-domain sources, corpora with upstream provenance opacity, and civic or governmental works.Tier 2(b) covers about 4% of the dataset and is isolated so users requiring strictly permissive generator and seed provenance can exclude it.Source: §2.1.4 · pp. 5–6

2.2.1 PERMISSIVENESS FILTERING

  • Web data is positively selected through allowlisted government and international domains, known permissively licensed sites, and license-keyword searches that exclude restrictive terms.The paper says this upfront approach minimizes the risk of including paywalled or opted-out content.Source: §2.2.1 · p. 6

2.2.2 SAFETY FILTERING

  • Safety filtering removes obscene, adult, and CSAM-related content and targets selected Wikipedia articles to reduce memorization of facts about people.The authors also evaluate the final model's safety profile through standard red-teaming.Source: §2.2.2 · p. 6

2.2.4 DEDUPLICATION

  • Deduplication removes exact boilerplate and repeated sentences but avoids cross-dataset fuzzy deduplication to preserve near-duplicates and stylistic and domain diversity.Documents with sentence duplication rates above 75% are discarded; training examples are then packed by base-URL clusters after within-document concatenation.Source: §2.2.4 · p. 6, §2.2.5 · p. 7

3 EXPERIMENTS

This section evaluates MixtureVitae in controlled dataset comparisons, reports its downstream performance, and tests whether that performance persists after decontamination and changes to mixture components.

3.1 EXPERIMENTAL SETUP

  • The controlled study fixes architecture and hyperparameters, varies the dataset, and separately trains models at 50B- and 300B-token budgets.Model sizes span 0.13B to 1.7B parameters; separate runs permit budget-specific learning-rate schedules.Source: §3.1 · p. 7

3.2 EXPERIMENT RESULTS

  • At 300B tokens, MixtureVitae outperforms all permissive baselines overall and approaches FineWeb-Edu and DCLM late in training, while non-permissive datasets retain the highest overall scores.MixtureVitae is particularly strong on MMLU; only Nemotron-CC-HQ catches up around 260B tokens and overtakes it after that point.Source: Figure 3 · p. 8, §3.2 · pp. 8–9
  • MixtureVitae outperforms other permissive datasets on MMLU, ARC Challenge, ARC Easy, and BoolQ, while matching Comma-0.1 on several other tasks and trailing it on LAMBADA.On PIQA, HellaSwag, Winogrande, and OpenBookQA, both MixtureVitae and Comma-0.1 trail non-permissive datasets.Source: §3.2 · p. 9

3.3 RESULTS ON PROBLEM SOLVING AND INSTRUCTION-BASED DOWNSTREAM TASKS

  • MixtureVitae scores 0.53 on GSM8K, 0.32 on HumanEval, and 0.38 on MBPP, exceeding SmolLM2-1.7B-Instruct on all three despite training on 300B rather than ≈11T tokens.These evaluations use the final 1.7B checkpoints trained under the open-sci-ref protocol.Source: §3.3 · pp. 9–10, Table 2 · p. 9

3.4 TEST LEAKAGE AND DECONTAMINATION

  • After removing overlapping test items, MixtureVitae retains its performance, including GSM8K at 0.54 versus 0.53 originally and MBPP at 0.38 in both versions.Retraining after removing shards responsible for most detected contamination also tracks closely with the full MixtureVitae model.Source: §3.4 · p. 10, Table 3 · p. 10, Figure 4 · p. 11

3.5 ABLATION STUDIES

  • Removing Instructions causes the largest average-performance drop, including GSM8K falling from 0.47 to 0.03; removing Web lowers GSM8K from 0.47 to 0.41.The ablation compares the full mixture with versions lacking Web or Instructions; the average covers 10 downstream tasks.Source: §3.5 · pp. 10–11, Figure 5 · p. 11

4 RELATED WORK

Large pretraining datasets have expanded alongside greater attention to provenance and licensing. This section situates MixtureVitae among web-scale, open, permissive, and synthetic-data efforts.

4.1 PIONEERING LARGE-SCALE DATASETS

  • Early web-scale corpora such as C4, The Pile, and ROOTS provided scale and diversity but often mix permissive material with content of unknown or non-commercial licensing.The section notes that this licensing complexity can create potential legal risks for commercial applications.Source: §4.1 · p. 12

4.2 OPEN AND REPRODUCIBLE DATASETS

  • Open and reproducible datasets expanded community access, while MixtureVitae adds explicit consideration of underlying copyright to this effort.The section names RedPajama-1T, Dolma, and SILO as examples of this openness and reproducibility push.Source: §4.2 · p. 12

4.3 PERMISSIVELY LICENSED AND SYNTHETIC DATA

  • Strictly permissive human-text corpora remain a challenge to scale and diversify, and the paper reports that they can lag in complex reasoning, math, and instruction following.The section frames synthetic instruction and reasoning data as a related response to scarce high-quality data.Source: §4.3 · p. 12
  • MixtureVitae extends synthetic instruction and reasoning-data efforts by combining targeted synthetic data with a curated, permissive-first, risk-mitigated corpus.The section contrasts instruction data for fine-tuning in Alpaca and OpenMathInstruct-1 with Phi-4's synthetic reasoning-task data.Source: §4.3 · p. 12
  • Unlike Apertus's breadth-oriented, retroactively filtered 15T-token, 1800+-language approach, MixtureVitae emphasizes English-centric reasoning density and positive inclusion.The section also distinguishes MixtureVitae's ready-to-use dataset release from Apertus's recipes and reconstruction scripts.Source: §4.3 · p. 12

4.4 MIXING REASONING DATA INTO PRE-TRAINING

  • Pretraining with diverse reasoning data has been reported to improve reasoning, math, and language understanding, and MixtureVitae extends this finding to a permissive dataset setting.The related work reports matched-compute, matched-token improvements from thinking-trace augmentation and says MixtureVitae achieves competitive performance against non-permissive baselines with a constrained token budget.Source: §4.4 · pp. 12–13

5 DISCUSSION & CONCLUSION

MixtureVitae tests whether permissively sourced real and synthetic data can achieve strong pretraining performance. The discussion reports competitiveness with non-permissive baselines and presents the corpus and its provenance scheme as a reusable model.

5 DISCUSSION & CONCLUSION

  • In controlled 300B-token experiments, MixtureVitae catches up to DCLM and FineWeb-Edu, and its 1.7B base model outperforms SmolLM2-1.7B-Instruct on GSM8K, HumanEval, and MBPP.SmolLM2-1.7B-Instruct was trained on approximately 11T tokens.Source: §5 · p. 13
  • Removing reasoning and instruction data causes substantially greater degradation across tasks than removing web data, while language-understanding evaluations remain strong.The discussion reports strong math and code performance and says language understanding matches non-permissive baselines while outperforming other permissive datasets.Source: §5 · p. 13
  • The three-tier licensing scheme and shard-level annotations are presented as a template for risk-mitigated mixtures and a reusable blueprint for compliant pretraining.The paper describes positive-inclusion pseudo-crawling, tiered provenance tracking, targeted synthetic generation with audited seeds, and decontamination.Source: §5 · p. 13

6 REPRODUCIBILITY STATEMENT

The reproducibility statement describes planned public data and script releases, a controlled training procedure, and a specified evaluation protocol. It also notes that checkpoints and training logs are not included in the initial submission.

6.1 DATASET AND CURATION RECIPES •

  • The full 422B-token dataset and 100B- and 50B-token subsets are slated for public release upon acceptance, with curation and math-problem-generation scripts included.The paper also says its data sources and composition are detailed in Figure 6.Source: §6.1 · p. 13

6.2 TRAINING PROCEDURE

  • Experiments use the open-sci-ref procedure, specified model scales and training schedules, Megatron-LM, and the GPT-NeoX-20B tokenizer.Training scripts are included with the submission.Source: §6.2 · p. 14

6.3 EVALUATION AND ANALYSIS

  • Evaluation uses the public LM Evaluation Harness, with benchmark settings, few-shot counts, and the 13-gram decontamination protocol specified.Evaluation and decontamination scripts are included with the submission.Source: §6.3 · p. 14
  • Model checkpoints and training logs are absent from the initial submission because of size and anonymity constraints, with release planned upon publication.The stated release plan is intended to facilitate future research.Source: §6.3 · p. 14

A DATASET COMPOSITION AND COMPARISON

The appendix compares dataset scale, data types, and licensing philosophies, then documents MixtureVitae's composition and experimental setup. Its source categories support transparency, licensing audits, and ablation analysis rather than prescribing a rigid training order.

A DATASET COMPOSITION AND COMPARISON

  • Table 4 compares evaluated pretraining datasets by size, data type, and licensing philosophy, highlighting MixtureVitae's risk-mitigated licensing and reasoning, coding, and instruction synthetic data.The appendix says its shard and category groupings support transparency, licensing audits, and ablation analysis rather than a fixed training order.Source: Table 4 · p. 24, A DATASET COMPOSITION AND COMPARISON · p. 24
  • Figure 6 color-codes the composition by category, including code and technology, reasoning and instruction, encyclopedic and scholarly text, math, and web sources.Named leading sources include Stack V1 and Ling-Coder, Open Thoughts, Wikipedia, Math and Science (Nemotron) and Prism-Math, and SEC Filings, Mega-Corpus, and FineFineWeb.Source: A DATASET COMPOSITION AND COMPARISON · pp. 24–25, Figure 6 · p. 25

B EXPERIMENT SETUP DETAILS

B.1 TRAINING SETUP PARAMETERS

  • Models use open-sci-ref architectures and tied embedding weights, with single-stage training and no post-training.The appendix defines the four model scales and describes training schedules for the 50B- and 300B-token runs.Source: Table 6 · p. 25, §B.1 · p. 26

B.2 EVALUATION SETTINGS

  • General evaluations use the LM Evaluation Harness, and reasoning-task settings are listed separately from general benchmark settings.Both evaluation-setting tables specify accuracy as the primary metric.Source: §B.2 · p. 26, Table 7 · p. 27, Table 8 · p. 27

C ADDITIONAL EXPERIMENTS

  • Additional experiments provide a granular breakdown of the 300B-token run, results at 50B tokens, and a red-teaming analysis of safety.The appendix describes these experiments as supplementing the main paper's findings.Source: C ADDITIONAL EXPERIMENTS · p. 26

C.1 300B EXPERIMENT - DETAILED RESULTS

At 300B tokens, MixtureVitae shows strong performance across reasoning-related and language tasks, while individual evaluations vary in how clearly they rank datasets. At 50B tokens, it already leads other permissive datasets, especially at larger model scales.

C.1 300B EXPERIMENT - DETAILED RESULTS

  • MixtureVitae performs strongly on reasoning-related and language tasks, although individual evaluations vary in how clearly they rank datasets.Figure 7 presents the detailed 300B-token comparisons.Source: §C.1 · p. 27, Figure 7 · p. 27

C.2 PERFORMANCE AT 50B TOKENS SCALE.

  • At 50B tokens, MixtureVitae leads permissive datasets consistently, especially at 1.3B and 1.7B scales, and is the only permissive dataset with a significant early MMLU learning signal.The authors suggest this early signal may reflect knowledge-rich and instruction-like content.Source: Figure 8 · p. 28, Figure 9 · p. 28, §C.2 · p. 29

C.3 MODEL RED TEAMING

  • The 300B-trained MixtureVitae model is reported as competitive with baselines on red-teaming benchmarks measuring attack success rate.Llama Guard-8B evaluates Do-Not-Answer and AdvBench; toxigen_roberta evaluates ToxiGen.Source: §C.3 · p. 29, Table 9 · p. 29

D.1 CONTAMINATION DETECTION PROTOCOL

  • The contamination-detection protocol has three stages: index construction, dataset scanning, and leakage reporting.It is designed to measure overlap between training data and reported evaluation benchmarks.Source: §D.1 · p. 29

D.1.1 INDEX CONSTRUCTION

  • Benchmark evaluation text is normalized, converted into 13-grams, filtered for boilerplate, and deduplicated between train and test splits to build the index.The index retains only 13-grams unique to the evaluation set.Source: §D.1.1 · p. 29

D.1.2 DATASET SCANNING

  • A training document is flagged when it has at least three distinct matching n-grams and matching coverage of at least 0.1%.Documents are processed using the same normalization, 13-gramming, and hashing pipeline used to construct the benchmark index.Source: §D.1.2 · p. 30

D.1.3 LEAKAGE REPORTING

  • Leak Percentage aggregates unique leaked n-grams found across scanned partitions and divides by the benchmark’s total unique n-grams.The final stage summarizes scan results in a report.Source: §D.1.3 · p. 30

D.2 CONTAMINATION REPORT

Decontamination analyses find that removing flagged data does not diminish MixtureVitae’s reported benchmark performance. Exact 13-gram matching offers a scalable, high-precision check but can miss paraphrased overlap.

D.2 CONTAMINATION REPORT

  • Document-level overlap is at or below 0.0003% for most benchmarks, but is higher for MMLU, BoolQ, HumanEval, and MBPP.The authors characterize the code-benchmark overlap as potentially reflecting common code snippets rather than direct test-set leakage.Source: §D.2 · p. 30

D.3 DECONTAMINATED TEST SET PERFORMANCE

  • MixtureVitae’s strong results are consistent on original and decontaminated benchmarks, aside from upward bias in decontaminated MBPP+ and IFEval.The authors report that strict benchmark decontamination did not affect MixtureVitae’s math and coding results.Source: §D.3 · pp. 30–31

D.4 DECONTAMINATION CASE STUDY

  • Removing three shards responsible for 27% of contaminated documents had no effect on MMLU performance.The removed shards were Misc-Instruct, DART-Math, and Nemotron Science & Math.Source: §D.4 · p. 31

D.5 Full Decontamination Experiment

  • The fully decontaminated 1.7B model performs slightly better than the original MixtureVitae model after 300B training tokens.The comparison is shown in Figure 10.Source: §D.5 · p. 31, Figure 10 · p. 32

D.5 DISCUSSION ON DECONTAMINATION METHODOLOGY

  • The 13-gram exact-match procedure is designed for high precision, scalability, and comparability, but can overlook paraphrased content.Approximate matching was avoided because it can produce false positives on common factual or algorithmic templates and may discard valuable instructional data.Source: §D.5 · p. 32

E SYNTHETIC MATH DATA GENERATION

The paper describes synthetic math data that pairs diverse problems with step-by-step solutions and training-oriented formatting. It also explains the legal rationale for including government web content while acknowledging residual risk.

E SYNTHETIC MATH DATA GENERATION

  • The synthetic math generator covers arithmetic, fractional expressions, algebra, and narrative word problems, producing detailed step-by-step solutions formatted as chain-of-thought.A post-processing step adds randomized descriptive headers to selected explanatory entries.Source: E SYNTHETIC MATH DATA GENERATION · pp. 32–33

F OUR POSITION ON USING GOVERNMENTAL AND OTHER WORKS UNDER FAIR USE AND RELATED ETHICAL AND LEGAL BASIS

  • The authors frame fair use, permissive upstream licensing, government-works doctrine, and EU text-and-data-mining exceptions as principles informing their risk-mitigated design, not as definitive legal advice.They state that provenance tracking and filtering do not eliminate all residual risks.Source: F OUR POSITION ON USING GOVERNMENTAL… · p. 33

F.1 FAIR USE OF GOVERNMENT WORKS

  • The dataset includes about 11B tokens of government website data; for content not explicitly public-domain or licensed, the authors rely on fair-use principles and EU text-and-data-mining exceptions to mitigate risk.They apply keyword filters to exclude pages containing terms such as “All Rights Reserved” and “Copyright ©” to further reduce risk.Source: §F.1 · pp. 33–34

F.2 OTHER TIER-2 DATA WITH OPAQUE OR MIXED PROVENANCE

The paper describes residual provenance and legal uncertainties in some Tier 2 data, reliance on EU text-and-data-mining rules for web sources, and limits to its risk mitigation. It also explains why The Stack v1 is included despite imperfect repository-level filtering.

F.2 OTHER TIER-2 DATA WITH OPAQUE OR MIXED PROVENANCE

  • The Stack v1 [Kocetkov et al., 2023] lacks a line-by-line copyright audit, and repository-level license filtering can leave mixed-license content and residual risk.The authors rely on fair use because the data train models rather than provide a substitutive or competing software product.Source: §F.2 · p. 34, G PROVENANCE AND RATIONALE FOR THE… · p. 35
  • Tier 2(b) covers about 4% of the dataset amid debate over upstream generator conditions; some P3 few-shot data also has components without explicit licenses.The authors regard the reformatted P3 material as transformative and unlikely to compete with underlying works.Source: §F.2 · p. 34

F.3 RELIANCE ON EU TEXT AND DATA MINING EXCEPTIONS

  • The authors rely in part on the EU text and data mining exception for web-crawled data and on Common Crawl’s practice of respecting robots.txt at crawl time.They do not consider retroactive recrawling legally necessary to check for later opt-outs.Source: §F.3 · p. 34

F.4 RESIDUAL COPYRIGHT AND TRADEMARK RISKS

  • Aggregate training could allow models to generate substantially more text than the short fair-use quotations present in sources such as Wikipedia.The paper calls for dataset copyright audits and model-level measures against reproducing substantial protected passages.Source: §F.4 · p. 35
  • Generated new stories using a character name such as “Spiderman” may implicate trademark rights even when their plots do not derive from existing stories.Thoroughly addressing these issues is beyond the work’s scope.Source: §F.4 · p. 35
  • The authors cannot guarantee that provenance tracking and standard filtering make the dataset free of legal risk.They recommend users consult counsel in their jurisdiction before commercial deployment.Source: §F.4 · p. 35

G PROVENANCE AND RATIONALE FOR THE STACK V1 (OPENRAIL-M AND TERMS OF USE)

  • The Stack v1’s repository-root license check is a heuristic, but the authors include it as a large public corpus curated for permissive filtering and aligned with risk mitigation.The dataset is classified as Tier 2.Source: G PROVENANCE AND RATIONALE FOR THE… · p. 35

H DATA FILTERING REASONING AND PROTOCOL

The authors describe transparent procedures for selecting and checking pseudo-crawled sources, with distinct protocols for governmental and NGO domains and a curated permissive-domain list.

H DATA FILTERING REASONING AND PROTOCOL

  • The paper presents its protocol for defining and checking the lists and content of the pseudo-crawled data.The following procedures cover government and NGO domain patterns as well as curated permissive domains.Source: H DATA FILTERING REASONING AND PROTOCOL · p. 35

H.1 GOVERNMENTAL AND NGO DOMAIN PATTERNS

  • The authors manually checked domain ownership, audited pseudo-crawled data for quality and spam, and filtered rights-reservation keywords on US federal sites while relying on Common Crawl’s robots.txt practice.The domain list accounts for varied international government domains and the possibility of spam sites mimicking .gov domains.Source: §H.1 · p. 36

H.2 CURATED PERMISSIVE DOMAIN LIST

  • The curated permissive-domain list contains about 50 sites whose terms or licensing were reviewed; most are CC-BY, and crawled data was also quality-checked and filtered for spam.The sites were selected for diverse content, and spam sites with similar names were added to blocklists.Source: §H.2 · p. 36

I SYNTHETIC DATA SOURCE PROVENANCE

This section documents synthetic-source provenance and tests whether Tier 2(b) data materially contributes to performance. It also outlines ways to scale the 422-billion-token corpus and records author contributions.

I SYNTHETIC DATA SOURCE PROVENANCE

J SCALING OUTLOOK AND FUTURE DIRECTIONS

  • At 422 billion tokens, MixtureVitae is smaller than frontier runs often exceeding 10 trillion tokens; the work frames its current scale as a proof of concept.The authors identify upsampling high-value subsets, such as curated reasoning data, as a way to increase their contribution to the training budget.Source: J SCALING OUTLOOK AND FUTURE DIRECTIONS · pp. 37, 39
  • The authors identify international permissively licensed sources and machine translation as routes to expand the primarily English-centric corpus.They describe multilingual expansion as an order-of-magnitude scaling opportunity.Source: J SCALING OUTLOOK AND FUTURE DIRECTIONS · p. 39
  • Procedural or LLM-based generation can expand the math and reasoning subsets by increasing compute, without the legal risks associated with scraping organic web data.The authors also propose rephrasing MixtureVitae web data to expand corpus volume while maintaining their stated safety and licensing standards.Source: J SCALING OUTLOOK AND FUTURE DIRECTIONS · p. 39

K AUTHOR CONTRIBUTIONS •

  • The contribution record spans dataset curation and engineering, model training and evaluation, decontamination analysis, policy design, supervision, and paper writing.Named contributions include overall pipeline engineering, baseline experiment design, and model training and evaluation.Source: K AUTHOR CONTRIBUTIONS • · p. 39
Loading 2509.25531v5…