Source-linked AI summary
Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training
Yiwei Qin, Zhen Huang, Tiantian Mi, Weiye Si, Chenyang Zhou, Qipeng Guo, Siyuan Feng, Pengfei Liu
TL;DR
Foundation-model data processing lacks a systematic framework, motivating Data Darwinism’s ten-level hierarchy and its scientific validation. The paper constructs and processes Darwin-Science through L0–L5, finding that higher-level processing improves performance over baselines after continued pre-training. The study is limited to scientific domains, L0–L5, and specific teacher models and training configurations.
Problem
Data processing lacks a unified framework for relating transformations to downstream model capabilities, hindering reproducibility and principled experimentation.
Method
The paper introduces a ten-level Data Darwinism hierarchy and applies L0–L5 processing to construct and evaluate the 900B-token Darwin-Science corpus.
Results
+2.12 points for 3B and +2.95 points for 7B are achieved across general benchmarks, rising to +5.60 and +8.40 on domain-aligned evaluation.
Takeaways & Limitations
+1.36 cumulative gain from L4–L5 indicates that hierarchy ascension unlocks value unavailable through basic filtering alone.
Takeaways & Limitations
The study covers scientific domains and L0–L5, using specific teacher models and training configurations whose broader generalizability remains to be tested.
Abstract
from arXiv · showhide
Data quality determines foundation model performance, yet systematic processing frameworks are lacking. We introduce Data Darwinism, a ten-level taxonomy (L0-L9) that conceptualizes data-model co-evolution: advanced models produce superior data for next-generation systems. We validate this on scientific literature by constructing Darwin-Science, a 900B-token corpus (L0-L5). We identify a learnability gap in raw scientific text, which we bridge via L4 (Generative Refinement) and L5 (Cognitive Completion) using frontier LLMs to explicate reasoning and terminology. To ensure rigorous attribution, we pre-trained daVinci-origin-3B/7B models from scratch, excluding scientific content to create contamination-free baselines. After 600B tokens of continued pre-training, Darwin-Science outperforms baselines by +2.12 (3B) and +2.95 (7B) points across 20+ benchmarks, rising to +5.60 and +8.40 points on domain-aligned tasks. Systematic progression to L5 yields a +1.36 total gain, confirming that higher-level processing unlocks latent data value. We release the Darwin-Science corpus and daVinci-origin models to enable principled, co-evolutionary development.
1. Introduction
The paper introduces Data Darwinism to organize data processing systematically and model its co-evolution with model capabilities. Scientific literature provides a test case because raw, highly compressed content exhibits a severe learnability gap that L4 and L5 processing aim to address.
- Data Darwinism defines a ten-level hierarchy for systematically organizing data operations.
- Lower levels filter and preserve original content, whereas higher levels rewrite, enrich, and generate data with increasingly machine-driven processing.
- More capable models enable more sophisticated processing that produces higher-quality training data for subsequent models.
- Scientific texts expose a severe learnability gap because high conceptual compression and implicit reasoning hinder model training value after preliminary filtering.
- L4 refinement and L5 cognitive completion are advanced to purify scientific content and make implicit reasoning and terminology explicit.
2. Data Processing Hierarchy in Data Darwinism
The hierarchy spans acquisition through increasingly transformative processing, with Darwin-Science operationalizing these levels for scientific data. The reported experiments show that higher-level processing yields larger gains than basic filtering.
- +1.36 cumulative gain results from advancing through L4 and L5, while L0–L3 yields negligible improvement.
- +2.12 points for 3B and +2.95 points for 7B are achieved across 20+ benchmarks, increasing to +5.60 and +8.40 on Darwin-Science-Eval.
- The work presents Darwin-Science, transparent base models, and hierarchy-driven evidence as a practical roadmap for co-evolutionary data-model development.
- The ten-level hierarchy tracks data from acquisition to simulated synthesis while trading volume for quality, density, and structural complexity.
- L0 preserves diverse raw data, while L1 normalizes heterogeneous formats into training-ready representations without filtering content.
- L2 applies deterministic filters, and L3 uses lightweight models for semantic judgments such as topic, domain, quality, and educational value.
L4 Generative Refinement
The hierarchy extends from lightweight filtering toward model-driven completion and synthesis. L5 specifically enriches existing content by exposing implicit reasoning and dependencies, while L9 remains an aspirational endpoint.
- L5 Cognitive Completion makes implicit reasoning and logical steps explicit in mathematical, scientific, and instructional content.
- L6 expands content with external references and prerequisite definitions to resolve implicit dependencies.
- L7 constructs executable environments, whereas L8 creates dynamic multi-agent ecosystems that generate interaction data.
- L9 aims to construct physically and socially coherent simulated worlds, but faces immense computational and theoretical challenges.
3 Dataset Construction
Darwin-Science is built from public academic resources and open-source scholarly collections through OCR, filtering, classification, refinement, and cognitive completion. The resulting corpus contains approximately 900B tokens and targets the learnability barriers of scientific text.
- The corpus combines public academic resources with PubMed Central, arXiv, and S2ORC scholarly collections.
- Scanned PDFs are converted into machine-readable text with olmOCR-7B-0225-preview before subsequent processing.
- Deduplication removes 22% of documents, followed by file-size, garbled-text, and language filtering.
- Documents receive multidimensional classification for content quality, educational value, discipline, and book-paper type.
- L4 removes structural noise and repairs defects, while L5 addresses compressed expert exposition by reconstructing reasoning and explicating terminology.
- The final Darwin-Science contains 50M documents totaling approximately 900B tokens, alongside a 601B-token corpus of original OCR-extracted text.
4 Evaluation of Darwin-Science
Darwin-Science-Eval is designed to measure advanced scientific comprehension through questions generated from scholarly text. Its construction pipeline selects knowledge-rich segments and produces multiple-choice questions.
- Darwin-Science-Eval targets specialized scientific knowledge that existing elementary-science benchmarks do not adequately capture.
- Q&A Generation: The pipeline segments documents into 4096-token chunks to preserve relative semantic completeness.
- Q&A Generation: Qwen3-32B identifies valuable knowledge points and generates questions from suitable text segments.
5. Foundation Model Training
The paper builds transparent, science-free foundation models and evaluates them across general, scientific, and newly constructed research-level benchmarks. The training setup combines large-scale web, mathematical, and code data with disclosed model and optimization configurations.
- Evaluation Data: Darwin-Science-Eval contains 140K book questions and 10K paper questions, with two 1,500-question held-out test sets.
- Foundation Models: daVinci-origin-3B and daVinci-origin-7B are trained from scratch without scientific content, creating transparent contamination-free baselines.
- Pretraining Dataset: The foundation-model dataset combines CC, Math, and Code for a total of 5.37T tokens.
- Evaluation: Evaluation covers general reasoning, scientific capabilities, and BookQA and PaperQA for research-level comprehension.
- Evaluation: The pretrained checkpoints provide established general and scientific capabilities for studying scientific-data integration.
6. Experiments
The experiments use controlled continued pre-training to isolate the contribution of hierarchy-processed scientific data. They compare a baseline mixture with Sci-Mix using transparent, science-free models and evaluate progression across processing levels.
- Main Results: The study compares scientific-data processing levels from L0–L3 through L4 and L5 to quantify hierarchical effects.
- Training Configurations: Both configurations apply 600B tokens of continued pre-training to daVinci-origin-3B and daVinci-origin-7B models.
- Training Configurations: The Baseline retains the original mixture of 80.2% CommonCrawl, 11.2% Code, and 8.5% Math.
- Training Configurations: Sci-Mix combines 50% hierarchy-processed scientific data with 50% of the baseline mixture.
7. Analysis
The analysis examines how processing depth, model scale, evaluation alignment, data composition, teacher quality, context length, and training stage affect scientific-data value. Results show that higher-level processing and aligned evaluation reveal the strongest gains.
- Processing Strategy: +1.36 points is the cumulative gain reached by L5 Cognitive Completion, exceeding the +0.38-point gain at L4.
- Model Capacity: +2.95 points for 7B versus +2.12 for 3B shows larger models derive greater average benefit from scientific data.
- Evaluation Alignment: +5.60–+8.40 points on Darwin-Science-Eval exceeds the 1.76–2.38-point gains on standard benchmarks.
- Data Composition: A 50% scientific-content ratio peaks on aggregated benchmarks, while aligned performance increases monotonically with the scientific ratio.
- Processing Strategy: L5 variants improve over L4 by +0.75 with OSS-120B and +1.27 with Qwen3-235B, while Qwen3-235B adds +0.52 over OSS-120B.
- Model Properties: +0.80 points favor the 32K context model after adaptation, and gains remain robust at both early and late training stages.
8 Related Work
Prior work has established scalable web-data curation, while specialized scientific pre-training increasingly relies on targeted corpora and controlled data mixtures.
- Open-domain resources such as C4, RefinedWeb, Dolma, and FineWeb demonstrate scalable, high-quality web curation pipelines.
- Scientific pre-training faces a scarcity of high-quality domain-specific corpora beyond general-purpose web data.
- Existing specialized-data work motivates controlled data mixtures for improving domain-specific pre-training.
9. Conclusion
The paper presents Data Darwinism as a systematic framework for processing scientific data, validates L0–L5 through controlled pre-training, and reports consistent gains from higher-level refinement. It also identifies scope limits and releases resources for further study.
- Data Darwinism organizes data transformations from selection to generation, preservation to transformation, and human-centric to machine-driven processing.
- 900B tokens of scientific literature were processed through L0–L5 to construct Darwin-Science, addressing a domain-specific data resource gap.
- +1.36 points accrued by advancing from basic filtering through L4 generative refinement to L5 cognitive completion.
- +2.12 points for 3B and +2.95 points for 7B models were achieved on general benchmarks after 600B continued-pretraining tokens.
- 3× stronger signals appeared in domain-matched assessment than in standard benchmarks, while performance sustained without saturation.
- Released Darwin-Science, daVinci-origin models, and Darwin-Science-Eval provide resources for principled data-model co-evolution.
- The work focuses on scientific domains and L0–L5, leaving higher levels, broader configurations, and deeper learnability analysis for future work.
B.4 L4 Processing Examples
The L4 examples show comprehensive removal of thesis front matter and formatting artifacts while preserving substantive research content, mathematical notation, and document structure. Even under severe OCR corruption, processing retains salvageable material but cannot reconstruct missing mathematics.
- L4 removed thesis front matter—including acknowledgments, affiliations, jury information, and tables of contents—while preserving English and French abstracts.
- L4 standardized mathematical notation and Unicode rendering across multilingual text while retaining technical symbols and LaTeX expressions.
- All academically valuable elements, including abstracts, keywords, mathematical definitions, and technical terminology, remained intact after cleaning.
- The processing preserved theorem structure, recoverable variables, operators, and scholarly narrative despite extreme corruption.
- L4 improved usability in severely corrupted documents by removing artifacts, preserving salvageable content, and maintaining document structure.
C.3 L5 Processing Examples
L5 processing rewrites dense scientific and experimental text into clearer, pedagogical explanations while preserving technical fidelity and making reasoning explicit.
- Mathematical example: Heavy-tailed equilibria invalidate the classical diffusion limit because their divergent second moment requires anomalous scaling and a fractional operator.The processed explanation links algebraic decay of M to the failure of the standard diffusion approximation.
- Mathematical example: L5 processing clarifies the transition from kinetic models to fractional diffusion limits by connecting microscopic dynamics with macroscopic equations.The rewrite explains weak convergence, the limiting density, and the role of fractional diffusion in the asymptotic analysis.
- Mathematical example: The rewrite unpacks weak convergence, diffusive scaling, and diffusion-matrix notation into stepwise reasoning units suitable for learning.It defines weak convergence through test functions and explains the outer-product notation in the diffusion tensor.
- Overall effect: Across examples, L5 reconstructs reasoning structure while preserving analytical rigor, technical terminology, and experimental content.The resulting expositions support advanced teaching, graduate-level study, and educational explanation of event-selection logic.
- Experimental-selection example: L5 transforms dense selection procedures into reproducible, didactic checklists that explain both experimental criteria and their purposes.The detector-selection example retains all experimental criteria while making event geometry and selection logic explicit.
D. Benchmark Construction SII-GAIR
SII-GAIR constructs expert-level multiple-choice questions from professional text, then filters them for standalone completeness and answer verifiability.
- Question generation: The generation prompt asks Qwen3-32B to identify core knowledge points and produce seven-option questions directly derived from suitable text.Questions, options, and correct answers must be grounded in the source content.
- Question generation: Unsuitable fragments, indexes, lists, or text lacking substantial technical content receive a direct “No QA” outcome.The workflow evaluates content suitability before attempting question creation.
- Validation filters: The completeness filter tests whether each question is independently assessable without external figures, tables, studies, references, or contextual phrases.It checks question independence and self-containment as a first validation stage.
- Validation filters: The correctness filter retains questions only when the labeled answer is explicitly stated or directly derivable from the original text and uniquely correct.The validator checks factual support, answer verifiability, and the absence of competing correct options.
- Worked example: An example applies the workflow to quasiconvex optimization and identifies the star-subdifferential as non-empty for every point.The example presents a domain-specific question with a text-supported answer.