Source-linked AI summary
daVinci-LLM:Towards the Science of Pretraining
Yiwei Qin, Yixiu Liu, Tiantian Mi, Muhang Xie, Zhen Huang, Weiye Si, Pengrui Lu, Siyuan Feng, Xia Wu, Liming Liu, Ye Luo, Jinlong Hou, Qipeng Guo, Yu Qiao, Pengfei Liu
TL;DR
Pretraining remains under-explored because industrial resources and research transparency rarely coincide. daVinci-LLM combines large-scale training with a fully open methodology, using Data Darwinism and 200+ controlled ablations to study processing, training dynamics, mixture design, and evaluation; the resulting evidence shows systematic effects of processing depth, domain saturation, data format, and evaluation protocol.
Problem
Pretraining research lacks systematic exploration because organizations with large computational resources face commercial disclosure pressures, while academic efforts face severe scale limitations.
Method
daVinci-LLM combines billion-parameter training resources with research freedom, releases complete pipelines and training evidence, and organizes data processing through the L0-L9 Data Darwinism framework while conducting 200+ controlled ablations.
Results
Across its investigations, daVinci-LLM finds that processing depth enhances capabilities, domains saturate differently, data strategies must adapt, and evaluation protocols can produce substantially different model rankings; its 3B model achieves 51.72 overall average across 19 tasks.
Takeaways & Limitations
Transparent, systematic pretraining exploration can support cumulative scientific knowledge, while evaluating base models under both PPL-based and generative protocols provides a more complete capability profile.
Takeaways & Limitations
Further domain-proportion adjustments yield only marginal gains once standard web, code, and science corpora approach saturation.
Abstract
from arXiv · showhide
The foundational pretraining phase determines a model's capability ceiling, as post-training struggles to overcome capability foundations established during pretraining, yet it remains critically under-explored. This stems from a structural paradox: organizations with computational resources operate under commercial pressures that inhibit transparent disclosure, while academic institutions possess research freedom but lack pretraining-scale computational resources. daVinci-LLM occupies this unexplored intersection, combining industrial-scale resources with full research freedom to advance the science of pretraining. We adopt a fully-open paradigm that treats openness as scientific methodology, releasing complete data processing pipelines, full training processes, and systematic exploration results. Recognizing that the field lacks systematic methodology for data processing, we employ the Data Darwinism framework, a principled L0-L9 taxonomy from filtering to synthesis. We train a 3B-parameter model from random initialization across 8T tokens using a two-stage adaptive curriculum that progressively shifts from foundational capabilities to reasoning-intensive enhancement. Through 200+ controlled ablations, we establish that: processing depth systematically enhances capabilities, establishing it as a critical dimension alongside volume scaling; different domains exhibit distinct saturation dynamics, necessitating adaptive strategies from proportion adjustments to format shifts; compositional balance enables targeted intensification while preventing performance collapse; how evaluation protocol choices shape our understanding of pretraining progress. By releasing the complete exploration process, we enable the community to build upon our findings and systematic methodologies to form accumulative scientific knowledge in pretraining.
1. Introduction
daVinci-LLM addresses a structural gap in pretraining research by combining large-scale computational resources with research freedom and full scientific transparency. It applies a systematic data framework, open training process, and controlled ablations to study pretraining decisions.
- Commercial models offer limited API access, while open-weight releases provide checkpoints but withhold design rationale and negative results.
- Academic efforts provide transparency and research freedom but lack the scale needed for systematic exploration with 200+ configurations.
- daVinci-LLM combines billion-parameter training resources with research freedom, releasing complete training trajectories, data specifications, and ablation results.
- The Data Darwinism framework organizes data-processing operations from basic filtering through content transformation and knowledge synthesis using an L0-L9 taxonomy.
- The project trains a 3B-parameter model from random initialization across 8T tokens using a two-stage curriculum that shifts from broad foundations toward reasoning-intensive enhancement.
- Through 200+ controlled ablations, the study investigates data-processing depth, training dynamics, mixture design, and evaluation validity.
2. Data: What We Used and How We Processed It
The corpus spans general, code, math, science, and QA data, with each source annotated by its Data Darwinism processing level. Processing ranges from acquisition and normalization to filtering, refinement, cognitive completion, and higher-order synthesis, with selected sources receiving deeper interventions.
- The project releases processed datasets, curation prompts and code, intermediate and final checkpoints, training logs, ablation results, and the evaluation suite.
- 2.1 Data Darwinism Framework: Data Darwinism defines a ten-level L0-L9 taxonomy for organizing processing operations and assessing whether further quality improvements justify additional investment.
- 2.1 Data Darwinism Framework: The hierarchy progresses from acquisition and format normalization to rule-based filtering, lightweight model filtering, generative refinement, cognitive completion, and higher-order synthesis.
- 2.1 Data Darwinism Framework: Processing levels can be applied repeatedly and need not follow a strict hierarchy when another ordering better serves data quality.
- The corpus contains five major categories: CC, Code, Math, Science, and QA, with sources annotated by Darwin Level.
- 2.2.3 Science: MegaMath-Web-Pro applies FineMath classification and LLM-driven refinement to remove noise and reorganize content while preserving original length, reaching Darwin Level L4 and contributing approximately 13B tokens.
3. Training Recipe: How We Trained It
daVinci-LLM uses transparent, multi-stage pretraining to track capability development and adapt data mixtures as training progresses. Across 8T tokens, the resulting 3B model improves reasoning while preserving broad capabilities and matches larger baselines.
- Architecture: The model uses a Qwen2-based decoder-only architecture trained from random initialization, with 3B parameters and modern efficiency-oriented design choices.The cited architectural passages specify Qwen2, causal language modeling, grouped-query attention, SwiGLU, and a 36-layer, 2048-dimension configuration.
- Evaluation: Evaluation tracks capability emergence at 5k-step intervals across 19 benchmarks spanning general knowledge, code, and science.The protocol examines saturation and benchmark stability while covering reasoning, code generation, and mathematical problem solving.
- Stage 1: Stage 1 training remains stable, with smooth loss convergence, stable gradient norms, and no significant gradient spikes or divergences.Stage 1 covers a 6T-token trajectory, with checkpoints evaluated every 5k steps.
- Stage 1: General benchmarks plateau within the first 1T tokens, whereas code and science continue growing, accelerating during the reasoning-heavy Stage 1-2 mixture.Stage 1 ends at an overall average of 39.58, motivating later reasoning-focused training.
- Stage 2: Stage 2 raises the overall average from 39.58 to 48.60 within 1T tokens, while deeper processing and reasoning-focused data substantially improve mathematical reasoning.The MATH score increases from 22.0 at the Stage 1 endpoint to 62.8.
- Final results: The final model reaches a 51.72 overall average, matching OLMo-3 7B despite using 3B rather than 7B parameters, while maintaining general capability.Code averages 55.99 and science reasoning reaches 48.30, with no reported catastrophic forgetting during specialized training.
- Findings: The ablations identify processing depth, adaptive training dynamics, and balanced data mixtures as central factors in improving reasoning without capability collapse.The findings support systematic investigation of data quality, capability-specific saturation, and mixture trade-offs during pretraining.
4. Exploration: Why We Trained It This Way
The paper turns pretraining design decisions into controlled questions, showing that processing depth, adaptive data strategies, and balanced intensification shape capability development. It also finds that evaluation protocols can alter apparent model quality and that standard corpus reallocation eventually requires format changes.
- 200+ controlled ablations transform pretraining decisions into an evidence-based record of what worked and how choices were informed.The exploration covers processing depth, training dynamics, mixture design, and evaluation validity.
- 4.1 Data Processing Depth: Processing depth systematically enhances capabilities: L3 gives modest consistent gains, L4 improves complex reasoning, and L5 enables domain-targeted synthesis.The reported examples include +3.40 on MBPP for L3 and +7.00 on MATH for L4.
- 4.1 Data Processing Depth: L3 filtering alone may be insufficient for sophisticated coding gains, making more intensive interventions such as generative refinement necessary.HumanEval and EvalPlus improvements remain minimal relative to the notable MBPP gain.
- 4.1 Data Processing Depth: +7.00 on MATH versus +1.37 on GSM8K shows that L4 refinement benefits complex mathematical reasoning unevenly.The paper attributes this pattern to transforming messy exposition into pedagogical logic.
- 4.1 Data Processing Depth: L5 synthesis produces strong source-target alignment: CodeQA improves programming benchmarks, while CC-QA primarily strengthens science reasoning.Its limited cross-domain transfer makes synthesis a precision steering tool rather than a universal booster.
- 4.2 Training Dynamics: Different capabilities saturate at different timescales, so training reallocates data from plateaued general knowledge toward still-improving code and science reasoning.Stage 1 reduced generic web data and increased code/science concentrations at the 4T-token checkpoint.
- 4.2 Training Dynamics: Once standard corpora approach saturation, proportion adjustments yield diminishing returns, motivating a shift from raw text to structured question-answer data.Stage 2 introduces explicit reasoning structures after later Stage 1 adjustments produced only marginal gains.
5. Conclusion
The conclusion presents daVinci-LLM as a transparent, controlled study of pretraining dynamics and releases its pipelines and exploration results. It argues that evaluation protocol choice materially affects apparent model quality and that cumulative open collaboration is needed for pretraining to become a rigorous scientific discipline.
- A 3.10% swing reverses the OLMO-2-7B and Qwen-2.5-3B ranking between PPL-based and generative evaluation.The gap changes from -0.20% under PPL to +2.90% under generative evaluation.
- Generative evaluation tests explicit answer production and reasoning, whereas PPL-based evaluation more closely measures latent knowledge access among presented options.QA-heavy pretraining can therefore create larger advantages under generative evaluation.
- Evaluation protocol should match deployment: generative tests suit direct answer generation, while PPL-based tests suit scoring or ranking applications.The paper cautions against treating either protocol as inherently superior.
- The paper combines Data Darwinism and 200+ controlled ablations with complete pipelines and exploration results to document pretraining decisions transparently.Its stated aim is to support broader accumulation of empirical knowledge.
A.1 Benchmark Descriptions
The appendix describes the 19 benchmarks used to evaluate the model, organized by capability domain.
- The evaluation covers 19 benchmarks across General, Code, and Science capability domains.General includes knowledge and commonsense tasks, Code includes Python synthesis, and Science includes mathematical and scientific reasoning benchmarks.
A.1.1 General Knowledge and Reasoning Benchmarks
These benchmarks assess broad knowledge, commonsense, reading comprehension, and physical-world reasoning across varied domains and difficulty levels.
- MMLU covers 57 subjects across STEM, humanities, social sciences, and other areas, with questions ranging from elementary to professional level.It evaluates broad knowledge and reasoning across domains including mathematics, history, computer science, and law.
- MMLU-Pro increases difficulty with reasoning-focused questions, ten answer choices, and removal of trivial or noisy items.It reports a 16-33% accuracy drop relative to MMLU and lower prompt sensitivity.
- AGIEval-EN evaluates general abilities using English portions of high-standard admission and qualification examinations.Its source exams include the SAT, law-school admissions tests, mathematics competitions, and civil-service examinations.
- HellaSwag, TriviaQA, and RACE test commonsense inference, multi-document reading comprehension, and passage understanding with implicit reasoning, respectively.TriviaQA contains over 650K question-answer-evidence triples, while RACE includes more than 28,000 passages and nearly 100,000 questions.
- WinoGrande, OpenBookQA, and PIQA evaluate commonsense reasoning, elementary science application, and physical interaction knowledge.OpenBookQA contains 5,957 multiple-choice science questions based on 1,326 core facts and broader common knowledge.
A.1 Benchmark Descriptions SII-GAIR
PIQA is designed to distinguish physical-world understanding from pattern matching.
- PIQA assesses whether models learn about the physical world rather than merely pattern matching.
A.1.2 Code Generation Benchmarks
The code benchmarks measure Python program synthesis and functional correctness, with EvalPlus applying stricter tests than HumanEval.
- HumanEval measures functional correctness when synthesizing Python programs from docstrings across 164 handwritten problems.Its pass@k metric measures the probability that at least one of k generated samples passes all unit tests.
- EvalPlus extends HumanEval with comprehensive test suites covering edge cases and corner conditions.The additional tests make high scores harder to achieve through superficial pattern matching.
- MBPP evaluates basic Python program synthesis through 974 short tasks designed to be solvable by entry-level programmers.Each task includes a natural-language description, reference solution, and three automated test cases.
A.1.3 Mathematics and STEM Reasoning Benchmarks
These benchmarks span grade-school to competition mathematics and graduate-level science, while testing robustness, advanced knowledge, and reasoning beyond pattern matching.
- GSM8K tests sequential reasoning on 8.5K linguistically diverse grade-school math word problems requiring multiple steps.
- GSM-Plus perturbs GSM8K problems to test whether mathematical solvers understand and apply knowledge rather than rely on shortcuts.Small changes to statements or targets can expose brittleness.
- MATH contains 12,500 challenging competition problems across algebra, geometry, probability, number theory, and related areas.Each problem includes a full step-by-step solution and requires sophisticated strategies beyond pattern matching.
- GPQA comprises 448 graduate-level biology, physics, and chemistry questions written by domain experts and designed to be Google-proof.PhD experts reach 65% accuracy, while GPT-4-based baselines achieve around 39%.
- SuperGPQA expands GPQA with additional challenging questions while retaining graduate-level difficulty and Google-proof design.
- MMLU-STEM covers STEM subjects including mathematics, physics, chemistry, computer science, engineering, and machine learning.
- MMLU-Pro-STEM retains MMLU-Pro’s increased difficulty and ten-option format for a more challenging STEM assessment.
B. Training Implementation Decisions: LR Decay and QA Masking SII-GAIR
The implementation comparisons examine learning-rate decay and QA masking as training choices affecting Stage 2-2 performance. Cosine decay changes the learning rate from 3e-5 to 3e-6, while masking is compared with unmasked QA training across capability dimensions.
- Cosine decay gradually reduces the learning rate from 3e-5 to 3e-6, whereas constant LR maintains 3e-5 throughout Stage 2-2.
- QA masking is evaluated across three capability dimensions by comparing masked and unmasked configurations during Stage 2-2.
- QA masking produces consistent but modest gains across all capability clusters.
B Training Implementation Decisions: LR Decay and QA Masking
The paper studies learning-rate decay and QA question masking as practical training choices in Stage 2-2. Cosine decay provides modest improvements in selected capabilities, while masking yields a small, consistent gain across general, code, and science domains.
- Learning rate decay: Cosine decay consistently improves general knowledge and code generation, while science reasoning remains largely unchanged.The comparison uses the 70% QA Stage 2-2 configuration.
- QA data masking policy: 49.14 overall at 30k steps with masking versus 48.77 without masking yields a +0.37 gain.The gains are reported for the 70% QA Stage 2-2 configuration with high-quality, diverse question data.
- QA data masking policy: Masking improves general, code, and science scores by +0.38, +0.35, and +0.35, respectively.
- QA data masking policy: The masking comparison trains either only on answer tokens or on all QA tokens as continuous text.
C Prompts for Dataset Construction
The dataset-construction prompts combine document cleaning, OCR correction, content preservation, and pedagogical transformation. They require outputs to remain faithful to the input while preserving academic structure and valuable technical material.
- Document cleaning: The cleaning prompts remove redundant, erroneous, navigational, metadata, reference-list, and other unwanted document content.They target items including OCR artifacts, garbage, duplicates, URLs, page markers, and structural navigation material.
- OCR correction: OCR correction repairs fragmented text, damaged structured content, character and encoding errors, whitespace, formatting, and punctuation.
- Content preservation: Protection rules preserve formulas, equations, proofs, specialized notation, citations, tables, code, educational content, and explanatory material.
- Output constraints: The cleaning output must contain only the corrected text inside the specified tags, or remain empty when the entire chunk should be deleted.
- Data Darwinism prompts: L5 cognitive completion makes implicit reasoning explicit through reconstruction, terminological explanation, and pedagogical bridging while retaining scientific fidelity.
- Contextual limitations: Partial-chunk processing must terminate exactly where the provided chunk ends, without inventing definitions or context beyond it.