Source-linked AI summary
ArXiv-to-Model: A Practical Study of Scientific LM Training
Anuj Gupta
TL;DR
The paper addresses the limited documentation of training scientific language models from raw open-access sources. It presents a transparent case study using a 1.36B-parameter model, an end-to-end arXiv-LaTeX pipeline, and constrained compute. The study finds that preprocessing, tokenization, infrastructure, and training-data scale materially shape usable data and stable specialization.
Problem
Training scientific language models directly from raw open-access sources is sparsely documented, despite the prevalence of curated or proprietary alternatives.
Method
The paper conducts a transparent case study training a 1.36B-parameter model from arXiv LaTeX through staged data engineering, tokenization, curriculum, and dense-model training experiments.
Results
Careful data engineering, tokenization design, and curriculum strategy supported stable domain specialization in a data-rich training regime.
Takeaways & Limitations
For small-to-mid-scale scientific models, reproducible preprocessing decisions and infrastructure planning are as important to report as architectural configuration.
Takeaways & Limitations
Preprocessing choices introduce pipeline-induced bias, and evaluation based mainly on held-out-corpus perplexity does not directly measure reasoning correctness or proof consistency.
Abstract
from arXiv · showhide
While frontier large language models demonstrate strong reasoning and mathematical capabilities, the practical process of training domain-specialized scientific language models from raw sources remains under-documented. In this work, we present a detailed case study of training a 1.36B-parameter scientific language model directly from raw arXiv LaTeX sources spanning mathematics, computer science, and theoretical physics. We describe an end-to-end pipeline covering metadata filtering, archive validation, LaTeX extraction, text normalization, domain-aware tokenization, and dense transformer training under constrained compute (2xA100 GPUs). Through 24 experimental runs, we analyze training stability, scaling behavior, data yield losses, and infrastructure bottlenecks. Our findings highlight how preprocessing decisions significantly affect usable token volume, how tokenization impacts symbolic stability, and how storage and I/O constraints can rival compute as limiting factors. We further analyze convergence dynamics and show stable training behavior in a data-rich regime (52B pretraining tokens). Rather than proposing a novel architecture, this work provides an engineering-grounded, transparent account of training a small scientific language model from scratch. We hope these insights support researchers operating under moderate compute budgets who seek to build domain-specialized models.
1 Introduction
The paper documents an engineering-focused, reproducible study of training a 1.36B-parameter scientific language model from raw arXiv sources. It examines dataset construction, tokenization, training dynamics, and scaling under constrained resources.
- Raw scientific-model training remains sparsely documented compared with systems using curated or proprietary corpora and undisclosed preprocessing.
- ArXiv’s heterogeneous LaTeX archives and inconsistent metadata make preprocessing choices important for dataset yield and training stability.
- The study trains a 1.36B-parameter scientific language model from raw arXiv sources without proposing a new architecture.
- The pipeline covers corpus construction, formula-aware tokenization, constrained-budget training dynamics, and empirical scaling behavior.
- 24 experimental runs analyze failure modes, optimization instabilities, preprocessing bottlenecks, hardware utilization, and 20GB versus 200GB data regimes.
- The contributions are primarily empirical and engineering-focused, including a reproducible corpus pipeline, quantitative yield-loss analysis, and tokenizer insights for symbolic-heavy text.
2 Training Data Scaling Considerations
The study constructs an 80GB arXiv-LaTeX corpus through staged extraction, filtering, cleaning, deduplication, and weighted mixture assembly. It emphasizes that usable data depends on preprocessing and infrastructure, while weighted sampling balances expertise with diversity.
- The corpus pipeline has four stages: source extraction, metadata filtering, LaTeX normalization and cleaning, and weighted mixture assembly.
- Raw arXiv data contain multiple versions, withdrawn manuscripts, non-English text, multi-file projects, custom macros, and external style dependencies.
- Filtering restricts subjects and publication dates while removing withdrawn or very short documents to preserve complete scientific arguments.
- LaTeX extraction causes unavoidable data loss from malformed sources or unconventional project layouts.
- Weighted sampling upsamples high-quality documents while retaining broader papers at lower weights to balance technical expertise and stylistic diversity.
- Scientific pretraining uses LaTeX corpora, while external conversational datasets are reserved for post-training alignment.
3 Tokenization
Scientific LaTeX’s symbolic density makes tokenization central to compression and stable representation learning. Exploratory tokenizer experiments informed a robust final choice, producing 52.18B scientific-pretraining tokens from approximately 200GB of curated data.
- Scientific corpora contain dense symbols, equations, operators, and specialized environments that generic tokenizers may fragment excessively.
- Fragmentation increases sequence length and weakens stable representations of formal reasoning patterns, making tokenization a core design decision.
- Tokenizer objectives were preserving LaTeX structure, reducing symbolic fragmentation, improving compression, and maintaining cross-domain consistency.
- Exploratory BPE and SentencePiece trials found that sampling strategy and vocabulary size significantly influenced symbolic segmentation patterns.
- The final model used a LLaMA-compatible SentencePiece tokenizer with approximately 102,400 tokens, favoring compatibility and stable initialization.
- The LLaMA tokenizer was sufficiently robust under current compute constraints, although domain-trained tokenizers remain promising future work.
- 52.18B tokens supported scientific pretraining, while 5B tokens were allocated to post-training and alignment data.
4 Model Architecture
The paper uses a dense, decoder-only transformer with approximately 1.36B parameters, configured for stable, deterministic training on constrained multi-GPU hardware.
- Architecture: The model is a dense, decoder-only transformer following the LLaMA architectural framework, with approximately 1.36B parameters and untied input and output embeddings.Separate embedding matrices increase representational flexibility at the cost of additional parameters.
- Architectural Specification: The configuration uses a 2048-dimensional hidden state, 16 key–value heads, a 5504-dimensional feed-forward layer, and a vocabulary of 102,400 tokens.The maximum context length is 4096 tokens, with SiLU activations.
- Architecture Choice: Dense architecture was selected for predictable convergence, deterministic compute per token, lower communication overhead, and domain specialization under a 2×A100 budget.The design prioritizes parameter efficiency over sparse capacity scaling.
5 Training Setup
Training combines a staged curriculum with memory- and throughput-oriented systems choices to adapt a 1.36B-parameter model to scientific text under moderate hardware constraints.
- Training Configuration: The pipeline balances scientific rigor, hardware constraints, and optimization stability on a dual A100 80GB GPU setup.
- Curriculum: A three-stage curriculum moves from textual warm-up to symbolic integration and then mixed prose–formula training.Abstracts, introductions, and conclusions precede full LaTeX bodies and a balanced mixture of content.
- Sequence Construction: Although the architecture supports a 4096-token context window, training sequences use 768 tokens to maximize batch throughput and stabilize memory utilization.
- Data Regime: The 52.18B-token corpus places the 1.36B-parameter model in a data-rich regime of approximately 38 tokens per parameter.This prioritizes domain robustness over strict compute optimality relative to the approximately 27B-token Chinchilla estimate.
6 Training Dynamics and Optimization Analysis
Across 24 runs, the experiments compare small- and full-data regimes, showing substantially more stable convergence and optimization in the 200GB setting while identifying practical infrastructure lessons.
- Iterative Optimization: 24 experimental runs varied dataset scale, learning rate, gradient accumulation, and preprocessing to refine optimization and hardware utilization.Early exploratory runs often terminated because of hyperparameters or memory constraints.
- Data-Scale Effects: 20GB training produced oscillating, high-plateau loss and unstable convergence, whereas full-data training reduced loss more smoothly with lower gradient noise.The small-data regime was used for pipeline validation before full-scale runs.
- Full-Data Convergence: Run 20 on 200GB showed smooth monotonic convergence with long-tail stabilization, while full-data runs exhibited classic transformer long-tail behavior.
- Validation and Overfitting: Validation loss decreased monotonically and remained aligned with training loss, with no sustained divergence or severe overfitting within the observed horizon.
- Gradient Stability: Gradient norms spiked during warm-up, stabilized below 1.0, and showed no late-stage explosion or vanishing gradients.The authors attribute this stable optimization behavior to the selected learning rate and batch configuration.
- Hardware Utilization: GPU utilization stayed above 95% with stable power, no ECC errors, and no persistent I/O stalls, indicating efficient pipeline throughput.
- Best Practices: The resulting practices emphasize conservative learning rates, full-scale data, gradient monitoring, storage throughput, and iterative small-scale debugging.The authors stress systematic experimentation and infrastructure-aware optimization for small scientific language models.
7 Evaluation
Evaluation primarily uses perplexity on held-out scientific data, showing strong familiarity with scientific notation and writing but no instruction-following or conversational behavior.
- Perplexity on held-out scientific validation data is the primary evaluation measure.
- The model demonstrates strong familiarity with mathematical notation, LaTeX structures, and formal scientific writing patterns.
- Raw scientific-corpus pretraining does not produce instruction-following or conversational behavior.
- The evaluation focuses on capabilities and limitations of a small domain-specialized model rather than competition with large instruction-tuned systems.
8 Empirical Observations
The training observations show that preprocessing and infrastructure materially shape usable data and practical training impact, with data engineering often rivaling architectural changes.
- Data yield depended more on preprocessing decisions than raw data availability.Archive validation, LaTeX cleaning, and filtering heuristics significantly affected usable token volume.
- Storage and I/O throughput were more limiting than raw compute capacity during early training stages.
- Applying language detection heuristics too early removed valid scientific documents containing dense symbolic content.
- Data engineering and pipeline design often outweighed architectural modifications in practical impact for small-scale training.
9 Limitations and Lessons Learned
The study’s limitations span infrastructure, preprocessing, scaling, context utilization, evaluation, domain coverage, and instruction alignment, constraining reproducibility and generality.
- Compute Constraints: 5,000–8,000 GPU-hours on dual A100 GPUs limited exploration of larger architectures, longer contexts, and broad hyperparameter sweeps.
- Storage and I/O Bottlenecks: Raw archives and intermediate artifacts required substantial storage capacity and high-throughput I/O, which became more limiting than GPU compute early on.
- Preprocessing Sensitivity: Small preprocessing heuristic changes caused large variations in usable token volume, introducing pipeline-induced bias.
- Scaling Regime Tradeoffs: 52.18B tokens with 1.36B parameters placed training in a data-heavy regime that may reduce marginal efficiency relative to strictly compute-optimal scaling.
- Context-Length Utilization: Training sequences were limited to 768 tokens despite a 4096-token context window, so long-context reasoning capacity may not be fully realized.
- Evaluation Scope: Perplexity on held-out scientific corpora does not directly measure reasoning correctness, theorem validity, or symbolic proof consistency.
- Domain Bias and Instruction Alignment: The dataset’s restriction to selected scientific categories limits general-domain adaptability, while the non-instruction-tuned model is unsuitable for direct conversational deployment.
- Reproducibility Constraints: Large-scale preprocessing, storage, and GPU requirements may limit exact reproducibility for researchers without similar infrastructure.
10 Conclusion
The paper concludes that transparent data engineering and tokenizer design are central to stable scientific specialization, while the implementation highlights configuration-dependent model and tokenizer behavior.
- 10 Conclusion: The work presents an end-to-end case study of training a 1.36B-parameter scientific model from raw arXiv LaTeX sources.
- 10 Conclusion: Careful data engineering can rival architectural scaling in impact for small-to-mid-scale models.
- 10 Conclusion: Transparent reporting of preprocessing, tokenization, and compute tradeoffs remains essential for reproducibility.
- 10 Conclusion: Future work includes longer-context training, instruction-aligned post-training, and formal mathematical reasoning benchmarks.
- Configuration and Tokenizer Behavior: In Hugging Face loading, model_type selects the configuration class and dispatches the corresponding model and tokenizer classes.
- Configuration and Tokenizer Behavior: Tokenizer files are loaded according to the selected tokenizer class, and mismatches can cause failure or incorrect behavior.
- Configuration and Tokenizer Behavior: If a model was trained with a different tokenizer, token IDs and embedding rows can become misaligned, producing meaningless output silently.
F.1.2 Model Architecture Effects
Using a LLaMA configuration for DeepSeek can produce shape mismatches or incorrect behavior because the architectures may differ. Tokenizer identity and correct model metadata are required for reliable loading and downstream compatibility.
- DeepSeek may differ from LLaMA in attention, rotary embedding scaling, mixture-of-experts layers, and RMSNorm usage.
- Tensor shape mismatches can cause hard failures, while successful loading may still produce incorrect behavior.
- A single token offset is sufficient to invalidate the model.
- DeepSeek can fork the LLaMA tokenizer while retaining identical vocabulary and compatible special tokens.
- Mislabeling model_type may appear to work but breaks chat templates, LoRA adapters, quantization configurations, and future library updates.
- Tokenizer files must exactly match those used during training, while model_type should remain deepseek when the LLaMA tokenizer is intentionally used.