Galactica: A Large Language Model for Science

1 Introduction

• Scientific papers and data have grown beyond individual researchers’ ability to read and organize, while search engines leave knowledge organization to costly human-curated resources.Galactica is proposed as a language-model interface that can store, combine, reason over, synthesize, and connect scientific knowledge across modalities.Source: §1 · pp. 1–2

1.1 Our Contribution

• Galactica is trained on over 48 million papers, textbooks, lecture notes, compounds, proteins, websites, encyclopedias, and other curated scientific sources.Its curated corpus supports multiple training epochs without overfitting, with upstream and downstream performance improving through repeated tokens.Source: §1.1 · p. 2
• A common markdown format, task-specific datasets and tokens let Galactica blend sources, predict citations, support reasoning, and interface with SMILES and protein sequences through natural language.The interface is designed to compose scientific knowledge into new task contexts and achieves state-of-the-art results across many scientific tasks.Source: §1.1 · p. 2
• 41.3% versus 35.7% on mathematical MMLU and 20.4% versus 8.8% on MATH show Galactica outperforming Chinchilla and PaLM 540B, respectively.The 30B model also beats PaLM 540B on MATH with 18 times fewer parameters.Source: §1.1 · p. 2
• 68.2% versus 49.0% on LaTeX equations and state-of-the-art scores of 77.6% on PubMedQA and 52.9% on MedMCQA dev demonstrate strong scientific knowledge performance.The LaTeX comparison is against the latest GPT-3; PubMedQA and MedMCQA dev are downstream scientific tasks.Source: §1.1 · p. 2
• Citation prediction improves smoothly with scale, approaches the reference citation distribution, and outperforms tuned sparse and dense retrieval approaches.This supports replacing document storage and retrieval with context-associative information stored in model weights.Source: §1.1 · p. 2

Large Language Models (LLMs)

• LLMs achieve broad task performance, few-shot learning, and improved reasoning with scale and suitable prompting (Brown et al., 2020; Rae et al., 2021; Hoffmann et al., 2022; Black et al., 2022; Zhang et al., 2022; Chowdhery et al., 2022; Kojima et al., 2022; Lewkowycz et al., 2022).They also support scientific knowledge tasks such as MMLU (Hendrycks et al., 2020).Source: §2 · p. 3

Dataset Selection

• Self-supervised general corpora can encode misinformation, stereotypes, and bias, waste compute on low-transfer tokens, and represent scientific terms and mathematics inefficiently (Sheng et al., 2019; Kurita et al., 2019; Dev et al., 2019; Blodgett et al., 2020; Sheng et al., 2021).For example, 50% of the PaLM corpus consists of social media conversations (Chowdhery et al., 2022).Source: §2 · p. 3

Scientific Language Models

• Curated scientific corpora benefit scientific language models, but prior datasets and models were typically much smaller than general-model corpora (Beltagy et al., 2019; Gu et al., 2020; Shin et al., 2020; Hong et al., 2022).Protein-sequence and SMILES Transformers also learn useful representations, although SMILES has descriptive limitations for chemical structure (Honda et al., 2019; Nijkamp et al., 2022).Source: §2 · p. 3

Scaling Laws

• Scaling-law work links loss to model size, data, and compute, while later analyses emphasize downstream performance and optimal data quantities (Kaplan et al., 2020; Tay et al., 2022a; Hoffmann et al., 2022).Galactica investigates whether training on repeated tokens can improve both upstream and downstream performance.Source: §2 · p. 3

Language Models as Knowledge Bases

• Large language models can serve as implicit knowledge bases for general and specialist tasks without external retrieval, but stored information may be unreliable because models blend facts and hallucinate (Brown et al., 2020; Hendrycks et al., 2020).Updating network knowledge and improving generation reliability remain active research questions (Scialom et al., 2022; Mitchell et al., 2022; Gao et al., 2022), while Galactica probes scientific knowledge depth and its scaling behavior.Source: §2 · p. 3

Retrieval-Augmented Models

• Retrieval-augmented models reduce the capacity required for knowledge storage but depend on retrieval infrastructure; fine-grained scientific knowledge may require retrieval even for larger models (Borgeaud et al., 2021).Galactica focuses on how far model weights alone can go while recognizing retrieval augmentation as important future work.Source: §2 · p. 3

3 Dataset

• Galactica trains on 106 billion tokens spanning scientific text, protein sequences, chemical formulae, LaTeX, and academic code in a small, curated corpus.The corpus combines papers, reference material, encyclopedias, textbooks, natural sequences, and computational-science data; full components and filtering are provided in the Appendix.Source: §3 · pp. 4–5

3.1 Tokenization

• Specialized tokenization represents citations, step-by-step reasoning, mathematics, numbers, chemical SMILES, amino-acid sequences, and DNA sequences with modality-specific tokens or character-level units.Citation markers use [START_REF] and [END_REF], while reasoning uses <work>; mathematical operations and digits are split into individual tokens, and biological or chemical sequences use character-based tokenization.Source: §3.1 · p. 5

3.1.1 Working Memory Token, <work>

• The <work> token wraps step-by-step reasoning and supports external computation when a calculation exceeds reliable single-pass processing.Its training datasets include programmatically generated, online-sourced, and transformed reasoning data, while prompt coverage is limited in size and diversity; Python offloading was not disabled in experiments.Source: §3.1.1 · pp. 6–7

3.1.2 Citation Token

• Citation tokens encode the implicit citation graph, while title-based identifiers improve citation prediction accuracy but are more prone to hallucination at smaller scales than alphanumeric IDs.The paper therefore uses title processing while noting the trade-off between citation accuracy and hallucination risk.Source: §3.1.2 · p. 8

3.2 Prompt Pre-Training

• Prompt pre-training adds diverse task prompts alongside general scientific data to strengthen task signals at lower model and data scales while preserving broad task generality.The paper distinguishes in-domain results, whose training data appears in pre-training, from out-of-domain results, whose training data does not, and does not present prompt pre-training as an alternative to instruction tuning.Source: §3.2 · pp. 8–9

4.1 Architecture

• Galactica uses a decoder-only Transformer with GeLU activations, a 2048-token context window, no biases, learned positional embeddings, and a 50k-token BPE vocabulary.The vocabulary was generated from a randomly selected 2% subset of the training data; ALiBi was tested at smaller scales but not adopted because it produced no large gains.Source: §4.1 · p. 10

4.2 Models

• Training uses AdamW with β1 = 0.9, β2 = 0.95, weight decay 0.1, gradient clipping at 1.0, linear learning-rate decay to 10%, and dropout p = 0.1.Longer warmup was important for the largest model early in training, potentially protecting against bad initialization, although the authors note that this advice may not generalize.Source: §4.2 · p. 10

4.3 Libraries and Infrastructure

• The 120B model trains on 128 NVIDIA A100 80GB nodes and requires one A100 node for inference, constraining the maximum model size for downstream accessibility.Training uses the metaseq library built by the NextSys team at Meta AI.Source: §4.3 · p. 10
• Validation loss continues falling through four training epochs for all model sizes, while the 120B model shows initial overfitting at the start of epoch five and is early-stopped.This behavior is reported in Figure 6.Source: Figure 6 · p. 11

5.1 Repeated Tokens Considered Not Harmful

• Four training epochs continue lowering validation loss, with the 120B model only beginning to overfit at the start of the fifth epoch.The 30B and 120B models show epoch-wise double descent, while BIG-bench performance shows no overfitting after four corpus repeats.Source: §5.1 · p. 11, Figure 8 · p. 12

5.2.1 LaTeX Equations

• Equation knowledge increases smoothly with scale, and Galactica outperforms larger general-corpus language models on zero-shot LaTeX equation probes.The probe contains 434 equations spanning chemistry, physics, mathematics, statistics, and economics.Source: §5.2.1 · p. 13, Table 6 · p. 13

5.2.2 Domain Probes

• Domain-probe performance generally scales steadily, especially for fine-grained factual queries, although BioLAMA remains an exception likely because of zero-shot prompt difficulty.The probes cover amino acids, biomedical facts, chemical reactions, galaxy-cluster classifications, and mineral groups.Source: §5.2.2 · pp. 13–14

5.2.3 Reasoning

• Galactica 30B outperforms PaLM 540B on MATH with both chain-of-thought and <work> prompts, despite being 18 times smaller.On mathematics MMLU, the <work> token boosts performance over Chinchilla, including for the smaller 30B model; Minerva outperforms base Galactica on direct comparisons.Source: §5.2.3 · p. 15, Table 9 · p. 15

5.3 Downstream Scientific NLP

• 77.6% on PubMedQA and 52.9% on MedMCQA dev exceed fine-tuned state-of-the-art scores of 72.2% and 41.0%, respectively.Performance is also close to state of the art on BioASQ and MedQA-USMLE, while Galactica tends to outperform Chinchilla on mathematical and graduate-level tasks but lags on some high-school and memorization-intensive tasks.Source: §5.3 · p. 16, Table 10 · p. 16

5.4 Citation Prediction

• At larger scales, Galactica outperforms retrieval-based citation approaches, while predicted citation distributions become closer to ground truth and less biased toward popular papers.Citation-prediction accuracy increases smoothly with scale, and the Kolmogorov-Smirnov distance declines as model size grows.Source: §5.4.1 · p. 17, Table 11 · p. 17, §5.4.2 · p. 18

5.5 General Capabilities

• Both Galactica 30B and 120B outperform the larger OPT and BLOOM models on 57 primarily non-scientific BIG-bench tasks.The evaluation uses five-shot prompts whose style was excluded from pre-training, making the comparison comparable across models.Source: §5.5 · p. 19, Table 12 · p. 19

5.6 Chemical Understanding

• Galactica learns chemical tasks such as IUPAC naming self-supervised, with naming accuracy improving smoothly with scale despite pretraining on only 2 million compounds.Attention patterns during prediction are interpretable: for example, the model attends to the −NH2 substituent when generating “amino.”Source: §5.6 · p. 19, §5.6.1 · p. 20, Table 13 · p. 20

5.6.2 MoleculeNet

• MoleculeNet performance scales with model size, but lags a specialist using explicit 3D information and ten times more molecules, suggesting weak supervision limits task signal.Training sets were converted to text and included during pre-training, with randomized natural-language prompts; evaluation used recommended DeepChem splits and ROC-AUC.Source: §5.6.2 · pp. 22–23, Table 15 · p. 23

5.7 Biological Understanding

• Galactica learns an implicit measure of protein-sequence similarity that supports functional annotation and free-form descriptions from biological modalities.The study investigates a single multimodal model using a restricted Swiss-Prot subset of 0.5 million proteins.Source: §5.7 · p. 24

5.7.1 Sequence Validation Perplexity

• Perplexity scales smoothly on three validation sets, whereas the sequence-similarity-constrained CASP set levels off; on a particular dataset, validation perplexity falls until the fourth epoch, then overfits.Source: §5.7.1 · p. 25

5.7.2 Functional Keyword Prediction

• F1 score increases with scale across protein keyword holdouts but saturates on CASPSimSeq, indicating dependence on sequence similarity to training data.Attention visualizations did not yield biologically interpretable patterns; the authors hypothesize keyword prediction relies on implicit sequence matching rather than identifiable domains.Source: §5.7.2 · p. 26, Table 17 · p. 26

5.7.3 Protein Function Description

• ROUGE-L increases smoothly across protein-function holdouts, and generated descriptions can match reference functions when similar sequences appear in training.The authors suggest sequence-based language models could complement search methods such as BLAST and MMseqs2.Source: §5.7.3 · p. 27, Table 18 · p. 27

6 Toxicity and Bias

• Galactica is significantly less biased and toxic than existing language models across benchmarks covering stereotypes, toxicity, and misinformation.Source: §6 · p. 27

6.1 Bias and Stereotypes

• The bias and stereotype evaluations use four widely used benchmarks to assess Galactica’s detection and generation of harmful stereotypes and hate speech.Source: §6.1 · p. 28

6.1.1 CrowS-Pairs

• Galactica attains a 60.5% overall CrowS-Pairs score and exhibits significantly lower stereotypical bias than GPT-3 and OPT 175B in most categories, except sexual orientation and age.Source: §6.1.1 · p. 28, Table 19 · p. 28

6.1.2 StereoSet

• Galactica outperforms all other models across every StereoSet category on the overall ICAT score.ICAT combines language-modeling and stereotype scores as a balanced measure of bias detection and language modeling.Source: §6.1.2 · p. 29, Table 20 · p. 29

6.1.3 Toxicity

• Galactica produces substantially lower toxicity rates than other models on RealToxicityPrompts, even as original prompt toxicity increases.The evaluation samples 25 20-token generations for each of 5,000 prompts using nucleus sampling with p=0.9.Source: §6.1.3 · p. 29, Figure 22 · p. 30

6.2 TruthfulQA

• Galactica exceeds other language models on TruthfulQA, but absolute performance remains low and improves with scale only slowly.TruthfulQA contains 817 questions spanning health, law, finance, and other categories.Source: §6.2 · pp. 29–30, Table 21 · p. 30

7.1 Limitations

• Open-access restrictions and limited molecule and protein coverage constrain the corpus; adding closed sources and larger modality corpora could considerably improve performance.The paper identifies both external and self-imposed corpus constraints.Source: §7.1 · p. 30
• Benchmark gains are not disentangled between pre-training prompts and the scientific corpus, leaving general language capabilities without prompt boosting unresolved.Future work should separate these effects to determine what the scientific corpus alone enables.Source: §7.1 · p. 31
• Citation bias toward popular papers remains at 120B scale, while prompt pre-training versus instruction tuning lacks direct ablation comparisons and may require production augmentation.The paper calls for comparing these training approaches and augmenting the model before production use.Source: §7.1 · p. 31
• Galactica is not intended for general-knowledge tasks and lacks geometric representations such as 3D atomic coordinates, despite broad societal knowledge and strong text-based scientific representations.The chemistry example emphasizes that text is not sufficient to capture all scientific modalities.Source: §7.1 · p. 31

7.2 Future Work

• Mixture-of-denoising objectives and context windows longer than 2048 tokens could improve scientific modalities, long-form documents, and long protein sequences.The paper particularly identifies left-to-right modeling as limiting for protein sequences.Source: §7.2 · p. 31
• Extending Galactica to images is a natural follow-up because scientific knowledge cannot be adequately captured without images, though architectural modification may be required.Existing multimodal work is cited as a possible direction.Source: §7.2 · p. 31
• Future work should broaden <work> prompts and benchmarks, while developing verification layers that ensure generations are correct and factual for production applications.The paper proposes more prompt diversity, additional benchmarks, and verification beyond scientific applications.Source: §7.2 · p. 31
• Further research should determine how to incorporate new scientific knowledge through continual learning and complement Transformer weight memory with retrieval augmentation for fine-grained knowledge.The paper leaves retraining from scratch versus older checkpoints as an open question.Source: §7.2 · p. 31

• The work targets the human-effort bottleneck in organizing scientific knowledge and releases its models so the open machine learning community can extend them.The authors envision language models taking on knowledge tasks that are currently human specialisms.Source: §8 · p. 32
• Language models absorb technical knowledge, scale with model size, and outperform tuned sparse and dense retrieval pipelines on citation prediction.Their context-associative power may give them long-term advantages over search engines for exploring scientific literature.Source: §8 · p. 32
• Working-memory tokens enable strong mathematical MMLU and MATH performance by composing curated knowledge step by step.High-quality step-by-step datasets remain a bottleneck, while human-like performance may require adaptive computation.Source: §8 · p. 32
• Galactica bridges scientific modalities and natural language through self-supervised IUPAC naming, prompted MoleculeNet drug discovery, and automatic protein annotation.More and larger datasets connecting natural language with natural sequences could further improve performance.Source: §8 · p. 32

A.1 Dataset Components

• The section covers the various components of the corpus.Source: §A.1 · p. 42

Sources and corpus scale

• Scientific papers are sourced from preprint servers such as arXiv and PMC, alongside other sources.Source: §A.1.1 · p. 42
• The Semantic Scholar full-text dataset captures the long tail of science, while quality filters exclude papers based on journal keywords and low impact factor.Filter details are provided in the Appendix.Source: §A.1.1 · p. 42
• 48 million papers, including abstracts and full texts, comprise the dataset through July 2022, with abstracts used when full texts are not open access.Source: §A.1.1 · p. 42

Document processing

• Modified GROBID converts PDFs to text and extracts metadata, while available LaTeX is combined with GROBID output to recover mathematical content.Source: §A.1.1 · p. 42
• Final paper documents are stored in Markdown to support knowledge blending across sources, with citations processed using the title-based approach of Section 2.2.Source: §A.1.1 · p. 42

A.1.2 Reference Material

• Reference material includes encyclopedias, textbooks, and educational resources that provide a base for model learning.The paper directs readers to Table 23 for further details.Source: §A.1.2 · p. 42
• Source-specific processing selects scientific StackExchange questions and applies prompt randomization to Papers with Code and IUPAC Goldbook content.Some sections are reframed as questions and answers, such as asking what a machine-learning method is.Source: §A.1.2 · p. 42
• KhanAcademy articles receive <work> tokens to mark step-by-step reasoning examples.The paper explains this tokenization approach in Section 2.4.Source: §A.1.2 · p. 42
• The processing pipeline preserves mathematical LaTeX and captures citations, including hyperlinks to papers.These preservation steps support retention of technical notation and references in the source material.Source: §A.1.2 · p. 42

Knowledge sources

• Fine-grained scientific knowledge bases provide a dedicated source of knowledge for the corpus, with the included resources detailed in Table 24.The corpus also includes reference material listed in Table 23.Source: §A.1.3 · p. 42, Table 24 · p. 43, Table 23 · p. 43

Data formatting and augmentation

• Chemistry and biology data wrap SMILES and protein sequences in specialized tokens, while UniProt documents receive format and order augmentation.Order randomization supports both sequence-to-property and property-to-sequence learning; format randomization sometimes converts descriptions into question-and-answer form.Source: §A.1.3 · p. 43
• For NASA Exoplanet, the characteristics undergo order randomization to vary their presentation.Source: §A.1.3 · p. 43

Dataset selection

• To reduce bias toward natural sequences over natural language, chemical and biological sequence datasets use small subsets of available entities.PubChem Compound contributes 2 million of 110 million compounds, and UniProt contributes 0.5 million reviewed Swiss-Prot proteins of 227 million total proteins.Source: §A.1.3 · p. 43

Scope and limitation

• The sequence-data constraint can be relaxed to build a much larger corpus, but this work focuses first on whether one model can learn effectively in a multimodal setting.Source: §A.1.3 · p. 43

A.1.4 Common Crawl

• Scientific Common Crawl data is filtered with fastText and manual annotation into 200 high-quality scientific and reference domains, while Academic Common Crawl PDFs are processed with GROBID.The extracted Common Crawl text is generally poor, motivating stringent filters and leaving improved capture of foundational scientific knowledge as future work.Source: §A.1.4 · p. 44

A.1.5 Code / A.1.6 <work> Datasets / A.2 Dataset Deduplication

• The corpus pipeline collects licensed academic GitHub repositories, constructs several <work> datasets, and deduplicates identical spans of at least 100 bytes by removing repeated paragraph-bounded content.The Workout dataset contains 921 examples and is described as the highest-quality, reasonably diverse <work> dataset, while GSM8k and OneSmallStep have error or diversity limitations.Source: §A.1.5 · p. 44, §A.1.6 · p. 44, §A.2 · p. 45

A.3 Citation Identifier Ablations

• Citation processing ablation uses 6.7 billion-size models on Papers with Code method and dataset names, finding that title processing has higher accuracy but greater hallucination risk.The evaluation includes 1,705 methods and 4,735 datasets.Source: Table 26 · p. 45, §A.3 · p. 45, Table 27 · p. 45

A.6 Prompt Pre-training Datasets / A.7 Scientific Evaluations

• The appendix includes prompt-based datasets and evaluations spanning chemical properties, docking, MMLU, entity and method description, sequence translation, question answering, and true-or-false tasks, while S2 excludes several fields, low-impact journals, and non-English papers.The S2 restrictions are acknowledged as potentially relaxable in future work.Source: §A.6 · p. 48, §A.7.1 · p. 52, §A.7.2 · p. 52

A.6.1 Chemical Property Prediction

• Scaling generally lowers chemical and physical property RMSE, but the improvement tails off at 120B, suggesting that more molecule data may be needed.The task uses 17,052 compounds and prompts based on PubChem document structure, predicting properties from SMILES sequences.Source: §A.6.1 · p. 50

A.6.2 Docking Regression

• Galactica infers docking scores from sequences for three targets from 1.3B parameters onward, but does not solve ESR2 or PGR, indicating that geometry may be needed.The text representation encodes a protein target and ligand as a natural-language question whose answer is the docking score.Source: §A.6.2 · p. 50, Figure 25 · p. 50

Introduction

• Hartree–Fock, also called the Self-Consistent Field method, derives orbitals variationally by minimizing the Slater-determinant energy and permits localized non-canonical orbitals through unitary transformations.Canonical Hartree–Fock orbitals need not be spatially localized, while infinitely many non-canonical orbitals share the same Hartree–Fock energy.Source: Introduction · p. 57

Single-Electron Approximation

• The single-electron approximation is introduced through the time-independent Schrödinger equation, whose Hamiltonian combines kinetic and potential-energy operators acting on the wavefunction.The kinetic-energy operator is represented by the Laplacian, a second spatial derivative.Source: Single-Electron Approximation · p. 57

A.8.8 I’m sorry Frank, I think you missed it

• The checkmate-in-one evaluation used ASCII chess boards and move prompts, reported accuracy in Table 38, and was described as state-of-the-art among other large language models, though further work was needed.The dataset comprised 19,426 public chess games included in pre-training, with the remainder reserved for validation.Source: Table 38 · p. 58, §A.8.8 · p. 58