PaperlayerView exact revision on arXiv

Source-linked AI summary

Galactica: A Large Language Model for Science

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, Robert Stojnic

arXiv:2211.09085v1cs.CLstat.ML

TL;DR

Scientific literature and data have become too extensive for researchers to fully read and organize, while search engines do not directly structure scientific knowledge. Galactica is a language model trained on a curated scientific corpus to store, combine, and reason over that knowledge. It performs strongly across technical, reasoning, knowledge-intensive, and cross-modal scientific tasks, supporting its potential as a new interface for science.

  • Problem

    The growth of scientific literature and data has made it impossible for one person to read all papers in a field or organize the underlying phenomena, while search engines do not organize knowledge directly.

  • Method

    Galactica is trained on a large curated scientific corpus and uses an interface designed to combine scientific knowledge across task contexts and modalities.

  • Results

    Galactica performs strongly across scientific knowledge, reasoning, citation prediction, question answering, and modality-bridging tasks, including mathematical MMLU, MATH, PubMedQA, MedMCQA, and protein annotation.

  • Takeaways & Limitations

    Language models may provide a valuable new interface for exploring scientific literature and taking on knowledge tasks that are currently human specialisms.

  • Takeaways & Limitations

    The corpus is limited by open-access availability and self-imposed restrictions on molecules and proteins, and benchmark gains are not disentangled between prompts and the scientific corpus.

Abstract

from arXiv · show

Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.

1 Introduction

Scientific literature and data have outgrown researchers’ ability to read, organize, and connect them, while search engines mainly point to human-produced secondary resources. Galactica addresses this bottleneck with a curated scientific language model designed to store, combine, reason over, and organize knowledge across scientific tasks.

  • Motivation: 516 papers per day were submitted to arXiv in May 2022, while GenBank contained 1.49 × 10^12 nucleotide bases by August 2022.The paper argues that this volume makes it impossible for one person to read all papers in a field or organize all underlying scientific data.
  • Motivation: Search engines provide access to scientific knowledge but do not organize it directly, leaving researchers dependent on costly human-created reviews, encyclopedias, and annotations.Examples include Wikipedia, UniProt, and PubChem Compound.
  • Contribution: Galactica is a large language model trained on a curated corpus of over 48 million papers, textbooks, lecture notes, compounds, proteins, websites, and encyclopedias.The authors report that repeated training epochs improve upstream and downstream performance without overfitting.
  • Contribution: Galactica uses a common markdown format, task-specific datasets and tokens, citation tokens, and step-by-step reasoning mechanisms to compose scientific knowledge into task contexts.The interface supports citation prediction and different types of scientific knowledge.
  • Results: 41.3% versus 35.7% was achieved against Chinchilla on mathematical MMLU, while Galactica scored 20.4% versus PaLM 540B’s 8.8% on MATH.The paper also reports 68.2% versus GPT-3’s 49.0% on LaTeX equations, plus 77.6% on PubMedQA and 52.9% on MedMCQA dev.
  • Results: Galactica’s interface improves citation prediction with scale, outperforms tuned sparse and dense retrieval, and supports multimodal tasks involving chemical formulas and protein sequences.Reported applications include drug-discovery prompting, IUPAC name prediction, and natural-language protein annotation.

2 Related Work

Prior work established the capabilities and scaling behavior of large language models, while also highlighting risks from uncurated data and unreliable weight-based knowledge. Scientific language models and sequence models showed the value of domain-specific corpora, but generally remained smaller in scale or scope; retrieval-augmented systems trade lower capacity needs for infrastructure requirements.

  • Large Language Models: Large language models achieve breakthrough NLP performance and learn in context through few-shot learning, including on scientific knowledge tasks such as MMLU.This work is presented against a background of increasingly capable general-purpose models.
  • Large Language Models: Uncurated training data can reproduce misinformation, stereotypes, and bias, while including tokens with limited transfer value for scientific use cases.The paper frames these issues as undesirable for scientific tasks that value truth and efficient compute use.
  • Scientific Language Models: Scientific language models such as SciBERT and BioLM show benefits from curated scientific corpora, but prior datasets and models were typically smaller in scale and scope than general-model corpora.Protein-sequence and SMILES transformers also suggested that natural representations can be learned beyond ordinary scientific text.
  • Scaling Laws: Scaling-law research links loss to model size, dataset size, and training compute, while later work shows upstream perplexity does not always correlate with downstream performance.Optimal data quantity is also treated as an important factor in scaling analyses.
  • Knowledge Bases and Retrieval: Weight-based knowledge is pliable for associating information but can blend facts and hallucinate, whereas retrieval-augmented models reduce capacity needs at the cost of supporting retrieval infrastructure.The paper identifies updating network knowledge and improving generation reliability as active research questions.
  • Scientific Language Models: S2ORC contains fewer than 20bn tokens, while GPT-3 and PaLM corpora contain at least 300bn tokens; ScholarBERT combines over 200bn tokens with a 770M-parameter model.These figures illustrate the scale gap between earlier scientific corpora and general-model training corpora.

3 Dataset

Galactica’s dataset combines curated scientific knowledge across natural-language and structured modalities, with specialized tokens and prompt data designed to support reasoning and task use. The approach is motivated by information overload and the limits of existing working-memory and task-learning strategies.

  • Corpus design: Galactica combines papers, textbooks, encyclopedias, scientific websites, proteins, compounds, code, and other sources into a unified scientific corpus.The corpus contains 106 billion tokens and blends natural-language sources with protein sequences and chemical formulae.
  • Corpus design: The dataset is deliberately small and curated relative to larger uncurated LLM corpora, testing whether a normative scientific corpus can support a working language model.The paper frames curation as a key question for designing purpose-specific language models.
  • Tokenization: Specialized tokenization represents citations, working-memory traces, mathematics, numbers, SMILES formulas, amino-acid sequences, and DNA sequences.The design uses modality-specific delimiters and character- or digit-level tokenization where appropriate.
  • Working memory: The <work> token wraps step-by-step reasoning, while difficult computations can be offloaded by writing and executing a Python program.This external working-memory approach is intended to bridge intermediate reasoning steps and specialized computation.
  • Prompt pre-training: Prompt examples are included during pre-training to strengthen task signals at lower scales while preserving broader task generality better than instruction tuning.The paper uses prompts alongside general data so users can apply the model to tasks such as question answering and summarization without fine-tuning.

4 Method

Galactica uses a decoder-only Transformer with fixed architectural choices and standard optimization procedures across model sizes. Training uses repeated epochs, with early stopping applied when the largest model begins to overfit.

  • Architecture: Galactica uses a decoder-only Transformer architecture with GeLU activations, a 2048-token context window, no biases, and learned positional embeddings.The same context-window choice is used for all model sizes; ALiBi was tested at smaller scales but not adopted.
  • Optimization: The models are trained with AdamW, gradient clipping, learning-rate decay, dropout, and attention dropout.Optimization uses β1 = 0.9, β2 = 0.95, weight decay 0.1, gradient norm clipping at 1.0, and dropout probabilities of 0.1.
  • Systems: Training uses the metaseq library and scales to 128 NVIDIA A100 80GB nodes for the 120B model.Inference for Galactica 120B requires a single A100 node, which constrains the maximum model size for downstream accessibility.
  • Training duration: Validation loss continues falling through four training epochs across model sizes, while the 120B model shows initial overfitting at the start of the fifth epoch.The 120B model is early-stopped at that point.

5 Results

Galactica shows strong scientific knowledge, reasoning, downstream, multimodal, and general capabilities, with performance often improving as model scale increases. The results also identify scope boundaries and unresolved questions around data composition, weak supervision, and interpretability.

  • Repeated tokens: Validation loss falls through four training epochs, while the 120B model begins overfitting only at the start of the fifth epoch.The 30B and 120B models also exhibit an epoch-wise double descent effect, with stronger effects at larger scale.
  • Repeated tokens: Galactica shows no signs of overfitting on 57 BIG-bench tasks, suggesting repeated tokens improve downstream and upstream performance.BIG-bench uses principally non-scientific tasks and prompt types excluded from pre-training.
  • Scientific knowledge: Equation knowledge increases smoothly with scale, and Galactica outperforms larger general-corpus language models on these knowledge probes.The authors attribute this result to the value of a curated dataset.
  • Reasoning: Galactica 30B outperforms PaLM 540B on both chain-of-thought and <work> prompts despite using 18 times fewer parameters.Galactica also outperforms the base PaLM model by a significant margin with both prompting methods.
  • Downstream scientific NLP: Galactica achieves state-of-the-art scores of 77.6% on PubMedQA and 52.9% on MedMCQA dev.Performance on BioASQ and MedQA-USMLE is close to fine-tuned state-of-the-art results.
  • General and multimodal capabilities: Performance increases smoothly with scale across literature, chemical, and protein tasks, but weaker supervision, limited molecular data, and graduate-knowledge bias constrain some results.The protein model learns an implicit sequence-similarity measure, while chemical attention can be interpreted through underlying atomic structure.

6 Toxicity and Bias

Galactica is evaluated on stereotypes, toxicity, and truthfulness, generally outperforming comparison language models while retaining low absolute performance on TruthfulQA. Its lower toxicity and bias rates are associated with a curated scientific corpus, but some benchmark exceptions remain.

  • Overview: Galactica is significantly less biased and toxic than existing language models across the evaluated benchmarks.The evaluation covers stereotypes, toxicity, and misinformation-related behavior.
  • Interpretation: The authors suggest that Galactica’s curated scientific corpus may contribute to lower stereotype and discriminatory-text incidence.This is presented as a likely explanation rather than a disentangled causal finding.
  • Stereotype bias: Galactica outperforms all other models across StereoSet categories on the overall ICAT score.ICAT balances bias detection with language-modeling performance.
  • Toxicity: Galactica produces substantially lower toxicity rates than other models on RealToxicityPrompts.The evaluation measures mean toxicity probability of generated continuations across prompt-toxicity buckets.
  • Truthfulness: Galactica exceeds other language models on TruthfulQA, but absolute performance remains low and improves slowly with scale.TruthfulQA measures the truthfulness of generated answers.

7 Limitations and Future Work

The paper identifies limitations in corpus coverage, attribution of performance gains, citation bias, modality coverage, and general-purpose use. Future work proposes broader data and modalities, improved objectives and context, verification, continual learning, and retrieval augmentation.

  • Corpus Limitations: Open-access restrictions exclude much scientific knowledge, while molecule and protein limits constrain corpus coverage and likely performance.The authors expect access to closed sources and larger modality corpora to improve results.
  • Corpus Effects vs Prompt Effects: Performance gains are not disentangled between pre-training prompts and the core scientific corpus.Future work should test whether a scientific corpus alone supports general language capabilities without prompt boosting.
  • Citation Bias: Citation prediction still favors popular papers at 120B scale and likely requires augmentation before production use.The model approaches the true citation distribution with scale, but residual popularity bias remains.
  • Prompt Pre-Training vs Instruction Tuning: The study does not directly compare prompt pre-training with instruction tuning through ablations, leaving their trade-offs unclear.The paper uses prompt pre-training and identifies instruction tuning as an untested alternative.
  • General Knowledge: Galactica is not intended for tasks requiring broad general knowledge despite absorbing some societal knowledge.The authors specifically caution against using it for this use case.
  • Text as a Modality: Text representations do not capture scientific knowledge adequately when geometry is fundamental, because Galactica lacks atomic 3D coordinates.Chemistry is given as the concrete example.
  • Future Work: Future directions include mixture-of-denoising objectives, longer context windows, image extensions, more reasoning examples, verification, continual learning, and retrieval augmentation.These directions target scientific modalities, long documents, multimodal knowledge, factual reliability, updating, and fine-grained knowledge.
  • Future Work: Retrieval augmentation is proposed to complement the Transformer’s flexible weight memory for fine-grained scientific knowledge.The paper presents retrieval as a continuing need even after absorbing large scientific corpora.

8 Discussion and Conclusion

The paper explores language models as a new interface for scientific knowledge, addressing the human effort required to reason over and organize information. Galactica shows strong technical, reasoning, and cross-modal capabilities, while the authors identify adaptive computation as an architectural boundary.

  • Language models could disrupt the store-and-retrieve paradigm by combining, reasoning about, and organizing scientific information.The paper frames this as a response to the knowledge throughput bottleneck created by reliance on human effort.
  • Language models absorb technical knowledge, including LaTeX equations and chemical reactions, with capabilities that scale smoothly with model size.
  • Galactica outperforms tuned sparse and dense retrieval pipelines on citation prediction.
  • Galactica composes curated knowledge for knowledge-intensive question answering and step-by-step reasoning on mathematical benchmarks.
  • Language models can bridge scientific modalities and natural language through tasks including IUPAC naming, MoleculeNet prompting, and protein annotation.The paper suggests that more datasets connecting natural language and natural sequences could improve performance.
  • The authors open-source Galactica models so the machine learning community can extend this work.

A.1 Dataset Components

The corpus section introduces the dataset components used to assemble Galactica’s scientific knowledge base.

  • The corpus combines multiple components to provide the scientific knowledge used by the model.

A.1.1 Papers

The paper component of the corpus draws on papers and abstracts from preprint servers, open-access sources, and Semantic Scholar, with quality filtering and standardized processing.

  • Scientific papers are sourced from preprint servers including arXiv and PMC, alongside other sources.
  • Semantic Scholar full text is included to capture the long tail of science, with filters excluding selected journals and low-impact-factor papers.
  • 48 million papers, abstracts, and full texts comprise the dataset through July 2022.
  • Table 22 summarizes the paper sources used in the corpus.
  • GROBID converts PDFs to text and extracts bibliographic information, while available LaTeX sources help recover mathematical content.
  • Documents are stored in markdown to support knowledge blending, and papers undergo title-based citation processing.

A.1.2 Reference Material

Reference material supplements the paper corpus with encyclopedias, textbooks, educational content, and specially processed instructional datasets.

  • Encyclopedias, textbooks, and educational material provide base reference content for the model.
  • The corpus applies source-specific processing to several datasets.
  • Scientific StackExchange questions are included, while Papers with Code and IUPAC Goldbook receive prompt-randomization augmentation.
  • KhanAcademy articles receive <work> tokens for step-by-step reasoning examples.
  • The processing preserves mathematical LaTeX and captures citations, including hyperlinks to papers.

A.1.3 Knowledge Bases

Galactica incorporates fine-grained scientific knowledge bases, adapting chemical and biological modalities for text-based learning. The datasets use selective sampling and format randomization, while one stated constraint is intended for future relaxation.

  • Fine-grained knowledge is sourced from scientific knowledge bases, with details summarized in Table 24.
  • Chemical and biological modalities such as SMILES and protein sequences receive specialized tokens.
  • UniProt augmentation randomizes sequence order and sometimes converts descriptions into question-and-answer format.These transformations support learning both sequence-to-property and property-to-sequence mappings.
  • Chemical and biological datasets use small subsets, including 2 million of 110 million PubChem compounds and 0.5 million of 227 million UniProt proteins.
  • The paper describes the dataset-size constraint as relaxable in future work while initially investigating whether one model can learn effectively across modalities.

A.1.4 Common Crawl

The paper constructs scientific and academic corpora from filtered Common Crawl material, repositories, and transformed problem datasets. It also documents data-quality, annotation, representation, and coverage limitations affecting future scaling and some tasks.

  • Common Crawl: Scientific Common Crawl webpages are filtered with a fasttext classifier and manual annotation to produce 200 high-quality scientific and reference domains.The classifier is trained using a noisy set of 600 domains.
  • Common Crawl: Academic Common Crawl processing extracts PDFs from academic domains and processes them with GROBID, without LaTeX-processing pages.
  • Common Crawl: Poor extracted-text quality in Common Crawl motivates stringent filters and is identified as an important area for future work.
  • Problem datasets: The authors report that OneSmallStep had limited diversity, making further gains likely with more annotation.
  • Problem datasets: Workout contains 921 internet-derived examples in <work> format across mathematics, chemistry, biology, astronomy, physics, geology, and history.The paper describes Workout as its highest-quality dataset with reasonable diversity and a candidate for future scaling.
  • Scientific tasks: For three targets, Galactica infers properties from sequences alone, but it does not solve ESR2 and PGR, suggesting text representation limitations.The passage notes that geometrical information may be needed for reasonable data-efficiency on harder targets.
Loading 2211.09085v1…