S2ORC: The Semantic Scholar Open Research Corpus

1 Introduction

• S2ORC contains three times more full-text papers than PubMed Central (OA), the next-largest bibliometrically enhanced corpus, while covering more academic disciplines.Its citations link to the full set of 81.1M S2ORC paper nodes.Source: Table 1 · p. 2
• S2ORC contains 81.1M academic papers across dozens of disciplines, addressing the small scale and narrow coverage of some existing corpora.The paper also notes that some existing resources do not provide usable full text.Source: §1 · p. 2
• The corpus provides structured, machine-readable full text for 8.1M open-access papers and LATEX parses for 1.5M papers, including linked citations and references to figures and tables.The LATEX parses also provide table content and mathematical formulas.Source: §1 · p. 2
• S2ORC is curated from Semantic Scholar papers and includes English-language papers with abstracts or full text available.The paper points to its corpus-construction section for the filtering details.Source: §1 · p. 2

2 Constructing the corpus

• The construction pipeline processes PDF and LATEX sources, selects the best metadata and full-text parses per cluster, filters clusters, and resolves bibliography links.The pipeline also addresses metadata aggregation, open-access identification, clustering, and text extraction and cleaning.Source: §2 · pp. 2–3

2.1 Processing PDFs

• PDF processing uses SCIENCEPARSE and GROBID to extract metadata, structured body text, captions, equations, table content, inline citations, bibliography entries, and citation-to-entry links.Regular-expression postprocessing removes certain false positives and expands bracket citation ranges before linking them to bibliography entries.Source: §2.1 · p. 3

2.2 Processing LATEX source

• LATEX source provides structured text, section headers, captions, tables, equations, citations, and references, with near-perfect detection of several structures.LATEX-derived metadata is lower quality than PDF-derived metadata and is excluded from metadata selection and clustering.Source: §2.2 · p. 3

2.3 Selecting canonical metadata

• Canonical metadata favors a selected PDF and publisher-provided fields, using majority voting when publisher metadata is incomplete and minimizing the number of sources when breaking ties.When publisher data lacks titles or authors, parsed PDF values are used, prioritizing SCIENCEPARSE over GROBID.Source: §2.3 · pp. 3–4

2.4 Assembling the corpus

• A valid GROBID parse is associated with a paper object when its PDF is open access; non-open-access PDFs contribute bibliography data, and available arXiv LATEX parses are also attached.Open access is assigned from specified sources or an open-access DOI in Unpaywall.Source: §2.4 · p. 4

2.5 Filtering paper clusters

• Clusters are filtered for missing titles or authors, fewer than 100 characters of abstract and body text, or non-English primary language; 95.5M clusters are removed.The authors report that 80.0M filtered clusters lack a publisher-provided abstract and an associated PDF.Source: §2.5 · p. 4

2.6 Linking bibliographies to papers

• Bibliography entries are linked to paper clusters using title similarity: normalized title character 3-grams feed a harmonic-mean score of Jaccard and containment, with links emitted above 0.8.Entries without a qualifying pair remain unlinked.Source: §2.6 · p. 4

• 81.1M papers; publisher-provided abstracts cover 90.4% (73.4M), PDFs cover 35.6% (28.9M), and every paper has an abstract, PDF, or both.Source: §3 · p. 4, Table 3 · p. 5
• 8.1M open-access PDFs receive full text, alongside 1.5M LaTeX parses; extracted bibliographies yield 380.5M resolved links, 156.5M tied to inline mentions.Source: §3 · p. 5
• LaTeX parses preserve line breaks, affecting paragraph counts; authors speculate linking differences reflect parsing and expansion challenges and title-less math and physics bibliographies.Source: §3 · p. 5, Table 4 · p. 5
• Figure 2 uses Microsoft Academic fields; about 677k papers with multiple primary fields appear only under their top field, while unmatched papers are Unclassified.Source: §3 · p. 5, Figure 2 · p. 5

• For metadata selection, 500 PDF-containing paper clusters were sampled to check whether selected canonical titles and authors match the canonical PDF.Source: §4 · p. 5
• GROBID achieved F1 scores comparable to prior reports for inline citation detection and bibliography parsing in an evaluation of 200 sampled S2ORC papers.Source: §4 · p. 6
• Bibliography-link evaluation sampled 500 GROBID PDF parses and 100 LaTeX parses, checking linked-entry titles and authors against the linked papers.Table 6 reports accuracy for paper clustering and bibliography linking on sampled sets.Source: §4 · p. 6, Table 6 · p. 6

5 Pretraining BERT on S2ORC

• S2ORC-SCIBERT uses a 16.4B-token pretraining corpus, nearly five times larger than SCIBERT’s, with a more balanced distribution across academic disciplines.Biomedical and computer science papers account for 42.7% and 7.2% of the S2ORC corpus, respectively; its cased WordPiece vocabulary has 31k entries.Source: §5 · pp. 6–7
• The model is pretrained from scratch with BERT-Base defaults and fine-tuned using Adam, cross-entropy loss, linear learning-rate decay, and 10% warm-up.Fine-tuning uses batch size 32 and dropout 0.1; hyperparameters are selected on each task’s development set from a grid of 1–4 epochs and four learning rates.Source: §5 · p. 7
• S2ORC-SCIBERT outperforms SCIBERT on many tasks despite including a large percentage of data outside biomedical and computer science.The paper also reports similar performance to SCIBERT across a diverse suite of scientific NLP tasks.Source: §5 · pp. 6–7

Citation-related tasks

• Linked inline citations make S2ORC usable without additional supervision for inline and document-level citation recommendation.The document-level setting described by Liu et al. (2015) uses inline citation contexts to filter recommendations.Source: §6 · p. 7
• S2ORC citation contexts support studying citation intent, sentiment, meaningful citations, key phrases, and citation-context summarization.The paper suggests pretraining on these contexts before fine-tuning and using wider contexts to augment document-level datasets.Source: §6 · p. 8

Citation contexts and paper similarity

• Citation-context word2vec embeddings cluster papers by arXiv machine-learning categories, and papers from different categories within a cluster share research themes.The paper proposes using similarity between papers’ citing contexts as a proxy for paper similarity; Figure 3 visualizes embeddings for 20k papers, while Table 7 samples regions A and B.Source: Figure 3 · p. 7, Table 7 · p. 7, §6 · p. 8

LaTeX subset

• Structured tables parsed from S2ORC’s LaTeX subset could support corpus-level results extraction and aggregation.Its fine-grained formula extraction and labeling could also support research on proof construction or symbol coreference resolution.Source: §6 · p. 8

7 Related work

• Existing resources include a 24.6k-paper computational-linguistics corpus, a 2.6M-paper biomedical corpus, and arXiv- or CiteSeerX-derived collections with differing text and citation data.The paper notes that RefSeer provides short citation snippets and that Saier and Färber (2019) provide extracted snippets without bibliography parses.Source: §7 · p. 8
• S2ORC is a significantly larger linked-paper dataset spanning broad scientific domains, built using PDF parsing in addition to LaTeX source.The paper contrasts it with resources that are smaller, domain-specific, or lack usable full text.Source: §7 · p. 8
• S2ORC provides clean full text with annotations for table and figure references and captions, alongside metadata and structured text adaptable to downstream tasks.Source: §7 · p. 8

8 Conclusion

• S2ORC contains 81.1M papers, 380.5M resolved citation links, and structured full text from 8.1M open-access PDFs and 1.5M LATEX source files.It aggregates metadata and abstracts from hundreds of trusted sources and augments full text with sections, citation mentions, and references to tables and figures.Source: §8 · p. 9
• The corpus pipeline was used to construct CORD-19, which saw fervent adoption for COVID-19 text mining, including more than 75K dataset downloads in its first month.The authors hope S2ORC will make similar text-mining resources available beyond periods of global crisis.Source: §8 · p. 9

A Background & Terminology

• A bibliography entry is a structured item identifying another paper, while an inline citation is a mention in the abstract or body that refers to a bibliography entry.Bibliography entries can include title, authors, year, venue, and document identifiers; citation formats vary.Source: A Background & Terminology · p. 13
• Inline citations may be narrative, parenthetical, bracket-numbered, or superscript styles; entries without numbered markers are typically cited in NAME-YEAR format.Narrative citations appear as noun phrases, whereas parenthetical citations are inserted as asides.Source: A Background & Terminology · p. 13
• An inline reference points to another part of the same paper, such as a figure, table, equation, proof, section, or appendix.These spans can help readers understand relationships between text and paper objects.Source: A Background & Terminology · p. 13

B PDF filters

• PDFs are filtered before GROBID if PyPDF2 processing errors, page counts exceed 50, page widths exceed heights, or pdfalto extraction fails.The removed-PDF counts are reported in Table 8.Source: B PDF filters · p. 13, Table 8 · p. 13

C The paper clustering problem

• A paper cluster groups highly similar documents that authors would generally regard as the same paper, such as an arXiv preprint and its published version.For construction, each cluster must have one canonical document representing it.Source: C The paper clustering problem · pp. 13–14
• Choosing a canonical draft can make a citation context misrepresent its rationale when the cited preprint differs from the selected peer-reviewed version; the authors believe such cases are rare.They state these cases do not affect the vast majority of citation contexts.Source: C The paper clustering problem · p. 14

D S2ORC evaluation criteria

• Paper-cluster titles are judged correct when they match the canonical PDF title, allowing capitalization, minor special-character, and whitespace differences.The evaluation compares the selected canonical Title field with the title on the PDF.Source: D S2ORC evaluation criteria · p. 14
• Paper-cluster authors are judged correct when every PDF author appears in the right order, allowing first-initial substitutions and omitted middle names or titles.The selected canonical Authors field is compared with the authors shown on the PDF.Source: D S2ORC evaluation criteria · p. 14
• For paper-bibliography links, titles use the same matching criteria, while authors match when their names substantially overlap, even if the linked paper lists fewer authors.The passage attributes most author-set differences to incorrectly written bibliography entries or publisher metadata mistakes.Source: D S2ORC evaluation criteria · p. 14

E Training corpus sizes for other language models

• 25B tokens of body text and 15B tokens of abstracts are estimated for S2ORC, while aggressively cleaned PDF-parsed body text still yields approximately 16.5B tokens.The authors say this volume is more than sufficient for pretraining models such as ELMO, BERT, ROBERTA, and GPT2.Source: E Training corpus sizes for other… · p. 14, Table 9 · p. 14

F Numeric representations in S2ORC-SCIBERT

• Numeric representations in S2ORC-SCIBERT shift from sentence position and magnitude in early layers toward contextual cues, including inline LATEX identifiers, citations, and PDF references, in later layers.Figure 4 visualizes layer 9 representations, with labels based on token-level heuristics.Source: Figure 4 · p. 14, F Numeric representations in… · p. 15

G MAG topic distribution

• The MAG topic distribution lists Medicine at 12.8M and 1.8M, Biology at 9.6M and 1.6M, and Chemistry at 8.7M and 484k in its two reported columns.The supplied distribution also lists discipline entries from Computer Science and Physics to Philosophy.Source: G MAG topic distribution · p. 15