Source-linked AI summary
The Semantic Scholar Open Data Platform
Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David Graham, Fangzhou Hu, Regan Huff, Daniel King, Sebastian Kohlmeier, Bailey Kuehl, Michael Langan, Daniel Lin, Haokun Liu, Kyle Lo, Jaron Lochner, Kelsey MacMillan, Tyler Murray, Chris Newell, Smita Rao, Shaurya Rohatgi, Paul Sayre, Zejiang Shen, Amanpreet Singh, Luca Soldaini, Shivashankar Subramanian, Amber Tanaka, Alex D. Wade, Linda Wagner, Lucy Lu Wang, Chris Wilhelm, Caroline Wu, Jiangjiang Yang, Angele Zamarron, Madeleine Van Zuylen, Daniel S. Weld
TL;DR
Scientific output creates a need for tools that help researchers discover and understand relevant literature. Semantic Scholar addresses this by combining diverse scholarly data sources with PDF extraction, knowledge-graph construction, and semantic-feature generation, and provides access through APIs and datasets. As of January 2023, S2AG covered many natural, physical, and social sciences, while the API served over 1.8 billion requests in 2024.
Problem
The volume of scientific output creates an urgent need for automated tools that help scientists keep up with developments and understand relevant literature.
Method
Semantic Scholar combines scholarly metadata and PDF content with extraction, knowledge-graph construction, entity disambiguation, and semantic features, distributing the resulting graph through APIs and datasets.
Results
As of January 2023, S2AG covered many natural, physical, and social sciences, and the API served over 1.8 billion requests in 2024.
Takeaways & Limitations
The platform provides open scholarly data and services for scientific NLP, science-of-science research, and applications supporting literature discovery and understanding.
Takeaways & Limitations
Author-affiliation links were still being populated, so the reported number represented the expected eventual count.
Abstract
from arXiv · showhide
The volume of scientific output is creating an urgent need for automated tools to help scientists keep up with developments in their field. Semantic Scholar (S2) is an open data platform and website aimed at accelerating science by helping scholars discover and understand scientific literature. We combine public and proprietary data sources using state-of-the-art techniques for scholarly PDF content extraction and automatic knowledge graph construction to build the Semantic Scholar Academic Graph, the largest open scientific literature graph to-date, with 200M+ papers, 80M+ authors, 550M+ paper-authorship edges, and 2.4B+ citation edges. The graph includes advanced semantic features such as structurally parsed text, natural language summaries, and vector embeddings. In this paper, we describe the components of the S2 data processing pipeline and the associated APIs offered by the platform. We will update this living document to reflect changes as we add new data offerings and improve existing services.
1 Introduction
Semantic Scholar was created to help scholars address information overload by discovering and understanding relevant literature. The platform combines an open scientific corpus with data downloads, open-source libraries, and APIs.
- Semantic Scholar was launched in 2015 by AI2 to help scholars combat information overload and discover and understand relevant research literature.
- The platform built a comprehensive open corpus through partnerships with scientific publishers and preprint services.
- Most website data and functionality are available through data downloads, open-source libraries, and API services.
- The paper overviews the technology used to build the corpus and the APIs and downloads provided to access it.
- The platform is intended to support scientific NLP, science-of-science research, and applications that help scientists discover and understand literature.
2 Platform Overview
The Semantic Scholar Open Data Platform builds and distributes S2AG, a disambiguated bibliographic knowledge graph covering scholarly entities and their relationships. The graph is constructed from varied sources and made available through public APIs and datasets.
- S2AG represents papers, authors, venues, and academic institutions as nodes in a disambiguated bibliographic knowledge graph.
- Its edges encode authorship, citation, publication venue, and institutional affiliation relationships.
- S2AG ingests varied data sources through a processing pipeline and is distributed through publicly available APIs and datasets.
- The current work updates an earlier Semantic Scholar pipeline while retaining its core graph structure and refining knowledge-graph construction components.
- As of January 2023, S2AG covered many natural, physical, and social sciences, with some papers assigned to multiple fields and many unclassified.
3 Data Processing Pipeline
Semantic Scholar’s processing pipeline continually ingests scholarly metadata and PDFs, extracts structured content, constructs and normalizes the academic graph, and adds semantic features. The pipeline combines multiple systems and makes many components openly available.
- The pipeline ingests academic metadata and PDF content, extracts structured information, constructs S2AG, and adds semantic features such as summaries and vector embeddings.
- Its processing stages include PDF extraction, normalization and disambiguation of scholarly entities, field-of-study classification, and textual summarization.
- Structured metadata and extracted PDF content are processed by knowledge-graph construction systems that build S2AG.
- The pipeline includes a caveat that the author-affiliation links were still being populated, so the reported number was an expected eventual count.
- Many pipeline components are available as open software or models, with associated code, models, datasets, or research articles where applicable.
3.1 Data Sources
The pipeline draws on more than 50 sources spanning nonprofit organizations, preprint servers, academic publishers, a web crawler, and human-created data. These sources provide metadata through varied formats and delivery mechanisms, with most updated daily.
- More than 50 input sources feed the pipeline, including Crossref, arXiv, academic publishers, an internet web crawler, and human-created data.
- Sources provide metadata in JATS or proprietary formats and deliver data through FTP, APIs, or bulk HTTP downloads.
- Most sources are updated daily.
3.2 PDF Content Extraction
The PDF content extraction pipeline converts difficult, unstructured PDF files into structured data through three stages, enabling fine-grained text and document-element information.
- Extraction outputs: PDF parsing produces structured bibliographies, section headers, paragraphs, figures, tables, and inline references from unstructured paper content.The resulting structure supports construction of the citation graph and detailed representation of complete paper text.
- Pipeline stages: The pipeline has three stages: Text Extraction, Visual Region Annotation, and Text Span Annotation.These stages respectively recover plain text, identify page-layout regions, and assign semantic labels to extracted tokens.
- Text Extraction: Text Extraction infers word boundaries and word order by converting page-position commands into plain text.The implementation uses open-source toolkits including pdfalto, PDFPlumber, and PDFMiner.
- Visual Region Annotation: Visual Region Annotation generates page images and labels bounding boxes as categories such as figures, tables, and paragraphs.It uses poppler for page rendering and an object detector from LayoutParser.
- Text Span Annotation: Text Span Annotation assigns semantic labels such as title, author name, section header, body text, figure caption, and bibliography to tokens.The I-VILA model, trained on S2-VL, performs the tagging.
- Structured output: PaperMage converts visual and token annotations into linked structured data and emits a JSON-suitable object containing metadata and detailed body text.It associates section headers with content, figures with captions, and bibliographies with constituent elements.
3.3 Knowledge Graph Construction
Knowledge Graph Construction assigns persistent identities and relationships to entities extracted from heterogeneous scholarly sources, using specialized normalization, deduplication, and linking systems.
- Entity resolution: Knowledge Graph Construction assigns IDs to authors, institutions, venues, and papers and links each plain-string mention to the appropriate real-world entity.The input combines structured source metadata with outputs from PDF processing.
- Paper Deduplication: S2APLER deduplicates papers by blocking similar titles and scoring pairwise similarity with title, abstract, author, and venue features.Its synthetic training data uses authoritative paper records associated with PDFs or DOIs.
- Citation Linking: Citation Linking finds references to papers in bibliographies and associates each citation link with the sentence containing it.The current system uses fuzzy title-and-author matching heuristics and may later adapt S2APLER.
- Venue normalization: Publication Venue Normalization matches unnormalized venue strings against a lookup table containing normalized and variant venue titles.The system applies regular-expression rules and exact matching, using Fatcat and MAG data.
- Author Disambiguation: S2AND resolves author mentions through candidate blocking, LightGBM similarity scoring, and clustering within each block.The similarity model is trained on a large author-disambiguation dataset.
- Affiliation normalization: S2AFF parses affiliations into components, retrieves 100 Jaccard-overlap candidates, and ranks them with a pairwise LightGBM model.The model is trained using internally collected human annotations and links organizations to ROR identifiers.
3.4 Semantic Features
Semantic Scholar adds semantic features to its knowledge graph, including paper summaries, field classifications, citation interpretations, influential-citation labels, embeddings, and recommendations.
- TLDRs: TLDRs provide short scientific-paper summaries intended to support faster understanding and decision making when scanning paper lists.The generation task requires high source compression and domain-specific expertise.
- TLDRs: A BART model trained with CATTS generates TLDRs using paper titles as a scaffolding task to address limited annotated training data.Training combines SciTLDR examples with separately collected biomedical-paper summaries.
- Citation semantics: SciCite classifies citations as background information, use of methods, or comparing results.The model and dataset are adopted from prior work and are distributed with their code and data.
- Citation semantics: Highly Influential citations are identified with eligibility rules and textual or structural heuristics, including repeated citation, cue terms, and table or figure references.Only citations between papers without overlapping authors are eligible.
- Fields of study: S2FOS classifies papers into fields of study with a multilabel linear SVM using character n-gram TF-IDF representations.Its training labels come from manually labeled publication venues propagated to papers, and the taxonomy adds Education, Law, and Linguistics after MAG deprecation.
- Embeddings: SPECTER produces document-level paper embeddings from titles and abstracts for author disambiguation, recommendations, and public reuse.The model is built from SciBERT and trained with a triplet margin loss.
- Recommendations: The recommender trains rankers, selects candidates, and ranks candidates from positive and negative paper annotations.It uses TF-IDF and SPECTER-based SVM models, FAISS search over ∼1M recent papers, and final ranking of ∼500 papers near the positive-paper centroid.
4 APIs and Datasets
Semantic Scholar distributes its processed graph and semantic features through APIs and downloadable datasets, supporting lookup, search, bulk retrieval, passage search, and recommendations.
- Platform access: API-served data may shift over time or combine models during system migrations, with live documentation updated when appropriate.This reflects ongoing development and refinement of the platform’s models.
- Platform access: The API served over 1.8 billion requests in 2024, including 60% paper-metadata lookups and 16% keyword searches.More than 3,000 authentication keys have been issued, while high-volume access requires a free key subject to terms of use.
- Graph API: The Graph API provides current S2AG data through identifier lookup, search endpoints, citation relationships, and author-based retrieval.Bulk dataset downloads are recommended for users needing large data volumes.
- Search APIs: Relevance search matches titles, abstracts, and author names, filters metadata, and reranks up to 1000 results with LightGBM.The reranker emphasizes direct title matches and highly cited papers with recent publication dates.
- Search APIs: Bulk search supports Boolean and metadata filters, citation-count or publication-date sorting, pagination, and up to 10M matching papers.It is designed for broad queries where ordering is not critical.
- Search APIs: Snippet search indexes over 275M passages from more than 12M papers, with titles, abstracts, body text, and metadata filters.Passages are limited to 480 tokens and overlap neighboring passages by up to 64 tokens where possible.
- Datasets: Datasets API releases monthly knowledge-graph snapshots and incremental diffs as collections of gzipped JSON files linked by record IDs.Available datasets include papers, abstracts, authors, citations, embeddings, paper IDs, TLDRs, and publication venues.
- Recommendations API: The Recommendations API returns relevance-ordered metadata for papers published within the past 60 days using positive and negative paper annotations.At least one positive paper ID is required, while negative examples are optional.
5 Related Work
The paper compares scholarly data providers by comprehensiveness, access, and services, positioning Semantic Scholar as a comprehensive open knowledge base with broad services.
- Providers differ in comprehensiveness, access, and services offered.
- Google Scholar offers no programmatic services, while MAG has been discontinued.
- PubMed Central provides parsed content but is not cross-disciplinary, and some other providers require subscriptions.
- Semantic Scholar is described as a comprehensive, open knowledge base with the widest array of services.
6 Conclusion and Future Work
The platform combines open code, datasets, APIs, and a large academic graph built through PDF extraction and knowledge-graph normalization. Future work focuses on expanding semantic services, personalization, corrections, annotations, and model quality.
- The platform offers code bases, datasets, and APIs covering scientific literature.
- The Semantic Scholar Academic Graph contains hundreds of millions of papers and billions of citation links.
- The graph is built with a state-of-the-art PDF extraction and knowledge graph normalization pipeline.
- Semantic features include summarization, vector embeddings, and recommendations.
- Future plans include exposing selected semantic features for users' own data and adding richer full-text semantic labels.
- The platform plans more personalization, human data corrections, external annotations, and continued improvements to graph and semantic-feature models.