Source-linked AI summary
Construction of the Literature Graph in Semantic Scholar
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, Oren Etzioni
TL;DR
Researchers need structured representations to support algorithmic discovery across scientific literature. The paper presents a deployed scalable system that builds a heterogeneous literature graph by applying NLP methods to scientific documents, reporting results across its extraction tasks. The resulting approach underlies semantic features in Semantic Scholar while exposing challenges from limited knowledge-base coverage and task assumptions.
Problem
Researchers remain unable to answer simple and complex literature questions, motivating structured data extraction for natural-language interfaces and academic-search ranking.
Method
The system constructs a directed literature graph and reduces graph construction to sequence labeling, entity linking, and relation extraction, with deployed models for authors, references, and entities.
Results
Entity-linking outputs are pooled because complementary approaches yield significantly higher yield than any individual approach, while the full entity-extraction model improves F1 by 4.2 points and a 15-model ensemble adds 1.1 points.
Takeaways & Limitations
The methods provide a deployed overview of Semantic Scholar’s approach for representing scientific literature and supporting algorithmic discovery.
Takeaways & Limitations
The draft does not discuss its relation-extraction models because of space constraints, and the approach faces limited knowledge-base coverage as scientific concepts evolve faster than curation.
Abstract
from arXiv · showhide
We describe a deployed scalable system for organizing published scientific literature into a heterogeneous graph to facilitate algorithmic manipulation and discovery. The resulting literature graph consists of more than 280M nodes, representing papers, authors, entities and various interactions between them (e.g., authorships, citations, entity mentions). We reduce literature graph construction into familiar NLP tasks (e.g., entity extraction and linking), point out research challenges due to differences from standard formulations of these tasks, and report empirical results for each task. The methods described in this paper are used to enable semantic features in www.semanticscholar.org
1 Introduction
The paper targets algorithmic discovery in scientific literature by extracting structured information into a scalable literature graph. It reframes graph construction as familiar NLP tasks while addressing assumptions that are unrealistic for many domains and applications.
- Scientific search systems still leave researchers unable to answer questions about clinical-trial demographics, coauthor expertise, and paper-specific effects.
- The system extracts structured information from scientific documents into a directed property graph that supports simple and complex literature queries.The graph can, for example, compute an author’s Erdős number through shortest-path queries over authorship edges.
- Literature graph construction is reduced to sequence labeling, entity linking, and relation extraction rather than treated as a single bespoke task.
- Standard NLP formulations often assume that test-set entity types match training-set labels, but the paper identifies this assumption as unrealistic for many domains and applications.
- The paper presents the approach deployed at www.semanticscholar.org as a step toward more intelligent academic search engines.
2 Structure of The Literature Graph
The literature graph is a directed property graph whose structured nodes and edges represent papers, authors, entities, mentions, citations, authorship, linking, and extracted relations. Its internal node and edge structure is designed to represent complex scientific data types.
- The graph uses directed property-graph edges, whose nodes and edges have internal structure for representing complex data types such as papers and entities.
- Nodes: 37M unique paper nodes store metadata such as titles, abstracts, full text, venues, and publication years after deterministic duplicate removal.
- Nodes: The graph contains 12M unique author nodes and 0.4M unique scientific-concept entity nodes with names, aliases, and descriptions.
- Nodes: It contains 237M entity-mention nodes representing textual references with mention text, context, and confidence attributes.
- Edges: Directed citation edges connect citing papers to referenced papers, while directed authorship edges connect authors to papers.
- Edges: Entity-linking edges connect extracted mentions to referenced entities, and mention–mention relations encode predicted relation types within a sentence.
- Edges: Entity–entity relations may be imported from a knowledge base or inferred from other edges in the graph.
3 Extracting Metadata
The paper describes ScienceParse, an RNN-based system that predicts paper metadata from raw PDFs when source metadata is incomplete or absent. It uses token, layout, and page-position features with a bidirectional LSTM and evaluates exact matches on held-out PubMed Central documents.
- The draft omits discussion of the relation-extraction models because of space constraints.
- ScienceParse predicts paper titles, authors, and references from raw PDFs using recurrent neural networks when metadata is incomplete or unavailable.Each reference includes a title, authors, venue, and year.
- Input and model: PDF pages are converted into token sequences with text, font-size, spacing, position, case, and page-number features before modeling.
- Input and model: Token representations pass through a fully connected layer and a two-layer bidirectional LSTM, followed by dense label prediction and label-bigram features.
- Training: The model is trained on 1.4M PubMed Central PDFs, with heuristic labeling succeeding for 76% of documents.
- Decoding: Viterbi decoding selects the most likely global label sequence, with title and author extraction determined from continuous labeled token spans.
- Evaluation: Evaluation uses about 54K held-out PubMed Central documents and counts an instance as correct only when it exactly matches the gold annotation.
4 Entity Extraction and Linking
The system builds literature-graph entities by combining statistical, rule-based, and off-the-shelf extraction and linking methods, while adapting standard NLP formulations to scientific documents. Experiments show trade-offs among precision, yield, and extraction accuracy, with pooled approaches and language-model enhancements improving deployed performance.
- 4 Entity Extraction and Linking: The deployed pipeline focuses entity extraction and linking on paper titles and abstracts, feeding extracted mentions into knowledge-base grounding.It uses multiple model instances and a rule-based module before entity linking.
- 4.1 Approaches: Statistical, hybrid, and off-the-shelf approaches provide alternative strategies for extracting mention spans and linking them to candidate knowledge-base entities.The hybrid approach combines deterministic string matching with statistical disambiguation, while off-the-shelf methods use TagMe or MetaMap Lite.
- 4.1 Approaches: The hybrid approach consistently gives the highest yield, while the statistical approach gives the highest precision and lowest yield.The pure rules-based approach was also tested but performed worse than the hybrid approach.
- 4.2 Entity Extraction Models: Entity extraction uses BILOU token labels without entity types, leaving further information about each entity to the linking module.The model combines character-based CNN representations, word embeddings, language-model embeddings, and a two-layer bidirectional LSTM.
- 4.2 Entity Extraction Models: 4.2 F1 points separate the full entity-extraction model from the variant without language-model embeddings, and an ensemble adds 1.1 F1 points.The reported comparisons use the development set of SemEval-2017 Task 10.
- 4.4 Entity Linking Models: Entity linking retrieves candidates through a token-to-entity index and ranks them with a neural model encoding mention context and candidate names or definitions.The candidate-selection index also serves as the baseline in the Bag of Concepts evaluation.
5 Other Research Problems
The paper identifies additional research problems around resolving author identities, matching concepts across knowledge bases, expanding coverage, extracting figures and tables, and understanding citations.
- Author disambiguation: Author disambiguation must handle shared names and inconsistent naming conventions across venues and sources.The authors train a supervised binary classifier for this problem.
- Ontology matching: Ontology matching identifies semantically equivalent entities represented differently across knowledge bases.For example, artificial neural networks has distinct identifiers in MESH and DBpedia.
- Limited KB coverage: Limited knowledge-base coverage leaves scientific concepts and relations behind the faster pace of new literature.The authors propose predicting textual relations and detailed concept descriptions to help close this gap.
- Figure and table extraction: Figure and table extraction remains constrained by the lack of large labeled datasets for scientific documents.One approach induced high-quality training labels without human intervention.
- Understanding and predicting citations: Citation edges support research on citation rates, citation importance, and predictions about papers or authors.The authors distinguish important from incidental citations and develop models for estimating potential.
6 Conclusion and Future Work
The paper presents a symbolic literature graph, deployed models for extracting key scholarly information, and experiments evaluating those models. It points toward improving graph quality, aggregating domain-specific extractions, and releasing resources to support future work.
- Conclusion: The paper constructs a graph that symbolically represents scientific literature and evaluates deployed models for identifying authors, references, and entities.These models are presented as part of a deployed system.
- Future work: Three future directions are improving graph quality, aggregating domain-specific extractions, and using the resulting corpus to understand literature-wide patterns.Examples include ontology matching, knowledge-base population, demographic-bias analysis, and empirical-result summarization.
- Resources: The authors publicly release metadata for over 20 million papers, a meaningful citations dataset, and models for figure, table, citation, and metadata extraction.The released resources are intended to help future research efforts.