PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction

Yi Luan, Luheng He, Mari Ostendorf, Hannaneh Hajishirzi

arXiv:1808.09602v1cs.CL

TL;DR

Scientific information extraction must identify entities and relations across documents despite costly annotation and cross-sentence coreference dependencies. The paper introduces SCIERC and a unified span-based multi-task SciIE framework, which outperforms prior scientific IE systems and supports knowledge-graph construction. The analysis highlights coreference links as important for producing a denser, more useful graph.

  • Problem

    Scientific information extraction requires costly expert annotation and must connect relations across sentences through coreferential mentions.

  • Method

    The paper creates SCIERC and develops a unified multi-task framework that shares span representations across entity recognition, relation extraction, and coreference resolution.

  • Results

    The unified model improves performance across all tasks, outperforms previous scientific IE systems on entity and relation extraction, and constructs a scientific knowledge graph.

  • Takeaways & Limitations

    Coreference links help organize extracted scientific information into a denser, more useful knowledge graph for analyzing scientific literature.

  • Takeaways & Limitations

    The model still has a large performance gap relative to human performance, confirming the challenges of scientific information extraction.

Abstract

from arXiv · show

We introduce a multi-task setup of identifying and classifying entities, relations, and coreference clusters in scientific articles. We create SciERC, a dataset that includes annotations for all three tasks and develop a unified framework called Scientific Information Extractor (SciIE) for with shared span representations. The multi-task setup reduces cascading errors between tasks and leverages cross-sentence relations through coreference links. Experiments show that our multi-task model outperforms previous models in scientific information extraction without using any domain-specific features. We further show that the framework supports construction of a scientific knowledge graph, which we use to analyze information in scientific literature.

1 Introduction

Scientific information extraction must handle costly expert annotation and relations that cross sentence boundaries through coreference. The paper addresses these challenges with a unified multi-task framework and a dataset for entities, relations, and coreference.

  • Motivation: Scientific information extraction supports organizing entities and relationships into structured knowledge bases, but annotation requires costly domain expertise.This limits available annotation resources.
  • Motivation: Cross-sentence relations can depend on coreferential mentions such as pronouns or generic phrases, which within-sentence systems cannot connect reliably.Without coreference, relation coverage can be much lower and the resulting knowledge base sparser.
  • Method: The paper develops a unified multi-task model for scientific entities, relations, and coreference instead of treating them as independent pipeline components.The model shares parameters across tasks and leverages document context through coreference links.
  • Method: The framework enumerates spans, detects overlapping spans, jointly models span and span-span relations, and avoids cascading errors between tasks.It uses shared span representations and document-level coreference links rather than a standard tagging-only design.
  • Contributions: The authors create SCIERC with annotations for scientific terms, relation categories, and coreference links, then use the framework to construct and analyze a scientific knowledge graph.Human evaluation indicates that propagating coreference significantly improves the automatically constructed graph.

2 Related Work

Prior scientific information extraction work includes citation, community, unsupervised, and supervised approaches, but recent neural systems often focus on single tasks or limited joint settings. This paper distinguishes its approach by propagating cross-task information through span representations.

  • Prior scientific IE: Earlier scientific information extraction research analyzed citations, research communities, and scientific information using unsupervised methods.Subsequent datasets enabled supervised and semi-supervised learning for scientific information extraction.
  • Prior scientific IE: Many recent neural models address single tasks or joint entity and relation extraction, and some assume that entity boundaries are given.This leaves broader joint treatment of entities, relations, and coreference less common.
  • Multi-task learning: Neural multi-task models commonly share word-level representations, while some use high-order cross-task factors.The paper instead propagates cross-task information through span representations.

3 Dataset

SCIERC is a 500-abstract dataset that jointly annotates scientific entities, relations, and coreference, extending prior datasets with broader relation coverage and cross-sentence links. Its annotation scheme defines six entity types and seven relation types, with expert agreement measured on a subset.

  • Dataset scope: SCIERC annotates scientific entities, their relations, and coreference clusters in 500 scientific abstracts from four AI communities.The abstracts come from 12 AI conference and workshop proceedings in the Semantic Scholar Corpus.
  • Dataset scope: SCIERC extends SemEval 17 and SemEval 18 with more entity and relation types, broader relation coverage, and cross-sentence relations linked through coreference.The cited comparison reports 9.4 versus 3.2 relations per abstract for SCIERC and SemEval 18, respectively.
  • Annotation scheme: The annotation scheme defines six entity types: Task, Method, Metric, Material, Other-ScientificTerm, and Generic.Generic entities are annotated only when involved in a relation or coreference link.
  • Annotation scheme: It defines seven relation types and annotates coreference links between identical scientific entities, accounting for directionality except for Conjunction and Compare.The seven relation types are Compare, Part-of, Conjunction, Evaluate-for, Feature-of, Used-for, and Hyponym-Of.
  • Annotation process: One domain expert annotated all documents, while 12% received duplicate annotation from four additional experts to assess agreement.Reported kappa scores are 76.9% for entities, 67.8% for relations, and 63.8% for coreference.

4 Model

SCIIE formulates entity recognition, relation extraction, and coreference resolution as jointly trained classification tasks over shared span representations. Its factorized scoring and pruning strategy supports document-level inference while controlling computational cost.

  • Model architecture: SCIIE treats entity recognition, relation extraction, and coreference resolution as multinomial classification tasks with shared span representations.Sharing representations allows information from coreference resolution to propagate across sentences without increasing inference complexity.
  • Problem definition: The model derives all possible within-sentence spans and predicts entity types for spans, relations for span pairs, and coreference links between spans.These outputs are represented as discrete random variables indexed by spans or span pairs.
  • Model definition: The conditional distribution P(E, R, C|D) is decomposed under a conditional-independence assumption over the observed document and candidate spans.The model uses independently normalized conditional probabilities for the task variables.
  • Training objective: The three-task objective is a weighted sum of negative log-likelihood losses, with λE, λR, and λC controlling task importance.Coreference training uses a marginalized loss because each mention may have multiple correct antecedents.
  • Scoring architecture: Feed-forward networks compute span and pairwise scores from shared fixed-length span vectors, including entity, relation-mention, coreference-mention, relation, and coreference scores.The model combines these scores into task-specific Φ functions; null-label scores are fixed at 0.
  • Inference and pruning: Beam pruning reduces pairwise span factors from O(n^4) to O(n^2), while a fixed maximum span width further reduces span factors to O(n).Separate beams retain spans ranked by coreference-mention and relation-mention scores.

5 Knowledge Graph Construction

SCIIE constructs a scientific knowledge graph from extracted entities and relations across 110k abstracts, using coreference links to normalize entities and resolve ambiguity. Nodes represent scientific entities, while typed edges represent relations selected by corpus-wide frequency.

  • Graph construction: The graph contains scientific-entity nodes and typed edges representing relation types between entity pairs.The corpus comprises all 110k abstracts from 12 AI conference proceedings in the Semantic Scholar Corpus.
  • Graph construction: SCIIE extracts entities, relations, and coreference clusters within each document before graph construction.The extracted information is used for heuristic phrase normalization through entity and coreference links.
  • Graph example: Figure 4 illustrates graph neighborhoods around statistical machine translation, including a Used-for edge to grammatical error correction.The figure abbreviates common terms and denotes reverse relations as Uses or Evaluated-by.
  • Entity normalization: Coreference-based normalization links cluster members and replaces them with the longest entity string, reducing ambiguous phrases.Generic terms can be replaced with nongeneric cluster members during normalization.
  • Relation assignment: For each entity pair, the graph assigns the most frequent relation type observed across the corpus.Entity pairs may occur in different contexts with different relation types, so relation frequencies determine the selected edge.

6 Experimental Setup

The experiments evaluate SCIIE on SCIERC and SemEval 2017, alongside neural extraction baselines and systems using additional features. The setup also examines knowledge-graph construction with and without coreference resolution.

  • Evaluation benchmarks: SCIIE is evaluated on SCIERC and SemEval 2017.The supplied setup identifies both datasets as evaluation benchmarks.
  • Knowledge-graph analysis: The knowledge-graph analysis uses 110k abstracts from the Semantic Scholar Corpus.The corpus covers 12 AI conference proceedings.
  • Baselines: The comparison includes LSTM+CRF, LSTM+CRF+ELMo, joint entity-relation systems, pipeline extraction, and end-to-end coreference baselines.These baselines cover tagging, relation extraction, pipeline variants, contextual embeddings, and coreference resolution.
  • Baselines: On SemEval, SCIIE is compared with a leaderboard system using gazetteers and model ensembles, and with a semi-supervised keyphrase extractor.The comparison includes systems with in-domain features and a neural tagging model.
  • Implementation: The implementation uses a one-layer BiLSTM, span enumeration up to 8 words, dropout, and beam-pruning weights for coreference and relation extraction.The model uses 200-dimensional hidden layers and separate pruning weights λC = 0.3 and λR = 0.4.

7 Experimental Results

SCIIE outperforms the baselines across entity recognition, relation extraction, and coreference resolution, while multi-task learning improves performance across tasks. The resulting knowledge graph supports scientific trend analysis and benefits from coreference links.

  • 7.1 IE Results: SCIIE outperforms all baselines across entity recognition, relation extraction, and coreference resolution on SCIERC.Relative improvements range from 1.3% to 13.1% over the reported comparison systems.
  • 7.1 IE Results: 13.1% relative improvement is observed for relation extraction over E2E Rel, while coreference resolution improves 4.5% over E2E Coref.Entity recognition improves by 1.3% and 2.4% over LSTM+CRF variants, and by 1.8% and 2.7% over E2E Rel variants.
  • 7.1 IE Results: 7.1% relative improvement occurs when relation extraction is multi-tasked with coreference resolution, while coreference resolution improves 4.9% with relation extraction.Entity recognition also benefits from adding coreference resolution and relation extraction to the objective.
  • 7.1 IE Results: SCIIE outperforms previous hand-featured systems on SemEval 17, with larger gains in span identification than keyphrase classification.The authors attribute the span-identification improvement to enumerating spans rather than using BIO tagging; relation extraction remains competitive.
  • 7.2 Knowledge Graph Analysis: The knowledge graph captures historical application trends for neural networks across speech, computer vision, and NLP from 1996 to 2016.Neural networks first gained popularity in NLP through language modeling, later extending to POS tagging and machine translation; computer vision applications appeared earlier for object recognition.
  • 7.2 Knowledge Graph Analysis: Coreference linking yields higher graph recall while maintaining high precision above 84% for both evaluated systems.Human evaluation used four domain experts and approximately 1.5k relation triples from each system.

8 Conclusion

The paper combines a new scientific information-extraction dataset with a multi-task model that builds knowledge graphs from scientific articles. It reports broad extraction improvements and identifies a remaining gap to human performance.

  • 8 Conclusion: The framework jointly identifies entities, relations, and coreference clusters while sharing span representations and leveraging cross-sentence information.It avoids hand-engineered features and pipeline processing.
  • 8 Conclusion: The model improves performance across all tasks and outperforms previous scientific information-extraction systems on entity and relation extraction.The paper also reports better span-boundary prediction.
  • 8 Conclusion: The extracted information is automatically organized into a knowledge graph from a large collection of scientific articles.The analysis shows the importance of coreference links for making the graph dense and useful.
  • 8 Conclusion: A large gap remains between the model's performance and human performance, motivating future work on semi-supervised learning and in-domain features.The authors also plan to extend the framework to information extraction in other domains.

A.1 Entity Category

The annotation scheme categorizes scientific entities and relations while marking coreference links under explicit span, relation, and coreference rules. It distinguishes informative scientific mentions from generic or excluded expressions.

  • A.1 Entity Category: The scheme defines six entity types: Task, Method, Metric, Material, Other-ScientificTerm, and Generic.Generic entities are annotated only when involved in a relation or coreference link.
  • A.1 Entity Category: Relations include five directed types—Part-of, Evaluate-for, Feature-of, Used-for, and Hyponym-Of—and two symmetric types, Compare and Conjunction.Asymmetric relations point from B to A, while Compare and Conjunction are symmetric.
  • A.1 Entity Category: Coreference links connect two entities that refer to the same scientific concept, including anaphoric and coreferring noun phrases.Examples include “The system” referring back to “a machine reading system” and “The POS tagger” referring to a part-of-speech tagging system.
  • A.1 Entity Category: Entity spans may be embedded in longer spans only when the shorter span participates in a relation.Determiners and adjective pronouns are excluded from spans, and generic phrases are omitted when unrelated.
  • A.1 Entity Category: The guidelines exclude relations involving variable-bound entities or the word “which,” and exclude coreference for generic uses or distinct examples of the same term.Negative relations are not labeled.

B Annotation and Knowledge Graph Examples

The paper illustrates its annotation interface and the automatically generated knowledge graph using examples from ACL papers. These examples connect the annotation process to graph construction.

  • B Annotation and Knowledge Graph Examples: Figure 9 shows a screenshot of the BRAT annotation interface for an ACL paper.The paper notes that additional examples are available on the project website.
  • B Annotation and Knowledge Graph Examples: Figure 10 reproduces the original graph figure and presents an automatically generated knowledge graph centered on statistical machine translation.The figure is described as an example of the paper's knowledge-graph output.
Loading 1808.09602v1…