PaperlayerView exact revision on arXiv

Source-linked AI summary

Global-to-Local Neural Networks for Document-Level Relation Extraction

Difeng Wang, Wei Hu, Ermei Cao, Weijian Sun

arXiv:2009.10359v1cs.CLcs.AI

TL;DR

Document-level relation extraction must reason across entities and mentions throughout a document. GLRE combines global and local entity representations with context-relation representations, and the paper reports superiority on two public datasets, particularly for long-distance relations and entities with multiple mentions.

  • Problem

    Document-level relation extraction requires reasoning across sentences and synthesizing context from multiple entities and mentions.

    Source: §1 · p. 1

  • Method

    GLRE encodes entity-global information with R-GCN, aggregates local mention context with multi-head attention, and uses self-attention over other relations.

    Source: §1 · p. 2

  • Results

    GLRE outperformed many comparative models on two public datasets, with especially strong results for long-distance relations and entities with multiple mentions.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The reported advantage is particularly evident for relations between distant entities and entities with multiple mentions.

    Source: §5 · p. 9

    GLRE lacked knowledge needed to connect fatigue with adverse effects in an indirect reasoning case, which the authors leave for future work.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Relation extraction (RE) aims to identify the semantic relations between named entities in text. Recent years have witnessed it raised to the document level, which requires complex reasoning with entities and mentions throughout an entire document. In this paper, we propose a novel model to document-level RE, by encoding the document information in terms of entity global and local representations as well as context relation representations. Entity global representations model the semantic information of all entities in the document, entity local representations aggregate the contextual information of multiple mentions of specific entities, and context relation representations encode the topic information of other relations. Experimental results demonstrate that our model achieves superior performance on two public datasets for document-level RE. It is particularly effective in extracting relations between entities of long distance and having multiple mentions.

1 Introduction

Document-level relation extraction must connect entities and mentions across an entire document, where relations may span sentences. GLRE addresses this with global, local, and context-relation representations and is reported to perform especially well on long-distance relations and entities with multiple mentions.

1 Introduction

  • At least 40.7% of Wikipedia relations were reported as extractable only at the document level (Yao et al., 2019).Document-level RE can use information distributed across sentences.Source: §1 · p. 1
  • Document-level RE must handle complex reasoning and interactions among entities and mentions, including entities mentioned multiple times under aliases.Identifying a relation may require combining context from multiple mentions across sentences.Source: §1 · p. 1
  • Earlier approaches were limited to short spans or two entity mentions, while some graph models risked including irrelevant information as noise.The cited example requires considering multiple entity mentions together.Source: §1 · pp. 1–2
  • GLRE uses BERT features and a heterogeneous graph, with R-GCN for entity-global representations and multi-head attention to aggregate entity-local mention context.It also uses self-attention over other relations to encode document topic information for final prediction.Source: §1 · p. 2
  • GLRE is reported to outperform many state-of-the-art competitors on two public datasets, especially for long-distance relations and entities with multiple mentions.The introduction attributes this result to detailed experiments and analysis.Source: §1 · p. 2

2 Related Work

Related work spans sentence-level extraction, document-level extensions, graph-based approaches, and reading-comprehension methods. The paper distinguishes GLRE by combining local entity representations with information from other document relations.

2 Related Work

  • Sentence-level RE methods use patterns, features, kernels, and neural networks, but are limited to extracting intra-sentential relations.The paper also notes that these methods ignore interactions with entities outside the target pair.Source: §2 · p. 2
  • Prior document-level work extended sentence-level models, incorporated coreference, or aggregated information hierarchically across words, sentences, and documents.Other graph approaches represented words as nodes with syntactic-dependency and discourse-relation edges.Source: §2 · p. 2
  • Some document-level graph models predict one mention pair at a time, while others aggregate mention-pair scores but consider one target entity pair per document.The latter approaches also rely on external syntactic analysis tools to construct their graphs.Source: §2 · p. 3
  • GLRE differs from graph-based relation extraction by adding entity-local representations and using other document relations to refine prediction.The cited comparison notes that some prior graph models integrate document information indiscriminately, potentially introducing noise.Source: §2 · p. 3
  • Reading-comprehension approaches to document-level RE require domain knowledge for question templates and may perform poorly in zero-answer or multi-answer scenarios.The paper describes both scenarios as common for relation extraction.Source: §2 · p. 3

3 Proposed Model

GLRE classifies document entity pairs using global entity representations, pair-specific local mention representations, and context representations of other relations.

3 Proposed Model

  • GLRE encodes the document, builds entity global and local representations, and combines them with context relation representations for final predictions.Its architecture uses a global heterogeneous graph, multi-head attention over mentions, and self-attention over relation representations.Source: §3 · p. 3

3.1 Encoding Layer

  • BERT encodes the input document into word representations; long documents are processed sequentially as short paragraphs to fit BERT's input limit.Source: §3.1 · p. 3

3.2 Global Representation Layer

  • A heterogeneous graph represents mentions, entities, and sentences with typed edges, and stacked R-GCN layers encode document-wide entity semantics.The graph omits entity-entity edges because those relations are the prediction targets.Source: §3.2 · pp. 3–4

3.3 Local Representation Layer

  • For each entity pair, multi-head attention aggregates mention representations into pair-specific local representations, guided by global entity queries and sentence keys.The values come from initial mention-node representations, while the keys come from initial sentence-node representations.Source: §3.3 · p. 4

3.4 Classifier Layer

  • The classifier combines global and local entity representations with binned relative distance, then uses relation self-attention and an FFNN for binary predictions.Context relation representations capture document topic information; multiple relations per entity pair are handled as separate binary classifications.Source: §3.4 · p. 5

4 Experiments and Results

This section evaluates GLRE on CDR and DocRED against sentence- and document-level models, then analyzes where its representations help and what errors remain.

4.1 Datasets; 4.3 Experiment Setup

  • Evaluation uses the chemical-induced disease CDR dataset and manually annotated DocRED; experiments also use CDR train+dev and report DocRED test F1 and Ign F1.Source: §4.1 · p. 5, §4.3 · p. 6

4.2 Comparative Models

  • Comparisons include five sentence-level models adapted to documents and nine document-level models, spanning CNN/LSTM, graph-based, and BERT-based approaches.Source: §4.2 · pp. 5–6

4.4 Main Results

  • GLRE achieved the best results among all models on CDR and DocRED; the authors credit entity global and local representations, with BERT and context relation representations also boosting performance.Source: Table 2 · p. 6, Table 3 · p. 6, §4.4 · p. 7

4.5 Detailed Analysis

  • GLRE improved extraction at longer entity distances, especially distances of at least 3 sentences, where global graph and local mention representations help model interactions and noisy context.Source: Figure 3 · p. 7, §4.5 · p. 7
  • GLRE improved for entity pairs averaging at least 2 mentions, especially at least 4; on CDR in the [1, 2) bin, it slightly trailed Christopoulou et al. (2019).Source: §4.5 · p. 7, Figure 4 · p. 8
  • Removing any of GLRE’s three representation layers lowered F1, with the largest drop after removing local representations; BERT mattered more on DocRED than CDR.Source: Table 4 · p. 8, §4.5 · p. 8
  • On DocRED train+dev, XLNet-Large reached Ign F1 58.5 and F1 60.5; XLNet-Large and ALBERT-xxLarge did not improve CDR without biomedical versions.Source: Table 5 · p. 8, §4.5 · p. 8
  • Among 132 mispredicted CDR inter-sentential relations, logical reasoning errors (40.9%) and missing components (28.8%) dominated; missing prior knowledge (13.6%) and coreference errors (12.9%) also occurred.Source: §4.5 · pp. 8–9

5 Conclusion

GLRE outperforms many comparative models, with particularly large leads on long-distance relations and relations involving multiple mentions. It combines global, local, and context-relation representations, and the authors identify knowledge graphs and hierarchical graphs as future directions.

5 Conclusion

  • GLRE shows superiority over many comparative models, especially for long-distance relations and relations involving multiple mentions.The authors plan to explore knowledge graphs and other document-graph designs, such as hierarchical graphs, to improve performance.Source: §5 · p. 9
  • GLRE models document-wide entity semantics with R-GCN, aggregates mention context selectively with multi-head attention, and encodes other relations' topic information with self-attention.Source: §5 · p. 9
  • The authors plan to integrate knowledge graphs and explore other document-graph modeling approaches, including hierarchical graphs.Source: §5 · p. 9

A Notations

Table 7 summarizes the key notations used in the paper.

A Notations

B Dataset Availability

The appendix gives availability information for CDR and DocRED, while noting that DocRED's test-set gold standard is unavailable and only F1 can be obtained online.

B Dataset Availability

C Experimental Setup

The experiments used PyTorch 1.5 on a server with two NVIDIA Tesla V100 GPUs and applied dataset-specific preprocessing and stated training protocols. The reported setup also specifies BERT and projection dimensions, with hyperparameters listed in Table 8.

C Experimental Setup

  • GLRE was implemented with PyTorch 1.5 and run on a server with an Intel Xeon Gold 5117 CPU, 120 GB memory, and two NVIDIA Tesla V100 GPUs.The server used Ubuntu 18.04.Source: C Experimental Setup · p. 11
  • CDR preprocessing included sentence splitting, word tokenization, and hypernym filtering.Source: C Experimental Setup · p. 11
  • For training-set-only and train-plus-development settings, the best epoch was selected using development-set F1 before test evaluation.In the train-plus-development setting, the model was then rerun on the union of the training and development sets; dropout and layer normalization were used to prevent overfitting.Source: C Experimental Setup · pp. 11–12
  • BERT hidden size was 768, and a transformation layer projected its output to 256 dimensions.The hyperparameter values are shown in Table 8.Source: Table 8 · p. 11, C Experimental Setup · p. 12

D Case Study on DocRED

DocRED case studies illustrate logical reasoning across mentions, coreference resolution, and the use of prior knowledge. Further examples contrast GLRE with versions lacking entity-local or context-relation representations.

D Case Study on DocRED

  • The cases require combining multiple mentions for logical reasoning, resolving a pronoun through coreference, and using prior knowledge about geographic categories.One example uses “Conrad Johnson” mentions in two sentences; another infers that North America is a continent and California a state.Source: D Case Study on DocRED · p. 12
  • GLRE identifies the relation between “Kunar” and “Afghanistan,” while removing entity-local representations leads to an “N/A” prediction.The example attributes GLRE’s focus on “Afghanistan” in S3 to entity-local representations.Source: D Case Study on DocRED · p. 12
  • For “The Breaking Dawn” and “Stephenie Meyer,” GLRE predicts “author,” whereas removing context-relation representations yields “creator.”The case connects the prediction to the context relation “author” between “Eclipse” and “Stephenie Meyer.”Source: D Case Study on DocRED · p. 12
Loading 2009.10359v1…