PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning with Latent Structure Refinement for Document-Level Relation Extraction

Guoshun Nan, Zhijiang Guo, Ivan Sekulić, Wei Lu

arXiv:2005.06312v3cs.CL

TL;DR

Document-level relation extraction must synthesize information across sentences, while static graphs based on syntax, co-references, or heuristics may miss complex non-local interactions. LSR automatically induces a latent document-level graph and iteratively refines it for multi-hop reasoning, outperforming existing approaches on DocRED and achieving new state-of-the-art results on biomedical datasets.

  • Problem

    Document-level relation extraction requires integrating information across multiple sentences and capturing complex interactions among inter-sentence entities.

  • Method

    LSR automatically induces a task-specific latent document-level graph and iteratively refines it to incrementally aggregate information for multi-hop reasoning.

  • Results

    LSR significantly outperforms existing approaches on DocRED and achieves new state-of-the-art results on the CDR and GDA datasets.

  • Takeaways & Limitations

    Analyses show that LSR discovers more accurate inter-sentence relations through multi-hop reasoning.

Abstract

from arXiv · show

Document-level relation extraction requires integrating information within and across multiple sentences of a document and capturing complex interactions between inter-sentence entities. However, effective aggregation of relevant information in the document remains a challenging research question. Existing approaches construct static document-level graphs based on syntactic trees, co-references or heuristics from the unstructured text to model the dependencies. Unlike previous methods that may not be able to capture rich non-local interactions for inference, we propose a novel model that empowers the relational reasoning across sentences by automatically inducing the latent document-level graph. We further develop a refinement strategy, which enables the model to incrementally aggregate relevant information for multi-hop reasoning. Specifically, our model achieves an F1 score of 59.05 on a large-scale document-level dataset (DocRED), significantly improving over the previous results, and also yields new state-of-the-art results on the CDR and GDA dataset. Furthermore, extensive analyses show that the model is able to discover more accurate inter-sentence relations.

1 Introduction

Document-level relation extraction requires synthesizing information across multiple sentences, but existing static structures based on syntax, co-reference, or heuristics may miss complex non-local interactions. LSR instead induces and iteratively refines a latent document-level graph for multi-hop reasoning.

  • Document-level extraction infers entity relations by synthesizing relevant information across multiple sentences.
  • The model uses iterative refinement to incrementally capture complex interactions and perform multi-hop reasoning.
  • Experiments report significant improvements over existing approaches on DocRED and new state-of-the-art results on two biomedical datasets.
  • LSR induces a document-level graph end-to-end without relying on co-references or rules, then dynamically refines it for information aggregation.

2 Model

The LSR model combines node construction, dynamic reasoning, and classification to induce and use a document-level structure for relation extraction.

  • LSR has three components: a node constructor, dynamic reasoner, and classifier.
  • The node constructor encodes each sentence and extracts mention and shortest-dependency-path token representations as graph nodes.
  • The dynamic reasoner induces a document-level structure from the extracted nodes and updates their representations.

2.1 Node Constructor

The node constructor encodes sentences contextually and builds mention, entity, and meta dependency path nodes for the document-level graph.

  • A context encoder produces contextualized representations for each word in every document sentence.
  • The document-level graph therefore contains mention nodes, entity nodes, and meta dependency path nodes.
  • Mention nodes represent entity mentions, while entity nodes are formed by averaging the representations of their mentions.
  • Meta dependency path nodes represent tokens on shortest dependency paths between mentions within a sentence.

2.2 Dynamic Reasoner

The dynamic reasoner jointly induces a latent document-level graph and performs multi-hop reasoning over it, repeatedly refining the structure from updated node representations. It uses structured attention with the Matrix-Tree Theorem for induction and densely connected GCNs for information aggregation.

  • Dynamic Reasoner: The dynamic reasoner alternates structure induction with multi-hop reasoning and stacks N blocks for iterative graph refinement.Each block induces a structure, updates node representations, and produces a latent structure for inference.
  • Structure Induction: The model treats document-level graph structure as a latent variable and induces it end-to-end rather than using co-reference links or heuristics.A variant of Kirchhoff’s Matrix-Tree Theorem generates task-specific dependency structures for non-local entity interactions.
  • Structure Induction: Pairwise node representations are transformed into unnormalized attention scores, while root scores identify candidate roots for the latent structure.The scores use feed-forward networks, a bilinear transformation, and a linear root-scoring transformation.
  • Structure Induction: The induced edge marginals form a weighted adjacency matrix A that feeds the multi-hop reasoning module.The model derives marginal dependency-edge probabilities using the Matrix-Tree formulation before updating node representations.
  • Multi-hop Reasoning: Dense graph convolutional networks update each node from neighboring information and support deeper reasoning over large document-level graphs.Dense connections help capture richer local and non-local information while using the previous layer’s node representations as input.
  • Iterative Refinement: Repeated refinement addresses the shallow structures produced by one-pass latent-structure induction by incorporating increasingly non-local information.Early iterations mainly propagate between neighboring nodes, whereas later refinements can generate more informative structures.

2.3 Classifier

After refinement, the classifier computes relation probabilities for each entity pair using a bilinear function over the final node representations.

  • Classifier: The classifier applies a bilinear function to the final representations of each entity pair to compute probabilities for every relation type.The relation probability uses trainable weights and bias, a sigmoid function, and relation-specific indexing.

3 Experiments

The experiments evaluate LSR on DocRED and compare it with sequence-based, graph-based, and BERT-based models. LSR achieves strong DocRED results, with gains over both sequence models and static or attention-based graph models.

  • 3.1 Data: The evaluation uses F1, Ign F1, and separate intra- and inter-sentence F1 scores across DocRED, CDR, and GDA.
  • 3.3 Main Results: 54.18 F1 is achieved by LSR with GloVe on the DocRED test set, the best result among models with GloVe.
  • 3.3 Main Results: 3.1 F1 points separate LSR from the best sequence-based model, BiLSTM, on DocRED.
  • 3.3 Main Results: 3.0 and 2.4 higher F1 are obtained by LSR than EoG on the development and test sets, respectively.
  • 3.3 Main Results: LSR consistently outperforms graph-based models using static graphs or attention mechanisms and also outperforms GAT and AGGCN.
  • 3.3 Main Results: 59.05 F1 is achieved by LSR with BERT on DocRED, establishing a new state-of-the-art result.

3.4 Intra- and inter-sentence performance

LSR performs better than other models for both intra- and inter-sentence entity pairs, with its largest advantages appearing when reasoning across sentences. These results support the role of its latent structure in synthesizing document-wide information.

  • 3.4 Intra- and inter-sentence performance: About 45% of DocRED development-set entity pairs require information aggregation over multiple sentences.
  • 3.4 Intra- and inter-sentence performance: LSR outperforms all other models in both intra- and inter-sentence settings.
  • 3.4 Intra- and inter-sentence performance: The differences between LSR and other models are larger for inter-sentence than intra-sentence entity pairs.
  • 3.4 Intra- and inter-sentence performance: The majority of LSR’s superiority comes from inter-sentence relational facts, indicating that its latent structure synthesizes information across multiple sentences.
  • 3.4 Intra- and inter-sentence performance: LSR with GloVe performs better than two BERT-based models in the inter-sentence setting.

3.5 Results on the Biomedical Datasets

On the biomedical datasets, simplified LSR achieves strong results on GDA, while the full LSR performs worse on CDR because parser-derived MDP nodes provide less informative context. In the DocRED refinement analysis, LSR outperforms competing graph structures overall and benefits from iterative refinement.

  • CDR: Full LSR performs worse than state-of-the-art models on CDR because spaCy-extracted MDP nodes contain less informative context than those from DocRED.Removing MDP nodes and building a fully connected graph consistently improves performance over sequence-based and graph-based models.
  • GDA: The simplified LSR model achieves the new state-of-the-art result on the distantly supervised GDA dataset.The comparison distinguishes the simplified LSR from the full EoG-based model and reports stronger overall performance for simplified LSR.
  • Structure comparison: LSR outperforms QAGCN, EoG, and AGGCN in overall F1 across settings with one to four refinement blocks.The comparison uses the same GCN model for multi-hop reasoning across the evaluated structures.
  • Refinement analysis: LSR achieves its best performance at the second refinement, improving overall F1 by 0.72% over the first induction.Additional iterations can reduce F1 because of over-fitting.

3.7 Ablation Study

The ablation study shows that LSR’s components generally contribute to performance, with structure induction having the largest observed impact. A case study illustrates how LSR focuses attention to support cross-sentence reasoning but can still produce a false relation.

  • Ablation results: Removing structure induction lowers F1 by 3.26 points, indicating that latent structure is central to LSR’s performance.The study reports performance deterioration when individual components are removed.
  • Case study: LSR predicts the Japan–World War II relation “participant of” by propagating information through intra- and inter-sentence structures.The case study examines the reasoning process over two refinement steps using attention to World War II mentions.
  • Case study: LSR attends strongly to New Ireland and consequently fails to predict that New Ireland and World War II have no relation.The example identifies a NIL relation that the model misses.

4 Related Work

Related work addresses document-level relation extraction through sequence models, structural representations, and multi-hop reasoning. Prior structural approaches construct document graphs using dependency trees, co-reference information, or rules.

  • Document-level relation extraction: Document-level relation extraction extends earlier sentence-level relation extraction to datasets requiring document understanding and multi-hop reasoning.The task involves relations among entities across multiple sentences and domains.
  • Document-level relation extraction: Prior approaches use sequence encoders such as CNNs, LSTMs, BiLSTMs, and attention-based models for document-level relation extraction.These methods encode sentences or document sequences without the structural graph constructions emphasized by graph-based approaches.
  • Structure-based relational reasoning: Structure-based methods model non-local dependencies with dependency graphs, co-reference links, heuristics, GCNs, or related graph reasoning architectures.These approaches have been applied in both question answering and relation extraction.

5 Conclusion

The paper introduces LSR for document-level relation extraction, dynamically learning document structures and making predictions end to end. It identifies extending node construction without an external parser as a direction for future work.

  • Conclusion: LSR dynamically learns a document-level structure and makes predictions in an end-to-end fashion instead of relying on syntactic trees, co-references, or heuristics.The model is presented as a latent structure refinement approach for document-level relation extraction.
  • Conclusion: Future work could extend structure induction to construct nodes without relying on an external parser.This is identified as one possible direction for broadening the model’s structure induction scope.
Loading 2005.06312v3…