Source-linked AI summary
Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extraction
Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, Zhendong Mao
TL;DR
Document-level relation extraction requires modeling dependencies among entity mentions, but prior approaches provide limited or delayed structural guidance. The paper formalizes these dependencies and introduces SSAN, a structurally guided self-attention network that integrates them throughout encoding, achieving state-of-the-art results on three datasets.
Problem
Document-level entities span multiple mentions with meaningful dependencies, while prior techniques have limited modeling ability and often omit dependencies beyond coreference.
Method
SSAN formalizes mention dependencies in a unified framework and incorporates them into self-attention within and throughout the encoding network.
Results
SSAN achieves new state-of-the-art results with consistent and substantial improvements over competitive baselines on DocRED, CDR, and GDA.
Takeaways & Limitations
The results demonstrate the usefulness of entity structure and SSAN for document-level relation extraction.
Takeaways & Limitations
The formulation assumes no crucial dependency for inter-sentential non-entity words and leaves more complex discourse-based interactions for future work.
Abstract
from arXiv · showhide
Entities, as the essential elements in relation extraction tasks, exhibit certain structure. In this work, we formulate such structure as distinctive dependencies between mention pairs. We then propose SSAN, which incorporates these structural dependencies within the standard self-attention mechanism and throughout the overall encoding stage. Specifically, we design two alternative transformation modules inside each self-attention building block to produce attentive biases so as to adaptively regularize its attention flow. Our experiments demonstrate the usefulness of the proposed entity structure and the effectiveness of SSAN. It significantly outperforms competitive baselines, achieving new state-of-the-art results on three popular document-level relation extraction datasets. We further provide ablation and visualization to show how the entity structure guides the model for better relation extraction. Our code is publicly available.
1 Introduction
Document-level relation extraction must reason over entities spanning multiple mentions and the dependencies among those mentions. SSAN incorporates this structure within and throughout encoding, achieving state-of-the-art results across three datasets.
- Motivation: Document-level entities span multiple mentions whose dependencies provide informative priors for relation extraction.Examples include local co-occurrence, coreference across sentences, and indirect associations through coreferential mentions.
- Limitations of prior work: Earlier methods mainly treated entity dependencies as extra features, pooled representations, or prediction aggregation, limiting their modeling ability.These approaches focused particularly on coreference and often used dependencies only before or after the main encoding process.
- Limitations of prior work: Graph-based methods perform structure reasoning after contextual encoding, so contextual representations cannot benefit from structural guidance during initial encoding.The heterogeneity between encoder and graph network separates context reasoning from structure reasoning.
- SSAN: SSAN formulates mention dependencies in a unified framework and incorporates them within self-attention throughout the encoding network.Its structural guidance supports simultaneous and interactive context and structure reasoning across network layers.
- Results: SSAN achieves consistent and substantial improvements over competitive baselines on DocRED, CDR, and GDA, establishing new state-of-the-art results.The experiments cover one entity-rich general-domain dataset and two biomedical-domain datasets.
2 Approach
SSAN models document-level entity structure as token-pair dependencies and incorporates that structure throughout Transformer-style self-attention. It uses structured attentive biases to regulate attention flow before aggregating contextual representations for relation extraction.
- 2.1 Entity Structure: Entity structure comprises co-occurrence and coreference relations, distinguishing intra- versus inter-sentential interactions and shared- versus distinct-entity mentions.These structures yield four mention dependencies; intra-sentential non-entity words are additionally categorized as intraNE, while other inter-sentential non-entity words are NA.
- 2.2 SSAN: SSAN extends the Transformer encoder with structured self-attention and two alternative transformation modules inside its building blocks.The encoder retains feedforward networks, residual connections, and layer normalization while adding structural guidance.
- 2.2 SSAN: The model represents entity structure as a token-level matrix whose entries specify one of six discrete dependencies between query and key tokens.Dependencies are assigned to subword pairs within mentions as intra+coref, extending mention-level structure for implementation.
- 2.2 SSAN: Structured attentive biases are combined with standard query-key attention scores to guide the final self-attention flow.The resulting attention is normalized with softmax and used to aggregate value vectors.
- 2.3 Transformation Module: Biaffine and Decomposed Linear Transformation modules model dependency-specific biases from contextualized query and key representations.Biaffine transformation attends to query and key vectors jointly, while decomposed transformation introduces separate query-conditioned, key-conditioned, and prior bias terms.
- 2.4 SSAN for Relation Extraction: For relation extraction, SSAN average-pools encoded mentions into entity representations and computes relation probabilities for entity pairs using a predefined schema.Training uses cross-entropy loss, and predictions are computed for all N × N entity pairs across M relations.
3 Experimental Setup
The experiments evaluate SSAN on three document-level relation extraction datasets spanning Wikipedia, chemical-disease, and gene-disease domains. Models use several pretrained language-model initializations, with hyperparameters selected on development data and test results reported for the best model.
- Datasets: SSAN is evaluated on DocRED, CDR, and GDA, three datasets requiring relational reasoning across multiple entities and sentences.DocRED is general-domain, while CDR and GDA are biomedical datasets.
- Datasets: DocRED contains 97 target relations and approximately 26 entities per document, with 3053 training, 1000 development, and 1000 test documents.It is constructed from Wikipedia and Wikidata with annotations for mentions, types, relations, and supporting evidence.
- Datasets: CDR contains 1500 human-annotated PubMed documents split equally into training, development, and test sets for binary chemical-to-disease relation classification.The target is an induced relation from a chemical entity to a disease entity.
- Datasets: GDA is a binary gene-disease relation task with 29192 distantly supervised training documents and 1000 test documents.The dataset is constructed from MEDLINE abstracts.
- Implementation: SSAN is initialized with BERT, RoBERTa, or SciBERT, with SciBERT used for the two biomedical datasets.The study compares each SSAN initialization with its corresponding baseline and selects test models using development-set grid search.
- Evaluation: Table 2 reports DocRED results for Decomposed Linear and Biaffine transformations, with official test results submitted to Codalab.The table caption identifies the transformation subscripts and notes a result marked with an asterisk from Nan et al. (2020).
4 Experiments and Results
Experiments on DocRED, CDR, and GDA show that SSAN consistently improves over baselines, while ablations and visualizations clarify the contributions of entity dependencies, bias terms, and layer placement.
- DocRED Results: SSAN consistently outperforms its baselines with both Biaffine and Decomp transformations.Biaffine usually provides larger gains than Decomp, indicating stronger structural-dependency modeling.
- DocRED Results: 63.78 Ign F1 and 65.92 F1 are achieved on the DocRED test set after distantly supervised pretraining and finetuning.The adapted model reaches the 1st leaderboard position at submission time.
- CDR and GDA Results: On CDR, SSAN gains +1.3 F1 with BERT Base, +1.7 F1 with BERT Large, and +2.9 F1 with SciBERT over corresponding baselines.Similar improvements are observed on GDA, supporting applicability across biomedical datasets.
- Ablation Study: All five entity dependencies contribute to DocRED performance, with intra+coref having the largest effect in the ablation.The “-all” condition degenerates to the RoBERTa Large baseline.
- Ablation Study: The biaffine bias term yields +1.38 Ign F1 alone, while key-conditioned bias outperforms query-conditioned bias in the decomposed transformation.The bias-term ablation is implemented across all layers.
- Visualization and Layer Analysis: Inter+coref biases are significantly positive in top layers, consistent with coreferential mentions supporting cross-sentence information passing.Intra+relate and inter+relate biases appear in contrast.
- Visualization and Layer Analysis: Top-layer structural biases alone retain a +0.89 F1 gain, while using all 24 layers reaches the best gain of +1.50 F1.Figure 3 averages attentive biases across attention heads and DocRED development instances.
5 Related Work
Related work has incorporated entity structure across NLP tasks, including document-level relation extraction, while structural priors have also been explored within neural encoders.
- Entity structure has been used in NLP for discourse analysis, language modeling, reading comprehension, and document-level relation extraction.
- Document-level relation extraction methods have merged coreferential predictions, modeled entity interactions through latent structure, or constructed graphs linking mentions, entities, and sentences.
- Neural-network research has explored injecting structural priors into LSTM and Transformer encoders, including relative-position information as attentive bias.
6 Conclusion and Future Work
The paper formalizes entity structure for document-level relation extraction and introduces SSAN to reason about context and entity structure simultaneously. It reports results on three datasets and identifies broader tasks and richer dependencies as future directions.
- The paper formalizes entity structure for document-level relation extraction and proposes SSAN to incorporate structural priors.
- SSAN performs contextual reasoning and entity-structure reasoning simultaneously and interactively.
- Results on three datasets demonstrate the usefulness of entity structure and the effectiveness of SSAN.
- Future work includes applying SSAN to reading comprehension and extending entity structure to more complex discourse-based interactions.
A Datasets
The three selected datasets contain substantial mention-level complexity, with more than two dozen mentions per document on average and approximately three mentions per sentence.
- More than two dozen mentions occur per document on average across the three selected datasets.
- Each sentence contains approximately three mentions on average across the three selected datasets.
- These statistics demonstrate the complexity of entity structure in document-level relation extraction tasks.
B Hyper-parameters Setting
The experiments use hyperparameters selected through grid search on the development set, with comparable results evaluated under the same search scope.
- All experiment results use hyperparameters selected through grid search on the development set.
- The development set serves as the basis for hyperparameter selection.
- All comparable results share the same hyperparameter search scope.