Source-linked AI summary
Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction
Tapas Nayak, Hwee Tou Ng
TL;DR
Relation extraction must handle multiple tuples whose entities may overlap, a challenge that pipeline systems address incompletely. The paper proposes two encoder-decoder approaches for jointly extracting entities and relations, and reports significantly improved F1 scores on New York Times datasets over prior work.
Problem
Pipeline systems identify entities before relations, missing interactions among multiple tuples and complicating extraction of overlapping entities with full names.
Method
The paper uses a tuple representation for word-level decoding and a pointer-network decoder that extracts two entity spans and their relation per step.
Results
3.9% and 4.1% higher F1 scores than HRL on NYT29 and NYT24, respectively, for WordDecoding; the improvements are statistically significant (p < 0.001).
Takeaways & Limitations
The proposed encoder-decoder models achieve improved state-of-the-art F1 scores for joint entity and relation extraction on the NYT corpus.
Abstract
from arXiv · showhide
A relation tuple consists of two entities and the relation between them, and often such tuples are found in unstructured text. There may be multiple relation tuples present in a text and they may share one or both entities among them. Extracting such relation tuples from a sentence is a difficult task and sharing of entities or overlapping entities among the tuples makes it more challenging. Most prior work adopted a pipeline approach where entities were identified first followed by finding the relations among them, thus missing the interaction among the relation tuples in a sentence. In this paper, we propose two approaches to use encoder-decoder architecture for jointly extracting entities and relations. In the first approach, we propose a representation scheme for relation tuples which enables the decoder to generate one word at a time like machine translation models and still finds all the tuples present in a sentence with full entity names of different length and with overlapping entities. Next, we propose a pointer network-based decoding approach where an entire tuple is generated at every time step. Experiments on the publicly available New York Times corpus show that our proposed approaches outperform previous work and achieve significantly higher F1 scores.
Introduction
Prior systems separate entity detection from relation classification, limiting their handling of interacting and overlapping relation tuples. This paper proposes encoder-decoder methods that jointly extract full entities and relations, including multiple tuples with overlapping entities.
- Pipeline systems identify entities before classifying relations, missing interactions among multiple relation tuples in a sentence.
- Joint neural models share parameters but still identify all entities before finding relations and do not fully capture tuple interactions.
- Sequence tagging cannot extract overlapping-entity tuples, while prior copying extracts only the last token rather than full entity names.
- The paper introduces a tuple representation enabling word-level decoding to recover multiple overlapping tuples and multi-token entities.
- A pointer-network decoder extracts both entity spans and their relation as an entire tuple at each decoding step.
Task Description
The task is to extract every relation tuple in a sentence, including tuples that share one or both entities. Sentences are categorized by whether tuples have no, pairwise, or single-entity overlap.
- A relation tuple contains two entities and one relation, with entities represented as text spans and relations drawn from a predefined set.
- No Entity Overlap contains tuples that share no entities, while Entity Pair Overlap shares both entities in the same or reverse order.
- Single Entity Overlap contains tuples sharing exactly one entity, and a sentence may belong to both overlap classes.
- The system must extract all relation tuples present in each sentence.
Encoder-Decoder Architecture
The paper uses two encoder-decoder designs to jointly extract relation tuples, combining sequence generation or pointer-based tuple decoding with entity and relation modeling. The pointer approach generates complete tuples and uses pointer networks to identify entity spans.
- Word-Level Decoding: The word-level approach represents each tuple as entity1 ; entity2 ; relation and separates multiple tuples with |.This uniform representation supports overlapping entities and entities of different lengths, while enabling tuple recovery after generation.
- Word-Level Decoding: The decoder generates one target token at a time, conditioning on source attention and the previous target embedding until EOS.A projection layer maps decoder output to the shared vocabulary, with inference masking restricting predictions to valid source, relation, and special tokens.
- Pointer Network-Based Decoding: The pointer-based approach represents each tuple with two entity spans and a relation, then generates an entire tuple at every decoding step.Its decoder combines an LSTM, two pointer networks, and a relation-classification network, while previously generated tuples are supplied to avoid duplicates.
- Pointer Network-Based Decoding: The first pointer network assigns start and end probabilities to source tokens for the first entity.Encoder and decoder representations are processed by a Bi-LSTM and feed-forward layers before softmax produces the token probabilities.
- Pointer Network-Based Decoding: A second pointer network identifies the second entity, while a feed-forward softmax network predicts the relation.The predicted relation is embedded and combined with the entity representations to form the tuple representation; decoding stops when EOS is predicted.
- Attention Modeling: The word-level decoder was evaluated with three attention mechanisms, and the single attention mechanism performed best.The single mechanism is the attention approach proposed by Bahdanau, Cho, and Bengio (2015).
Experiments
Experiments evaluate the models on two NYT datasets designed to contain many overlapping relation tuples, using set-based precision, recall, and F1. Both proposed decoders outperform HRL, with statistically significant gains and further improvements from ensembling.
- Datasets: The experiments use NYT24 with 24 relations and NYT29 with 29 relations, selected for their larger numbers of overlapping relation tuples.NYT24 is the version used by Zeng et al. (2018), while NYT29 is the version used by Takanobu et al. (2019).
- Baselines: The comparison includes SPTree, Tagging, CopyR, and HRL as state-of-the-art joint entity and relation extraction baselines.HRL achieves the highest F1 scores among the baselines on both datasets.
- Evaluation: Extracted tuples are deduplicated and counted correct only when both full entity names and the relation match; precision, recall, and F1 are reported.
- Results: WDec exceeds HRL by 3.9% and 4.1% F1 on NYT29 and NYT24, while PNDec exceeds HRL by 3.0% and 1.3%, respectively.These gains are statistically significant under bootstrap-paired t-tests with p < 0.001.
- Results: Ensembling five runs raises WDec's F1 advantage over HRL to 4.2% and 3.5%, and PNDec's to 4.2% and 2.9% on NYT29 and NYT24.The ensemble includes tuples extracted by at least three of five runs and significantly increases precision with a small recall improvement.
Analysis and Discussion
The analysis compares WordDecoding and PtrNetDecoding through attention, copy, dataset, error, and efficiency results. PtrNetDecoding offers lower-resource tuple-level decoding, while WordDecoding achieves stronger F1 in the reported ablations.
- Ablation Studies: WordDecoding with single attention achieves the highest F1 score on both datasets.The comparison appears in the ablation analysis of attention mechanisms.
- Ablation Studies: The masking-based copy mechanism improves F1 scores by around 4–7% across attention mechanisms and both datasets.
- Ablation Studies: PtrNetDecoding achieves its highest F1 scores when attention uses both the previous decoder hidden vector and previously extracted tuples.
- Performance Analysis: NYT24 yields significantly higher F1 scores than NYT29 because 72.0% of its test tuples overlap with training tuples, versus 41.7% for NYT29.The paper attributes the difference to memorization by deep neural networks.
- Performance Analysis: PtrNetDecoding extracts an entire tuple at each decoding step, whereas WordDecoding requires about eight steps per tuple under the stated average entity length.The pointer model therefore uses fewer decoding steps.
- Performance Analysis: PtrNetDecoding is more than two times faster and uses one-third of WordDecoding’s GPU memory during training and inference.The reported efficiency advantage is linked to PtrNetDecoding’s fewer decoding steps.
Related Work
Earlier relation extraction systems commonly separated entity detection from relation classification, while later models shared parameters or jointly decoded entities and relations. Prior approaches nevertheless retained limitations for overlapping entities or full entity-name extraction, motivating the paper’s encoder-decoder designs.
- Pipeline and Joint Extraction: Traditional systems first detect entities with NER and then classify relations between them in a pipeline.
- Pipeline and Joint Extraction: Neural joint models share parameters for entity and relation extraction but still identify relations after identifying all entities.They therefore do not fully capture interaction among multiple tuples.
- This Paper’s Position: This paper uses pointer networks with an encoder-decoder model to extract relation tuples from sentences for the first time.
Conclusion
The paper addresses sentence-level relation tuple extraction with two encoder-decoder approaches designed for multiple, overlapping tuples and entities of different lengths. Experiments on NYT report significantly improved state-of-the-art F1 scores, with document-level extraction identified as future work.
- Conclusion: The proposed encoder-decoder models address multiple tuples, overlapping entities, and entities of different lengths in sentence-level extraction.
- Conclusion: Experiments on the New York Times corpus show significantly improved new state-of-the-art F1 scores.
- Conclusion: Future work will explore the proposed models for document-level tuple extraction.