Source-linked AI summary
End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures
Makoto Miwa, Mohit Bansal
TL;DR
End-to-end relation extraction requires entities and relations to be modeled together while using both word-sequence and dependency-tree information. The paper introduces a shared-parameter LSTM-RNN model with entity-focused training, improving over feature-based end-to-end systems and comparing favorably with CNNs for nominal relation classification.
Problem
Prior relation-extraction models used limited linguistic structures, and existing end-to-end joint systems were feature-based rather than neural-network-based.
Method
The model stacks bidirectional sequential and tree-structured LSTM-RNNs, jointly decoding entities and relations with entity pretraining and scheduled sampling.
Results
The model achieves 12.1% and 5.7% relative error reductions in F1-score on ACE2005 and ACE2004, respectively, and compares favorably with a CNN-based model on SemEval-2010 Task 8.
Takeaways & Limitations
Using both word sequences and dependency trees, shared parameters, entity pretraining, and scheduled sampling improves end-to-end relation extraction performance.
Takeaways & Limitations
The evaluation did not tune precision-recall trade-offs, although doing so could specifically improve precision further.
Abstract
from arXiv · showhide
We present a novel end-to-end neural model to extract entities and relations between them. Our recurrent neural network based model captures both word sequence and dependency tree substructure information by stacking bidirectional tree-structured LSTM-RNNs on bidirectional sequential LSTM-RNNs. This allows our model to jointly represent both entities and relations with shared parameters in a single model. We further encourage detection of entities during training and use of entity information in relation extraction via entity pretraining and scheduled sampling. Our model improves over the state-of-the-art feature-based model on end-to-end relation extraction, achieving 12.1% and 5.7% relative error reductions in F1-score on ACE2005 and ACE2004, respectively. We also show that our LSTM-RNN based model compares favorably to the state-of-the-art CNN based model (in F1-score) on nominal relation classification (SemEval-2010 Task 8). Finally, we present an extensive ablation analysis of several model components.
1 Introduction
The paper targets end-to-end relation extraction by jointly modeling entities and relations over complementary sequence and dependency-tree information. Its neural model combines bidirectional sequential and tree-structured LSTM-RNNs with entity-focused training, improving over prior systems.
- Joint modeling of entities and relations is important because entity types and contextual relation cues interact closely.For example, entity types identify the participants in an Organization-Affiliation relation, while surrounding words can encourage their extraction.
- Prior LSTM-based relation classifiers underperformed CNN-based systems while using limited linguistic structures and not jointly modeling entities and relations.
- The proposed model jointly represents word sequences and dependency trees with bidirectional sequential and bidirectional tree-structured LSTM-RNNs.
- Entity pretraining and scheduled sampling address weak early entity detection and provide entity information to downstream relation classification.
- 12.1% and 5.7% relative error reductions in F1-score are achieved over the state-of-the-art feature-based model on ACE2005 and ACE2004, respectively.
2 Related Work
Earlier neural relation models often used only one linguistic structure or lacked joint entity-relation modeling. This work addresses those gaps with bidirectional sequence and tree-structured LSTM-RNNs in an end-to-end model.
- LSTM-RNNs had been widely used for sequential labeling tasks including clause identification, phonetic labeling, and named entity recognition.
- Neural relation-classification research included embedding-based, CNN-based, and RNN-based models for SemEval-2010 Task 8.
- Earlier tree-structured LSTM-RNNs fixed bottom-to-top information flow and could not handle arbitrary numbers of typed children in dependency trees.
- No earlier RNN-based relation-classification model simultaneously used word-sequence and dependency-tree information.
- Existing end-to-end joint relation-extraction models were feature-based rather than neural-network-based.
3 Model
The model jointly extracts entities and relations by combining bidirectional sequential and dependency-tree LSTM-RNN representations. It incrementally detects entities, builds relation candidates, and classifies them using shared neural representations, entity information, and scheduled-sampling training.
- Architecture: Three stacked representation layers encode embeddings, word sequences, and dependency subtrees for end-to-end relation extraction.The sequence layer maintains sentential context and entities, while the dependency layer provides relation-specific representations.
- Sequence layer: Bidirectional sequential LSTM-RNNs represent words in both directions and pass concatenated hidden states to subsequent layers.The sequence layer receives word and POS embeddings and maintains contextual representations for entity detection.
- Entity detection: Entities are decoded greedily from left to right as BILOU sequence labels, with previous predicted labels informing subsequent predictions.The entity detector uses a two-layer neural network over the sequence layer and incorporates previous label embeddings.
- Dependency layer: Bidirectional tree-structured LSTM-RNNs propagate information bottom-up and top-down around dependency-tree relation candidates.The model primarily uses shortest paths, while SubTree and FullTree add modifier or full-sentence context; a single connected-root variant performed slightly worse initially.
- Relation classification: Relation candidates combine the final words of detected entities, and relation classification uses dependency-tree representations stacked on sequential representations.The candidate vector includes the lowest common ancestor’s bottom-up state and both target words’ top-down states; entity-level sequence states are also concatenated directly.
- Training: Entity pretraining and scheduled sampling encourage entity detection and the construction of positive relation instances during training.Scheduled sampling uses gold labels with inverse-sigmoid decay, while entity pretraining initializes the entity detection model before joint training.
4 Results and Discussion
Experiments on ACE05, ACE04, and SemEval-2010 Task 8 evaluate the model against prior systems and isolate the effects of its training and representation choices. The results show gains over the feature-based ACE baseline, favorable SemEval comparison, and strong dependence on entity-focused training and shortest-path tree representations.
- End-to-end relation extraction: The model outperforms the state-of-the-art feature-based model on ACE05 and ACE04 end-to-end relation extraction.Table 1 compares the model with Li and Ji (2014) on the ACE05 test set and ACE04 dataset.
- Training and parameter sharing: Removing entity pretraining significantly degrades performance, while removing scheduled sampling causes a smaller degradation on ACE05.Removing both enhancements is also significantly worse than SPTree (p<0.05).
- Training and parameter sharing: Entity label information helps relation detection, while removing shared parameters slightly lowers entity detection and relation-classification performance.The shared-parameter differences are not significant, but removing all enhancements is significantly worse than SPTree (p<0.01).
- LSTM-RNN structures: Selecting the shortest-path tree representation matters more than choosing a sequential versus tree-based LSTM-RNN on that input.This trend holds for both end-to-end extraction and relation classification; FullTree without shortest-path distinctions significantly hurts performance (p<0.05).
- Relation classification: The model is comparable in F1-score to state-of-the-art CNN-based models on SemEval-2010 Task 8.This comparison holds for models with or without external WordNet information.
- Relation classification: Removing both the sequence layer and entity-related sequence information significantly reduces SemEval performance, while adding WordNet yields only a small improvement.Replacing the Stanford neural parser with the Stanford PCFG parser also causes only a small drop.
5 Conclusion
The model jointly represents entities and relations using bidirectional sequential and tree-structured LSTM-RNNs. Evaluation found gains over feature-based end-to-end extraction and favorable comparison with CNN-based nominal relation classification, while ablations identified effective components.
- Bidirectional sequential and tree-structured LSTM-RNNs represent word sequences and dependency trees within one end-to-end model.
- The shared model represents entities and relations together, improving over the state-of-the-art feature-based system on ACE04 and ACE05.
- The model compares favorably with recent state-of-the-art CNN-based models on nominal relation classification in SemEval-2010 Task 8.
- Using both word sequence and dependency tree structures is effective for relation extraction.
- Shared-parameter training improves relation extraction accuracy, particularly with entity pretraining, scheduled sampling, and label embeddings.
- The shortest dependency path is appropriate for representing tree structures in neural LSTM models.
A.1 Data and Task Settings
The evaluation uses ACE04, ACE05, and SemEval-2010 Task 8, each with defined entity or relation categories and dataset-specific preprocessing or split procedures.
- ACE05: ACE05 defines 7 coarse-grained entity types and 6 coarse-grained relation types, using a 351/80/80 train/dev/test split after preprocessing.
- ACE04: ACE04 defines 7 coarse-grained entity types and 7 coarse-grained relation types, removes DISC, and uses 5-fold cross-validation on 348 documents.
- SemEval-2010 Task 8: SemEval-2010 Task 8 defines 9 nominal relation types plus Other, treats Other as negative, ignores direction, and contains 8,000 training and 2,717 test sentences.
A.2 Hyper-parameter Settings
Hyper-parameters were tuned over specified ranges covering optimization, regularization, dropout, gradient clipping, scheduled sampling, training duration, and WordNet hypernym embeddings.
- The tuning ranges include six initial learning rates, four regularization values, and six dropout probabilities.
- Training and entity-pretraining duration was tested up to 100 epochs, and WordNet hypernym embedding dimensions ranged from 5 to 30.