Source-linked AI summary
Semantic Relation Classification via Bidirectional LSTM Networks with Entity-aware Attention using Latent Entity Typing
Joohong Lee, Sangwoo Seo, Yong Suk Choi
TL;DR
Relation-classification models often depend on high-level NLP-tool features, while attention-based models may not fully use tagged entities. This paper combines entity-aware attention and latent entity typing in an end-to-end recurrent model; on SemEval-2010 Task 8, it reports 85.2% F1-score and says it outperforms existing state-of-the-art methods without high-level features.
Problem
Relation-classification models often rely on high-level NLP-tool features, and attention-based models may not fully use tagged entities, which can be strong relation cues.
Source: §1 · p. 1Method
The paper proposes an end-to-end recurrent model with entity-aware attention and latent entity typing, using bidirectional LSTMs and self-attention for sentence representations.
Source: §1 · p. 1, §3.1 · p. 3, §3.4.2 · p. 4Results
85.2% F1-score on SemEval-2010 Task 8; the authors report outperforming existing state-of-the-art methods without high-level NLP-tool features.
Source: §6 · p. 7Takeaways & Limitations
Visualizations of the model’s attention mechanisms and LET results are presented as evidence that its decisions are more interpretable.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Classifying semantic relations between entity pairs in sentences is an important task in Natural Language Processing (NLP). Most previous models for relation classification rely on the high-level lexical and syntactic features obtained by NLP tools such as WordNet, dependency parser, part-of-speech (POS) tagger, and named entity recognizers (NER). In addition, state-of-the-art neural models based on attention mechanisms do not fully utilize information of entity that may be the most crucial features for relation classification. To address these issues, we propose a novel end-to-end recurrent neural model which incorporates an entity-aware attention mechanism with a latent entity typing (LET) method. Our model not only utilizes entities and their latent types as features effectively but also is more interpretable by visualizing attention mechanisms applied to our model and results of LET. Experimental results on the SemEval-2010 Task 8, one of the most popular relation classification task, demonstrate that our model outperforms existing state-of-the-art models without any high-level features.
1 Introduction
Relation classification predicts the semantic relationship between two tagged entities, a task used in several NLP applications. The paper proposes an end-to-end model that combines entity-aware attention with latent entity typing to address tool dependence and underuse of entity information.
1 Introduction
- Previous models often rely on NLP-tool features, which can propagate implicit errors and add computational expense.Source: §1 · p. 1
- Attention-based neural models may underuse tagged entities, although entity pairs can provide strong clues to their relation.The introduction contrasts Cause-Effect with Component-Whole for the pair crash and attack.Source: §1 · p. 1
- The proposed end-to-end recurrent model combines bidirectional LSTMs, entity-aware attention, and latent entity typing.Its attention uses entity-pair positions and latent types to focus on important sentence information.Source: §1 · p. 1
- The model is reported to obtain 85.2% F1-score on SemEval-2010 Task 8.The introduction’s result statement is truncated after beginning a comparison, so it does not establish a winner.Source: §1 · p. 1
- The authors say visualizing self-attention, entity-aware attention, and LET makes the model’s decision process more interpretable.Source: §1 · p. 2
2 Related Work
Prior relation-classification work ranges from handcrafted features and kernels to end-to-end neural models, attention mechanisms, and dependency-path approaches.
2 Related Work
- Early methods used handcrafted features from NLP tools or manually designed kernels, with tool-based approaches vulnerable to propagated errors.Source: §2 · p. 2
- End-to-end neural models use raw sentences and pretrained word representations, with CNNs, ranking-loss models, and bidirectional RNNs among the approaches described.Source: §2 · p. 2
- Other neural approaches combine attention with BLSTMs, hierarchical entity-specific attention, or word-level attention over CNN features.Source: §2 · p. 2
- Dependency-path systems such as SDP-LSTM and DRNNs select meaningful words along the shortest dependency path between entity pairs.Source: §2 · p. 2
3 Model
The model combines contextual word representations and bidirectional sequence encoding with entity-aware attention that incorporates entity positions and latent types. The resulting sentence representation is classified into a relation label.
3 Model
- The model has four stages: word representation, multi-head self-attention, BLSTM encoding, and entity-aware attention, followed by averaging across time steps.The averaged sentence-level features are used for classification.Source: §3 · p. 3
3.1 Word Representation
- Multi-head self-attention contextualizes word representations; in this layer, queries, keys, and values all use the same word vectors X.The attention uses parallel heads to attend to different parts of the input.Source: §3.1 · p. 3, Figure 3 · p. 3
3.3 Bidirectional LSTM Network
- The BLSTM processes self-attention outputs in both forward and reverse sentence order, concatenating the two hidden states at each time step.Each direction contributes a hidden state of dimension dh.Source: §3.3 · p. 4
3.4 Entity-aware Attention Mechanism
- Entity-aware attention calculates weights using BLSTM outputs together with relative-position and entity features derived using LET.Its output is the sentence representation.Source: §3.4 · p. 4, §3.4.2 · p. 5
3.4.1 Relative Position Features
- Relative-position features represent each word’s position relative to each entity, using learnable embeddings.The two position vectors are concatenated with the BLSTM state before a linear transformation.Source: §3.4.1 · p. 4
3.4.2 Entity Features with Latent Type
- LET constructs latent entity-type representations by weighting K latent type vectors; these representations join the entity-position hidden states as entity features.The combined entity features are linearly transformed and added to the BLSTM representations.Source: §3.4.2 · pp. 4–5
3.5 Classification and Training
- A fully connected softmax layer maps the entity-aware sentence representation to probabilities over relation classes, and training minimizes cross-entropy with AdaDelta.L2 regularization and dropout after word embedding, the LSTM, and entity-aware attention are used to alleviate overfitting.Source: §3.5 · p. 5
4 Experiments
The experiments evaluate the model on SemEval-2010 Task 8 and compare it with prior systems. They report an 85.2% F1-score and a 0.5% performance gain from latent entity typing.
4.1 Dataset and Evaluation Metrics
- SemEval-2010 Task 8 has 10 base relations (19 directional classes) across 10,717 sentences; evaluation uses macro-averaged F1 excluding Other and accounting for directionality.Source: §4.1 · p. 5
4.2 Implementation Details
- Hyperparameters were tuned on an 800-sentence validation set; word embeddings use pre-trained GloVe, while other weights use zero-mean Gaussian initialization.Table 1 presents the selected hyperparameters.Source: §4.2 · p. 5, Table 1 · p. 5
4.3 Experimental Results
- 85.2% F1-score outperforms all competing state-of-the-art approaches except depLCNN+NS, DRNNs, and Attention-CNN, which rely on high-level NLP-tool features.The comparison is on SemEval-2010 Task 8.Source: Table 2 · p. 6, §4.3 · p. 6
- LET improves performance by 0.5%, with the model performing best using three types.Source: §4.3 · p. 6
5 Visualization
The visualizations illustrate how self-attention and entity-aware attention weight words, and how latent entity types organize entity representations. The examples show attention on relation-relevant words and groups of entities with similar characteristics.
5.1 Self Attention
- Self-attention enriches word representations using within-sentence correlations; Figure 4 shows pollution and shipwreck attending to “was,” “caused,” and each other.Source: Figure 4 · p. 6, §5.1 · p. 6
5.2 Entity-aware Attention
- Figure 5 highlights important words; “inside” is the strongest cue for Component-whole(e1,e2), while “using” represents Instrument-Agency(e2,e1) better than “assess.”Source: §5.2 · pp. 6–7, Figure 5 · p. 7
5.3 Latent Entity Type
- Figure 6 uses t-SNE to show latent type vectors and entity representations, which are generally well divided and almost uniformly distributed.Red points mark latent type vectors; remaining points are entity representations colored by their closest type vector.Source: Figure 6 · p. 7, §5.3 · p. 7
- Figure 7 groups 50 entities near each of three latent type vectors; the groups show similar characteristics, including jobs and foods, engineering, and disaster- or drug-related words.Source: Figure 7 · p. 7, §5.3 · p. 7
6 Conclusion
The model combines entity-aware attention with latent entity typing for relation classification. On SemEval-2010 Task 8, it achieves 85.2% F1 using raw sentences and word embeddings, without high-level NLP-tool features, and outperforms existing state-of-the-art methods.
6 Conclusion
- 85.2% F1 on SemEval-2010 Task 8, using raw sentences and word embeddings without high-level NLP-tool features, outperforms existing state-of-the-art methods.Source: §6 · p. 7
- Three visualizations of the model’s attention mechanisms demonstrate greater interpretability than previous models.Source: §6 · p. 7
- The authors propose extending latent entity typing to sequence modeling without NER and developing question-answering or knowledge-base-population methods based on extracted entity relations.Source: §6 · p. 7