Source-linked AI summary
Leveraging Knowledge Bases in LSTMs for Improving Machine Reading
Bishan Yang, Tom Mitchell
TL;DR
Machine reading models often disregard background knowledge or use sparse symbolic KB features that generalize poorly and can be misleading across contexts. KBLSTM integrates continuous KB concept embeddings into BiLSTMs through context-sensitive attention and a sentinel, and achieves state-of-the-art results for entity and event extraction on ACE2005.
Problem
Existing KB integration relies on sparse symbolic features with poor generalization and can mislead recurrent models when knowledge relevance varies by context.
Method
KBLSTM retrieves WordNet and NELL concept embeddings and uses attention with a sentinel to select relevant knowledge or rely on the processed text.
Results
KBLSTM achieves state-of-the-art performance on both entity extraction and event extraction on the ACE2005 dataset.
Takeaways & Limitations
The architecture provides an approach for adaptively blending background KB knowledge with information from the currently processed text in recurrent machine reading.
Takeaways & Limitations
The evaluation covers entity and event extraction, while the model is described as potentially useful for other machine reading tasks and richer KBs as extensions.
Abstract
from arXiv · showhide
This paper focuses on how to take advantage of external knowledge bases (KBs) to improve recurrent neural networks for machine reading. Traditional methods that exploit knowledge from KBs encode knowledge as discrete indicator features. Not only do these features generalize poorly, but they require task-specific feature engineering to achieve good performance. We propose KBLSTM, a novel neural model that leverages continuous representations of KBs to enhance the learning of recurrent neural networks for machine reading. To effectively integrate background knowledge with information from the currently processed text, our model employs an attention mechanism with a sentinel to adaptively decide whether to attend to background knowledge and which information from KBs is useful. Experimental results show that our model achieves accuracies that surpass the previous state-of-the-art results for both entity extraction and event extraction on the widely used ACE2005 dataset.
1 Introduction
KBLSTM addresses the poor generalization and context-insensitivity of symbolic KB features by integrating continuous KB representations into BiLSTMs with adaptive attention. Evaluated on entity and event extraction, it achieves state-of-the-art performance on ACE2005.
- Motivation: Symbolic KB features can generalize poorly, become highly sparse, and mislead models when their usefulness varies across textual contexts.Examples include millions of WordNet indicator features and ambiguity such as “Clinton” referring to a person or town.
- Motivation: The paper asks whether recurrent neural networks can learn to adaptively leverage KB knowledge to improve machine reading.
- Method: KBLSTM extends BiLSTMs with retrieved KB concepts, attention over their semantic relevance, and a sentinel that decides whether to use background knowledge.
- Method: The model learns distributed embeddings for WordNet and NELL concepts and integrates them with BiLSTM state vectors.WordNet supplies lexical concepts and relations, while NELL stores beliefs about entities organized by semantic categories and relations.
- Evaluation: The model is evaluated on entity extraction and event extraction, including identifying entity mentions and event triggers such as elections, attacks, and travels.
- Results: Extensive experiments yield significant improvements over feature-based methods and knowledge-disregarding LSTMs, establishing new state-of-the-art results on ACE2005.
2 Related Work
Related work combines recurrent text models with distributed knowledge representations and attention, while prior connections often use shallow text representations. KBLSTM instead targets dynamic use of relevant knowledge in recurrent reading.
- Recurrent text models: LSTMs and GRUs help recurrent networks handle long-term dependencies by adaptively memorizing information for different durations.Bidirectional variants incorporate both past and future information and have supported several NLP tasks.
- Knowledge representations: Neural methods learn distributed representations of structured knowledge from large knowledge bases, making symbolic knowledge easier to use in statistical learning.
- Knowledge-text integration: Prior approaches connecting knowledge representations with text address knowledge base completion, relation extraction, and question answering, but often model text with triples or bag-of-words.
- Attention: Attention mechanisms motivate KBLSTM’s dynamic selection of knowledge relevant to the current textual context.
3 Method
KBLSTM extends BiLSTMs with continuous KB concept representations, attention, and a sentinel to integrate context-relevant external knowledge during reading. It retrieves candidate concepts from WordNet and NELL, learns their embeddings, and combines knowledge with recurrent states for prediction.
- Knowledge-aware BiLSTMs: At each time step, a knowledge module retrieves candidate concepts related to the current input and computes a knowledge state from their embeddings and the current context.The knowledge state integrates candidate concept vectors with the context vector through weighted combination.
- Knowledge-aware BiLSTMs: KBLSTM extends BiLSTMs by encoding KB concepts as continuous embeddings and integrating retrieved concepts into the recurrent state for knowledge-aware predictions.The model retrieves concepts related to the current word and combines their information with the BiLSTM state.
- Knowledge-aware BiLSTMs: Attention weights computed from the reading context determine the relevance of candidate KB concepts.Each candidate concept has an embedding, and a bilinear operator assigns its attention weight.
- Knowledge-aware BiLSTMs: A sentinel records current-context information and mixes it with KB-derived information, allowing the model to reduce reliance on misleading candidate concepts.The paper motivates this mechanism with cases where a KB assigns an incorrect or incomplete sense to an ambiguous word.
- Embedding Knowledge Base Concepts: Candidate concepts come from NELL entity matches and related categories, or from WordNet synsets associated with the current word.The paper treats WordNet synsets and NELL concept categories as KB concepts.
- Embedding Knowledge Base Concepts: Knowledge graph embeddings represent entities and relations in KB triples using a bilinear scoring function trained with a max-margin ranking objective.The embeddings are learned for WordNet and NELL concepts and their relations; the implementation also compares alternative embedding methods.
4 Experiments
The experiments evaluate KBLSTM on entity and event extraction using BiLSTM and feature-based baselines across ACE2005 and OntoNotes 5.0. KBLSTM-CRF achieves the best reported performance on ACE2005, while knowledge benefits are smaller on OntoNotes.
- Entity extraction: Entity extraction is performed in two stages: BiLSTM-CRF mention chunking followed by supervised chunk-level entity-type labeling.Experiments include both gold-standard and predicted mention boundaries.
- Entity extraction: KBLSTM-CRF significantly improves over BiLSTM-Fea-CRF on entity extraction with gold-standard mention boundaries.The CRF objective generally outperforms the softmax objective in this setting.
- Entity extraction: KBLSTM-CRF outperforms BiLSTM-Fea-CRF across all tested KB settings, with both KBs producing the best performance.The ablation compares different knowledge-base configurations.
- Entity extraction: KBLSTM-CRF achieves the best performance among compared models and new state-of-the-art results on ACE2005 entity extraction with predicted mention boundaries.The comparison includes feature-based linear models and sequence- and tree-structured LSTMs.
- Entity extraction: On OntoNotes 5.0, KBLSTM-CRF is competitive with prior results and significantly improves over BiLSTM and BiLSTM-Fea, but KB benefits are smaller than on ACE2005.The paper attributes this partly to lower NELL mention coverage in OntoNotes: 57% versus 65% in ACE2005, and to its larger training set.
- Event extraction: For event extraction on ACE2005, KBLSTM models significantly outperform BiLSTM and BiLSTM-Fea, and KBLSTM-CRF beats prior state-of-the-art methods without gold-standard entities.The event-extraction experiments integrate learned WordNet synset embeddings during training.
- Model analysis: Attention visualizations show that KBLSTM selects different KB word senses for “clinton” across contexts and can attend to relevant WordNet synsets for event-type disambiguation.The sentinel weight measures the tradeoff between knowledge and context.
5 Conclusion
KBLSTM incorporates background knowledge into recurrent neural networks through adaptive attention and a sentinel, achieving state-of-the-art performance on entity and event extraction benchmarks. The paper also identifies broader opportunities to integrate richer knowledge sources and apply the approach to other machine reading tasks.
- 5 Conclusion: KBLSTM blends knowledge-base information with processed text using adaptive attention and a sentinel to select relevant concepts and control the knowledge–context tradeoff.The mechanism can select appropriate semantic categories for ambiguous words such as “clinton” as a town or person.
- 5 Conclusion: State-of-the-art performance is reported for both entity extraction and event extraction on standard benchmarks.
- 5 Conclusion: The approach is evaluated on entity extraction and event extraction but may also be useful for other machine reading tasks.
- 5 Conclusion: The model could be extended to use a richer set of knowledge bases to capture more diverse and deeper background knowledge.