Source-linked AI summary
Dynamic Integration of Background Knowledge in Neural NLU Systems
Dirk Weissenborn, Tomáš Kočiský, Chris Dyer
TL;DR
Static training corpora may not capture all useful background knowledge and can become outdated as facts change. The paper dynamically integrates retrieved external knowledge as text, refining word representations before task-specific NLU models process the inputs. Results show improvements across benchmarks, a new state of the art on TriviaQA, and semantically appropriate knowledge use.
Problem
Neural NLU systems typically acquire background knowledge implicitly from training corpora, which may not capture all relevant information and is difficult to keep current as facts change.
Source: §1 · p. 1Method
A jointly trained reading module reads task inputs and retrieved supplementary text to refine word embeddings for a task-specific NLU architecture.
Source: §1 · p. 1Results
Embedding refinement helps across four benchmarks, reaches a new state of the art on TriviaQA, and the analysis finds semantically appropriate knowledge use.
Source: §1 · p. 2Takeaways & Limitations
The reading architecture can augment virtually any NLU architecture that takes word embeddings, and simple task models can become competitive with task-specific state-of-the-art systems.
Source: §7 · p. 8The approach assumes supplementary knowledge is available as natural-language statements or can be recoded into that form.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Common-sense and background knowledge is required to understand natural language, but in most neural natural language understanding (NLU) systems, this knowledge must be acquired from training corpora during learning, and then it is static at test time. We introduce a new architecture for the dynamic integration of explicit background knowledge in NLU models. A general-purpose reading module reads background knowledge in the form of free-text statements (together with task-specific text inputs) and yields refined word representations to a task-specific NLU architecture that reprocesses the task inputs with these representations. Experiments on document question answering (DQA) and recognizing textual entailment (RTE) demonstrate the effectiveness and flexibility of the approach. Analysis shows that our model learns to exploit knowledge in a semantically appropriate way.
1 Introduction
Background knowledge is difficult to capture completely in static training corpora and keep current as facts change. The paper proposes reading external knowledge alongside task inputs to refine word representations, and reports gains across four benchmarks alongside semantically appropriate knowledge use.
1 Introduction
- Static training corpora may omit useful background knowledge and become outdated as relevant facts change, while building and maintaining suitably large corpora is described as impractical.The paper contrasts this constraint with dynamic incorporation of external knowledge.Source: §1 · p. 1
- The architecture reads retrieved supplementary text with task inputs, produces contextually refined word embeddings, and passes them to a task-specific NLU model trained jointly end-to-end.The proposed reading module is intended to work with any task architecture that reads sequences of word embeddings.Source: §1 · p. 1
- Embedding refinement improves performance on four benchmarks, with a new state of the art on TriviaQA and evidence of semantically appropriate knowledge use.The authors also report that adding background knowledge helps beyond refining embeddings using task context alone.Source: §1 · p. 2
- The authors report analysis showing that the model uses background knowledge semantically appropriately, including appropriate counterfactual inferences from alternative facts.They also describe evidence from information about candidate answers in DQA that the system uses heterogeneous knowledge sources sensibly.Source: §1 · p. 2
2 External Knowledge as Supplementary Text Inputs
The paper treats supplementary knowledge as text that can be read alongside the task input, whether it originates as natural language or is recoded from structured sources. It focuses on integrating retrieved information rather than learning which information to retrieve.
2 External Knowledge as Supplementary Text Inputs
- The approach assumes supplementary information is available as natural-language statements or can be recoded into that form, enabling use of both structured facts and unstructured sources such as encyclopedias.The paper gives converting a structured triple into a simple textual assertion as an example.Source: §2 · p. 2
- Selecting contextually relevant supplementary knowledge is left as an open question because retrieving and preparing it is a complex research topic.The paper notes statistical, neural, and reinforcement-learning approaches to retrieval, citing Manning et al. (2008), Mitra and Craswell (2017), and Nogueira and Cho (2017).Source: §2 · p. 2
- Rather than jointly learning relevance selection and integration, the study uses heuristic retrieval and concentrates on the integration model.Source: §2 · p. 2
3 Refining Word Embeddings by Reading
The paper refines word embeddings by reading task inputs and supplementary text in successive steps, so task architectures can consume representations informed by both. The process starts from uncontextualized embeddings, reads each text sequence, pools information across matching lemmas, and gates updates to the representations.
3 Refining Word Embeddings by Reading
- The general strategy replaces static embeddings with representations that depend on task and supplementary inputs, while remaining transparent to task architectures that use word embeddings.At each reading step, the previous embedding matrix is refined using specified text sequences.Source: §3 · p. 2
3.2 Refined Word Embeddings (Eℓ, ℓ≥1)
- At each refinement step, a BiLSTM reads text sequences embedded with the previous matrix, and the model pools representations at occurrences matching each word's lemma before a gated update.Lemmatization allows tokens of the same lemma to interact; pooling across passages can connect otherwise separate texts and mitigate long-distance dependency problems.Source: §3 · p. 2, §3.2 · p. 4
3.1 Unrefined Word Embeddings (E0)
- Unrefined representations begin as non-contextual word embeddings built from fixed pretrained vectors, combined using a gated operation and a convolutional network with max pooling.The supplied passages specify a single-layer convolutional neural network with n width-5 filters.Source: §3.1 · p. 3
4 Experimental Setup
The experiments evaluate embedding refinement on document question answering and textual entailment using four benchmarks and mostly simple task architectures. Supplementary knowledge is retrieved heuristically from ConceptNet and, for DQA, Wikipedia abstracts.
4 Experimental Setup
- Most experiments use single-layer BiLSTM encoders with feed-forward prediction layers, while the RTE experiments also add refinement to an ESIM reimplementation.Models are trained end-to-end with the refinement module; the TriviaQA embedding dimensionality is reduced to 150 for memory constraints, versus 300 otherwise.Source: §4 · p. 4
- The setup covers SQuAD and TriviaQA for document question answering, and SNLI and MultiNLI for recognizing textual entailment.TriviaQA includes a large, noisy distant-supervision test set and a much smaller human-verified set; results are reported on both.Source: §4 · pp. 4–5
- ConceptNet assertions are retrieved when their subject and object connect words or phrases across the task's two texts, then ranked and truncated to a task-specific top-k.The retained limits are k = 50 for DQA and k = 20 for RTE, and some examples yield fewer or no assertions.Source: §4 · p. 5
- For DQA, the model first predicts 16 answer spans without Wikipedia, then retrieves abstracts for the three most frequently linked Wikipedia entries per answer string and reads their initial 50 tokens.Using multiple linked entries is intended to mitigate ambiguity from polysemous entity names, while requiring the refinement model to select relevant information.Source: §4 · p. 5
5 Results
Across DQA and RTE, refinement-based systems are competitive, with the full TriviaQA model slightly surpassing current state of the art. Qualitative tests indicate that models respond to assertion semantics and sometimes use retrieved answer information appropriately.
5 Results
- Final systems are competitive on TriviaQA and MultiNLI; Wikipedia slightly outperforms TriviaQA state of the art, while ConceptNet substantially helps BiLSTM RTE but ESIM mainly on MultiNLI.The results section also reports competitive SNLI results and that refinement often helps even without external knowledge.Source: §5 · p. 5, Table 2 · p. 6, §5.1 · p. 6
5.1 Question Answering
- A two-layer BiLSTM with similar computational complexity and parameter count did not improve performance, supporting contextual pooling as a simple refinement technique.The comparison controlled for the possibility that refinement gains came from additional computation over the inputs.Source: §5.1 · p. 6
5.2 Recognizing Textual Entailment
- The ESIM reimplementation fell 0.8% short of its reported 88% result on SNLI but performed about 5% better on MultiNLI; this instability suggests possible SNLI overfitting.The authors note that the results may indicate current state-of-the-art RTE models are overfit to SNLI.Source: §5.2 · p. 7
5.3 Qualitative Analysis
- Swapping synonym and antonym predicates in MultiNLI assertions caused about a 10% performance drop when either type was present, showing sensitivity to assertion semantics.The authors report the drop for both BiLSTM and ESIM models and describe prediction changes as appropriate counterfactual inferences.Source: §5.3 · p. 7
- Across 25 DQA answer changes after adding Wikipedia abstracts, 14 corrected an incorrect answer and 11 changed a correct answer to an incorrect one.In 11 of the 14 corrections, the abstracts contained obvious information reinforcing the correct answer; unrelated information was present in 8 of the 11 harmful changes.Source: §5.3 · p. 7
- The authors interpret counterfactual RTE inferences and DQA’s use of reinforcing candidate-answer information as evidence of semantically sensible use of heterogeneous knowledge sources.The DQA examples also reveal cases where the model struggled with unrelated or plausibly relevant background information.Source: §5.3 · pp. 7–8
6 Related Work
Prior neural NLU work used external knowledge in task- or representation-specific ways, while this paper presents a general-purpose reader for heterogeneous knowledge expressed in natural-language statements. It also situates dynamic word-representation updates among task-specific memory approaches and pre-trained contextualized representations.
6 Related Work
- Earlier neural NLU systems had begun using background knowledge, but prior efforts targeted particular tasks or knowledge types [Ahn et al., 2016; Dhingra et al., 2017].The role of background knowledge had long been discussed in classical AI.Source: §6 · p. 8
- The proposed general-purpose reading architecture handles heterogeneous, fine-grained concept assertions encoded as simple natural-language statements.Examples include statements such as “abdication is a type of resignation.”Source: §6 · p. 8
- Other work reads word definitions or incorporates word-sense information before downstream NLU, whereas this approach integrates varied fine-grained assertions about relevant concepts.The passage describes definition-reading approaches by Bahdanau et al. (2017) and Long et al. (2017), and word-sense representations by Pilehvar et al. (2017).Source: §6 · p. 8
- Dynamic memories often track concepts, entities, or sentences for particular tasks, while this approach is task-agnostic and permits external background knowledge.Pre-trained contextualized token representations are another related line of work [Peters et al., 2017; McCann et al., 2017].Source: §6 · p. 8
7 Conclusion
The paper concludes that reading supplementary text to refine word representations enables dynamic background-knowledge integration across neural NLU architectures. Its results and analysis indicate improved performance and semantically appropriate use of the supplied knowledge.
7 Conclusion
- Incrementally refining word representations from supplementary inputs offers a flexible way to add dynamic background knowledge to NLU architectures that use word embeddings.The authors say the approach can be used with virtually any such existing architecture.Source: §7 · p. 8
- Refinement using task inputs and supplementary external text can yield large improvements, and simple BiLSTM task architectures can become competitive with task-specific state-of-the-art systems.Source: §7 · p. 8
- The analysis finds that the model learns to exploit provided background knowledge in a semantically appropriate way.Source: §7 · p. 8
A Implementation Details
The implementation details report the use of three random seeds and describe the initialization of the word-representation update gate and fully connected layers.
A Implementation Details
- Models were trained with three random seeds, and the top performance was reported.The passage directs readers to Table 5 for hyperparameters and introduces the task-specific baseline implementations.Source: A Implementation Details · p. 11
- The input sequences q and p are embedded as Q and P, respectively, and the fully connected layer is defined as FC(z) = Wz + b.The passage defines the hidden dimensionality as n and gives W and b dimensions.Source: A Implementation Details · p. 11
- The word-representation update gate starts with bias 1 so that representations are refined only slightly at the beginning of training.Source: A Implementation Details · p. 11
A.1 Question Answering
For question answering, the implementation encodes question and passage sequences with BiLSTMs and predicts answer spans, with a multi-paragraph setup for TriviaQA. For textual entailment, it uses conditional encoding and predicts among entailment, contradiction, and neutral.
A.1 Question Answering
- For QA, identical parallel BiLSTMs encode the question and passage, followed by a linear projection and tanh; the projection matrix U is initialized as [I; I].Source: §A.1 · p. 11
- The QA model forms a weighted question representation and predicts answer-start and answer-end probabilities with a two-layer MLP.Training maximizes correct-span likelihood; evaluation selects the best-scoring span, capped at 16 tokens for SQuAD and 8 for TriviaQA.Source: §A.1 · p. 11
- Reported result variations were less than one percentage point across experiments.Source: §A.1 · p. 11
- TriviaQA documents are split and merged into paragraphs capped at 600 tokens, ranked by question-paragraph TF-IDF similarity, with the top six retained for prediction.Training samples two paragraphs from the top four or six for the Web or Wikipedia datasets; prediction normalizes over tokens across the retained paragraphs.Source: §A.1 · p. 11
- The table lists ADAM optimization, 300-dimensional GloVe embeddings, and dropout settings, with additional dropout on QA projections.The learning rate is 10^-3 and is halved when performance drops between checkpoint intervals.Source: Table 5 · p. 12
A.2 Recognizing Textual Entailment
- For RTE, hypothesis BiLSTM end states initialize the premise BiLSTM; premise outputs are transformed, max-pooled, and used to predict entailment, contradiction, or neutral.The model is trained to maximize the likelihood of the correct category.Source: §A.2 · p. 12
B Reducing Training Data & Dimensionality of Pre-trained Word Embeddings
The authors hypothesize that external knowledge matters more when training data and pretrained embeddings are reduced. In those settings, contextual refinement helps most overall, while ConceptNet offers a slightly larger benefit in the most impoverished conditions.
B Reducing Training Data & Dimensionality of Pre-trained Word Embeddings
- The authors hypothesized that external knowledge would help more as training data and pretrained word representations were reduced.They expected ESIM’s attention and pretrained representations to recover some lexical relations, with less opportunity for recovery under impoverished settings.Source: B Reducing Training Data &… · p. 12
- Contextual refinement produced the biggest overall impact over baseline ESIM, especially with 1k and 3k training examples.The benefit from ConceptNet was slightly larger in more impoverished settings, with its largest improvements around 10k examples and dimensionality reduced to 10.Source: B Reducing Training Data &… · p. 12
- Jointly reducing training examples and embedding dimensionality produced a slightly larger ConceptNet benefit in more impoverished settings.Reducing either data or dimensionality alone showed similar but less pronounced results.Source: B Reducing Training Data &… · p. 12
C Further Analysis of Knowledge Utilization in RTE
Removing knowledge at test time reduced performance on MultiNLI and SQuAD, indicating that the model usefully refines representations with supplied assertions. The analysis finds sensitivity varies by relation type and task: related-to assertions have the largest total impact, while antonym and synonym assertions have the strongest normalized impact across tasks.
C Further Analysis of Knowledge Utilization in RTE
- Withholding knowledge at test time reduced MultiNLI accuracy by 3.7–3.9% and SQuAD F1 by 4%.The authors interpret this as evidence that provided assertions usefully refine representations.Source: C Further Analysis of Knowledge… · p. 12
- Related-to assertions caused the largest total performance drop, while antonym and synonym assertions had the greatest normalized impact across tasks.The normalized drops are measured only on examples containing the respective relation; is-a and derived-from assertions also mattered for some tasks.Source: C Further Analysis of Knowledge… · p. 13, Figure 2 · p. 13
- The authors suggest related-to assertions help connect the two input sequences and bridge long-range dependencies, similarly to attention.They note that related-to is a prominent predicate that appears much more frequently than other assertions.Source: C Further Analysis of Knowledge… · p. 13