Source-linked AI summary
Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge
Todor Mihaylov, Anette Frank
TL;DR
The paper addresses cloze-style reading comprehension when the story omits information needed to infer an answer. It introduces a single-hop neural reader that retrieves and attends to external knowledge before combining it with context. External knowledge reduces relative error on Common Nouns and makes the reasoning evidence inspectable.
Problem
Cloze-style reading comprehension sometimes requires information beyond the story, motivating models that can use relevant external commonsense or factual knowledge.
Method
A two-module neural system retrieves relevant facts and uses attention to integrate selected external knowledge with story and question representations before answer selection.
Results
9% relative error-rate reduction on Common Nouns makes the single-hop model competitive with more complex reading-comprehension models.
Takeaways & Limitations
Explicit knowledge integration improves common-noun answering and enables analysis of the facts considered for specific questions.
Abstract
from arXiv · showhide
We introduce a neural reading comprehension model that integrates external commonsense knowledge, encoded as a key-value memory, in a cloze-style setting. Instead of relying only on document-to-question interaction or discrete features as in prior work, our model attends to relevant external knowledge and combines this knowledge with the context representation before inferring the answer. This allows the model to attract and imply knowledge from an external knowledge source that is not explicitly stated in the text, but that is relevant for inferring the answer. Our model improves results over a very strong baseline on a hard Common Nouns dataset, making it a strong competitor of much more complex models. By including knowledge explicitly, our model can also provide evidence about the background knowledge used in the RC process.
1 Introduction
The paper targets cloze-style reading comprehension by augmenting text-based representations with relevant external commonsense knowledge. Its model explicitly attends to and integrates that knowledge before predicting answers, while exposing the knowledge used.
- Cloze-style reading comprehension replaces a token in a story sentence with a placeholder that forms the question.
- The model addresses cases where most answer information appears in the story but additional information must be retrieved from a knowledge base.
- It encodes pre-selected commonsense or factual knowledge in memory and learns to enrich the context representation with it.
- Unlike approaches relying only on document-question interaction or discrete features across multiple hops, the model attends to selected external knowledge and combines it with context in one hop.
- The approach integrates knowledge into the Attention-Sum Reader and examines WordNet and ConceptNet for common-noun and named-entity questions.
- Knowledge facts can be added directly to text-only representations, and case studies and data statistics provide qualitative evidence of the injected knowledge’s effectiveness.
2 Reading Comprehension with Background Knowledge Sources
The system retrieves relevant background facts, encodes them in a knowledge memory, and combines attended knowledge with document and question representations for cloze-style answer selection.
- Knowledge Retrieval: The system uses Knowledge Retrieval to select facts relevant to the story, question, and candidate answers, then feeds them to the Knowledgeable Reader.Retrieved facts are selected from tokens appearing in the document, question, or answer candidates.
- Neural Model: The Knowledgeable Reader extends the Attention-Sum Reader with a pre-selected knowledge fact memory for single-hop reading comprehension.It ranks candidates using attention between the question-placeholder representation and candidate occurrences in the document.
- Knowledge Encoding: Each fact is encoded as a subject-relation-object triple, with a BiGRU preserving triple directionality and placing fact attributes in the same vector space as text tokens.The relation representation initializes the encoding of the object, filtering subject information through the relation type.
- Knowledge Retrieval: Key-Value retrieval attends over encoded facts for each document and question token, producing a weighted knowledge representation in the context-token vector space.Keys can be subject representations and values can be object representations; attention weights are normalized before summation.
- Context and Knowledge: The model combines plain context and context-plus-knowledge attentions because the task does not supervise whether external knowledge is needed.The learned scalar weights combine both representations before candidate scoring.
3 Data and Task Description
The experiments evaluate knowledge-enhanced cloze reading comprehension on the Common Nouns and Named Entities partitions of the Children’s Book Test using ConceptNet-derived commonsense facts.
- Dataset: The experiments use the Common Nouns and Named Entities partitions of the Children’s Book Test dataset.The dataset contains children’s-story contexts and cloze questions with candidate answers drawn from the document.
- Task: In the CBT task, a system reads 20 story sentences and predicts the placeholder in a 21st sentence from 10 candidate words.The Common Nouns setup is difficult: reported human accuracy without the full context is 64.4%, while a language model alone reaches 57.7%.
- Task: Other cloze-style datasets such as CNN/Daily Mail and Who-DidWhat focus mainly on named entities, where commonsense knowledge may offer more limited benefit.This motivates evaluating the Common Nouns partition alongside Named Entities.
- Knowledge Source: The study uses Open Mind Common Sense from ConceptNet 5.0, containing about 630k fact triples, with CN5All, CN5WN3, and CN5Sel experimental subsets.CN5WN3 contains 213k triples, while CN5Sel excludes selected WordNet relations.
4 Related Work
Prior work includes cloze models, neural systems that integrate background knowledge, and rare-entity prediction; this paper differs by explicitly attending to unordered knowledge facts for each context token.
- Cloze-Style Reading Comprehension: Cloze-style reading comprehension has progressed from MCTest to large-scale datasets such as CNN/Daily Mail and the Children’s Book Test.The related work contrasts simple single-turn models with more complex reading-comprehension approaches.
- Integrating Background Knowledge: Earlier neural systems incorporated background knowledge for tasks including recipe generation, language modeling, event extraction, and entity extraction.Examples use ingredient lists, retrieved information, or knowledge embeddings from WordNet and NELL.
- Related Cloze Tasks: Rare Entity Prediction uses paragraph context, entity descriptions derived from Freebase, and pre-selected candidates to fill missing entities.The task is superficially similar to cloze-style reading comprehension but differs substantially in its setting.
- Knowledge Integration: Compared with Weissenborn et al., this model uses attention over unordered ConceptNet fact triples and integrates relevant knowledge explicitly for each context token.The cited comparison states that the other model reads knowledge sequentially and transfers it implicitly by refining document and question word embeddings.
5 Experiments and Results
The experiments evaluate knowledge sources, memory size, key-value strategies, and context interactions for KnReader. Results show that combining context and knowledge interactions improves performance, especially on Common Nouns, while ConceptNet configurations and retrieval strategies vary across datasets.
- Experimental setup: The experiments compare knowledge sources, memory sizes, key-value strategies, and combinations of context and context-plus-knowledge interactions.Accuracy is reported on Dev and Test sets, with Dev results used to prune experiments.
- Knowledge sources: CN5Sel performs best on the Dev set, whereas CN5WN3 performs much better on Test; subsequent experiments use CN5Sel.These results are reported for the CBT Common Nouns dataset with the Full model and 50 facts.
- Number of facts: 50 facts produce the best Dev result on Common Nouns, so this memory size is used in subsequent experiments.The comparison uses the Full model with CN5Sel.
- Component ablations: Combining interactions between ctx and ctx+kn yields clear improvement over the no-knowledge setup, particularly on Common Nouns.The ensemble combines attention interactions between document and question representations with and without injected knowledge.
- Key-value selection: Subj/Obj performs slightly better than Obj/Obj on Common Nouns, while the two strategies perform equally well on Named Entities Dev and Subj/Obj is slightly better on Named Entities Test.Subj/Obj attends over fact subjects and uses weighted fact objects as values.
- Comparison to previous work: KnReader clearly outperforms prior single-hop models on both datasets, although it does not improve over the state of the art and remains competitive with multi-hop models.The comparison uses settings selected from the Dev sets for the Named Entities and Common Nouns datasets.
6 Discussion and Analysis
The analysis examines how knowledge changes predictions and how the model uses retrieved facts. Knowledge helps correct frequency-biased answers, supports non-trivial semantic links, and improves Common Nouns more than Named Entities.
- Empirical results: Knowledge provides only small improvements on Named Entities, while the single-hop model remains competitive with most multi-hop neural architectures.The analysis treats Common Nouns as the clearest setting for measuring the value of injected knowledge.
- Empirical results: Knowledge clearly helps the Common Nouns task, but different knowledge sources affect Dev and Test performance differently.The analysis attributes this variation to sensitivity of the model or data subsets to knowledge types and retrieved knowledge.
- Component analysis: The contribution ranking from attention-based prediction changes matches the component-importance ablation ranking on the Dev set.The reported ordering is Dctx, Qctx+kn > Dctx+kn, Qctx > Dctx+kn, Qctx+kn.
- Case analysis: Enriching only the question is crucial when the question suffices, whereas enriching both question and document is required when the story context is needed.This distinction is reported across human and machine reading-comprehension cases.
- Case analysis: When appropriate knowledge is selected, the model can correct a frequency bias toward an incorrect answer and predict infrequent candidates.The analysis reports this behavior in several cases, including one where relevant CN5 knowledge supports an infrequent candidate.
- Knowledge limitations: The analysis identifies missing event knowledge, such as take off versus put on clothes and climb up, as a source of limited coverage.Relevant CN5 facts can nevertheless help predict infrequent candidates in some cases.
- Knowledge interpretation: Fact-context encoding lets the model identify semantically related knowledge that is not surface-near to the question or document, including paraphrases such as undressed–naked.The analysis also describes a fact such as ear PartOf head receiving high question attention and supporting candidate head.
7 Conclusion and Future Work
The paper concludes that explicitly integrating external commonsense knowledge into a single-turn cloze-style reader improves Common Nouns performance and makes the model competitive with more complex systems. Attention over injected facts also provides interpretable evidence and supports adaptation to other paired-input tasks.
- Conclusion: A single-turn neural cloze-style reader with external commonsense knowledge achieves a 9% relative error-rate reduction on Common Nouns.The model competes with more complex reading-comprehension models, and ConceptNet knowledge is useful.
- Conclusion: Explicit knowledge integration makes the model transparent by allowing the facts considered during prediction to be traced and analyzed.The paper presents quantitative and qualitative evidence for the effectiveness of selecting relevant knowledge.
- Future work: Because background knowledge is integrated with document and question representations, the approach can be adapted to paired-input settings such as entailment and question answering.The paper identifies investigating knowledge sources needed for different datasets and domains as future work.
A Model and Implementation Details
The model encodes retrieved knowledge triples with a BiGRU into a separate memory for each instance. This encoding preserves shared token representations, triple directionality, and relation-guided information flow.
- Knowledge Encoding: Each retrieved fact is represented as a subject–relation–object triple, with multi-word subject and object expressions and a single-token relation.The dataset produces a separate knowledge memory for each instance.
- Knowledge Encoding: A BiGRU encodes the triple attributes into context-aware representations used in the knowledge memory.The final hidden states initialize encoding of the next triple attribute in left-to-right order.
- Knowledge Encoding: The relation is encoded as a single relation-type word, such as /r/IsUsedFor.This relation representation participates in the left-to-right encoding of each fact.
- Knowledge Encoding: The encoding places knowledge attributes in the same vector space as plain tokens while preserving triple directionality.The relation type also filters subject information used to initialize the object representation.
Model Implementation Parameters
The implementation uses ConceptNet commonsense facts, BiGRU-based representations, pretrained word embeddings, and fixed training and optimization settings for the CBT experiments.
- Data and Preprocessing: Experiments use Common Nouns and Named Entities subsets of the Children’s Book Test, with each instance containing 20 story sentences and one question sentence.The story tokens are concatenated into a single document-token list after preprocessing.
- Knowledge Source: The knowledge source is the OMCS portion of ConceptNet, a crowd-sourced commonsense resource containing approximately 630k facts.The vocabulary is extended with words retrieved from the knowledge source.
- Embeddings: The model uses 100-dimensional GloVe embeddings pretrained on 6B Wikipedia and Gigaword5 tokens, with out-of-vocabulary initialization in [−0.1, 0.1].Embeddings are optimized during the first 8000 training steps.
- Encoder: The GRU encoder uses hidden size 256, producing 512-dimensional bidirectional outputs.This setting follows a configuration reported to work well for the Attention Sum Reader.
- Training: Training uses batches of 64, 60 epochs, validation every 1000 steps, and the highest-validation-accuracy model for Test predictions.Optimization uses cross-entropy loss, Adam with learning rate 0.001, and gradient clipping to [−10, 10].
Additional Ablation Experiments
Additional experiments examine larger knowledge memories, ensemble prediction, and comparisons with multi-hop and re-ranking models.
- Ablation Experiments: Using 100 facts produces a similar tendency to the 50-fact experiments, but omitting knowledge enrichment gives the best Common Nouns results in this setting.These are additional ablation results reported for different model-component interactions.
- Ensembling: The Ensemble model combines the best 11 runs for each dataset and predicts answers by majority voting.This procedure is used to produce the ensemble result.
- Model Comparisons: The comparison with multi-hop models reports Dev and Test Accuracy rounded to one decimal place, while noting that most neural models are not directly comparable.AoA Reader uses re-ranking as a post-processing step.
C Manual Analysis and Visualization
Manual analyses visualize how retrieved knowledge changes candidate scores and document–question interactions. The cases show both useful corrections and harmful reversals, often reflecting missing event or location knowledge.
- Case 1: Knowledge-enhanced attention can raise the correct answer’s score by adding weighted object representations and related facts to the question or document context.In the example, knowledge about head is approximated through a retrieved ear–head fact and related animal and bird facts.
- Case 2: A retrieved clothes–undressed fact helps reverse a text-only cloth prediction when the story describes the king’s nonexistent clothes.The example cannot be answered from story context alone because the relevant clothes are imaginary.
- Case 3: Knowledge can overcome story frequency bias, as the model selects father over magician in one example.The story contains four occurrences of magician, yet the model chooses the more plausible answer using a father-related fact.
- Case 4: Adding retrieved knowledge can harm prediction when the model cannot combine a relevant people–help fact with the question context.In this case, the knowledge-enhanced model reverses a correct initial prediction and selects an incorrect answer.
- Case 5: Location knowledge can also reverse a correct cliff prediction to sea, while the authors identify missing knowledge about actions such as climbing steep locations.The authors plan to explore sources containing more information about events.