Source-linked AI summary
Key-Value Memory Networks for Directly Reading Documents
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, Jason Weston
TL;DR
The paper tackles the difficulty of answering questions directly from documents, whose information is less structured than KBs even though KBs are incomplete and schema-limited. It introduces Key-Value Memory Networks and the WikiMovies comparison tool, finding reduced gaps across knowledge-source settings and state-of-the-art WikiQA results.
Problem
KBs are effective for QA but incomplete and schema-limited, while directly reading documents is harder because information is less structured, ambiguous, and scattered.
Method
Key-Value Memory Networks store facts as key-value pairs, using keys to address relevant memories and values to predict answers across KB and document knowledge sources.
Results
KV-MemNN reduces the gap among human-annotated KB, automatically extracted KB, and Wikipedia settings on WikiMovies, and achieves state-of-the-art results on WikiQA.
Takeaways & Limitations
WikiMovies enables comparison of QA across KBs, extracted KBs, and raw documents, while key-value memory supports reading documents or KBs for question answering.
Abstract
from arXiv · showhide
Directly reading documents and being able to answer questions from them is an unsolved challenge. To avoid its inherent difficulty, question answering (QA) has been directed towards using Knowledge Bases (KBs) instead, which has proven effective. Unfortunately KBs often suffer from being too restrictive, as the schema cannot support certain types of answers, and too sparse, e.g. Wikipedia contains much more information than Freebase. In this work we introduce a new method, Key-Value Memory Networks, that makes reading documents more viable by utilizing different encodings in the addressing and output stages of the memory read operation. To compare using KBs, information extraction or Wikipedia documents directly in a single framework we construct an analysis tool, WikiMovies, a QA dataset that contains raw text alongside a preprocessed KB, in the domain of movies. Our method reduces the gap between all three settings. It also achieves state-of-the-art results on the existing WikiQA benchmark.
1 Introduction
The paper addresses the difficulty of answering questions directly from unstructured documents, where KBs are incomplete and schema-limited but raw text contains more information. It introduces WikiMovies and Key-Value Memory Networks to compare and improve document- and KB-based QA.
- KBs have incomplete information and fixed schemas that cannot support all varieties of answers, while raw textual resources contain more information.
- WikiMovies provides approximately 100k movie-domain questions answerable from a perfect KB, Wikipedia pages, or an automatically extracted KB.
- Key-Value Memory Networks store facts as key-value pairs, using keys to address memories relevant to the question and values to predict answers.
- The architecture supports either knowledge source while allowing learned transformations between keys and values through standard backpropagation.
- KV-MemNN consistently outperforms the original Memory Network on WikiMovies and reduces the gap among human-annotated KB, extracted KB, and Wikipedia settings.
- On WikiQA, KV-MemNN reaches state-of-the-art results and surpasses the most recent attention-based neural network models.
2 Related Work
Earlier QA emphasized structured KBs and semantic parsing, but KB sparsity shifted attention toward information extraction and direct text-based QA. Existing text benchmarks and attention-based neural models address this setting, while KV-MemNN introduces structured key-value memory for greater flexibility.
- Semantic-parsing QA systems use large-scale KBs to return precise answers to complicated compositional questions.
- KB sparsity shifts the challenge toward developing efficient information-extraction methods to populate KBs automatically.
- TRECQA and WikiQA return to direct text-based QA, but both benchmarks focus on answer sentence selection rather than returning the actual answer.
- The best published TRECQA and WikiQA results came from convolutional or recurrent neural networks, usually with attention mechanisms.
- KV-MemNNs operate symbolic memories structured as key-value pairs, a structure not employed in existing attention-based neural QA architectures.
- The structured memory gives the model greater flexibility for encoding knowledge sources and helps shrink the gap between document reading and KB-based QA.
3 Key-Value Memory Networks
Key-Value Memory Networks separate memory addressing from value reading, enabling different representations for matching questions and producing answers. The model iteratively retrieves and combines evidence, supporting knowledge-base and document-based question answering.
- Model Description: KV-MemNNs store each memory as a key-value pair, using keys to address relevant memories and values to produce the returned result.This separates question-matching features from answer-producing features.
- Model Description: The model uses a controller and multiple hops, updating the query with retrieved context before repeating memory access.Each hop uses a different transformation matrix, and hashing is performed only initially.
- Model Description: Key hashing preselects memories sharing question words, key addressing assigns relevance probabilities, and value reading computes their weighted sum.The hashing threshold ignores words with frequency at least F = 1000.
- Model Description: After a fixed number H of hops, the controller predicts an answer from candidate outputs and is trained end-to-end with cross-entropy loss.Candidate outputs may be entities in a knowledge base or answer sentences in WIKIQA.
- Key-Value Memories: Other representations include KB triples, sentence-level memories, and window-title pairs, with title encoding limited to datasets having meaningful document titles.KB triples use subject-relation keys and object values, including reversed relations for different question directions.
- Key-Value Memories: Window-level document memories encode the entire word window as the key and its center entity as the value, aligning question matching with answer selection.Window-plus-center encoding further distinguishes the center from surrounding words.
4 The WikiMovies Benchmark
WIKIMOVIES is a movie-domain QA benchmark designed to compare raw documents, human-annotated KBs, and IE-derived KBs under controlled answerability. It supports analysis across question types and knowledge representations.
- WIKIMOVIES contains movie-domain question-answer pairs designed to provide ample training examples and enable performance breakdowns by question type.
- The benchmark compares three knowledge sources: raw Wikipedia documents, a classical KB built from OMDb and MovieLens, and an IE-based KB extracted from Wikipedia.The IE representation uses precise, compact subject-verb-object facts but may lose recall when triplets are malformed or missing.
- Questions are constructed to be potentially answerable from either the KB or the original Wikipedia documents, reducing data-sparsity differences between sources.This design supports direct analysis of methods for closing the gap between document reading and structured knowledge sources.
- The document source comprises approximately 17k Wikipedia movie articles, retaining each article’s title and first section before the contents box.
- The KB contains nine relation types and approximately 43k entities, with IMDb ratings and votes discretized into textual categories.The relations include director, writer, actor, release year, language, genre, tags, IMDb rating, and IMDb votes.
- Question classes range from specific actor–movie and movie–actor relations to broader tag–movie relations, and some questions have multiple correct answers.
5 Experiments
The experiments compare QA from KBs, information-extracted KBs, and Wikipedia documents using several learning methods, with KV-MemNNs evaluated on WIKIMOVIES and WIKIQA. Results show KV-MemNNs perform strongly across sources and benchmarks, while document structure remains important.
- WIKIMOVIES: KV-MemNNs outperform all other methods on KB, IE, and Wikipedia document sources in WIKIMOVIES.
- WIKIMOVIES: 93.9 vs. 76.2: directly reading Wikipedia outperforms IE-based KBs but remains below a human-annotated KB.
- WIKIMOVIES: Window-level + Center Encoding + Title performs best for document reading, while sentence-level representations are inferior.The best setting uses W = 7 and H = 2; center encoding and title features both help window-level representations.
- Question-type breakdown: IE especially loses to Doc and KB on Writer, Director, and Actor to Movie questions, where coreference may be difficult.Only 56% of subject-object pairs in IE match the original KB triples, so information-extraction losses are expected.
- Synthetic document analysis: 93.9% for KB versus 82.9% for One Template Sentence: representing facts as sentences directly causes part of the performance loss.The remaining drop is split roughly equally between conjunctions at 74% and coreference at 76%; the hardest synthetic setting reaches 72.5% versus 76.2% for real Wikipedia.
- WIKIQA: KV-MemNNs outperform a large set of methods on WIKIQA, with results very similar to the L.D.C. method.Memory Networks perform much worse, highlighting the importance of pairing windows with sentences through key-value memories.
6 Conclusion
The paper studies the performance gap between directly reading documents and using human-annotated or automatically constructed KBs. It presents Key-Value Memory Networks, which outperform several methods across WIKIMOVIES and WIKIQA, while some gap remains.
- The paper analyzes the gap between direct document reading and human-annotated or automatically constructed KBs.
- Key-Value Memory Networks bridge this gap and outperform several other methods across WIKIMOVIES and WIKIQA.
- Some performance gap remains, motivating future work to close it further.
- KV-MemNNs can read documents or KBs while encoding task-specific prior knowledge in separate key and value memories.