Source-linked AI summary
Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph
Amrita Saha, Vardaan Pahuja, Mitesh M. Khapra, Karthik Sankaranarayanan, Sarath Chandar
TL;DR
Existing QA and dialog systems are studied largely independently, despite chatbot scenarios combining context-aware conversation with factual reasoning over large knowledge graphs. The paper introduces CSQA and builds linked QA conversations through crowdsourced and in-house annotation, template modification, and semi-automatic instantiation. Its hybrid dialog-and-QA model is inadequate for CSQA, highlighting the need for further research.
Problem
Existing QA and dialog systems are studied largely independently, despite chatbot scenarios combining context-aware conversation with factual reasoning over large knowledge graphs.
Method
The paper introduces CSQA and builds linked QA conversations through crowdsourced and in-house annotation, template modification, and semi-automatic instantiation.
Results
The hybrid dialog-and-QA model is inadequate for CSQA, whose questions require complex reasoning and conversational context.
Takeaways & Limitations
CSQA provides a resource for studying complex sequential QA and highlights the need for further research beyond existing models.
Takeaways & Limitations
Generic Wikidata relations and overlapping types leave some generated questions unnatural or repetitive despite filtering.
Abstract
from arXiv · showhide
While conversing with chatbots, humans typically tend to ask many questions, a significant portion of which can be answered by referring to large-scale knowledge graphs (KG). While Question Answering (QA) and dialog systems have been studied independently, there is a need to study them closely to evaluate such real-world scenarios faced by bots involving both these tasks. Towards this end, we introduce the task of Complex Sequential QA which combines the two tasks of (i) answering factual questions through complex inferencing over a realistic-sized KG of millions of entities, and (ii) learning to converse through a series of coherently linked QA pairs. Through a labor intensive semi-automatic process, involving in-house and crowdsourced workers, we created a dataset containing around 200K dialogs with a total of 1.6M turns. Further, unlike existing large scale QA datasets which contain simple questions that can be answered from a single tuple, the questions in our dialogs require a larger subgraph of the KG. Specifically, our dataset has questions which require logical, quantitative, and comparative reasoning as well as their combinations. This calls for models which can: (i) parse complex natural language questions, (ii) use conversation context to resolve coreferences and ellipsis in utterances, (iii) ask for clarifications for ambiguous queries, and finally (iv) retrieve relevant subgraphs of the KG to answer such questions. However, our experiments with a combination of state of the art dialog and QA models show that they clearly do not achieve the above objectives and are inadequate for dealing with such complex real world settings. We believe that this new dataset coupled with the limitations of existing models as reported in this paper should encourage further research in Complex Sequential QA.
Introduction
CSQA targets chatbot conversations that combine context-aware factual QA with complex reasoning over large knowledge graphs. The paper introduces a dataset and hybrid model to expose gaps in existing QA and dialog approaches.
- Large knowledge graphs could support context-aware factual question answering in domain-specific chatbot conversations.
- CSQA conversations link questions through shared predicates or entities, requiring the system to use prior turns for contextual interpretation.The example requires retaining a predicate while changing the subject in a later question.
- Unlike simple QA, CSQA questions may require logical, quantitative, or comparative reasoning over larger knowledge-graph subgraphs.For example, answering which rivers flow through India and China requires constructing two sets and intersecting them.
- The dataset-creation process combines crowdsourced question generation with in-house conversational templates and challenge-oriented modifications.
- The paper proposes a CSQA model combining a hierarchical conversation model with a key-value memory network and uses it to demonstrate inadequacies in current methods.The stated objectives are to introduce the task and show that existing state-of-the-art QA and dialog methods are insufficient.
Related Work
Prior QA and dialog datasets address related tasks separately, but existing resources do not combine linked conversational QA with explicit knowledge-graph reasoning. CSQA therefore extends beyond single-tuple factoid questions.
- Existing QA research spans document-based, structured-KG, reading-comprehension, cloze, and multiple-choice tasks.
- Existing factoid QA datasets mainly contain Simple Questions answerable from a single knowledge-graph tuple.
- CSQA instead requires logical, quantitative, and comparative reasoning involving larger knowledge-graph subgraphs.
- The paper experiments with Memory Network architectures while arguing that better architectures are needed beyond simple QA.
- Prior dialog datasets such as Twitter, Ubuntu, and Movie-Dic lack the CSQA flavor and an explicit associated knowledge graph.
Dataset Creation
The dataset is built by generating simple KG-grounded questions, extending them with complex reasoning operators, and linking them into context-dependent conversations. Manual annotation and semi-automatic instantiation produce a large-scale corpus with controlled splits.
- Knowledge Graph: The corpus uses Wikidata, whose cited 14-Nov-2016 dump contains 5.2K relations, 12.8M entities, and 52.3M facts.
- Simple Questions: Annotators created subject-based and object-based simple-question templates, while relation-based questions were excluded as often unnatural.
- Simple Questions: 1531 subject-based and 1450 object-based templates, including paraphrases, were collected for semi-automatic question instantiation.
- Complex Questions: Complex questions were generated with logical operators and quantitative aggregations over larger KG subgraphs, including AND, OR, NOT, max, min, and count.
- Linked Sequential QA: Conversations were formed by linking consecutive QA pairs that share a relation or entity, then adding indirect references, ambiguity, ellipsis, and clarification patterns.
- Linked Sequential QA: 200 K dialogs and 1.6 M turns were created through crowdsourced and in-house work plus semi-automatic instantiation, with validation and test tuples separated from training.
Some peculiar characteristics of Wikidata
Wikidata’s scale and heterogeneous schema introduce artifacts that can make generated CSQA questions unnatural or unevenly distributed. Although filtering removes many cases, some noisy, repetitive, rare-relation, and questionable peer-type questions remain.
- Generic Wikidata predicates can allow broad object types, producing unnatural instantiated questions that filtering cannot completely remove.Examples include lake outflow and fabrication method; the authors expect some noisy questions in the final dataset.
- Overlapping predicate and subject types can create repetitive questions, despite rule-based post-processing.Religion and sport can function as both predicates and subject types in Wikidata.
- A long tail of subject types and relations makes dominant relations frequent while rarer relations appear infrequently in linked questions.The authors state that this behavior is characteristic of any reasonably large knowledge base and cannot really be avoided.
- Wikidata’s peer subject types can yield logically valid but potentially odd questions that average users may not regard as peers.Religion and social group are both subclasses of belief system, motivating such cases.
Proposed Model
The proposed CSQA model combines hierarchical dialogue encoding with key-value memory-based QA. It represents context and KG entities, narrows the KG to candidate tuples, performs memory-based reasoning, and decodes varied answer types.
- The model combines an HRED-style hierarchical encoder with a key-value memory network for QA.This cross-task design targets the combination of conversational context and factual reasoning in CSQA.
- The hierarchical encoder uses a word-level RNN followed by a higher-level encoder that represents the current dialogue context.The higher-level encoder computes the context representation q1 from utterance representations.
- Handling Large Vocabulary: Question words concatenate available GloVe embeddings with offline TransE KG embeddings to represent entities without an unmanageably large vocabulary.Unavailable embedding types are replaced with zero vectors before concatenation.
- Candidate generation: Longest entity n-gram matching filters the KG before attention, retaining tuples where the matched entity is a subject or object.This candidate-generation step avoids computing attention over the entire large-scale KG.
- Key Value Memory Network: The key-value memory stores candidate tuples with relation-subject keys and object values using TransE embeddings.The model then makes multiple passes over the candidate memory to update its query representation.
- Decoder: The decoder is designed to generate counts, verification answers, clarification responses, and lists of KG entities.This output flexibility supports multiple answer formats required by CSQA.
Results
The experiments tune the proposed model and evaluate its performance across dialogue question types. The reported analysis emphasizes that context-dependent indirect or incomplete questions remain especially challenging.
- Adam optimization tunes learning rate, representation sizes, batch size, and dialogue context size using the validation set.The tested learning rates were 1e-3 and 4e-4; embedding and hidden sizes included 256 and 512, with context size 2.
- 10K candidate tuples are produced on average, motivating a cap on the number retained for subsequent memory processing.The passage states that the candidate-generation step produces approximately 10K tuples on average.
- Clarification questions are evaluated with entity Precision/Recall and overall utterance BLEU.These metrics separately assess KG-entity prediction and similarity of the complete natural-language response.
Discussions
The discussion identifies shortcomings of existing CSQA methods across complex reasoning, contextual question resolution, candidate generation, and memory organization, motivating further research and released resources.
- Simple v/s Complex Questions: Existing models perform very poorly on complex questions because they lack explicit aggregation functions for quantitative, comparative, and logical reasoning.Designing such aggregation functions for an end-to-end solution remains non-trivial.
- Simple v/s Complex Questions: Complex-question encoders may fail to decompose queries into subquestions and produce representations suitable for retrieving and combining relevant knowledge-graph tuples.The example requires separately finding rivers through India and China before intersecting the resulting sets.
- Direct v/s Indirect Questions: Model performance drops on indirect or incomplete questions that require contextual resolution of ellipsis and coreferences, especially when named entities and relations carry crucial information.The discussion calls for better modeling of these context elements.
- Candidate Generation: N-gram-based candidate generation can miss paraphrased answer entities, lowering recall and potentially causing the key-value memory to grow excessively.The resulting memory expansion leads to poor and inefficient training and inference.
- Better organization of the memory: Around 15% of questions require more than 100K candidate tuples, making flat key-value memories costly in capacity and softmax computation.The paper calls for better memory organization and approximate softmax methods.
- Resources: The released resources include train/validation/test splits, a processed WikiData dump, scripts for extracting question-type splits, and model-evaluation scripts.These resources are intended to facilitate further research on CSQA.
Conclusion
The paper introduces CSQA, a large-scale dataset of linked question-answer conversations requiring complex reasoning over a large knowledge graph. Its model exposes substantial challenges beyond existing dialog and QA datasets, motivating further research.
- Conclusion: CSQA contains 200K dialogs and 1.6M turns collected through a manually intensive semi-automated process.It is presented as a large-scale dataset of conversations over linked question-answer pairs.
- Conclusion: The dataset is presented as the first of its kind to combine linked QA conversations with logical, quantitative, and comparative reasoning over a knowledge graph containing millions of tuples.The paper positions this combination as distinct from prior datasets.
- Conclusion: The proposed model combines state-of-the-art dialog and QA approaches but remains inadequate for the challenges of CSQA.The paper states that addressing all identified challenges is beyond the scope of a single paper.