Source-linked AI summary
Neural Generative Question Answering
Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, Xiaoming Li
TL;DR
Generative QA must provide precise answers while handling flexible natural-language questions and storing discrete knowledge. The paper proposes GENQA, an encoder-decoder model that queries a knowledge-base during generation and trains on question-answer pairs linked to triples. Experiments show it generates right and natural answers and outperforms an embedding-based QA model and a neural dialogue model trained on the same data.
Problem
Generation-based QA models handle language variation but struggle to store the discrete, isolated knowledge required for real-world precision and coverage.
Method
GENQA jointly trains an encoder-decoder sequence-to-sequence model to generate answers while querying a knowledge-base linked to question-answer pairs and triples.
Results
GENQA generates right and natural answers by referring to knowledge-base facts and outperforms an embedding-based QA model and a neural dialogue model trained on the same data.
Takeaways & Limitations
GENQA demonstrates that a unified neural model can combine flexible language generation with knowledge-base facts for simple factoid question answering.
Takeaways & Limitations
The study considers only simple factoid questions associated with a single knowledge-base triple and focuses on forward-relation QA.
Abstract
from arXiv · showhide
This paper presents an end-to-end neural network model, named Neural Generative Question Answering (GENQA), that can generate answers to simple factoid questions, based on the facts in a knowledge-base. More specifically, the model is built on the encoder-decoder framework for sequence-to-sequence learning, while equipped with the ability to enquire the knowledge-base, and is trained on a corpus of question-answer pairs, with their associated triples in the knowledge-base. Empirical study shows the proposed model can effectively deal with the variations of questions and answers, and generate right and natural answers by referring to the facts in the knowledge-base. The experiment on question answering demonstrates that the proposed model can outperform an embedding-based QA model as well as a neural dialogue model trained on the same data.
1 Introduction
GENQA addresses the difficulty of combining neural language generation with precise access to discrete knowledge. It connects an encoder-decoder model to a knowledge-base so it can generate natural answers using retrieved terms and facts.
- Motivation: Fully distributed neural representations model language flexibility well but are unsuitable for storing discrete, isolated concepts needed for precise real-world QA.This limits the ability of generation-based QA systems to store all required knowledge with desired precision and coverage.
- Proposed approach: GENQA generates answers to simple factoid questions by querying a knowledge-base within an encoder-decoder sequence-to-sequence model.Its decoder can switch between generating common words and outputting terms retrieved from the knowledge-base.
- Proposed approach: The model is jointly trained on real-world question-answer pairs associated with knowledge-base triples.All components are tuned together while the model learns to generate answers referring to the associated facts.
- Empirical findings: GENQA can capture language variation and generate right, natural answers by referring to knowledge-base facts.The reported QA experiment found GENQA outperforming an embedding-based QA model and a neural dialogue model trained on the same data.
2 Task Description
The task trains a sequence-to-sequence system to answer simple factoid questions by retrieving relevant knowledge-base facts. The dataset links web-derived question-answer pairs to triples and tests generalization to unseen facts amid noisy language.
- 2.1 The learning task: Generative QA maps a natural-language question sequence to an answer sequence while querying a knowledge-base for relevant facts.The generated answer combines common words with knowledge-base words retrieved from an appropriate fact.
- 2.1 The learning task: Each training instance pairs a question and answer with a single knowledge-base triple, focusing on forward relations from subject and predicate to object.The answer contains the object of the associated triple as the knowledge-base word.
- 2.2 Data: The dataset combines triples mined from three Chinese encyclopedia websites with question-answer pairs collected from two Chinese community QA sites.The extracted entities and triples are normalized and aggregated into a knowledge-base.
- 2.2 Data: Training and test data are partitioned by triple, so test questions concern facts unseen during training.Many knowledge-base facts are absent from both partitions, underscoring the need to generalize to unseen facts.
- 2.3 Challenges: The central challenge is jointly learning question understanding, answer generation, and relevant-fact retrieval from noisy, informal data.Typos, nonstandard expressions, and broad language variation can obstruct acquisition of correct question-answer patterns.
3 The GENQA Model
GENQA combines an encoder-decoder with an external knowledge-base: it retrieves relevant triples and generates answers by mixing ordinary language generation with knowledge-base terms. Its components support end-to-end training through a unified probabilistic model.
- Architecture: GENQA consists of Interpreter, Enquirer, Answerer, and an external knowledge-base for generating answers from retrieved facts.Interpreter stores question information in short-term memory; Enquirer retrieves and summarizes relevant triples from long-term memory for Answerer.
- Interpreter: The Interpreter encodes the question with a bidirectional GRU into vector representations used by downstream components.It concatenates forward and backward recurrent states with word embeddings and one-hot representations.
- Enquirer: The Enquirer retrieves candidate triples, scores their relevance to the question, and represents the scores as a probability vector r_Q.It supports bilinear matching and a CNN-based matching model; candidate retrieval limits the matching computation to at most several hundreds of triples in the data.
- Answerer: The Answerer conditions an RNN decoder on question memory H_Q and retrieved-fact scores r_Q to generate the answer sequence.Its mixture model switches between common vocabulary words and objects from the knowledge-base vocabulary.
- Training: GENQA is trained end-to-end by maximizing observed-data likelihood with regularization over the RNN, Enquirer, and shared embedding parameters.The implementation uses stochastic gradient descent with mini-batches on GPUs.
4 Experiments
GENQA is evaluated against neural dialogue, retrieval-based, and embedding-based QA baselines for accuracy and fluency. It outperforms the competitors on accuracy, while generating fluent answers that combine knowledge-base terms with common words.
- Evaluation: The evaluation measures accuracy and answer fluency on 300 manually filtered test questions.Nearly duplicate and mistaken cases, including non-factoid questions, were removed before evaluation.
- Accuracy: NRM has the lowest accuracy, while the embedding-based model improves on it through generalization from distributed representations.The retrieval-based method has moderate accuracy but suffers from word mismatch between questions and knowledge-base triples.
- Accuracy: GENQA and GENQACNN outperform the competing QA models on test accuracy, with GENQACNN getting over half of the questions right.The authors attribute GENQACNN’s best accuracy to its convolution layer capturing salient matching features.
- Fluency: GENQA’s fluency does not differ significantly from NRM’s, and generation-based models usually produce correct sentences.This comparison concerns fluency rather than factual accuracy.
- Case Study: GENQA smoothly blends knowledge-base words with common words in generated answers through a unified neural model that selects between them.Figure 4 presents generated-answer examples with knowledge-base-generated words underlined.
- Case Study: Roughly 8% of correctly generated answers contain improper surrounding words, and some other cases fail to match the correct triples.Length-normalized beam search generally favors short answers, while incorrect triple matching can produce completely wrong answers.
5 Related Work
The work draws on neural machine translation, neural dialogue, knowledge-base embeddings, and memory networks. These lines of research supply sequence generation, representation-based retrieval, and memory-based reasoning, while motivating knowledge integration in QA.
- Neural Generation: Encoder-decoder models and attention mechanisms provide the sequence-generation framework underlying neural translation and dialogue systems.Neural dialogue models are trained end-to-end on conversation data but lack a mechanism to incorporate knowledge.
- Knowledge-Base QA: Knowledge-base embedding methods map questions and knowledge-base constituents into a shared low-dimensional space to rank relevant triples or subgraphs.Related work also transforms natural questions into logical forms using joint relational embeddings.
- Memory-Based QA: Memory Networks combine large memories with learning components that read and write memory for question-answering reasoning.Embedding-based QA under this framework suggests incorporating additional inference schemes into QA.
6 Conclusion
GENQA is an end-to-end generative QA model that queries a knowledge-base to produce natural, correct answers, with future work targeting interactive and complex-KB settings.
- GENQA generates natural and correct answers by querying a knowledge-base for relevant facts.The proposed model uses an encoder-decoder architecture equipped to query a knowledge-base.
- Future work extends GENQA toward iterative multi-turn question answering and question answering over complex-structured knowledge-bases.