Source-linked AI summary
Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus
Iulian Vlad Serban, Alberto García-Durán, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, Yoshua Bengio
TL;DR
QA research has lacked large-scale labeled question-answer corpora, despite their importance for modeling natural-language variation. The paper uses neural models to transduce Freebase facts into questions, handling knowledge-base sparsity and rare words, and builds the 30M corpus. Across automatic and human evaluations, the neural questions outperform template-based questions and appear comparable to human-generated questions.
Problem
QA systems, particularly neural systems, lack large-scale labeled question-answer pairs that capture paraphrases and ambiguity.
Method
Neural models transduce Freebase facts into natural-language questions, using knowledge-base embeddings for sparsity and placeholders for rare words.
Results
Across all automatic metrics, neural network models outperform the template-based baseline; human evaluators also cannot distinguish the best neural questions from human-generated questions on average.
Takeaways & Limitations
The best-performing model produces the freely available 30M Factoid Question-Answer Corpus, whose pairs are suggested to be useful for training QA systems.
Takeaways & Limitations
The authors believe some SimpleQuestions questions were produced by nonnative English speakers.
Abstract
from arXiv · showhide
Over the past decade, large-scale supervised learning corpora have enabled machine learning researchers to make substantial advances. However, to this date, there are no large-scale question-answer corpora available. In this paper we present the 30M Factoid Question-Answer Corpus, an enormous question answer pair corpus produced by applying a novel neural network architecture on the knowledge base Freebase to transduce facts into natural language questions. The produced question answer pairs are evaluated both by human evaluators and using automatic evaluation metrics, including well-established machine translation and sentence similarity metrics. Across all evaluation criteria the question-generation model outperforms the competing template-based baseline. Furthermore, when presented to human evaluators, the generated questions appear comparable in quality to real human-generated questions.
1 Introduction
The paper addresses the shortage of labeled question-answer pairs by generating natural-language questions from Freebase facts with neural models. The resulting pairs outperform a template-based baseline and appear comparable to human-generated questions in human evaluation.
- Labeled question-answer data are a major obstacle for training QA systems, especially neural systems that need examples of paraphrase and ambiguity.
- The paper frames question generation as transducing each Freebase fact into a question whose answer is the fact’s object.
- The proposed neural models are inspired by neural machine translation and use techniques for handling rare words.
- The models are evaluated with human judgments and automatic metrics including BLEU, METEOR, and sentence similarity.
- The generated questions outperform the template-based baseline and appear indistinguishable from real human-generated questions to untrained evaluators.
- The best model is used to construct and freely release the 30M Factoid Question-Answer Corpus.
2 Related Work
Earlier question-generation work largely mapped unstructured text or symbolic representations to questions using rules or templates. This paper is positioned as the inverse mapping from structured facts to natural-language questions.
- Rule-based approaches use syntactic operations such as wh-fronting or wh-inversion, without fully using words’ semantic content.
- Methods mapping unstructured text to questions typically transform text into a symbolic representation before generating the question.
- Triple-based systems use relationship-specific templates and replace placeholder tokens with triple entities.
- The paper’s template baseline derives multiple relationship-specific templates from the different question framings observed in training pairs.
- Unlike logical-form realization work, this approach takes a fact as input and outputs its associated question.
- Rule- and template-based synthetic questions have been used to address the shortage of question-answer pairs for QA training.
3 Task Definition
The task treats a knowledge-base fact as structured input and generates a question about its subject and relationship, with the object as a valid answer. Models are trained on human-authored SimpleQuestions pairs derived from Freebase facts.
- 3.1 Knowledge Bases: A Freebase knowledge base is represented as a graph whose nodes are entities and whose directed edges are relationships connecting pairs of entities.
- 3.1 Knowledge Bases: A fact is a subject-relationship-object triple representing a complete piece of information.
- 3.2 Transducing Facts to Questions: The generated question must concern the fact’s subject and relationship, while its object must be a valid answer.
- 3.3 Dataset: SimpleQuestions contains over 100K English question-answer pairs created by Mechanical Turk users from Freebase facts.
- 3.3 Dataset: Annotators were shown a complete Freebase fact and asked to phrase a question whose answer was the fact’s object.
- 3.3 Dataset: The dataset’s questions can include information about the target object and are often ambiguous.
4 Model
The model transduces Freebase facts into questions using an encoder–decoder architecture with attention, pretrained knowledge-base embeddings, and placeholders for sparse or unseen tokens.
- 4 Model: The system treats question generation as lossy translation from Freebase triples into natural-language questions whose answers are the triples’ objects.Each fact consists of a subject, relationship, and object; the model intentionally omits some structured information, such as the object name, from the generated question.
- 4.1 Encoder: Each fact atom is embedded and linearly transformed, then the three resulting representations are concatenated into the fact embedding Enc(F).The input embedding matrix can be pretrained separately and fixed during model training.
- 4.2 Decoder: A GRU decoder with attention generates the question token by token, using a context vector that weights subject, relationship, and object representations according to the previous hidden state.The decoder initializes its hidden state from the fact embedding and produces token probabilities with a softmax layer.
- 4.3 Modeling the Source Language: TransE initializes subject, relationship, and object embeddings from the knowledge base to address sparse graphs and previously unseen entities or relationships.The model uses the energy function f(s, r, o) = ||es + er − eo||2 to assign low values to true facts and high values otherwise.
- 4.3 Modeling the Source Language: Additional Freebase-derived category and geographical information is used to make entity embeddings capture semantic types, professions, nationalities, and locations.These signals provide coarse and fine-grained structure that may be difficult to learn from the sparse SimpleQuestions graph alone.
- 4.4 Generating Questions: The Single-Placeholder model replaces question words overlapping with the subject string by <placeholder>, restoring the subject string after generation.This reduces target-side sparsity while preserving entity and relationship information on the input side.
5 Experiments
The experiments evaluate question-generation models with automatic metrics and human preferences, comparing neural systems against a template-based baseline and human-written questions.
- Automatic Evaluation Metrics: The study uses BLEU, METEOR, and Embedding Greedy to evaluate generated questions automatically.Embedding Greedy measures sentence similarity through word alignments and Word2Vec cosine similarity.
- Automatic Evaluation Metrics: The neural network models outperform the template-based baseline by a clear margin across all reported automatic metrics.The baseline uses relationship-specific templates, while the neural models include variants with single or multiple placeholders and TransE++ embeddings.
- Automatic Evaluation Metrics: The strongest-performing models use TransE embeddings trained on the largest set of triples, which includes approximately 30 million facts.These facts were used to generate the 30M Factoid Question-Answer Corpus.
- Human Evaluation Study: Human evaluation records preferences among human-generated questions, template-baseline questions, and multiple-placeholder TransE++ questions.Evaluators judge which question is most relevant and natural for a given fact, with a comparable class for tied preferences.
- Human Evaluation Study: Across 3,810 preferences from 63 evaluators, humans strongly prefer human or neural questions over the template-based baseline.Human evaluators cannot distinguish human-generated questions from neural questions on average, with a slight preference for the latter.
- Human Evaluation Study: The authors attribute some evaluation behavior to the model penalizing uncommon or unnatural question framings and sometimes including specific information about the target object.The source questions may also include examples produced by nonnative English speakers.
6 Conclusion
The paper proposes neural models that map knowledge-base facts to natural-language questions and uses the best model to create a 30M question-answer corpus.
- 6 Conclusion: The models combine neural machine translation architectures, multi-relational knowledge-base embeddings, and placeholders for rare words.The resulting question-answer pairs are evaluated with BLEU, METEOR, sentence similarity, and untrained human judgments.
- 6 Conclusion: The generated question-answer pairs outperform a template-based baseline on automatic metrics and appear comparable in quality to human-generated questions.These findings are reported across automatic evaluation and untrained human evaluation.
- 6 Conclusion: The best-performing neural model generates a freely available corpus containing 30M question-answer pairs.The authors hope the corpus will enable future improvements to question-answering systems.
A Supplemental Material: Generated Questions
The supplemental material presents additional test examples comparing the template-based baseline with the MP Triples TransE++ model.
- A Supplemental Material: Generated Questions: Table 6 provides test examples and corresponding questions from the template-based baseline and MP Triples TransE++.
- A Supplemental Material: Generated Questions: Table 7 provides additional test examples and corresponding questions from the template-based baseline and MP Triples TransE++.