Source-linked AI summary
Machine Comprehension by Text-to-Text Neural Question Generation
Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman, Sandeep Subramanian, Saizheng Zhang, Adam Trischler
TL;DR
Question generation from text is studied as a way to support information acquisition and potentially improve question answering. The paper develops an end-to-end recurrent model conditioned on documents and answers, trains it with maximum likelihood and policy-gradient rewards, and reports improved generated-question rewards. The approach remains subject to concerns about reward exploitation and the difficulty of evaluating diverse valid questions with overlap metrics.
Problem
Question generation from documents is important for information acquisition and may support question answering, but existing QA work primarily predicts answers from document-question pairs.
Method
A recurrent encoder-decoder generates questions conditioned on a document and answer, using copying, answer suppression, maximum likelihood, and policy-gradient optimization of fluency and QA rewards.
Results
Policy gradient increased QA reward by 8.9% in accuracy over the improved Seq2Seq model and improved most other metrics, while combined PPL and QA rewards improved QA performance and recovered some PPL.
Takeaways & Limitations
The study demonstrates an end-to-end text-to-text question-generation model and shows that reward combinations affect generated-question quality quantitatively and qualitatively.
Takeaways & Limitations
QA-reward optimization can produce overly simplistic or exploitative questions, and overlap metrics inadequately evaluate diverse valid questions.
Abstract
from arXiv · showhide
We propose a recurrent neural model that generates natural-language questions from documents, conditioned on answers. We show how to train the model using a combination of supervised and reinforcement learning. After teacher forcing for standard maximum likelihood training, we fine-tune the model using policy gradient techniques to maximize several rewards that measure question quality. Most notably, one of these rewards is the performance of a question-answering system. We motivate question generation as a means to improve the performance of question answering systems. Our model is trained and evaluated on the recent question-answering dataset SQuAD.
1 Introduction
The paper reframes machine comprehension as generating natural-language questions from documents and predefined answers. It presents question generation as both an information-acquisition task and a potential means to improve question answering.
- The model teaches machines to ask questions about documents rather than answer questions about them.
- Question generation is motivated by information acquisition, improved question answering, and the abstractive nature of asking questions.Answering is typically extractive, whereas question asking requires generating text that may not appear in the document.
- The paper adapts sequence-to-sequence modeling to generate questions conditioned on a document and answer.The encoder processes the document and answer, while the decoder outputs question words sequentially.
- Training combines maximum likelihood with policy-gradient optimization of fluency and question-answering rewards.The authors report quantitative improvements in test-time rewards and provide qualitative comparisons across training schemes.
2 Related Work
Earlier question-generation systems relied on rules, templates, or structured inputs, while newer neural approaches support end-to-end generation. This paper instead generates questions from unstructured text conditioned on an answer and evaluates the questions themselves.
- Traditional systems parse individual sentences and reformulate questions with hand-crafted rules, producing high-overlap questions tied to sentence structure.
- Neural question-generation work includes systems converting knowledge-base triples or images into questions, alongside question generation used as a QA auxiliary task.
- This paper conditions question generation on a known answer while extending the context from structured triples to unstructured text.
- Unlike typical extractive QA systems, the proposed decoder generates questions rather than extracting answers from documents.
- The model applies a REINFORCE-style algorithm to maximize rewards earned by generated questions.
3 Encoder-Decoder Model for Question Generation
The encoder-decoder model represents both a document and an extractive answer, then generates a question sequentially using attention and copying. Training combines teacher-forced likelihood with answer suppression, entropy regularization, and related objectives.
- 3.1 Encoder: The encoder processes document and answer sequences, using answer positions in the document to construct an extractive condition encoding.Document words receive answer-membership features, and a second bidirectional LSTM encodes the answer span with its document annotations.
- 3.2 Decoder: The decoder generates question tokens sequentially from representations of the document and answer.At each step, it models a conditional distribution over the next output word.
- 3.2 Decoder: Pointer-softmax interpolates between generating vocabulary words and copying tokens from the document.A source-switching network controls the balance between the shortlist and location softmax distributions.
- 3.2 Decoder: Attention computes document relevance weights at each decoding step to identify words relevant to the question.These alignment scores also provide the document-token distribution for copying.
- 3.3 Training: Initial training minimizes negative log-likelihood with teacher forcing, where previous decoder tokens come from the source sequence.
- 3.3 Training: Additional objectives suppress answer words and maximize entropy to encourage questions that avoid answer leakage and use more varied outputs.Answer suppression uses a penalty over answer words absent from the ground-truth question; entropy regularization counters common-output degeneracy.
4 Policy Gradient Optimization
The model fine-tunes a teacher-forced question generator with policy gradients and rewards tied to QA performance and fluency. Beam search, reward normalization, and continued likelihood training support optimization while limiting sampling variance and forgetting.
- Motivation: Teacher forcing creates a training–testing mismatch because training uses ground-truth sequences, whereas testing uses the model’s own outputs.It also prevents the model from making and learning from mistakes during training.
- REINFORCE: Policy gradient fine-tuning follows maximum-likelihood pretraining and optimizes scalar rewards correlated with question quality.The approach uses REINFORCE over model-generated token sequences initialized from likelihood-trained parameters.
- Rewards: QA reward uses a pretrained MPCM system’s answer accuracy on generated questions, with F1 measuring overlap between predicted and reference answers.MPCM processes the generated question and document, then points to answer start and end tokens.
- Rewards: Answer-word masking prevents the QA reward from favoring questions that inject answer words or exploit quirks of the MPCM model.The authors identify overly simplistic or adversarially favorable questions as a risk of optimizing the QA reward.
- Rewards: Fluency reward uses the negative perplexity assigned by an LSTM language model trained on over 80,000 human-generated SQuAD questions.The negation rewards lower perplexity, corresponding to more fluent generated questions under the language model.
- Reward Combination: The combined reward weights QA and fluency terms, while reward models remain black boxes and are not jointly optimized or differentiated through.The weighting coefficients λQA and λPPL are hyperparameters, and future work is left to explore joint optimization.
- Optimization: Reward whitening uses running reward mean and standard deviation to stabilize learning, while maximum-likelihood updates are combined with policy-gradient updates.The likelihood term is retained during policy-gradient optimization to prevent overwriting earlier training.
- Optimization: Beam search replaces random sampling for expectation estimation because sampling was less effective empirically and could increase policy-gradient variance.Generated beam sequences are teacher-forced to recreate their model states before updating the model with the reward-weighted log-likelihood.
5 Experiments
Experiments on SQuAD evaluate question generation with automatic metrics and qualitative analyses. Policy-gradient rewards improve answerability and other metrics, but expose trade-offs with fluency, abstraction, and evaluation by word overlap.
- Experimental setup: SQuAD experiments use a Seq2Seq baseline and automatic metrics including BLEU, F1, QA performance, and perplexity.The dataset contains document, question, and answer triples; Table 2 evaluates generated questions on its development set.
- Discussion: Generated questions can look reasonable despite low BLEU because a document-answer pair may support multiple semantically distinct questions.This makes overlap-based evaluation difficult when valid generated questions share few words with the reference.
- Results: QA and fluency can compete: PPL-focused policy-gradient training improves perplexity but decreases QA, F1, and BLEU.A more fluent example does not refer to the desired answer.
- Results: Policy-gradient training with the QA reward gains 8.9% accuracy over the improved Seq2Seq model and improves most other metrics.The QA score is also 3.7% higher than the score obtained on ground-truth questions, while reported perplexity is higher.
- Results: Combining PPL and QA rewards improves QA performance over maximum-likelihood training while recovering some perplexity.A generated example remains specific to the answer but ends prematurely.
- Discussion: Qualitative analysis identifies entity and verb substitutions, limited abstraction, absent commonsense and relational reasoning, and pathologies requiring future work.The authors also note that QA rewards do not appear beneficial for abstraction, while NLL training yields more abstraction at lower specificity.
6 Conclusion
The paper proposes recurrent neural question generation conditioned on text passages and predefined answers, trained with maximum likelihood and policy-gradient optimization. It evaluates how reward combinations affect generated outputs quantitatively and qualitatively.
- The proposed recurrent neural model generates natural-language questions conditioned on text passages and predefined answers.
- Training combines maximum likelihood with policy-gradient optimization.
- The study evaluates how several reward combinations affect generated outputs using quantitative and qualitative analyses.
A Implementation details
The models use Keras with a Theano backend, Adam optimization, GloVe initialization, and RNN hidden size 768. Training and decoding use specified regularization, reward-weighting, and beam-search settings.
- The models are implemented in Keras with a Theano backend, using Adam at an initial learning rate of 2e-4 for both maximum likelihood and policy gradient updates.GloVe vectors initialize the word embeddings, which are updated during training.
- All recurrent neural networks use a hidden size of 768, with GloVe-initialized embeddings updated during training.
- Dropout with rate 0.3 is applied to embedding layers and all RNNs, including input-hidden and hidden-hidden connections.
- Answer-suppression and entropy-maximization coefficients are both set to 0.01.
- Beam search uses a beam size of 32 in all experiments, while policy-gradient reward weights are listed in Table 6.Table 6 provides the hyperparameter settings for policy gradient training.