Source-linked AI summary
Learning to Ask: Neural Question Generation for Reading Comprehension
Xinya Du, Junru Shao, Claire Cardie
TL;DR
Automatic question generation for reading comprehension must produce natural questions without relying on manually engineered rules. The paper uses an attention-based neural sequence-to-sequence model, including sentence- and paragraph-level variants, and reports stronger automatic and human evaluations than prior systems.
Problem
Prior question-generation systems relied heavily on handcrafted rules and could produce questions that closely overlapped the source, making them easy to answer.
Method
The paper trains an attention-based conditional neural sequence-to-sequence model end to end and investigates sentence- versus paragraph-level passage encoding.
Results
The best model achieves state-of-the-art performance in both automatic and human evaluations, with questions rated more grammatical, fluent, and challenging than the rule-based comparison system.
Takeaways & Limitations
End-to-end neural question generation can produce higher-quality and more difficult reading-comprehension questions than the evaluated rule-based system.
Takeaways & Limitations
The paragraph-level model does not achieve the best performance across all question categories.
Abstract
from arXiv · showhide
We study automatic question generation for sentences from text passages in reading comprehension. We introduce an attention-based sequence learning model for the task and investigate the effect of encoding sentence- vs. paragraph-level information. In contrast to all previous work, our model does not rely on hand-crafted rules or a sophisticated NLP pipeline; it is instead trainable end-to-end via sequence-to-sequence learning. Automatic evaluation results show that our system significantly outperforms the state-of-the-art rule-based system. In human evaluations, questions generated by our system are also rated as being more natural (i.e., grammaticality, fluency) and as more difficult to answer (in terms of syntactic and lexical divergence from the original text and reasoning needed to answer).
1 Introduction
Question generation for reading comprehension requires natural, sufficiently difficult questions that may use paraphrase, context, and world knowledge. The paper proposes an end-to-end attention-based sequence-to-sequence approach and evaluates it against prior systems.
- Motivation: Question generation creates natural questions from sentences or paragraphs, with reading-comprehension education as a key application.Other applications include chatbot interaction and developing annotated NLP datasets.
- Prior limitations: Earlier systems mainly used manually designed declarative-to-interrogative rules, while overgenerate-and-rank methods still depended on crafted rules and features.Generated questions could overlap the input word for word, making them easy to answer.
- Motivation: Natural questions may compress source sentences, use synonyms, refer to preceding context, or require world knowledge rather than merely transform syntax.These properties motivate an abstractive approach capable of fluent phrasings that do not exactly match the source text.
- Contribution: The paper frames question generation as sequence-to-sequence learning that maps a passage sentence directly to a question without manually generated rules.The model is fully data-driven and uses a conditional neural language model with global attention.
- Contribution: The experiments compare sentence- and paragraph-level information and variations involving pre-trained versus learned word embeddings.The study evaluates these neural model variations in the reading-comprehension setting.
- Results: On SQuAD, the system significantly outperforms strong information-retrieval, statistical-translation, and overgenerate-and-rank baselines, while human raters find its questions more grammatical, fluent, and challenging.Human judgments assess syntactic divergence from the passage and reasoning needed to answer.
2 Related Work
Related work spans synthetic and natural reading-comprehension datasets, rule-based question generation, and structured-to-text systems. The paper identifies a gap: prior reading-comprehension QG work had not used end-to-end deep sequence-to-sequence learning.
- Reading-comprehension datasets: Machine reading comprehension requires natural-language understanding and world knowledge, while existing datasets vary in syntheticity, size, and question quality.SQuAD addresses earlier size and semi-synthetic limitations with crowd-worker questions of relatively high quality.
- Research gap: Prior rule-based systems often produce questions through syntactic transformation, whereas the present work targets natural questions for reading comprehension.SQuAD provides the corpus used for this natural-question generation setting.
- Rule-based generation: Question-generation research commonly transforms syntactic representations into interrogative sentences using manually constructed templates or rules.These approaches generally emphasize syntactic rather than semantic roles.
- Rule-based generation: Heilman and Smith’s overgenerate-and-rank system learns to rank questions but remains critically dependent on manually constructed generating rules.Other work includes visual question generation and factoid questions generated from subject-relation-object triples.
- Research gap: The authors state that previous work had neither framed reading-comprehension QG end to end nor used deep sequence-to-sequence learning for question generation.This positions the proposed neural formulation as a distinct approach to the task.
3 Task Definition
The task maps an input sentence to a natural question related to information in that sentence. It selects an output sequence by maximizing its conditional likelihood given the input.
- Task formulation: Given an input sentence x, the goal is to generate a natural question y related to information in x.The question sequence may have arbitrary length.
- Task formulation: The input sentence is represented as a token sequence [x1, ..., xM], and the generated question as [y1, ..., y|y|].M denotes the input sentence length, while the output length is unrestricted.
- Objective: The task chooses the question sequence y that maximizes P(y|x), the conditional log-likelihood of the predicted question given the input.The paper later elaborates on modeling this conditional probability with global attention.
4 Model
The model generates questions with an attention-based sequence-to-sequence architecture, using sentence information and optionally paragraph context. It combines recurrent encoding and decoding with attention, paragraph truncation, and decoding procedures for question generation.
- Model architecture: An RNN encoder-decoder with global attention focuses on relevant input elements while generating each question.The decoder uses its recurrent state together with an attention-based encoding of the input.
- Model variants: Two variants encode either only the sentence or both the sentence and paragraph-level information.The paragraph-level model initializes decoding from the concatenated sentence and paragraph representations in a Y-shaped network.
- Model architecture: The system frames question generation as sequence-to-sequence learning, predicting each question word from previously generated words and the input sentence.The conditional probability is factorized into word-level predictions, with each prediction conditioned on y<t and x.
- Sentence encoding: The sentence encoder is a bidirectional LSTM whose token representations support attention and whose final forward and backward states initialize the decoder.Attention weights use bilinear scoring with softmax normalization, while the sentence representation concatenates the final directional states.
- Paragraph encoding: The paragraph encoder is another bidirectional LSTM, and its concatenated final directional states provide the paragraph representation.Paragraph information is used only in the paragraph-level model; long paragraphs are truncated at a threshold L.
- Training and decoding: Training minimizes the data negative log-likelihood, while inference uses beam search and replaces unknown tokens with highly attended input tokens.The beam search uses k possible paths, and each decoded UNK is replaced by the input token receiving the highest attention score.
5 Experimental Setup
The experiments use processed SQuAD sentence–question pairs, article-level train/development/test splits, and automatic and human evaluation. The study compares neural question-generation models with several retrieval, rule-based, direct-copy, and sequence-learning baselines.
- Dataset: The processed SQuAD corpus contains over 100k questions from 536 articles, with approximately 70k training samples after preprocessing.Sentences average about 30 tokens, questions about 10 tokens, and each sentence has 1.4 questions on average.
- Dataset: Answers identify the input sentence, while pairs with no shared non-stop-word are pruned from the training data.Answer spans crossing sentences are handled by concatenating those sentences; the accessible ∼90% of SQuAD is treated as the full dataset.
- Dataset: Data are divided by article into 80% training, 10% development, and 10% test sets, with results reported on the test split.Hyperparameters are tuned on the development set and final results are reported on held-out test data.
- Implementation Details: The neural models use 45k source and 28k target vocabulary limits, 300-dimensional GloVe embeddings, two-layer 600-unit LSTMs, dropout, and beam search.Training uses SGD with minibatches of 64; decoding uses beam size 3.
6 Results and Analysis
The neural system performs strongly in automatic and human evaluations, while paragraph information helps on questions requiring context beyond the sentence but can add noise overall. Qualitative analyses show better paraphrasing and alignment than H&S, alongside some imperfect or overly challenging outputs.
- Automatic evaluation: The sentence-level model achieves the best performance across all automatic metrics, while IR performs poorly and DirectIn and H&S remain comparable.H&S largely performs syntactic changes without paraphrasing, consistent with its high input-question overlap.
- Automatic evaluation: Adding pre-trained embeddings generally helps, whereas encoding paragraph information slightly reduces overall performance because paragraphs also introduce noise.
- Human evaluation: Our system is ranked best in 38.4% of human evaluations, with an average ranking of 1.94 and higher ratings than H&S across all modalities.The reported overall inter-rater agreement is Krippendorff’s Alpha 0.236.
- Qualitative analysis: Qualitative examples show stronger focus capture and paraphrasing than H&S, although some generated questions introduce less-related wording while becoming more challenging to answer.Attention weights align output keywords such as “introduced” and “teletext” with corresponding input words.
- Dataset analysis: The dataset analysis categorizes questions by whether sentence, paragraph, article, or world knowledge is needed to generate them.The analysis randomly sampled 346 sentence-question pairs from the development set.
- Per-category analysis: The paragraph-information model performs best on “w/ paragraph” questions, supporting its effectiveness when generating questions requires information outside the sentence.
7 Conclusion and Future Work
The paper presents a fully data-driven attention-based neural approach to question generation and examines sentence- versus paragraph-level information. Its best model achieves state-of-the-art performance in automatic and human evaluations, while future work targets better use of paragraph context and additional generation mechanisms.
- Conclusion: The paper presents a fully data-driven attention-based neural network approach for reading-comprehension question generation.
- Conclusion: The models investigate the effect of encoding sentence-level versus paragraph-level information.
- Conclusion: The best model achieves state-of-the-art performance in both automatic and human evaluations.
- Future work: Future work will seek to use paragraph information more effectively across question categories and incorporate mechanisms such as copying to improve generated-question quality.