Source-linked AI summary
Improving Neural Question Generation using Answer Separation
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, Kyomin Jung
TL;DR
Existing NQG models can include target-answer words in generated questions, producing unintended questions. The paper introduces answer-separated seq2seq, which masks the answer in the passage while separately using answer information and keyword-net. The model substantially improves over previous state-of-the-art NQG models and reduces improper answer inclusion.
Problem
NQG models often generate questions containing words from the target answer, resulting in unintended questions.
Method
Answer-separated seq2seq masks the target answer in the passage, processes passage and answer information separately, and uses keyword-net to extract key answer features.
Results
The model substantially outperforms previous state-of-the-art NQG models and significantly reduces target-answer inclusion in generated questions.
Takeaways & Limitations
Separating answer information helps generate the intended question while retaining answer-specific information through contextual cues and keyword extraction.
Abstract
from arXiv · showhide
Neural question generation (NQG) is the task of generating a question from a given passage with deep neural networks. Previous NQG models suffer from a problem that a significant proportion of the generated questions include words in the question target, resulting in the generation of unintended questions. In this paper, we propose answer-separated seq2seq, which better utilizes the information from both the passage and the target answer. By replacing the target answer in the original passage with a special token, our model learns to identify which interrogative word should be used. We also propose a new module termed keyword-net, which helps the model better capture the key information in the target answer and generate an appropriate question. Experimental results demonstrate that our answer separation method significantly reduces the number of improper questions which include answers. Consequently, our model significantly outperforms previous state-of-the-art NQG models.
Introduction
NQG models often struggle to focus questions on a specified answer and may reproduce answer words. The paper addresses this by separating the answer from the passage while retaining answer information through contextual cues and keyword extraction.
- NQG generates questions from passages with deep neural networks and supports educational materials, question answering, and chatbot conversations.
- Without explicit target-answer handling, models may generate questions without a specific target.A passage can contain multiple plausible targets, such as a person, location, or number.
- Answer-position features can still produce questions containing words from the target answer, creating improper questions.The paper gives “Who was elected John Francis?” as an example.
- Replacing the target answer with a special token preserves contextual clues for selecting the interrogative word while preventing answer-word inclusion.The masked passage retains information such as the target’s position or role.
- Answer-separated seq2seq processes the passage and target answer separately, while keyword-net extracts key answer information to supplement the separated representation.The model also uses a retrieval-style word generator.
- On SQuAD, the model outperforms previous state-of-the-art NQG models and shows fewer target-answer words in generated questions.The paper also reports better interrogative-word prediction and higher attention from the mask token to interrogative words.
Related Work
Earlier NQG work introduced end-to-end sequence-to-sequence models and answer-aware features, but target-answer words could still appear in generated questions. This paper focuses on using the answer as separated knowledge.
- Early end-to-end NQG used attention-based sequence-to-sequence learning but did not explicitly model the target answer.Its generated questions were described as containing substantial randomness.
- Later models incorporated answer positions, named-entity and part-of-speech features, or passage-answer interaction to make generation more answer-aware.These approaches also used copy mechanisms to reflect passage-word reuse.
- Previous answer-aware systems did not resolve the frequent inclusion of target-answer words, and copy mechanisms could intensify the problem.The paper therefore treats the target answer as separated knowledge.
- Question generation has also been used to create additional data for training question-answering systems.Earlier work generated questions through retrieval-based and generation-based methods.
- The paper identifies target-answer inclusion as an issue not previously emphasized by earlier NQG work.
Task Definition
The NQG task takes a passage and a target answer as input and generates a question about that answer. Formally, it seeks the question with maximum conditional likelihood given both inputs and the preceding generated words.
- Given a passage Xp and target answer Xa, the NQG model generates a question Y asking about Xa.
- The task selects the best question Y by maximizing its conditional likelihood given Xp, Xa, and previously generated words y<t.
Base Model: Encoder-Decoder with Attention
The base model uses an RNN encoder-decoder with attention to generate a question from a passage. A bidirectional LSTM encoder represents the passage, while an attention-based decoder uses relevant input features to produce the output sequence.
- Base architecture: The base model takes a passage Xp as input and generates a relevant question Y without using the target answer.It is an RNN-based sequence-to-sequence model.
- Encoder: The encoder represents a variable-length input sequence as a fixed-length vector containing contextual features and token dependencies.
- Attention: Attention computes normalized alignment weights and forms a context vector from weighted encoder states.This lets the decoder focus on relevant parts of the input while generating the sequence.
- Encoder: A one-layer bidirectional LSTM extracts contextual features from the input passage.Forward and backward hidden states are concatenated into the bidirectional hidden state.
- Decoder: The one-layer unidirectional LSTM decoder uses attention to generate the corresponding question.At each step, it combines the previous output token, previous hidden state, and current context vector.
Answer-Separated Seq2seq
Answer-separated seq2seq encodes the passage and target answer separately, replacing the answer in the passage with a special token. Its decoder combines both representations and uses keyword-net and retrieval-style word generation to produce more appropriate questions.
- Architecture: Answer-separated seq2seq treats the passage and target answer separately to prevent the decoder from copying the answer into generated questions.The model uses two encoders and an answer-separated decoder.
- Encoders: Two one-layer bidirectional LSTM encoders independently encode the passage and target answer.The answer encoder combines its final forward and backward states into an overall answer representation.
- Answer separation: The passage encoder replaces the target answer with a special <a> token, preserving its position and contextual information.This teaches the model which passage region should be associated with the generated question without exposing the answer words.
- Decoder: The answer-separated decoder combines passage and answer information through decoder initialization and an answer-aware decoding process.Its state is initialized with the final answer vector.
- Keyword-net: Keyword-net extracts key information from the target answer by matching decoder outputs with answer hidden states.The resulting keyword feature is initialized from the current context vector and refined across layers.
- Word generation: A retrieval-style word generator queries distributed word representations to capture word meaning during generation.The query combines the decoder hidden state and context vector, and normalized relevance scores determine word-generation probabilities.
Experimental Settings
The experiments use processed SQuAD data, standard question-generation metrics, and ablations of the proposed modules. Training and decoding settings specify the model vocabulary, embeddings, optimization procedure, and beam search, while additional analyses measure answer inclusion and interrogative-word recall.
- Dataset: Experiments use two processed SQuAD data splits created from passages containing target answers.The original dataset contains 23,215 paragraphs, over 100k questions, and 536 articles.
- Model settings: The proposed implementation uses one-layer encoders and decoder with 350 hidden units, a 34k-word vocabulary, and frozen 300-dimensional pretrained GloVe embeddings.
- Training: Training uses Adam with an initial learning rate of 0.001, batch size 128, and up to 17 epochs.
- Inference: Inference uses beam search with width 10 and length-penalty weight 2.1, stopping when <EOS> is generated.Reported performance is given as mean ± standard deviation.
- Additional analyses: Additional analyses measure complete or partial target-answer inclusion and recall across eight interrogative-word categories.The categories include what, how, when, which, where, who, why, and yes/no.
- Evaluation metrics: Models are evaluated with BLEU-4, METEOR, and ROUGE-L.BLEU-4 counts matching 4-grams, METEOR measures several word and phrase matching types, and ROUGE-L uses longest common subsequence.
Results
Answer-separated seq2seq outperforms prior NQG models and reduces answer inclusion by separating target-answer information from passage context. Analyses link its gains to improved interrogative prediction, attention from the mask token, and keyword-net.
- Performance comparison: ASs2s outperforms all previous NQG models on both data splits, while removing answer masking, keyword-net, or the answer-separated decoder substantially reduces performance.The complete model benefits from separate target-answer utilization, keyword-net, and answer-separated decoding.
- Answer copying: The percentage of generated questions containing target answers is significantly lower for ASs2s than for comparison systems.Song et al. (2018) produces more irrelevant target-answer questions than seq2seq+AP, indicating an adverse effect from copying.
- Interrogative word prediction: Answer-separated seq2seq achieves higher recall than seq2seq+AP for all eight interrogative-word categories.The largest improvements occur for “how,” “when,” “where,” and “who,” while rare “why” and “yes/no” categories remain difficult.
- Attention analysis: In the attention example, the mask token gives most attention to “who,” whereas seq2seq+AP attends only to part of the answer and generates “john francis.”The comparison illustrates how answer separation can preserve target-related context without copying answer words into the question.
- Machine comprehension: Training machine comprehension only on synthetic questions yields EM/F1 of 22.72/31.58 on the public SQuAD dev set and 49.09/56.57 on its single-named-entity subset.The synthetic data contains about 50k questions, and the subset result is compared with 68.78/78.56 when training uses the original training set.
Conclusion
The paper addresses unintended answer inclusion in neural question generation with an architecture that separately treats passages and target answers. Experiments show stronger target-specific question generation and substantial improvement over previous state-of-the-art models.
- Conclusion: Answer-separated seq2seq treats the passage and target answer separately to better use information from both sides.The architecture replaces the target answer with a special token while retaining the answer separately.
- Conclusion: Experimental results show stronger generation of the right question for the target answer and substantial improvement over previous state-of-the-art models.The conclusion frames answer separation as the basis for these reported gains.