PaperlayerView exact revision on arXiv

Source-linked AI summary

Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

Shiyue Zhang, Mohit Bansal

arXiv:1909.06356v1cs.CLcs.AIcs.LG

TL;DR

Question generation models can produce questions that drift from their context and answer, while traditional metrics may not capture answerability or valid paraphrases. The paper uses semantic rewards, QA-based evaluation, and synthetic QA data strategies; it reports state-of-the-art QG results and improvements over QA baselines, even without new articles.

  • Problem

    Generated questions can drift from their context and answer, and traditional metrics may miss answerability and valid paraphrases.

    Source: §1 · pp. 1–2

  • Method

    The paper regularizes QG with paraphrase and answerability rewards, evaluates generated QA data through a QA-based measure, and uses filtered synthetic data with mixing mini-batch training.

    Source: §1 · p. 2

  • Results

    The approach achieves new state-of-the-art QG performance and improves BiDAF and BERT QA baselines, including without introducing new articles.

    Source: §7 · p. 9

  • Takeaways & Limitations

    QG-generated data can improve semi-supervised QA performance even when no new articles are introduced.

    Source: §1 · p. 2

Abstract

from arXiv · show

Text-based Question Generation (QG) aims at generating natural and relevant questions that can be answered by a given answer in some context. Existing QG models suffer from a "semantic drift" problem, i.e., the semantics of the model-generated question drifts away from the given context and answer. In this paper, we first propose two semantics-enhanced rewards obtained from downstream question paraphrasing and question answering tasks to regularize the QG model to generate semantically valid questions. Second, since the traditional evaluation metrics (e.g., BLEU) often fall short in evaluating the quality of generated questions, we propose a QA-based evaluation method which measures the QG model's ability to mimic human annotators in generating QA training data. Experiments show that our method achieves the new state-of-the-art performance w.r.t. traditional metrics, and also performs best on our QA-based evaluation metrics. Further, we investigate how to use our QG model to augment QA datasets and enable semi-supervised QA. We propose two ways to generate synthetic QA pairs: generate new questions from existing articles or collect QA pairs from new articles. We also propose two empirically effective strategies, a data filter and mixing mini-batch training, to properly use the QG-generated data for QA. Experiments show that our method improves over both BiDAF and BERT QA baselines, even without introducing new articles.

1 Introduction

The paper addresses semantic drift in question generation and proposes semantic rewards, a QA-based evaluation, and methods for using generated questions in semi-supervised QA. It reports state-of-the-art QG performance and improvements over QA baselines, including without new articles.

1 Introduction

  • QG remains understudied despite its potential to augment QA datasets and support conversation and education systems.The paper also notes QG’s potential role in encouraging machine comprehension through semantic reasoning and syntactic variation.Source: §1 · p. 1
  • Semantic drift occurs when generated questions depart from the context and answer, as in a baseline question with nearly contrary meaning to its ground-truth counterpart.The generated phrase “the principle of enlightenment” does not make sense in context.Source: §1 · pp. 1–2
  • The authors propose QPP and QAP rewards, applied through reinforcement learning, to encourage paraphrase agreement and answerability.QPP estimates whether a generated question paraphrases the ground-truth question; QAP estimates whether the given answer correctly answers the generated question.Source: §1 · p. 2
  • Because traditional metrics can miss answerability and valid paraphrases, the paper evaluates whether QG systems can mimic human annotators producing QA training data.The semantics-reinforced system performs best on this QA-based evaluation, but its improvement is relatively minor compared with gains on other QG metrics.Source: §1 · p. 2
  • The proposed approach achieves new state-of-the-art QG performance and improves both BiDAF and BERT QA baselines, even without introducing new articles.The QA gains are 1.69/1.27 EM/F1 points for BiDAF and 1.19/0.56 for BERT; without new articles, they are 1.51/1.13 and 0.95/0.13, respectively.Source: §1 · p. 2
  • The paper generates synthetic QA pairs from existing articles or new Wikipedia articles, then uses a QAP-based filter and mixing mini-batch training to use them.Directly mixing synthetic and ground-truth data does not improve QA performance; the reported improvements include settings without new articles.Source: §1 · p. 2

2 Related Works

Prior QG work moves from rule-based transformations to neural models using answer sentences or paragraph context, while reinforcement learning and QG-based QA augmentation explore other approaches. The paper distinguishes its methods through semantic rewards, QA-based evaluation, and synthetic-data strategies.

2 Related Works

  • Neural QG replaced earlier rule-based transformations, first using answer sentences and later leveraging whole-paragraph context.Prior approaches either extracted additional paragraph information or directly used the whole paragraph as input.Source: §2 · p. 2
  • Earlier reinforcement-learning QG work used evaluation metrics or language-model and QA rewards, but reported limited or no quality gains.Yuan et al. (2017) used QA accuracy, while this paper uses QA probability; Hosking and Riedel (2019) reported that none of the rewards they compared improved question quality.Source: §2 · p. 3
  • Prior QG-based semi-supervised QA included joint learning, GANs, self-training, and cloze-pair pretraining, with some methods reducing ground-truth data or simplifying span prediction.The paper instead follows back-translation to augment SQuAD training data.Source: §2 · p. 3
  • This work adds a data filter and mixing mini-batch training, and differs from concurrent SQuADv2.0 work by generating questions from existing articles.The filter removes poorly generated examples, while the training strategy uses synthetic data alongside ground-truth data.Source: §2 · p. 3

3 Question Generation

The section builds a paragraph-and-answer-conditioned QG model, then adds paraphrase- and answerability-based rewards to address semantic drift. It also proposes evaluating QG through downstream QA performance.

3.1 Base Model

  • The base model combines word, answer, POS, and NER representations with contextual word vectors, a bidirectional LSTM encoder, and gated self-attention.The authors report that these additions surpass Zhao et al. (2018) by significant margins.Source: §3.1 · pp. 3–4
  • A two-layer unidirectional LSTM decoder attends to encoded context at each step, while a pointer network enables copying input words.The target-word probability is computed by a maxout neural network.Source: §3.1 · p. 4

3.2 Semantics-Reinforced Model

  • QPP rewards paraphrase probability from a pretrained classifier, treating semantic similarity more fairly than surface-overlap metrics.The QPC model is trained on Quora Question Pairs and scores generated questions against ground-truth questions.Source: §3.2 · p. 4
  • QAP rewards the probability that a pretrained QA model recovers the answer from a sampled question, supporting novel generation while enforcing answerability.The authors caution that greedy copying near the answer can yield high QAP rewards while weakening question generation; they use a lower QAP mixing ratio for more teacher-forcing regularization.Source: §3.2 · pp. 4–5
  • The two semantics-enhanced rewards improve QG separately or jointly, and the resulting model achieves new state-of-the-art QG performance.The rewards are designed to regularize generation toward semantically valid questions.Source: §3.2 · pp. 4–5

3.3 QA-Based QG Evaluation

  • The QA-based evaluation trains a QA model on questions generated for answer spans in unlabeled Wikipedia paragraphs, then scores it on SQuAD development data.Higher QA performance is taken as evidence that the QG model better mimics human question-asking ability.Source: §3.3 · p. 5

4 Semi-Supervised Question Answering

The section builds synthetic QA data from existing or new articles and introduces filtering and balanced mini-batch training to use it alongside human-labeled data.

4.1 Synthetic Data Generation

  • Synthetic questions come from existing SQuAD context-answer pairs or HarvestingQA paragraphs with answer spans, avoiding the need to build an answer extractor.For existing pairs, all beam-search outputs are kept; for HarvestingQA, the authors use its paragraphs and answer-span labels but generate questions with their own QG models.Source: §4.1 · p. 6

4.2 Synthetic Data Usage

  • The QAP filter removes synthetic pairs with answer-given-question probability below a dataset-tuned threshold ε before augmentation.The authors treat low-QAP pairs as likely mismatches.Source: §4.2 · p. 6, Figure 4 · p. 6
  • 20.76/24.20/48.91 BLEU4/METEOR/ROUGE-L was achieved by QPP&QAP in the reversed dev-test setup.The authors note this was the reversed setup rather than the original one.Source: §4.2 · p. 6
  • Mixing mini-batches combine equal amounts of human-labeled and synthetic data, keeping ground-truth gradients in training regardless of dataset-size ratio.This is proposed because fine-tuning after synthetic-data pretraining can cause catastrophic forgetting when the synthetic set is small.Source: §4.2 · pp. 6–7

5 Experiment Setup

Experiments use SQuAD and HarvestingQA for question generation and QA-based evaluation, with SQuADv1.1 as the semi-supervised QA task.

5 Experiment Setup

  • SQuADv1.1 supplies the base QA task, with its development set split in half for development and testing; QG experiments use SQuAD and HarvestingQA.HarvestingQA examples are sampled at 10%–100% in the experiments.Source: §5 · p. 7
  • QG is evaluated with BLEU4, METEOR, ROUGE-L, Q-BLEU1, and pairwise MTurk judgments; QA evaluations use Exact Match and F1.Table 2 covers pairwise human evaluation of the baseline and QPP&QAP model, while Table 3 reports QA-based evaluation scores.Source: §5 · p. 7, Table 2 · p. 7, Table 3 · p. 7

6 Results

Across QG and semi-supervised QA experiments, semantics-enhanced rewards yielded state-of-the-art QG results, while generated data improved BiDAF and BERT QA, including without new articles.

6.1 Question Generation

  • QPP and QAP separately significantly improved all metrics over the baseline, while multi-reward optimization performed best on BLEU4, METEOR, ROUGE-L, and Q-BLEU1.The baseline itself improved over the previous best QG system, with POS/NER features, contextualized vectors, and a tied output projection.Source: §6.1 · pp. 7–8
  • Both the baseline and best QG model improved synthetic-data QA performance, but the best model’s advantage over baseline was minor.Thus, stronger QG metrics did not guarantee significantly better question annotation ability.Source: §6.1 · p. 8

6.2 Semi-Supervised Question Answering

  • The QAP-based filter significantly improved QA performance with synthetic data alone; gains in semi-supervised QA were smaller but stable.Table 4 reports QA development-set performance for filtering examples below a QAP threshold.Source: §6.2 · p. 8, Table 4 · p. 8
  • 1.51/1.13 absolute points improved the BiDAF-QA testing results with Beam10, while +H8 with new articles improved them by 1.69/1.27 points.Combining Beam10 and H8 did not outperform either setting used separately.Source: §6.2 · p. 8, Table 5 · p. 8
  • Beam sizes and diverse beam search performed similarly, while increasing HarvestingQA data from H1 to H10 saturated around H2–H4.The authors conjectured that synthetic-data gains may be limited by QG quality; with no or less new data, their methods improved more than Dhingra et al. (2018) over a stronger baseline.Source: §6.2 · p. 9

6.3 QG and QA Results with BERT

  • BERT-based QPP/QAP rewards achieved new state-of-the-art QG performance on both traditional metrics and QA-based evaluation.The BERT-QG baseline also improved on the previous ELMo-QG baseline.Source: §6.3 · p. 9, Table 7 · p. 9
  • 1.19/0.56 absolute points improved the BERT-QA testing results, and gains without new articles were 0.95/0.13 absolute points.The method used the QAP-reinforced BERT-QG model.Source: §6.3 · p. 9, Table 8 · p. 9
  • The QAP-reinforced BERT-QG model also improved the strong BERT-QA baseline through the same semi-supervised QA method.The reported gains were smaller than for the BiDAF-QA baseline.Source: §6.3 · p. 9

7 Conclusion

The paper combines semantics-enhanced QG rewards and QA-based evaluation with synthetic-data strategies for semi-supervised QA, reporting improvements over BiDAF and BERT baselines even without new articles.

7 Conclusion

  • Two semantics-enhanced rewards regularize QG models to generate semantically valid questions.The paper also introduces a QA-based evaluation of a QG model’s ability to mimic human annotators in generating QA training data.Source: §7 · p. 9
  • The QG system augments QA datasets through two synthetic-data generation methods, a data filter, and mixing mini-batch training.Source: §7 · p. 9
  • The approach improved both BiDAF and BERT QA baselines, even without introducing new articles.Source: §7 · p. 9

A.1 Dataset

The appendix describes the QG and QA datasets and the data splits and synthetic-data sources used in the experiments.

A.1 Dataset

  • The QG dataset derives from SQuADv1.1, with 10% of training articles sampled for testing because the original test set was not open.The original development set remained the validation set.Source: §A.1 · p. 12
  • For QA-based QG evaluation, HarvestingQA paragraphs with pre-extracted answer spans supplied inputs for QG models acting as question annotators.The models’ generated questions were used to create synthetic datasets for QA training.Source: §A.1 · pp. 12–13
  • The QA experiments used SQuADv1.1 and split its original development set in half for validation and testing to retain the full training set.Source: §A.1 · p. 13
  • Semi-supervised QA used either beam-search questions for SQuAD training pairs or questions generated for HarvestingQA paragraphs with pre-extracted answer spans.HarvestingQA subsets from 10% to 100% were labeled H1–H10.Source: §A.1 · p. 13

A.2 Evaluation Metrics

The appendix specifies the automatic metrics used to evaluate QG and QA performance.

A.2 Evaluation Metrics

  • QG evaluation used BLEU4, METEOR, ROUGE-L, Q-BLEU1, QPP and QAP rewards, and pairwise human comparison.Q-BLEU1 was selected from the Q-metrics because it had the highest reported correlation with human judgments on SQuAD.Source: §A.2 · p. 13
  • QA performance was evaluated using Exact Match (EM) and F1.Source: §A.2 · p. 13

A.3 Human Evaluation

The human evaluation compared the baseline with the QPP&QAP multi-reward model using anonymized generated-question pairs. Annotators judged answerability, contextual sense, and overall quality without seeing ground-truth questions.

A.3 Human Evaluation

  • Annotators were US-based, had approval rates above 98%, and had at least 10,000 approved HITs.The study used pairwise human evaluation on Amazon Mechanical Turk.Source: §A.3 · p. 13
  • They judged whether each question was answerable by the given answer, made sense in context, and was fluent, non-ambiguous, and semantically compact.The input paragraph showed the answer in bold, and the two model questions were randomly shuffled to anonymize model identities.Source: §A.3 · p. 13
  • Ground-truth questions were withheld to prevent annotators from relying on simple matching with the reference.Source: §A.3 · p. 13

B Implementation Details

The implementation details describe the QG, QA, and paraphrase-classification models and report example outputs. The examples include useful paraphrases as well as questions that can limit semi-supervised QA performance.

B Implementation Details

  • ELMo-QG uses 2-layer LSTMs with hidden size 600, is pretrained with teacher forcing before mixed-loss fine-tuning, and uses beam search with size 10.Its tuned γ_qpp and γ_qap are 0.99 and 0.97; BERT-QG uses WordPiece tokenization and the same reward values.Source: B Implementation Details · p. 13
  • BiDAF-QA uses BiDAF with self-attention, GRUs, and ELMo, whereas BERT-QA fine-tunes uncased BERT-base on QA datasets.Source: B Implementation Details · pp. 13–14
  • ELMo-QPC embeds questions with ELMo, encodes them with separate bidirectional LSTMs, and predicts paraphrase status from pooled question representations.The model is trained on Quora Question Pairs.Source: B Implementation Details · p. 14
  • BERT-QPC obtained 90% accuracy on QQP development, while ELMo-QPC obtained 86% accuracy.Source: B Implementation Details · p. 15

C Examples

  • The SQuAD examples include paraphrased or novel questions that enrich the dataset, alongside easier or wrong questions that limit semi-supervised QA performance.On HarvestingQA, the models produce better questions than Du and Cardie (2018), but still generate some wrong questions.Source: Figure 5 · p. 14, C Examples · p. 15
Loading 1909.06356v1…