PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems

Siamak Shakeri, Cicero Nogueira dos Santos, Henry Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, Bing Xiang

arXiv:2010.06028v1cs.CL

TL;DR

Synthetic QA generation has often produced limited downstream gains, while recent successful approaches typically rely on multiple costly modules. This paper uses one encoder-decoder to generate answers and questions and scores them with generation likelihood; QAGen2S yields the best overall results among its variants and baselines across four target domains.

  • Problem

    Earlier question-generation strategies had limited or no success improving downstream QA, while recent synthetic-data approaches typically use separate span detection, question generation, and filtering modules.

    Source: §1 · p. 1

  • Method

    A fine-tuned encoder-decoder generates answers and questions from a passage, and generation likelihood serves as a filtering score without a separate filtering model.

    Source: §1 · p. 1, §2.2 · p. 2, §2.4 · p. 3

  • Results

    QAGen2S provides the best overall results across four target datasets, improving the SQuAD baseline on NQ by more than 8 points in EM and more than 7 points in F1.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments show that proper decoding with a single transformer-based model can achieve significant improvements in domain adaptation of reading-comprehension models.

    Source: §5 · p. 9

    The likelihood filtering score is noisy, and selecting only samples with F1 scores of 1 requires a high threshold that drops most samples.

    Source: Correlation between LM and F1 Scores · p. 9

Abstract

from arXiv · show

We propose an end-to-end approach for synthetic QA data generation. Our model comprises a single transformer-based encoder-decoder network that is trained end-to-end to generate both answers and questions. In a nutshell, we feed a passage to the encoder and ask the decoder to generate a question and an answer token-by-token. The likelihood produced in the generation process is used as a filtering score, which avoids the need for a separate filtering model. Our generator is trained by fine-tuning a pretrained LM using maximum likelihood estimation. The experimental results indicate significant improvements in the domain adaptation of QA models outperforming current state-of-the-art methods.

1 Introduction

The introduction presents end-to-end synthetic QA generation as a response to prior systems’ limited downstream gains and the complexity of multi-stage pipelines. It proposes one model to generate answer-question pairs and reports improved domain adaptation across four target datasets.

1 Introduction

  • Prior synthetic QA systems typically use separate span-detection, question-generation, and filtering modules, whose errors can propagate and whose large transformer networks make each module expensive.Span detection normally combines a pretrained QA model with handcrafted heuristics, while filtering uses another reading-comprehension model.Source: §1 · p. 1
  • The proposed single encoder-decoder generates both answers and questions token-by-token, using generation likelihood as a filtering score instead of a separate filtering model.The generator is fine-tuned from a pretrained language model using maximum likelihood estimation.Source: §1 · p. 1
  • The experiments compare three generation orders: answer then question, question then answer, and question followed by answer generation in a second pass.The two-step variant reuses the same encoder-decoder and concatenates the generated question to the passage before generating the answer.Source: §1 · p. 1
  • QAGen performs better than AQGen and the baselines on all four target datasets, while QAGen2S provides the best overall results.On NQ, QAGen2S improves the SQuAD baseline by more than 8 points in EM and more than 7 points in F1; NewsQA and BioASQ each gain more than 4 points in EM.Source: §1 · p. 2

2 End-to-End Model for Question and Answer Generation and Filtering

The section formulates synthetic QA generation as conditional language modeling with an encoder-decoder, then presents three output-ordering variants and likelihood-based filtering. Decoding uses sampling to encourage diverse pairs, with extractive answers retained.

2.1 Enc-Dec Conditional Language Models

  • An encoder-decoder conditional language model encodes context with bidirectional self-attention and autoregressively decodes tokens while attending to prior outputs and encoded context.Conditional generation incorporates context c when modeling the output sequence.Source: §2.1 · p. 2

2.2 Question-Answer Generation

  • The model learns the joint conditional distribution of an answer span and question given a passage by fine-tuning an encoder-decoder with maximum likelihood estimation.The training set consists of passage, question, and answer triples.Source: §2.2 · p. 2
  • AQGen generates answers before questions, conditioning questions on the passage and generated answer; QAGen reverses the order, conditioning answers on the passage and generated question.The output ordering determines which generated component conditions the other.Source: §2.2 · pp. 2–3
  • QAGen 2S generates a question, concatenates it with the passage, then uses the same encoder-decoder in a second pass to generate the answer.Control codes indicate whether the decoder should generate a question or an answer.Source: §2.2 · p. 3

2.3 Decoding

  • Topk+Nucleus sampling uses k=20 and a 95% probability-mass cutoff; QAGen2S answer generation instead uses greedy decoding, and pairs with non-extractive answers are discarded.The authors report that 10% to 15% of generated samples were dropped because their answers did not occur in the input passage.Source: §2.3 · p. 3

2.4 Filtering

  • LM filtering ranks sampled QA pairs by generation likelihood, avoiding a separate RC filtering model; QAGen and QAGen2S use answer-only scores, while AQGen uses answer and question scores.For AQGen, likelihood summation worked slightly better than averaging in experiments.Source: §2.4 · pp. 3–4

3 Related Work

Prior question-generation work often treated question generation as a standalone task, while synthetic QA approaches using pretrained language models improved reading comprehension but relied on additional models.

3 Related Work

  • Earlier neural question-generation work commonly evaluated standalone question quality, while some studies reported downstream gains for answer sentence selection or reading comprehension under limited labeled-data conditions.Alberti et al. (2019) and Puri et al. (2020) used pretrained language models for reading-comprehension gains but required two additional BERT models for answer-span identification and filtering.Source: §3 · p. 5

4 Experimental Setup and Results

The experiments evaluate synthetic QA generation across four target domains and compare generator variants, filtering, sampling, and dataset sizes. QAGen and QAGen2S outperform baselines broadly, with gains varying by domain and remaining below supervised target-domain performance in some settings.

4.4 Domain Adaptation Results

  • QAGen and QAGen2S outperform SQuAD-only, unsupervised domain-adaptation, and three-stage QGen baselines; BioASQ is an exception to the reported point-wise gains over Nishida et al. (2019).For BioASQ, Nishida et al. report 4.3 EM and 4.2 F1 points versus 4 EM and 2.2 F1 points with the proposed method.Source: §4.4 · p. 6, Table 2 · p. 7
  • Same-domain synthetic passages usually yield the best or near-best target-domain scores, while mixing all four domains gives the best F1 results across datasets.Using all-domain synthetic data with SQuAD improves SQuAD dev performance by 3.8 EM and 2.7 F1 points.Source: §4.4 · p. 7, Table 3 · p. 7

4.5 Comparison of AQGen, QAGen and QAGen2S models

  • QAGen outperforms AQGen, and QAGen2S performs best overall because it permits bidirectional attention between passage and question.The authors attribute QAGen's advantage to answer generation conditioned on the question, which improves answer spans for downstream RC.Source: §4.5 · p. 7

4.6 Ablation Studies Sampling Design Choices

  • Topk+Nucleus sampling outperforms beam search; beam search produces less diverse samples, fewer distinct spans, and minor variations of the same question.The comparison uses QAGen2S with LM filtering on Natural Questions.Source: Sampling Design Choices · p. 8, Table 4 · p. 8
  • Sampling 10 pairs per document gives the best target-domain EM/F1; larger samples increase cost without improvement, while top-ranked pairs can become too similar.The RC model uses the top five LM-scored samples per passage.Source: Sampling Design Choices · p. 8

LM Filtering

  • LM filtering improves F1/EM in most experiments, especially for AQGen, which generates lower-quality data than the other variants.Five of ten sampled pairs per passage are selected in the described configuration.Source: LM Filtering · p. 8, Table 5 · p. 8

Correlation between LM and F1 Scores

  • Answer likelihood scores correlate strongly with round-trip F1, but are noisy; selecting only samples with F1 of 1 requires a high threshold that drops most samples.Generated samples are sorted by answer LM score and grouped into contiguous buckets of 200 to measure average F1.Source: Correlation between LM and F1 Scores · pp. 8–9, Figure 5 · p. 9

Impact of Synthetic Dataset Size

5 Conclusions

The paper concludes that a single transformer-based model can generate question–answer pairs and that suitable decoding yields significant improvements in domain adaptation. LM-score filtering improves pair quality, though it remains behind round-trip filtering in some target domains, and generated question types are limited by SQuAD training.

5 Conclusions

  • The approach generates question–answer pairs with a single transformer-based model.The conclusion identifies improving LM-score-based filtering as a future direction.Source: §5 · p. 9
  • Suitable decoding with the single-model generator achieves significant improvements in domain adaptation of reading-comprehension models.The authors also conclude that LM filtering improves synthetic question–answer pair quality, though it still trails round-trip filtering in some target domains.Source: §5 · p. 9
  • The generator produces diverse, high-quality, challenging target-domain samples, but its question types remain limited to those found in SQuAD.The authors attribute this boundary to training the generative models on SQuAD and suggest adapting them to target-domain question types as future work.Source: §5 · p. 9

A Additional Details Regarding the Datasets Used

The experiments use several reading-comprehension datasets and corresponding unlabeled target-domain passages, with dataset choice reflecting domain and question characteristics. The appendix also specifies validation-set conventions.

A Additional Details Regarding the Datasets Used

  • SQuAD 1.1 supplies generator training data and in-domain supervised data, while target-domain passages come from Wikipedia, CNN/Daily Mail, PubMed abstracts, and IMDB movie plots.The datasets include Natural Questions, NewsQA, BioASQ, and DuoRC; NewsQA excludes unanswerable or non-consensus questions in the MRQA version used.Source: A Additional Details Regarding the… · p. 12
  • Natural Questions questions have less answer overlap than SQuAD questions, and its passages can contain HTML tables and tags.DuoRC ParaphraseRC was designed for minimal question–answer lexical overlap.Source: A Additional Details Regarding the… · p. 12
  • The appendix uses the same validation sets as Nishida et al. for the out-of-domain tasks, except for DuoRC, which uses the MRQA-formatted development set.Source: A Additional Details Regarding the… · p. 12

B.1 Performance on SQuAD 1.1 with Different Filtering Approaches

On SQuAD 1.1, adding synthetic data generated from IMDB passages improves development performance, with QAGen2S performing best among the three generator variants. The appendix reports that the filtering conclusions also hold for NewsQA and DuoRC.

B.1 Performance on SQuAD 1.1 with Different Filtering Approaches

  • Strong source-domain performance matters for applications requiring strong performance on both source and target domains.Source: §B.1 · p. 12
  • 3.1 EM and 2.2 F1 gains are achieved with QAGen2S when synthetic IMDB samples are added to SQuAD training data.The passage also reports that synthetic-only samples from the same model outperform the SQuAD baseline.Source: §B.1 · p. 12
  • QAGen2S outperforms QAGen, which in turn outperforms AQGen on SQuAD 1.1.Source: §B.1 · p. 12

B.2 Comparison of Using Filtering vs. No Filtering

  • The appendix reports that the filtering conclusions from Section 4.7 also hold for NewsQA and DuoRC.Table 8 presents results comparing LM filtering with no filtering across target domains.Source: §B.2 · p. 12

B.3 Impact of Language Model Score Pooling

The appendix compares sum and average pooling for aggregating language-model scores used in filtering. Summation generally performs better, while the authors speculate that averaging favors longer, potentially incorrect pairs.

B.3 Impact of Language Model Score Pooling

  • The pooling comparison evaluates sum and average aggregation of token scores for question–answer pairs.Source: §B.3 · p. 12
  • Summation generally outperforms averaging when aggregating language-model token scores for filtering.Source: §B.3 · p. 12
  • The authors speculate that average pooling favors longer pairs, which are more likely to be incorrect, while summation favors shorter pairs during filtering.Source: §B.3 · p. 13

C Examples of Generated Samples

The examples contrast decoding strategies and show how answer likelihood scores can flag incorrect generated pairs. Beam search produces repetitive samples that underperform Topk+Nucleus samples for downstream RC.

C.1 Illustration of Answer LM Score

  • The last two Topk+Nucleus samples are incorrect and would be filtered by the LM approach; the irrelevant final answer has a considerably lower answer LM score.The samples are generated with QAGen2S from a randomly selected Natural Questions passage.Source: §C.1 · p. 13
  • In the illustrated samples, answer LM scores are used to rank generated question-answer pairs in decreasing order.The examples include beam search and Topk+Nucleus decoding.Source: §C.1 · p. 13
  • Beam search samples have high likelihood but lack diversity, and RC models trained on them underperform those trained on Topk+Nucleus samples.The passage attributes this to beam search's high repetition.Source: §C.1 · p. 13

C.2 Comparison of Generated Samples by AQGen, QAGen and QAGen2S

The examples show quality and diversity differences among the generators, residual errors after sampling, and QAGen2S outputs on HTML-formatted tables. In the table example, QAGen2S produces mostly correct pairs where QGen selects spans containing HTML tags.

C.2 Comparison of Generated Samples by AQGen, QAGen and QAGen2S

  • QAGen2S generates more diverse, longer answer spans than QAGen, while AQGen samples are lower quality and only 3 of 6 correctly generated samples are correct pairs.The example also shows more repeated spans from QAGen than QAGen2S.Source: §C.2 · p. 13
  • Topk+Nucleus improves diversity, but repetitions and incorrect pairs remain; the authors believe LM-score filtering discards the vast majority of incorrect pairs, leaving room to improve the generators.Source: §C.2 · p. 13

C.3 Question Answers from Table

  • On an HTML-formatted table passage, QGen selects spans containing HTML tags, whereas QAGen2S produces high-quality spans and questions, with one incorrect sample.The authors suggest question-conditioned span generation sharply lowers the likelihood of spans containing spurious tokens such as HTML tags.Source: §C.3 · p. 14

D Training and Platform Details

The generative models and downstream RC models were trained on GPU instances with fixed hyperparameters. Training details include epoch durations and the point at which the best generative model was usually reached.

D Training and Platform Details

  • Generative-model epochs took 2 to 3 hours on 3 GPUs, and the best model was usually achieved within the first two epochs.Experiments used p3.8xlarge, p3.16xlarge, and p3dn.24xlarge instances.Source: D Training and Platform Details · p. 14
  • Downstream RC-model training epochs took 2 to 12 hours on 1 GPU, depending on the average length of target-domain passages.Synthetic samples were combined with SQuAD training data and randomly shuffled.Source: D Training and Platform Details · p. 14
  • Hyperparameters for both generative and downstream RC models were fixed, with tuning limited to those specified in the paper.Source: D Training and Platform Details · p. 14
  • The CNN/Daily Mail samples and Natural Questions table samples are sorted by LM scores in their respective tables.The supplied captions identify these as Tables 11 and 12.Source: Table 11 · p. 15, Table 12 · p. 16
Loading 2010.06028v1…