PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Question Answering Models From Synthetic Data

Raul Puri, Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro

arXiv:2002.09599v1cs.CLcs.AI

TL;DR

Limited labeled data and a persistent quality gap constrain synthetic question-answer generation for QA. The paper scales pretrained generative models and synthetic data, achieving strong SQuAD performance using entirely synthetic training data, including 88.4 EM and 93.9 F1 on SQuAD1.1.

  • Problem

    Limited and costly labeled data, along with a persistent quality gap between synthetic and human-generated QA pairs, motivates improved synthetic data generation.

  • Method

    The paper uses a three-step answer generation, question generation, and filtration pipeline with large pretrained GPT-2 models, overgeneration, and synthetic corpora.

  • Results

    88.4 EM and 93.9 F1 are achieved on SQuAD1.1 using QA data generated entirely from a synthetic corpus, while scaling and improved methods also yield a 1.7-point SQuAD2.0 EM gain over the fully supervised baseline.

  • Takeaways & Limitations

    QA models can be trained from fully synthetic questions and answers and can surpass models trained on human-labeled data on SQuAD1.1.

Abstract

from arXiv · show

Question and answer generation is a data augmentation method that aims to improve question answering (QA) models given the limited amount of human labeled data. However, a considerable gap remains between synthetic and human-generated question-answer pairs. This work aims to narrow this gap by taking advantage of large language models and explores several factors such as model size, quality of pretrained models, scale of data synthesized, and algorithmic choices. On the SQuAD1.1 question answering task, we achieve higher accuracy using solely synthetic questions and answers than when using the SQuAD1.1 training set questions alone. Removing access to real Wikipedia data, we synthesize questions and answers from a synthetic corpus generated by an 8.3 billion parameter GPT-2 model. With no access to human supervision and only access to other models, we are able to train state of the art question answering networks on entirely model-generated data that achieve 88.4 Exact Match (EM) and 93.9 F1 score on the SQuAD1.1 dev set. We further apply our methodology to SQuAD2.0 and show a 2.8 absolute gain on EM score compared to prior work using synthetic data.

1. Introduction

The paper addresses the gap between synthetic and human-generated QA data by improving question-and-answer generation with larger pretrained models and scalable filtering. Its synthetic data can match or exceed supervised QA performance, including when both corpus and QA examples are model-generated.

  • The paper targets the persistent quality gap between synthetic and human-generated question-answer pairs caused by the difficulty of generating natural-language questions and answers.
  • The approach scales pretrained GPT-2 models, uses large synthetic datasets, and applies answer generation, question generation, and filtration to improve synthetic QA data.The pipeline also uses overgeneration and filtering, while larger generative transformers improve question quality.
  • 88.4 EM and 94.1 F1 are achieved on SQuAD1.1 using purely synthetic questions and answers from unlabeled data, compared with 87.7 EM and 94.0 F1 for supervised training.Finetuning this model on real SQuAD1.1 data further reaches 89.4 EM and 95.1 F1.
  • 88.4 EM and 93.9 F1 are achieved when QA data are generated from a corpus produced by an unconditional GPT-2 model, without human language supervision or real data.This recovers 100.3% of fully supervised EM and F1 scores.
  • The work reports a 1.7 absolute gain on SQuAD2.0 EM over prior synthetic-data work by scaling models, improving pretrained models, and expanding synthetic data.
  • Answer generation is foundational to question quality, and aligning the answer distribution boosts scores by 19.8 EM points.

2. Method

The method synthesizes extractive QA training triples through answer generation, conditional question generation, and roundtrip filtration. It improves fidelity through large pretrained generators and a two-question filtration strategy.

  • Pipeline: The pipeline samples answer candidates, generates questions conditioned on answers and contexts, filters pairs with roundtrip consistency, and trains a BERT QA model.The synthesized triples are filtered before QA-model training.
  • Answer Generation: Answer candidate generation acts as a prior for the dataset’s answer distribution, which must be matched for strong dataset-specific performance.The answer generator is trained without question tokens to predict extractive spans from context.
  • Answer Generation: The answer extractor jointly models the start and end tokens of an answer span.Joint modeling performed better than independently modeling start and end tokens.
  • Question Generation: The question generator concatenates context, answer, and question tokens, using segment embeddings to distinguish input components and highlight the answer.It is trained with pretrained GPT-2 decoder models and a next-token-prediction objective.
  • Roundtrip Filtration: Roundtrip filtration retains a generated pair when a QA model’s predicted answer matches the generated answer.Generating two questions per candidate reduces losses from valid questions being discarded; each question is filtered independently.

3. Experiment Setup

The experiments use large transformer models, mixed-precision GPU training, and document-level data partitioning to separate synthetic-data generation from QA-model training.

  • Models and Data: GPT-2 question-generation models were pretrained on a 174GB corpus combining Wikipedia, OpenWebText, RealNews, and CC-Stories.The implementation relies on Megatron-LM and HuggingFace Transformers codebases.
  • Models and Data: BERT models were pretrained with n-gram masked language modeling and sentence order prediction, using GPT-2-style residual and normalization ordering.The authors report improved stability and training of larger BERT models without training instabilities and overfitting.
  • Models and Data: The study used BERT-345M and BERT-1.2B models, with the larger model configured with 24 layers, hidden size 2048, and 32 attention heads.The reported finetuning schedule used learning rate 1e-5 with cosine decay over two epochs.
  • Data Partitioning: SQuAD training documents were randomly split into equal halves: one trained generation and filtration models, while the other produced synthetic data for QA finetuning.This partitioning separates model training documents from documents used to generate QA training examples.
  • Compute: Training used mixed precision on NVIDIA V100 GPUs, with pretraining distributed across 4 to 32 DGX-2H servers.Finetuning generally required one DGX-1V, except for the 8.3B question generator, which required eight.

4. Results

Synthetic data matched or surpassed real-data performance on SQuAD1.1, while larger labeled corpora and improved generation also benefited SQuAD2.0 comparisons.

  • SQuAD1.1: Completely synthetic SQuAD1.1 data outperformed training with real data, and subsequent finetuning on real SQuAD1.1 data further boosted performance.The synthetic corpus was generated by an 8.3B GPT-2 model and question-answer pairs by 1.2B-parameter models.
  • SQuAD1.1: 1.7 EM and 1.2 F1 point boosts followed finetuning the synthetic-data model on actual SQuAD1.1 data.This result is reported for the BERT-345M QA model.
  • Data Scale: Purely synthetic-data performance followed a log-linear relationship with labeled text and began saturating at approximately 100 MB.Post-finetuning on labeled SQuAD1.1 data continued improving beyond 500 MB.
  • SQuAD2.0: The synthetic-data approach outperformed prior work on SQuAD2.0 despite generating no unanswerable questions.The advantage persisted with a similar number of questions and with more labeled data.

5. Model Scale

The paper studies how model scale and pretraining quality affect synthetic QA generation, evaluating downstream SQuAD1.1 performance across answer, question, and filtration components.

  • Model Scale: Larger models, improved pretraining, and increased scale improve synthetic-data SQuAD1.1 performance across the generation pipeline.The study tracks downstream QA performance while varying answer generation, question generation, and filtration models.
  • Model Scale: Pretraining large generative transformer models up to 8.3B parameters improves the quality of generated questions.
  • Question Generation: Question-generation scale is evaluated using ground-truth answers, without filtration, before finetuning a BERT model on the generated questions.
  • Scaling Answer Generation: Answer-generation improvements between BERT-Large and the 345M model yield 1.4 EM and 0.3 F1 gains, while scaling to 1.2B adds 0.4 F1 and 0.1 EM.
  • Question Filtration: Question filtration responds more strongly to model-size increases than to improved pretraining tasks.The authors report that pretraining-task improvements produce modest filtration gains, whereas increasing model size produces more substantive improvements.

6. Modeling Choices

The modeling-choice studies examine ablations across the synthetic-data pipeline, identifying answer generation, sampling, and filtration choices that materially affect downstream QA performance.

  • Modeling Choices: The ablation program varies one model component at a time, while noting that it does not capture second-order effects from combining hyperparameters.
  • Answer Generation: Answer generation is a significant performance bottleneck, and the selected answer generator must represent SQuAD1.1’s diverse answer distribution.
  • Answer Generation: Joint span modeling improves answer extraction over independent span modeling, reaching 79.1, while paragraph-level joint modeling performs worse than sentence-level independent modeling.
  • Answer Sampling: k = 5 answers per sentence is the reported optimum for top-k answer sampling within a p = 0.9 nucleus.
  • Question Filtration: Roundtrip consistency improves synthetic-answer training by 7.2 EM points relative to no filtration, but filtering can discard valid questions generated from ground-truth answers.
  • Question Filtration: Overgeneration with both top-k and nucleus sampling improves performance by approximately one point with and without ground-truth answers.

7. Related Work

Prior work develops question generation through rules, recurrent and transformer models, answer-consistency filtering, and methods for broader QA settings, while synthetic-data quality remains below leading performance.

  • Related Work: Early question-generation work overgenerates and reranks questions using regression models over handcrafted linguistic features.
  • Related Work: Later approaches use LSTM and seq2seq models, unified transformers, and roundtrip consistency to generate or filter synthetic QA data.
  • Answer Generation: NER- and parser-based answer selection covers only a subset of SQuAD1.1 because Named Entity Answers comprise 52% of the dataset.
  • Related Work: Prior synthetic-data systems remain below the leaderboard, motivating improved language-model quality and larger pretraining resources.
  • Related Work: Question answering research spans multihop, yes-no, trivia, analytical, conversational, generative, and unanswerable question datasets.

8. Conclusion

The paper scales synthetic QA generation to unlabeled and fully synthetic corpora, achieving strong SQuAD1.1 performance and further gains after real-data finetuning.

  • Conclusion: A 345M BERT-style model trained on synthetic QA from unlabeled Wikipedia achieves 88.4 EM on SQuAD1.1.
  • Conclusion: Finetuning that model on real SQuAD1.1 data raises EM to 89.4, a 1.7-point improvement over the fully supervised baseline.
  • Conclusion: Using synthetic text from a Wikipedia-finetuned GPT-2 model enables similar QA accuracy without directly using real data.
  • Conclusion: The fully synthetic approach requires scaling answer generators, question generators, and filtration models.

A. Samples Generated from Wikipedia Documents

The Wikipedia-derived samples pair questions with contexts covering factual entities, events, works, science, and technical topics. Several examples include explicit answers or answer-bearing context passages.

  • Sample coverage: Questions span people, fictional characters, sports results, historical events, military technology, medicine, music, and literature.Examples include Miss United Kingdom, the Phantom, Tottenham, Ernst Sekunna, the Chiyoda, cefalotin, Bon Jovi, and Othello.
  • Answer-bearing contexts: Several contexts state answers directly, including Vicki-Lee Walberg, a 2-1 Tottenham victory, Harvey armor, first-generation cephalosporin, and a Billboard Hot 100 peak of #7.These examples illustrate answer-bearing passages paired with generated questions.

B. Samples Generated from GPT-2 Documents

The GPT-2-derived samples cover graph concepts, chemistry, espionage, medieval history, comics, television, music, astronomy, theater, and highways. Their contexts vary from direct answers to partial or absent supporting text.

  • Sample coverage: Questions address topics ranging from cliques in DAGs and chemical isomers to espionage, Norman history, comics, television, music, stars, theater, and roads.The sample set is broad and mixes factual and narrative domains.
  • Presentation: Other entries provide incomplete contexts or questions without accompanying answer evidence in the displayed samples.Examples include the clique, SOE, Henry II, wedding, star, and production-style items.
  • Answer-bearing contexts: Several contexts directly support answers, including Nightwing as Dick Grayson’s identity, Get Rich or Die Tryin’ as 50 Cent’s debut album, and US 290 as a controlled-access road.These examples connect generated questions to answer-bearing synthetic contexts.
Loading 2002.09599v1…