Source-linked AI summary
Synthetic QA Corpora Generation with Roundtrip Consistency
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, Michael Collins
TL;DR
QA needs auxiliary tasks that are better aligned with question answering and can be built from widely available text, while generated QA-pair accuracy remains relatively unexplored. The paper generates and filters synthetic QA data using roundtrip consistency, then pretrains QA models on it, improving SQuAD2 and NQ performance, including a new state of the art on NQ.
Problem
Existing auxiliary tasks may be poorly suited to QA, and the accuracy of automatically generated question-answer pairs remains relatively unexplored.
Method
The method generates synthetic QA examples by applying separately trained question-generation, answer-extraction, and roundtrip-filtering models to unlabeled passages, using publicly available BERT models for reproducible generation.
Results
Pretraining on synthetic data significantly improves SQuAD2 and NQ performance, achieves a new state of the art on NQ, and brings the strongest SQuAD2 result within 0.1−0.4% of human performance.
Takeaways & Limitations
Roundtrip-consistent synthetic QA data provides a demonstrated way to improve QA systems across SQuAD2 and NQ.
Takeaways & Limitations
The paper offers only a speculative, preliminary account of why roundtrip consistency improves QA accuracy, leaving formal grounding for future work.
Abstract
from arXiv · showhide
We introduce a novel method of generating synthetic question answering corpora by combining models of question generation and answer extraction, and by filtering the results to ensure roundtrip consistency. By pretraining on the resulting corpora we obtain significant improvements on SQuAD2 and NQ, establishing a new state-of-the-art on the latter. Our synthetic data generation models, for both question generation and answer extraction, can be fully reproduced by finetuning a publicly available BERT model on the extractive subsets of SQuAD2 and NQ. We also describe a more powerful variant that does full sequence-to-sequence pretraining for question generation, obtaining exact match and F1 at less than 0.1% and 0.4% from human performance on SQuAD2.
1 Introduction
The paper constructs QA-specific auxiliary tasks from unlabeled passages by generating and validating synthetic question-answer-context triples. Pretraining on these roundtrip-consistent examples improves performance on SQuAD2 and NQ.
- 1 Introduction: QA-specific auxiliary tasks generate millions of synthetic question-answer-context triples from unlabeled passages for pretraining.The resulting model is then finetuned on a labeled QA dataset.
- 1 Introduction: Each synthetic example starts with an extractive answer sampled from a passage, followed by question generation conditioned on the answer and context.A separate model then predicts an extractive answer from the generated question and context.
- 1 Introduction: Examples are retained only when the predicted answer A′ matches the original answer A, enforcing roundtrip consistency before emission.Table 1 illustrates this filtering decision.
- 1 Introduction: Separate models are trained for answer extraction, question generation, and question answering, then applied sequentially to many unlabeled passages.The procedure uses the generated triples for synthetic-data pretraining.
- 1 Introduction: Pretraining on the synthetic data significantly improves results on SQuAD2 and NQ, achieving a new state of the art on NQ.The paper reports improvements on both challenging datasets.
2 Related Work
Prior work uses generated question-answer pairs and consistency-based synthetic data, but validating generated QA pairs remains relatively unexplored. This paper is also related to approaches that model question-answer distributions directly.
- 2 Related Work: Question generation has been used to improve QA systems, with large gains reported in low-resource settings with few gold labeled examples.These approaches use generated question-answer pairs as additional training data.
- 2 Related Work: Validating and improving the accuracy of generated QA pairs has received comparatively little study.The paper identifies this as an underexplored aspect of synthetic QA data.
- 2 Related Work: Dual learning and back-translation improve machine translation by modeling consistency across both translation directions.Back-translation adds synthetically generated parallel data and has achieved state-of-the-art results in supervised and unsupervised settings.
- 2 Related Work: Lewis and Fan model the joint distribution of questions and answers given context, whereas this work generates synthetic data for pretraining.The paper suggests combining the two approaches as future work.
3 Model
The paper models answer extraction, question generation, and question answering as complementary components for creating roundtrip-consistent synthetic QA data. It uses these models sequentially, with variants based on fine-tuned BERT encoders and fully pretrained sequence-to-sequence generation.
- 3 Model: Three models represent answer extraction p(a|c; θA), question generation p(q|c, a; θQ), and question answering p(a|c, q; θA′).The models are trained from context-question-answer datasets and used as the core components of the synthetic-data pipeline.
- 3.1 Question (Un)Conditional Extractive QA: Answer spans are scored jointly in the question-unconditional extractor because contexts often contain multiple acceptable answers.Independent start and end scoring cannot determine the span reliably when the question is unavailable.
- 3.2 Question Generation: Fine-tuning Only: The fine-tuning-only question generator adapts publicly available BERT without additional pretraining or newly initialized parameters.It is trained by fine-tuning BERT on extractive subsets of datasets such as SQuAD2 and NQ, using a left-to-right language-model formulation with one encoder stack.
- 3 Model: Synthetic examples are emitted only when the generated question and re-extracted answer reproduce the sampled answer, enforcing roundtrip consistency.The procedure samples A from C, generates Q from A and C, predicts A′ from Q and C, and retains (C, Q, A) when A and A′ match.
- 3.3 Question Generation: Full Pretraining: The alternative generator fully pretrains a sequence-to-sequence Transformer, retaining a BERT-pretrained encoder and training a decoder to output the next sentence.During fine-tuning, the input is (C, A) and the output is Q; synthetic triples are sampled with beam and Monte Carlo search before consistency filtering.
- 3.4 Why Does Roundtrip Consistency Work?: Roundtrip consistency is presented as a plausible constrained-likelihood interpretation, but its formal explanation remains an open question for future work.The supplementary argument connects pretraining on generated triples with constraining an unlabeled-data consistency measure β(θA′) to exceed γ.
4 Experiments
Experiments evaluate synthetic-data pretraining on SQuAD2 and NQ, using fine-tuning-only and full-pretraining settings. Results show gains from roundtrip-filtered data, strong SQuAD2 performance, and substantial NQ improvement, while data-source benefits differ by task.
- Experimental setup: The experiments use SQuAD2 and NQ, with NQ's 1B-word training text serving as unlabeled data.Fine-tuning-only experiments train separate model triples on each dataset's extractive subset and generate synthetic examples from unlabeled NQ windows.
- Experimental setup: 2.4M and 3.2M synthetic positive instances were generated from 8M unlabeled 512-token NQ windows using SQuAD2-trained and NQ-trained models, respectively.Each window produced one example from each model set, with roundtrip consistency used for filtering.
- Experimental setup: The full-pretraining setting created a 50M-example, roundtrip-filtered corpus after pretraining question generation on all BERT pretraining data.Models were fine-tuned on SQuAD2, with both single-model and six-model ensemble settings evaluated.
- Results: Pretraining on synthetic SQuAD2 and NQ data significantly improves fine-tuned performance, with NQ short-answer headroom reduced 50% to a single human and 10% to a human ensemble.Synthetic-data pretraining benefits both evaluated datasets, but the reported human-relative comparison is for NQ short-answer performance.
- Results: The union of SQuAD2- and NQ-style synthetic data benefits SQuAD2 but does not improve NQ results.This shows that broader synthetic-data sources do not transfer uniformly across the two tasks.
- Results: The full-pretraining ensemble achieves the highest listed SQuAD2 EM and F1, within 0.1−0.4% of human performance.It was the third-best model on the SQuAD2 leaderboard as of 5/31/19.
- Results: Generated pairs differ by source: SQuAD2 models target specific entity properties, whereas NQ models pursue popular themes and combine information across passages.The comparison uses the same passage to generate both styles of question-answer pair.
5 Conclusion
The paper presents a synthetic QA-instance generation method and reports improvements on both SQuAD2 and NQ. It also proposes formal grounding of the roundtrip method as future work.
- Conclusion: The paper presents a novel method for generating synthetic QA instances and demonstrates improvements on SQuAD2 and NQ.The authors additionally propose formal grounding of the method as a future research direction.
A Supplementary Material: a Sketch of a Formal Justification for the Approach
The sketch proposes combining labeled-data likelihood with an auxiliary roundtrip-consistency measure on unlabeled examples. Constraining this measure may reduce the effective parameter space while retaining well-performing parameters, though the formal justification remains speculative.
- The section is intentionally speculative and presents possible directions toward a more formal grounding of the roundtrip method.It is inspired by work on learning with labeled and unlabeled data.
- The proposed objective combines labeled-example log-likelihood with a constraint requiring roundtrip consistency β(θA′) on unlabeled data to exceed γ.γ can be estimated using tuning or development data.
- The constraint β(θA′) ≥ γ is intended to eliminate substantial parameter space and reduce sample complexity while retaining good parameter values that fit labeled data.The sketch connects this intuition to a potentially less complex hypothesis set H′ than H.
- The auxiliary function β(θA′) evaluates consistency using triples derived from unlabeled data, where a significant proportion are assumed correct question-answer pairs.One proposed form aggregates a function f over the answer probabilities p(a(j)|q(j), c(j); θA′).
- The function f is chosen to reward high answer probabilities, with log probability and threshold-margin functions given as examples.These choices encourage parameters to fit the unlabeled triples.
- The optimization can use gradient ascent on a weighted objective or pretrain on β(θA′) before fine-tuning on labeled-data likelihood.The relative weight λ may be estimated by cross-validation, while the pretraining approach is presented as related to the paper’s algorithms.