Simple and Effective Semi-Supervised Question Answering

1 Introduction

• The approach targets the scarcity and expense of large annotated corpora, especially for domain-specific QA and low-resource settings.It hypothesizes particular benefit for factual questions lexically similar to the base documents.Source: §1 · p. 1
• The system generates cloze questions from base documents, pretrains an extractive QA model on them, and fine-tunes it using a small set of labeled question-answer pairs.The user specifies the base documents and labeled examples.Source: §1 · p. 1
• More than 50% F1 on SQuAD and TriviaQA is achieved with 1% of training data, and the system improves low-resource performance across all three datasets.The datasets are SQuAD, TriviaQA-Web, and BioASQ.Source: §1 · p. 1

2 Related Work

• GDAN uses reinforcement learning to generate questions from passages, whereas this work uses a simpler heuristic and reports superior performance to GDAN.Other natural-question generation approaches have not shown significant improvement from generated questions in semi-supervised QA.Source: §2 · p. 2
• Earlier QA transfer learning uses manually curated source datasets, while this work automatically constructs a source dataset from the target domain and reports better performance.Prior examples transfer from SQuAD to BioASQ or from Book Test to SQuAD.Source: §2 · p. 2
• Cloze datasets have been constructed with heuristics, and this work examines their use in a semi-supervised QA setting.The cited cloze datasets include those of Hermann et al. (2015), Onishi et al. (2016), and Hill et al. (2016).Source: §2 · p. 2

3 Methodology

• Cloze generation assumes documents have introductions that summarize their bodies, using the first 20% when no clear boundary exists; matched answer candidates are drawn from body passages.The heuristic is noisy, but the authors found it beneficial; exact matching spans must be recognized phrases or named entities, and triples with fewer than two non-stopword overlaps are pruned.Source: §3 · p. 2
• The generated Wikipedia and PubMed corpora contain 2.2M and 1M clozes, respectively; manual analysis found their passages answerable in 76% and 80% of cases.The clozes often paraphrase passage information.Source: §3 · p. 3
• Using the entire cloze corpus performed better than selecting subsets by sentence-passage Jaccard similarity, answer-candidate tf-idf, or answer length.Source: §3 · p. 3
• The cloze corpus pretrains GA Reader and BiDAF+SA; BiDAF+SA reaches 0.58 F1 and 0.55 EM on Wikipedia cloze development questions.The authors interpret this as evidence that the cloze corpus is neither too easy nor too difficult to answer.Source: §3 · p. 3
• The pretrained model is fine-tuned on a small set of labeled question-answer pairs, a step the authors say is crucial for achieving substantial performance with few examples.The paper contrasts this with the tens of thousands of questions typically used in training.Source: §3 · p. 3

4.1 Datasets

• Evaluation covers SQuAD and TriviaQA span QA plus BioASQ factoid and list questions, using training-data subsets and five-fold cross-validation for BioASQ.The BioASQ setup uses all 899 available training questions.Source: §4.1 · p. 3

4.2 Main Results

• With 1% of training data (454 SQuAD questions and 746 TriviaQA questions), BiDAF+SA exceeds 50% F1; at 10%, it approaches 90% of best performance.The comparison covers SQuAD and TriviaQA across training-data fractions.Source: Table 2 · p. 4, §4.2 · p. 4
• Cloze pretraining outperforms GDAN on SQuAD and same-data language-model pretraining; its gains over supervised learning shrink as labeled data grows.The authors attribute the language-model baseline's weaker performance partly to only a subset of its parameters being pretrained.Source: Table 2 · p. 4, §4.2 · p. 4
• On BioASQ, cloze pretraining significantly improves over supervised learning, with a larger gain than prior SQuAD pretraining on that target.The authors believe the improvement is smaller than on SQuAD and TriviaQA because BioASQ questions are generally more difficult.Source: Table 3 · p. 4, §4.2 · p. 4

4.3 Analysis

• In SQuAD regression analysis, answer-sentence lexical overlap predicts larger pretraining gains, while whole-passage overlap predicts the opposite; short answers and rare tokens also benefit.The authors connect these patterns to cloze construction, which tends to produce short answer spans and covers a broader domain than SQuAD training data.Source: §4.3 · pp. 4–5, Figure 1 · p. 5
• Across question types, pretraining improves broadly, especially for abbreviation questions, while why questions improve least, consistent with their reasoning or world-knowledge demands.The analysis uses SQuAD's 10% training split and two question classifications.Source: Figure 2 · p. 5, §4.3 · p. 5

• Automatically constructed cloze-question pre-training significantly improves QA performance, especially when labeled examples are scarce.Source: §5 · p. 5
• Models trained only on cloze questions perform poorly, supporting the need to fine-tune them on labeled examples.Source: §5 · p. 5
• Regression analysis finds pre-training helps with factual questions whose answers are located in a specific part of the context.Source: §5 · p. 5
• Future work will explore active learning for choosing passages or question types to annotate, and adapting cloze pre-training to other NLP tasks.Source: §5 · p. 5