Unsupervised Question Answering by Cloze Translation

• The approach addresses limited access to QA training data, which can be costly to collect for new domains or languages.Source: §1 · p. 1
• It generates training examples by sampling paragraphs and candidate answers, masking answers into clozes, and translating clozes into natural questions.Source: §1 · p. 2
• Cloze-to-natural question translation is the most challenging stage; the proposed seq2seq model learns from unaligned corpora using back-translation and denoising autoencoding.Source: §1 · p. 2
• With modern QA architectures, unsupervised QA can surpass early supervised approaches, and the unsupervised seq2seq translator outperforms noise- and rule-based systems.Source: §1 · p. 2
• 59.3 F1 with 32 labelled examples compares with 40.0 F1 without the method in the few-shot setting.Source: §1 · p. 2
• The paper frames its contributions as an unsupervised QA approach based on cloze translation, experiments on translation choices, and few-shot EQA evaluation.Source: §1 · p. 2

2 Unsupervised Extractive QA

• The generator samples a context, samples an answer within it, and generates a question; its synthetic examples train a discriminative QA model.Source: §2 · p. 2

2.1 Context and Answer Generation

• The context generator samples paragraphs, while the answer prior selects spans from noun phrases or named entities; named entities reduce candidate variety.Source: §2.1 · pp. 2–3

2.2 Question Generation

• Question generation first forms a cloze from the answer and its context, then translates that cloze into a question.Source: §2.2 · p. 3

2.2.1 Cloze Generation

• Cloze generation masks the answer in its sentence, optionally narrowing the context to the containing sub-clause using a syntactic parser.Source: §2.2.1 · p. 3

2.2.2 Cloze Translation

• Translation variants include identity and noisy clozes, rule-based question generation, and an unsupervised seq2seq translator between cloze and natural-question domains.Source: §2.2.2 · p. 3

2.3 Question Answering

• A separate QA system can train on generated examples, while posterior inference selects the answer candidate most likely to have generated the question.Source: §2.3 · p. 3

2.4 Unsupervised Cloze Translation

• The translator learns from unaligned cloze and question corpora using denoising autoencoding and online back-translation; language-model pretraining improved fluency but did not significantly change QA performance.Source: §2.4 · p. 4
• Training data includes five million clozes and five million mined questions; answer-type tokens and a wh-word heuristic guide question generation.Source: §2.4 · p. 4

3.1 Unsupervised QA Experiments

• The experiments evaluated BERT and BiDAF+SA using Exact Match and F1, with QA training stopped when performance on held-out synthetic data plateaued.The SQuAD development set was used to assess model components, and only the best system was submitted to the test server.Source: §3.1 · pp. 4–5
• 54.7 F1 on the SQuAD test set came from the best approach, while an ensemble of five models achieved 56.4 F1 and surpassed early supervised methods.The approach also significantly outperformed baseline systems and Dhingra et al. (2018).Source: Table 1 · p. 5, §3.1 · p. 5

3.2 Ablation Studies and Analysis

• Named entities outperformed noun phrases by an average of 8.9 F1, while subclause generation improved equivalent sentence-level BERT-Base models by 4.0 F1.Subclauses produced shorter questions and shorter context overlaps, more closely matching SQuAD question distributions; the study reports 14 named entities versus 33 noun phrases per context on average.Source: §3.2 · p. 5, Figure 2 · p. 6
• Noisy cloze translation improved over identity translation by a mean 9.8 F1, and unsupervised NMT added 1.8 F1 over otherwise equivalent noisy-cloze models.BERT-Base outperformed BiDAF+SA, and BERT-Large improved the best configuration by 6.9 F1; training QA models was more effective than posterior maximization.Source: §3.2 · pp. 5–6

3.3 Error Analysis

• On SQuAD, the best QA model scored 64.5 F1 for questions whose correct answers were automatically tagged NEs and 47.9 F1 for questions without automatically tagged NE answers.It performed best on “when” questions and TEMPORAL answers, struggled with “what” questions, and predicted some spans the NER tagger did not detect despite training solely on NE answer spans.Source: §3.3 · pp. 6–7

3.4 UNMT-generated Question Analysis

• UNMT questions copied an average 9.1-token context subsequence (56.9% of each question), versus 4.7 tokens (46.1%) for SQuAD questions.The model sometimes changed syntax but also produced semantic drift; 68% of its questions were classified as well-formed, rising to 78.5% with language-model pretraining.Source: §3.4 · p. 7

3.5 Few-Shot Question Answering

• Unsupervised QA pretraining performed best in very data-poor few-shot regimes, while BERT-Large alone reached about 60% F1 with 1% of available data.The authors caution that development sets used for early stopping could be larger than the training sets, so these results do not reflect true low-data performance.Source: §3.5 · p. 8

Unsupervised Learning in NLP

• Unsupervised NMT and style-transfer work informs this approach, which uses such models for training-data generation rather than applying them directly.The background also notes representation-learning approaches based on latent variables or language-model-inspired criteria.Source: §4 · p. 8

Semi-supervised QA

• Prior semi-supervised QA methods often require labelled data, use Wikipedia summaries, or target related tasks rather than unsupervised QA with natural questions.Dhingra et al. (2018) remove supervised question-generation requirements but rely on Wikipedia summary paragraphs and do not attempt to generate natural questions.Source: §4 · pp. 8–9

Question Generation

• Question generation has been studied with symbolic templates and rules, ranking models, and supervised neural models trained on SQuAD question-context-answer pairs.The passage cites Rus et al. (2010), Heilman and Smith (2010), Yao et al. (2012), Olney et al. (2012), and several neural approaches.Source: §4 · p. 9

5 Discussion

• The best-performing approach relies on an NER system, a constituency parser, and a language-specific wh* heuristic, limiting flexibility where these resources or engineering are unavailable.The NER system indirectly uses OntoNotes 5 labels, and the parser was trained on the Penn Treebank.Source: §5 · p. 9
• The approach is unsupervised in that it requires no labelled question-answer or question-context pairs, although it uses linguistic resources.The authors characterize these pairs as usually the most challenging aspects of annotating large-scale QA training datasets.Source: §5 · p. 9
• Noisy cloze performs nearly as well as the more complex system despite only 2.7% of its questions being classified as well-formed.The authors suggest natural questions may be less important for SQuAD and that strong question-context word matching may suffice, consistent with Jia and Liang (2017).Source: §5 · p. 9
• The generated questions require no multi-hop or multi-sentence reasoning, yet achieve non-trivial SQuAD performance.The passage notes that Min et al. (2018) report 90% of SQuAD questions require only one context sentence, while Sugawara et al. (2018) find 76% have answers in the sentence with highest token overlap.Source: §5 · p. 9

6 Conclusion

• 56.4% F1 on SQuAD and 64.5% F1 on the named-entity-answer subset show that the approach surpasses simple supervised systems.The conclusion describes the evaluated task as relatively simple and says further work is needed for more challenging QA elements.Source: §6 · p. 9
• The work explores learning extractive QA behaviour without labelled QA data and reports that this is possible.The conclusion states that the approach surpasses simple supervised systems and strongly outperforms other approaches that do not use labelled data.Source: §6 · p. 9
• Further work is needed to handle more challenging QA elements and reduce reliance on linguistic resources and heuristics.Source: §6 · p. 9

A.1 Cloze Question Featurization and Translation

• Cloze questions are tokenized, discarded if longer than 40 tokens, and have the answer blank replaced by a mask token.Source: §A.1 · p. 13
• Noun-phrase-chunker answers use a single MASK token because they have no specific answer-entity typing.Source: §A.1 · p. 13
• Named-entity answers use masks for higher-level entity categories, such as TEMPORAL for the DATE answer “2018.”Source: §A.1 · p. 13

A.2 Unsupervised NMT Training Setup Details

• The model uses four-layer Transformer encoders and decoders, each with one language-specific layer and otherwise shared layers, plus a 60,000-token FastBPE vocabulary.The English tokenizer is Moses, and input embeddings are initialized with FastText vectors trained on the cloze and natural-question corpora.Source: §A.2 · p. 13
• Training uses 5M cloze questions and natural questions, with equal autoencoding and backtranslation loss weights initially; the autoencoding weight falls to 0.1 by 100K steps and 0 by 300K.The model is initialized with random weights, apart from FastText-initialized input embeddings.Source: §A.2 · p. 13
• Training usually stops around 300K optimization steps when BLEU between back-translated and input questions stops improving; greedy decoding is used, and beam size 5 did not significantly change QA performance or fluency.Source: §A.2 · p. 13

A.3 Wh* Heuristic

• The wh* heuristic maps each high-level answer category to a preferred wh* word and prepends the matching category token to corresponding training questions.For numeric answers, it randomly selects “How much” or “How many.”Source: §A.3 · p. 13

A.4 QA Model Setup Details

• BiDAF + Self Attention and BERT models are trained with default settings; BERT-Base uses synthetic-development-set early stopping, while BERT-Large is halted manually at a performance plateau.Source: §A.4 · p. 13

A.5 Question Well-Formedness

• 68.0% of questions from the best unsupervised generator were classified as well-formed, versus 75.6% for the rule-based generator.Source: Table 6 · p. 14, §A.5 · p. 14
• The well-formedness classifier reached 80.9% test accuracy, compared with the prior state of the art at 70.7%; it classified 92.3% of SQuAD questions as well-formed.The evaluation dataset contains 25,100 search-engine queries annotated for whether each is a well-formed question.Source: §A.5 · pp. 13–14

A.6 Language Model Pretraining

• MLM-pretrained models were classified as well-formed 10.5% more often than models without pretraining, surpassing the rule-based generator, but QA performance did not differ significantly.Source: §A.6 · p. 14

A.7 More Examples of Unsupervised NMT Cloze Translations

• Table 7 provides further UNMT cloze translations with subclause boundaries and the wh* heuristic, including the generated question “What are the mentions of Bab?”Source: Table 7 · p. 15, §A.7 · p. 15