PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study

Divyansh Kaushik, Douwe Kiela, Zachary C. Lipton, Wen-tau Yih

arXiv:2106.00872v1cs.CLcs.AIcs.LG

TL;DR

Whether adversarial collection makes QA models more robust to distribution shifts remains unclear. In a randomized study comparing model-in-loop and standard question collection, adversarial training generally helps on adversarial evaluations but tends to hurt out-of-domain performance; the paper also identifies differences in the questions produced.

  • Problem

    It remains unclear whether adversarial data collection improves out-of-distribution generalization over standard collection, how worker behavior contributes, and whether adversarial data is better used alone or as augmentation.

    Source: §1 · p. 1

  • Method

    The authors randomly assigned workers to generate QA questions with BERT or ELECTRA in the loop, or without a model, then evaluated models trained on the resulting data.

    Source: §1 · pp. 1–2

  • Results

    Across models and datasets, adversarial training tends to improve performance on similarly collected adversarial evaluations but tends to worsen performance on out-of-domain evaluations.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results suggest the adversarial collection process may matter more than whether workers successfully fool the model, and ADC and standard collection yield different question types.

    Source: §6 · p. 9

    ADC is more time-consuming and expensive than standard collection and requires additional worker incentives.

    Source: §6 · p. 9

Abstract

from arXiv · show

In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Researchers hope that models trained on these more challenging datasets will rely less on superficial patterns, and thus be less brittle. However, despite ADC's intuitive appeal, it remains unclear when training on adversarial datasets produces more robust models. In this paper, we conduct a large-scale controlled study focused on question answering, assigning workers at random to compose questions either (i) adversarially (with a model in the loop); or (ii) in the standard fashion (without a model). Across a variety of models and datasets, we find that models trained on adversarial data usually perform better on other adversarial datasets but worse on a diverse collection of out-of-domain evaluation sets. Finally, we provide a qualitative analysis of adversarial (vs standard) data, identifying key differences and offering guidance for future research.

1 Introduction

The study tests whether adversarial question collection improves QA robustness, finding gains on adversarial evaluations but typically worse out-of-domain performance, alongside differences in the questions workers create.

1 Introduction

  • The study addresses whether adversarial collection improves out-of-distribution generalization, how worker behavior contributes, and whether adversarial data works better alone or as augmentation.Prior work found that models can exploit dataset-specific spurious associations despite strong i.i.d. performance, making plausible domain shifts a concern.Source: §1 · p. 1
  • Workers were randomly assigned to generate questions with BERT, with ELECTRA, or without a model, using the same contexts.The study fine-tuned BERT, RoBERTa, and ELECTRA on the resulting datasets.Source: §1 · pp. 1–2
  • Adversarial-data training usually improves performance on previously collected adversarial data but typically worsens performance across a diverse collection of out-of-domain datasets.The evaluation includes 12 MRQA datasets.Source: §1 · p. 2
  • More questions in the adversarial collection require numerical reasoning than in the standard-collection sample.The authors present this qualitative analysis as possible guidance for future research.Source: §1 · p. 2

2 Related Work

Prior work spans model-in-the-loop collection, adversarial crowdsourcing and iterative rounds, with reported outcomes varying by task and evaluation; related approaches also include counterfactual augmentation and analyses of distribution-shift assumptions.

2 Related Work

  • Early work explored model-assisted candidate answers, proposed adversarial QA, and used builder-breaker tasks to create challenging examples (Ettinger et al., 2017).Later studies asked workers or community members to create questions that fool QA models (Wallace et al., 2019; Dua et al., 2019; Dasigi et al., 2019).Source: §2 · p. 2
  • Multi-round ADC studies report that models become harder to fool or improve on adversarial evaluations, with some gains on original evaluations when adversarial and original training data are mixed (Dinan et al., 2019; Nie et al., 2020).Yang et al. (2018b) used workers first as adversaries and later as collaborators.Source: §2 · pp. 2–3
  • For SQuAD, ADC improved adversarial-test performance, but trends varied with the models used to collect and train data; counterfactual data collection was reported to improve out-of-domain generalization (Bartolo et al., 2020; Kaushik et al., 2020, 2021).Related work also notes similar model-mismatch issues in active learning (Lowell et al., 2019).Source: §2 · p. 3
  • Robust learning under distribution shift is impossible without further assumptions, while NLP work often implicitly assumes deterministic labels and covariate shift.The passage notes that label shift can change p(y|x), requiring adaptation to each environment (Schölkopf et al., 2012; Lipton et al., 2018).Source: §2 · p. 3

3 Study Design

The study compares model-in-loop and standard question collection on shared Wikipedia contexts, using randomized worker assignments, defined incentives, validation, and span-based QA models and metrics.

Context passages and task

  • Workers generated five questions and highlighted answer spans for each context; the study used the first 100 words of Wikipedia articles and sampled 10,000 contexts from a 23.1k-example Natural Questions subset.The pool was restricted to examples whose context was the first 100 words of a Wikipedia article.Source: §3 · p. 3

Models in the loop

  • BERTlarge and ELECTRAlarge served as the in-loop models, fine-tuned for span-based QA on 23.1k examples; their i.i.d. validation F1 scores were 80.5 and 81.2, respectively.Their validation EM scores were 73.1 for BERT and 74.2 for ELECTRA.Source: §3 · p. 3

Measures and data resources

  • The study reports word-overlap F1 and Exact Match for passage-based QA, and Table 1 summarizes unique passages and question-answer pairs for each data resource.The correct answer is a span in the given passage.Source: §3 · p. 3, Table 1 · p. 4

Crowdsourcing protocol

  • The study randomly assigned workers to fool BERT, fool ELECTRA, or work without a model, presenting the same passages and five-question task across groups.A total of 984 workers participated, including 741 who created questions.Source: §3 · p. 4

Incentive structure

  • All workers received $0.75 per HIT, while ADC workers earned an extra $0.15 for each question that led to a model prediction with F1 below 40%.The bonus could double a worker’s pay if all five questions induced incorrect predictions.Source: §3 · p. 4

Quality control

  • About 20% of questions were checked by validation workers, incorrect work from workers with at least 20% validation errors was discarded, and the study collected over 150k pairs before final filtering.The process, including pilots, cost about $50k and lasted seven months.Source: §3 · p. 4

4 Experiments and Results

ADC-trained models generally outperform SDC-trained models on prior adversarial evaluations, but results on MRQA vary by dataset. The experiments also suggest that the adversarial collection process matters beyond whether workers successfully fool the model, while augmentation changes the comparison.

Datasets

  • The study creates six datasets by matching the number of questions per context across fooled, randomly sampled ADC, and SDC subsets for each collection model.The BERT subsets contain 11.3k examples each, and the ELECTRA subsets contain 14.7k each.Source: §4 · p. 4

Models

  • On matched held-out tests, ADC-trained models generally perform better on ADC test data, while SDC-trained models generally perform better on SDC test data.The paper reports means and standard deviations across 10 runs; Table 2 covers EM and F1 on adversarial and non-adversarial datasets.Source: §4 · p. 4, Table 2 · p. 5

Out-of-domain generalization to adversarial data

  • 9.1, 9.3, and 6.2 EM points: RoBERTa fine-tuned on BERTfooled outperforms its SDC-trained counterpart on DRoBERTa, DBERT, and DBiDAF, respectively.Across the adversarial test sets, ADC-trained models generally outperform SDC-trained models; ELECTRA gains disappear after augmentation in some comparisons.Source: §4 · p. 5, Table 3 · p. 6

Out-of-domain generalization to MRQA

  • On MRQA, BERT fine-tuned on SDC scores 35.1 EM and 55.7 F1 on BioASQ, versus 23.5 EM and 30.3 F1 for BERTfooled; ADC can improve DROP by about 7 points.On Natural Questions, ADC- and SDC-trained models are generally comparable; the paper notes that both collections use passages from Natural Questions.Source: §4 · pp. 5–6

The adversarial process versus adversarial success

  • Randomly sampled ADC data typically outperforms fooled-only ADC data on prior adversarial tests and MRQA, suggesting the collection process matters more than successful fooling.The same pattern is observed when ADC data is augmented with the original training data.Source: §4 · p. 6
  • On SearchQA, BERT's EM is 8.7 with fooled ADC plus original data and 11.2 with random ADC plus original data, versus 13.6 with SDC plus original data.Augmentation makes ADC competitive on more datasets than ADC-only training, but does not always help.Source: §4 · p. 6

5 Qualitative Analysis

The qualitative analysis found differences between ADC and SDC question mixes, including more numerical-reasoning questions in BERT ADC samples and some prohibited passage-structure questions in the final data.

5 Qualitative Analysis

  • The analysis categorized 100 questions per dataset using Hovy et al.'s taxonomy and examined the first word of wh-questions in development sets.Figure 3 presents frequencies of question types under the taxonomy.Source: §5 · pp. 7–8, Figure 3 · p. 8
  • BERT ADC dev sets had more what-questions than SDC (587 in BERTfooled and 492 in BERTrandom versus 416); SDC had more when- and who-questions than BERTfooled.BERTrandom also had more when- and who-questions than BERTfooled (103 and 182 versus 50 and 159, respectively).Source: §5 · p. 8
  • Numerical reasoning appeared in 11 BERTfooled and 18 BERTrandom questions, compared with 7 in SDC.The authors suggest model performance on numerical reasoning may have influenced question generation.Source: §5 · p. 8
  • ELECTRA ADC dev sets had more what-questions than SDC (641 in ELECTRAfooled and 619 in ELECTRArandom versus 542); ELECTRArandom had 126 how-questions versus 101 in SDC.The analysis examined the first word of wh-questions in development sets.Source: §5 · p. 8
  • Some questions about passage structure appeared despite instructions against them, and some remained in the final data after validation.The authors suggest asking workers to edit initial questions as a possible remedy for ADC's incentive-related skew in question distributions.Source: §5 · p. 9

6 Conclusion

Across models and datasets, training on adversarial data improved performance on similarly collected evaluation sets but tended to reduce performance on non-adversarial out-of-domain sets. The authors also report that the collection process may matter beyond whether workers successfully fool a model.

6 Conclusion

  • Across models and datasets, adversarial training performed better on similarly collected evaluation sets but tended to perform worse on non-adversarial out-of-domain sets.The study also found qualitative differences in the kinds of questions generated through ADC and SDC.Source: §6 · p. 9
  • The results suggest the ADC process may matter more than successfully fooling a model.The authors present this as a suggestion, not a definitive causal finding.Source: §6 · p. 9
  • ADC is more time-consuming and expensive than SDC, and requires additional financial incentives for workers.The authors suggest asking workers to edit questions rather than generate them as a possible remedy.Source: §6 · p. 9
  • Future work could test question-creation constraints, domain complexity, passage length, and incentive structure.The authors frame these as factors to investigate when extending the study.Source: §6 · p. 9

Ethical Considerations

The released datasets use English passages and were created for a human study, not to design NLP applications.

Ethical Considerations

A Appendix

The appendix captions describe evaluation tables for adversarial and non-adversarial datasets, plus tables of validation-set examples.

A Appendix

  • Table 5 reports EM and F1 scores for various models evaluated on adversarial datasets collected with ELECTRAlarge and on non-adversarial datasets.Bold indicates results statistically significant against the SDC setting, or vice versa, at p < 0.05.Source: Table 5 · p. 13
  • Table 6 reports EM and F1 scores for various models on Bartolo et al. (2020) development datasets, with bold marking statistically significant comparisons against SDC at p < 0.05.Source: Table 6 · p. 13
  • Table 7 reports EM and F1 scores for various models on MRQA development and test sets, with bold marking statistically significant comparisons against SDC at p < 0.05.Source: Table 7 · p. 14
  • Tables 8 and 9 provide validation-set question examples from different resources and highlight correct answers in red.Source: Table 8 · p. 15, Table 9 · p. 16
Loading 2106.00872v1…