PaperlayerView exact revision on arXiv

Source-linked AI summary

Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension

Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, Pontus Stenetorp

arXiv:2002.00293v2cs.CL

TL;DR

Reading-comprehension datasets need methods that challenge models beyond superficial cues, motivating adversarial human annotation with a model in the loop. The paper applies this protocol with progressively stronger models across three datasets and evaluates reproducibility and transfer. Stronger models learn from weaker-adversary data, while generalisation to standard data remains strong but deteriorates as adversaries strengthen.

  • Problem

    The paper asks whether adversarially constructed RC datasets remain useful as models grow stronger and whether they generalise to standard questions.

  • Method

    The authors have annotators compose questions that models answer incorrectly, collecting three 12,000-sample datasets with BiDAF, BERT, and RoBERTa adversaries.

  • Results

    Stronger models learn from weaker-adversary data, and model-in-the-loop training generalises well to SQuAD and NaturalQuestions while performance gradually deteriorates with stronger adversaries.

  • Takeaways & Limitations

    Model-in-the-loop annotation produces challenging RC questions and shows promise for future dataset creation.

  • Takeaways & Limitations

    The collected datasets become progressively harder to answer for humans, with answerability scores of 87.95%, 85.41%, and 82.63% for DBiDAF, DBERT, and DRoBERTa.

Abstract

from arXiv · show

Innovations in annotation methodology have been a catalyst for Reading Comprehension (RC) datasets and models. One recent trend to challenge current RC models is to involve a model in the annotation process: humans create questions adversarially, such that the model fails to answer them correctly. In this work we investigate this annotation methodology and apply it in three different settings, collecting a total of 36,000 samples with progressively stronger models in the annotation loop. This allows us to explore questions such as the reproducibility of the adversarial effect, transfer from data collected with varying model-in-the-loop strengths, and generalisation to data collected without a model. We find that training on adversarially collected samples leads to strong generalisation to non-adversarially collected datasets, yet with progressive performance deterioration with increasingly stronger models-in-the-loop. Furthermore, we find that stronger models can still learn from datasets collected with substantially weaker models-in-the-loop. When trained on data collected with a BiDAF model in the loop, RoBERTa achieves 39.9F1 on questions that it cannot answer when trained on SQuAD - only marginally lower than when trained on data collected using RoBERTa itself (41.0F1).

1 Introduction

The paper investigates model-in-the-loop adversarial annotation for reading comprehension, where humans create questions that current models answer incorrectly. Across progressively stronger adversaries, it studies reproducibility, generalisation, and how question characteristics change.

  • 1 Introduction: Model-in-the-loop annotation retains only questions the model fails to answer correctly.Annotators compose questions and provide answers; unsuccessful model predictions determine which samples enter the dataset.
  • 1 Introduction: 36,000 samples were collected across three annotation settings using progressively stronger models in the loop.The study compares datasets produced with three distinct adversarial models and examines performance across adversary strengths.
  • 1 Introduction: Adversarially collected training data generalises to non-adversarial SQuAD and NaturalQuestions questions at a similar degree to SQuAD training.The data also improves performance across collected model-in-the-loop datasets and by more than 20.0F1 for BERT and RoBERTa on extractive DROP.
  • 1 Introduction: Questions created with a model in the loop are more diverse and rely more on paraphrasing, multi-hop inference, comparisons, and background knowledge.They are generally less answerable through direct matching of an explicit statement containing the required information.
  • 1 Introduction: The paper contributes analyses of adversary strength, question characteristics, reproducibility, and generalisation in adversarial RC data collection.These investigations compare datasets constructed with progressively stronger models and datasets produced without a model adversary.

2 Related Work

Related work has increasingly focused on constructing challenging reading-comprehension datasets, including adversarial methods that use models to expose weaknesses. This paper distinguishes interactive annotation from offline filtering and studies annotators probing models in the original task setting.

  • Constructing Challenging Datasets: RC dataset construction has expanded beyond SQuAD with unanswerable, multi-hop, context-aware, and answer-type-controlled questions.These methods aim to make natural-language understanding more necessary for generalisation than superficial cues.
  • Adversarial Annotation: Adversarial annotation uses a model to select examples it performs poorly on, resembling active learning only superficially.The paper distinguishes offline adversarial filtering from interactive model-in-the-loop annotation.
  • Adversarial Annotation: Interactive feedback lets annotators probe model weaknesses and formulate targeted adversarial attacks.The paper focuses on annotators interacting with the model rather than merely filtering easy samples after generation.
  • Adversarial Annotation: Unlike constrained DROP and Quoref setups, this work keeps annotators in the original RC task setting to explore varied natural adversarial attacks.DROP limits answer types, while Quoref focuses on coreference.

3 Annotation Methodology

The annotation protocol extends SQuAD1.1 with a model adversary: annotators generate answerable questions until the model’s predicted answer fails to match the human-selected answer. Three progressively stronger adversaries produce datasets that are filtered, validated, and compared under consistent data-splitting procedures.

  • Adversarial annotation: Annotators generate a question and highlight its answer span, then retain the sample only when the model’s answer fails the word-overlap criterion.A model F1 score above 40% counts as a model win; the process repeats until the human wins.
  • Models in the annotation loop: BiDAF, BERTLARGE, and RoBERTaLARGE serve as progressively stronger adversaries, producing the DBiDAF, DBERT, and DRoBERTa datasets.Their SQuAD1.1 validation EM/F1 scores are 65.5%/77.5%, 82.7%/90.3%, and 86.9%/93.6%, respectively.
  • Experimental controls: The study preserves passage consistency across splits and uses majority-vote SQuAD1.1 answers to ensure one valid answer per question for direct comparisons.Half of the existing SQuAD1.1 validation set is held out as an official test set because the original test set is unavailable.
  • Quality control: Human validation retains questions answerable by at least one of three additional validators, while discarding unanswerable questions and data from workers below the answerability threshold.Answerability scores are 87.95% for DBiDAF, 85.41% for DBERT, and 82.63% for DRoBERTa; the procedure can discard valid questions.
  • Dataset statistics: Stronger adversaries yield longer answers and reduce the largest passage-question n-gram overlap from 3 tokens for DSQuAD to 2 tokens for DRoBERTa.Model-in-the-loop datasets also contain more diverse question types, including more which, where, and why questions than DSQuAD.

4 Experiments

The experiments test reproducibility, cross-model generalisation, transfer to standard data, and transfer to DROP and NaturalQuestions. Adversarial training helps some models overcome blind spots and generalise broadly, but performance declines as the annotation-loop model becomes stronger, while weaker-model datasets can become outdated.

  • Reproducibility: 0.0EM on same-model adversarial datasets does not reliably reproduce after retraining, with BERT reaching 19.7EM.Random initialisation and minibatch order can substantially affect the adversarial annotation process.
  • Adversarial generalisation: Stronger-model evaluation datasets generally produce progressively lower performance across training setups; RoBERTa scores 72.1, 57.1, 49.5, and 41.0F1 from DSQuAD through DRoBERTa.This pattern holds for all but BiDAF across the reported training configurations and evaluation datasets.
  • Adversarial generalisation: BERT and RoBERTa partially overcome blind spots through model-in-the-loop training, whereas BiDAF remains consistently poor on adversarial evaluations.BERT reaches 47.9F1 on DBERT and RoBERTa reaches 41.0F1 on DRoBERTa; these gains exceed random retraining.
  • Cross-strength transfer: 39.9F1 on DRoBERTa is achieved by RoBERTa trained on DBiDAF, versus 22.1F1 when trained on 10,000 SQuAD samples.Training on data from either stronger or weaker adversaries can support transfer across adversarial datasets.
  • Non-adversarial generalisation: 82.8, 80.0, 75.1, and 72.1F1 on DSQuAD result when RoBERTa is trained on DSQuAD(10K), DBiDAF, DBERT, and DRoBERTa respectively.The authors suggest this may reflect a gradual shift in composed-question distributions as the model in the loop strengthens.
  • Combined training: 86.9, 74.1, 65.1, and 52.7F1 are achieved by RoBERTa trained on all datasets for DSQuAD, DBiDAF, DBERT, and DRoBERTa respectively.Combined training provides additional gains and surpasses the best previous performance on all adversarial datasets.
  • Limitations: 58.2EM/73.2F1 for RoBERTa on DBiDAF contrasts with 0.0EM/5.5F1 for BiDAF, indicating that weaker-adversary datasets risk becoming outdated.RoBERTa’s result approaches the reported non-expert human performance of 62.6EM/78.5F1.
  • Transfer to DROP and NaturalQuestions: 69.8F1 is obtained on DNQ by BERT trained on DBiDAF, while transfer to non-adversarial data generally deteriorates with stronger annotation-loop models.The filtered DROP and NaturalQuestions experiments evaluate transfer under SQuAD-like structural constraints.

5 Qualitative Analysis

The qualitative analysis develops a non-exclusive, non-comprehensive taxonomy to compare comprehension requirements across standard, adversarial, DROP, and NaturalQuestions data. Model-in-the-loop questions are less literal and increasingly demand diverse inference and external knowledge as the adversary strengthens.

  • Taxonomy and analysis: The analysis uses a 13-label taxonomy, assigning up to three salient, non-exclusive labels to each question.The catalogue is designed for the analysed datasets rather than as a universally accepted formalism.
  • Comprehension requirements: 57% of SQuAD answers are explicit, compared with 8% for DBiDAF, while model-in-the-loop questions require more external knowledge and multi-hop inference.The analysis covers 100 validation samples from each dataset.
  • Comprehension requirements: Model-in-the-loop questions include non-negligible spatial and temporal inference, which SQuAD questions rarely require.These inference types go beyond explicitly stated spatial or temporal information.
  • Dataset comparisons: DROP contains 60% comparative questions, while NaturalQuestions combines elements of the SQuAD and DBiDAF distributions.The mixed NaturalQuestions distribution may help explain strong transfer from DSQuAD(10K) and DBiDAF.
  • Dataset comparisons: Question distributions shift progressively away from SQuAD and NaturalQuestions as the model-in-the-loop becomes stronger.This qualitative shift mirrors the reported decrease in performance on those datasets after training with stronger adversaries.

6 Discussion and Conclusions

The paper evaluates model-in-the-loop annotation with progressively stronger adversaries and finds that adversarial data transfers well, including to non-adversarial datasets. Stronger models can learn from weaker-adversary data, but stronger adversaries bring gradual performance deterioration and weaker-adversary datasets may age.

  • Conclusions: Three datasets are produced using progressively stronger adversaries—BiDAF, BERT, and RoBERTa—to study reproducibility, transfer, and generalisation.The datasets support comparisons across adversary strength and between adversarial and non-adversarial data.
  • Implications: Stronger models can learn from weaker-adversary data and generalise better to samples composed with stronger adversaries.The authors present this as an encouraging implication for model-in-the-loop annotation.
  • Conclusions: Models trained on model-in-the-loop data generalise well to SQuAD and NaturalQuestions, while performance gradually deteriorates with stronger adversaries.This conclusion covers both non-adversarial transfer settings reported in the experiments.
  • Implications: The method is presented as especially promising for mature datasets where models exceed human performance and struggle on selected samples.The paper focuses on RC and SQuAD, while suggesting the paradigm may inform other tasks using similar annotation procedures.

A Additional Dataset Statistics

Model-in-the-loop datasets contain more diverse questions and progressively longer, more variable answers as model strength increases. These distributions differ from SQuAD1.1 in both question and answer characteristics.

  • Question statistics: Question diversity generally increases with model-in-the-loop strength, despite similar mean question lengths to SQuAD1.1.The model-in-the-loop distributions also show greater question-length variability.
  • Answer statistics: Answers in all model-in-the-loop datasets tend to be longer than answers in SQuAD.
  • Answer statistics: Answer length and variability increase with model-in-the-loop strength.

B Annotation Interface Details

Dataset construction combines worker training and qualification, adversarial question generation, and answer validation. Workers generate questions and answers, while subsequent validation retains questions with at least one matching answer.

  • Process: Dataset construction has three steps: training and qualification, “Beat the AI” annotation, and answer validation.
  • Training and Qualification: Training and qualification require workers to generate questions, highlight answers, and complete a sample “Beat the AI” task.The combined interface was introduced to reduce the bottleneck between qualification layers.
  • Training and Qualification: 1,386 workers completed training and qualification, with 752 assigned the qualification.
  • Answer Validation: Validation submits retained questions to at least three annotators and keeps questions with at least one valid matching answer.

C Catalogue of Comprehension Requirements

The paper defines a catalogue of comprehension requirements for qualitative analysis of adversarial model-in-the-loop datasets. The catalogue’s categories are not mutually exclusive, and the accompanying figures visualize question structures for several datasets.

  • Catalogue: The comprehension-requirement catalogue provides definitions and examples used for qualitative analysis.
  • Question visualizations: Sunburst plots display question-token patterns separately for DSQuAD, DBERT, DBiDAF, and DRoBERTa.
  • Catalogue: The catalogue types are not mutually exclusive, allowing one question to receive multiple comprehension-requirement labels.
Loading 2002.00293v2…