Trick Me If You Can: Human-in-the-loop Generation of Adversarial Examples for Question Answering

• Trivia enthusiasts use model interpretations in an interactive interface to craft diverse Quizbowl questions that stump existing QA models.Source: §1 · p. 1
• Although adversarial questions remain as easy for humans as regular questions, strong QA models’ relative accuracy drops by as much as 40%.Source: §1 · p. 1
• Analysis of the edits uncovers phenomena people can solve but computers cannot, making the dataset a challenging resource for future QA research.Source: §1 · p. 1
• Automatic adversarial evaluation often targets specific phenomena, limiting example diversity and requiring human checks to ensure validity.Source: §1 · p. 1, Figure 1 · p. 2

2 Adversarial Evaluation for NLP

• Simple meaning-preserving attacks include syntax edits, typos (Belinkov and Bisk, 2018), distractor sentences (Jia and Liang, 2017), and character replacements (Ebrahimi et al., 2018).Source: §2 · p. 2
• Neural generators can diversify examples through syntax, captions, or NLI premises, but require human verification and still target specific phenomena (Iyyer et al., 2018; Zellers et al., 2018; Zhao et al., 2018).Source: §2 · p. 2
• 3% of Quizbowl paraphrases from syntax paraphrase networks are valid, illustrating the difficulty of applying sentence perturbations to paragraph inputs (Iyyer et al., 2018).Source: §2 · p. 2

2.1 Putting a Human in the Loop

• Authors create questions that break a specific QA system while remaining answerable by humans, using exposed model predictions and interpretations to find confusing edits.Source: §2.1 · p. 2
• The interface makes authoring interactive and model-driven, deliberately producing a dataset that exposes a model’s limitations rather than examples written independently of a model (Ettinger et al., 2017).Source: §2.1 · p. 2
• Human-in-the-loop generation can replace or aid model-based generation, and human edit histories can inform models trained to mimic those edits.Source: §2.1 · p. 2

3 Our QA Testbed: Quizbowl

• Quizbowl uses pyramidal, interruptible questions whose clues become progressively easier, with well-known entities as typical answers.Source: §3 · p. 2

3.1 Known Exploits of Quizbowl Questions

• Human-oriented clues can yield trivial computer triggers, while harder questions demand coreference, multiple knowledge types, or wordplay.Source: §3.1 · p. 3

3.2 Models and Datasets

• Authors attack an ElasticSearch-based IR baseline first, then an IR or bidirectional GRU QA model trained as answer-entity classification with 300-dimensional GloVe inputs.Source: §3.2 · p. 3

3.3 Interpreting Quizbowl Models

• Word saliency highlights influential question terms: neural scores approximate removal through gradients, while IR scores come from query matches in an inverted index.Source: §3.3 · p. 3

3.4 Adversarial Writing Interface

• The interface shows top-five guesses and evidence as questions are written, enabling authors to make models wrong or delay the first correct answer.Edit histories also support analysis of recurring failures and how interpretations guide authors.Source: §3.4 · pp. 3–4

3.5 Question Authors

• Quizbowl specialists write grammatical, semantically difficult questions under familiar format and giveaway-clue conventions, going beyond character noise or syntax changes.Source: §3.5 · p. 4

3.6 How an Author Writes a Question

• In the Brahms example, authors replace an early-answering name with a positional clue, then use added ambiguity and German wording to mislead the model.Source: §3.6 · pp. 4–5

4 A New Adversarially-Authored Dataset

• The dataset contains 1,213 questions and 6,541 sentences across topics, authored by 115 people; 807 target IR and 406 target the neural model.Source: §4 · p. 5

4.1 Validating Questions with Quizbowlers

• Validation filtered length, vulgarity, and duplicates, then compared 60 adversarial with 60 standard high-school championship questions, category-matched and read to players for buzz position and accuracy.Intermediate and expert players participated; buzz position was recorded as a percentage of the question, with lower values indicating earlier answers.Source: §4.1 · p. 5
• 41.6% question remaining at buzz and 89.7% accuracy on adversarial questions, versus 28.3% and 84.2% on standard questions.Accuracy difference was not significant (p = 0.16, Fisher’s exact test), while adversarial questions elicited earlier buzzing (p = 0.0047, two-sided t-test).Source: §4.1 · p. 5

5 Computer Experiments

• The computer experiments evaluate IR, RNN, and DAN models across two rounds of attacks, followed by live matches against Studio Ousia.The first round targets IR; the second targets both IR and RNN.Source: §5 · p. 5

5.1 First Round Attacks: IR Adversarial Questions Transfer To All Models

• DAN accuracy falls from 54.1% to 32.4% on full questions in the first round, and IR-targeted questions are significantly harder for IR, RNN, and DAN models.Adversarial questions are especially difficult in the middle third, where Quizbowl buzzes most often.Source: §5.1 · pp. 5–6, Figure 4 · p. 6
• The first-round questions begin with difficult clues like regular questions, but are significantly harder in the middle third.For both question types, accuracy rises toward the end as clues become give-aways.Source: Figure 4 · p. 6, §5.1 · p. 6

5.2 Second Round Attacks: RNN Adversarial Questions are Brittle

• IR-targeted questions also degrade neural models, but RNN-targeted questions challenge neural models while leaving the IR model largely unaffected.The second-round comparison used models trained on a larger dataset.Source: Figure 5 · p. 6, §5.2 · p. 6

5.3 Humans vs. Computer, Live!

• The human team beat the state-of-the-art Studio Ousia model 300–30 on IR adversarial questions, reversing its prior 475–200 win over expert humans on regular questions.The live match used a forty-question, tossup-only Quizbowl format.Source: §5.3 · p. 6
• Human teams spanning high-school to national-level players prevailed in every match against IR- or neural-based models across the larger live event.Seven teams played on more than 400 RNN-targeted questions; a few games between weaker teams and models were close.Source: §5.3 · pp. 6–7
• Humans find adversarial questions about as difficult as regular questions across intermediate, expert, and national player levels.Human accuracy rises quickly in the last half of both regular and adversarial questions.Source: Figure 6 · p. 7, §5.3 · p. 7
• Studio Ousia’s accuracy degrades on adversarial questions despite never being directly targeted, extending the findings beyond the IR and RNN models.Models do well on regular questions in the first half but struggle on adversarial questions there.Source: Figure 7 · p. 7, §5.3 · p. 7

6.1 Quantitative Differences in Questions

• Adversarial questions have 14.9 mean training examples per answer versus 16.9 for regular test questions, so answer scarcity does not explain their difficulty.Source: §6.1 · p. 7
• Adversarial questions have roughly equal unigram overlap but higher bigram overlap, and are shorter with fewer named entities while having a roughly equivalent named-entity proportion.The overlap measure counts test n-grams also found in training questions with the same answer.Source: §6.1 · p. 7
• IR-targeted questions show a larger named-entity decrease than RNN-targeted questions; the authors suggest this may explain their generalization, given IR sensitivity to specific words and RNN sensitivity to phrasing.Source: §6.1 · p. 7

6.2 Categorizing Adversarial Phenomena

• The authors manually classify edit logs from a random sample of 100 questions into six phenomena across two broad categories, double-counting questions when applicable.Table 3 summarizes the phenomenon breakdown.Source: §6.2 · p. 7, Table 3 · p. 8

6.2.1 Adversarial Category 1: Reasoning

• Reasoning questions include composing seen clues, applying logic or calculations, and making multiple reasoning steps between entities.Examples require linking Tecumseh to the Battle of the Thames, adding 150 to 300 Spartans, or moving from the “I Have A Dream” speech to Lincoln.Source: §6.2.1 · p. 8

6.2.2 Adversarial Category 2: Distracting Clues

• Distracting clues include paraphrases that remove exact n-gram matches, entity-type distractors that prompt the wrong answer type, and novel clues not uniquely associated with an answer.Novel clues can be absent from Wikipedia or Quizbowl data and may confuse neural models while IR models largely ignore them.Source: §6.2.2 · p. 8
• A Bill Clinton saxophone clue in training does not prevent the RNN from predicting Don Cheadle when a lead-in distracts it toward an actor answer.Source: §6.2.2 · p. 8
• A novel shaving-history clue makes the RNN miss a previously recognizable “World Economic Herald” clue, while IR-targeted edits are described as generalizing to neural models.Source: §6.2.2 · p. 9

7 How Do Interpretations Help?

• The section analyzes edit logs to examine how authors modify questions in response to model interpretations.The logs record the interactive authoring process.Source: §7 · p. 9
• In one edit sequence, removing a sentence makes the IR model answer earlier, but a subsequent interpretation-guided modification breaks it.The figure tracks question length and the position where the model is first correct; lower buzzing position is better.Source: §7 · p. 9, Figure 9 · p. 10
• Interpretation-guided edits can break the IR model, but the illustrated targeted edits fail to trick the RNN.The examples include replacing highlighted “ellipse” and “parabola” with a phrase associated with Sphere, and replacing “molecules” with “species.”Source: Figure 8 · p. 9, §7 · pp. 9–10, Figure 9 · p. 10

7.1 Interviews With Adversarial Authors

• Authors interviewed at live events said the interface helped reveal stock clues they had not recognized, including a Kraków clue the model immediately recognized.One author described the interface as revealing clues they had not thought were stock.Source: §7.1 · p. 10
• A national Quizbowl champion said computers handled direct battle or poetry references better, while author-specific writing styles made questions harder for them.The interface’s evidence panel lets authors read existing clues and encourages these distinctive stylistic choices.Source: §7.1 · p. 10

8 Related Work

• The work uses human adversaries to expose model weaknesses across phenomena including paraphrases and multi-hop reasoning, unlike datasets focused on a particular task or genre.The cited dataset examples include book-passage contexts and questions users trust search engines to answer.Source: §8 · p. 10
• Human adversaries generate hard questions rather than filtering out easy ones, extending prior human-adversary work with model interpretations and a ready pool of Quizbowl writers.Other approaches filter examples using baseline QA models or stylistic classifiers; Ettinger et al. (2017) also use human adversaries.Source: §8 · p. 10
• Recent NLP adversarial methods often target one phenomenon, such as syntactic modification, and complement this human-in-the-loop approach.Related work also describes human-computer collaborations in chess, Starcraft, and simultaneous interpretation.Source: §8 · pp. 10–11

9 Conclusion

• The conclusion brings together visualizations of model capabilities and human-crafted adversarial examples that reveal remaining gaps from human performance.Source: §9 · p. 11
• Neural and IR QA models still struggle to synthesize clues, handle distracting information, and adapt to unfamiliar data.The authors describe their adversarially authored dataset as one of many possible iterations for examining future systems.Source: §9 · p. 11
• The authors say the procedure may apply beyond QA when a task has talented authors who write text with specific goals.They suggest future work could create adversarially authored datasets for tasks meeting those conditions.Source: §9 · p. 11

A Failure of Syntactically Controlled Paraphrase Networks

• Only six of 200 sampled SCPN paraphrases preserved all original information, even when paraphrasing the shortest sentence from each question.The model processes sentences independently, cannot paraphrase paragraphs, and often produces ungrammatical, repetitive, or incomplete paraphrases.Source: A Failure of Syntactically Controlled… · p. 15
• SCPN paraphrases can fail to preserve named entities, which the authors say is vital to Quizbowl question validity.They note this was surprising because SCPN incorporates a copy mechanism.Source: A Failure of Syntactically Controlled… · p. 15

B Studio Ousia Quizbowl Model

• Studio Ousia combines neural text classification and IR scores with entity-type matching, knowledge-graph entities, and customized word vectors.Its entity-type classifier predicts types such as religious leader or government agency.Source: B Studio Ousia Quizbowl Model · p. 15

C Failed Adversarial Attempt

• The author’s repeated modifications fail to break the neural Quizbowl model.Figure 10 shows the user's failed attempt to break the system.Source: Figure 10 · p. 15, C Failed Adversarial Attempt · p. 15
• The author modifies the question multiple times during the attempt.Source: Figure 10 · p. 15
• The modifications replace words suggested by the model’s interpretation.Source: Figure 10 · p. 15