Source-linked AI summary
longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks
Venelin Kovatchev, Trina Chatterjee, Venkata S Govindarajan, Jifan Chen, Eunsol Choi, Gabriella Chronis, Anubrata Das, Katrin Erk, Matthew Lease, Junyi Jessy Li, Yating Wu, Kyle Mahowald
TL;DR
Longhorns addressed how to create adversarial questions that fool extractive QA models while remaining answerable by people. They used a systematic, linguistically informed approach, and reported first place with a 62% Model Error Rate, pending expert validation.
Problem
The paper addresses how to generate successful adversarial examples for NLP systems, a task for which the authors say there is no straightforward recipe.
Source: §1 · p. 1Method
Longhorns used linguistically motivated attacks targeting lexical knowledge, syntax, discourse, and reasoning phenomena.
Source: §4 · p. 3Results
Longhorns reported first place in DADC Task 1 with a 62% Model Error Rate, pending expert validation of the results.
Source: §1 · p. 1Takeaways & Limitations
The authors argue that linguistically unusual but human-understandable questions are fair targets for improving systems and that their approach may extend beyond QA.
Source: §1 · p. 2, §7 · p. 8Most passages were short, simple declarative texts with few named entities and little information, limiting the complexity of the evaluated setting.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team "longhorns" on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first, with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.
1 Introduction
Longhorns used a systematic, linguistically informed process to develop adversarial questions for extractive QA. The team reported first place in the competition with a 62% Model Error Rate, pending expert validation.
1 Introduction
- Dynamic benchmarking uses human-created examples that fool a state-of-the-art model but not a human, to reveal system strengths and weaknesses.The authors suggest their theoretically motivated approach may extend to NLU tasks beyond QA.Source: §1 · pp. 1–2
- 62% Model Error Rate and first place were reported for Longhorns, pending expert validation of the results.The competition result was reported at the time of paper submission.Source: §1 · p. 1
- The team compiled linguistically motivated attack strategies, analyzed model biases and performance in pilot studies, and used the most successful attacks for its official submission.The approach was designed to be systematic and analytical.Source: §1 · p. 1
- Attacks exploiting strong model priors included same-type distractors, which could draw answers toward salient or lexically overlapping text.The model often identified the expected entity type but favored salient answers or answers with high lexical overlap.Source: §1 · pp. 1–2
- Questions using processing-difficult constructions often fooled the model, though the team collected no human data to measure processing difficulty.The authors suggest these questions may challenge real-time human processing while remaining answerable with enough time.Source: §1 · p. 2
- Questions requiring numerical, temporal, commonsense, or list reasoning were also difficult for the model.The authors relate these tasks to reasoning beyond linguistic processing.Source: §1 · p. 2
2 Task Definition
Task 1, “Better Annotators,” asked teams to produce extractive QA examples that a model would get wrong but people could answer. Entries were submitted through Dynabench for evaluation.
2 Task Definition
- Each team submitted 100 official extractive QA examples through Dynabench.The shared task provided short passages as contexts.Source: §2 · p. 2
- Questions had to be answerable from a span in the passage and could not be answered with yes or no.Source: §2 · p. 2
- The winning team was the one with the highest validated model error rate, with examples intended to fool the model but remain answerable by competent humans.Dynabench is described as supporting human-and-model-in-the-loop dataset creation.Source: §2 · p. 2
3 Approach
Longhorns developed a broad list of hypothesized linguistic and reasoning challenges, used pilot experiments to compare attacks, and selected promising strategies for the official submission. Team members tailored attacks to the passages and often reached decisions by consensus.
3 Approach
- The team drew on computational, psycholinguistic, and theoretical linguistics to compile a non-exhaustive list of potential attack strategies.The authors say the list could guide adversarial-example creation for other tasks and systems.Source: §3 · p. 2
- Separate pilot experiments explored strategies before or after the official task to preserve the fairness of the competition submission.Source: §3 · p. 3
- Pilots measured each strategy’s Model Error Rate on a small sample and how frequently it could be used; the strongest strategies informed the official submission.The team adapted its focus to the passage, and participants submitted questions by consensus when working together.Source: §3 · p. 3
- Generating questions took 5–10 minutes per passage, and the team sometimes skipped unpromising passages or generated multiple questions for promising ones.The authors anecdotally found attacks more successful when multiple team members considered model behavior from different angles.Source: §3 · p. 3
4 Pilot Experiments: Evaluating Adversarial Strategies
The pilot strategy space covered lexical, syntactic, discourse, and reasoning phenomena, alongside attacks using distractors. The team assessed both model error rates and how often each strategy could be applied.
4 Pilot Experiments: Evaluating Adversarial Strategies
- Pilot attacks targeted lexical knowledge, including entailments, named-entity properties, nominalization, relations, polarity, comparisons, modality, and quantifiers.Source: §4 · p. 3
- Syntax and discourse attacks tested genitives, datives, relative clauses, conjunction and disjunction scope, and intersectivity and restrictivity.Source: §4 · p. 3
- Other attacks targeted prepositional-phrase attachment, argument structure, repeated verbs, ellipsis, anaphora, and coreference.Source: §4 · pp. 3–4
- A nonsensical question could elicit an answer with high semantic overlap with the question, illustrating semantic-similarity bias.Source: Figure 1 · p. 4
- Negation attacks tested whether the model could handle forms such as adverbial, morphological, lexical, implicit, and double negation and identify their scope.Source: §4 · p. 4
- Garden Path questions and questions about people’s mental states drew on syntactic processing difficulty and challenges in reasoning about others’ beliefs.The examples distinguish why someone thinks another person is leaving from why that person is leaving.Source: §4 · p. 4
- Reasoning attacks included conditionals, hypothetical situations, numerical and temporal reasoning, commonsense reasoning, and list manipulation.Source: §4 · p. 4
- Distractors added information to steer the model toward a wrong answer, while noise increased question complexity; the team measured error rate and attack frequency.The strategies could occur in the passage, question, or both, and not every strategy fit every passage.Source: §4 · p. 4
5 Exploring Model Biases
The authors examined shortcuts behind the model’s unexpectedly strong performance on preliminary attacks, including semantic similarity and answer-type heuristics. Their official submission used these biases to steer the model toward wrong answers or make its heuristics harder to use.
5 Exploring Model Biases
- When unsure, the model matches question and context by semantic similarity, sometimes answering correctly without understanding, especially in short passages.The paper illustrates this with a nonsensical question answered using a neighboring context word.Source: §5 · pp. 4–5
- The model often returns a salient answer of the right entity type regardless of the question; short passages with few matching tokens can artificially boost performance.Figure 2 shows it assigning Stephen Austin a Golden Globe award despite the passage being about Texas colonization.Source: §5 · p. 5, Figure 2 · p. 5
- The official submission used these biases to direct the model toward wrong answers or rephrased questions to impede its heuristics when distractors were unavailable.Source: §5 · p. 5
6 Official Submission
The official submission combined model-prior attacks with linguistically difficult questions and non-linguistic reasoning challenges. The examples show where these strategies can fool the model, and where passage properties make them easier or harder to use.
6 Official Submission
- The team focused its official submission on distractors, numerical and temporal reasoning, garden paths, coreference, list manipulation, and common-sense reasoning, using biases to confuse the model or limit its heuristics.Source: §6 · p. 5
6.1 Taking advantage of model priors
- Distractors were among the most successful and easy-to-use strategies; they work best when the correct answer is less salient and many same-type alternatives appear.The question wording can resemble a passage detail while the correct answer depends on another feature, such as a size boundary.Source: §6.1 · p. 5
6.2 Linguistically difficult utterances
- A garden-path question delays its intended interpretation until the last word; the model answers Dion instead of Holly, though such questions are easier to generate than numerical or temporal ones.The authors note that the model often handles complex syntax, possibly because passages are short and it can rely on question-type bias.Source: §6.2 · p. 6
- Coreference attacks require resolving a pronoun to its antecedent, but are relatively infrequent because they need multiple entities and potentially ambiguous references.In the example, the model returns a salient named entity instead of resolving “their” to “the founders.”Source: §6.2 · p. 6
6.3 Non-linguistic reasoning
- Numerical attacks require simple calculations or comparisons, while temporal attacks ask the model to place an unstated date between events; temporal examples are rare but very successful when available.The example compares “between 80 and 90” with a 50-AU boundary; another asks about June 13, 1935, between moves in 1934 and 1936.Source: §6.3 · p. 7
- List attacks either request a complete list or a criterion-matching subset; the example asks for nine composers but elicits only three, and list attacks are infrequent in single-passage settings.Source: §6.3 · p. 8
- Common-sense attacks test whether the model can apply a naming convention to a hypothetical case, such as inferring that a 737-300C is cargo-equipped because “C” denotes cargo.Source: §6.3 · p. 8
7 Discussion
The most successful attacks depart from ordinary cooperative language through surprising content, difficult constructions, and reasoning demands, yet can still be meaningful to people. The discussion connects these attacks to model weaknesses while noting that results depend on both question strategy and passage difficulty.
7 Discussion
- Successful attacks depart from cooperative language by asking about surprising aspects of a passage and using complex language or reasoning.Examples include distractors, garden-path questions, complex coreference, and temporal or numeric reasoning.Source: §7 · p. 8
- The authors consider linguistically unusual adversarial questions a fair target because language can express subtle, complicated meanings beyond the average case.They connect this view to language’s flexibility and generativity.Source: §7 · p. 8
- Larger groups generated questions more successfully, which the authors associate with a wider diversity of ideas.They suggest that generating these questions may require creative thinking outside ordinary linguistic experience.Source: §7 · p. 8
- Although such questions may be less common in extractive QA datasets, the authors argue they remain understandable and useful in real-world settings such as automated assistants.They may ask where someone was at a particular time or compare two aircraft models.Source: §7 · pp. 8–9
- High MER on attacks targeting coreference, numerical reasoning, and temporal reasoning indicates model underperformance on those capacities.The authors describe these attacks as having practical as well as scientific implications.Source: §7 · p. 9
- Only a small subset of strategies consistently exceeded 50% MER, and those strategies did not work for every question.Fewer possible entities matching a question type also meant fewer available distractors.Source: §7 · p. 9
- The model’s performance depends on passage difficulty; the mostly short, simple passages made question generation and answering relatively trivial.The authors expect more complex paragraphs to lower model performance and allow more creative, successful attacks.Source: §7 · p. 9
8 Conclusions
The longhorns team presents a systematic, linguistically informed, collaborative approach to generating adversarial examples. The authors report better competition results than prior work, insights into model behavior, and strategies they believe can extend beyond extractive QA.
8 Conclusions
- The team reports that its systematic, linguistically informed, collaborative approach obtained the competition’s best results.The approach was used to generate high-quality adversarial examples for Task 1 of DADC.Source: §8 · p. 9
- The approach outperformed prior work in model error rate and provided insights from pilot analyses of linguistic phenomena and model biases.The authors also say the systematic approach offers insight into data quality and difficulty, though the supplied passage does not specify further details.Source: §8 · p. 9
- The authors say their strategies are not limited to extractive QA and could be adopted at larger scale across tasks to improve model and data quality.They describe the work as opening research directions with scientific and practical implications.Source: §8 · p. 9