Source-linked AI summary
Adversarial Examples for Evaluating Reading Comprehension Systems
Robin Jia, Percy Liang
TL;DR
Standard average-case evaluation may reward reading-comprehension models that rely on superficial cues rather than deeper language understanding. The paper introduces adversarial evaluation for SQuAD by adding compatible, automatically generated distracting sentences, and finds substantial F1 declines across published models. These results motivate evaluation methods and models that better distinguish answer-bearing sentences from merely related ones.
Problem
Average F1 can reward models that exploit superficial cues, leaving their genuine language understanding unclear.
Method
The paper evaluates SQuAD systems on paragraphs augmented with adversarial sentences that do not contradict the correct answer.
Results
Across sixteen published open-source models, grammatical adversarial sentences reduce average F1 score from 75% to 36%.
Takeaways & Limitations
Adversarial evaluation reveals that existing reading-comprehension models are overly stable to semantic changes and helps distinguish shallow pattern matching from language understanding.
Takeaways & Limitations
ADDANY does not ensure that generated sentences are compatible with the original answer; in practice, its outputs are gibberish without semantic content.
Abstract
from arXiv · showhide
Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear. To reward systems with real language understanding abilities, we propose an adversarial evaluation scheme for the Stanford Question Answering Dataset (SQuAD). Our method tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences, which are automatically generated to distract computer systems without changing the correct answer or misleading humans. In this adversarial setting, the accuracy of sixteen published models drops from an average of $75\%$ F1 score to $36\%$; when the adversary is allowed to add ungrammatical sequences of words, average accuracy on four models decreases further to $7\%$. We hope our insights will motivate the development of new models that understand language more precisely.
1 Introduction
The paper argues that average test accuracy can reward superficial pattern matching rather than genuine language understanding, and proposes adversarial evaluation for SQuAD to expose this weakness. Its adversarial sentences distract models without changing the correct answer, producing large performance drops.
- Standard held-out accuracy can be high when models exploit predictive patterns while ignoring deeper language phenomena.
- Adversarial evaluation tests NLP systems on adversarially chosen inputs rather than only average-case test examples.
- SQuAD evaluates systems that answer questions about Wikipedia paragraphs, with state-of-the-art performance at 84.7% F1 versus 91.2% for humans.
- Adversarial sentences are automatically generated to confuse models without contradicting the correct answer or confusing humans.
- Across sixteen published open-source models, grammatical adversarial sentences reduce average F1 score from 75% to 36%.
2 The SQuAD Task and Models
SQuAD is a large Wikipedia-based reading-comprehension benchmark in which each question targets an answer span in one paragraph. The paper evaluates several published neural architectures and uses standard F1-based accuracy over paragraph-question pairs.
- 2.1 Task: SQuAD contains 107,785 human-generated questions about Wikipedia articles, each linked to one paragraph and an answer span in that paragraph.
- 2.2 Models: The development experiments use BiDAF and Match-LSTM, each in single and ensemble versions, for four systems total.
- 2.2 Models: Twelve additional published models with public test-time code form a held-out set for validating the generality of the approach.
- 2.3 Standard Evaluation: Standard accuracy averages the F1 score between the true answer and the model prediction over the test set.
3 Adversarial Evaluation
The paper defines adversarial evaluation for reading comprehension by appending compatible, semantics-altering sentences that distract models while leaving the question and answer unchanged. It introduces ADDSENT and ADDANY, with procedures ranging from question-like grammatical distractors to arbitrary word sequences.
- General Framework: Adversarial accuracy measures the fraction of examples on which a model remains correct under adversarially chosen alterations.Valid alterations must preserve the human-correct answer and remain close to the original example.
- Concatenative Adversaries: Concatenative adversaries transform an example into (p + s, q, a) by appending a sentence compatible with the original answer.The question and answer remain unchanged, while the added sentence may alter semantics without contradicting the answer.
- ADDSENT: ADDSENT’s distractors use semantics-altering question changes while remaining compatible with the correct answer, exposing models’ difficulty distinguishing related distractors from answers.The procedure replaces nouns and adjectives with antonyms and changes named entities or numbers to nearby same-part-of-speech words before generating the sentence.
- ADDSENT and ADDANY: ADDSENT adds grammatical sentences resembling the question, whereas ADDANY adds arbitrary sequences of English words with greater power to confuse models.ADDSENT uses four steps: alter question semantics, generate a same-type fake answer, convert the result to declarative form, and crowdsource grammatical edits.
- Variants: ADDONESENT is model-independent because it adds a random human-approved sentence without access to the evaluated model or its training data.ADDCOMMON provides another ADDANY variant that adds only common words rather than incorporating question words.
- ADDANY: ADDANY does not ensure compatibility with the original answer; its generated sentences are typically gibberish containing question words but no semantic content.This provides a stronger but less linguistically constrained distraction setting than grammatical adversarial sentences.
4.1 Setup
The experiments evaluate adversarial F1 on 1,000 randomly sampled SQuAD development examples, using downsampling because some adversaries require thousands of model queries per example.
- 1,000 randomly sampled SQuAD development examples were used for all experiments.The test set was unavailable, so evaluation used the development set.
- Adversarial F1 score was the evaluation metric.
- Downsampling reduced runtime for ADDANY and ADDCOMMON, which can issue thousands of model queries per example.The authors report that the measured effects remained statistically significant.
4.2 Main Experiments
Adversarial evaluation substantially reduced model performance, with grammatical and nongrammatical attacks producing large F1 declines across evaluated systems. The adversaries also transferred across models, while Mnemonic Reader performed better than other models.
- 75.7% to 31.3%: ADDSENT reduced average F1 across four models.The evaluated models were Match-LSTM and BiDAF variants.
- 6.7%: ADDANY reduced average F1 across the four models.ADDANY was more effective than ADDSENT in these experiments.
- 46.1%: ADDCOMMON reduced average F1 despite adding only common words.
- Mnemonic Reader models outperformed the other models by about 6 F1 points.The authors hypothesize that self-alignment helps locate supporting evidence and distinguish correct from fake answers.
4.3 Human Evaluation
Human evaluation indicates that ADDSENT affects people much less than computer systems, and that much of the human accuracy loss reflects ordinary mistakes rather than adversarial sentences.
- Much of the ADDSENT human decrease was attributable to mistakes unrelated to the adversarial sentences.ADDSENT selects the worst case over up to five paragraph-question pairs, increasing opportunities for ordinary human error.
- 3.4 F1 points: human accuracy dropped on ADDONESENT.ADDONESENT provides a more meaningful human evaluation because it does not select among up to five adversarial pairs.
4.4 Analysis
The analysis examined attack validity, model and human error patterns, perturbation types, factors associated with robustness, and transferability across models. Successes were associated with lexical overlap and shorter questions, while adversarial examples transferred effectively between models.
- 4.4.1 Manual verification: Only one of 100 manually checked BiDAF Ensemble failures contained a sentence interpretable as answering the question.The check found seven minor grammar errors, but none materially impeded understanding.
- 4.4.2 Model errors: 96.6% of model failures involved predicting a span in the adversarial sentence.BiDAF models sometimes produced very long spans because their start and end positions are predicted independently.
- 4.4.2 Human errors: Humans predicted from the adversarial sentence in only 27.3% of human error cases.This supports the conclusion that many human errors were ordinary mistakes unrelated to the adversarial sentence.
- 4.4.3 Perturbation types: In 100 BiDAF Ensemble failures, entity names changed in 75 cases, numbers or dates in 17, and antonyms of question words appeared in 33.Counts exceed 100 because multiple words could be altered in one example.
- 4.4.4 Reasons for model successes: 41.5% of BiDAF Ensemble successes had a 4-gram match with the original paragraph, compared with 21.0% of failures.Across models and n values, successes were more likely than failures to contain question n-gram matches.
- 4.4.5 Transferability: ADDSENT adversarial examples transferred effectively between models, whereas ADDANY examples showed more limited transferability.Transfer was slightly better between single and ensemble versions of the same model.
- 4.4.4 Reasons for model successes: 32.7% of BiDAF Ensemble successes involved questions of 8 words or fewer, compared with 11.8% of failures.Successes generally involved shorter questions, partly because changing one word leaves more overlap in long questions.
4.5 Transferability across Models
Adversarial examples generated against one reading-comprehension model can transfer to others, but transferability depends on the adversary-generation method.
- ADDONESENT examples transfer across models because the method always adds the same adversarial sentence regardless of the model.
- ADDSENT examples transfer between models quite effectively and are harder than ADDONESENT examples.
- ADDSENT and ADDANY differ in transferability, with ADDANY examples exhibiting more limited transfer between models.
4.6 Training on Adversarial Examples
Training BiDAF on raw adversarial examples appears to improve performance against ADDSENT, but the benefit is limited because the model can overfit the adversary’s specific patterns.
- Training augmented with raw ADDSENT examples largely protects BiDAF against ADDSENT.The augmented training set combined original examples with raw adversarial examples generated without crowdsourcing.
- On ADDSENTMOD, the retrained model performs almost as badly as the original model.ADDSENTMOD changes the fake answers and prepends the adversarial sentence rather than appending it.
- The ADDSENTMOD result suggests that the retrained model learned to ignore the last sentence and reject ADDSENT’s usual fake answers.
- Effective adversarial training requires preventing the model from overfitting to the particular adversary.
5 Discussion and Related Work
The discussion argues that adversarial evaluation exposes weaknesses missed by standard metrics, supports broader out-of-distribution testing, and motivates more robust evaluation and training strategies.
- Discussion: Existing reading-comprehension systems perform poorly under adversarial evaluation despite appearing successful under standard metrics.The paper attributes standard evaluation’s leniency to models relying on superficial cues.
- Related Work: Adversarial evaluation metrics may require new training strategies, but adversarial training can be infeasible when generating examples is slow.
- Related Work: Adversarial evaluation differs from generative adversarial models: the former evaluates language understanding, whereas the latter pits generation against discrimination.
- Discussion and Related Work: Perturbing test examples can introduce out-of-distribution phenomena while leveraging existing data-collection efforts.
- Discussion: Concatenative adversaries suit reading comprehension, while other tasks may require methods such as paraphrases, meaning-altering perturbations, or recombination.
- Discussion: The paper releases scripts for ADDSENT and code for ADDANY to support adversarial evaluation of SQuAD systems.