Source-linked AI summary
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Rowan Zellers, Yonatan Bisk, Roy Schwartz, Yejin Choi
TL;DR
The paper addresses grounded commonsense inference: predicting plausible future events from situated language while reducing annotation artifacts that can inflate benchmark performance. It introduces SWAG and Adversarial Filtering, then finds that humans achieve 88% accuracy while state-of-the-art NLI models remain below 60%.
Problem
Grounded inference requires commonsense reasoning beyond linguistic entailment, while human-biased datasets risk overstating performance and vulnerability to adversarial examples.
Method
The paper constructs SWAG by oversampling counterfactual endings with a language model and aggressively filtering them using an ensemble of stylistic classifiers.
Results
88% human accuracy contrasts with below-60% performance from current state-of-the-art NLI models on SWAG.
Takeaways & Limitations
Adversarial Filtering supports cost-effective construction of large-scale benchmarks while substantially reducing known annotation artifacts.
Takeaways & Limitations
The filtering models focus on stylistic features in the second sentence, so subtle artifacts likely remain in SWAG.
Abstract
from arXiv · showhide
Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine"). In this paper, we introduce the task of grounded commonsense inference, unifying natural language inference and commonsense reasoning. We present SWAG, a new dataset with 113k multiple choice questions about a rich spectrum of grounded situations. To address the recurring challenges of the annotation artifacts and human biases found in many existing datasets, we propose Adversarial Filtering (AF), a novel procedure that constructs a de-biased dataset by iteratively training an ensemble of stylistic classifiers, and using them to filter the data. To account for the aggressive adversarial filtering, we use state-of-the-art language models to massively oversample a diverse set of potential counterfactuals. Empirical results demonstrate that while humans can solve the resulting inference problems with high accuracy (88%), various competitive models struggle on our task. We provide comprehensive analysis that indicates significant opportunities for future research.
1 Introduction
SWAG frames inference as anticipating plausible future events from grounded situations, extending entailment with commonsense reasoning. It uses adversarial filtering to reduce stylistic artifacts while constructing a large-scale benchmark that remains difficult for models but easy for humans.
- Task motivation: Grounded commonsense inference asks whether a multiple-choice ending describes a possible future world anticipated from a situation, not merely a strictly entailed hypothesis.The task requires understanding everyday physical situations, including object affordances and frame semantics.
- Dataset motivation: Human-biased datasets can overestimate model performance and remain vulnerable to adversarial or out-of-domain examples.
- Approach: Adversarial Filtering automatically detects and reduces stylistic artifacts while constructing Swag, a dataset with 113k multiple-choice questions.
- Results: 88% human accuracy contrasts with below-60% performance from current state-of-the-art NLI models on grounded commonsense inference.The comparison presents SWAG as easy for humans but hard for competitive models.
- Results: Adversarial Filtering makes large-scale dataset construction more cost-effective while substantially reducing known annotation artifacts.The paper presents the method as general enough to support future benchmark construction.
2 Swag: Our new dataset
SWAG is a dataset and task for predicting which event most likely occurs next in a video-derived situation. Its examples use partial sentence contexts and candidate verb-phrase endings, with the dataset built from sequential captions and adversarially generated alternatives.
- Task: The task provides a complete sentence, a noun phrase beginning a second sentence, and four possible verb-phrase endings.The model selects the most appropriate ending.
- Dataset: SWAG is short for Situations With Adversarial Generations.
- Data collection: For each pair of sequential captions, the second caption is split into noun and verb phrases before negative endings are generated and difficult examples are human-annotated.
- Dataset: SWAG contains 113k multiple-choice questions derived from pairs of consecutive video captions.The source material includes ActivityNet Captions and the Large Scale Movie Description Challenge.
3 A solution to annotation artifacts
The paper addresses annotation artifacts by adversarially filtering generated counterfactual endings with stylistic models, then validating the resulting examples with human annotators. The procedure reduces stylistic predictability while retaining physically plausible alternatives for large-scale grounded inference.
- Adversarial Filtering: Adversarial Filtering iteratively refines negative-example assignments to increase dataset difficulty for a chosen family of classifiers.Each iteration trains on one split, identifies negatives classified correctly as easy, and replaces them with misclassified negatives from outside the current assignment.
- Adversarial Filtering: The formal construction pairs each context-specific positive example with a filtered k-subset of a much larger population of negative examples.The target is a filtered dataset with k much smaller than the original negative population.
- Generating candidate endings: The counterfactual generator samples 1023 unique endings for each partial caption using an LSTM language model conditioned on video-caption contexts.The model is pretrained on BookCorpus, fine-tuned on video-caption datasets, and uses greedy sampling to avoid easily distinguishable low-perplexity outputs.
- Stylistic models for adversarial filtering: The final adversarial-filtering ensemble combines four stylistic models using perplexity, length, lexical, convolutional, and recurrent features, and trains them jointly each iteration.The ensemble includes an MLP, bag-of-words model, one-layer CNN, and bidirectional LSTM targeting low-level stylistic patterns.
- Stylistic models for adversarial filtering: 60% to close to random chance: ensemble accuracy falls across AF iterations, with the substantial drop occurring only after multiple stylistic models are included.Confusing the perplexity-based MLP alone is insufficient to lower ensemble performance, whereas adding the other stylistic models reduces it substantially.
- Human verification: 73.7% of the time: Turkers ranked the found ending as best or second best, allowing it to serve as a gold example with generated negatives.Workers evaluated one found ending and five adversarially sampled endings, ranking each as likely, unlikely, or gibberish and selecting the best two.
4 Experiments
The experiments evaluate models for selecting plausible event endings in SWAG, testing context use, representation choices, and human agreement. Models improve with additional context but remain below human performance.
- Task and models: SWAG models select a verb-phrase ending from four candidates given a sentence and noun-phrase context.Training minimizes multiclass cross-entropy using GloVe, Numberbatch, or ELMo representations.
- Task and models: The experiments also test whether models can predict answers from only the ending verb phrase or the entire second sentence.These input restrictions are used to study dataset bias.
- Results: 43.6% accuracy is achieved by the best ending-only model, an LSTM sequence model with ELMo embeddings.Adding the initial noun phrase improves performance by 3.1%, and adding the first sentence yields an additional 4%.
- Results: 59.2% accuracy is achieved by ESIM+ELMo when fully trained on SWAG, the strongest reported pairwise NLI result.Pairwise representations improve over DualBoW at 35.1% and InferSent-Bilinear at 40.5%.
- Human performance: 88% accuracy is achieved by an ensemble of five Mechanical Turk workers, exceeding the in-house expert annotator and indicating substantial machine headroom.Workers answered 100 questions by majority vote, while all models substantially underperform humans.
5 Analysis
SWAG differs from conventional NLI through temporal, dynamic, and physically grounded reasoning, while adversarial filtering reduces stylistic shortcuts. Humans strongly outperform current models, whose errors chiefly reflect missing grounded physical understanding.
- Dataset differences: SNLI-ESIM reaches only 36.1% accuracy on SWAG, indicating that SWAG requires temporal reasoning distinct from SNLI entailment.
- Dataset differences: Compared with SNLI, SWAG contains more active verbs such as “pull” and “hit” and fewer static verbs such as “sit” and “wear.”
- Dataset differences: FastText achieves 67.0% on SNLI but only 29.0% on SWAG, indicating substantially fewer lexical biases in SWAG.
- Error analysis: Humans preferred the gold ending in 94% of model-error cases, with situational reasoning the most common justification at 52.3%.Weirdness accounted for 17.5%, plausibility for 14.4%, and ambiguity for 12.0%.
- Error analysis: Qualitative errors show that sentence-alignment strategies are insufficient when answering requires object knowledge and intuitive physical reasoning.The examples include knowledge about how bumper cars differ from regular cars and intuitive physical reasoning about situations.
- Limitations: SWAG remains vulnerable to subtle artifacts because its filtering models are stylistic and focus on the second sentence.The authors propose applying stronger language and filtering models to create more adversarial versions.
6 Related Work
SWAG extends NLI and commonsense reasoning benchmarks with larger-scale, physically grounded inference over longer descriptions. It also addresses annotation artifacts, while complementing prior temporal, visual, demographic-bias, and physical-knowledge resources.
- Entailment NLI: Earlier NLI benchmarks focus on linguistic entailment, whereas SWAG targets grounded commonsense reasoning and annotation artifacts at larger scale.
- Commonsense NLI: Commonsense NLI datasets study causes, story endings, or hypothesis likelihood, but COPA and RocStories are relatively small at 1k and 10k examples.
- Commonsense NLI: Unlike sentence-completion tasks that predict one missing word, SWAG requires reasoning over longer textual descriptions.
- Vision datasets: Prior visual resources study temporal inference in images, photo albums, sports videos, and egocentric video, providing related but distinct settings.
- Reducing gender/racial bias: SWAG seeks to minimize elicitation-based demographic biases, but movie data can still contain gender and racial biases.
- Physical knowledge: SWAG evaluates both grounded physical knowledge and script-like event-chain representations developed in earlier work.
- Commonsense NLI: Heuristic perplexity filtering in prior sentence-completion work remained gameable, motivating more robust filtering approaches.
7 Conclusion
The paper introduces physically situated commonsense inference as a broader NLI challenge and provides SWAG, a large-scale benchmark constructed with Adversarial Filtering. AF is designed to scale dataset construction while reducing detectable annotation artifacts.
- 7 Conclusion: The paper proposes physically situated commonsense inference, broadening natural language inference with commonsense reasoning.
- 7 Conclusion: SWAG contains 113k multiple-choice questions and is constructed using Adversarial Filtering to reduce annotation artifacts detected by strong baseline models.
- 7 Conclusion: Adversarial Filtering is presented as a general paradigm for robust, cost-effective dataset construction that can apply beyond SWAG.
A.1 More detail about video datasets
The video sources differ in structure and temporal reliability, so the dataset construction uses consecutive captions selectively. ActivityNet supplies clip-level activity descriptions, while LSMDC captions may be separated by substantial time gaps.
- Video sources: ActivityNet provides videos already divided into clips, whereas LSMDC contains captions spanning entire movies.
- Caption selection: LSMDC’s temporally adjacent captions can describe events far apart in time, so pairs with more than 25 seconds between captions are excluded.
- Dataset choices: DiDeMo was considered but omitted because many referring expressions are sentence fragments.
- Dataset choices: Visual Madlibs was omitted because its worker-written hypothetical next-event captions differ fundamentally from the rest of SWAG’s data.
A.2 Details of the language model
The language model ties input and output embeddings, uses 512-dimensional layers and recurrent dropout, and adds a backward model sharing embeddings. Held-out development perplexities were 31.2 forward and 30.4 backward, while more complex architectures overfit without improving results.
- The model ties input and output embeddings and sets all embedding and hidden layers to 512 dimensions.
- Recurrent dropout is applied to hidden states and the embedding layer.
- A backward language model shares embedding parameters with the forward model, adding embedding supervision and another scoring direction.
- 31.2 forward and 30.4 backward perplexity were obtained on the final development set.
- More complicated language-model architectures did not improve performance because of overfitting.
A.3 Language model features for the MLP, during adversarial filtering
During adversarial filtering, the bidirectional language model supplies perplexity and token-probability features from both context and ending directions. These features evaluate the context, ending conditioned on context, context conditioned on ending, ending alone, and sentence termination.
- Perplexities are extracted for the context alone and the ending conditioned on the context in the forward direction.
- Backward features include perplexity for the context given the ending and for the ending by itself.
- The system also records the final generated token’s forward probability to detect unnatural sentence endings caused by the 25-token limit.
A.4 Refinining the generated answers to four distractors
The refinement procedure reduced a large pool of generated negatives to a small set of worker-reviewed distractors, supported by screening and agreement controls. The appendix also reports dataset statistics, topical diversity, qualitative examples, and verb-distribution comparisons with MultiNLI.
- A.4 Refining the generated answers to four distractors: 1023 generated negatives per example were filtered to 9, of which 5 were sent to Mechanical Turk workers.
- A.4 Refining the generated answers to four distractors: 62% of examples retained a fourth distractor after filtering, with the second-best turker-selected option always removed.
- A.4 Refining the generated answers to four distractors: 79% pairwise agreement was achieved for classifying whether an ending belonged in the Top 2.
- A.4 Refining the generated answers to four distractors: Turkers were screened before annotation and periodically dequalified when their agreement with gold endings was low.
- A.4 Refining the generated answers to four distractors: The dataset cost $23,000, averaging 20 cents per example, while median annotator pay was $8.57 per hour.
- A.4 Refining the generated answers to four distractors: The final dataset contains a vocabulary of 21,000 words and includes topic-model coverage visualizations.
- A.8 Comparing the distribution of verbs with MultiNLI: SWAG’s verb distribution is less skewed than MultiNLI’s up to roughly 120 verbs, after which MultiNLI is slightly less skewed.
- A.8 Comparing the distribution of verbs with MultiNLI: The comparison may reflect MultiNLI’s broader domains, while video data underrepresents words common in newswire text.