PaperlayerView exact revision on arXiv

Source-linked AI summary

HellaSwag: Can a Machine Really Finish Your Sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi

arXiv:1905.07830v1cs.CL

TL;DR

Whether pretrained models truly perform commonsense inference remained unclear despite BERT’s near-human SWAG results. HellaSwag uses adversarial filtering to create a harder benchmark, showing that humans achieve 95.6% accuracy while machines remain below 50%.

  • Problem

    The paper asks whether deep pretrained models genuinely perform commonsense natural language inference despite BERT’s near-human performance on SWAG.

  • Method

    The authors introduce HellaSwag, using Adversarial Filtering to select challenging machine-generated wrong answers for a commonsense NLI benchmark.

  • Results

    95.6% human accuracy versus below 50% machine accuracy shows that HellaSwag remains challenging for state-of-the-art models, even with extensive pretraining.

  • Takeaways & Limitations

    HellaSwag provides a challenging testbed indicating that strong NLI models still lack robust commonsense reasoning ability.

  • Takeaways & Limitations

    The benchmark’s effectiveness depends on a Goldilocks zone in which generators make mistakes while discriminators fail to detect them, tying it to progress in language generation.

Abstract

from arXiv · show

Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys." With the introduction of BERT, near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (>95% accuracy), state-of-the-art models struggle (<48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.

1 Introduction

The introduction argues that strong performance on SWAG did not establish robust commonsense inference, motivating HellaSwag as an adversarially constructed benchmark that remains easy for humans but difficult for machines. It further presents evolving, model-aware datasets as necessary for measuring progress on commonsense NLI.

  • Motivation: BERT exceeded 86% on SWAG, approaching humans’ 88%, after earlier state-of-the-art models scored ≤60%.This apparent success prompted the paper’s question about whether deep pretrained models genuinely perform commonsense NLI.
  • Problem: The paper concludes that the underlying commonsense NLI task remains unsolved because BERT relies on dataset-specific distributional biases rather than robust reasoning.The authors characterize deep models as rapid surface learners for particular datasets.
  • Contribution: HellaSwag uses Adversarial Filtering to select challenging generated wrong answers, producing 70k problems with 95.6% human accuracy yet machine performance below 50%.The result remains difficult for machines even with many training examples.
  • Method: The benchmark combines state-of-the-art generators, BERT discriminators, and diverse WikiHow text, targeting a Goldilocks zone of roughly three context sentences and two generated sentences.This design increases context diversity and generation length while making outputs nonsensical to humans but challenging for models.
  • Broader Implications: The paper advocates iterative dataset construction that uses modeling advances to filter spurious biases and keeps benchmarks evolving with the state of the art.The underlying task, rather than an individual dataset, is considered solved only when this cycle becomes impossible.

2 Background

SWAG frames commonsense NLI as choosing the actual next video-caption ending from four candidates, but human-written endings can introduce annotation artifacts. Adversarial Filtering addresses this by iteratively selecting difficult language-model-generated negatives, producing datasets that remain challenging for models across dataset splits and training conditions.

  • SWAG: SWAG asks models to select the actual next video-caption ending from four choices, with exactly one correct answer.
  • Limitations of human-written negatives: Human-written NLI endings can introduce subtle, strong class-conditional annotation artifacts that inflate model performance.Prior work also links such human-induced biases to unwanted gender and race biases.
  • Adversarial Filtering: Adversarial Filtering generates many incorrect candidates with an in-domain language model and iteratively selects them using an ensemble trained to distinguish real from generated endings.Each iteration randomly partitions the dataset into D_train and D_test before adversarial selection replaces easy-to-classify generated candidates.
  • Adversarial Filtering: AF produces datasets that challenge models regardless of the final dataset split, even with substantial training data from the same distribution as the test set.The paper uses AF to construct an NLI dataset that is easy for humans yet challenging for machines.

3 Investigating SWAG

The analysis finds that BERT’s strong SWAG performance depends on dataset scale but remains robust when context or word order is removed. Adversarial Filtering exposes a major generator effect: SWAG’s machine-generated endings remain distinguishable from human text, whereas GPT’s generations are substantially harder to discriminate.

  • Data scale: 64 examples let BERT outperform ESIM+ELMo’s 59% result, but it needs upwards of 16k examples to approach human performance before plateauing.ESIM+ELMo requires the entire training set to reach 59%.
  • Context and structure: 86.7%Ñ74.8%: omitting context reduces BERT’s performance by only 11.9 points, indicating bias in the endings themselves.The result suggests human- and machine-generated endings differ markedly even without contextual information.
  • Context and structure: Less than 10%: randomly shuffling ending words while retaining context barely reduces BERT’s performance, suggesting lexical reasoning over each pair.BERT adapts despite never seeing randomly shuffled text during pretraining.
  • Context and structure: 60.4%: removing context and shuffling endings still exceeds ELMo’s ~60% performance, implying that distributional stylistic patterns drive much of finetuning.Neither context nor structure is needed to distinguish human- from machine-written endings in most cases.
  • Adversarial Filtering: Roughly 75%: AF with BERT-Large never reduces SWAG-generation accuracy to chance, while GPT generations drive accuracy below 30% across many random subsplits.The contrast reveals that the generator strongly affects AF difficulty.

4 HellaSwag

HellaSwag removes artifacts from SWAG by combining ActivityNet and WikiHow contexts with adversarially filtered machine-generated endings. Human validation and zero-shot evaluation show that the resulting commonsense benchmark remains difficult for models despite high human agreement.

  • Dataset construction: HellaSwag uses ActivityNet video captions and their activity labels as additional structure for testing generalization.The dataset uses ActivityNet rather than LSMDC captions and includes labels such as “jumping rope.”
  • Dataset construction: 80k WikiHow context–follow-up paragraph pairs span diverse how-to topics, with each context and follow-up limited to three sentences.Examples range from making an origami owl to surviving a bank robbery.
  • Adversarial filtering: 35%, 40%, and 50% are the final accuracies for one-, two-, and three-sentence WikiHow endings after Adversarial Filtering.BERT begins at 70-90% in these settings, but filtering lowers accuracy considerably; the one-sentence setting converges only slightly above random.
  • Human and model performance: 93.5% is human performance on WikiHow, compared with 41.1% for BERT, while ActivityNet yields 60% for humans and 25.5% for BERT.The contrast shows that WikiHow is nearly trivial for humans, whereas ActivityNet is difficult for both groups.
  • Human validation: 94% human performance is reached through repeated filtering, retaining 25k ActivityNet and 45k WikiHow contexts with the highest worker agreement.Workers choose the true ending from six choices and replace machine endings judged realistic.
  • Generalization evaluation: 70k dataset examples support evaluation on 5k in-domain and 5k zero-shot examples for each validation or test set.Zero-shot subsets use randomly selected held-out categories from WikiHow and ActivityNet.

5 Results

HellaSwag is difficult for machines despite strong baselines and massive pretraining: humans exceed 95% accuracy, while every model remains below 50%, with BERT-Large reaching 47.3%. Transfer results further show limited cross-dataset generalization, especially for SWAG-trained models on HellaSwag and WikiHow.

  • Overall performance: >95% human accuracy contrasts with <50% performance for every evaluated model, while BERT-Large achieves 47.3% overall despite serving as the adversarial filter.Human answers were combined by majority vote from five independent crowd workers solving the same four-way multiple-choice problems.
  • Overall performance: 59% ESIM+ELMo accuracy on SWAG falls to 33.3% on HellaSwag, indicating that adversarial filtering affects other models as well.The passage states that making HellaSwag adversarial for BERT also made it adversarial for every other model.
  • Domain difficulty: 45% BERT-Large performance on WikiHow remains far below 96.5% for humans, making WikiHow harder for machines than ActivityNet.OpenAI GPT outperforms BERT on WikiHow, whereas BERT performs better on ActivityNet.
  • Transfer experiments: 12% lower performance results when training on SWAG and evaluating on HellaSwag, while the reverse transfer lowers performance by 15%.The best models are trained on the same dataset on which they are evaluated.
  • Transfer experiments: 69% HellaSwag-model accuracy on the missing LSMDC domain contrasts with 28% for SWAG models on their missing WikiHow domain.The passage concludes that SWAG models do not generalize at all to WikiHow, suggesting that general commonsense reasoning was hardly necessary to solve SWAG.

6 Discussion

HellaSwag remains challenging for pretrained state-of-the-art NLI models because its Goldilocks-zone examples are nonsensical to humans yet difficult for models to distinguish. The discussion argues that future progress may require algorithmic advances, substantial compute, and continually adversarial benchmark construction, while data limitations remain fundamental.

  • HellaSwag is a challenging testbed for state-of-the-art NLI models, including those built on extensive pretraining.
  • The Goldilocks zone combines generations that are nonsensical to humans with text that existing state-of-the-art NLP models cannot reliably distinguish.
  • 109 GPU-hours of pretraining are estimated to be required to reach human-level HellaSwag performance without algorithmic or computational improvements.The estimate uses RTX 2080 Ti-equivalent computation and corresponds to over 100k GPU years.
  • Algorithmic improvements may include architectural advances and better pretraining objectives, but reporting bias can make some commonsense answers require an exponential number of samples.
  • Models that solve HellaSwag through spurious patterns could be challenged by developing a new benchmark that uses those models as adversaries.
  • When HellaSwag is solved, the authors propose crowdsourcing another dataset in the same format to identify where models fail.Using GPT as generator and BERT-Large as discriminator produced a dataset adversarial to all accessible models.

7 Conclusion

The paper introduces HellaSwag, a physically situated commonsense reasoning dataset constructed through adversarial filtering and designed to challenge robust models. It also uses the dataset to illuminate pretrained models and motivate benchmarks that co-evolve adversarially with state-of-the-art systems.

  • HellaSwag is a new dataset for physically situated commonsense reasoning.
  • Adversarial filtering, combined with state-of-the-art language generation and discrimination models, produces a dataset adversarial to robust models.
  • The findings provide insight into pretrained models and suggest benchmarks that adversarially co-evolve with evolving state-of-the-art models.

Supplemental Material … C BERT setup

The supplemental material details the Adversarial Filtering, GPT generation, and BERT configurations used to construct and evaluate HellaSwag. AF uses iterative dataset splits and multi-way BERT discrimination, while GPT sampling and BERT input formats define example generation and scoring.

  • Supplemental Material: The supplemental material provides additional details about the Adversarial Filtering experiments.
  • A Adversarial Filtering Setup: 80% of each iteration’s dataset is used for training and 20% for testing, without special handling of video or article IDs.
  • A Adversarial Filtering Setup: ActivityNet uses k = 9 assigned indices per example, whereas WikiHow uses k = 5 because longer sequences yield fewer good generated endings.
  • A Adversarial Filtering Setup: AF trains discriminators in a multi-way fashion using BERT-Large, matching the model configuration used for SWAG.
  • A Adversarial Filtering Setup: When BERT fails to converge at around 25% accuracy, the procedure skips reassignment.
  • B GPT Setup: GPT is finetuned for 2 epochs on WikiHow and 5 epochs on ActivityNet, with random language-model sampling rather than beam search.
  • B GPT Setup: WikiHow endings use Nucleus Sampling with p = 0.98, preventing generation from being biased toward common words.
  • C BERT setup: BERT retains its original architecture and pretraining, using context-plus-ending inputs or a single-span ending-only format.The contextual format places the passage between [CLS] and [SEP] markers before the candidate ending; ending-only inputs use the BERT-style single-span setting.

D A discussion on BERT Hyperparameters and Instability · E Human validation

The authors report that BERT hyperparameters that worked well on SWAG became unstable on HellaSwag, motivating modified training schedules and randomized adversarial-filtering settings. Human validation used six-answer choices, iterative replacement of confusing answers, and worker-quality controls, with multiple rounds especially helpful on ActivityNet.

  • D A discussion on BERT Hyperparameters and Instability: Hyperparameters that generally worked well on SWAG became somewhat unstable when transferred to HellaSwag.The SWAG experiments used Devlin et al. (2018)'s hyperparameters.
  • D A discussion on BERT Hyperparameters and Instability: A batch size of 64 examples and 20% warmup helped the model adapt to SWAG more gradually without diverging early.These settings replaced a batch size of 16 and 10% warmup.
  • D A discussion on BERT Hyperparameters and Instability: Adversarial-filtering experiments randomized learning rates between 1e-5 and 4e-5 using a log-uniform distribution and selected the cased model with probability 0.5.These choices were applied on WikiHow and ActivityNet, with ranges recommended by the original BERT paper.
  • D A discussion on BERT Hyperparameters and Instability: Increasing adversarial-filtering training beyond 3 epochs helped final HellaSwag fine-tuning, with 10 epochs as the best configuration.The passage contrasts the initial 3-epoch setting with the best 10-epoch configuration.
  • D A discussion on BERT Hyperparameters and Instability: HellaSwag fine-tuning used a learning rate of 2e-5.This was the reported fine-tuning learning rate.
  • E Human validation: Human validation presented six answers per example, with exactly one true ending and five adversarial-filtering endings.The setup matched Zellers et al. (2018).
  • E Human validation: Multiple validation rounds were especially helpful on ActivityNet, while replacing a confusing answer before the next worker improved the validation procedure.Workers were paid fairly, screened with an autograded qualification HIT, and dequalified when they preferred generated endings over real ones.

F Human Evaluation · G More examples

Human evaluation matched the model task, using majority votes from five Turkers across WikiHow and ActivityNet examples. The paper also reports additional validation examples in Figure 2.

  • F Human Evaluation: Five Turkers jointly judged each item by selecting the best of five endings.Human evaluation used the exact same task given to models, with majority voting across the five workers.
  • F Human Evaluation: Human difficulty differed by domain, with WikiHow easier than ActivityNet.The paper attributes the largest differences in human difficulty to the dataset domain.
  • F Human Evaluation: 200 WikiHow and 200 ActivityNet examples were evaluated for each validator setting: 0, 1, or 2 previous validators.These evaluations correspond to the conditions shown in Figure 7.
  • F Human Evaluation: Mixed-split accuracy combines WikiHow and ActivityNet accuracies weighted by their example counts.The formula uses each dataset’s human accuracy and the number of examples from that dataset in the split.
  • F Human Evaluation: The formula’s acc term denotes human-judged accuracy for each dataset.The notation distinguishes dataset-specific accuracy from the number of examples in the split.
  • G More examples: Additional validation examples are reported in Figure 2.The paper notes these examples separately from the human-evaluation procedure.

H In-Domain and Zero-Shot categories

The in-domain validation examples span category labels such as Youth and Family Life, with evaluation equally weighting categories seen during training and out-of-domain categories. The examples illustrate both in-domain and zero-shot settings through practical how-to questions and candidate continuations with model probabilities.

  • Youth (in-domain): The Youth in-domain example asks how to make a believable excuse for unfinished homework, with “Blame technology” as the demonstrated step.The source presents this Wikihow category, title, and step explicitly.
  • Family Life (zero-shot): The Family Life zero-shot example asks how to raise children to be helpers, demonstrating the step “Call them helpers when you ask for things.”The source labels the category as zero-shot and gives the title and step.
  • Family Life (zero-shot): In the Family Life example, candidate b receives 99.9%, while candidate a receives 0.1%.These percentages are reported for the displayed candidate continuations in the supplied passage.
  • Category coverage: Figure 13 groups in-domain HellaSwag validation examples by category label and equally weights seen and out-of-domain categories in evaluation.The figure caption identifies the grouped validation examples and the evaluation weighting setup.
Loading 1905.07830v1…