Source-linked AI summary
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman
TL;DR
GLUE’s rapid progress has reduced its usefulness for measuring further advances in general-purpose language understanding. The paper introduces SuperGLUE, a harder benchmark with diverse tasks, evaluation infrastructure, and analysis tools; BERT-based baselines still lag humans by nearly 20 points.
Problem
GLUE’s state-of-the-art performance surpassed the human-performance estimate, leaving the original benchmark unsuitable for quantifying further progress.
Method
The paper introduces SuperGLUE, an eight-task benchmark that retains GLUE’s leaderboard and single-number metric while selecting more challenging and diverse tasks.
Results
BERT-based baselines still lag behind humans by nearly 20 points on SuperGLUE.
Takeaways & Limitations
SuperGLUE offers a challenging testbed for developing general-purpose language-understanding methods.
Takeaways & Limitations
GLUE diagnostics show that pretrained models remain weak on some semantic phenomena, including restrictivity, disjunction, and downward monotonicity.
Abstract
from arXiv · showhide
In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.
1 Introduction
SuperGLUE responds to GLUE’s diminishing headroom by introducing a more rigorous benchmark that retains GLUE’s basic structure while adding harder, more diverse tasks and supporting tools.
- GLUE’s state-of-the-art score reached 88.4, exceeding the 87.1 human-performance estimate by 1.3 points and surpassing humans on four tasks.
- SuperGLUE introduces a more rigorous test of English language understanding while preserving GLUE’s goal of measuring general-purpose progress.
- SuperGLUE uses eight language-understanding tasks, a public leaderboard, a single-number metric, existing data, and an analysis toolkit.
- The benchmark selects difficult tasks and expands beyond sentence and sentence-pair classification to include formats such as coreference resolution and question answering.
- Human-performance estimates are included for all tasks, and they show substantial headroom over a strong BERT-based baseline.
- SuperGLUE provides a modular toolkit, revised leaderboard rules, and publicly available benchmark resources.
2 Related Work
Prior work established pretrained transfer learning and exposed both rapid GLUE progress and persistent semantic weaknesses in current models.
- Pretraining and transfer methods produced representations that transfer across many NLP tasks, motivating GLUE as a common evaluation framework.
- GPT and BERT substantially improved GLUE scores, while recent models surpassed estimates of non-expert human performance.
- GLUE’s diagnostic entailment performance remained far below humans, with models near or below chance on some linguistic phenomena.
- Existing pretrained models still struggle to extract semantic details requiring appropriate supervision.
WiC
Mark told Pete many lies about himself, and Pete included those lies in his book. The passage concludes that Mark should have been more truthful.
- Mark told Pete many lies about himself.
- Pete included Mark’s lies in his book.
- The passage states that Mark should have been more truthful.
3 SuperGLUE Overview
SuperGLUE selects eight challenging English language-understanding tasks using criteria emphasizing substance, difficulty, evaluability, public data, simple formats, and licensing. It broadens GLUE’s task coverage with longer inputs, question answering, coreference, diagnostics, and bias analysis.
- Task criteria: The benchmark requires tasks to test English text understanding, exceed current systems’ abilities, remain solvable by college-educated speakers, and support automatic evaluation.It excludes tasks requiring domain-specific knowledge and notes that some generation tasks lack reliable automatic metrics.
- Task formats: SuperGLUE expands beyond GLUE’s sentence and sentence-pair formats to include longer inputs, coreference resolution, and question answering.The resulting tasks require understanding tokens in context, sentences, inter-sentence relations, and paragraphs.
- Task selection: SuperGLUE selects eight tasks after filtering proposals and testing remaining candidates with BERT and human baselines.Tasks that were unsuitable because of licensing, complex formats, insufficient headroom, or inappropriate difficulty were filtered out.
- Selected tasks: The task suite spans Boolean, entailment, causal, multi-answer, cloze, word-sense, and commonsense coreference problems.Examples include BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC.
- Selected tasks: WSC tests whether systems resolve pronouns among candidate noun phrases using everyday knowledge and commonsense reasoning.SuperGLUE uses a binary-classification version with training and validation data from original and affiliated sources, and fiction-derived test examples.
- Diagnostics: SuperGLUE retains a broad-coverage diagnostic and adds Winogender to analyze linguistic, world-knowledge, and gender-bias behavior.Winogender reports accuracy and gender parity, while the broad diagnostic collapses contradiction and neutral into not_entailment.
4 Using SuperGLUE
SuperGLUE provides a toolkit, open submission rules, standardized task data, and reporting requirements to support reproducible evaluation. Table 3 summarizes baseline performance across tasks and diagnostics using task-specific metrics.
- Software tools: jiant is a modular PyTorch-based toolkit that implements baselines and supports evaluating custom models and training methods on SuperGLUE.It supports pretrained models including OpenAI GPT and BERT, as well as multistage and multitask methods.
- Eligibility: Any system capable of producing predictions for the tasks may submit, without restrictions on method type, parameter sharing, or shared initialization.Submission policies limit overfitting to private test data, including a maximum submission frequency.
- Data: Each task provides standardized training, development, and unlabeled test sets through the SuperGLUE site and toolkit download script.Systems may use public or private development data subject to restrictions on the distributed task versions.
- Reporting: Leaderboard papers must name and cite the specific datasets used, including benchmark datasets, to support credit assignment.This citation requirement is enforced for leaderboard listing.
- Baseline evaluation: Table 3 reports task-specific baseline metrics, diagnostic scores, and an overall average across non-AX tasks.CB uses accuracy and macro-F1, MultiRC uses answer-option F1 and exact match, AXb uses MCC, and AXg uses accuracy and gender parity.
5 Experiments
The experiments compare simple and transfer-enhanced baselines across SuperGLUE, finding substantial gains from BERT and additional supervision while persistent human-performance gaps remain.
- Results: BERT raises the average SuperGLUE score by 25 points and produces significant gains on every benchmark task.The largest gains occur on MultiRC, ReCoRD, and RTE.
- Transfer baselines: MultiNLI transfer improves BoolQ, CB, and RTE by 2–5 points, while SWAG transfer improves COPA by 8 points.These results indicate benefits from task-related intermediate supervision.
- Baselines: The majority-class and CBOW baselines perform near chance on several tasks, whereas BERT substantially improves average performance.The CBOW baseline represents inputs by averaged 300-dimensional GloVe vectors.
- Results: BERT performs worse than simple baselines on WSC, likely because the dataset is small and lacks data augmentation.This is the main task-specific exception to BERT’s overall gains.
- Human comparison: BERT++ remains nearly 20 points below human performance on average, with the largest gap on WSC at 35 points.The smallest gaps are around 10 points on BoolQ, CB, RTE, and WiC.
6 Conclusion
SuperGLUE is introduced as a challenging benchmark for general-purpose language understanding, emphasizing diverse formats and low-data tasks. BERT-based baselines remain far below human performance, motivating further methodological progress.
- Conclusion: SuperGLUE evaluates general-purpose language understanding with eight challenging tasks emphasizing diverse formats and low-data settings.Nearly half the tasks have fewer than 1k examples, and all but one have fewer than 10k.
- Conclusion: BERT-based baselines lag human performance by nearly 20 points on average.The authors expect progress in multitask, transfer, and unsupervised or self-supervised learning to be necessary for approaching human-level performance.
A Development Set Results
The development-set results for the SuperGLUE baselines are presented in Table 4.
- A Development Set Results: Table 4 presents baseline results on the SuperGLUE development sets.
B Performance on GLUE Diagnostics
Figure 2 reports GLUE leaderboard systems’ performance on selected diagnostic categories using the R3 metric scaled by 100. It highlights categories that improved and categories that remain difficult or adversarial.
- B Performance on GLUE Diagnostics: Figure 2 reports GLUE submissions’ performance across selected diagnostic categories using R3 scaled up by 100.The categories are drawn from the GLUE diagnostic set.
- B Performance on GLUE Diagnostics: Double negation improved with GLUE advances, while restrictivity remains hard and disjunction and downward monotone categories remain adversarial.
C Human Performance Estimation
Human-performance estimates are constructed through trained crowdworker annotation, with task-specific instructions and majority voting used to produce benchmark baselines.
- $23.75/hr was the average pay rate for crowdworkers during both training and annotation across tasks.
- Workers received task instructions, an FAQ link, and up to 30 development examples during training.
- Qualification required annotators to complete at least five HITs and achieve at least the median training performance across workers.
- Human performance was estimated from 100 randomly sampled test examples per task, except WSC, which used its full test set.
- Five workers annotated each example, and majority voting produced the human-performance estimate.
- Task-specific instructions were provided during training and annotation, while additional human estimates covered several tasks later excluded from the benchmark.
C.3 Task Specific Details
Task-specific human-performance procedures adapt label formats, task reformulations, and training data to the characteristics of each evaluated dataset.
- WSC and COPA were presented as two-way classification tasks, with human performance calculated by majority vote across annotations.
- CommitmentBank annotations began with seven classes and were collapsed into three using the authors’ established bucketing ranges before majority voting.
- CommitmentBank training used randomly sampled low-agreement examples outside the benchmark because they were generally more ambiguous.
- Diagnostic-dataset workers trained on RTE development examples because RTE is the most closely related textual-entailment task.
- Ultrafine Entity Typing was reformulated as binary classification of whether a proposed tag applied to a word or phrase.
- For GAP, noun-phrase spans were supplied and workers chose span A, span B, or neither in a three-way classification task.
D Excluded Tasks
Tasks were excluded when their format encouraged task-specific design, human annotation was difficult or conservative baselines were low, or machine performance left too little headroom.
- Some medical datasets were excluded because access generally required explicit permission and credentials from data owners.
- QuAC and STREUSLE were excluded because their formats might incentivize task-specific model designs instead of general-purpose techniques.
- Tasks were filtered using BERT and human baselines, excluding datasets that were too difficult for humans or too easy for machine baselines.
- BERT scored 67.2 F1 on Quora Insincere Questions versus 66.7 for the human baseline, so the task was excluded.
- Query Well-Formedness, recast Ultrafine Entity Typing, and GAP were excluded because machine–human performance gaps were below the tolerated margin.
- Discovering Ongoing Conversations was excluded after BERT reached 51.9 F1 on a sentence-pair formulation and humans found the task challenging.