PaperlayerView exact revision on arXiv

Source-linked AI summary

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman

arXiv:1804.07461v3cs.CL

TL;DR

NLU models are often task-specific and struggle with out-of-domain data, motivating more general evaluation. GLUE benchmarks diverse tasks and diagnostics, finding that joint training slightly improves aggregate performance but current models still struggle with deeper linguistic structure.

  • Problem

    Most NLU models are task-specific and struggle with out-of-domain data, motivating unified models that handle diverse linguistic tasks and domains.

  • Method

    GLUE combines nine diverse NLU tasks with a model-agnostic evaluation platform and hand-crafted diagnostic examples for linguistic analysis.

  • Results

    Jointly trained models slightly outperform separately trained models in aggregate, but baselines handle strong lexical signals better than deeper logical structure.

  • Takeaways & Limitations

    The design of general-purpose NLU models remains an unanswered question, with current systems leaving substantial room for improvement.

  • Takeaways & Limitations

    The diagnostic set is intended for model comparison and error analysis, not as a benchmark of overall performance or downstream generalization.

Abstract

from arXiv · show

For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems.

1 INTRODUCTION

GLUE is introduced as a model-agnostic benchmark and online platform for evaluating and analyzing models across nine diverse NLU tasks. It also provides a hand-crafted diagnostic dataset, while initial multi-task models slightly outperform separate-task training but still achieve a fairly low absolute score.

  • Motivation: Most above-word-level NLU models remain task-specific and struggle with out-of-domain data, motivating unified models that can execute diverse linguistic tasks across domains.The introduction frames general, flexible, and robust language understanding as the target.
  • GLUE benchmark: The model-agnostic platform evaluates, compares, and analyzes any method that processes single-sentence and sentence-pair inputs and predicts on all nine tasks.Evaluation relies primarily on privately held test data and includes a leaderboard.
  • Diagnostic evaluation: GLUE includes hand-crafted analysis examples that probe learned knowledge and highlight challenges involving world knowledge and logical operators.The examples are intended to encourage linguistically meaningful solution strategies.
  • Initial experiments: Unified multi-task models slightly outperform comparable models trained separately on each task, but the best multi-task model still achieves a fairly low absolute score.The best multi-task model uses ELMo, a recently proposed pre-training technique.
  • GLUE benchmark: GLUE comprises nine sentence or sentence-pair NLU tasks spanning diverse text genres, dataset sizes, and difficulty levels.The tasks are built on established annotated datasets.

2 RELATED WORK

Prior work has pursued general NLU through multi-task learning, sentence-to-vector encoders, and supervision from unlabeled and labeled data. GLUE builds on these directions while differing from SentEval and decaNLP through a diverse, difficult benchmark and accompanying analysis tools.

  • Multi-task learning: Multi-task models have jointly learned several core NLP tasks, while later work studied hierarchical supervision and automatically learned cross-task sharing mechanisms.Examples include joint learning for POS tagging, chunking, named entity recognition, and semantic role labeling.
  • General NLU evaluation: General NLU research has developed sentence-to-vector encoders using unlabeled data, labeled data, or combinations of both.SentEval codified a standard evaluation practice for this line of work.
  • GLUE versus SentEval: GLUE differs from SentEval by selecting tasks intended to make the benchmark both diverse and difficult.The paper characterizes many SentEval tasks as closely related to sentiment analysis and others as relatively uninformative because they are nearly solved.
  • GLUE versus decaNLP: decaNLP also evaluates systems across multiple datasets, recasting ten tasks as question answering through automatic transformations.GLUE distinguishes itself by providing a leaderboard and error analysis toolkit, while viewing decaNLP as more ambitious but less immediately practical.

3 TASKS

GLUE comprises nine English sentence-understanding tasks spanning varied domains, data quantities, and difficulties, designed to encourage substantial knowledge sharing across tasks alongside task-specific components. The benchmark evaluates submissions per task and aggregates their metrics into a macro-average leaderboard score.

  • Benchmark design: GLUE covers nine English sentence-understanding tasks across diverse domains, data quantities, and difficulty levels.The benchmark is designed so strong performance requires sharing substantial knowledge across tasks while retaining some task-specific components.
  • Task descriptions: CoLA evaluates whether sentences are grammatically acceptable using Matthews correlation coefficient for unbalanced binary classification.The corpus contains acceptability judgments from books and journal articles on linguistic theory; the metric ranges from -1 to 1, with 0 representing uninformed performance.
  • Task descriptions: SST-2 predicts positive or negative sentiment from sentence-level labels assigned to movie-review sentences.The task uses the two-way class split and excludes phrase-level labels.
  • Task descriptions: MRPC and QQP determine whether sentence pairs are semantically equivalent, reporting both accuracy and F1 because their class distributions are imbalanced.MRPC contains automatically extracted online-news sentence pairs and is 68% positive, while QQP contains Quora question pairs and is 63% negative; QQP uses a standard test set with private labels.
  • Task descriptions: STS-B predicts human similarity scores from 1 to 5 for sentence pairs and evaluates them with Pearson and Spearman correlation coefficients.The pairs come from news headlines, video and image captions, and natural language inference data.
  • Task descriptions: MNLI, QNLI, RTE, and WNLI formulate entailment, question-answering, textual-entailment, and pronoun-reference problems as sentence-pair classification tasks.MNLI uses entailment, contradiction, and neutral labels across ten sources; QNLI pairs questions with context sentences; RTE uses a two-class entailment split; WNLI converts Winograd pronoun resolution into sentence pairs and has a 65% not-entailment test distribution.
  • Evaluation: The benchmark reports per-task scores and a macro-average, using an unweighted average when a task has multiple metrics.Systems run on provided test data and upload predictions to gluebenchmark.com for scoring and leaderboard placement.

4 DIAGNOSTIC DATASET

The diagnostic dataset is a manually curated NLI test set designed to analyze predefined linguistic phenomena through diverse, naturally grounded examples. It supports model comparison and error analysis rather than measuring overall performance or downstream generalization.

  • Dataset Design: The diagnostic dataset is a small, manually curated test set highlighting predefined linguistic phenomena that models should capture.Its design draws inspiration from the FraCaS suite and the Build-It-Break-It competition.
  • Dataset Design: Each example is an NLI sentence pair tagged with demonstrated phenomena, spanning skills from syntactic ambiguity resolution to pragmatic reasoning with world knowledge.Examples are based on naturally occurring sentences from several domains to promote diversity.
  • Annotation Process: Examples use minimal edits and high lexical and structural overlap to limit superficial cues while expressing target inference phenomena.The annotation process begins from a target phenomenon set based roughly on FraCaS.
  • Evaluation: R3, a three-class generalization of the Matthews correlation coefficient, evaluates the diagnostically imbalanced class distribution.The metric is selected because the diagnostic set’s class distribution is not balanced.
  • Human Baseline: 0.80 average R3 was achieved by six NLP researchers annotating 50 randomly sampled sentence pairs, with Fleiss’s κ of 0.73.The human R3 score was much higher than any baseline system described in Section 5.
  • Intended Use: The diagnostic set is intended for error analysis, qualitative model comparison, and development, with performance compared across models but not categories.Because examples are hand-picked and NLI has no natural input distribution, scores should not represent overall performance or downstream generalization.

5 BASELINES

The paper evaluates multi-task and single-task GLUE baselines built around shared BiLSTM encoders, attention-based sentence-pair modeling, and pretrained representations. It also benchmarks several trained sentence-to-vector encoders using task-specific classifiers.

  • Baselines: The baseline suite includes a multi-task GLUE model, several pretraining variants, and separately trained single-task models.The models are implemented in AllenNLP, and the single-task models share no parameters with other tasks.
  • Architecture: The simplest architecture uses 300D GloVe embeddings and a two-layer 1500D-per-direction BiLSTM with max pooling.Single-sentence tasks classify the encoded sentence vector, while sentence-pair tasks process both sentence vectors.
  • Architecture: An attention variant explicitly models interactions between all word pairs before applying a second BiLSTM with max pooling.This sentence-pair architecture falls outside the sentence-to-vector paradigm.
  • Pre-Training: The base model is augmented with ELMo or CoVe pretrained representations, using ELMo instead of other embeddings and concatenating CoVe with GloVe.ELMo is derived from two-layer language models, whereas CoVe comes from a two-layer BiLSTM trained for English-to-German translation.
  • Sentence Representation Models: The benchmark also evaluates CBoW, Skip-Thought, InferSent, DisSent, and GenSen by training only task-specific classifiers on their sentence representations.These are trained sentence-to-vector encoder models evaluated through the GLUE benchmark.

6 BENCHMARK RESULTS

Benchmark results show that multi-task training, attention, and ELMo can improve aggregate performance, while pre-trained sentence representations progress from CBoW through GenSen. Performance varies substantially by task, with notable failures on CoLA and WNLI and strong sentence-representation performance on STS-B.

  • Evaluation setup: Models were evaluated using the run with the best macro-average development-set performance, with single-task and sentence-representation models selected separately per task.Three runs were trained for each model.
  • Training and representations: Multi-task training improves overall scores for attention- or ELMo-based models, while attention is negligible or negative in single-task training but helpful in multi-task training.ELMo consistently improves over GloVe or CoVe, particularly on single-sentence tasks; CoVe has mixed effects over GloVe.
  • Training and representations: Pre-trained sentence representations show consistent gains from CBoW to Skip-Thought to InferSent and GenSen, with GenSen outperforming all but the two best directly trained models.InferSent is competitive with models trained directly on GLUE tasks.
  • Per-task results: 65.1% is the most-frequent-class guessing rate on WNLI, and no model exceeds it; sentence representations underperform on CoLA, while direct models lag on STS-B.These results indicate substantial task-to-task variation in model performance.

7 ANALYSIS

The diagnostic analysis finds poor overall linguistic performance, with single-task models usually outperforming multi-task models. Fine-grained results reveal reliance on lexical cues, model-specific weaknesses, and mixed effects from attention and transfer methods.

  • Analysis: The models’ MNLI classifiers were evaluated on the diagnostic set to assess their linguistic capabilities.Results are presented in Table 5.
  • Coarse Categories: The highest total score is 28, indicating poor absolute performance; scores tend to be higher for Predicate-Argument Structure and lower for Logic.Numbers across coarse categories are not closely comparable.
  • Coarse Categories: Single-task models almost always outperform multi-task models on the diagnostic analysis, unlike on the main benchmark.The authors attribute this partly to their simple multi-task training regime and likely destructive interference.
  • Fine-Grained Subcategories: Most models perform relatively well on universal quantification and morphological negation, often using lexical cues such as “all” as useful signals.Relevant examples suggest that lexical cues can suffice for these phenomena.
  • Fine-Grained Subcategories: Double negation is especially difficult for GloVe-only GLUE-trained models, while ELMo ameliorates this weakness and CoVe helps to some degree.Attention has mixed effects, and attention models tend to struggle with downward monotonicity.
  • Analysis: Attention may aid out-of-domain performance, while ELMo and CoVe encode supervision-specific linguistic information; greater capacity can also produce overfitting.Attention models’ failures in downward-monotone contexts exemplify this risk, alongside sensitivity to hypernym/hyponym substitution and word deletion as entailment signals.

8 CONCLUSION

The paper introduces GLUE as a platform for evaluating and analyzing natural language understanding systems. It finds benefits from joint training, attention mechanisms, and transfer learning, while showing that these methods still leave room for improvement.

  • GLUE is a platform and resource collection for evaluating and analyzing natural language understanding systems.
  • Jointly trained models achieve better aggregate performance than models trained separately for each task.
  • Attention mechanisms and transfer learning methods such as ELMo improve NLU systems and together outperform the best sentence representation models on GLUE.

A ADDITIONAL BENCHMARK DETAILS · B ADDITIONAL BASELINE DETAILS · B.1 ATTENTION MECHANISM

The additional details describe how QNLI is balanced and converted into NLI format, and specify the attention mechanism used to compare two sequences before classification.

  • A ADDITIONAL BENCHMARK DETAILS: QNLI balances examples where the most similar sentence is not the answer with cases where the answer is most similar but a distractor is a close second.The dataset includes equal amounts of both cases.
  • A ADDITIONAL BENCHMARK DETAILS: QNLI sentence similarity is computed from CBoW representations using pre-trained GloVe embeddings.The conversion of pre-existing datasets into NLI format is closely related to work by White et al.
  • B.1 ATTENTION MECHANISM: The attention mechanism computes H_ij as the dot product u_i · v_j between hidden states from two sequences.The sequences contain hidden states u_1 through u_M and v_1 through v_N.
  • B.1 ATTENTION MECHANISM: For each u_i, attention weights α_i are obtained by applying a softmax over the corresponding row of H.The resulting weights form a context vector by weighting the v_j states.
  • B.1 ATTENTION MECHANISM: The context vectors are formed as attention-weighted sums of the v_j hidden states.Each context vector is paired with its corresponding u_i state.
  • B.1 ATTENTION MECHANISM: A second BiLSTM with max pooling processes the concatenated pairs [u_i; ˜v_i] to produce u′, with an analogous process producing v′.The mechanism processes both sequence directions symmetrically.
  • B.1 ATTENTION MECHANISM: The classifier receives [u′; v′; |u′ − v′|; u′ ∗ v′] as its input representation.This combines the two pooled representations, their absolute difference, and their elementwise product.

B.2 TRAINING · B.3 SENTENCE REPRESENTATION MODELS · C DEVELOPMENT SET RESULTS

The GLUE baselines use shared BiLSTM encoders with task-specific classifiers, compare several sentence representation models, and report best development-set results as a reference while limiting submissions to reduce private-test overfitting.

  • B.2 TRAINING: Models share a BiLSTM sentence encoder and post-attention BiLSTMs across tasks, while classifiers are trained separately for each task.Training samples tasks with probability proportional to training-set size and scales each task’s loss inversely to its example count.
  • B.2 TRAINING: Each training update samples a task with probability proportional to that task’s number of training examples.The training procedure also scales each task’s loss inversely proportional to its number of examples.
  • B.3 SENTENCE REPRESENTATION MODELS: The evaluated sentence representations include CBoW, Skip-Thought, InferSent, and DisSent.CBoW averages GloVe token embeddings; Skip-Thought generates neighboring sentences; InferSent and DisSent use BiLSTMs with max-pooling.
  • B.3 SENTENCE REPRESENTATION MODELS: Skip-Thought uses the original pre-trained model trained on sentence sequences from the Toronto Book Corpus.InferSent is trained on MNLI and SNLI, as specified in the representation-model descriptions.
  • B.3 SENTENCE REPRESENTATION MODELS: Task-specific classifiers are trained on top of frozen sentence encoders using SentEval’s default parameters.The passage directs readers to SentEval for implementation details and code.
  • C DEVELOPMENT SET RESULTS: The GLUE website limits users to two submissions per day to reduce overfitting to private test data.The authors provide their baselines’ best development-set results in Table 6 as a reference for future GLUE work.

D BENCHMARK WEBSITE DETAILS

GLUE’s online platform uses a modern web and cloud stack to store submissions and run grading. Its leaderboard presents baseline results and expandable submission details, including descriptions and parameter counts.

  • Platform implementation: The platform is built with React, Redux, and TypeScript, using Google Firebase for storage and Google Cloud Functions to run grading scripts on submission.The grading script is hosted and executed through Google Cloud Functions.
  • Leaderboard: The leaderboard visually presents baseline results and provides expanded submission details, including a prose description and parameter count.These features are shown in the benchmark website leaderboard.

E ADDITIONAL DIAGNOSTIC DATA DETAILS … E.4 KNOWLEDGE

The diagnostic suite is organized into four broad phenomena categories and finer-grained subcategories to analyze diverse levels of natural language understanding. It is intended as an analysis tool for error analysis, qualitative comparison, and adversarial-example development rather than as a conventional benchmark.

  • E ADDITIONAL DIAGNOSTIC DATA DETAILS: The diagnostic dataset spans Lexical Semantics, Predicate-Argument Structure, Logic, and Knowledge, each divided into finer-grained phenomena.These categories range from word meaning and sentence structure to reasoning and world knowledge.
  • E ADDITIONAL DIAGNOSTIC DATA DETAILS: The dataset supports error analysis, qualitative model comparison, and adversarial-example development, but performance should not be compared across categories.It is designed to illustrate phenomena models may or may not capture, not to serve as a benchmark.
  • E.1 LEXICAL SEMANTICS: Lexical Semantics covers word-level entailment and contradiction, morphological negation, factivity, symmetry, redundancy, named entities, and quantifier meaning.Examples include “dog” entailing “animal,” “affordable” contrasting with “unaffordable,” and “SNL” expanding to “Saturday Night Live.”
  • E.2 PREDICATE-ARGUMENT STRUCTURE: Predicate-Argument Structure addresses how sentence parts compose through syntactic ambiguity, argument alternations, ellipsis, coreference, modifier interpretation, and restrictivity.The categories include prepositional-phrase attachment, active/passive and dative alternations, omitted arguments, anaphora, intersectivity, and restrictive versus non-restrictive modifiers.
  • E.3 LOGIC: Logic covers propositional operations, quantification, monotonicity, numerical and temporal reasoning, and entailments derived from these structures.The suite distinguishes upward, downward, and non-monotone restrictor positions for “a,” “no,” and “exactly one.”
  • E.4 KNOWLEDGE: Knowledge examples test entailments requiring extra information beyond sentence disambiguation, including concrete world knowledge and common sense.The Knowledge categories focus on facts and broader geographical, legal, political, technical, cultural, physical, social, and lexical understanding.
  • E.4 KNOWLEDGE: World Knowledge includes factual and broader domain knowledge, such as inferring Japan from Mt. Fuji or heat from an exothermic reaction.The examples also distinguish supported geographic inferences from unsupported alternatives, such as Nepal.
  • E.4 KNOWLEDGE: Common Sense covers broadly expected physical, social, and lexical knowledge, such as interpreting “seeing his barber” as getting a haircut.It also includes inferences about social expectations and functional relationships, such as shock waves implying lack of prior preparation.
Loading 1804.07461v3…