PaperlayerView exact revision on arXiv

Source-linked AI summary

GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective

Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, Yue Zhang

arXiv:2211.08073v4cs.CLcs.AIcs.LGcs.PF

TL;DR

NLP OOD generalization remains challenging, while existing evaluations often cover too few tasks for unified assessment. GLUE-X benchmarks cross-domain generalization across standard classification tasks and evaluates PLMs and training strategies. The results show that PLMs remain below human performance on OOD evaluations.

  • Problem

    NLP OOD generalization remains a challenge, and existing evaluations covering only one or a few tasks leave a gap in unified assessment.

    Source: §1 · p. 1

  • Method

    GLUE-X evaluates single-source OOD domain generalization across eight standard tasks using multiple-domain test data and compares PLMs and training strategies.

    Source: §1 · p. 1, §5 · p. 9

  • Results

    PLMs’ average OOD accuracy remains below human performance: 80.14% for humans versus 74.62% for models.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    The ID–OOD performance correlation usually holds in most cases, but its coefficient is primarily related to the selection of OOD datasets.

    Source: §5 · p. 9

    GLUE-X focuses on text classification and does not include language-generation tasks such as machine translation, summarization, and dialogue; extending it to more real-world domain datasets is important.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Pre-trained language models (PLMs) are known to improve the generalization performance of natural language understanding models by leveraging large amounts of data during the pre-training phase. However, the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks, limiting the real-world deployment of these methods. This paper presents the first attempt at creating a unified benchmark named GLUE-X for evaluating OOD robustness in NLP models, highlighting the importance of OOD robustness and providing insights on how to measure the robustness of a model and how to improve it. The benchmark includes 13 publicly available datasets for OOD testing, and evaluations are conducted on 8 classic NLP tasks over 21 popularly used PLMs, including GPT-3 and GPT-3.5. Our findings confirm the need for improved OOD accuracy in NLP tasks, as significant performance degradation was observed in all settings compared to in-distribution (ID) accuracy.

1 Introduction

GLUE-X addresses the lack of unified, multi-task evaluation of NLP models under distribution shift by benchmarking cross-domain generalization across standard text-classification tasks. Its evaluations find that PLMs still trail humans on cross-domain accuracy, while results vary by task and model characteristics.

1 Introduction

  • The benchmark responds to a gap in unified NLP OOD evaluation: existing evaluations and methods cover only one or a few tasks, which may leave model limitations inadequately captured.The authors describe their work as, to their knowledge, the first systematic cross-distribution evaluation on genuine data compared with human performance.Source: §1 · pp. 1–2
  • GLUE-X evaluates single-source domain generalization across eight GLUE tasks using test data from multiple domains, averaging two test domains per task.Models trained on a source domain are evaluated on target domains without labeled or unlabeled target-domain data.Source: §1 · p. 1
  • The study evaluates 21 PLMs under unified conditions, compares three tuning strategies, and analyzes model–human rationale overlap for feature-level insight into OOD robustness.The tuning strategies are linear probing, fine-tuning, and linear probing followed by fine-tuning.Source: §1 · pp. 1–2
  • 81.3% human versus 74.6% model average accuracy on cross-domain evaluations, and no included GLUE-X backbone surpasses humans under the same setting.The analysis also reports that the average human ID-to-OOD performance decay is lower than that of even the best-performing model.Source: §1 · p. 2
  • Across tasks, no single backbone consistently outperforms others; architecture influence is more significant than model parameters, and LP-FT slightly improves OOD performance over standard fine-tuning.The introduction also reports a linear ID–OOD performance correlation in most text-classification cases.Source: §1 · p. 2

2 Related Work

Prior NLP OOD studies and benchmarks often focus on a few tasks or challenge-set robustness rather than standardized, multi-task cross-distribution generalization. GLUE-X is positioned as a unified evaluation in a single-source domain-generalization setting.

2 Related Work

  • OOD evaluation has received less attention in multi-task NLP than in areas such as computer vision, OCR, time series, and AGI, where OOD benchmarks are available.The related-work discussion identifies benchmark development across those areas while noting relatively little multi-task NLP OOD evaluation.Source: §2 · p. 2
  • Adversarial NLI, Dynabench, Contrastive Set, and AdvGLUE create difficult examples, but GLUE-X instead tests cross-distribution generalization using off-the-shelf and self-collected datasets.The cited challenge datasets use iterative, adversarial, and human-and-model-in-the-loop procedures [Nie et al., 2020; Kiela et al., 2021; Gardner et al., 2020; Wang et al., 2021a].Source: §2 · p. 2
  • Unlike benchmarks such as AdvGLUE, GLUE-X evaluates OOD performance in a multi-task setting where test data arise from one or more different distributions.The paper describes GLUE-X as focusing on single-source domain generalization, aligning with the GLUE leaderboard setting.Source: §2 · p. 3
  • Existing NLP OOD evaluations and methods often cover only one or a few tasks, while standardized multi-task generalization testing remains lacking.The paper argues that narrow evaluations may inadequately capture model limitations and can result in inflated test accuracy.Source: §2 · p. 3

3 Data and Settings

GLUE-X extends GLUE's training setup with varied out-of-distribution tests to evaluate domain generalization. The section describes its task coverage, dataset curation, metrics, feature analysis, and model-training comparisons.

3.1 Overview of GLUE-X

  • GLUE-X covers eight GLUE tasks, using the same in-domain training data and evaluation metrics while adding tests from multiple domains.The tasks span sentiment analysis, natural language inference, textual entailment, paraphrase, textual similarity, and linguistic acceptability.Source: §3.1 · p. 3

3.2 Dataset Curation

  • Fifteen label-compatible OOD datasets include public and newly collected tests; WNLI is excluded because there is insufficient in-domain data for OOD test construction.NewsQA instances are reformatted for QNLI, SICK similarity labels are rounded to five classes, and MRPC and QQP serve as each other's OOD datasets.Source: §3.2 · pp. 3–4

3.3 Metrics

  • The benchmark averages task metrics for overall OOD performance and reports both OOD-to-ID decay rankings and Friedman ranks across tasks.The Friedman rank uses each model's performance rank on each task; the example specifies n = 8 tasks.Source: §3.3 · p. 4

3.4 Post-hoc Analysis

  • Post-hoc analysis on sentiment analysis and natural language inference measures overlap between model and human rationales as a partial representation of model trust.The analysis masks a phrase and surrounding context to assess changes in the ground-truth-class prediction logits; for sentiment, removing a non-causal word such as “movie” should not change a robust model's prediction.Source: §3.4 · p. 4

3.5 Models and Training Strategies

  • The evaluation includes discriminative, generative, and cost-efficient PLMs, including GPT-3 and GPT-3.5, and compares standard fine-tuning, linear probing, and linear probing followed by fine-tuning.The paper reports training costs and inference speed estimates using a single V100, and estimates more than 10,000 GPU hours for its experiments.Source: §3.5 · pp. 4–5, Table 2 · p. 4

4 Experiments

The experiments use GLUE-X to examine OOD performance, compare it with human performance, and analyze factors associated with robustness. Across the benchmark, models show substantial ID-to-OOD decay, with differences by task, architecture, and tuning strategy.

4.1 Human Annotation

  • Human annotators predicted labels on 1,000 test samples per OOD dataset after receiving ID-based instructions and examples; average inter-annotator agreement across 15 datasets was 0.857.For rationale marking, two labelers annotated 6,000 sampled instances each for IMDB, Yelp, and Amazon; F1-based agreement was 0.874, 0.871, and 0.840, respectively.Source: §4.1 · p. 5

4.2 Prediction Results

  • 20.05% average performance decay separates ID from OOD tests across the evaluated PLMs; ELECTRA-large leads both, at 89.18% ID and 74.62% OOD.Table 3 reports overall performance and robustness rankings; the paper also finds no significant OOD-robustness difference between generative and discriminative models for text classification.Source: §4.2 · p. 5, Table 3 · p. 6
  • ALBERT-base (11M parameters) achieved 65.30% OOD performance, above DistilBERT-base’s 61.94% and several moderate-sized models.The paper identifies lightweight and compressed models as especially challenged on GLUE-X.Source: §4.2 · p. 6

4.3 Discussion

  • 80.14% human OOD performance exceeded the best model’s 74.62%, while human performance decay was 7.82% versus 16.33% for that model.The human baseline outperformed state-of-the-art results on five of eight tasks under the controlled unseen-sample evaluation.Source: §4.3 · p. 6
  • No model consistently leads across tasks; CoLA is especially challenging, while the authors report that greater distribution shift exacerbates OOD performance decay.ELECTRA-large leads on four of eight tasks, with 94.67% accuracy on SST-2 but 37.85% Matthew’s Corr on CoLA.Source: §4.3 · p. 6
  • Model architecture may influence OOD performance more than parameter size: T5 decreased ratios are 16.98% (large), 18.47% (base), and 18.57% (small).The authors suggest architecture design and training methods as directions for improving OOD robustness.Source: §4.3 · p. 7
  • ID and OOD performance are higher and lower, respectively, across the examined tasks, but their correlation varies by task, including an inverse correlation on MRPC.Discriminative models show stronger linear correlation than generative models; SST-2 and MNLI have smaller ID–OOD gaps than CoLA.Source: §4.3 · p. 8, Figure 1 · p. 8
  • On MNLI with RoBERTa-base, linear probing has relatively low ID and OOD accuracy, while LP-FT slightly improves OOD robustness over standard fine-tuning.The authors note that freezing pretrained features may hinder generalization on complex NLP tasks.Source: §4.3 · p. 8, Figure 2 · p. 8

5 Conclusion

GLUE-X evaluates 21 pre-trained models across eight classification tasks under consistent multi-domain conditions. The authors report persistent human-level OOD gaps, dataset-dependent ID/OOD relationships, and benefits from stronger architectures.

5 Conclusion

  • GLUE-X evaluates 21 pre-trained models on eight classification tasks using three tuning strategies and post-hoc analyses of OOD robustness.The benchmark aims to support fair, consistent evaluation across datasets and domains.Source: §5 · p. 9
  • Current PLMs still lag far behind human-level OOD robustness, while stronger architectures bring performance benefits, especially on OOD performance.The authors also report that ID and OOD performance usually correlate linearly, with the relationship depending primarily on the selected OOD datasets.Source: §5 · p. 9
  • ID and OOD performance usually hold a linear correlation in most cases, with the correlation coefficient primarily related to the selected OOD datasets.Source: §5 · p. 9
  • Stronger architectures bring decent performance benefits, especially for OOD performance.Source: §5 · p. 9

Limitation

GLUE-X focuses on OOD robustness in text classification and identifies broader task and domain coverage as future work.

Limitation

  • The benchmark's primary focus is OOD robustness for text classification tasks.Source: Limitation · p. 9
  • GLUE-X does not currently include language-generation tasks such as machine translation, summarization, or dialogue.Source: Limitation · p. 9
  • The authors identify adding more real-world datasets from different domains as important and aim to maintain GLUE-X continuously.Source: Limitation · p. 9

Ethics Statement

The paper reports using public datasets, no private data, paid annotators, and an open-source release under a specified license.

Ethics Statement

  • Publicly available datasets were used to establish the GLUE-X leaderboard, and no private data was used.Source: Ethics Statement · p. 9
  • Crowdsourcing annotators received labor fees corresponding to the number of instances they annotated.Source: Ethics Statement · p. 9
  • The code and data are open-sourced under the CC-BY-NC-SA license.Source: Ethics Statement · p. 9

A Data Collection

The CoLA-OOD Grammar Test is derived from public high-school English exams, recast as CoLA-style examples with grammar-acceptability labels.

A Data Collection

  • CoLA-OOD is derived from the Public High School English Exam, which contains 304,277 examples.Source: A Data Collection · p. 14
  • The exam's multiple-choice fill-in questions are converted into CoLA-style examples, with correct answers labeled positive and incorrect answers negative.The golden answer is supplied by an English teacher who is a native speaker or holds an English Teaching degree.Source: A Data Collection · p. 14
  • Each Grammar Test input is a text span containing a question-answer pair or a few sentences, and the output label marks whether its grammar is acceptable.Source: A Data Collection · p. 14

B Training Details

This section describes GLUE-X training and cross-domain evaluation settings, measures dataset shifts through word overlap and MMD, and evaluates rationale overlap between models and humans.

B Training Details

  • A single dataset forms the source domain, while target domains may combine datasets from multiple domains; grid-searched checkpoints selected on ID data are tested on corresponding OOD datasets.The training procedure keeps the best-performing checkpoint on ID datasets.Source: B Training Details · p. 14

D Rationale Overlap

  • 34.98% F1: base ELECTRA leads e-SNLI rationale overlap, which is higher than on sentiment datasets; shorter e-SNLI instances may explain the gap.Across models, e-SNLI scores vary from 30.93 to 34.98. Precision measures predicted rationale words also marked by humans, while recall measures human-marked words also predicted.Source: Table 6 · p. 14, D Rationale Overlap · p. 15

C.1 Word Overlap

C Domain Distributions

  • The section uses MMD for semantic distribution shift and Word Overlap Rate for superficial similarity between datasets.Source: C Domain Distributions · p. 14

C.2 MMD Distance

  • MMD measures semantic distribution shift; equal-sized sentence samples are encoded by PLMs and sampled repeatedly to estimate an average score, with lower scores indicating higher dataset correlation.Source: §C.2 · p. 14, Figure 5 · p. 15

E The In-domain Evaluation Results

The section reports in-domain performance across seven tasks and examines how ID and OOD performance relate overall and by task.

E The In-domain Evaluation Results

F The Correlation between ID and OOD Performance

  • ID and OOD performance are linearly correlated overall for discriminative and generative models, but the relationship varies by task: it is weak for MRPC and QQP and significant for STSB and QNLI.Source: F The Correlation between ID and OOD… · p. 16

G The ID and OOD Performance of GPT-3 and GPT-3.5

The section evaluates GPT-3 and GPT-3.5 on GLUE-X using in-context instructions and reports substantial OOD performance decay. The authors caution that the results may be affected by training-data overlap and are not directly comparable with fine-tuned PLMs.

G The ID and OOD Performance of GPT-3 and GPT-3.5

  • GPT-3 is evaluated with in-domain instances as instructions before OOD testing, using in-context learning and 1,000 sampled ID instances for its in-domain result.Source: G The ID and OOD Performance of GPT-3… · p. 16
  • 11.49% for GPT-3.5 and 11.51% for GPT-3 are the reported decay ratios, versus 7.82% for humans; GPT-3.5 OOD performance is reported below ELECTRA-large and humans.The passage reports 66.90% for GPT-3 versus 80.14% for humans, and 69.68% for ELECTRA-large.Source: G The ID and OOD Performance of GPT-3… · p. 16
  • The GPT-3 and GPT-3.5 results are for reference because GLUE-X data may have appeared in their training corpora, and their instruction-based evaluation is not comparable with PLM fine-tuning.Source: G The ID and OOD Performance of GPT-3… · p. 16
Loading 2211.08073v4…