Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Ruiqi Zhong; Kristy Lee; Zheng Zhang; Dan Klein

Source-linked AI summary

Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Ruiqi Zhong, Kristy Lee, Zheng Zhang, Dan Klein

arXiv:2104.04670v5Apr 10, 2021cs.CLcs.AI

TL;DR

Zero-shot classification benefits from answering label-description prompts, but language models’ next-word prediction objective is misaligned with this task. The paper meta-tunes models on QA-formatted classification datasets and reports better zero-shot performance than general QA models, with larger models improving average AUC-ROC by 6.3%.

Problem
Next-word prediction is correlated with but misaligned with the objective of answering zero-shot classification prompts.
Method
The paper meta-tunes language models on a collection of classification datasets unified into a “Yes”/“No” QA format, evaluating on held-out groups of similar datasets.
Results
Meta-tuned models outperform general-purpose QA models in zero-shot classification, while increasing model size from 220 M to 770 M improves average AUC-ROC by 6.3%.
Takeaways & Limitations
The results suggest that out-of-the-box zero-shot measurements may underestimate language models’ potential and that larger pretrained models may improve meta-tuned zero-shot learning.
Takeaways & Limitations
The authors acknowledge that their similarity-based definition of unseen tasks may still be lenient despite being harsher than prior definitions.

Abstract

from arXiv · show

Large pre-trained language models (LMs) such as GPT-3 have acquired a surprising ability to perform zero-shot learning. For example, to classify sentiment without any training examples, we can "prompt" the LM with the review and the label description "Does the user like this movie?", and ask whether the next word is "yes" or "no". However, the next word prediction training objective is still misaligned with the target zero-shot learning objective. To address this weakness, we propose meta-tuning, which directly optimizes the zero-shot learning objective by fine-tuning pre-trained language models on a collection of datasets. We focus on classification tasks, and construct the meta-dataset by aggregating 43 existing datasets and annotating 441 label descriptions in a question-answering (QA) format. When evaluated on unseen tasks, meta-tuned models outperform a same-sized QA model and the previous SOTA zero-shot learning system based on natural language inference. Additionally, increasing parameter count from 220M to 770M improves AUC-ROC scores by 6.3%, and we forecast that even larger models would perform better. Therefore, measuring zero-shot learning performance on language models out-of-the-box might underestimate their true potential, and community-wide efforts on aggregating datasets and unifying their formats can help build models that answer prompts better.

1 Introduction

Zero-shot classification uses label descriptions without examples, but next-word prediction is misaligned with answering prompts. The paper addresses this by unifying diverse classification datasets into QA format and meta-tuning language models, while defining unseen tasks to exclude similar datasets.

Zero-shot classification predicts labels from descriptions without providing examples, often by prompting a language model to answer “Yes” or “No”.
The paper directly optimizes zero-shot classification because next-word prediction is correlated with but misaligned with the target prompt-answering objective.
The authors unify binary classification tasks into a “Yes”/“No” QA format and aggregate 43 datasets containing 204 labels with 441 manually annotated label descriptions.
Unseen-task evaluation excludes similar datasets, using a stricter boundary than merely excluding the same dataset or labels.
Meta-tuning improves zero-shot classification over UnifiedQA for most labels and outperforms the previous state of the art based on natural language inference.
The results suggest that out-of-the-box prompt-based measurements may underestimate language models’ zero-shot potential, while prompt collection raises security, privacy, and fairness concerns.

2 Data

The paper builds a diverse binary-classification meta-dataset by converting tasks into QA, annotating label questions, and grouping similar datasets to enforce demanding unseen-task evaluations. It acknowledges that dataset similarity remains subjective and that the collection includes noisy or unsuitable datasets reserved for training.

Classification datasets are converted into a “Yes”/“No” QA format, with inputs as contexts and label information in questions.
The collection spans English tasks from Kaggle, Hugging Face, SemEval, and papers, covering sentiment, topic, grammaticality, paraphrase, stance, and other classification goals.
Some noisy, unreviewed, or overly complicated datasets are used for training but excluded from testing to make evaluation more informative and avoid selection bias.
Three authors manually annotated 441 questions for 204 unique labels, with each question proofread by another author.
The notion of unseen tasks is subjective because datasets can appear different in labels and inputs yet equivalent after QA unification.
Datasets sharing the same property tags are grouped, and training and testing prohibit datasets from the same group to exclude similar tasks.
The grouping procedure is not bullet-proof and may still define unseen tasks too leniently, so the authors perform additional relevance-based robustness checks.

3 Metrics

Performance is evaluated per label description with AUC-ROC for “Yes” as the positive class, then summarized across label descriptions using changes, threshold exceedance, variability, and scatter plots. Model superiority requires consistent improvements across multiple thresholds and weighting schemes.

The study avoids reducing performance across datasets to a single scalar and does not report F-score or accuracy because both depend strongly on decision cutoffs.
Each label description receives an AUC-ROC score by treating the “Yes” answer as the positive class.
The evaluation summarizes average change, threshold exceedance, negative changes, and standard deviation across label descriptions.
A model is considered better only when average improvement is positive and positive changes exceed negative changes at 1%, 5%, and 10% thresholds under all weighting schemes.
Scatter plots place model X scores on the x-axis and model Y scores on the y-axis, so points above y = x indicate better performance by model Y.

4 Model

The model concatenates each context and question for a T5-based QA architecture, scores the first decoded “Yes”/“No” token, and is meta-tuned on balanced samples from held-out groups of similar datasets.

The architecture concatenates context and question with a “[SEP]” token before encoding them with T5.
Answer scores are obtained by normalizing the first decoded token’s “Yes” and “No” probabilities.
Unless otherwise noted, experiments initialize the model with T5-Large containing 770 million parameters.
Meta-tuning balances datasets, label descriptions, and “Yes”/“No” answers while preventing duplicate label-description and input combinations.
Evaluation leaves out one group of similar datasets and trains on the remaining groups to measure zero-shot performance.

5 Results

Meta-tuned models outperform UnifiedQA and prior zero-shot classification systems, while larger pretrained models perform better. Pretraining is essential, and performance also depends on dataset similarity, label-description ensembling, initialization, and early stopping.

Meta-tuned models improve average AUC-ROC by 3.3% over same-sized UnifiedQA models.
Increasing model size from 220M to 770M parameters improves average AUC-ROC by 6.3%.
Randomly initialized models cannot beat the random baseline, with average AUC-ROC 0.503, showing that pretraining is essential.
Training on similar datasets improves average AUC-ROC by 0.7%, while ensembling label descriptions and initializing with UnifiedQA provide additional slight improvements.
Training too long decreases performance on unseen tasks, including a 3% AUC-ROC drop, although performance remains above random guessing.
Meta-tuned RoBERTa outperforms the strongest prior system across all three benchmark datasets using the same pretraining method.

6 Discussion and Future Directions

The paper frames meta-tuning as a way to adapt language models for zero-shot classification and argues that it may reveal more capability than prompt-based evaluation alone. It also identifies dataset curation, prompt quality, task scope, and probe rigor as important boundaries and future directions.

Meta-tuned models outperform a general-purpose QA model and prior natural-language-inference-based zero-shot learning system.
Meta-tuning may underestimate models’ zero-shot potential less than out-of-the-box prompting, while dataset aggregation could scale training and evaluation.
Probe validity depends on preventing training prompts from being too similar to test prompts.
Short or natural label descriptions can miss exact label semantics, motivating scalable generation of more accurate descriptions.
Prompt optimization remains complementary because semantically similar descriptions can produce substantially different accuracy.
Extending the framework to multi-label or few-shot learning requires label-resolution procedures or longer context windows and more computation.

A Ethics

The ethics discussion warns that collecting user prompts for meta-tuning could create privacy, security, fairness, and misuse risks. These risks may propagate across tasks and users when models are broadly deployed.

User prompts could become training data for inference providers, creating incentives to collect them for meta-tuning.
Meta-tuning on user data may expose prompts, enable data poisoning, and introduce security, privacy, or fairness concerns.
Bias or poisoned data from one task or user population could compromise another task’s fairness and performance at scale.
Because many tasks remain below 0.9 AUC-ROC, the system should not be applied to high-stakes scenarios.

B Datasets

The meta-dataset covers diverse zero-shot classification tasks, including sentiment, emotion, topic, situation, ethics, stance, question type, irony, and definition detection. Examples span social-media posts, reviews, legislation, news, and scientific text.

The collection includes sentiment classification for movie reviews, tweets, and financial text.
Emotion-related tasks classify inputs into predefined emotion categories, including nine broad emotions and four tweet-level emotions.
Topic-oriented tasks classify articles into domains such as health, sports, science, mathematics, and family relationships.
The collection also covers situation classification, ethics judgments, stance, and offensive or hate-speech detection.
Other tasks identify question types, irony, clickbait, abstract domains, definitions, and whether text concerns European legislation.

C Dataset Property Tags

Dataset similarity is defined using shared property tags, and evaluation excludes meta-tuning datasets judged similar to the evaluation dataset. Tags describe source, topic, input genre, and task semantics.

Datasets are considered similar when they share the same set of property tags.
The evaluation protocol disallows meta-tuning on datasets similar to the evaluation dataset.
Tags identify whether inputs come from social media, papers, reviews, or questions.
Other tags describe political or social relevance, topic classification, good-versus-bad judgments, and emotion classification.
Datasets judged clearly different from other tasks, such as definition extraction, may remain eligible for meta-tuning without tags.

E Robustness Checks

The robustness analysis reports descriptive comparisons across model sizes and weighting schemes, but the evaluation uses only around 20 datasets, so most results are not statistically significant.

Around 20 evaluation datasets limit the statistical significance of most reported robustness results.
The analysis compares T5-small with T5-base, BERT-medium with BERT-Base, and BERT-Base with BERT Large.
Results are reported under three types of description weighting, with additional comparisons across model-size pairs.

E.1 Different Description Weighting

Across nearly all weighting comparisons, performance changes are positive, and improvements above positive thresholds occur more often than declines below corresponding negative thresholds. The authors conclude that ensembling label descriptions is more likely to improve performance.

The mean change is positive under almost all comparisons across different weighting schemes.
Performance increases above positive thresholds occur more frequently than decreases below matching negative thresholds.
The ensemble comparison is the sole exception, with slightly more datasets showing changes below -1%.
Ensembling label descriptions is more likely to improve model performance.

E.2 Larger T5 Models are Better

Larger pretrained models perform better across the reported metrics. The strongest jump occurs when scaling from T5-small to T5-base.

Across all metrics, larger T5 models are significantly better.
Sometimes 15% increase in ¯∆ occurs when increasing model size from T5-small to T5-base.
Across all metrics, larger BERT models are significantly better across the tested sizes.

F Most Relevant Datasets

The evaluation separates training and test datasets with matching property tags, identifies relevant training datasets, and reports zero-shot performance using AUC-ROC-based summaries and visualizations. The supplied materials also document label-description groupings and table conventions, while warning that accuracy results are unreliable or not directly comparable.

Most Relevant Datasets: Training and test datasets are disallowed from sharing the same set of dataset property tags.This defines the paper’s stricter notion of an unseen task.
Most Relevant Datasets: For each evaluated dataset, the authors list the most relevant training task or mark it N/A when none is particularly relevant.
Label-description groups: The label-description groups contain related questions, and an initial “*” marks the dataset being evaluated.
Accuracy caveat: Accuracy is considered unreliable with imbalanced labels and is not directly comparable when some evaluations use test-set subsamples.