Source-linked AI summary
Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
Wenpeng Yin, Jamaal Hay, Dan Roth
TL;DR
0Shot-TC lacks unified datasets and evaluations across diverse aspects, while classification must handle unseen labels and potentially unknown task domains. This work benchmarks the problem with diverse datasets, standardized partially- and fully-unseen evaluations, and textual-entailment models, finding robust performance across aspects without task-specific classification training.
Problem
0Shot-TC has received little attention, with prior studies and experiments largely limited to topical categorization and lacking uniform comparison.
Method
The paper provides datasets across topic, emotion, and situation aspects, defines label-partially-unseen and label-fully-unseen evaluations, and formulates 0Shot-TC as textual entailment.
Results
Textual-entailment models perform robustly across the three 0Shot-TC aspects without text-classification training or knowledge of test domains and aspects.
Takeaways & Limitations
The benchmark establishes standardized resources and evaluations for diverse 0Shot-TC settings and indicates textual entailment is a promising approach for this problem.
Abstract
from arXiv · showhide
Zero-shot text classification (0Shot-TC) is a challenging NLU problem to which little attention has been paid by the research community. 0Shot-TC aims to associate an appropriate label with a piece of text, irrespective of the text domain and the aspect (e.g., topic, emotion, event, etc.) described by the label. And there are only a few articles studying 0Shot-TC, all focusing only on topical categorization which, we argue, is just the tip of the iceberg in 0Shot-TC. In addition, the chaotic experiments in literature make no uniform comparison, which blurs the progress. This work benchmarks the 0Shot-TC problem by providing unified datasets, standardized evaluations, and state-of-the-art baselines. Our contributions include: i) The datasets we provide facilitate studying 0Shot-TC relative to conceptually different and diverse aspects: the ``topic'' aspect includes ``sports'' and ``politics'' as labels; the ``emotion'' aspect includes ``joy'' and ``anger''; the ``situation'' aspect includes ``medical assistance'' and ``water shortage''. ii) We extend the existing evaluation setup (label-partially-unseen) -- given a dataset, train on some labels, test on all labels -- to include a more challenging yet realistic evaluation label-fully-unseen 0Shot-TC (Chang et al., 2008), aiming at classifying text snippets without seeing task specific training data at all. iii) We unify the 0Shot-TC of diverse aspects within a textual entailment formulation and study it this way. Code & Data: https://github.com/yinwenpeng/BenchmarkingZeroShot
1 Introduction
The paper broadens zero-shot text classification beyond topical categorization and introduces a benchmark spanning diverse aspects and evaluation settings. It formulates the task as textual entailment to support classification without task-specific labeled data.
- Motivation: Zero-shot text classification receives limited attention and must handle compound, fine-grained, changing classes across topics, emotions, and other aspects.Existing work mainly studies topical categorization.
- Existing limitations: Conventional studies often assume training data for some seen classes and evaluate only unseen labels within the same aspect.This is the paper’s Definition-Restrictive setting.
- Existing limitations: Prior studies use different datasets, evaluation setups, and metrics, making fair comparison difficult.Examples include medical data evaluated with R@K and SNIPS intent detection evaluated only over unseen intents.
- Contributions: The benchmark covers topic, emotion, and situation aspects, with labels interpreting text through distinct aspect-specific meanings.Examples include sports or politics, joy or anger, and medical assistance or shelter.
- Contributions: The paper defines label-fully-unseen classification as learning without any task-specific labeled data, extending label-partially-unseen evaluation.The broader setup also allows labels and aspects to be unknown during model development.
- Contributions: It unifies diverse-aspect zero-shot classification as textual entailment and releases standardized datasets, evaluations, and an entailment system.The approach is intended to model human interpretation of aspect-defined problems and label meanings.
2 Related Work
Related work develops zero-shot classification through label representations, semantic spaces, and indirect supervision from other tasks. This paper extends that line by studying multiple aspects and standardized evaluations with textual entailment.
- Dataless classification: Dataless classification maps texts and labels into a common semantic space and selects the label with the highest matching score.Explicit Semantic Analysis was an early method in this paradigm.
- Label semantics: Later methods use pretrained word embeddings, label attention, generative models, class descriptions, hierarchies, knowledge graphs, or explanatory constraints.These approaches supply models with knowledge about label meanings.
- Indirect supervision: Some zero-shot relation-extraction studies obtain indirect supervision by converting the task into machine comprehension or textual entailment.A supervised model pretrained on an existing dataset then performs inference.
- This work: This work differs by formulating zero-shot text classification across multiple aspects and evaluation setups.Its broader benchmark includes datasets beyond topical categorization.
3 Benchmark the dataset
The benchmark standardizes datasets and separates label-partially-unseen from label-fully-unseen evaluation across topic, emotion, and situation classification. It includes both single-label and multi-label tasks, with realistic none categories where applicable.
- Benchmark design: The benchmark standardizes topic, emotion, and situation datasets with train, development, and test splits plus seen/unseen class separation.Both partial-unseen and fully-unseen settings are defined for systematic comparison.
- Evaluation setups: Label-partially-unseen training uses some classes while testing over the full label set; label-fully-unseen provides no training set.The latter supports test-agnostic evaluation and comparison across training-data conditions.
- Topic detection: Yahoo supplies a 10-class topic dataset with an original 1.4M/60k train/test split and balanced labels.The benchmark reorganizes its data into fixed development and test sets and two non-overlapping training versions.
- Emotion detection: UnifyEmotion contains nine emotion types plus none, draws from multiple domains, removes approximately 4k multi-label instances, and uses label-weighted evaluation.Partial-unseen training versions contain five and four classes, while fully-unseen evaluation has no training set.
- Situation detection: Situation Typing contains 5,956 labeled instances across 11 situation types plus none and is evaluated with label-wise weighted F1.It is a multi-label event-type task originally designed for low-resource detection without annotated data.
- Benchmark design: Across the benchmark, topic and emotion are single-label tasks, whereas situation is multi-label; emotion and situation include none categories when predefined types do not apply.These design choices aim to make the benchmark more realistic.
4 Benchmark the evaluation
The benchmark evaluates zero-shot text classification under two settings that differ in whether task-specific labeled data are available for some labels or none.
- These two setups are designed to approximate classification with open-defined labels across domains and aspects, while reflecting different levels of supervision.
- Label-partially-unseen training uses labeled examples from some labels and tests across the complete label space.
- Label-fully-unseen removes annotated data for every label and tests models on datasets spanning open aspects.
- The fully unseen setup follows dataless learning, encouraging models trained with open or task-agnostic data to generalize across domains and tasks.
5 An entailment model for 0SHOT-TC
The paper formulates zero-shot text classification as textual entailment by converting aspects and labels into hypotheses, then applies entailment models under partially and fully unseen settings.
- Textual entailment lets classification evaluate whether a text supports hypotheses formed from aspect interpretations and candidate labels.
- Labels are converted into hypotheses by completing aspect-specific interpretations with either label names or WordNet definitions.
- Classification data become entailment data: each text is a premise, with positive and negative labels supplying corresponding hypotheses.
- BERT is pretrained on MNLI, GLUE RTE, and FEVER as binary entailment tasks, then applied directly or fine-tuned depending on the evaluation setup.
- Seen labels receive a harsher selection policy than unseen labels when both are predicted positive.
- FEVER supplies ground-truth evidence sentences as premises for entailment decisions in this implementation.
6 Experiments
The experiments compare unsupervised, supervised, Wikipedia-based, and textual-entailment systems under partially and fully unseen-label settings across multiple aspects. Entailment models are generally robust across tasks, with performance depending on pretraining, fine-tuning, hypothesis generation, and label semantics.
- Evaluation: The benchmark evaluates label-partially-unseen and label-fully-unseen settings using topic, emotion, and situation classification tasks.The partially unseen setup includes seen and unseen labels, while the fully unseen setup removes task-specific labeled data.
- Baselines: Binary-BERT performs strongly on seen labels but substantially worse on unseen labels.It learns label-specific models from training data, which benefits seen-class prediction but limits generalization to unseen classes.
- Label-partially-unseen evaluation: Entailment models are competitive with Binary-BERT on seen classes and improve unseen-class performance by large margins.MNLI-pretrained entailment is slightly worse on topic and emotion seen classes but clearly better on situation; fine-tuning on MNLI appears more powerful.
- Label-fully-unseen evaluation: Entailment models pretrained on MNLI, FEVER, and RTE perform more robustly across the three aspects without task-specific classification training.The ensemble further boosts performance across all three tasks, although RTE is an exception on emotion.
- Label-fully-unseen evaluation: For fully unseen labels, pretrained models rank RTE > FEVER > MNLI, whereas fine-tuning on partially unseen data makes MNLI perform best.The authors suggest dataset similarity or fine-tuning overfitting may explain this reversal and leave deeper exploration for future work.
- Hypothesis generation: Definition-only hypotheses usually underperform, while word versus word-and-definition preferences vary by task and entailment model.The ensemble reaches top performance when combining words and definitions, motivating further study of ensemble and hypothesis-generation strategies.
- Hypothesis generation: Situation classes such as evacuation, infrastructure, and regime change remain difficult, while water, shelter, and food are recognized relatively well.The authors associate difficulty with abstract meanings and stronger recognition with common, more frequent labels.
7 Summary
The paper identifies restrictive definitions, weak label-meaning understanding, and inconsistent datasets and evaluations as central problems in zero-shot text classification. It responds by standardizing benchmarks and proposing textual entailment that can operate with or without seen-label annotations.
- Existing zero-shot text classification research is limited by restrictive problem definitions, weak label-meaning understanding, and chaotic datasets and evaluation setups.
- The paper standardizes datasets and evaluations across three aspects and proposes textual entailment for broader zero-shot classification.The framework is designed to work with or without annotated data for seen labels.