PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot Text Classification With Generative Language Models

Raul Puri, Bryan Catanzaro

arXiv:1912.10165v1cs.CL

TL;DR

Zero-shot classification raises the question of whether natural-language task descriptions can support adaptation to new tasks without separate classification heads. This work trains a generative language model with weak supervision on title prediction and prompts it with class descriptions; it improves over random on some tasks, with a 45.2% absolute accuracy improvement over random on DBPedia. The paper concludes that natural language can serve as a learning and adaptation signal, while identifying controllability, available data, and task diversity as limitations.

  • Problem

    Prior zero-shot language-model work does not explicitly train for zero-shot text classification or establish adaptation to unseen task descriptors across domains.

    Source: §2 · p. 3

  • Method

    The model is pretrained on title prediction from web documents and receives downstream classes as natural-language multiple-choice descriptions, generating a class answer.

    Source: §1 · pp. 1–2, §1.1 · p. 3

  • Results

    45.2% absolute accuracy improvement over random is reported on DBPedia with the 355M GPT-2 model, while multiclass tasks remain below 50% and no task approaches fine-tuned or state-of-the-art accuracy.

    Source: §3 · p. 4

  • Takeaways & Limitations

    The results highlight natural language’s potential as a learning and adaptation signal for future applications.

    Source: §4 · p. 4

    Extensions beyond zero-shot classification require addressing controllability, available data, and task diversity.

    Source: §4 · p. 4

Abstract

from arXiv · show

This work investigates the use of natural language to enable zero-shot model adaptation to new tasks. We use text and metadata from social commenting platforms as a source for a simple pretraining task. We then provide the language model with natural language descriptions of classification tasks as input and train it to generate the correct answer in natural language via a language modeling objective. This allows the model to generalize to new classification tasks without the need for multiple multitask classification heads. We show the zero-shot performance of these generative language models, trained with weak supervision, on six benchmark text classification datasets from the torchtext library. Despite no access to training data, we achieve up to a 45% absolute improvement in classification accuracy over random or majority class baselines. These results show that natural language can serve as simple and powerful descriptors for task adaptation. We believe this points the way to new metalearning strategies for text problems.

1 Method

The method casts classification as multiple-choice question answering: a language model receives natural-language class descriptions and generates its answer. It is pretrained on title prediction from annotated web documents, then prompted with downstream task classes for zero-shot transfer.

1 Method

  • The model treats classification as multiple-choice question answering, using natural-language class descriptions as input and generating the answer in natural language.This replaces separate task-specific output heads with task descriptors prepended to the text.Source: §1 · p. 1, Figure 1 · p. 2
  • Title prediction on OpenWebText trains a pretrained GPT-2 to select among rich natural-language titles or descriptions, simulating many N-way classification tasks.The authors report that title-prediction language was needed for adaptation in initial studies; subreddit prediction did not transfer.Source: §1 · p. 1, §1.1 · p. 2
  • For pretraining, each document is paired with 2–15 sampled titles, including its correct title, and the model generates the answer to the resulting question.“None of the above” replaces one title half the time, and occasionally replaces the correct title.Source: §1 · p. 1
  • Training minimizes next-token prediction loss over concatenated question, reference text, and answer, with type tokens marking those segments.Questions are generated according to a grammar.Source: §1 · p. 1
  • Zero-shot transfer covers six torchtext datasets; the model receives each task’s classes as a multiple-choice question and generates a class using greedy decoding.Downstream tasks exclude “none of the above”; a 2,000-example validation set per task is used to evaluate design choices before final test accuracies.Source: §1 · p. 2, §1.1 · p. 3

2 Related Work

Prior work studies zero-shot language generation, meta-learning, and unified multitask models, but the cited approaches do not establish adaptation to unseen task descriptors across domains. This work combines natural-language task prompts with architecture-based meta-learning.

2 Related Work

  • Radford et al. (2018, 2019) use prompted autoregressive language models for zero-shot multitask behavior, but do not explicitly train for zero-shot text classification or use task descriptors to modulate output.Their classification proposal appends a prompt token and restricts the output vocabulary, effectively creating a pretrained task-specific classification head.Source: §2 · p. 3
  • The Multitask Question Answering Network examines supervised transfer between same-domain tasks with identical prompts, limiting evidence about adaptation to unseen descriptors.The cited evaluation focuses on SST-2 and Amazon-2.Source: §2 · p. 3
  • Architecture-based meta-learning uses memory-augmented networks or attentive learners, whereas this work supplies weakly supervised class labels and natural-language questions instead of similar class examples.The paper presents this combination of NLP and architecture-based meta-learning as a way to adapt to new language classification tasks.Source: §2 · p. 3
  • Concurrent approaches handle tasks with varying label counts or unify many task types, but the cited works do not examine zero-shot adaptation to new task descriptors.Raffel et al. (2019) demonstrate a unified model across classification, inference, question answering, and summarization; Bansal et al. (2019) combine prototypical networks and MAML.Source: §2 · p. 3

3 Results

Zero-shot transfer improves over random on some multiclass tasks, but remains below fine-tuned and state-of-the-art performance. Larger models and more pretraining data improve performance and reduce invalid generated answers.

3 Results

  • Increasing model size improves downstream performance, and restricting the pretraining dataset and document-title pairs reduces average performance across tasks.Table 2 reports zero-shot transfer test accuracies and, when applicable, the percentage of out-of-vocabulary test answers; its baselines include random guessing, majority class, and fine-tuning.Source: §3 · pp. 3–4, Table 2 · p. 4
  • On DBPedia, the 355M GPT-2 model achieves a 45.2% absolute accuracy improvement over random; across multiclass tasks, performance exceeds random but struggles to break past 50% and does not approach fine-tuned or state-of-the-art accuracy.Source: §3 · p. 4
  • Out-of-vocabulary answers are rare with greedy decoding and diminish as pretrained model and data size increase, but become more frequent with top-k or top-p sampling.Invalid outputs include empty strings and rearrangements or combinations of valid class names.Source: §3 · p. 4
  • Studies of descriptions and tokenization show adaptation to different descriptions but expose controllability issues, shifting practitioner effort toward data problem design.Source: §3 · p. 4

4 Conclusion and Future Work

The paper presents generative language modeling as a zero-shot classification method that avoids multiple task-specific heads. It identifies controllability, available data, and task diversity as limitations for extending the approach beyond its current setting.

4 Conclusion and Future Work

  • Generating classifications in natural language removes the need for multiple task-specific classification heads and makes the model more general and flexible.The conclusion says that increasing model and data scale demonstrates the capability of recent transformers to extract useful representations for generalizing and adapting to new tasks.Source: §4 · p. 4
  • The current work is limited to zero-shot classification; extensions to other settings require addressing controllability, available data, and task diversity.Proposed future directions include K-shot support-based learning, gradient steps on task descriptors, question answering, and instruction following.Source: §4 · p. 4

A.1.1 Input Tokens

The input representation concatenates question, text, and answer tokens, using distinct segment markers and embeddings to distinguish the fields and generated output.

A.1.1 Input Tokens

  • Question, text, and answer tokens are concatenated, each with an <|endoftext|> token and its own prepended prompt token.The markers are <|question|>, <|text|>, and <|answer|>.Source: §A.1.1 · p. 7
  • Each of the three segments receives a distinct type-token embedding.Source: §A.1.1 · p. 7
  • Two position-embedding ranges distinguish the context from generated output, with the second range restarting at zero for answer tokens.Source: §A.1.1 · p. 7

A.1.2 Multiple Choice Format

The multiple-choice format varies natural-language question templates and fills them with the task’s class choices.

A.1.2 Multiple Choice Format

  • Approximately 25 question formats are randomly sampled during training and evaluation, with brackets filled by the desired classes.Source: §A.1.2 · p. 7
  • Class choices are comma-separated and double-quoted, with spaces around answers and quotation marks to avoid unwanted byte-pair merges.Source: §A.1.2 · p. 7
  • Templates vary wording across document and text prompts, including category and choice formulations.Source: §A.1.2 · pp. 7–8

A.2 Training Hyperparameters

Training uses a 512-token maximum sequence length, dataset-dependent learning rates and batch sizes, and a cosine-decay schedule over 10 epochs.

A.2 Training Hyperparameters

  • All models use a maximum sequence length of 512 tokens; full-dataset training uses a 4 × 10^-5 learning rate and batch size 128, while quarter-dataset training uses 3 × 10^-5 and batch size 32.Source: §A.2 · p. 8
  • The learning rate warms up over 1% of iterations, then follows a single-cycle cosine decay schedule over 10 epochs.Source: §A.2 · p. 8
  • Training uses Adam with decoupled weight decay λ = 0.01, gradient-norm clipping at 1.0, and attention and hidden-state dropout of 0.1.Models use mixed precision with dynamic loss scaling on V100 GPUs.Source: §A.2 · p. 8

A.3 Training Data Statistics

The pretraining data spans many subreddits with a long-tailed frequency distribution, while its most common subreddits show a political skew that may bias training.

A.3 Training Data Statistics

  • Subreddit frequency follows a power law, clustering below 1,000 samples and extending to 245,000 samples for a subreddit.Figure 2 plots subreddit frequency against the number of subreddits at each frequency; Figure 3 enlarges the distribution.Source: §A.3 · p. 8, Figure 2 · p. 8, Figure 3 · p. 9
  • Of 50,700 subreddits, approximately 9,400 have at least 20 samples, and two-thirds of those have fewer than 100.Source: §A.3 · pp. 8–9
  • The authors describe this diversity as ideal for a meta-learning or domain-adaptation dataset.Source: §A.3 · p. 9
  • Half of the 15 most common subreddits are political, a skew that may bias training and plausibly reflects filtering for reputable outbound links.Source: §A.3 · p. 9

A.4 Downstream Task Setup

The appendix examines how natural-language class descriptions shape downstream classification, reporting that poor descriptions can sharply reduce accuracy and increase invalid answers.

A.4.1 Class Descriptors

  • Appendix A.4.1 lists the class descriptions used for each classification task.Source: §A.4.1 · p. 10

A.4.2 Descriptor Selection

  • For the 355 million parameter model, descriptor variants used positive/negative for sentiment, joined DBPedia class names, and shortened AGNews labels.The AGNews changes were World News to World and Science & Technology to Sci/Tech.Source: §A.4.2 · p. 10
  • The authors hypothesize that incomplete labels omit task meaning, while concatenated names can change bytepair tokenization and semantic understanding.They report that the model may avoid selecting affected classes and that some subword embeddings may not have been trained.Source: §A.4.2 · p. 10
  • In the worst case, poor class descriptions can lead to a 27% absolute drop in accuracy and a 44% increase in out-of-vocabulary answers.Table 4 compares validation accuracy and out-of-vocabulary answer percentages for good and bad descriptors.Source: Table 4 · p. 10, §A.4.2 · p. 10
  • Figure 4 maps its columns to the first and second descriptor rows in Table A.4.2, with color showing prediction frequency.Green marks the highest frequency, red the lowest, and yellow the middle.Source: Figure 4 · p. 11
Loading 1912.10165v1…