Source-linked AI summary
ZeroGen: Efficient Zero-shot Learning via Dataset Generation
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, Lingpeng Kong
TL;DR
Zero-shot learning with PLMs can be unstable across prompts and expensive because inference relies on gigantic models. ZEROGEN generates task data without annotations, trains a tiny task model, and reports effectiveness across multiple NLP tasks, while identifying prompt stability and synthetic-data noise as limitations.
Problem
Prompt-based zero-shot learning is unstable across prompt choices and relies heavily on gigantic PLMs during inference.
Method
ZEROGEN generates a task-specific dataset from scratch with a PLM and trains a tiny, flexible task model on the synthesized data without human annotations.
Results
ZEROGEN significantly surpasses PLM counterparts with only ∼0.4% of their parameters and can outperform human-annotation supervision in some low-resource settings.
Takeaways & Limitations
ZEROGEN is presented as a viable approach to flexible and efficient zero-shot learning, with potential for data-free model-agnostic distillation and unreferenced text evaluation.
Takeaways & Limitations
Prompt-choice stability remains unsatisfactory, with ZEROGEN underperforming PROMPTING for some prompts and preferences varying across tasks.
Abstract
from arXiv · showhide
There is a growing interest in dataset generation recently due to the superior generative capacity of large pre-trained language models (PLMs). In this paper, we study a flexible and efficient zero-short learning method, \textsc{ZeroGen}. Given a zero-shot task, we first generate a dataset from scratch using PLMs in an unsupervised manner. Then, we train a tiny task model (e.g., LSTM) under the supervision of the synthesized dataset. This approach allows highly efficient inference as the final task model only has orders of magnitude fewer parameters comparing to PLMs (e.g., GPT2-XL). Apart from being annotation-free and efficient, we argue that \textsc{ZeroGen} can also provide useful insights from the perspective of data-free model-agnostic knowledge distillation, and unreferenced text generation evaluation. Experiments and analysis on different NLP tasks, namely, text classification, question answering, and natural language inference, show the effectiveness of \textsc{ZeroGen}.
1 Introduction
ZEROGEN addresses unstable, inefficient prompt-based zero-shot learning by generating task data with PLMs and training a tiny model without human annotations. The paper evaluates this framework across text classification, question answering, and natural language inference.
- ZEROGEN generates downstream training data from scratch with a PLM, guided by task-specific prompts.
- A tiny task model learns from synthesized data, using orders of magnitude fewer parameters than PLMs for efficient inference.The task model can use flexible architectures, losses, and task-specific inductive biases.
- ZEROGEN requires no human annotations and connects the PLM to the final task model through machine-generated text.
- The framework is also studied as data-free model-agnostic knowledge distillation, unreferenced text evaluation, and a setting for investigating prompt engineering.
- ZEROGEN is evaluated on text classification, question answering, and natural language inference across six datasets.
- ZEROGEN’s zero-shot performance significantly surpasses PLM counterparts with only ∼0.4% of their parameters.
2 Preliminary: Prompt-based Zero-Shot Learning
Prompt-based zero-shot learning combines each input with task prompts and uses a frozen PLM to score candidate classes. Although PLM knowledge enables this approach, prompt-based inference depends heavily on gigantic models and prompt choice.
- PROMPTING instantiates a prompt with each input and asks a PLM to complete the resulting natural-language sequence.
- A verbalizer maps each class label to one or more words in the PLM vocabulary.For sentiment analysis, “positive” and “negative” represent the two classes.
- PROMPTING freezes the pretrained PLM weights and requires no training.
- Despite relying on linguistic and factual knowledge encoded in PLMs, PROMPTING relies heavily on gigantic PLMs during inference and does not fully exploit their capacity.
3 ZEROGEN
ZEROGEN uses a PLM to synthesize a pseudo-dataset, trains a flexible tiny task model on it, and then performs efficient zero-shot inference. The generation procedure adapts to single-sentence and sentence-pair tasks.
- 3 ZEROGEN: ZEROGEN has three stages: pseudo-dataset generation, pseudo-supervised training of a TAM, and efficient inference.The process uses no human annotations.
- Pseudo dataset generation: For single-sentence classification, ZEROGEN samples labels, wraps them in label-descriptive prompts, and generates matching texts with a frozen left-to-right PLM.
- Pseudo dataset generation: Sampling algorithms such as Top-k and nucleus sampling increase the diversity of generated examples before pairing each text with its label.
- Pseudo dataset generation: For sentence-pair tasks, the method generates or samples a context, combines it with a label, and prompts generation of the related sentence.The procedure covers premise–hypothesis pairs in NLI and context–question pairs in QA.
- Pseudo-supervised training: A TAM can use any architecture, loss, and training strategy, while its much smaller size enables extremely efficient inference.
- Evaluation: Table 1 evaluates ZEROGEN across three PLM scales, two TAM scales, and 200k synthetic examples per task.
4 Experiments
Experiments across six datasets and three NLP tasks show that ZEROGEN generally outperforms prompt-based zero-shot learning while using much smaller task models. Performance depends on synthetic-data scale, generation quality, decoding strategy, prompt design, and PLM size.
- Experimental Setup: ZEROGEN was evaluated on six datasets spanning text classification, question answering, and natural language inference.The datasets include SST-2, IMDb, QNLI, RTE, SQuAD1.1, and AdversarialQA.
- ZEROGEN vs. PROMPTING: ZEROGEN outperformed PROMPTING in 15 of 18 DistilBERT head-to-head comparisons across three generators and six tasks.The advantage was reported as consistent across different PLM generators and task-model architectures.
- ZEROGEN vs. PROMPTING: ZEROGEN achieved comparable LSTM and better DistilBERT performance than PROMPTING with more than 200 times and 20 times fewer parameters, respectively.The synthesized dataset connects the large PLM to a lightweight task model, preserving efficient inference and serving.
- ZEROGEN vs. SUPERVISED: 200k synthesized samples enabled ZEROGEN to outperform SUPERVISED training on SST-2 and RTE, compared with 6.9k and 2.5k human annotations.The paper attributes this result to the larger automatically generated training sets and identifies low-resource settings as a favorable scenario.
- ZEROGEN vs. SUPERVISED: Generating 10k samples outperformed PROMPTING on most datasets, while increasing data volume improved performance continuously with diminishing returns.ZEROGEN still retained a performance gap against SUPERVISED on datasets such as SQuAD and QNLI.
- ZEROGEN as Text Generation Evaluator: More diverse decoding did not consistently improve downstream performance: nucleus sampling was nearly 6% and 3% below the best strategy on SQuAD and QNLI.Sampling methods were superior to greedy decoding on IMDb, but greedy decoding outperformed some sampling strategies on other tasks.
- Prompt Engineering in ZEROGEN: Natural-language prompts were favored, and ZEROGEN was more robust than PROMPTING to a minor prompt revision, reducing an IMDb accuracy drop from 16.2% to 9.4%.The paper links this robustness to using synthesized data as a medium between the PLM and TAM rather than directly instructing label words.
- Prompt Engineering in ZEROGEN: Larger PLMs benefited both PROMPTING and ZEROGEN on hard tasks such as SQuAD but degraded results on simpler text-classification tasks.The comparison used GPT2-XL and OPT with 175B parameters, showing that prompt selection remains important across model scales.
5 Related Work
Related work uses PLMs to generate pseudo-datasets, but ZeroGen takes dataset generation to an extreme by starting from scratch and targeting broader NLP tasks with tiny models.
- PLM-based dataset generation: Recent dataset-generation methods use PLMs to create training examples, often through data augmentation or semi-supervised training.Earlier approaches commonly fine-tune generative models with human annotations before combining generated and human data.
- ZeroGen’s distinction: ZeroGen generates task data from scratch with carefully designed prompts rather than relying on human annotations or unlabeled in-domain examples.This positions the framework as an extreme form of dataset generation for zero-shot learning.
- ZeroGen’s distinction: Compared with concurrent dataset-generation work, ZeroGen studies extremely small edge models, broader NLP tasks, and question answering.The paper explicitly contrasts its scope with prior work focused on semantic textual similarity, text classification, and natural language inference.
6 Conclusion and Future Directions
The paper concludes that ZeroGen is effective for annotation-free, efficient zero-shot learning, while identifying prompt instability and synthetic-data noise as future challenges.
- Conclusion: ZeroGen is presented as a viable and promising framework for flexible and efficient zero-shot learning in NLP.The paper also identifies potential applications in data-free model-agnostic knowledge distillation and unreferenced text evaluation.
- Conclusion: An LSTM trained without human annotations can surpass PLM zero-shot performance and, in some settings, a similarly trained supervised model.The reported comparison includes GPT2-XL as a PLM counterpart.
- Future directions: Prompt-choice stability remains unsatisfactory on natural language inference tasks despite positive results on text classification.The paper suggests multi-task prompt-based pre-training as one future direction.
- Future directions: Synthetic examples become noisier on difficult tasks such as natural language inference and question answering when decoding diversity increases.The authors call for decoding methods that preserve diversity while ensuring label correctness.
Limitations
The paper’s main limitations concern prompt robustness and noisy synthetic examples, especially on difficult tasks and under diverse decoding.
- Prompt robustness: ZEROGEN’s prompt-choice stability is limited, and it underperforms PROMPTING for some selected prompts.Prompt engineering also shows different preferences across tasks.
- Synthetic-data quality: Synthetic datasets contain noisy examples on difficult NLI and QA tasks, with noise increasing under more diverse decoding such as Nucleus Sampling.The proposed remedies are improved decoding strategies and noisy-label learning methods.
A Experimental Setup
The experiments generate synthetic datasets with specified sampling and filtering procedures, then train LSTM- and DistilBERT-based tiny task models from scratch.
- Dataset generation: Nucleus Sampling with p = 0.9 is the default generation strategy, using 200k synthetic examples for main results and 100k for analysis experiments.The setup also manually designs multiple prompts for each task and reports the best prompt for PROMPTING.
- Dataset generation: Generated text is filtered by removing overly short or long sentences and sentences without an ending quotation mark.These filters are part of the dataset-generation preprocessing.
- Task models: TAM implementations include an LSTM-based model and DistilBERT, with task-specific BiLSTM or BiDAF architectures for classification and question answering.The LSTM models use random-initialized embeddings rather than pretrained word embeddings.
B Additional Results on Prompt Design
Prompt suitability varies across tasks: natural-language prompts work best for QA and RTE, while control-code prompts perform better for QNLI. ZEROGEN also compares favorably with conventional knowledge distillation while using a data-free, dataset-based paradigm.
- B Additional Results on Prompt Design: Natural-language prompts achieve the highest scores for Question Answering in both PROMPTING and ZEROGEN settings.
- B Additional Results on Prompt Design: RTE favors natural-language prompts, whereas QNLI performs significantly better with control-code prompts.
- C Additional Related Work on Efficient Inference of PLMs: Efficient PLM inference has been studied through pruning, low-rank factorization, quantization, knowledge distillation, and parallel decoding.
- C Additional Related Work on Efficient Inference of PLMs: ZEROGEN generates data from PLMs without predefined labeled data and can produce a student model with superior zero-shot performance to the teacher.
- D ZEROGEN as Knowledge Distillation: ZEROGEN is compared with vanilla hard- and soft-label knowledge distillation using labels generated by GPT2-XL on an unlabeled training set.
- D ZEROGEN as Knowledge Distillation: The comparison tests whether dataset-based distillation better utilizes large PLMs by transferring knowledge through many input-output pairs.
E ZEROGEN for Data Augmentation
ZEROGEN’s synthetic data can supplement human annotations and support task-model training. The paper also examines whether larger models benefit from self-generated data and whether synthetic-data volume can be traded against performance.
- Synthetic data improves model performance when added to human-annotated data as augmentation.
- AUG-200k and AUG-500k denote augmentation with 200k and 500k synthetic examples, respectively.
- A tiny task model can outperform a large PLM after training on a synthetic dataset.
- Using a PLM as TAM tests whether the model can improve after fine-tuning on data it generated itself.
- A larger TAM further boosts performance, and PLMs can improve their own performance after fine-tuning on self-generated datasets.
G Generated Examples
ZEROGEN produces task-conditioned examples across classification, inference, and question answering. The examples illustrate both prompt adherence and the ability to generate varied training samples for TAMs.
- SST-2 generations generally match the prompts and sentiment polarity using descriptive movie-related tokens.
- QNLI and RTE generations respond to different prompts and relate to their supplied contexts, despite the complexity of natural language inference.
- For SQuAD, ZEROGEN generates a question that corresponds to a preset answer despite the context’s length and information density.
- The generated examples support producing useful and arbitrary numbers of training samples for TAM training.
- Table 10 reports prompt-design results for Question Answering and Natural Language Inference on development sets.
- Table 11 presents generated examples for each task and bolds the input conditions in each prompt.