Source-linked AI summary
Not Enough Data? Deep Learning to the Rescue!
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, Naama Zwerdling
TL;DR
Text classification can require abundant labeled data, yet specialized applications often have scarce and costly labels. The paper proposes LAMBADA, which fine-tunes a language model to generate and classifier-filter labeled sentences. Across small-data settings, LAMBADA improves classifier performance and beats state-of-the-art text augmentation methods.
Problem
Effective text classification may require abundant labeled data, but labels are scarce and costly in many specialized applications.
Method
LAMBADA fine-tunes a pre-trained language model on small labeled data, generates class-conditioned sentences, and filters them with a classifier trained on the original data.
Results
LAMBADA statistically improves classifiers’ performance on small datasets and outperforms state-of-the-art data augmentation techniques across reported comparisons.
Takeaways & Limitations
LAMBADA provides a data-synthesis approach for improving text classifiers when only a few labeled examples are available.
Takeaways & Limitations
Repeated augmentation iterations may cause data drifting, allowing biased synthesized samples to dominate the training dataset.
Abstract
from arXiv · showhide
Based on recent advances in natural language modeling and those in text generation capabilities, we propose a novel data augmentation method for text classification tasks. We use a powerful pre-trained neural network model to artificially synthesize new labeled data for supervised learning. We mainly focus on cases with scarce labeled data. Our method, referred to as language-model-based data augmentation (LAMBADA), involves fine-tuning a state-of-the-art language generator to a specific task through an initial training phase on the existing (usually small) labeled data. Using the fine-tuned model and given a class label, new sentences for the class are generated. Our process then filters these new sentences by using a classifier trained on the original data. In a series of experiments, we show that LAMBADA improves classifiers' performance on a variety of datasets. Moreover, LAMBADA significantly improves upon the state-of-the-art techniques for data augmentation, specifically those applicable to text classification tasks with little data.
1 Introduction
LAMBADA addresses scarce labeled data in text classification by generating and filtering labeled sentences with a fine-tuned language model. Experiments report improved classifier performance and advantages over existing augmentation methods.
- Motivation: Scarce and costly labeled data can make training effective text classifiers difficult, especially for specialized applications.Chatbot intent classification may begin with only two, three, or five samples per class, or none.
- Motivation: Text augmentation is common for scarce-data problems, but textual transformations can distort grammar and semantics or introduce harmful synthetic examples.Existing textual methods commonly replace or delete words or change word order.
- LAMBADA: LAMBADA fine-tunes GPT on small labeled data, generates class-conditioned sentences, and filters them with a classifier trained on the original dataset.The pipeline uses a pre-trained generative model to synthesize labeled data before filtering lower-quality instances.
- Contributions: LAMBADA statistically improves classifier accuracy and outperforms state-of-the-art text augmentation methods in scarce-data situations.The reported contribution covers both classifier improvement and comparison with existing augmentation techniques.
- Contributions: LAMBADA is presented as a compelling alternative to semi-supervised techniques when unlabeled data does not exist.This claim is stated as one of the method’s three main contributions.
2 Related Work
Related work includes local sentence alterations, whole-sentence generation, and pre-trained language models. These developments motivate testing whether fine-tuned generative models can produce useful labeled text from very small datasets.
- Textual augmentation: Earlier textual augmentation methods alter individual sentences while preserving their original class, often through synonym replacement or other local edits.The related-work discussion identifies sample alteration as a dominant approach.
- Textual augmentation: Other approaches generate whole sentences using VAEs, round-trip translation, paraphrasing, generative adversarial networks, or data noising.These methods differ from local editing by synthesizing broader sentence-level variations.
- Pre-trained modeling: Pre-trained models such as ELMO, GPT, BERT, and GPT-2 use large-scale pre-training followed by task-specific fine-tuning.The shared training pattern combines publicly available text with task-specific data and objectives.
- Pre-trained modeling: GPT produced high-quality sentences after fine-tuning on small training data, supporting its use for scarce-data text augmentation.The paper illustrates this setting with generated sentences conditioned on class labels from five sentences per class.
- Motivation: The resulting research question is whether fine-tuning a pre-trained model can generate high-quality sentences that improve text-classifier performance.This question connects advances in text generation to supervised classification with limited labels.
3 Problem Definition
The paper formulates text classification as supervised learning over labeled strings and seeks a general augmentation method for small training sets. The augmented data should improve a classifier while better reflecting the underlying distribution without excessive noise.
- Problem setup: The training set contains labeled sentences, each represented as a token sequence and assigned one of q classes.Each sentence is drawn from an unknown distribution over strings, with labels determined by an unknown class function.
- Problem setup: The classification objective is to learn a function that approximates the unknown labeling function across the entire space of strings.Generalization extends beyond the observed training dataset to the full input domain.
- Evaluation: Classifier quality is estimated using test-set accuracy, where the Kronecker delta counts correct predictions.For a large test set, accuracy approximates the probability that the classifier matches the underlying labeling function.
- Data augmentation objective: Small training sets can substantially affect classifier performance, motivating synthetic training pairs that better reflect the data distribution without excessive noise.This is the stated problem that data augmentation is intended to address.
- Data augmentation objective: The paper targets a general method that augments Dtrain for an arbitrary classification algorithm A, producing a classifier trained on Dtrain ∪ Dsynthesized.LAMBADA is specifically tailored to small or even minuscule training sets with only a few examples per class.
4 LAMBADA Method
LAMBADA augments scarce text-classification datasets by fine-tuning a language model to generate labeled sentences, then filtering them with a baseline classifier. The method is designed to synthesize data for a specified classification algorithm and control the number of additions per class.
- Method overview: LAMBADA combines language-model fine-tuning and classifier-based filtering to retain high-quality synthesized labeled sentences.The language model is fine-tuned on the task data, and a classifier trained on the original data filters generated instances.
- Inputs: LAMBADA accepts a labeled training dataset, classification algorithm, language model, and desired synthesis counts per class as inputs.The method is designed to synthesize data for the algorithm that will train the augmented classifier.
- Objective: LAMBADA targets an augmented classifier A(D_train ∪ D_synthesized) that outperforms the baseline classifier trained only on D_train.The paper presents GPT-2 as the language model used in its method and reports higher performance in scarce-data settings.
- Fine-tuning and synthesis: GPT-2 is fine-tuned on the small task dataset and its class-labeled format before generating new sentences conditioned on each class label.The training sequence uses SEP between labels and sentences and EOS to terminate sentences and separate successive examples.
- Filtering: For each class, LAMBADA generates candidate sentences, verifies their predicted labels, ranks them by classifier confidence, and retains the top N_y instances.The resulting synthesized dataset has the same labeled structure as the original training set.
- Control and scope: The method can generate any number of sentences per class, allowing class balancing or control over the ratio of generated examples.Its stated scope is small or even minuscule training sets with only a few examples per class.
5 Experimental Results
Experiments evaluate LAMBADA across classifiers, datasets, sample sizes, and competing augmentation methods. The results show significant improvements over baselines and other generators, especially with scarce labeled data.
- Experimental Setup: Experiments use BERT, SVM, and LSTM classifiers across ATIS, TREC, and WVA datasets, varying training samples per class.Comparisons include baseline and alternative generative augmentation methods, with statistical validation using McNemar’s test.
- Number of Samples and Classifiers: LAMBADA outperforms the baseline for all classifiers and sample sizes up to 50 examples per class on ATIS.At 100 examples per class, LSTM and SVM accuracy does not improve as much from augmentation.
- Datasets: LAMBADA significantly improves all classifiers across ATIS, TREC, and WVA when trained with five samples per class.The authors attribute especially large ATIS gains for BERT and SVM partly to its imbalanced classes and generated data for under-represented classes.
- Comparison of Generative Models: LAMBADA is statistically superior to other generation algorithms across ATIS and WVA for all classifiers, while matching EDA for SVM on TREC.On TREC, BERT results are significantly better than those of all other methods.
- LAMBADA vs. Unlabeled Data: For ATIS with five samples per class, LAMBADA generally achieves better accuracy than weak labeling from unlabeled data.The reported analysis attributes the improvement to generated labels and the ability to control class-specific sample counts.
6 Discussion and Future Work
LAMBADA improves classifiers by fine-tuning a language model to synthesize labeled data and filtering generated sentences. The discussion highlights weak-labeling applications, possible zero-shot generation, and risks from iterative training.
- 6 Discussion and Future Work: LAMBADA combines language-model fine-tuning, conditional labeled-sentence generation, and filtering to improve classifiers on small datasets.The method’s two key ingredients are fine-tuning for synthesis and filtering for quality control.
- 6 Discussion and Future Work: LAMBADA’s filtering heuristic uses the generated label, the filtering classifier’s label and confidence, and the number of sentences per class.The authors plan to investigate other filtering heuristics and approaches.
- 6 Discussion and Future Work: LAMBADA produces weakly labeled corpora that can be incorporated into weak-labeling or semi-supervised frameworks.The synthesized corpus can also be used when unlabeled data is unavailable, with comparable results expected.
- 6 Discussion and Future Work: Because LAMBADA uses the semantic meaning of class labels, it may generate samples for meaningful unseen classes as a zero-shot learning approach.The authors identify this possibility as future research rather than an established result.
- 6 Discussion and Future Work: Repeating augmentation iterations may cause data drifting, whereby biased synthesized samples dominate the training dataset.A single augmentation step may already improve the classifier, but repeated iterations introduce this hazard.