Source-linked AI summary
Data Augmentation for Intent Classification with Off-the-shelf Large Language Models
Gaurav Sahu, Pau Rodriguez, Issam H. Laradji, Parmida Atighehchian, David Vazquez, Dzmitry Bahdanau
TL;DR
Collecting and labelling intent-classification data is costly, and distinguishing numerous fine-grained intents is challenging. The paper prompts off-the-shelf language models to generate labelled training data, finding greater benefit when intents are distinct and less benefit when they are semantically close. It also reports preliminary evidence for filtering generated examples with a prompting-based classifier.
Problem
Intent classification needs labelled training data, which is tedious and expensive to collect, and can involve many semantically close intents.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
Generated data can help as additional labelled data for some tasks, while other tasks may need it relabelled or filtered; GPT-based filtering can be effective.
Source: §6 · p. 9The main data-scarce evaluation setup uses K = 10 training examples per intent.
Source: §3.2 · p. 2
Abstract
from arXiv · showhide
Data augmentation is a widely employed technique to alleviate the problem of data scarcity. In this work, we propose a prompting-based approach to generate labelled training data for intent classification with off-the-shelf language models (LMs) such as GPT-3. An advantage of this method is that no task-specific LM-fine-tuning for data generation is required; hence the method requires no hyper-parameter tuning and is applicable even when the available training data is very scarce. We evaluate the proposed method in a few-shot setting on four diverse intent classification tasks. We find that GPT-generated data significantly boosts the performance of intent classifiers when intents in consideration are sufficiently distinct from each other. In tasks with semantically close intents, we observe that the generated data is less helpful. Our analysis shows that this is because GPT often generates utterances that belong to a closely-related intent instead of the desired one. We present preliminary evidence that a prompting-based GPT classifier could be helpful in filtering the generated data to enhance its quality.
1 Introduction
The introduction frames intent-classification augmentation as a response to costly data collection and the difficulty of distinguishing many fine-grained intents. It proposes prompting off-the-shelf language models and reports that usefulness depends on intent separation, with filtering and human relabelling offering potential improvements.
1 Introduction
- Manual authoring and crowdsourcing make training-data collection tedious and expensive, motivating data augmentation as a way to alleviate acquisition costs.Prior off-the-shelf GPT3Mix results covered only up to six classes with easy-to-grasp boundaries.Source: §1 · pp. 1–2
- Task-specific fine-tuning can become a bottleneck when task data are scarce, motivating off-the-shelf language models that generate text without task-specific fine-tuning.The introduction describes fine-tuned generation as one paradigm and off-the-shelf generation as another.Source: §1 · pp. 1–2
- Intent classification is difficult to augment because it may involve hundreds of semantically close intents, making approaches that prompt with many class names poorly suited.The proposed prompt instead focuses on one seed intent and combines its name with all available examples.Source: §1 · p. 2
- GPT-generated samples boost accuracy when intents are well-distinguished, but help less on granular datasets; preliminary findings suggest GPT classification can filter unfaithful examples.The experiments also find that human relabelling improves classifier performance and generated data has greater relabelling potential than EDA.Source: §1 · p. 2
2 Method
The method augments intent-classification training data by prompting a large pretrained language model with examples available for an intent. An intent is a request type supported by the conversational agent.
2 Method
- An intent is a type of request that the conversational agent supports, such as changing the conversation language or transferring money.The classifier’s goal is to predict the user’s intent from an utterance.Source: §2 · p. 2
- The method creates prompts from the available examples for an intent and feeds them to a large language model such as GPT-3 to generate augmented data.The process applies to an intent with K available examples.Source: §2 · p. 2
- The augmentation targets the training data for each supported intent, addressing the difficulty and expense of collecting many utterances expressing the same intent.The method uses the available examples as the basis for generation.Source: §2 · p. 2
3 Experimental Setup
The experiments cover four intent-classification datasets and use few-shot settings with ten examples per intent. BERT-large classifiers are evaluated across baseline and augmentation scenarios, with additional measures of generated-data fidelity and out-of-scope recall.
3.1 Datasets
- The four datasets span multidomain CLINC150 and HWU64, fine-grained banking intents in Banking77, and seven smart-speaker intents in SNIPS.CLINC150 also includes out-of-scope utterances; Table 1 provides dataset statistics.Source: §3.1 · p. 2, Table 1 · p. 3
3.2 Setup
- The main few-shot setup uses K = 10 training examples per intent, and augmentation expands few-shot intents to the median number of examples per intent.The partial few-shot setup retains full training data for data-rich intents while limiting data for selected few-shot intents.Source: §3.2 · pp. 2–3
3.3 Training and Evaluation
- BERT-large classifiers are evaluated in full- and partial-few-shot baseline and augmentation scenarios, with accuracy and, for CLINC150, out-of-scope recall reported.Full few-shot comparisons also include EDA; partial few-shot comparisons include upsampling, and runs are averaged across repetitions.Source: §3.3 · pp. 3–4
- An oracle trained on the full training set assesses generated-sample fidelity and estimates gains from human labeling.Figure 2 reports few-shot accuracy, out-of-scope recall, and fidelity for different GPT-3 models and temperatures.Source: §3.3 · p. 3, Figure 2 · p. 4
4 Experimental Results
GPT augmentation improves full few-shot accuracy most on CLINC150 and SNIPS, while gains are limited on Banking77 and HWU64. Analysis links weaker results to close-intent confusion and finds that GPT-3 filtering can improve generated-data fidelity.
4 Experimental Results
- Oracle relabelling of GPT- and EDA-generated sentences significantly boosts accuracy across datasets; GPT data improves more, and relabelling also improves CLINC150 out-of-scope recall.The authors suggest GPT generations may be more diverse and useful after careful human inspection.Source: §4 · p. 4
- Accuracy improves by up to approximately 3.7% on CLINC150 and 6% on SNIPS with GPT-3 augmentation, exceeding EDA’s 2.4% and 2.9% gains, while Banking77 and HWU64 show little or no improvement.GPT augmentation also lowers CLINC150 out-of-scope recall.Source: §4 · p. 4, Table 2 · p. 5
- Few-shot accuracy improves by up to 2.92% on CLINC150 and 18.14% on SNIPS in the partial few-shot setup, with results competitive with Ex2.The method uses BERT-large with 340 million parameters, compared with Ex2’s T5-XL at nearly 3 billion parameters.Source: §4 · pp. 4–5, Table 3 · p. 5
4.1 Analysis
- Around temperature 1.0, generated data yields the highest classification accuracy; lower temperatures raise fidelity, while higher temperatures increase diversity.For all datasets, lower-temperature GPT-J generations have higher fidelity, and low-temperature data also improves model performance.Source: Figure 3 · p. 4, §4.1 · p. 6
- Generated examples can mix close intents or combine their features, as in a music request assigned to “music_likeness” or a failed top-up mentioning a card.The paper gives these as examples of confounding between seed intents and alternatives.Source: Table 4 · p. 6, §4.1 · p. 7
4.2 Can GPT Models Understand Close Intents?
- GPT-3 rejects generations it classifies outside the seed intent, significantly improving fidelity for the selected close-intent triplets in HWU64 and Banking77.Its three-way classification accuracy on these triplets falls between BERT-large trained on 10 examples and on the full training set.Source: §4.2 · pp. 7–8, Table 5 · p. 7
4.3 Comparison with GPT3Mix
- On CLINC150, GPT3Mix and a mixed prompt variant perform poorly, with in-scope accuracies of 86.33% and 86.05% and fidelities of 27.96% and 33.56%, respectively.The authors interpret the results as indicating that GPT cannot handle the prompt’s long list of 150 intent names.Source: §4.3 · p. 8
5 Related Work
Prior augmentation approaches include edits, backtranslation, and task-specific fine-tuning, while off-the-shelf LM methods avoid the fine-tuning bottleneck. The paper’s prompt focuses on one seed intent, and its GPT-3 filtering rejects rather than assigns labels.
5 Related Work
- Edit-based methods such as Easy Data Augmentation alter utterances, backtranslation translates them out and back, and fine-tuned LMs generate examples from task-specific data.The paper’s method instead uses off-the-shelf LMs without task-specific fine-tuning.Source: §5 · p. 8
- GPT3Mix prompts with a list of all class names and asks the LM to select a class and generate a labelled example, but the paper says this does not scale to intent tasks with hundreds of intents.The paper instead uses a prompt centered on one seed intent and its examples.Source: §5 · p. 8
- Unlike Wang et al. (2021), which uses GPT-3 to propose labels for unlabelled samples, this work uses it to reject generated samples with incorrect labels.Source: §5 · p. 8
6 Conclusion
The paper proposes prompting large pretrained language models to generate intent-classification data and finds generated data helpful for some tasks but in need of relabeling or filtering for others.
6 Conclusion
- Prompting large pretrained language models is the proposed approach for generating intent-classification data.Source: §6 · p. 8
- Prompt-based filtering that recasts GPT as a classifier can be effective, but it requires knowing which other intents generated utterances may belong to.Future work could develop heuristics to identify those likely actual intents.Source: §6 · p. 9
- Generated data can help as additional labelled data for some tasks, while other tasks require relabeling or filtering for it to be helpful.Source: §6 · p. 9
7 Ethical Considerations
The authors caution that generated samples require careful consideration because large language models can exhibit social biases and produce toxic content.
7 Ethical Considerations
- Generated samples may exhibit social biases or toxic content, so the authors say they need to be considered carefully.Source: §7 · p. 9
- Human inspection is described as the most reliable way to identify and filter problematic generations; practitioners may also debias the language model before generation.Source: §7 · p. 9
A Human Evaluation
A human evaluation tests whether people can distinguish GPT-3-generated sentences from human examples of the same intent. Evaluators made more mistakes at lower temperatures and with higher-capacity models.
A Human Evaluation
- Higher human error rates indicate greater sample fidelity; evaluators made more mistakes at lower temperatures, and higher-capacity models produced more indistinguishable sentences.The authors report that this agrees with the oracle-based fidelity results.Source: Figure 6 · p. 11, A Human Evaluation · p. 11
- Fidelity was framed as human inability to distinguish generated from same-intent human sentences, and the reported error rate was failed tasks divided by total evaluated.The study also compared human judgments with an oracle intent classifier's performance on generated samples.Source: A Human Evaluation · p. 11
- For each CLINC150 intent, evaluators saw five examples, with one same-intent GPT-3 sentence substituted at random or none substituted.The study used 15 voluntary evaluators, who annotated 500 tasks; Figure 7 shows an example question.Source: A Human Evaluation · p. 11, Figure 7 · p. 11