PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Augmentation for Spoken Language Understanding via Pretrained Language Models

Baolin Peng, Chenguang Zhu, Michael Zeng, Jianfeng Gao

arXiv:2004.13952v2cs.CL

TL;DR

SLU models face data scarcity, and earlier augmentation work focused on scant paired data despite other semi-supervised scenarios. This paper uses pretrained language models for semantically controlled synthetic-data generation and proposes solutions for ontology-rich and utterance-rich settings. On ATIS and Snips, the method outperforms baseline systems, with reported gains on ATIS-Small.

  • Problem

    SLU training suffers from data scarcity, and prior augmentation work focused on scant paired examples despite two other commonly arising semi-supervised scenarios.

    Source: §1 · p. 1

  • Method

    The paper uses pretrained SC-GPT for semantically controlled utterance generation and proposes SC-GPT-NLU to generate semantic labels for unlabelled utterances.

    Source: §1 · p. 1

  • Results

    On ATIS and Snips, the method outperforms baseline systems; on ATIS-Small, it achieves 0.5 points higher slot F1 and 3.02 points higher intent accuracy.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Experiments on ATIS and Snips show the method can leverage auxiliary unlabeled data to produce synthetic training data for higher-accuracy SLU models.

    Source: §5 · p. 4

    The Rich-in-Ontology and Rich-in-Utterance setups assume, respectively, many available dialogue acts or many available unlabelled utterances alongside labelled pairs.

    Source: §3.2.1 · p. 2, §3.2.2 · p. 2

Abstract

from arXiv · show

The training of spoken language understanding (SLU) models often faces the problem of data scarcity. In this paper, we put forward a data augmentation method using pretrained language models to boost the variability and accuracy of generated utterances. Furthermore, we investigate and propose solutions to two previously overlooked semi-supervised learning scenarios of data scarcity in SLU: i) Rich-in-Ontology: ontology information with numerous valid dialogue acts is given; ii) Rich-in-Utterance: a large number of unlabelled utterances are available. Empirical results show that our method can produce synthetic training data that boosts the performance of language understanding models in various scenarios.

1. Introduction

SLU data scarcity limits system applicability, while conventional augmentation trained on small domain datasets may not capture broader language variation. The paper uses pretrained models for semantically controlled augmentation and addresses two additional data-scarcity scenarios, reporting gains on benchmark datasets.

1. Introduction

  • Conventional augmentation models trained on small domain-specific datasets may have limited language expressibility beyond their training data.The paper motivates pretrained language models as a way to adapt with modest domain-specific data.Source: §1 · p. 1
  • The method frames augmentation as semantically controlled generation, using pretrained SC-GPT to generate utterances from dialogue acts.It combines pretrained language syntax and semantics with domain-specific utterance generation to increase variability and accuracy.Source: §1 · p. 1
  • Earlier SLU augmentation work focused on scant labelled utterance-semantic-label pairs, leaving two other commonly arising semi-supervised scenarios overlooked.The paper proposes corresponding augmentation solutions for Rich-in-Ontology and Rich-in-Utterance.Source: §1 · p. 1
  • Rich-in-Ontology provides valid dialogue-act combinations but lacks corresponding natural-language utterances, and the method generates examples for valid intent-slot combinations.SC-GPT is fine-tuned on paired training data before generating utterances from ontology combinations.Source: §1 · p. 1
  • Rich-in-Utterance provides abundant unlabelled utterances, and SC-GPT-NLU generates intent and slot sequences to label them.The model is pretrained on the SC-GPT corpus with sources and targets flipped.Source: §1 · p. 1
  • On ATIS and Snips, the proposed method outperforms baseline systems; on ATIS-Small, it yields 0.5 points higher slot F1 and 3.02 points higher intent accuracy.The paper also reports significant performance boosts in Rich-in-Ontology and Rich-in-Utterance settings.Source: §1 · p. 1

2. Related Work

Prior SLU augmentation work explores diverse ways to generate or select training utterances, while pretrained language models provide capabilities for language understanding and generation. The paper situates its approach among both strands.

2.1. SLU Data Augmentation

  • SLU augmentation methods include perturbing decoder states, VAE-based generation, substitution and paraphrasing, generate-filter-rank pipelines, reinforcement learning, templates, and pseudo-labelling.The paper distinguishes its data-augmentation focus from semi-supervised learning independent of downstream models.Source: §2.1 · p. 2
  • Prior pretrained-model augmentation blends labels and input sentences during training, making the amount of generated data per class hard to control [17].Source: §2.1 · p. 2

2.2. Pretraining

  • Pretrained models including ELMo, GPT-2, BERT, and UniLM use different language-modeling approaches and have supported NLP applications such as question answering and summarization [18] [6] [20] [21].Source: §2.2 · p. 2
  • SC-GPT builds on GPT-2 and is further pretrained on a large dialogue corpus; it outperforms many baselines in few-shot task-oriented dialogue generation [8].Source: §2.2 · p. 2

3. Data Augmentation

The paper treats augmentation as generation conditioned on dialogue acts, using pretrained models to address small paired datasets and two additional semi-supervised data settings. It describes generating utterances from ontology acts and generating dialogue acts for unlabeled utterances.

3.1. Traditional Augmentation Scenario

  • SC-GPT is initialized with GPT-2, pretrained on 400K dialogue-act–utterance pairs, and fine-tuned on paired data to generate utterances from dialogue acts.The augmenter generates candidate utterances for dialogue acts and filters them to retain required slot-values.Source: §3.1 · p. 2, Table 1 · p. 2

3.2. More Data Augmentation Scenarios

  • The authors identify two overlooked settings where information beyond paired training data is available for augmentation.These settings are Rich-in-Ontology and Rich-in-Utterance.Source: §3.2 · p. 2

3.2.1. Rich In Ontology

  • For Rich-in-Ontology, SC-GPT generates utterances for valid dialogue acts in the ontology that are not paired with training utterances, then filters for required slot-values.Source: §3.2.1 · p. 2

3.2.2. Rich In Utterance

  • For Rich-in-Utterance, SC-GPT-NLU reverses the generation direction, producing dialogue acts for unlabeled utterances using a model trained on paired data.Generated intent, slot, and value names are mapped to the ontology by string matching, and outputs are filtered for slot-value consistency.Source: §3.2.2 · pp. 2–3
  • Figure 1 depicts augmentation across all three scenarios, with models initialized from GPT-2, pretrained on a 400K dialogue corpus, and fine-tuned on paired data.Source: Figure 1 · p. 3

4. Experiments

Experiments evaluate augmentation on ATIS and Snips using slot F1 and intent accuracy, with results spanning paired-data-only and two semi-supervised settings. The reported comparisons favor the proposed method, particularly with small training sets, and examples illustrate both generation directions.

4.1. Datasets and Metrics

  • The experiments use ATIS and Snips, measure slot tagging with F1 and intent classification with accuracy, and simulate Small and Medium few-shot training splits.ATIS contains about 5.8K utterances; Snips contains 14K, with 120 and 123 slot labels respectively.Source: §4.1 · p. 3

4.2. Model Details

  • The shared downstream model is a two-layer bidirectional LSTM; augmentation adds generated utterances or dialogue-act pairs, with filtering before training.SC-GPT and SC-GPT-NLU are fine-tuned for five epochs, and nucleus sampling uses top-p 0.9.Source: §4.2 · pp. 3–4

4.4. Examples of Augmented Data

  • Table 3 gives Snips examples of SC-GPT utterances conditioned on dialogue acts and SC-GPT-NLU dialogue acts predicted from unlabelled utterances.The examples are described as coherent and variable, covering required intent, slots, and values, while predicted dialogue acts capture input semantics.Source: Table 3 · p. 4, §4.4 · p. 4

4.3. Results

  • On ATIS Paired-Data-Only, augmentation increases slot F1 by 6.51 points (Small) and 1.31 (Medium), and intent accuracy by 1.68 (Small) and 0.56 (Medium).The reported gains are especially pronounced with smaller training sets.Source: §4.3 · p. 4
  • Under Rich-in-Ontology and Rich-in-Utterance, augmentation raises slot F1 by up to 7 points and intent accuracy by up to 2.4 points; ontology information yields the highest accuracy scores.For Paired-Data-Only, the method outperforms all baselines on ATIS-Small and is comparable on ATIS-Medium.Source: §4.3 · p. 4

5. Conclusion

The paper addresses SLU data scarcity with pretrained language models and reports that synthetic training data boosts downstream model performance. It also proposes solutions for two semi-supervised scarcity scenarios; extending the approach to other speech-language tasks is future work.

5. Conclusion

  • On ATIS and Snips, pretrained-model augmentation produces synthetic data that improves downstream SLU performance, including in Rich-in-Ontology and Rich-in-Utterance settings.The paper identifies extension to information retrieval and summarization as future work.Source: §5 · p. 4

Related papers

Loading 2004.13952v2…