Source-linked AI summary
Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding
Yutai Hou, Yijia Liu, Wanxiang Che, Ting Liu
TL;DR
The paper studies data augmentation for language understanding when task-oriented dialogue systems have scarce labeled data and existing methods overlook relationships among utterances. It proposes a seq2seq framework that uses same-semantic alternatives and diversity ranks to generate varied augmented utterances. The method reports improvements of 6.38 and 10.04 F-scores in low-data settings.
Problem
Scarce labeled data makes task-oriented dialogue language understanding vulnerable to unseen utterances, while prior augmentation can ignore relationships among semantically related utterances.
Method
A seq2seq framework learns from utterances sharing semantic frames and uses diversity ranks to generate and filter diverse alternatives.
Results
6.38 and 10.04 F-scores are reported as improvements on ATIS and Stanford dialogue data when only hundreds of training utterances are represented.
Takeaways & Limitations
The framework produces diverse alternative expressions and improves language understanding performance when labeled training data is scarce.
Takeaways & Limitations
The method's effectiveness is sensitive to training conditions: neural-network results vary across random seeds, and augmentation can introduce noise when the full ATIS dataset is used.
Abstract
from arXiv · showhide
In this paper, we study the problem of data augmentation for language understanding in task-oriented dialogue system. In contrast to previous work which augments an utterance without considering its relation with other utterances, we propose a sequence-to-sequence generation based data augmentation framework that leverages one utterance's same semantic alternatives in the training data. A novel diversity rank is incorporated into the utterance representation to make the model produce diverse utterances and these diversely augmented utterances help to improve the language understanding module. Experimental results on the Airline Travel Information System dataset and a newly created semantic frame annotation on Stanford Multi-turn, Multidomain Dialogue Dataset show that our framework achieves significant improvements of 6.38 and 10.04 F-scores respectively when only a training set of hundreds utterances is represented. Case studies also confirm that our method generates diverse utterances.
1 Introduction
The paper addresses scarce labeled data for task-oriented dialogue language understanding by modeling relationships among semantically equivalent utterances. Its seq2seq augmentation framework uses diversity ranks to generate varied alternatives and improves LU especially in low-data settings.
- Motivation: Limited labeled data leaves LU vulnerable to unseen utterances that differ syntactically but remain semantically related to training examples.This vulnerability is especially important when moving to new domains where large-scale labeled data is usually unavailable.
- Prior work: Prior augmentation work perturbed individual utterances without modeling their relationships to other utterances, while related paraphrasing methods explicitly use shared outputs.The paper positions its approach between noise-based augmentation and paraphrase-aware augmentation.
- Approach: The proposed framework uses a seq2seq model to generate lexical and syntactic alternatives from utterances sharing the same semantic frame.It operates on delexicalised utterances and incorporates diversity rank information into the representation.
- Approach: A diversity rank encourages varied generation and filters over-alike alternative pairs during seq2seq training.The resulting augmented data is intended to improve LU performance in domains with scarce labeled data.
- Results: 6.38 and 10.04 F-scores are the reported improvements on ATIS and Stanford dialogue data when only hundreds of training utterances are available.The experiments cover ATIS and a newly annotated semantic-frame layer for the Stanford Multi-turn, Multidomain Dialogue Dataset.
2 Overview of the Approach
The framework generates semantically consistent alternatives through a seq2seq model, controls their diversity with ranked representations, and lexicalizes the resulting delexicalised utterances using context-sensitive slot-value mappings.
- Problem formulation: LU maps natural-language utterances to semantic frames, with slot filling treated as sequence classification over contiguous slot-value spans.The approach focuses on assigning slot-type labels to word sequences representing slot values.
- Augmentation workflow: Given an utterance and its semantic frame, the augmentation process generates new utterances paired with corresponding semantic frames from the training data.The framework traverses training instances to construct augmented examples.
- Seq2seq generation: The seq2seq model takes a delexicalised utterance and a specified diversity rank, then generates an alternative delexicalised utterance.The rank is appended to the input representation, and beam search with size 10 yields multiple candidate translations during testing.
- Training data: Training pairs are formed from utterances sharing a semantic frame, while diversity information helps avoid producing only lexical paraphrases or over-alike examples.The method uses same-frame alternatives for bidirectional generation and filters less diverse pairs.
- Surface realisation: Surface realisation replaces delexicalised slot types with slot values selected using the slot type and its surrounding context.If an exact context is unavailable, the method uses the most similar context by edit distance; multiple matching values are sampled randomly.
3 Diversity Ranks in Utterance Representations
The diversity rank models utterance diversity and conditions seq2seq generation on rank-specific alternatives. It combines edit distance with a length difference penalty and is added to the utterance representation as a generation controller.
- Rank construction: The method generates delexicalised utterances at ranks from 1 to N_s, where N_s equals half the number of training instances sharing semantic frame s.The rank range is governed by the size of the same-frame instance cluster.
- Rank construction: Diversity scores rank same-frame utterances using edit distance and a length difference penalty.The length penalty reduces the impact of length differences so the score emphasizes lexical and syntactical differences.
- Rank representation: The rank value is appended as an additional final token in the seq2seq model’s input representation.This incorporates the selected diversity level directly into generation.
- Rank representation: The diversity rank acts as an utterance-independent controller for target-generation diversity.The paper uses the rank to encourage diverse outputs during both augmentation and training.
4 Filtering the Alike Instances
The framework filters alike utterance pairs before seq2seq training, retaining only the most diverse translations. This reduces training ambiguity caused by mapping one utterance to multiple same-frame alternatives without rank information.
- Pair filtering: The method filters less diverse utterance pairs before training the seq2seq model.The filtering targets pairs whose usefulness for generating diverse augmented data is considered questionable.
- Pair filtering: Only the most diverse half of an utterance’s translations are used as seq2seq training data.The selected pairs are drawn using the diversity ranks.
- Model training: The filtered pair set is used to train the seq2seq model.
- Model training: Adding a unique rank to each training pair resolves ambiguities that otherwise arise when one utterance translates to multiple same-frame utterances.Without the rank, the paper states that learning can become intractable.
5 Experiments
Experiments evaluate the augmentation framework on ATIS and Stanford dialogue data, including small-data settings, ablations, cross-domain transfer, and diversity-focused analyses. The method improves language understanding most when training data is scarce, while syntactic generation, diversity ranks, and filtering contribute to performance and diversity.
- Experimental settings: The experiments use ATIS and a newly slot-annotated Stanford dialogue dataset, with F-score evaluation and five differently seeded runs.The Stanford annotation covers navigation, scheduling, and weather; the ATIS evaluation uses slot tagging.
- Results on ATIS: 6.38 F-score improvement is achieved on the small ATIS training proportion, while the medium-proportion improvement is 2.02 F-score.On the full ATIS data, augmentation slightly lags the baseline, which the authors attribute to sufficient data and introduced noise.
- Results on ATIS: The augmentation outperforms Kurata et al. on all ATIS proportions, with significant differences on the small and medium proportions.The comparison attributes the advantage to delexicalised generation and training from same-semantic-frame utterance pairs.
- Results on Stanford Dialogue Dataset: 10.04 average F-score improvement is observed with 100 Stanford training utterances, compared with 0.47 with 500 utterances.The larger gain in the smaller training setting follows the trend observed in the ATIS experiments.
- Training-data size and diversity analysis: The method is purely data-driven and transfers to new domains using a mapping from slot-type context to slot values constructed from training data.Its diversity rank also produces different utterances rather than repeating same-rank training utterances, unlike the compared Kurata augmentation.
- Ablation analysis: A 0.84 F-score drop occurs without seq2seq generation, while removing diversity ranks causes a 0.40 F-score drop.The ablations associate seq2seq generation with syntactic alternatives and diversity ranks with more newly and diversely generated utterances.
- Ablation analysis: Without filtering alike instances, performance drops by 0.65 F-score while producing more new utterances with smaller edit distances.The authors interpret this as evidence that improperly filtered training data introduces more noise.
- Training-data size and diversity analysis: Improvements are achieved on ATIS training sizes below one thousand, indicating applicability when only hundreds of LU training instances are available.Case studies also show syntactically different alternatives that preserve the source semantic frame and remain similar in length.
6 Related work
Prior language data augmentation commonly generates alternative expressions, but generic paraphrasing may not suit specific problems. Existing approaches motivate domain-aware alternatives, while GAN-based augmentation remains unexplored for language understanding.
- Language data augmentation usually generates alternative expressions because direct signal transformations can disrupt word-order-dependent syntax and semantics.
- Generic paraphrasing has limited usefulness for specific problems, so successful augmentation methods often use task-tailored paraphrasing techniques.
- Prior paraphrase-based augmentation includes word-level substitution, question-template bootstrapping, and grammar-based sampling.
- These approaches assume input paraphrases share an output, whereas this paper uses data-driven seq2seq generation without special tailored domain knowledge.
- GANs attract interest for adversarial data augmentation, but their application beyond computer vision, including language understanding, remains an open question.
7 Conclusion
The paper proposes data-driven seq2seq augmentation that uses same-semantic alternatives and diversity ranking to generate varied training utterances. On small training sets, it substantially improves language understanding and produces diverse alternatives.
- The framework trains a seq2seq model on alternative expressions sharing the same semantic frame and uses diversity rank to encourage variation and filter alike instances.
- 6.38 and 10.04 F-scores are achieved on small training sets in the reported experiments.
- Case studies show that the framework generates diverse alternative expressions.