Source-linked AI summary
Few-shot Natural Language Generation for Task-Oriented Dialog
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, Jianfeng Gao
TL;DR
Task-oriented NLG must generate fluent responses from semantic dialog acts despite limited labeled data in new domains. The paper introduces the FEWSHOTWOZ few-shot benchmark and SC-GPT, which uses staged pre-training and few-shot fine-tuning. SC-GPT achieves the best overall automatic and human-evaluation performance on FEWSHOTWOZ and MultiWOZ, including a 4-point BLEU improvement on MultiWOZ.
Problem
Existing NLG benchmarks use extensive domain labeling, unlike real new-domain settings where only limited labeled data are available.
Method
The paper introduces FEWSHOTWOZ and trains SC-GPT through plain-text pre-training, dialog-act-controlled pre-training, and few-shot target-domain fine-tuning.
Results
SC-GPT achieves the best overall automatic and human-evaluation performance on FEWSHOTWOZ and MultiWOZ, outperforming previous MultiWOZ models by 4 points in BLEU.
Takeaways & Limitations
FEWSHOTWOZ provides a benchmark for few-shot NLG adaptation, while SC-GPT demonstrates strong semantic control and generalization with scarce domain labels.
Takeaways & Limitations
The paper identifies generating more interpersonal responses and extending generative pre-training to all four dialog-system modules as future work.
Abstract
from arXiv · showhide
As a crucial component in task-oriented dialog systems, the Natural Language Generation (NLG) module converts a dialog act represented in a semantic form into a response in natural language. The success of traditional template-based or statistical models typically relies on heavily annotated data, which is infeasible for new domains. Therefore, it is pivotal for an NLG system to generalize well with limited labelled data in real applications. To this end, we present FewShotWoz, the first NLG benchmark to simulate the few-shot learning setting in task-oriented dialog systems. Further, we develop the SC-GPT model. It is pre-trained on a large set of annotated NLG corpus to acquire the controllable generation ability, and fine-tuned with only a few domain-specific labels to adapt to new domains. Experiments on FewShotWoz and the large Multi-Domain-WOZ datasets show that the proposed SC-GPT significantly outperforms existing methods, measured by various automatic metrics and human evaluations.
1 Introduction
Task-oriented NLG converts semantic dialog acts into fluent natural-language responses, but existing approaches depend on extensive domain labeling. The paper introduces FEWSHOTWOZ and SC-GPT to study and address few-shot adaptation.
- Motivation: Task-oriented NLG converts semantic system actions into natural-language responses that should preserve meaning and remain fluent.Traditional template-based and statistical approaches face limitations in fluency, scalability, or labeling requirements.
- Benchmark: FEWSHOTWOZ simulates few-shot adaptation with seven domains and fewer than 50 labeled utterances per domain.The benchmark is based on existing MultiWOZ and Cambridge NLG resources.
- Model: SC-GPT is a multilayer Transformer trained on plain text, dialog-act-labeled utterances, and limited target-domain labels.The staged training is designed to acquire controllable generation before few-shot domain adaptation.
- Model: SC-GPT generates semantically controlled responses while requiring fewer domain labels to generalize to new domains than SC-LSTM.Its generation is conditioned on the supplied semantic form.
- Results: On MultiWOZ, SC-GPT establishes a new state of the art by outperforming previous models by 4 points in BLEU.On FEWSHOTWOZ, it also outperforms SC-LSTM and HDSA in adapting to new domains with smaller amounts of in-domain labels.
2 Background
A task-oriented dialog system processes user input through understanding, state tracking, policy, and generation. The NLG module converts a dialog act—an intent with slot-value pairs—into a response sequence.
- Pipeline: A typical task-oriented dialog pipeline passes user transcriptions through NLU, DST, dialog policy, and NLG.The policy produces a dialog act from information maintained by dialogue state tracking.
- Dialog acts: A dialog act combines an intent with slot-value pairs, whose number can vary across dialog acts.Intents distinguish system-action types, while slot-value pairs specify the information category and content to express.
- SC-GPT: SC-GPT generates each new response token by attending to the entire dialog act and the preceding response tokens.The figure illustrates this process with tokens such as “confirm” and “center.”
- NLG: The NLG goal is to translate dialog act A into a natural-language response sequence x = [x1, ···, xT].For example, confirm(name=Hilton, area=center) becomes a response confirming the requested hotel and area.
3 Semantically Conditioned GPT
SC-GPT combines GPT-2-style language modeling with dialog-act conditioning and limited domain-specific fine-tuning. Its design targets semantically adequate, fluent generation that can generalize to new domains with few labels.
- Training procedure: SC-GPT is trained in three stages: plain-text pre-training, dialog-act controlled pre-training, and target-domain fine-tuning.The controlled pre-training uses annotated dialog-act and response pairs to acquire controllable generation before adaptation.
- Semantic conditioning: Dialog acts are converted into control-code sequences and concatenated with augmented responses before being fed into GPT-2.The prediction loss is computed only for the response sequence, while the dialog act supplies the conditioning information.
- Fine-tuning: The model fine-tunes on only a few dozens of domain labels, despite novel intents or slot-value pairs in new domains.This procedure is intended to adapt the semantically controlled generator using limited domain-specific annotations.
- Model properties: SC-GPT is designed for flexibility, controllability, and generalizability in task-oriented response generation.It avoids delexicalization, preserves semantic intent and slot-value information, and is reported to generalize better than SC-LSTM.
4 Dataset: FEWSHOTWOZ
FEWSHOTWOZ is introduced to evaluate task-oriented NLG under realistic few-shot conditions. It expands domain coverage while reducing training data and training/testing overlap, requiring generalization to new intent compositions.
- Motivation: FEWSHOTWOZ addresses benchmarks with extensive per-domain labeling and high training/testing overlap that poorly measure new-domain generalization.Existing datasets contain hundreds to tens of thousands of labeled samples per domain, while some test dialog acts are fully covered during training.
- Dataset characteristics: FEWSHOTWOZ contains seven domains, more than existing NLG datasets described in the paper.The benchmark is designed to broaden evaluation across multiple task-oriented domains.
- Dataset characteristics: FEWSHOTWOZ uses fewer training instances per domain and only 8.82% training/testing overlap.Other datasets reportedly have more than 90% overlap, making FEWSHOTWOZ a lower-overlap few-shot setting.
- Evaluation setting: The benchmark requires models to generalize over new compositions of intents, including examples with multiple intents in some domains.Attraction, Taxi, and Train average 2, 1.33, and 2.05 intents per instance, respectively.
- Collection protocol: The dataset is constructed from RNNLG and MultiWOZ by grouping utterances by delexicalized dialog acts and sampling 50 training examples per domain, except Taxi with 40.The protocol retains one target utterance per dialog act and selects MultiWOZ utterances whose acts occur in only one domain.
5 Related Work
Prior task-oriented NLG methods include templates and statistical or neural generators, while pre-trained language models provide a related route to transfer. The paper positions SC-GPT as addressing semantic control and low-label adaptation together.
- Pre-trained models: GPT-2 demonstrated strong language modeling and transfer capabilities but lacked high-level semantic control during generation.CTRL and Grover use conditioning codes, but their codes are not readily applicable to task-oriented dialog NLG.
- Task-oriented NLG: Earlier neural NLG architectures and objectives generally require large amounts of annotated data to reach satisfactory performance.This limitation is especially relevant when dialog acts must be labeled from scratch for new domains.
- Paper focus: The paper focuses on improving sample efficiency for deploying NLG models to new domains with limited annotations.FEWSHOTWOZ and SC-GPT are proposed to formalize and address this research scenario.
6 Experiments
Experiments evaluate SC-GPT against established baselines on FEWSHOTWOZ and MultiWOZ using automatic and human assessments. Results show strong few-shot generalization, semantic controllability, and fluency, with performance advantages especially under limited in-domain data.
- Experimental setup: SC-GPT is evaluated against SC-LSTM, GPT-2, and HDSA on FEWSHOTWOZ and MultiWOZ using automatic and human metrics.The evaluation includes BLEU, entity coverage-related measures, informativeness, and naturalness assessments.
- FEWSHOTWOZ: SC-GPT outperforms GPT-2 and SC-LSTM significantly in FEWSHOTWOZ human evaluations for informativeness and naturalness.The results indicate better semantic control while maintaining good fluency, although a gap from human annotation remains.
- MultiWOZ: SC-GPT achieves the best BLEU performance on MultiWOZ, while GPT-2 performs similarly on the full dataset containing 57k utterances.The authors attribute GPT-2's comparable full-data performance to the amount of annotated data available.
- MultiWOZ: SC-GPT performs consistently better than GPT-2, HDSA, and SC-LSTM across MultiWOZ training sizes, with larger gains when fewer labels are available.Training sizes range from 0.1% (50 examples) to 50% of the dataset.
- FEWSHOTWOZ: SC-GPT reduces ERR to 4.96 on the unseen set, an order of magnitude below SC-LSTM and one-third of GPT-2.This result is associated with accurately grounding responses in novel dialog acts while composing fluent utterances.
- Analysis: Generated examples show SC-GPT preserving dialog-act slot information more precisely than baselines while producing fluent responses.Baseline examples may over-generate or omit important slots, whereas SC-GPT matches the semantic conditions more closely.
7 Conclusion and Future Work
The paper contributes FEWSHOTWOZ for evaluating NLG with scarce domain labels and SC-GPT for semantically controlled, generalizable generation. Future work targets more interpersonal responses and generative pre-training across the full dialog pipeline.
- Conclusion: FEWSHOTWOZ simulates few-shot learning with scarce labeled data, while SC-GPT targets semantic control and generalization in NLG.The paper reports best overall performance in automatic and human evaluations on FEWSHOTWOZ and MultiWOZ.
- Future Work: Future work includes mechanisms for generating more interpersonal responses to improve user experiences.The paper identifies interpersonal response generation as one of two directions for further study.
- Future Work: Another direction is extending generative pre-training to all four dialog-system modules for end-to-end training.The proposed organization treats module inputs and outputs as segments for a segment-level autoregressive model.