PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context Learning for Few-Shot Dialogue State Tracking

Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, Mari Ostendorf

arXiv:2203.08568v3cs.CL

TL;DR

Collecting and annotating task-oriented dialogues is costly, motivating flexible zero- and few-shot dialogue state tracking. IC-DST uses in-context learning with text-to-SQL and retrieved exemplars, achieving new MultiWOZ state-of-the-art results in few-shot settings and large improvements over prior zero-shot results.

  • Problem

    Dialogue-state annotations are costly, and changing schemas and domains call for DST strategies that require less data and remain flexible.

    Source: §1 · p. 1

  • Method

    IC-DST prompts a large pretrained language model with task context and exemplars, reformulates DST as text-to-SQL, and decodes states without parameter updates.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    IC-DST achieves state-of-the-art MultiWOZ results in few-shot settings using 1–10% training data and improves zero-shot state-of-the-art accuracy by 10–30% absolute on each domain.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On MultiWOZ, analyses indicate that each design innovation benefits performance, while applying the framework to wider dialogue tasks remains future work.

    Source: §7 · p. 9

    Performance depends strongly on the inference language model, which may limit use on speech transcripts or other languages lacking such pretraining data.

    Source: §9 · p. 10

Abstract

from arXiv · show

Collecting and annotating task-oriented dialogues is time-consuming and costly; thus, zero and few shot learning could greatly benefit dialogue state tracking (DST). In this work, we propose an in-context learning (ICL) framework for zero-shot and few-shot learning DST, where a large pre-trained language model (LM) takes a test instance and a few exemplars as input, and directly decodes the dialogue state without any parameter updates. To better leverage a tabular domain description in the LM prompt, we reformulate DST into a text-to-SQL problem. We also propose a novel approach to retrieve annotated dialogues as exemplars. Empirical results on MultiWOZ show that our method IC-DST substantially outperforms previous fine-tuned state-of-the-art models in few-shot settings. In addition, we test IC-DST in zero-shot settings, in which the model only takes a fixed task instruction as input, finding that it outperforms previous zero-shot methods by a large margin.

1 Introduction

Costly dialogue-state annotation and changing schemas motivate flexible DST strategies that require less data. IC-DST applies in-context learning through text-to-SQL, state-based context, and exemplar retrieval, reaching new MultiWOZ few-shot state of the art and improving zero-shot accuracy by 10–30% absolute per domain.

1 Introduction

  • DST requires flexible, scalable learning strategies because turn-level dialogue states are costly to collect and schemas commonly expand to new domains.The task extracts user intentions as slot values under a predefined schema.Source: §1 · p. 1
  • IC-DST achieves a new state of the art on MultiWOZ few-shot settings using 1–10% training data, and improves zero-shot accuracy by 10–30% absolute on each domain.Source: §1 · p. 2
  • IC-DST reformulates DST as text-to-SQL and includes a tabular ontology description in the prompt, matching the knowledge-grounded task and leveraging code-pretrained LMs.The named pretrained models are Codex, GPT-Neo, and CodeGen.Source: §1 · p. 2
  • IC-DST represents dialogue context with the dialogue state rather than the full conversation history, making context more efficient and better suited to domain changes.The dialogue state summarizes the prior interaction.Source: §1 · p. 2
  • For few-shot retrieval, IC-DST learns a similarity score trained to match similarity based on dialogue-state changes.The paper also analyzes the impact of its design innovations.Source: §1 · p. 2
  • The paper presents IC-DST as the first successful application of in-context learning to DST, using retrieved exemplars without parameter updates.This approach is intended to handle new-domain queries through exemplar retrieval without retraining.Source: §1 · pp. 1–2

2 DST System Design

IC-DST frames dialogue state tracking as in-context learning, using compact dialogue-state context and SQL representations. Its retrieval approach selects examples based on similarity between their state changes and the test turn's.

2.1 DST Task Framing

  • Predicting state changes addresses sparse few-shot coverage because each turn typically changes only a small number of slots.The predicted additions, deletions, and value changes are applied to the previous state.Source: §2.1 · pp. 2–3
  • The SQL representation treats each domain as a table, each slot as a column, and slot-values as WHERE-clause conditions.For turns involving multiple domains, the domains are renamed d1 through dm.Source: §2.1 · p. 3
  • Representing context as the previous state plus the current turn replaces the full dialogue history with a shorter summary.The representation is [y_{t-1}, A_t, U_t].Source: §2.1 · p. 3

2.2 In-Context Learning

  • In-context learning predicts from a prompt with task instructions, examples, and a test instance while keeping pretrained parameters fixed.Few-shot prompts use retrieved labeled turns; zero-shot prompts use a formatting example and no labeled data.Source: §2.2 · pp. 3–4

2.3 Dialogue Retriever

  • The retriever is trained to rank examples by state-change similarity, using contrastive learning to separate positive and negative pairs.SBERT performed best among the tested retrievers and is used for the reported IC-DST results.Source: §2.3 · pp. 4–5

3 Experiments

The experiments compare IC-DST with prior DST approaches on MultiWOZ and describe few-shot and zero-shot settings. They report joint goal accuracy as the main evaluation metric, but the supplied passages do not include table values.

3.1 Datasets

  • MultiWOZ contains over 10K multi-domain human-human dialogues; version 2.4 is described as a cleaner version of 2.1.The paper notes that DST results are generally higher on 2.4 than on 2.1.Source: §3.1 · p. 5

3.2 Baselines

  • The comparisons include TRADE, SGP-DST, TransferQA (Lin et al., 2021a), and DS2 (Shin et al., 2022).DS2-T5 is described as the strongest few-shot model in the literature, though it requires finetuning on DST labels.Source: §3.2 · p. 5

3.3 Experimental settings

  • Few-shot experiments sample 1%, 5%, or 10% of training data as a selection pool, while zero-shot prompts include one formatting example.The retriever is fine-tuned only on the few-shot selection pool.Source: §3.3 · p. 5

3.4 Experimental Details

  • Joint goal accuracy requires exact slot matches across all domains; the paper also reports slot-value-pair F1 for analysis.Few-shot results use all-domain JGA, while zero-shot results use per-domain JGA.Source: §3.4 · p. 5
  • Table 1 covers MultiWOZ 2.1 and 2.4 at 1%, 5%, 10%, and 100% of training data, averaging three runs.Codex experiments use 10 context exemplars, and GPT-Neo experiments use 5.Source: Table 1 · p. 6, §3.4 · p. 6

4 Results

On MultiWOZ, IC-DST achieves state-of-the-art results in few-shot settings and substantially outperforms prior zero-shot methods. Its zero-shot setting requires no DST training.

Few-shot DST on MultiWOZ

  • IC-DST achieves state-of-the-art performance with Codex at 1%, 5%, and 10% of training data, and performance improves with more retrieval candidates.GPT-Neo and CodeGen show a similar trend but are generally worse, suggesting model size matters for ICL.Source: §4 · p. 6
  • 43.1% vs. 23.5% against UnifiedSKG with about 80 labeled dialogues, and 50.6% vs. 13.9% against Few-Shot Bot on MultiWOZ 2.1.The prior ICL DST systems did not report standard few-shot configurations, so they were not included among the main baselines.Source: §4 · p. 6

Zero-shot DST on MultiWOZ

  • 35.3% multi-domain JGA on MultiWOZ 2.4 in zero-shot, compared with 48.35% for a system using 1% training data.IC-DST outperforms previous zero-shot results by a large margin and needs no training; adding a domain or slot requires updating SQL tables and adding a demonstration.Source: §4 · p. 6

5 Analysis

The analyses examine how input format, dialogue context, state-change prediction, and retrieval affect IC-DST. They also identify common error types.

Representation of dialogue context

  • Using the previous dialogue state and current turn gives the best performance for both retrieval objectives; full dialogue histories can require truncation and lower performance.The comparison varies dialogue context representation and retrieval objective in the 5% training-data setting with Codex.Source: §5 · pp. 6–7, Table 3 · p. 7

Dialogue states vs. state changes

  • State changes improve prediction: they contain fewer slots than full dialogue states, and more turns share the same changes, making good exemplars easier to retrieve.Across turns, state-change JGA stays high, while full-state JGA decreases as states gain slots; accumulated predicted changes outperform direct full-state prediction.Source: §5 · p. 7, Figure 5 · p. 7

Performance dependence on the retriever

  • Retriever quality strongly affects Codex performance, and the copy baseline’s F1 correlates with end-task JGA; the oracle retriever indicates room for improvement.The copy baseline performs much worse than IC-DST, indicating that the LM generalizes rather than simply copying example labels.Source: §5 · pp. 7–8, Table 4 · p. 8

Effect of DST as Text-to-SQL

  • Text-to-SQL format improves GPT-Neo, CodeGen, and Codex over the traditional slot-value-pair format, helping ICL use structured schema knowledge.The ablations use 100 random MultiWOZ 2.4 development dialogues in the 5% few-shot setting.Source: §5 · pp. 6, 8, Table 5 · p. 8

Error analysis

  • The examined errors arise from noisy training examples, retrieval failures that omit needed slots, or failures to generalize from examples.Sparse annotated data and retriever errors can both contribute to poor exemplars.Source: §5 · p. 8

6 Related Work

Prior DST work includes varied state representations and methods for reducing labeled-data needs. Earlier ICL applications to DST underperformed other methods, while dialogue retrieval differs from sentence- and document-focused retrieval work.

Dialogue state tracking

  • DST work has used naive and hierarchical state representations, while few-shot approaches include a summarization formulation; this paper instead represents states as SQL for text-to-SQL..Source: §6 · p. 8
  • Prior zero-shot methods use domain transfer, including related task-oriented domains and question-answering datasets, and their performance is described as quite low.Source: §6 · pp. 8–9

In-context learning

  • ICL has been applied to semantic parsing and other sentence-level tasks, but earlier DST systems underperformed other methods.Prior studies also explored retrieved examples with finetuning or prompted program generation.Source: §6 · p. 9

Retrieval

  • Retrieval research distinguishes similarity-based and LMscore-based methods, while dialogue retrieval studies cited here address different tasks and settings.This paper's retrieval task involves dialogues rather than sentences or documents.Source: §6 · p. 9

7 Conclusion

The paper applies ICL to DST through new context representation, retriever training, and text-to-SQL formulation, achieving state-of-the-art results on MultiWOZ in few-shot and zero-shot settings.

7 Conclusion

  • On MultiWOZ, IC-DST achieves state-of-the-art results in both few-shot and zero-shot settings, with analyses indicating that each innovation benefits performance.The innovations include dialogue-context representation, a retriever-training objective, and reformulating DST as text-to-SQL; future work may extend ICL to other dialogue tasks.Source: §7 · p. 9

9 Limitations

The section identifies dependence on the inference language model as a limitation, then provides prompt examples for few-shot, zero-shot, and traditional-format settings.

9 Limitations

  • The framework’s performance depends heavily on the inference language model, which may limit its use, especially for speech transcripts or languages underrepresented in pretraining.The authors also note a trade-off between avoiding fine-tuning costs and inference costs.Source: §9 · p. 10

A.1 Prompt in Few-Shot Settings

  • The few-shot prompt uses five retrieved examples from a 5% MultiWOZ training subset, followed by a sixth test instance whose SQL label the LM completes.The prompt also includes SQL-style domain tables and example rows.Source: §A.1 · pp. 13–14

A.2 Prompt in Zero-Shot Settings

  • The zero-shot prompt includes a crafted demonstration turn and a test instance, alongside the task prompt and domain tables.The test instance is marked as Example #2.Source: §A.2 · pp. 14–15

A.3 Prompt Example for Traditional Format

  • The traditional-format prompt lists the ontology in a dialogue-state-like format and includes five retrieved examples from a 5% MultiWOZ training subset.The test instance is Example #6, with its answer completed by the LM.Source: §A.3 · p. 15

B Details on Example Errors

The error examples illustrate three failure sources: noisy training labels, poor exemplar retrieval, and failure to generalize from an example.

B Details on Example Errors

  • A noisy training example with a missing attraction-type label illustrates one source of error.The section identifies the missing slot in the exemplar label as the concrete example.Source: B Details on Example Errors · p. 15
  • Poor example retrieval can leave the model mimicking an exemplar, predicting attraction-name but missing hotel-pricerange.The paper presents this as a second error type.Source: B Details on Example Errors · p. 15
  • The model may fail to generalize the demonstrated “don’t care” value, omitting hotel-area: don’t care.The paper presents this as a third error type.Source: B Details on Example Errors · p. 15

C Computing Details

The computing details distinguish inference from retriever training, reporting inference latency and the compute and tuning requirements for retriever experiments.

C Computing Details

  • Inference requires no model training; each turn takes about 2 seconds with GPT-Neo or CodeGen and about 4 seconds with Codex.GPT-Neo and CodeGen use one NVIDIA A10G graphics card; Codex uses the OpenAI API.Source: C Computing Details · pp. 15–16
  • Retriever training on one NVIDIA A10G takes about 10 minutes per epoch for the 5% setting and 6 hours for two epochs in the full-data setting.The 5% setting uses 10 epochs, and training time is linear in data size.Source: C Computing Details · p. 16
  • Retriever hyperparameters are manually tuned, and few-shot test results average three evaluation runs.The retriever is trained once per hyperparameter configuration.Source: C Computing Details · p. 16

D Dataset Details

The dataset details specify the MultiWOZ domains and dataset scale used in the work.

D Dataset Details

  • The experiments use five MultiWOZ domains—hotel, taxi, attraction, restaurant, and train—with 8,438 training dialogues and 1,000 each for development and test.The source dataset has seven domains; the selected five-domain setup averages 13.46 turns per dialogue and 13.13 tokens per turn.Source: D Dataset Details · p. 16

E Detailed Results

Table 7 reports detailed test JGA for individual few-shot runs on MultiWOZ 2.1 and 2.4.

E Detailed Results

Loading 2203.08568v3…