Source-linked AI summary
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, Pranav Khaitan
TL;DR
Virtual assistants need dialogue systems that scale across many heterogeneous services, while existing datasets provide limited coverage of this setting. The paper introduces SGD and a schema-guided modeling paradigm, then presents a state tracker that generalizes to unseen services and APIs while achieving competitive results. The authors also identify difficulty conducting direct side-by-side dataset comparisons because SGD covers many new domains at this scale.
Problem
Existing task-oriented dialogue datasets cover few domains and do not represent multiple heterogeneous services with overlapping functionality.
Method
The paper introduces SGD and conditions a unified dialogue model on service schemas to predict dynamic intents and slots across services.
Results
The proposed dialogue state tracker generalizes to unseen services and APIs while achieving competitive results on original and updated MultiWOZ datasets.
Takeaways & Limitations
SGD provides a benchmark for intent prediction, slot filling, state tracking, and language generation in large-scale virtual assistants.
Takeaways & Limitations
The authors could not conduct a side-by-side comparison with existing datasets because SGD covers many new domains at this scale.
Abstract
from arXiv · showhide
Virtual assistants such as Google Assistant, Alexa and Siri provide a conversational interface to a large number of services and APIs spanning multiple domains. Such systems need to support an ever-increasing number of services with possibly overlapping functionality. Furthermore, some of these services have little to no training data available. Existing public datasets for task-oriented dialogue do not sufficiently capture these challenges since they cover few domains and assume a single static ontology per domain. In this work, we introduce the the Schema-Guided Dialogue (SGD) dataset, containing over 16k multi-domain conversations spanning 16 domains. Our dataset exceeds the existing task-oriented dialogue corpora in scale, while also highlighting the challenges associated with building large-scale virtual assistants. It provides a challenging testbed for a number of tasks including language understanding, slot filling, dialogue state tracking and response generation. Along the same lines, we present a schema-guided paradigm for task-oriented dialogue, in which predictions are made over a dynamic set of intents and slots, provided as input, using their natural language descriptions. This allows a single dialogue system to easily support a large number of services and facilitates simple integration of new services without requiring additional training data. Building upon the proposed paradigm, we release a model for dialogue state tracking capable of zero-shot generalization to new APIs, while remaining competitive in the regular setting.
1 Introduction
Virtual assistants must scale across an expanding set of services and domains, but existing dialogue datasets underrepresent this setting. SGD addresses the gap with a large, schema-guided benchmark and a unified model designed to generalize across services and APIs.
- Motivation: Virtual assistants increasingly support many web services across expanding application domains.Frameworks such as Actions on Google and Alexa Skills make adding new services easier, increasing the number assistants must support.
- Limitations of Existing Datasets: Existing multi-domain dialogue datasets cover few domains and typically assume one static API per domain.Real assistants instead encounter multiple services with overlapping functionality and heterogeneous interfaces.
- SGD Dataset: SGD contains over 16000 training dialogues spanning 26 services across 16 domains, with unseen services and domains in evaluation.It supports evaluation of intent prediction, slot filling, dialogue state tracking, and language generation.
- Schema-Guided Modeling: The schema-guided paradigm uses each service schema as input so one dialogue model predicts over dynamic intents and slots.The paradigm is intended to share knowledge across services and support unseen services and APIs.
- Schema-Guided Modeling: A proposed dialogue state tracker generalizes to unseen services and remains competitive on original and updated MultiWOZ datasets.The model uses large pre-trained models such as BERT and is designed to remain robust to API changes.
2 Related Work
Related work progressed from single-domain task-oriented datasets to increasingly complex multi-domain corpora and models. Recent approaches address zero-shot transfer, domain adaptation, and dynamic handling of slot values.
- Dataset Development: Early task-oriented dialogue datasets were limited to a single domain, exemplified by ATIS for flight-related language understanding.Later challenges and datasets increased the complexity of dialogue modeling.
- Data Collection: Wizard-of-Oz collection uses two crowd workers to act as user and system while the system queries a database according to user preferences.WOZ2.0, FRAMES, and MultiWOZ used this setup.
- Data Collection: Simulation-based collection generates complete dialogue flows through simulated user and system roles, followed by crowd paraphrasing.The framework can reduce annotation complexity and cost because annotations are generated automatically.
- Scalable Modeling: Recent research explores zero-shot modeling, domain adaptation, and transfer learning as virtual assistants expand across diverse domains.Deep-learning approaches have achieved state-of-the-art performance on dialogue state tracking tasks.
- Scalable Modeling: Dynamic-vocabulary approaches represent slot values without restricting models to a fixed value vocabulary.Such methods have been proposed for dialogue state tracking.
3 The Schema-Guided Dialogue Dataset
The SGD dataset is designed as a scalable benchmark for virtual assistants, combining multiple services, domains, schemas, and unseen APIs with simulated and paraphrased conversations. Its construction preserves structured dialogue annotations while introducing realistic service constraints and natural-language variation.
- Dataset scope: SGD exceeds other public datasets in scale, with more domains, slots, slot values, and multiple services per domain.Its evaluation sets include services and slots absent from training to test performance on unseen services.
- Dataset construction: The dataset contains synthetic implementations of 45 services or APIs across 20 domains, with simulator-generated dialogue outlines paraphrased by crowd workers.The paraphrasing procedure preserves simulator annotations without requiring additional annotations after collection.
- Services and schemas: Schemas represent services through intents, slots, and constraints, while service data are implemented with a SQL engine and populated primarily from Freebase entities.Some values, including event dates, times, and ticket counts, are synthetically sampled when unavailable in Freebase.
- Services and schemas: Non-categorical slots omit exhaustive value lists, and evaluation includes many values unseen during training; categorical values may differ across services.This design covers open-ended values such as dates and names while preserving service-specific categorical vocabularies.
- Dialogue simulation: Supported service calls are encoded as intents with required slots, preventing the simulator from invoking combinations unsupported by real APIs.The simulator uses user and system agents, dialogue acts, probabilistic trajectories, and scenarios containing up to five intents.
- Dialogue paraphrasing: The paraphrasing pipeline converts canonical simulator actions into utterances using templates, value variations, and one crowd worker’s dialogue-level paraphrasing.Variation keeps substituted slot values consistent across user turns while crowd workers improve naturalness and coherence.
- Dataset analysis: The released dataset contains over 16000 annotated training dialogues with active intents, dialogue states, and system actions for benchmarking several dialogue tasks.Single-domain dialogues average 15.3 turns, while multi-domain dialogues average 23 turns.
4 The Schema-Guided Approach
The schema-guided approach uses each service’s schema as input to a single dialogue model, enabling predictions over service-specific intents and slots. Semantic schema representations support transfer across related services and interaction with unseen APIs.
- Schema-Guided Approach: The schema-guided approach gives each service a schema listing supported intents and slots with natural-language descriptions.The assistant conditions predictions on these service-provided schema elements rather than a single unified schema.
- Schema-Guided Approach: A single unified model makes predictions over the dynamic set of intents and slots specified by the input schema.The model contains no domain- or service-specific parameters.
- Schema-Guided Approach: Semantic representations of schema elements facilitate knowledge transfer across related services.The approach relates common concepts across services while avoiding the need for one schema covering all use cases.
- Schema-Guided Approach: The model can interface with unseen services or APIs and accommodate additions of new intents or slots.These capabilities follow from using semantic representations of schema elements as model input.
5 Zero-Shot Dialogue State Tracking
The paper presents a prototype zero-shot schema-guided dialogue state tracker that conditions one shared model on service schemas and utterances. It predicts active intents, requested slots, and user-goal updates, including categorical values and spans for non-categorical slots.
- Model Architecture: A single model shared across services and domains predicts over dynamic schema elements conditioned on their embeddings.The model encodes intents, slots, and categorical slot values, allowing schemas with different numbers of elements.
- Model Architecture: Pretrained BERT encodes schema-element sequence pairs and user-system utterance pairs into representations used for dialogue state tracking.The schema embedding uses BERT’s uCLS output, while utterance encoding provides sequence-pair and token-level representations.
- Active Intent: Active-intent prediction normalizes logits over service intents and a special “NONE” intent with softmax, then selects the highest-probability intent.The “NONE” intent represents cases where no service intent is currently being processed.
- Requested Slots: Requested-slot prediction applies sigmoid scores to slot logits and marks every slot with score > 0.5 as requested.The requested-slot projection produces one logit for each slot.
- User Goal: User-goal tracking predicts turn-level updates using slot status first, then categorical values or token-span boundaries when needed.Slot status distinguishes none, dontcare, and active; categorical values use softmax, while non-categorical values use start and end token distributions.
1. Active Intent
The model performs well on intent and requested-slot prediction across seen and unseen services, while goal accuracy is stronger for seen services and varies with domain-specific OOV rates and slot ambiguity.
- The model achieves a joint goal accuracy of 0.489 on MultiWOZ 2.1, exceeding the reported best-known result of 0.456.These experiments use exact matching rather than fuzzy matching for comparability.
- The model performs well for Active Intent Accuracy and Requested Slots F1 across both seen and unseen services.
- Joint goal and average goal accuracy are higher on seen services than on unseen services.The reported explanation is a significantly higher OOV rate for slot values in unseen services.
- Domain performance is generally better when test services were seen during training.Rental-Cars and Buses have high slot-value OOV rates, while RideSharing has many possible slot values.
- Categorical slots with similar values, such as “Psychologist” and “Psychiatrist,” produce weaker performance because the model has little signal to distinguish classes.
6 Discussion
The paper argues that simulation-based collection is preferable for this dataset because it reduces annotation burden and cost while supporting broad, targeted coverage, though naturalness was guided rather than directly compared.
- Simulation-based collection avoids skilled, domain-specific annotation tasks and makes collecting diverse datasets less time-consuming.
- Automatically generated annotations make annotation errors rare compared with Wizard-of-Oz collection, where 40% of MultiWOZ 2.0 turns reportedly contained errors.
- The simple paraphrasing task and lack of annotation work greatly reduce data collection costs.
- Simulation supports varied dialogue flows and targeted use cases, providing better coverage for a large dataset.
- Naturalness was guided by conversational flows from MultiWOZ 2.0 and WOZ2.0, but no side-by-side comparison with existing datasets was conducted.The authors identify such a comparison as future work.
7 Conclusions
The paper presents SGD and a schema-guided paradigm for scalable virtual assistants, with a dialogue-state-tracking model that achieves competitive zero-shot results.
- The SGD dataset is intended to encourage scalable modeling approaches for virtual assistants.
- The schema-guided paradigm simplifies integrating new services and APIs with large-scale virtual assistants.
- A dialogue-state-tracking model built on this paradigm achieves competitive results in zero-shot settings.
A Appendix
The appendix provides example dialogues illustrating single-domain, multi-domain, and paraphrasing scenarios across the dataset’s service domains.
- The appendix includes a single-domain dialogue example from the Events domain.
- The Services example includes searching, rejecting an option, requesting contact information, and confirming an Oakland salon.
- A second single-domain example illustrates a Services-domain dialogue about finding and booking a salon appointment.
- A multi-domain example combines Movies, Restaurants, and RideSharing domains.
- Another multi-domain example combines Buses, RentalCars, and Media domains.
- The paraphrasing interface requires workers to preserve all highlighted non-categorical slot values in the conversational dialogue.