Source-linked AI summary
Towards Zero-Shot Frame Semantic Parsing for Domain Scaling
Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur, Larry Heck
TL;DR
Slot-filling systems need large labeled datasets, making it difficult to bootstrap semantic parsing for new domains. This paper uses natural-language slot descriptions to transfer slot representations across domains. Its concept tagger provides reasonable zero-data bootstrap performance for most domains and performs better than or on par with a multi-task baseline at most learning-curve points.
Problem
Slot-filling models typically require large amounts of labeled data, while bootstrapping a semantic parser for a new domain from its semantic frame remains a challenge.
Source: §1 · p. 1Method
The concept tagger encodes natural-language slot descriptions and combines them with utterance representations to identify slots across domains.
Source: §3.3 · p. 3Results
Takeaways & Limitations
Slot descriptions can support new-domain slot filling without labeled or unlabeled in-domain examples or explicit schema alignment.
Source: §1 · p. 1Performance can suffer when a slot description yields an incompatible representation, as with pickup location, where a competing slot may contribute to degradation.
Source: §5 · p. 4
Abstract
from arXiv · showhide
State-of-the-art slot filling models for goal-oriented human/machine conversational language understanding systems rely on deep learning methods. While multi-task training of such models alleviates the need for large in-domain annotated datasets, bootstrapping a semantic parsing model for a new domain using only the semantic frame, such as the back-end API or knowledge graph schema, is still one of the holy grail tasks of language understanding for dialogue systems. This paper proposes a deep learning based approach that can utilize only the slot description in context without the need for any labeled or unlabeled in-domain examples, to quickly bootstrap a new domain. The main idea of this paper is to leverage the encoding of the slot names and descriptions within a multi-task deep learned slot filling model, to implicitly align slots across domains. The proposed approach is promising for solving the domain scaling problem and eliminating the need for any manually annotated data or explicit schema alignment. Furthermore, our experiments on multiple domains show that this approach results in significantly better slot-filling performance when compared to using only in-domain data, especially in the low data regime.
1. Introduction
Goal-oriented dialogue systems use semantic frames to interpret utterances, but building systems for many domains remains constrained by the need for large labeled datasets. This paper proposes using natural-language slot descriptions to transfer slot-filling capability to new domains without in-domain examples or explicit schema alignment.
1. Introduction
- Dialogue systems must detect an utterance’s domain and intent and identify slot-value tokens, while current models still require massive amounts of labeled data.The paper identifies broad domain coverage as an important challenge for assistants.Source: §1 · p. 1
- Earlier multi-task models shared features across domains [6], and shared slot-filling features improved over single-domain baselines, especially with little data [9].Source: Figure 1 · p. 1
- Natural-language slot descriptions let the model transfer slot-filling capability to new domains without in-domain examples or explicit schema alignment.The approach trains on small in-domain datasets across multiple domains and uses slot encodings to identify slots in new domains.Source: §1 · p. 1
- Table 1 presents sample utterances from each domain.Source: Table 1 · p. 2
2. Slot Filling
Slot filling identifies token spans that instantiate typed slots in a domain’s semantic frame. The paper describes its ten-domain crowd-sourced dataset and the data collection and labeling process.
2. Slot Filling
- Slot filling estimates an IOB tag sequence for the input tokens, marking the spans that instantiate slots in a semantic frame.A frame may include typed slots such as departure city, arrival city, or departure date.Source: §2 · p. 2
- The authors crowd-sourced utterances for ten domains, prompting workers with slot-value combinations generated from the Google knowledge graph.Utterances were automatically labeled after verbatim slot-value matches or sent to a second group of raters for span labeling.Source: §2 · p. 2
- The collected datasets have fewer fine-grained slot types than ATIS, which contains more than a hundred distinct slots.Source: §2 · p. 2
3. Models
The paper compares single-task and multi-task sequence taggers with a concept tagger that incorporates natural-language slot descriptions. The concept tagger uses those descriptions to represent slots across domains.
3. Models
- The experiments compare the three architectures at varying amounts of target-domain training data, using weighted token F1 for evaluation.The evaluation averages token F1 across slots in the target domain.Source: §3 · p. 2
3.1. Baseline single task model
- The single-task baseline uses a bidirectional LSTM followed by a token-level softmax layer to predict IOB labels.Source: §3.1 · p. 2
3.2. Multi-task model
- The multi-task model shares two bidirectional LSTM layers across domains and uses domain-specific softmax layers.It trains on alternating domain batches, with batch representation proportional to the logarithm of each domain’s training-set size.Source: §3.2 · pp. 2–3
3.3. Zero-Shot Concept Tagging Model
- The concept tagger encodes each slot description and combines that representation with contextual utterance tokens to predict slot tags.Slot-description embeddings are averaged in these experiments; predictions are made independently for each slot, using positive and negative examples.Source: §3.3 · p. 3
4. Experiments and Results
The concept tagger provides reasonable zero-data bootstrap performance for most domains and performs better than or on par with the multi-task baseline at most learning-curve points. Experiments vary target-domain data while using out-of-domain data.
4. Experiments and Results
- The learning-curve evaluation uses all available out-of-domain data while varying the amount of target-domain training data.Performance is averaged over ten runs with training samples drawn from different parts of each domain dataset.Source: §4 · p. 3
- Hyperparameters are tuned using dev-set performance averaged across domains, weighted by the logarithm of each domain’s training-sample count.Each domain’s dev set contains 20% of the training examples for that data point.Source: §4 · p. 3
- CT provides reasonable zero-data bootstrap performance for most domains, beats ST by significant margins, and beats or matches MT at most learning-curve points.Source: Table 3 · p. 4, §4 · p. 4
5. Discussion
The concept tagger often matches or exceeds the multi-task model when slots share semantics across domains, but performance depends on slot descriptions and can lag for domain-specific or poorly represented slots.
5. Discussion
- Shared semantics let the concept tagger match or outperform the multi-task model on most slots, with gains on slots related to slots in other domains.Domain-specific slots such as bus-ticket discount type usually need more training samples to reach the same performance, which the authors attribute to a lack of slot-specific parameters.Source: §5 · p. 4
- Appointment time reaches within 10% of peak performance without in-domain training, while # seniors ramps up with little data despite the competing # adults slot.The authors attribute appointment-time performance to shared features from departure time, reservation time, and related slots in other domains.Source: §5 · p. 4
- Pickup location performs worse than the multi-task baseline, possibly because its description is poorly represented and competes with dropoff location.The authors identify susceptibility to incomplete or misleading slot descriptions and weak description embeddings; they suggest fine-tuning slot representations with small amounts of in-domain data as a possible remedy.Source: §5 · p. 4
- The approach’s appeal is its simplicity and minimal supervision, with possible extensions including entity linking, knowledge bases, and domain-adversarial use of unlabeled target-domain data.The authors propose a plug-and-play system with reduced need for expensive labeled data for each additional domain.Source: §5 · p. 4
6. Conclusions
The paper proposes concept tagging, which shares feature extraction and slot representations across domains through natural-language slot descriptions. Across ten domains, it outperforms a strong multi-task baseline, especially with little training data, and shows potential for new-domain slot filling without in-domain training or explicit schema alignment.
6. Conclusions
- Across ten domains, concept tagging performs significantly better than a strong multi-task baseline, especially in the low-data regime.The authors report that it can leverage shared features and slot semantics from other domains, with potential for reasonable performance on new-domain slots without in-domain training or explicit schema alignment.Source: §6 · p. 4