Source-linked AI summary
TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking
Michael Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, Milica Gašić
TL;DR
Fixed candidate lists and extractive spans are limited for scalable, multi-domain, open-vocabulary dialog state tracking. TripPy combines span prediction with system-inform and dialogue-state copying, achieving state-of-the-art performance across four datasets, including 55.3% JGA on MultiWOZ 2.1. The approach handles unseen values without a predefined vocabulary, although its tracking remains limited to ontology-predefined slots.
Problem
Multi-domain and open-vocabulary DST requires scalable tracking, while fixed ontologies cannot reliably accommodate unseen values and changing vocabularies.
Method
TripPy fills slots through user-span extraction, system-inform memory copying, and dialogue-state copying, selecting the mechanism for each slot and turn.
Results
TripPy achieves state-of-the-art performance on all four evaluated datasets, including 55.3% JGA on MultiWOZ 2.1.
Takeaways & Limitations
The value-agnostic copy mechanisms handle unseen values without a predefined vocabulary and support challenging DST scenarios.
Takeaways & Limitations
TripPy’s tracking abilities remain limited to slots predefined in the ontology.
Abstract
from arXiv · showhide
Task-oriented dialog systems rely on dialog state tracking (DST) to monitor the user's goal during the course of an interaction. Multi-domain and open-vocabulary settings complicate the task considerably and demand scalable solutions. In this paper we present a new approach to DST which makes use of various copy mechanisms to fill slots with values. Our model has no need to maintain a list of candidate values. Instead, all values are extracted from the dialog context on-the-fly. A slot is filled by one of three copy mechanisms: (1) Span prediction may extract values directly from the user input; (2) a value may be copied from a system inform memory that keeps track of the system's inform operations; (3) a value may be copied over from a different slot that is already contained in the dialog state to resolve coreferences within and across domains. Our approach combines the advantages of span-based slot filling methods with memory methods to avoid the use of value picklists altogether. We argue that our strategy simplifies the DST task while at the same time achieving state of the art performance on various popular evaluation sets including Multiwoz 2.1, where we achieve a joint goal accuracy beyond 55%.
1 Introduction
Dialog state tracking must update users’ goals accurately, but fixed ontologies struggle with changing, open-vocabulary, multi-domain conversations. TripPy addresses this with three context-based copy mechanisms that avoid predefined value lists.
- Motivation: DST extracts domain, slot, and value information needed to maintain a belief over the dialog state.Accurate tracking matters because the next system action depends on the current conversational state.
- Limitations of Existing DST: Fixed-ontology methods cannot capture unseen slot values, scale efficiently with ontology size, or accommodate constantly changing real-world ontologies.These limitations become more significant on complex tasks.
- Limitations of Existing DST: Span-based methods support open-vocabulary extraction but struggle with values expressed implicitly, through coreference, or through value sharing.They often rely on picklists to handle values that are not explicit token sequences.
- Contribution: TripPy fills slots by extracting user-mentioned spans and copying values from system-inform and dialogue-state memories.The memories are created and maintained on-the-fly.
- Contribution: TripPy reports strong generalization to out-of-vocabulary and rare values during testing.The paper also analyzes the roles of the individual model components.
2 Related Work
Prior DST methods trade off value independence against robustness: span methods handle open vocabulary but miss implicit references, while picklists and generation introduce other constraints. TripPy instead creates candidate values from dialog context through memory mechanisms.
- Challenges: Complex multi-domain DST requires long, diverse conversations with many slots and largely unrestricted vocabulary, challenging candidate-list-based systems.Traditional approaches face scalability and generalization concerns in these settings.
- Existing Approaches: Span-based approaches extract values from context but remain limited when values are implicit or absent verbatim from the input.Contextual encoders can mitigate some expression variation through rephrasing generalization.
- Existing Approaches: Generative approaches combine copying with vocabulary distributions but can produce invalid values through repetitions or omissions.Examples include TRADE, MA-DST, and SOM-DST.
- Existing Approaches: Hybrid span-and-picklist systems alleviate some span limitations but still depend partly on predefined candidate values.DS-DST uses picklists for a subset of slots.
- TripPy’s Position: TripPy creates value candidates on-the-fly from dialog context, avoids predefined picklists, and selects among copy strategies per slot and turn.Its slot-value copying also supports value sharing across slots and domains.
3 TripPy: Triple Copy Strategy for DST
TripPy tracks each domain-slot pair from the current turn and dialog history, then fills values using span prediction or memory-based copying. A BERT context encoder supplies representations to specialized DST components.
- Task Formulation: Given a sequence of user and preceding system turns, TripPy detects active domain-slot pairs, predicts their values, and tracks the dialog state over time.The model processes each turn t across the dialog length T.
- Triple Copy Strategy: TripPy uses three copy cases: user spans, system-expressed values referenced by the user, and values previously assigned to another domain-slot pair.These mechanisms target explicit mentions, confirmation or rejection, and coreference.
- Context Encoder: BERT encodes the current user turn, preceding system turn, and dialog history into contextual representations.The history contains earlier user-system turn pairs before the current turn.
- Context Encoder: The aggregate sequence representation feeds slot gates, while token-level representations feed span predictors.Both representation types support downstream DST classification tasks.
3.2 Slot Gates
TripPy assigns every domain-slot pair a gate that chooses whether to leave it empty, mark it as unconstrained, or fill it through one of three copy mechanisms. Boolean slots use a separate gate.
- Gate Classes: Each domain-slot gate assigns one of none, dontcare, span, inform, or refer at every turn.The first two labels represent special cases; the remaining labels select copy mechanisms.
- Copy Decisions: span extracts a value from the current user utterance, inform copies a system-mentioned value, and refer copies a value already present in the dialog state.The three labels correspond directly to the model’s copy mechanisms.
- Gate Architecture: The slot-gate input is the contextual [CLS] representation and each gate is implemented as a trainable linear classification head.The gate predicts a probability distribution over its class set.
- Boolean Slots: Boolean slots use the separate class set none, dontcare, true, false.Their gate predicts binary slot states rather than copied values.
3.3 Span-based Value Prediction
For each slot, TripPy uses a domain-slot-specific span predictor over the entire dialog context to identify value boundaries. Parallel softmax layers predict start and end positions, with invalid ends producing an empty span.
- Each slot-specific span predictor takes token representations of the entire dialog context as input.
- A trainable linear BERT classification head feeds two parallel softmax layers that predict span start and end positions.
- Erroneously predicted span ends receive no special handling and simply yield an empty span.
3.4 System Inform Memory for Value Prediction
TripPy maintains a system inform memory containing values previously informed by the system. When users refer to such values without expressing them explicitly, the model copies them into the dialog state.
- The system inform memory tracks all slot values informed by the system during the current dialog turn.
- If a user positively refers to an informed value without expressing it, the corresponding slot is filled by copying that value into the dialog state.
- For the restaurant-name slot, the slot gate predicts inform when the user refers to a system-mentioned value.
3.5 DS Memory for Coreference Resolution
TripPy uses the dialog state as a memory for resolving coreferences, allowing a slot to copy a value already assigned to another slot. Auxiliary features indicate recently informed and previously filled slots.
- Coreferences become more likely in complex dialogs and may link values across domains and multiple turns.
- The DS memory resolves coreferences by selecting the slot that already contains the value referred to by the user.
- TripPy adds binary features indicating whether slots were recently informed or already filled during the dialog.
- These auxiliary vectors are fused late with BERT outputs to determine slot-gate probabilities.
3.7 Partial Masking
TripPy partially masks values in past system utterances with BERT’s [UNK] token so the model focuses on historical context rather than memorized values. The model then updates slots only when a non-none value is detected.
- Partial masking replaces values only in past system utterances with BERT’s generic [UNK] token.
- The masking is intended to make representations focus on historical context rather than the sighting of specific values.
- A rule-based update mechanism changes a slot when a value other than none is detected.
- If a slot-value is predicted as none, the slot remains unchanged.
4 Experimental Setup
The experiments evaluate the model across four datasets, emphasizing challenging multi-domain and open-vocabulary settings while using consistent joint goal accuracy and implementation choices.
- Four datasets are used: MultiWOZ 2.1, WOZ 2.0, sim-M, and sim-R.
- MultiWOZ 2.1 is the most challenging benchmark, containing over 10,000 multi-domain dialogs across five domains and 30 domain-slot pairs.
- The smaller single-domain datasets provide sanity checks, while sim-M and sim-R test value-independent DST under high out-of-vocabulary rates.
- Joint goal accuracy measures the ratio of dialog turns where every slot has the correct ground-truth value, with absent values requiring no prediction.
- Results are averaged over three random seeds, and MultiWOZ evaluation accepts identified valid label variants such as “centre” and “center.”
- The model uses BERT-base-uncased as its context encoder and trains with a joint loss combining gate, span, and refer components.
5 Experimental Results
TripPy achieves state-of-the-art performance across all four evaluated datasets, including 55.3% JGA on MultiWOZ 2.1. Ablations and generalization analyses show that triple-copy mechanisms, dialog history, auxiliary memories, and sufficient training exposure support performance on complex and unseen values.
- 55.3% JGA on MultiWOZ 2.1 is 2.0% absolute above the previous top-scoring method.TripPy achieves state-of-the-art performance on all four evaluated datasets.
- Ablation studies: The triple-copy mechanism pushes performance close to 50%, surpassing TRADE and approaching leading hybrid methods.Copying between slots also resolves coreferences that span prediction alone cannot handle.
- Ablation studies: Dialog history improves slot-gate performance by reducing contextual ambiguity and enables TripPy to outperform DS-DST while matching SOM-DST.It especially benefits dontcare and boolean values and improves assignment of values to the correct slot.
- Ablation studies: Adding memory-based auxiliary features surpasses SOM-DST and DST-picklist despite tracking only informed-slot and previously seen-slot identities.These features improve slot-gate and referral-gate classification.
- Generalization study: TripPy detects OOV values as well as less common values, with performance benefiting substantially from roughly 50 training samples per value.The span-prediction baseline struggles more with OOV and rare values and generally needs more samples for good recall.
- Generalization study: After about 84% of over 1,000 unique MultiWOZ test slot-value pairs were replaced with OOV values, most slots maintained high accuracy, although the train domain declined more.The result held across value types such as named entities and times of day.
6 Conclusion
TripPy handles challenging dialog state tracking scenarios using value-agnostic memory-based copy mechanisms, but remains limited to ontology-predefined slots and motivates future schema-guided extensions.
- 6 Conclusion: TripPy handles challenging dialog state tracking scenarios without substantial general-performance impairment from unseen values.The model’s information extraction remains effective even in extreme cases discussed by the paper.
- 6 Conclusion: Its memory-based copy mechanisms are value agnostic and therefore are not limited by a predefined vocabulary.The memories support extracting values without requiring a fixed candidate vocabulary.
- 6 Conclusion: TripPy’s tracking abilities remain limited to slots predefined in the ontology.The authors identify slot independence as a direction for improving the model’s DST capabilities.
- 6 Conclusion: Future extensions aim to adopt a schema-guided paradigm and a more sophisticated update strategy, including partial forgetting.The authors frame these capabilities as directions for future work.