PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Dialogue State Tracking by Selectively Overwriting Memory

Sungdong Kim, Sohee Yang, Gyuwan Kim, Sang-Woo Lee

arXiv:1911.03906v2cs.CL

TL;DR

Open vocabulary-based DST improves scalability and handling of unseen values, but many methods recompute the dialogue state from scratch at every turn. SOM-DST treats the state as selectively overwritable memory, separating operation prediction from value generation. It achieves state-of-the-art joint goal accuracy on MultiWOZ 2.0 and 2.1, while analysis identifies state operation prediction as a key improvement direction.

  • Problem

    Open vocabulary-based DST addresses ontology scalability and unseen slot values, but many methods inefficiently predict the dialogue state from scratch at every turn.

  • Method

    SOM-DST treats dialogue state as explicit memory, predicts state operations for slots, and generates new values only for slots selected for updating.

  • Results

    SOM-DST achieves state-of-the-art joint goal accuracy on MultiWOZ 2.0 and MultiWOZ 2.1 in an open vocabulary-based setting.

  • Takeaways & Limitations

    Selective overwriting makes DST generation efficient, and improving state operation prediction may substantially increase overall DST performance.

  • Takeaways & Limitations

    A large share of errors comes from the state operation predictor, with class imbalance identified as one possible improvement target.

Abstract

from arXiv · show

Recent works in dialogue state tracking (DST) focus on an open vocabulary-based setting to resolve scalability and generalization issues of the predefined ontology-based approaches. However, they are inefficient in that they predict the dialogue state at every turn from scratch. Here, we consider dialogue state as an explicit fixed-sized memory and propose a selectively overwriting mechanism for more efficient DST. This mechanism consists of two steps: (1) predicting state operation on each of the memory slots, and (2) overwriting the memory with new values, of which only a few are generated according to the predicted state operations. Our method decomposes DST into two sub-tasks and guides the decoder to focus only on one of the tasks, thus reducing the burden of the decoder. This enhances the effectiveness of training and DST performance. Our SOM-DST (Selectively Overwriting Memory for Dialogue State Tracking) model achieves state-of-the-art joint goal accuracy with 51.72% in MultiWOZ 2.0 and 53.01% in MultiWOZ 2.1 in an open vocabulary-based DST setting. In addition, we analyze the accuracy gaps between the current and the ground truth-given situations and suggest that it is a promising direction to improve state operation prediction to boost the DST performance.

1 Introduction

SOM-DST addresses inefficiency in open vocabulary-based dialogue state tracking by treating the dialogue state as selectively overwritable memory. It separates operation prediction from generating only the slot values that need updating, achieving state-of-the-art performance on MultiWOZ benchmarks.

  • Motivation: Open vocabulary-based DST handles unseen slot values more robustly, but many methods predict the entire dialogue state from scratch at every turn.Predefined ontology-based DST also requires advance candidate inventories and does not scale well to large ontologies.
  • Method: SOM-DST represents dialogue state as memory and predicts an operation for each slot before generating values for only a minimal subset.The two sub-tasks are state operation prediction and slot value generation.
  • Method: SOM-DST separates slot selection from value generation, allowing the encoder to select slots while the decoder focuses on generating their values.This decomposition is intended to reduce decoder difficulty in open-vocabulary DST.
  • Results: SOM-DST achieves state-of-the-art joint goal accuracy in open vocabulary-based DST on MultiWOZ 2.0 and MultiWOZ 2.1.The contribution statement reports state-of-the-art performance on both datasets.
  • Analysis: Error analysis identifies state operation prediction as a promising direction for improving final DST accuracy.The authors report that improving this component may substantially increase overall performance.
  • Contribution: The method enables efficient DST by using the previous dialogue state and generating values only for slots that require changes.This selectively overwriting mechanism is the paper’s central efficiency contribution.

2 Previous Open Vocabulary-based DST

Prior open vocabulary-based DST methods improve scalability and handling of unseen values, but differ in how they generate or extract slot values and how efficiently they use dialogue context. SOM-DST instead uses previous dialogue state memory with discrete operations and selective value generation.

  • Generation Efficiency: Wu et al. perform autoregressive generation for all slots at every turn, making their encoder-decoder method computationally inefficient.The inefficiency comes from generating every slot value regardless of whether it changes.
  • Generation Efficiency: Ren et al. use a hierarchical decoder to address generating all slot values, and introduce Inference Time Complexity based on slot and value counts.Their efficiency measure is calculated using the number of slots J and corresponding slot values M.
  • Generation Efficiency: Le et al. use a non-autoregressive decoder to generate current-state slot values simultaneously, but do not use the previous turn dialogue state as input.Their approach encodes dialogue and delexicalized dialogue context together.
  • Related Approaches: Le et al. process context at domain and slot levels and report a performance gain from joint response-generation training, while still generating every slot value each turn.The method retains the efficiency limitation of full-slot generation.
  • Discrete Operations: Gao et al. formulate DST as reading comprehension and use a slot carryover module to decide whether to retain a slot value from the previous state.SOM-DST’s discrete operations are inspired by this carryover idea.
  • Comparison: SOM-DST achieves state-of-the-art performance in an open vocabulary-based setting, unlike a picklist-based result that remains constrained by a predefined ontology.The comparison highlights the distinction between benchmark performance and ontology-free scalability.

3 Selectively Overwriting Memory for Dialogue State Tracking

SOM-DST represents dialogue state as fixed-sized slot-value memory and selectively overwrites it through operation prediction and targeted value generation. Its components use prior state and dialogue context to update only slots requiring new values.

  • Dialogue State: Dialogue state is represented as fixed-sized memory containing slot keys and corresponding values at each turn.The number of slots is denoted J, and each slot combines a domain name with a slot name.
  • Operation: Each slot receives one of four operations: CARRYOVER, DELETE, DONTCARE, or UPDATE.CARRYOVER preserves the previous value; the other operations change the slot value.
  • Operation: DELETE and DONTCARE assign special values, while UPDATE generates a new value different from the previous value and special values.DELETE sets a slot to NULL, and DONTCARE sets it to DONTCARE; UPDATE invokes the slot value generator.
  • State Operation Prediction: The state operation predictor classifies operations for each slot, and the slot value generator runs only for slots predicted as UPDATE.The two components are jointly trained to predict the current-turn dialogue state.
  • Input Representation: The model uses previous dialogue utterances, current utterances, and the previous dialogue state as input.The previous state provides an explicit compact representation of dialogue history under a Markov assumption.
  • Slot Value Generation: Because the generator produces values for J′_t slots rather than all J slots, and J′_t is usually much smaller than J, computation is more efficient.A GRU decoder generates values autoregressively for only the selected slots.
  • Objective Function: Training jointly optimizes operation classification, domain classification, and slot value generation with a summed joint loss.The loss is defined as L_joint,t = L_opr,t + L_dom,t + L_svg,t.

4 Experimental Setup

The experiments use MultiWOZ 2.0 and 2.1, five domains, and 30 slots, comparing SOM-DST with ontology-based and open-vocabulary baselines.

  • Datasets: Experiments use MultiWOZ 2.0 and MultiWOZ 2.1, large multi-domain datasets with about 10,000 dialogues across seven domains.The experiments exclude hospital and police, which do not appear in the test set.
  • Datasets: The experimental setting retains five domains—restaurant, train, hotel, taxi, and attraction—and 30 slots.These choices follow Wu et al. (2019).
  • Training: The state operation predictor uses pretrained BERT-base-uncased, while the slot value generator uses a GRU decoder with greedy decoding.The encoder and decoder use separate learning-rate schemes because the decoder is trained from scratch.
  • Training: The two components are trained jointly for 30 epochs, using ground-truth state operations and the previous-turn dialogue state during training.Slot order is randomly shuffled with rate 0.5 when dialogue state is provided.
  • Training: Reported SOM-DST results are averaged over ten runs.Experiments were performed on the NAVER Smart Machine Learning platform.
  • Baseline Models: SOM-DST is compared with predefined ontology-based, open-vocabulary, and hybrid DST models.The comparison includes models such as FJST, HJST, SUMBT, DST Reader, TRADE, COMER, NADST, ML-BST, DS-DST, and HyST.

5 Experimental Results

SOM-DST achieves state-of-the-art open-vocabulary DST performance, with particularly strong results in taxi and train domains and robustness on challenging conversations.

  • Joint Goal Accuracy: SOM-DST achieves state-of-the-art joint goal accuracy in the open-vocabulary setting on MultiWOZ 2.0 and 2.1.Joint goal accuracy requires every predicted slot value at a turn to exactly match the ground truth.
  • Joint Goal Accuracy: MultiWOZ 2.1 changes about 32% of MultiWOZ 2.0 state annotations through correction of annotation errors.The authors suggest SOM-DST benefits from these corrected state annotations because it explicitly uses dialogue state labels as input.
  • Domain-Specific Accuracy: SOM-DST outperforms other models in the taxi and train domains, while matching or slightly trailing them in other domains.These two domains contribute mainly to the model’s state-of-the-art joint goal accuracy.
  • Domain-Specific Accuracy: Taxi and train conversations contain more diverse slot values and more than one domain change, making them challenging dialogue settings.Among dialogues with more than one domain switch, those ending in taxi occur ten times more often than other cases.
  • Domain-Specific Accuracy: The authors assume SOM-DST is relatively more robust in challenging conversations because it explicitly utilizes the previous-turn dialogue state.The memory representation can preserve information mentioned near the beginning of a conversation and copy values when needed.

6 Analysis

Analysis shows that state operation prediction is the main source of DST error, while selective overwriting substantially reduces the number of generated slot values and inference time.

  • State Operation Analysis: Two-way, three-way, and six-way state operation variants all reduce joint goal accuracy relative to the four-way formulation.The two-way variant merges DELETE, DONTCARE, and UPDATE into a generate-new-value class, although these operations differ in how values appear.
  • Error Analysis: 92.85% of the 46.99% absolute error under error propagation is attributed to the state operation predictor, compared with 21.6% to slot value generation.The corresponding ranges indicate that at least 78.4%–92.85% comes from state operation prediction and 7.15%–21.6% from generation.
  • Error Analysis: 90.53% of the 19% absolute error without error propagation is attributed to the state operation predictor, compared with 19.63% to slot value generation.The corresponding ranges indicate that at least 80.37%–90.53% comes from state operation prediction and 9.47%–19.63% from generation.
  • Error Analysis: Error propagation increases the error 2.47 times when the predicted previous-turn dialogue state is used instead of ground truth.State operation prediction remains the larger error source both with and without error propagation.
  • Error Analysis: The analysis identifies state operation prediction as the main component with room for improvement and suggests class-imbalance mitigation as one possible direction.The authors state that improving this component may increase overall DST performance by a large margin.
  • Generation Efficiency: SOM-DST generates values for at most 9 slots and 1.14 slots on average per turn in MultiWOZ 2.1.TRADE and ML-BST generate values for all 30 slots per turn, while COMER generates up to 18 and 5.72 on average.
  • Inference Efficiency: SOM-DST inference is about 12.5 times faster than TRADE and has latency compatible with NADST.The comparison uses inference time per MultiWOZ 2.1 dialogue turn on Tesla V100 with batch size 1.

7 Conclusion

SOM-DST treats dialogue state as selectively overwritable explicit memory and decomposes tracking into operation prediction and slot value generation. This makes generation efficient and achieves state-of-the-art joint goal accuracy in open vocabulary-based DST.

  • SOM-DST regards dialogue state as an explicit memory that can be selectively overwritten.
  • The model decomposes dialogue state tracking into state operation prediction and slot value generation.
  • Generating values for only a minimal subset of slots makes the tracking process efficient at each dialogue turn.
  • SOM-DST achieves state-of-the-art joint goal accuracy on MultiWOZ 2.0 and MultiWOZ 2.1 in an open vocabulary-based setting.
  • Further analysis identifies improving state operation prediction as a promising direction for increasing overall DST performance.

A Data Statistics

The appendix reports dataset, vocabulary, and domain-transition statistics for MultiWOZ 2.1. It specifically identifies dialogues with multiple domain transitions, including those ending in the taxi domain.

  • Data Statistics: Table 8 reports data statistics for MultiWOZ 2.1.
  • Data Statistics: Table 9 reports statistics for the slot value vocabulary size in MultiWOZ 2.1.
  • Data Statistics: 175 test-set dialogues have more than one domain transition and are treated as complicated dialogues.
  • Data Statistics: 140 dialogues have more than one domain transition and end with the taxi domain.
  • Data Statistics: Table 10 bolds cases where the domain switches more than once and the dialogue ends in taxi.

B Inference Time Complexity (ITC)

The paper compares inference-time complexity using the number of slots and values per slot, reporting best- and worst-case complexity. SOM-DST can achieve constant best-case complexity when no slot is updated.

  • Inference Time Complexity (ITC): Inference Time Complexity uses J for the number of slots and M for the number of values of a slot.
  • Inference Time Complexity (ITC): The reported ITC comparison includes both best-case and worst-case complexity.
  • Inference Time Complexity (ITC): SOM-DST generates values only for slots whose predicted operation is UPDATE.
  • Inference Time Complexity (ITC): Ω(1) is SOM-DST’s best-case complexity when no slot requires an UPDATE operation.

C Sample Outputs

Figures 3 and 4 show SOM-DST outputs for two MultiWOZ 2.1 test-set dialogues. Blue marks the parts changed from the previous dialogue state, while NULL-valued slots are omitted.

  • Sample Outputs: Figure 3 presents SOM-DST output for test-set dialogue MUL2499 from MultiWOZ 2.1.
  • Sample Outputs: In both figures, blue marks the parts changed from the previous dialogue state.
  • Sample Outputs: Slots with NULL values are omitted from both figures to save space.
  • Sample Outputs: Figure 4 presents SOM-DST output for test-set dialogue PMUL3748 from MultiWOZ 2.1.
Loading 1911.03906v2…