Source-linked AI summary
MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation
Fanghua Ye, Jarana Manotumruksa, Emine Yilmaz
TL;DR
MultiWOZ 2.1 still contains incorrect and inconsistent state annotations, making model performance difficult to assess fairly. MultiWOZ 2.4 refines its validation and test annotations while retaining the training annotations, and all eight benchmarked dialogue state tracking models perform much better than on MultiWOZ 2.1.
Problem
Incorrect and inconsistent MultiWOZ 2.1 state annotations, especially in validation and test sets, make dialogue state tracking performance difficult to assess fairly.
Source: §1 · p. 1Method
Results
All eight benchmark models demonstrate much better performance on MultiWOZ 2.4 than on MultiWOZ 2.1.
Source: §6 · p. 5Takeaways & Limitations
The benchmark is intended to facilitate future research on dialogue state tracking.
Source: §6 · p. 5The authors say it is unlikely that they corrected all annotation errors in the validation and test sets.
Source: §5 · p. 5
Abstract
from arXiv · showhide
The MultiWOZ 2.0 dataset has greatly stimulated the research of task-oriented dialogue systems. However, its state annotations contain substantial noise, which hinders a proper evaluation of model performance. To address this issue, massive efforts were devoted to correcting the annotations. Three improved versions (i.e., MultiWOZ 2.1-2.3) have then been released. Nonetheless, there are still plenty of incorrect and inconsistent annotations. This work introduces MultiWOZ 2.4, which refines the annotations in the validation set and test set of MultiWOZ 2.1. The annotations in the training set remain unchanged (same as MultiWOZ 2.1) to elicit robust and noise-resilient model training. We benchmark eight state-of-the-art dialogue state tracking models on MultiWOZ 2.4. All of them demonstrate much higher performance than on MultiWOZ 2.1.
1 Introduction
MultiWOZ 2.0 helped advance task-oriented dialogue research, but noise persisted in its state annotations and in later versions, complicating fair model evaluation. MultiWOZ 2.4 refines the validation and test annotations of MultiWOZ 2.1 while leaving training annotations unchanged.
1 Introduction
- MultiWOZ 2.0 is the first large-scale dataset spanning multiple domains and has attracted substantial research attention.The paper situates it among publicly available dialogue datasets associated with advances in task-oriented dialogue research.Source: §1 · p. 1
- MultiWOZ 2.1 still contains incorrect and inconsistent annotations, and its joint goal accuracy benchmark is around 60%.Noise in its validation and test sets makes performance difficult to assess fairly; model-specific preprocessing strategies have also led to unfair comparisons.Source: §1 · p. 1
- MultiWOZ 2.2 misses value variants and introduces inconsistencies, while MultiWOZ 2.3 focuses on dialogue acts without fully resolving state-annotation noise.Source: §1 · p. 1
- MultiWOZ 2.4 corrects validation and test annotations, changing states in more than 41% of turns across over 65% of dialogues.Training annotations remain unchanged, and the paper reports better performance than on earlier versions.Source: §1 · p. 1
- Keeping noisy training annotations unchanged reflects the authors’ interest in robust, noise-resilient training, given the cost and labor of collecting noise-free multi-domain dialogue data.Source: §1 · pp. 1–2
2 Annotation Refinement
The authors refine MultiWOZ 2.1 because its single-value state representation avoids the incompleteness and ambiguity of multiple value variants. Their procedure corrects six annotation error types in validation and test dialogues by checking turn-active slots, and reports broad annotation changes.
2 Annotation Refinement
- MultiWOZ 2.1 is the refinement basis because MultiWOZ 2.2’s multiple value variants are difficult to enumerate, can be ambiguous, and did not clearly improve benchmark performance.MultiWOZ 2.1 had already improved annotation quality over MultiWOZ 2.0.Source: §2 · p. 2
2.1 Annotation Error Types
- The six error types concern context mismatch, missing or unmentioned values, incomplete values, implicit time processing, and unnecessary annotations.For time values, the refinement copies the specified time directly because deciding an exact offset is difficult; unnecessary system-provided values are generally removed, except for name-related requests.Source: §2.1 · pp. 2–3
2.2 Annotation Refinement Procedure
- Annotators checked turn-active slots in 2,000 validation and test dialogues spanning five domains and 30 slots, then accumulated those states across turns.A turn-active slot is determined by the current-turn context rather than inherited from earlier turns; the average counts were 1.16 before and 1.18 after refinement.Source: §2.2 · p. 3
2.3 Statistics on Refined Annotations
- More than 41% of turns across over 65% of dialogues have corrected states; an average of 1.53 slots are rectified at each refined turn.Source: §2.3 · p. 3, Table 2 · p. 3
- Missing Annotation and Not Mentioned are the most frequent error types, while Unnecessary Annotation accounts for more than 10% of errors.Source: §2.3 · p. 3, Figure 2 · p. 3
3 Benchmark Evaluation
All eight benchmarked dialogue state tracking models perform better on MultiWOZ 2.4 than on MultiWOZ 2.1, and SUMBT and TRADE also outperform earlier dataset versions. Results cover joint goal accuracy, slot accuracy, and domain-specific accuracy.
3.1 Benchmark Models
- The benchmark covers eight models, including ontology-based and open-vocabulary approaches.The evaluated models are SUMBT, STAR, TRADE, PIN, SOM-DST, SimpleTOD, SAVN, and TripPy.Source: §3.1 · pp. 3–4
3.2 Benchmark Results
- All models achieve higher joint goal accuracy and slot accuracy on MultiWOZ 2.4 than on MultiWOZ 2.1, and SUMBT and TRADE outperform all previous versions.Table 3 reports joint goal and slot accuracy; Table 4 compares test-set joint goal accuracy across dataset versions.Source: §3.2 · p. 4, Table 3 · p. 4, Table 4 · p. 4
- Joint goal accuracy measures turns with every slot value predicted correctly, while slot accuracy averages accuracy across slots.The evaluation uses joint goal accuracy and slot accuracy.Source: §3.2 · p. 4
- SimpleTOD shows the least improvement, while SAVN also has a low increase.The authors suggest direct state-value generation may explain SimpleTOD’s smaller gain, while SAVN already uses value normalization.Source: §3.2 · p. 4
- Except for SOM-DST in the taxi domain, SOM-DST and STAR perform better in each domain on MultiWOZ 2.4.Table 5 presents domain-specific test-set joint goal accuracy.Source: §3.2 · p. 4, Table 5 · p. 4
4 Human Evaluation
The human evaluation compares refined annotations with MultiWOZ 2.1 on sampled test-set dialogues. Raters generally favored the refined annotations, while reporting high agreement; an ambiguous hotel-type slot illustrates a remaining annotation challenge.
4 Human Evaluation
- Five computer science students rated turns from 50 randomly sampled test-set dialogues by comparing refined annotations with MultiWOZ 2.1.Scores distinguished cases where either version was correct or both were correct or problematic.Source: §4 · p. 4
- The refined annotations received an average score of 0.1653, and raters reached Fleiss’ kappa κ = 0.9226.The authors interpret the average score as indicating more accurate refined annotations and κ as almost perfect agreement.Source: §4 · p. 4
- Around 20% of annotations in MultiWOZ 2.4 were judged more accurate than MultiWOZ 2.1, versus about 1% judged better in MultiWOZ 2.1.In most cases, raters considered both versions correct and no change necessary.Source: §4 · p. 5
- The hotel-type slot is ambiguous because “hotel” can be a candidate value even when the user means a place to stay generally.The refinement procedure considered the whole dialogue session to infer intent, whereas raters tended to consider only dialogue history.Source: §4 · p. 5
5 Caveats and Lessons Learned
The authors note that MultiWOZ 2.4 may still contain annotation errors because some value choices, user intentions, and multi-value cases are difficult to resolve. They argue that clearer slot definitions, candidate values, and annotation instructions can support more consistent labeling.
5 Caveats and Lessons Learned
- Some annotation errors likely remain because the refinement challenges are difficult to overcome.The authors state that they have not corrected all errors.Source: §5 · p. 5
- Confusing candidate values, ambiguous user intentions, and slots with multiple values make consistent state annotation difficult.For example, “does not need internet” can mean either internet=no or internet=dontcare.Source: §5 · p. 5
- When a system lists many possible entities, it can be unclear whether the corresponding slot should be annotated.The paper gives the example of a user asking whether any of ten recommended museums has zero entrance fee.Source: §5 · p. 5
- Turning user utterances into structured dialogue states inevitably loses some information because language and user intentions are complex.The authors describe dialogue state annotation as inherently challenging.Source: §5 · p. 5
- The authors recommend defining unambiguous slots and candidate values and providing annotators with full slot-specific instructions.These measures are intended to facilitate annotation and consistency.Source: §5 · p. 5
6 Conclusion
MultiWOZ 2.4 refines validation- and test-set annotations from MultiWOZ 2.1 while leaving the training annotations unchanged to encourage robust training. In benchmarks of eight state-of-the-art dialogue state tracking models, all performed better on MultiWOZ 2.4 than on MultiWOZ 2.1.
6 Conclusion
- MultiWOZ 2.4 is an updated version of MultiWOZ 2.1 that rectifies (almost) all annotation errors in the validation and test sets, while keeping training annotations unchanged.Source: §6 · p. 5
- All eight benchmarked state-of-the-art dialogue state tracking models performed much better on MultiWOZ 2.4 than on MultiWOZ 2.1.The authors present the benchmark as support for the refined version.Source: §6 · p. 5
- The refined dataset can also be used to train better overall dialogue systems, for example by using its clean validation set to generate high-quality training data through data augmentation.This is presented as a possible application.Source: §6 · p. 5
A Additional Statistics on the Refined Annotations
The additional statistics compare slot-value vocabulary sizes and value-change ratios in the validation and test sets of MultiWOZ 2.1 and 2.4. Changes reflect both normalization or error correction and the addition of labels containing multiple values.
A Additional Statistics on the Refined Annotations
- Slot value vocabulary sizes decrease for some slots after value normalization and error correction, but increase for others when labels containing multiple values are added.Table 6 compares the validation and test sets of MultiWOZ 2.1 and 2.4 and reports value-change ratios.Source: A Additional Statistics on the Refined… · p. 8, Table 6 · p. 8
- Name-related slots have the highest value-change ratios, which the authors link to annotators making incomplete or inconsistent annotations for longer values.The reported change ratio is based on the slots' value vocabularies.Source: A Additional Statistics on the Refined… · p. 8
B Per-Slot (Slot-Specific) Accuracy
The per-slot analysis reports that most slots have higher accuracy on MultiWOZ 2.4, and SUMBT improves over MultiWOZ 2.1 on 26 of 30 slots. MultiWOZ 2.3-cof has the best performance in only seven slots despite its added co-reference annotations.
B Per-Slot (Slot-Specific) Accuracy
- SUMBT achieves higher slot-specific accuracy on MultiWOZ 2.4 than on MultiWOZ 2.1 in 26 of 30 slots.The authors say these results confirm the utility and validity of the refined version.Source: B Per-Slot (Slot-Specific) Accuracy · p. 8
- 21 of 30 slots show higher accuracy on MultiWOZ 2.4, while MultiWOZ 2.3-cof has the best performance in seven slots despite added co-reference annotations.The comparison is for SUMBT across different MultiWOZ versions.Source: B Per-Slot (Slot-Specific) Accuracy · p. 8
- Slot-specific accuracy is the ratio of dialogue turns in which a particular slot's value is correctly predicted.The paper reports this measure in Table 7.Source: B Per-Slot (Slot-Specific) Accuracy · p. 8
C Case Study
The case study shows that incorrect MultiWOZ 2.1 annotations can underestimate dialogue-state-tracking performance, while MultiWOZ 2.4 better reflects predictions aligned with dialogue context. SUMBT also shows stronger per-slot accuracy on most slots in MultiWOZ 2.4 than on the other versions.
C Case Study
- Invalid MultiWOZ 2.1 annotations underestimate existing trackers' performance, while MultiWOZ 2.4 better manifests model performance with context-aligned annotations.In the first four examples, SOM-DST and STAR predictions match MultiWOZ 2.4 annotations; in the last, STAR matches while SOM-DST does not.Source: C Case Study · p. 8
- In the last example, SOM-DST predicts a slot value that differs from both versions' annotations, whereas STAR matches the MultiWOZ 2.4 annotation.Source: C Case Study · p. 8
- Most slots show stronger SUMBT per-slot accuracy on MultiWOZ 2.4 than on the other MultiWOZ versions.Source: Table 7 · p. 9
D Discussion
The benchmark models are retrained on the unchanged, noisy MultiWOZ 2.1 training set, with cleaned validation and test sets used for selection and reporting. Their better performance on MultiWOZ 2.4 supports the need for further annotation refinement, while the dataset also reflects a common noisy-training, clean-evaluation setting.
D Discussion
- Benchmark models achieve better performance on MultiWOZ 2.4 despite training on the unchanged noisy set, using cleaned validation data for model selection and cleaned test data for reporting.Source: D Discussion · p. 9
- MultiWOZ 2.4's superiority indicates that earlier refined versions left incorrect and inconsistent annotations unresolved.Source: D Discussion · p. 9
- Combining MultiWOZ 2.4's refined validation and test sets with MultiWOZ 2.3's cleanest training set may yield even higher performance.Source: D Discussion · p. 9
- The examples show MultiWOZ 2.1 annotations conflicting with dialogue context while MultiWOZ 2.4 annotations and many model predictions align with it.Source: Table 8 · p. 10
- A large noisy training set with small clean validation and test sets reflects a common practical dataset setting, where collecting high-quality multi-domain, multi-turn dialogue data is costly.Source: D Discussion · pp. 9–10
E Potential Impacts
MultiWOZ 2.4's cleaned validation and test sets are intended to support more proper and fair evaluation of dialogue-state-tracking models. Its real training-set noise and multi-label structure may also make it useful for noisy-label-learning research.
E Potential Impacts
- Cleaned validation and test sets can help evaluate dialogue-state-tracking models more properly and fairly.Source: E Potential Impacts · p. 10
- MultiWOZ 2.4 may serve as a dataset for noisy-label-learning research because its training set contains real noise.Source: E Potential Impacts · p. 10
- Its multi-label structure connects MultiWOZ 2.4 to the identified research direction of noisy-label learning for multi-label classification.Source: E Potential Impacts · p. 10