Source-linked AI summary
MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, Jindong Chen
TL;DR
MultiWOZ is a widely used dialogue-state-tracking benchmark whose annotations and ontology retain substantial noise after MultiWOZ 2.1, including missing span annotations. The paper introduces MultiWOZ 2.2 by correcting annotations, redefining the ontology, adding standardized spans and dialogue annotations, and benchmarking tracking models; model performance is similar across versions. It also discusses data-collection practices and complex annotation cases that remain challenging.
Problem
MultiWOZ 2.1 retains annotation errors, ontology issues, and missing span annotations that make model comparison difficult.
Method
The paper corrects the dataset, defines a new schema, standardizes slot values and spans, adds intents and requested slots, and benchmarks three dialogue-state-tracking models.
Results
Model performance is similar between MultiWOZ 2.1 and MultiWOZ 2.2 across the evaluated dialogue-state-tracking models.
Takeaways & Limitations
The corrected dataset supports fairer comparison of dialogue-state-tracking models and includes standardized annotations for future work.
Takeaways & Limitations
Complex logical-expression annotations are disproportionately rare, making them difficult for models to handle and challenging to represent expressively.
Abstract
from arXiv · showhide
MultiWOZ is a well-known task-oriented dialogue dataset containing over 10,000 annotated dialogues spanning 8 domains. It is extensively used as a benchmark for dialogue state tracking. However, recent works have reported presence of substantial noise in the dialogue state annotations. MultiWOZ 2.1 identified and fixed many of these erroneous annotations and user utterances, resulting in an improved version of this dataset. This work introduces MultiWOZ 2.2, which is a yet another improved version of this dataset. Firstly, we identify and fix dialogue state annotation errors across 17.3% of the utterances on top of MultiWOZ 2.1. Secondly, we redefine the ontology by disallowing vocabularies of slots with a large number of possible values (e.g., restaurant name, time of booking). In addition, we introduce slot span annotations for these slots to standardize them across recent models, which previously used custom string matching heuristics to generate them. We also benchmark a few state of the art dialogue state tracking models on the corrected dataset to facilitate comparison for future work. In the end, we discuss best practices for dialogue data collection that can help avoid annotation errors.
1 Introduction
MultiWOZ is a widely used dialogue-state-tracking benchmark, but its annotations contain unresolved errors and missing span annotations. MultiWOZ 2.2 addresses these limitations by correcting the dataset, redefining its ontology, adding annotations, and benchmarking tracking models.
- MultiWOZ contains over 10,000 dialogues spanning eight domains and is the most widely used benchmark for dialogue state tracking.
- Despite MultiWOZ 2.1’s corrections, many annotation errors remained, while missing span annotations forced models to use custom string-matching heuristics.
- MultiWOZ 2.2 identifies annotation errors, inconsistencies, and ontology issues in MultiWOZ 2.1 and publishes an improved version.
- The updated dataset adds slot-span annotations for user and system utterances, plus active user intents and requested slots for each user utterance.
- The paper benchmarks state-of-the-art dialogue-state-tracking models on the corrected dataset to facilitate future comparisons.
2 Annotation Errors
MultiWOZ 2.1 contains noisy and inconsistent state annotations, including values absent from the dialogue and divergent tracking strategies. These problems arise from human annotation and multiple value sources, creating inconsistent supervision and evaluation.
- Wizard-of-Oz collection produces natural conversations but relies heavily on humans for annotations, making the procedure prone to noise.
- Hallucinated values appear in dialogue states without being specified in the dialogue history, including early markups, database-derived values, typos, and implicit time processing.
- 3128 turns across 948 dialogues contain hallucinated values in MultiWOZ 2.1.
- State inconsistencies result from multiple value sources, paraphrased slot values, and differing crowd-worker tracking strategies.
- Different annotation strategies can track semantically equivalent values inconsistently, unfairly penalizing equivalent predictions during evaluation.
3 Ontology Issues
MultiWOZ 2.1’s ontology contains duplicate, database-unmatched, and logically complex values. These issues make evaluation and training harder by introducing ambiguity and noise.
- MultiWOZ 2.1’s rebuilt ontology remains incomplete and includes multiple values with the same semantics.
- 51% of hotel-name values are not semantically unique, with similar duplication reported for restaurant-name and attraction-name.
- 21.0% of ontology slot values cannot be mapped directly to database values through exact string matching.
- Logical expressions such as “cheap|moderate” and “NOT(hamilton lodge)” add noise during training and raise representation and generalization challenges.
4 Correction Procedure
MultiWOZ 2.2 corrects inconsistent dialogue-state and action annotations while redefining the ontology through categorical and non-categorical slots. It adds span and copy-source annotations for values that must be extracted from dialogue history and reports the scale of the resulting corrections.
- 4.1 Schema Definition: The new schema is defined before correction to guide annotators and reduce annotation inconsistencies and ontology corruption.It separates slots with small finite value sets from slots with large or dynamic value sets.
- 4.1 Schema Definition: Categorical slots use schema-defined candidate values, while non-categorical slot values are extracted from dialogue history.Slots with fewer than 50 training-set values are generally categorical; Bus and Police use analogous domains because their training data are sparse.
- 4.3 Non-categorical Slots: Non-categorical annotations preserve natural utterances while locating semantically matching spans despite typos and paraphrases.When multiple matches exist, the most recently mentioned value is selected and added to the dialogue state while preserving the original value.
- 4.3 Non-categorical Slots: When a slot value is copied from another slot, MultiWOZ 2.2 records the source slot so models can trace the copy chain.This provides information that may benefit state trackers using copy mechanisms.
- 4.4 Action Annotations: Action annotations are supplemented and normalized, including crowdsourced annotations for 8,333 previously missing action turns.Domain and Booking prefixes are removed to provide a uniform action representation.
- 4.5 Statistics: 17.3% of user utterances across 28.2% of dialogues receive modified dialogue-state annotations.Among 12,375 modified utterance annotations, 1,497 corrections involved two or more slots; pre-existing actions were verified or fixed for around 10% of utterances.
5 Additional annotations
MultiWOZ 2.2 adds active-intent and requested-slot annotations for every user turn, extending the corrected dataset with targets for additional dialogue understanding tasks. These annotations also support dialogue-policy development and intent or API prediction.
- Additional tasks: The annotations create sub-tasks for evaluating active-intent and requested-slot prediction, while intent or API prediction supports efficiency in systems with many APIs.They are added for every user turn.
- Active intents: Active user intents identify all intents expressed in each user utterance, including multiple intents within one turn.The defined intent inventory varies by domain, with search and booking intents available in selected domains.
- Requested slots: Requested slots identify the information the user asks the system to provide.The field is generated from user actions and directly applies to dialogue-policy models.
6 Dialogue State Tracking Benchmarks
The benchmark evaluates three dialogue state tracking models using joint goal accuracy on MultiWOZ versions and on categorical versus non-categorical slots. Performance is generally similar between MultiWOZ 2.1 and 2.2, while TRADE and SGD-baseline perform better on non-categorical slots than categorical slots.
- Models: Three mixed-method dialogue state tracking models—SGD-baseline, TRADE, and DS-DST—are benchmarked on the updated dataset.TRADE combines categorical and non-categorical generation, whereas SGD-baseline uses separate strategies for the two slot types.
- Metric: Joint goal accuracy measures the average proportion of turns for which all slot values are predicted correctly.
- Overall results: Performance is generally similar across MultiWOZ 2.1 and MultiWOZ 2.2 for all three models.Table 4 reports joint goal accuracy for TRADE, SGD-baseline, and DS-DST across the dataset versions.
- Slot-type results: TRADE and SGD-baseline perform considerably better on non-categorical slots than on categorical slots.The authors infer that corrections ensured non-categorical state values also appeared in the dialogue history.
7 Discussion
The discussion emphasizes reducing annotation noise through better Wizard-of-Oz collection procedures and validation. It also highlights that rare logical-expression annotations remain difficult for current models and raise representation questions.
- Data collection: Wizard-of-Oz dialogue collection produces natural conversations but relies heavily on humans, making annotations prone to noise.
- Data collection: Defining an ontology before collection should distinguish categorical slots and constrain annotators to valid values.The proposed interface should list domain and API schemas and allow categorical values only from the predefined set.
- Data collection: Validation checks followed by crowd-sourcing correction can identify errors such as “thursday” for a time slot or “no” for a hotel name.
- Complex constraints: Logical-expression annotations occur in fewer than 1% of dialogues, making them difficult for models to handle and raising questions about expressive state representations.Examples encode preferences or sets of acceptable values using strings such as “cheap>moderate” and pipe-separated alternatives.
8 Conclusion
MultiWOZ 2.2 corrects remaining annotation and ontology issues in MultiWOZ 2.1 while adding standardized spans, intents, requested slots, and dialogue actions. Benchmarks show similar model performance across the two versions, and the cleaned dataset is intended to support fairer comparisons.
- MultiWOZ 2.2 corrects annotation errors, inconsistencies, and ontology issues left unaddressed in MultiWOZ 2.1.
- The release adds a schema, standardized slot values and span annotations, active intents, requested slots, and missing user and system actions.
- Experimental results show similar performance for the benchmarked state tracking models on MultiWOZ 2.1 and MultiWOZ 2.2.
- The cleaned dataset is intended to facilitate fairer model comparisons and further dialogue-system research.