CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset

1 Introduction

• Existing dialogue datasets remain limited in size, language variation, or task complexity, while real-life conversations naturally transition across domains while maintaining coherent contexts.The introduction notes that MultiWOZ has noisy state annotations, lacks user-side dialogue acts, and uses pre-specified rather than dynamically chosen cross-domain constraints.Source: §1 · p. 1
• Unlike fixed shared constraints, a choice in one domain can affect choices in related domains, such as requiring a hotel near a previously chosen attraction.The example dialogue links an attraction, nearby hotel, and taxi trip, with cross-domain informable slots changing during conversation.Source: §1 · pp. 1–2, Figure 1 · p. 2
• CrossWOZ introduces a large-scale Chinese cross-domain task-oriented dialogue corpus, addressing the shortage of widely recognized Chinese task-oriented data.It contains 6K sessions and 102K utterances across attraction, restaurant, hotel, metro, and taxi domains.Source: §1 · pp. 1–2
• Dialogue states and dialogue acts are annotated on both user and system sides, enabling tracking from the user’s perspective.The user-state annotation can support more elaborate user simulators.Source: §1 · p. 2
• The paper provides benchmark models for pipelined task-oriented dialogue modules and a user simulator to support model comparison and evaluation.The benchmark modules include natural language understanding, dialogue state tracking, policy learning, and natural language generation.Source: §1 · p. 2

2 Related Work

• Human-to-human collection can yield natural, diverse dialogues but requires substantial human effort; machine-to-machine collection needs less effort but is limited by simulator policy complexity and diversity.Human-to-machine data collection requires a ready dialogue system, whose performance influences data quality.Source: §2 · pp. 2–3
• MultiWOZ and Schema provide multi-domain dialogues, but their dependencies use pre-specified shared constraints rather than dynamically determined cross-domain choices.CrossWOZ examples link a hotel to an attraction chosen in earlier turns, so system recommendations and user selections can dynamically affect the dialogue.Source: §2 · pp. 3–4, Table 2 · p. 3
• CrossWOZ has a comparable training-set scale to MultiWOZ, with more domains and turns per dialogue on average, and permits repeated instances of the same domain in a user goal.For example, a tourist may want to visit more than one attraction.Source: §2 · p. 4
• User-state annotation complements system states and dialogue acts by recording subgoal completion from the user’s perspective, which can support user-simulator construction.The user state can be used to predict user actions.Source: §2 · p. 4

3 Data Collection

• The collection covers hotel, attraction, restaurant, metro, and taxi scenarios, with Web-crawled Beijing data for the first three domains and metro data derived from their nearest stations.Taxi information is accessed through an API when needed.Source: §3 · p. 4
• Cross-domain goals link targets through proximity or metro/taxi travel, and task descriptions are generated from structured goals using hand-crafted templates.The three HAR domains are hotel, attraction, and restaurant.Source: §3 · p. 4, §3.2 · p. 5

3.2 Goal Generation

• Goals use semantic tuples for sub-goals, domains, slots, and values, with cross-domain informable slots referring to other sub-goals so their constraints depend on conversation context.Each goal has at most five sub-goals; common informable values are sampled from the database, while requestable values are gathered through conversation.Source: §3.2 · pp. 5–6
• Goal generation adds linked HAR sub-goals, possible metro or taxi trips between HAR entities, and an ordering that places each sub-goal near the sub-goal it refers to.The process begins with independent HAR sub-goals and uses probabilities to generate linked goals and traffic-domain sub-goals.Source: §3.2 · p. 5

3.3 Dialogue Collection

• Two trained workers converse synchronously as tourist and wizard, while the wizard searches the database and responds to help the user complete an allocated goal.Workers received feedback on practice dialogues before formal collection.Source: §3.3 · p. 6

3.3.1 User Side

• The user state tracks goal completion and changes, while the system state records current domain constraints and can preserve both original and relaxed queries when no entity meets them.Users may revise constraints when no result is available; the wizard may relax constraints and issue multiple queries.Source: §3.3.1 · p. 6, §3.3.2 · p. 6

3.4 Dialogue Annotation

• Dialogue acts are automatically annotated from state updates, dialogue history, and keyword matching, and a binary label records whether each user-state tuple has been expressed.Three experts’ comparison with the annotations yielded average dialogue-act F1 of 95.36% and state accuracy of 94.95% on 50 dialogues (806 utterances).Source: §3.4 · pp. 6–7

4 Statistics

• 3.24 average sub-goals per dialogue in CrossWOZ, compared with 1.80 in MultiWOZ and 1.84 in Schema; its average 16.9 turns also exceeds MultiWOZ’s 13.7.The paper reports these statistics as indicating more complex dialogue data.Source: §4 · p. 7

Independent multi-domain + traffic (M+T)

• About 60% of dialogues in M+T, CM, and CM+T have cross-domain informable slots, and complexity increases in the orders S<M<CM and M<M+T<CM+T.The reported ordering is supported by average sub-goals, semantic tuples, and turns per dialogue.Source: Independent multi-domain + traffic (M+T) · pp. 7–8
• CM and CM+T are more challenging because strict cross-domain constraints can produce NoOffer situations, prompting constraint relaxation, multiple queries, and user goal changes.The paper describes NoOffer, multi-query, and goal-change rates as capturing this negotiation process.Source: Independent multi-domain + traffic (M+T) · p. 8
• Most single-domain dialogues end within 10 turns, while M and M+T length curves are similar and the traffic task requires two additional turns on average.The paper attributes the less similar CM and CM+T curves partly to CM goals with five sub-goals that cannot add a traffic sub-goal.Source: Independent multi-domain + traffic (M+T) · p. 8

5 Corpus Features

• Complex user goals favor inter-domain dependency and natural transitions between domains, yielding dialogues described as more complex and natural for cross-domain tasks.Figure 2 presents dialogue-length distributions for different training-set goal types.Source: §5 · p. 8, Figure 2 · p. 8
• A well-controlled synchronous setting is used to collect human-to-human dialogues, which the paper says ensures high-quality collected dialogues.Source: §5 · p. 8
• Explicit annotations cover both system and user sides, allowing user behaviors to be modeled or user simulators to be developed more easily.Source: §5 · p. 8

6.1 Natural Language Understanding

• BERTNLU performs well with context, but Select intent remains substantially harder to predict than other intent types.Context improves Select prediction; the authors identify context use for recognizing domain transitions as a need for NLU models.Source: §6.1 · p. 10

6.2 Dialogue State Tracking

• On user turns with Select intent, joint state accuracy is 11.6% for RuleDST and 12.0% for TRADE, and both models perform worse on cross multi-domain dialogues.The authors report that these results show the models do not track domain transitions well.Source: §6.2 · p. 10

6.3 Dialogue Policy Learning

• With Select intent in preceding user acts, exact-dialogue-act F1 is 41.53% and delexicalized-dialogue-act F1 is 54.39%; cross multi-domain dialogues are harder.The gap between exact and delexicalized act scores points to the need for a stronger state tracker to identify correct entities.Source: §6.3 · pp. 10–11

6.4 Natural Language Generation

• TemplateNLG BLEU is 0.5780 for user utterances and 0.6828 for system utterances; SC-LSTM scores 0.7858 and 0.8595, respectively.The authors attribute the gap to SC-LSTM generating common patterns while TemplateNLG retrieves more specific original sentences.Source: §6.4 · p. 11

6.5 User Simulator

• The rule-based user simulator performs markedly worse on complex CM and CM+T dialogues than on simpler goal types.It maintains user state with hand-crafted rules and outputs dialogue acts from system dialogue acts; user-side state and act annotations support development of more elaborate simulators.Source: §6.5 · pp. 11–12

6.6 Evaluation with User Simulation

• Cross multi-domain tasks are much harder to finish, and added traffic-domain sub-goals remain difficult despite good module performance.“Task finish” means all goal slots are filled with real values, not that the system provided correct information.Source: §6.6 · p. 12
• System-level performance is limited by RuleDST and the supervised-learning policy, while NLU and NLG also harm performance despite strong corpus-based results.The authors conclude that more powerful models are needed for all components of a pipelined dialogue system.Source: §6.6 · p. 12
• TemplateNLG has lower BLEU than SC-LSTM but performs better in natural-language-level simulation, which the authors attribute to BERTNLU preferring retrieved templates.Source: §6.6 · p. 12

• CrossWOZ contains 6K dialogues and 102K utterances across five domains, with dialogue-state and dialogue-act annotations on both user and system sides.Source: §7 · p. 12
• About 60% of dialogues have cross-domain user goals that encourage transitions between related domains.Source: §7 · p. 12
• Experiments find cross-domain constraints challenging across tasks, especially transitions between related domains, and system-level evaluation indicates that stronger models are needed across pipeline components.Source: §7 · p. 12