Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions

1 Introduction

• Ambiguous requests call for identifying when clarification is needed and asking a suitable question instead of answering directly.The introduction frames a lack of suitable datasets as a barrier to studying this problem.Source: §1 · pp. 1–2
• The dataset covers about 300 topics, with approximately 15K single-turn and 1.5M multi-turn conversations, supporting study of when and what to ask.Source: §1 · p. 2
• The paper benchmarks state-of-the-art neural models and proposes a two-stage evaluation pipeline: offline automatic evaluation and online human-in-the-loop conversation.Source: §1 · p. 2
• The work releases the dataset, offline evaluation pipeline, and code for running the explored neural models as task baselines.Source: §1 · p. 2

2 Related work

• The paper situates its contribution across learning to ask clarifying questions in open-domain conversations and evaluating dialogue systems.Source: §2 · p. 2

2.1 Learning to ask clarifying questions

• Prior clarification research includes search-oriented query clarification and datasets, alongside dialogue work on when to ask and which question to ask.Source: §2.1 · p. 2

2.2 Evaluating Dialogue Systems

• Open-domain dialogue evaluation remains under exploration, and the current standard commonly uses crowdsourced human assessments or research-competition volunteers.Source: §2.2 · pp. 2–3

3 Problem Setting

• User Request Understanding decides whether a request goes to Clarifying Question Generation or Answer Generation; this work focuses on the first two modules.Source: §3 · p. 3
• Each data item combines a conversational user request, a clarification-need label, candidate questions addressing request facets, and corresponding user answers.Source: §3 · p. 3
• The coherent triples (U, Q, A) can be transformed into single-turn conversations, with requests spanning topics and including both ambiguous and unambiguous cases.Source: §3 · p. 3
• Collected single-turn conversations train agents; the two best-performing agents then converse with crowdsourced workers for multi-turn collection and Acute-eval assessment.Source: §3 · p. 3

4.1 P1: Crowdsourcing Single-Turn Dialogues

• Single-turn requests start from 300 TREC Web topics; expert annotators convert keyword descriptions into conversational requests and label clarification need on a 1–4 scale.Two annotators assess each request; disagreements receive an additional assessment, with Cohen’s κ = 0.78.Source: §4.1 · p. 4
• Two workers generate 12 clarifying questions per request; experts remove invalid or duplicate questions, map questions to facets, and add questions for uncovered facets.Workers also provide answers conditioned on a facet; quality checks remove invalid submissions, yielding single-turn (U, Q, A) triples.Source: §4.1 · pp. 4–5

4.2 P3: Crowdsourcing Multi-Turn Dialogues

• For multi-turn collection, workers answer an agent’s final question using dialogue context that may include 1–2 prior question–answer rounds.Unclear or improper-language questions are removed; synthetic training conversations are also created from combinations of two or three questions and their answers.Source: §4.2 · pp. 5–6

5.1 P2: Evaluating Single-Turn Agents

• Experiments use topic-based 70/10/20 training, validation, and test splits shared across single- and multi-turn settings, with requests evaluated against an all-topic question bank.Source: §5.1 · p. 6

5.1.1 Predicting Clarification Need

• Clarification-need prediction assigns 1–4 necessity scores, but the authors characterize the task as difficult despite reasonable baseline performance.The compared predictors include RoBERTa-, BART-, and BERT-based classifiers.Source: §5.1.1 · pp. 6–7

5.1.2 Returning Clarifying Question

• Single-turn question selection is evaluated by retrieval impact (document relevance) and by ranking relevant questions from the question bank (question relevance).Question relevance is measured with Recall@[10,20,30].Source: §5.1.2 · p. 7
• The best-performing fine-tuned neural single-turn models perform reasonably and are used for multi-turn evaluation.Tables 5 and 6 report document- and question-relevance metrics, respectively.Source: Table 5 · p. 7, Table 6 · p. 8, §5.1.2 · p. 8

5.2 P4: Evaluating Multi-Turn Conversations

• Multi-turn evaluation uses full conversation history and document relevance; the two best single-turn models are fine-tuned on synthetic data for this task.Question relevance is not evaluated for multi-turn models.Source: §5.2 · p. 8
• Multi-turn methods are solid baselines: ELECTRA beats RoBERTa automatically and wins most human judgments across five aspects.The human comparison covered 120 dialogue pairs; automatic and human evaluations were reported as aligned.Source: Table 7 · p. 8, Table 8 · p. 8, §5.2 · pp. 8–9

• The dataset includes single- and multi-turn conversations and covers almost 300 different topics, providing a foundation for further research.Source: §6 · p. 9
• The authors conclude that fine-tuned state-of-the-art neural models are solid baselines for future research on clarifying questions.Source: §6 · p. 9
• The proposed offline automatic evaluation pipeline agrees with human-in-loop evaluation.Source: §6 · p. 9
• The authors publicly release the datasets, baseline-training code, and evaluation procedure.Source: §6 · p. 9