PaperlayerView exact revision on arXiv

Source-linked AI summary

ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)

Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, Mikhail Burtsev

arXiv:2009.11352v1cs.CLcs.IR

TL;DR

ClariQ addresses ambiguous requests in dialogue, where systems must identify ambiguity and ask clarifying questions rather than answer directly. It provides a challenge framework for evaluating clarification decisions and question ranking, treating clarification-question understanding and generation as central to conversational information-seeking.

  • Problem

    Ambiguous requests may admit multiple answers, so dialogue systems must identify ambiguity and decide when to clarify rather than answer directly.

    Source: §1 · p. 1

  • Method

    ClariQ organizes a challenge around scoring clarification necessity and ranking a suitable question for requests needing clarification.

    Source: §3.1 · p. 3

  • Results

    The paper presents clarification-question understanding and generation as major components of conversational information-seeking and describes ClariQ as a timely challenge.

    Source: §2 · p. 2

  • Takeaways & Limitations

    The authors frame shared clarification-question datasets as a means to facilitate further research in this direction.

    Source: §2 · p. 2

    Separating relevance from naturalness assumes users have an information need and avoids treating dialogue as a yes/no puzzle.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

This document presents a detailed description of the challenge on clarifying questions for dialogue systems (ClariQ). The challenge is organized as part of the Conversational AI challenge series (ConvAI3) at Search Oriented Conversational AI (SCAI) EMNLP workshop in 2020. The main aim of the conversational systems is to return an appropriate answer in response to the user requests. However, some user requests might be ambiguous. In IR settings such a situation is handled mainly thought the diversification of the search result page. It is however much more challenging in dialogue settings with limited bandwidth. Therefore, in this challenge, we provide a common evaluation framework to evaluate mixed-initiative conversations. Participants are asked to rank clarifying questions in an information-seeking conversations. The challenge is organized in two stages where in Stage 1 we evaluate the submissions in an offline setting and single-turn conversations. Top participants of Stage 1 get the chance to have their model tested by human annotators.

1 Introduction

ClariQ studies ambiguous dialogue requests, where a system must recognize that a request admits multiple answers and ask a clarifying question instead of answering directly. The challenge focuses on when to clarify and how to generate clarifying questions.

1 Introduction

  • ClariQ studies dialogue requests with multiple possible answers, requiring systems to recognize ambiguity and ask a clarifying question rather than answer directly.Source: §1 · p. 1
  • The challenge asks when systems should ask clarifying questions during dialogue and how they should generate them.Source: §1 · p. 2

2 Related work

Prior work studied clarification questions in narrow Stack Exchange topics and open-domain information retrieval, while datasets supported follow-up research. ClariQ extends the challenge to open-domain dialogue settings.

2 Related work

  • Stack Exchange studies examined clarification-question patterns and ranked candidate questions, but their dataset covers narrow topics and limits understanding of open-domain conversations.Source: §2 · p. 2
  • Open-domain information retrieval studies prompted conversational questions for ambiguous keyword queries; Qulac and MIMICS provide datasets, including real-user queries and behavioral signals such as clicks.Source: §2 · p. 2
  • Clarification-question understanding and generation are recognized as components of conversational information-seeking, making ClariQ a timely dialogue challenge.The paper also notes that shared datasets facilitate further research.Source: §2 · p. 2

3 Challenge Design

ClariQ combines an offline, single-turn stage with later human interaction to evaluate clarification decisions and question selection. Its design describes the Stage 1 dataset and tasks, Stage 2 system requirements, and user-based ratings.

3 Challenge Design

  • The challenge has two stages: Stage 1 uses static datasets for training, validation, and testing, while Stage 2 exposes top-ranked systems to real users.Source: §3 · pp. 2–3

3.1 Stage 1: initial dataset

  • Stage 1 collects conversational user requests, candidate clarification questions, and corresponding user answers for its dataset.The collected data is split into 70% training and 30% validation; testing supplies requests and a question bank.Source: §3.1 · p. 3
  • Stage 1 asks systems to score clarification necessity from 1 to 4 and, for requests needing clarification, return the most suitable question.Source: §3.1 · p. 3
  • Automatic evaluation measures question impact on document retrieval and ranks relevant questions, using MRR, precision, nDCG, and Recall metrics.The listed metrics are MRR, P@[1,3,5,10,20], nDCG@[1,3,5,20], and Recall@[10,20,30].Source: §3.1 · p. 3

3.2 Stage 2: human-in-the-loop

  • Stage 2 requires systems to answer simple questions, decide when to clarify, ask appropriate questions, and interpret user answers; they must balance over-questioning against irrelevant answers.The setup may call for a generative model or retrieval and ranking of candidate answers.Source: §3.2 · p. 4

3.3 User-based evaluation

  • User-based evaluation rates responses for relevance to the information need and naturalness in the dialogue context.The separation assumes users have an information need and avoids treating the interaction as a yes/no puzzle.Source: §3.3 · p. 5

Related papers

Loading 2009.11352v1…