Source-linked AI summary
Asking Clarifying Questions in Open-Domain Information-Seeking Conversations
Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, W. Bruce Croft
TL;DR
The paper addresses how conversational systems can clarify complex or ambiguous information needs that users cannot express in one query. It proposes an offline evaluation protocol, constructs the Qulac dataset, and develops retrieval models; experiments show strong gains from clarifying questions and significant improvements over baselines.
Problem
Users may express complex or ambiguous information needs through queries that support multiple interpretations, motivating systems that clarify the intended facet.
Method
The paper introduces a facet-based offline evaluation protocol, crowdsources the Qulac dataset, and develops models for question retrieval, question selection, and document retrieval.
Results
Asking one good clarifying question produces over 150% relative improvement in P@1, while NeuQS significantly outperforms learning-to-rank baselines.
Takeaways & Limitations
The experiments demonstrate substantial potential for clarifying questions to improve retrieval performance in open-domain information-seeking conversations.
Takeaways & Limitations
Qulac was not collected from actual conversations because pairing workers for question generation and answering was substantially more costly.
Abstract
from arXiv · showhide
Users often fail to formulate their complex information needs in a single query. As a consequence, they may need to scan multiple result pages or reformulate their queries, which may be a frustrating experience. Alternatively, systems can improve user satisfaction by proactively asking questions of the users to clarify their information needs. Asking clarifying questions is especially important in conversational systems since they can only return a limited number of (often only one) result(s). In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems. To this end, we propose an offline evaluation methodology for the task and collect a dataset, called Qulac, through crowdsourcing. Our dataset is built on top of the TREC Web Track 2009-2012 data and consists of over 10K question-answer pairs for 198 TREC topics with 762 facets. Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of P@1, which clearly demonstrates the potential impact of the task. We further propose a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. In particular, our question selection model takes into account the original query and previous question-answer interactions while selecting the next question. Our model significantly outperforms competitive baselines. To foster research in this area, we have made Qulac publicly available.
1 INTRODUCTION
Users often cannot express complex information needs in one query, forcing repeated result scanning or query reformulation. Systems can instead ask proactive questions to clarify intent before presenting results.
- Complex information needs often exceed what users can formulate in a single Web query.
- Users may consequently scan multiple result pages or reformulate their queries.
- Proactive clarification questions offer an alternative by helping systems clarify users’ intent before returning results.
W. Bruce Croft University of Massachusetts Amherst
The paper motivates clarifying-question conversations through examples where identical queries conceal different information needs. It proposes an offline evaluation framework, Qulac, and a three-component retrieval framework for question-driven conversational search.
- Identical queries such as “dinosaur” can represent completely different user information needs.
- The system begins with the same question when it lacks prior context, then adapts subsequent questions to users’ answers.
- Qulac contains over 10K question-answer pairs for 198 TREC topics and 762 facets.
- The proposed retrieval framework combines question retrieval, question selection, and document retrieval.
- The question selection model uses both the user query and conversation context to choose the next question.
2 RELATED WORK
Related work spans conversational information retrieval, clarification, recommendation, and neural question ranking or generation. This paper distinguishes itself by studying clarification-question asking in an information-retrieval setting.
- Early conversational IR systems supported multi-turn information seeking through expert mediation, strategy modeling, and search-session choices.
- Prior studies examined clarification in question answering, voice search, recommendation, and conversational agents.
- NLP research has addressed neural question ranking and generation using chatbot data, query logs, and QA websites.
- Unlike these studies, the paper investigates asking clarification questions when the user’s request is an information-retrieval query.
3 PROBLEM STATEMENT
The task is to select and ask clarifying questions when retrieved results are uncertain, using an offline protocol built around topic facets and crowdsourced question-answer coverage.
- Conversational search can ask questions proactively to better represent information needs and improve confidence in retrieved results.
- The workflow retrieves documents, estimates result confidence, and asks questions when confidence is insufficient.
- Offline evaluation replaces an unavailable question-generation process with a bank containing possible questions and crowdsourced answers.
- Qulac adapts TREC Web Track topics, facets, and facet-level relevance judgments into topic-facet conversation scenarios.
4 DATA COLLECTION
Qulac is collected through a four-step crowdsourcing process that defines topic facets, gathers and verifies clarifying questions, and collects facet-specific answers. The dataset uses TREC Web Track topics and treats facets as distinct information needs.
- Collection procedure: Qulac is built in four steps: define topics and facets, collect candidate questions, verify facet relevance, and collect answers.Questions are added for facets requiring more specific coverage.
- Topics and facets: TREC Web Track 2009–2012 topics are split into facets, with each facet representing a different user’s information need.The average number of facets per topic is 3.85 ± 1.05.
- Question generation: Human annotators generate questions from search results and query auto-complete suggestions to address multiple facets of each topic.Workers were asked to generate six questions related to the query after examining results and suggestions.
- Question verification: Two expert annotators remove invalid or duplicate questions, match questions to relevant facets, and identify facets needing additional questions.A question is relevant when its answer would address the facet.
- Answer collection: Answers are collected for each query–facet–question triplet, with instructions asking workers to answer as users whose information need matches the given facet.Quality checks manually inspect 10% of each worker’s submissions; invalid answers are removed and reassigned.
5 SELECTING CLARIFYING QUESTIONS
The retrieval framework separates candidate question retrieval, context-aware question selection, and document retrieval. It uses query, question, conversation, retrieval, and performance-prediction signals to choose questions and incorporate answers into ranking.
- Framework: The framework retrieves candidate questions, selects the best question from them, and ranks documents using the user’s answer.The workflow uses the original query and previous question–answer interactions as conversational context.
- Question retrieval: BERT-LeaQuR estimates whether a candidate question should be retrieved, aiming to maximize recall among the top k retrieved questions.It learns representations for the query and questions and applies a matching component to produce a retrieval score.
- Question retrieval: The BERT-based retrieval model maps word sequences to d-dimensional representations and fine-tunes pretrained parameters on Qulac.BERT is used to incorporate information from unlabeled data while modeling short queries and questions.
- Question selection: NeuQS selects questions using query, question, conversation-context, retrieval, and query-performance representations.Its challenge is adapting to divergence from the query after negative answers and maintaining focus when previous answers support the current direction.
- Document retrieval: The document model linearly interpolates the original query language model with the model of exchanged questions and answers.The interpolation weight α is tuned on the development set, and documents are ranked after an answer is received.
6 EXPERIMENTS
Experiments evaluate question retrieval and selection on Qulac using topic- and facet-based splits, multiple metrics, and multi-turn conversations. BERT-LeaQuR and NeuQS outperform baselines, while oracle and facet analyses quantify when clarifying questions help.
- Question retrieval: BERT-LeaQuR outperforms all baselines across question-retrieval metrics, including term-matching models such as BM25.Its improving performance as more questions are retrieved suggests it captures relevance when queries and questions lack common terms.
- Oracle question selection: Asking one best question yields over 100% relative improvement across MRR, P@1, and nDCG metrics.The largest relative gains are nDCG@1 at 173% and P@1 at 158%.
- Oracle question selection: Shorter queries and ambiguous topics benefit more from clarification: relative improvement decreases with query length, while ambiguous topics have ∆MRR of 0.3858 versus 0.2898 for faceted topics.The topic-type difference is statistically significant (2-way ANOVA, p ≪0.001).
- Question selection: NeuQS consistently outperforms all baselines on both Qulac splits and achieves statistically significant improvements across evaluation metrics.The evaluation emphasizes MRR, P@1, and nDCG@1 for first-result conversational settings, alongside nDCG@5 and nDCG@20 for Web search.
- Multi-turn conversations: All models improve over multiple conversation turns, and NeuQS remains superior to the baselines at k ∈{1, 2, 3}.The results indicate robust performance as conversations advance.
- Facet impact: NeuQS improves retrieval for 45% of facets but worsens performance for 19% across the evaluated data splits.Facet-level results average NeuQS performance over different conversation turns.
7 LIMITATIONS AND FUTURE WORK
The paper identifies limits in Qulac’s data collection and task scope, and proposes future work on broader conversations, retrieval, and adaptive stopping.
- Qulac was not collected from actual conversations because pairing workers was costly and question generation required nearly 10 times more effort than answering.The authors suggest offering workers a limited set of pre-generated questions as a lower-complexity alternative.
- Qulac supports single-turn conversations, without guaranteeing that questions asked sequentially involve the same participants.
- Question quality depends partly on how well the selected commercial search engine diversifies its result list.The authors attempted to reduce this bias by requiring workers to scan at least three result pages and by using expert questions to cover all facets.
- The task scope extends beyond faceted and ambiguous queries to exploratory search, where questions may increase user engagement.
- Future work includes improving document retrieval as new information arrives and modeling confidence to decide when to stop asking questions.
8 CONCLUSIONS
The paper introduces clarifying-question interaction for open-domain conversational search, an offline evaluation framework, the Qulac dataset, and the NeuQS retrieval framework. Its experiments show substantial gains from effective clarification and continued improvements with additional questions.
- The paper introduces offline evaluation for clarifying questions and releases Qulac with 762 topic-facet pairs and over 10K question-answer pairs.
- NeuQS, a neural question selection model within a question-and-document retrieval framework, significantly outperforms the LTR baselines.
- Over 150% relative improvement in P@1 and nDCG@1 results from asking only one good clarifying question in oracle-model experiments.
- Clarifying questions improve performance for a substantial percentage of facets, although some failure cases suggest document retrieval can be improved.
- Asking more clarifying questions leads to better results in the reported experiments.