Source-linked AI summary
Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search
Antonios Minas Krasakis, Mohammad Aliannejadi, Nikos Voskarides, Evangelos Kanoulas
TL;DR
Conversational ranking models’ ability to use clarifying questions and answers has not been broadly analyzed, despite mixed-initiative systems’ use of such questions. The paper analyzes lexical ranking on Qulac and introduces an answer-type heuristic; it significantly outperforms the naive baselines, with improvement over Q0 + Q + A significant at p < 0.001.
Problem
How clarifying-question answers affect document ranking remains insufficiently analyzed, while noisy language, mixed signals, and partially relevant questions complicate their use.
Source: §1 · p. 1Method
The authors analyze a query-likelihood model on Qulac and develop a heuristic ranker that uses answer polarity and length to select which clarification content to include.
Source: §1 · p. 1, §3.3 · p. 4Results
The heuristic ranker outperforms all baselines, with improvement over Q0 + Q + A significant at p < 0.001.
Source: §1 · p. 1, §3.3 · p. 4Takeaways & Limitations
The findings support fine-grained treatment of clarification conversations that incorporates explicit user feedback.
Source: §4 · p. 4Qulac’s question-collection process may bias comparisons for negative answers because workers composed questions after viewing the first two result pages for the original query.
Source: §3.1.2 · p. 3
Abstract
from arXiv · showhide
Recent research on conversational search highlights the importance of mixed-initiative in conversations. To enable mixed-initiative, the system should be able to ask clarifying questions to the user. However, the ability of the underlying ranking models (which support conversational search) to account for these clarifying questions and answers has not been analysed when ranking documents, at large. To this end, we analyse the performance of a lexical ranking model on a conversational search dataset with clarifying questions. We investigate, both quantitatively and qualitatively, how different aspects of clarifying questions and user answers affect the quality of ranking. We argue that there needs to be some fine-grained treatment of the entire conversational round of clarification, based on the explicit feedback which is present in such mixed-initiative settings. Informed by our findings, we introduce a simple heuristic-based lexical baseline, that significantly outperforms the existing naive baselines. Our work aims to enhance our understanding of the challenges present in this particular task and inform the design of more appropriate conversational ranking models.
1 INTRODUCTION
The paper studies how clarifying-question answers affect conversational document ranking, where noisy language, mixed signals, and partially relevant questions make feedback difficult to use. It analyzes answer polarity and length and introduces a heuristic ranker based on answer type.
1 INTRODUCTION
- Users often begin answers with “yes” or “no” but add information, motivating the hypothesis that explicit feedback can improve ranking even when questions are partly or wholly irrelevant.The paper illustrates common answer patterns using frequent 4-grams from Qulac conversations.Source: §1 · p. 1, Figure 1 · p. 2
- Clarifying-question answers can combine positive and negative signals, while rejected questions may still contain partially relevant information.For example, a user may answer “No” to a question about Thomas Jefferson when the information need concerns the Declaration of Independence.Source: §1 · p. 1
- The study asks how answer polarity, informativeness, and length—and question length—affect term-matching ranking, and how a rule-based ranker compares with baselines.It analyzes the Qulac dataset with a query-likelihood model chosen for simplicity and transparency.Source: §1 · p. 1
- The authors introduce a heuristic ranker that significantly outperforms the baseline lexical model by incorporating information about answer type.The analysis categorizes answers by polarity and length.Source: §1 · p. 1
2 EXPERIMENTAL SETUP
The experiments evaluate document ranking on Qulac’s single-round conversations, using an initial query followed by a system clarification and user answer. The setup reserves 40 topics for testing the heuristic ranker and uses query-likelihood scoring with equal-weight interpolation.
2 EXPERIMENTAL SETUP
- Qulac provides 10K query-question-answer tuples across 198 TREC topics and 762 facets, with each Q&A corresponding to a facet.The corpus is built on TREC Web Track 2009–2012 data.Source: §2 · p. 2
- The experiments use Qulac’s single-round conversations, with 40 randomly selected topics held out to test the heuristic ranker and the remaining topics used for analysis.The ranking task follows an initial query Q0, a system question Q, and a user answer A.Source: §2 · p. 2
- The query-likelihood model linearly interpolates Q0 with the concatenated clarification round Q and A at equal weight, and performance is evaluated using NDCG@20.The authors report that expanding Q0 with Q and A initially performed poorly because it underestimated the topic’s importance.Source: §2 · p. 2
3 RESULTS
The results show that clarification effects depend on answer polarity, answer length, and whether questions and answers are used together. A heuristic that selects which clarification content to use improves on all tested baselines.
3.1.1 Ranking with full conversation.
- Positive answers improve ranking for both single-word and multi-word replies, while single-word negative answers substantially reduce performance.Multi-word negative answers, by contrast, significantly improve ranking; multi-word “Other” answers also improve it.Source: §3.1.1 · p. 2
- For “I don’t know” answers, ignoring the clarification round is preferable, likely because the dataset instructed workers to use that response for unanswerable or irrelevant questions.The authors also report low original-query performance for negative-answer facets and hypothesize a data-collection bias toward popular information needs; they find a correlation between facet performance and the share of positive or negative answers.Source: §3.1.1 · p. 2, §3.1.2 · p. 3
3.1.2 Ranking with part of the conversation.
- For positive multi-word answers, using only the question or answer diminishes the gains from the full clarification round, consistent with complementary information.When question and answer trends agree, their combination generally follows that trend; when they conflict, outcomes vary, with a more robust positive impact when the question harms but the answer helps.Source: §3.1.2 · pp. 2–3
3.2 Clarifying Question and Answer Length
- Clarification-question and answer length both correlate with NDCG improvement, with the correlation strongest for answers.Questions can also help despite negative, single-word replies: contextually relevant terms may occur in relevant documents and act like query expansion, even when added terms are not strictly relevant to the need.Source: §3.2 · pp. 3–4, Table 4 · p. 4
3.3 Heuristic Ranker
- The heuristic ranker outperforms all baselines, significantly improving over Q0 + Q + A (p < 0.001).It uses the answer’s polarity and length to choose whether to add the answer, the full clarification round, or neither.Source: Table 5 · p. 4, §3.3 · p. 4
4 CONCLUSIONS AND FUTURE WORK
The paper offers insights into document ranking with clarification-based conversations and argues that explicit feedback requires fine-grained treatment. It proposes a simple heuristic ranker that addresses part of the problem and outlines future work on neural ranking models and answer classification.
4 CONCLUSIONS AND FUTURE WORK
- The paper argues that successful conversational search requires fine-grained treatment of clarification rounds and their explicit feedback.Source: §4 · p. 4
- The proposed heuristic ranking model addresses part of the clarification-feedback problem despite its simplicity.Source: §4 · p. 4
- Future work will examine neural ranking models, more sophisticated answer classification, and ranking models that better incorporate explicit feedback.Source: §4 · p. 4