Source-linked AI summary
Harnessing Evolution of Multi-Turn Conversations for Effective Answer Retrieval
Mohammad Aliannejadi, Manajit Chakraborty, Esteban Andrés Ríssola, Fabio Crestani
TL;DR
Multi-turn search requires recovering prior turns that clarify an evolving information need, but user-turn dependencies remain insufficiently understood. The paper annotates TREC CAsT 2019 turn relevance and proposes BERT-CUR for context selection; related-turn reformulation improves retrieval, and PrU outperforms baselines across three retrieval models.
Problem
How users’ utterances depend on one another and how information needs evolve across multi-turn conversations remain insufficiently understood.
Source: §1 · p. 1Method
The paper annotates turn relevance in TREC CAsT 2019 and proposes BERT-CUR to select relevant prior utterances as context.
Source: §1 · p. 2, §4.1 · p. 5Results
BERT-CUR outperformed competitive classification baselines, and PrU outperformed baselines across all three retrieval models.
Source: §5.2 · p. 8, §6 · p. 9Takeaways & Limitations
Publicly available CAsTUR annotations provide a research resource for studying turn relevance in TREC CAsT 2019 conversations.
Source: §1 · p. 2The retrieval experiments do not evaluate neural ranking or reranking; their validity is limited to term-matching approaches because CAsT pooling and labeling used term-matching models.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
With the improvements in speech recognition and voice generation technologies over the last years, a lot of companies have sought to develop conversation understanding systems that run on mobile phones or smart home devices through natural language interfaces. Conversational assistants, such as Google Assistant and Microsoft Cortana, can help users to complete various types of tasks. This requires an accurate understanding of the user's information need as the conversation evolves into multiple turns. Finding relevant context in a conversation's history is challenging because of the complexity of natural language and the evolution of a user's information need. In this work, we present an extensive analysis of language, relevance, dependency of user utterances in a multi-turn information-seeking conversation. To this aim, we have annotated relevant utterances in the conversations released by the TREC CaST 2019 track. The annotation labels determine which of the previous utterances in a conversation can be used to improve the current one. Furthermore, we propose a neural utterance relevance model based on BERT fine-tuning, outperforming competitive baselines. We study and compare the performance of multiple retrieval models, utilizing different strategies to incorporate the user's context. The experimental results on both classification and retrieval tasks show that our proposed approach can effectively identify and incorporate the conversation context. We show that processing the current utterance using the predicted relevant utterance leads to a 38% relative improvement in terms of nDCG@20. Finally, to foster research in this area, we have released the dataset of the annotations.
1 INTRODUCTION
The introduction frames evolving information needs and prior-turn dependencies as challenges in multi-turn search. It presents turn-relevance annotations, BERT-based context modeling, and experiments on classification and retrieval.
1 INTRODUCTION
- The introduction identifies limited understanding of user interaction and evolving information needs in multi-turn conversations as an open research area.It notes that prior work has studied multi-turn question answering, but less is known about how users interact and how their utterances convey information needs.Source: §1 · p. 1
- In the example, utterances u1 and u6 provide context needed to answer the current utterance u8.The paper treats a prior utterance as relevant when it clarifies, augments, or entails the current one.Source: §1 · p. 2
- BERT-CUR predicts related conversation turns and is evaluated both as a classifier and as part of an information-retrieval system.The model uses BERT fine-tuning to represent the utterance and its position in the conversation.Source: §1 · p. 2
- CAsTUR provides publicly available turn-relevance annotations for TREC CAsT 2019, identifying prior turns relevant to a current utterance.Relevance includes turns that clarify, augment, or entail the current utterance.Source: §1 · p. 2
- 38% relative improvement in nDCG@20 follows updating utterances with related turns; first-turn relevance also motivates a simple retrieval heuristic.The authors report that most turns relate to the conversation’s first turn.Source: §1 · p. 2
2 RELATED WORK
Related work spans conversational information retrieval systems, user behavior, and intent clarification. The section connects this work to research on maintaining context across turns and understanding conversational agents.
2 RELATED WORK
- Conversational IR research spans dialogue-based document search, airline retrieval, expert interfaces, and later learning-based and deep-learning approaches.The section traces early work from Oddy, Levinson and Shipley, and Croft and Thompson.Source: §2 · p. 2
- Related work covers conversational applications, user behavior in voice-only systems, and frameworks for multi-turn search that narrows information needs.The section also notes research on recommendation, e-health, and conversational AI.Source: §2 · p. 2
- Research on conversational agents highlights maintaining conversational context, while other work examines intent clarification and agent actions.The paper identifies context maintenance as a focal point of its own study.Source: §2 · p. 3
3 DATA ANNOTATION AND ANALYSIS
The section describes the TREC CAsT benchmark and the construction of CAsTUR, then analyzes question categories, relevant-turn patterns, and conversation evolution. Its findings show that relevant context often comes from early or nearby turns, while question categories and language reveal how users extend topics.
3.1 TREC CAsT
- TREC CAsT was designed to benchmark open-domain conversational search; experiments used MS MARCO and TREC-CAR, excluding WAPO because of data inconsistencies.The 2019 track included 30 training and 50 evaluation conversations.Source: §3.1 · p. 3
3.2 Data Annotation
- Three annotators selected prior utterances that clarify, augment, or entail the current information need, resolving disagreements through discussion.Of 748 utterances, 625 were annotated without major disagreement; the remaining 123 were resolved through discussion, and the resulting CAsTUR annotations were released publicly.Source: §3.2 · p. 3
3.3 Analysis
- 18% of utterances have no related utterances, most have one relevant turn, and fewer than 1% have three or more.The section also reports that conversations average over nine turns, with seven terms per turn.Source: §3.3 · p. 3
- What questions lead 67 conversations, How questions lead 12, and Why questions lead the remaining conversation.The nine categories include What, When, Where, Which, Who, Why, How, Yes/No, and Compare.Source: §3.3 · pp. 3–4, Figure 2 · p. 4
- Half of CAsT utterances contain an English pronoun, and 42% of those pronouns occur as the third word.Many questions begin with “What,” often requesting details or additional information; the third word is usually a pronoun.Source: §3.3 · p. 4, Figure 3 · p. 4
- Analyzing relevant-turn positions across conversations, the study finds that for more than 50% of total turns, a relevant turn is found at the first position in the conversation.Source: §3.3 · p. 4
- More than 30% of utterances have a relevant turn immediately before them, and relevant turns are generally nearby.The analysis suggests that early turns often depend on the first turn, while later turns are less dependent on it.Source: §3.3 · p. 4, Figure 5 · p. 5, Figure 6 · p. 5
- Question-category transitions commonly begin with What, which is likely to be followed by another What; Which is followed by more varied categories.The analysis models category sequences with a Markov model, using categories as nodes and START and END placeholders.Source: §3.3 · p. 5, Figure 6 · p. 5
4 RETRIEVAL FRAMEWORK
The framework predicts relevant prior turns, adds their context to current queries, and evaluates retrieval with term-matching models. BERT-CUR models utterance content and conversational position, while the evaluation scope is limited to term-matching approaches.
4.1 Utterance Relevance Prediction Model
- BERT-CUR jointly models current and candidate utterance language and their positions to estimate whether a candidate turn is relevant.Its relevance score is produced by matching the utterance and position representations.Source: §4.1 · p. 5
- BERT encodes current and candidate utterances, while turn positions receive separate representations; a feed-forward matcher uses ReLU and softmax, trained with cross-entropy.The paper fine-tunes uncased base and large BERT models on CAsTUR.Source: §4.1 · p. 6
4.2 Utterance Reformulation and Document Retrieval
- Reformulation adds context to current queries; FU applies coreference resolution, whereas BERT-CUR omits that step and preserves conversational turn order.Coreference resolution leaves self-contained and first turns unchanged.Source: §4.2 · p. 6
- The retrieval evaluation compares BERT-CUR, original utterances, first-utterance reformulations, and oracle reformulations across BM25, QL, and DFR using four utterance-selection strategies.The first utterance serves as a heuristic, while oracle comparisons assess annotation effectiveness and an upper bound for retrieval improvement.Source: §4.2 · p. 6
- Evaluation validity is limited to term-matching approaches because CAsT relevance judgments were pooled using term-matching models, potentially excluding neural rankers’ retrieved documents.The authors therefore do not evaluate neural ranking or reranking approaches.Source: §4.2 · p. 6
5 EXPERIMENTS
The experiments evaluate utterance-relevance prediction and context-based retrieval, finding that BERT-CUR and relevant-turn reformulation improve performance. They also analyze how results vary with turn position and question category.
5.1 Experimental Setup
- Because TREC relevance judgments were unavailable for the 50 evaluation topics, the 30 training topics served as the test set and the evaluation topics as BERT-CUR training data.Ten percent of the 50 training topics were randomly selected for development.Source: §5.1 · p. 7
5.2 Results and Discussion
- BERT-CUR outperforms the prediction baselines, while FirstUtterance is the closest baseline in F1 and the strongest heuristic.BERT-CUR BASE matches SVM-AWE in precision but exceeds it in recall and F1; the authors attribute better base-model performance than large-model performance to the smaller dataset.Source: §5.2 · p. 7, Table 3 · p. 7
- All reformulation strategies improve retrieval over the original utterance, but no heuristic method achieves a significant improvement.Even selecting the first or previous utterance improves retrieval, while the reported heuristic gains are not statistically significant.Source: §5.2 · p. 8, Table 4 · p. 9
- PrU outperforms all baselines across the three retrieval models; gains are consistently significant for DFR and mostly significant for QL, but less consistent for BM25.The reported exception is MRR for QL; PrU is also comparable to the oracle, with no significant difference reported.Source: §5.2 · p. 8, Table 4 · p. 9
- Retrieval performance generally diminishes as conversation turns progress, with exceptions; from turn 10 onward, the potential relationship between current and relevant questions decreases.The authors distinguish these conversations, which lack clarifying questions, from settings where clarification may improve information-need understanding.Source: §5.2 · p. 8, Figure 8 · p. 8
- Adding relevant questions yields no effect for Which questions, but improves When-question retrieval by at least 60% over the original baseline with PrU or the oracle.The authors link the When-question pattern to those questions being preceded by other question categories.Source: §5.2 · p. 8, Figure 9 · p. 8
- Precision-oriented PrUP yields the best retrieval performance of the three selection runs despite having the lowest recall and F1.The comparison covers precision-, recall-, and F1-oriented relevant-question selection.Source: §5.2 · p. 9, Table 5 · p. 9
6 CONCLUSIONS AND FUTURE WORK
The authors conclude that annotated utterance dependencies reveal patterns in multi-turn information-seeking conversations, and that BERT-CUR can predict relevant turns and improve retrieval. Their findings also indicate that context dependence varies across turns and favors precision-oriented selection.
6 CONCLUSIONS AND FUTURE WORK
- The authors annotated TREC CAsT 2019 conversations and found a strong negative correlation between utterance relevance and distance.They suggest that users tend to ask about information relevant to more recent utterances.Source: §6 · p. 9
- BERT-CUR outperformed competitive classification baselines, and incorporating conversation context positively affected document retrieval performance.The highest context dependency for their proposed model occurred at the third turn; experiments also suggest relevant-turn prediction is precision-oriented rather than recall-oriented.Source: §6 · p. 9
- Future work will extend the analysis to information-seeking conversations in which systems can ask clarifying questions.The authors plan to investigate effective modeling of relevant utterances in that setting.Source: §6 · p. 9