Source-linked AI summary
TREC CAsT 2019: The Conversational Assistance Track Overview
Jeffrey Dalton, Chenyan Xiong, Jamie Callan
TL;DR
CAsT addresses the need for reusable evaluation resources for conversational information seeking, where systems must interpret dialogue context to retrieve relevant passages. It establishes a benchmark with curated conversations, shared resources, and contextual evaluation, and finds a substantial effectiveness gap between manually resolved and automatic queries.
Problem
CAsT addresses the need for a reusable test collection for conversational search, where effective response selection requires understanding dialogue history.
Method
The track constructs conversational topics and passage-based tasks, provides rewritten utterances and other training resources, and evaluates systems by ranking depth and conversational turn depth.
Results
A 35% relative difference separates the best manual and automatic runs, while automatic effectiveness declines by 23% from the first to the eighth turn.
Takeaways & Limitations
Clean conversational context can maintain or improve effectiveness, whereas automatic systems decline with turn depth; BERT-based ranking is leading but has mixed results.
Takeaways & Limitations
Assessment was constrained by incomplete pooling because only the top two prioritized runs from each group were pooled.
Abstract
from arXiv · showhide
The Conversational Assistance Track (CAsT) is a new track for TREC 2019 to facilitate Conversational Information Seeking (CIS) research and to create a large-scale reusable test collection for conversational search systems. The document corpus is 38,426,252 passages from the TREC Complex Answer Retrieval (CAR) and Microsoft MAchine Reading COmprehension (MARCO) datasets. Eighty information seeking dialogues (30 train, 50 test) are an average of 9 to 10 questions long. Relevance assessments are provided for 30 training topics and 20 test topics. This year 21 groups submitted a total of 65 runs using varying methods for conversational query understanding and ranking. Methods include traditional retrieval based methods, feature based learning-to-rank, neural models, and knowledge enhanced methods. A common theme through the runs is the use of BERT-based neural reranking methods. Leading methods also employed document expansion, conversational query expansion, and generative language models for conversational query rewriting (GPT-2). The results show a gap between automatic systems and those using the manually resolved utterances, with a 35% relative improvement of manual rewrites over the best automatic system.
1 INTRODUCTION
CAsT establishes a reusable benchmark for conversational information seeking, where effective response selection depends on dialogue context. Its first-year scope emphasizes context understanding and passage retrieval, while anticipating richer mixed-initiative interactions.
- CAsT was introduced to facilitate Conversational Information Seeking research and create a large-scale reusable test collection.
- The task requires understanding a question’s dialogue history to select effective responses.It directs attention to user modeling, prior retrieval results, and transforming conversational questions into queries.
- The first-year benchmark uses predetermined conversation trajectories and passage responses across coherent, multi-round information needs.The initial focus is understanding conversational information needs and finding relevant passages using context.
- CAsT’s long-term vision includes mixed-initiative actions such as informing, clarifying, and suggesting, but Year 1 returns simple text passages.Future work covers richer actions, response formats, and interactions.
2 TASK DESCRIPTION
Year 1 defines conversational search as retrieving brief, relevant passages for each turn while using prior utterances as context. The benchmark combines curated exploratory dialogues with large passage collections, though WaPo is excluded from final evaluation because of document-ID errors.
- Task Definition: CAsT retrieves brief text passages suitable for voice interfaces or mobile screens rather than lists of documents.
- Task Definition: Given conversational turns T = {u1, ...ui...un}, systems identify relevant passages Pi for each utterance using earlier context u<i = u1 : ui−1.
- Information Needs: Topics are open-domain exploratory information needs selected for complexity, diversity, and answerability.
- Conversational Sequences: Conversation sequences begin with broad topics and develop manually formulated trajectories whose later turns depend only on previous utterances.
- Conversational Sequences: The dialogues include coherent transitions, coreference, omission, subtopic comparisons, and many turns requiring more than a factoid answer.
- Conversational Sequences: The first-year release contains 30 training and 50 evaluation topics, each with about ten turns, spanning diverse open-domain subjects.
- Passage Collection: The passage collection draws on MS MARCO, TREC CAR, and Washington Post passages, including approximately 30 million unique Wikipedia paragraphs and roughly 8 million MARCO passages.
- Passage Collection: WaPo passages were removed from pooling and evaluation after deduplication produced ambiguous document IDs, eliminating less than 5% of returned results.
3 RESOURCES
CAsT supplied training data, conversational rewrites, search-session resources, deduplication information, and software to support CIS system development. These resources were designed to reduce entry barriers and make the collection reusable.
- The organizers provided resources for model training, participant development, and future public CIS research.
- Training resources included topics with incomplete judgments, MS MARCO conversational sessions, manually rewritten topics, and near-duplicate files.
- The organizers created 30 training topics, including five with manually produced relevance assessments on a compressed three-point scale.
- MS MARCO and TREC CAR enabled participants to reuse existing non-conversational relevance labels for single-shot relevance training.
- The MS MARCO Conversational Search Session dataset aligns released MS MARCO queries to Bing sessions using a Generic Intent Encoder and approximate nearest-neighbor search.
- Manual resolved utterances rewrite conversational turns so each contains the information needed to represent the underlying single-turn need.
- Each utterance was rewritten by two organizers, adjudicated to a canonical form, and required about 5–10 minutes per ten-turn topic.
- Near-duplicate detection clustered MARCO and WaPo passages so only one passage per duplicate cluster was evaluated.
4 EVALUATION METHODOLOGIES
CAsT evaluates passage ranking at early positions and across conversational turns using context-sensitive relevance judgments. Its first-year assessment used pooled system outputs and manual rewrites, while limited labeling resources constrained the evaluation design.
- Relevance Criteria: Conversational relevance depends on preceding turns because a passage’s meaning and relevance may be contextualized.
- Relevance Criteria: CAsT turns commonly contain coreference and omission, so organizers supplied resolved versions that make implicit context explicit.
- Relevance Criteria: Good responses should be concise while completely answering the turn, matching the needs of spoken or chat interfaces.
- Relevance Criteria: The five-point relevance scale ranges from fully meeting the turn to failing to meet it.
- Assessment Process: Assessments used TREC-style pooling from two highest-priority runs per participant, judged to depth 10, yielding 33,614 unique paragraphs.
- Assessment Process: NIST assessors labeled 20 conversational topics, averaging eight judged rounds per topic, using both raw and manually resolved utterances.
- Evaluation Metrics: Evaluation varies both ranking depth and conversational turn depth, with deeper-turn performance indicating context-understanding ability.
- Evaluation Metrics: Mean NDCG@3 is the primary metric, supplemented by turn-depth NDCG@3&N, MAP, and mean reciprocal rank.
5 PARTICIPANTS
Twenty-one teams submitted 65 runs using diverse conversational retrieval approaches, including traditional retrieval, learning-to-rank, neural models, and knowledge-enhanced methods. Many systems used BERT-based reranking alongside query expansion, rewriting, or other ranking components.
- Participants and runs: 21 teams submitted 65 runs, with participants documenting properties of their conversational retrieval systems.The submissions are summarized in the participant and run tables.
- Query processing and reranking: Several systems combined conversational query rewriting or expansion with first-stage retrieval and neural reranking.Examples include sequence-to-sequence rewriting, historical query or answer expansion, and BERT-based reranking.
- Approach diversity: The approaches span traditional retrieval, feature-based learning-to-rank, neural models, and knowledge-enhanced methods.These categories characterize the diversity of submitted systems.
- Neural reranking: BERT-based reranking was a common theme across the submitted approaches.Systems used BERT to produce or combine passage-matching scores after initial retrieval.
6 OVERALL RESULTS
Results compare automatic and manual conversational retrieval runs using NDCG@3, with BERT prominent among top systems and performance varying across conversation depth. Manual runs substantially outperform automatic runs, while incomplete pooling and corpus filtering qualify interpretation.
- NDCG@3 is the primary measure because the evaluation emphasizes graded relevance at the top ranks.
- Automatic run results: The median automatic run scores 0.286 NDCG@3, while nine of the top ten runs use BERT for ranking.The best non-BERT run, clacBase, scores 0.360; the top two teams use contextual query rewriting and expansion.
- Manual run results: The best manual run reaches 0.589 NDCG@3, and the best manual and automatic runs differ by 35% relatively.The median manual run scores 0.361, and top manual runs also leverage BERT reranking.
- Evaluation considerations: Filtering Washington Post passages affects runs without WaPo results; about 10% of pool candidates come from WaPo.Four of the eight runs without WaPo results are among the top-performing runs.
- Evaluation considerations: Incomplete pooling changes NDCG by about 10% absolutely, although the source of the difference is unclear because teams likely selected their best runs for pooling.The top-performing run is not included in the pool, and unjudged passages remain among top results.
- Results by turn depth: Automatic-run effectiveness declines from approximately 0.3 at turn one to 0.23 by turn eight, a statistically significant 23% decrease.Early depths 1 and 3 perform best, while depth 7 is statistically equivalent to the first turn.
- Results by turn depth: Manual effectiveness remains statistically equivalent from the conversation start to its end, while the manual–automatic effectiveness gap increases by more than 100%.Manual runs dip slightly at turn two, rise through turn four, and drop at turn five.
7 RESULTS ANALYSES
Automatic-run analyses show that conversational query understanding remains difficult, while training data and contextual resources have uneven effects on effectiveness. Neural ranking methods show promise but perform more consistently on manually resolved queries.
- 7.1 Query Understanding: 57% of automatic runs used deep learning for query understanding, but performed 8% worse on average than runs not using it.
- 7.1 Query Understanding: The strongest query-understanding gains came from manually designed term reweighting, conversational stopword removal, and expansion using previous-turn results.
- 7.2 Retrieval and Ranking: Training data improved effectiveness overall: MS MARCO supervision produced a greater than 20% average improvement, while unsupervised runs were 8% worse.
- 7.2 Retrieval and Ranking: Neural methods were used by half of the runs and appeared in nine of the ten best runs, yet showed no average advantage over non-neural methods.
- 7.2 Retrieval and Ranking: Neural methods yielded a 19% relative improvement for manual runs versus 1% for automatic runs, suggesting greater effectiveness with manually resolved queries.
- 7.3 Conversational Context: Previous turns were used by 86% of automatic runs and were considered crucial for resolving current-turn contextual dependence; conversation titles were the most effective context.
8 CONCLUSION
CAsT identifies conversational language understanding, context, and ranking as central challenges in conversational search. Manual query resolution creates a substantial advantage, while neural ranking is promising but inconsistent.
- Conversational Language Understanding: Manual query resolution produced an approximately 35% gap over the best automatic system, highlighting the difficulty of conversational language understanding.
- Conversational Context: Clean context maintained or improved effectiveness across conversation turns, whereas automatic runs declined as turn depth increased.
- Ranking: BERT-based neural models led response ranking across manual and automatic methods, but many were outperformed by simpler traditional ranking approaches.