Source-linked AI summary
TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview
Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffery Dalton, Leif Azzopardi
TL;DR
Conversational search agents need to use a person’s prior interactions and context to tailor information to that user’s needs. iKAT provides tasks and resources for studying such personalized search, and its first-year results show generate-retrieve-generate approaches outperforming retrieve-then-generate approaches in passage ranking and response generation.
Problem
Conversational search must account for personal context because similar questions can require different responses for users with different profiles and preferences.
Source: §1 · p. 1Method
iKAT evaluates personalized conversational search through PTKB-informed statement ranking, passage retrieval, and response generation tasks.
Source: §2.1 · p. 2Results
The top four automatic submissions for passage ranking and response generation used generate-retrieve-generate pipelines, which outperformed retrieve-then-generate approaches.
Source: §6 · p. 11Takeaways & Limitations
The first iKAT edition provides resources for studying personalized conversational information seeking and shows that pipeline strategy differed in passage ranking and response generation versus PTKB statement ranking.
Source: §5.2.1 · p. 9, §6 · p. 11The response evaluation required human assessments to provide an independent evaluation.
Source: §5.3 · p. 11
Abstract
from arXiv · showhide
Conversational Information Seeking has evolved rapidly in the last few years with the development of Large Language Models providing the basis for interpreting and responding in a naturalistic manner to user requests. iKAT emphasizes the creation and research of conversational search agents that adapt responses based on the user's prior interactions and present context. This means that the same question might yield varied answers, contingent on the user's profile and preferences. The challenge lies in enabling Conversational Search Agents (CSA) to incorporate personalized context to effectively guide users through the relevant information to them. iKAT's first year attracted seven teams and a total of 24 runs. Most of the runs leveraged Large Language Models (LLMs) in their pipelines, with a few focusing on a generate-then-retrieve approach.
1 INTRODUCTION
iKAT focuses on conversational search agents that adapt answers to a person’s prior interactions, knowledge, objectives, tastes, and limitations. Its central challenge is guiding users through information relevant to their personalized, often decision-oriented needs.
1 INTRODUCTION
- Alice’s environmental concerns and veganism differ from Bob’s diabetes, nut allergy, and lactose intolerance, so their conversations about milk alternatives would evolve differently.What each person learns and decides about alternatives may vary with their persona.Source: §1 · p. 1
- iKAT asks agents to use personal context to guide users through relevant information, including when they compare options or weigh pros and cons.Different personas can produce different conversation trajectories because similar questions may call for different answers.Source: §1 · p. 1
- The track represents user-specific context with a Personal Text Knowledge Base and investigates context navigation, personalization, elicitation, and dependence on prior context and responses.The supplied passages state these research questions in fragments.Source: §1 · p. 1
2 TRACK, TASKS, DATA, AND RESOURCES
iKAT 2023 provides tasks, topics, a passage collection, baselines, and relevance and response assessments for studying personalized conversational search. Systems use the user’s PTKB alongside conversational context to retrieve relevant passages and generate responses.
2.1 Track and Tasks
- The track combines statement ranking, passage ranking, and response generation, with user information in the PTKB informing retrieval and answers.Responses should be fluent, meet the information need, avoid redundant material, and may summarize relevant passages.Source: §2.1 · p. 2
2.2 Topics
- The 25 test topics and 11 training topics include one to three persona-specific conversations per topic; a personalized turn has at least one relevant PTKB statement.Topic creation guidelines included persona-, turn-, and conversation-level quality checks, with topics checked and refined by multiple experts.Source: §2.2 · p. 2
2.3 Collection
- The collection subset contains 116,838,987 passages, selected from ClueWeb22-B after manual domain inspection prioritizing domain diversity and excluding irrelevant domains.Documents were shortened to 10,000 characters and segmented with sliding windows of 10 sentences and a five-sentence stride.Source: §2.3 · p. 3
2.4 Baselines
- Four organizer baselines include BM25+RM3 with manual or automatic query and PTKB selection, a multi-step LLaMA-2 pipeline, and ColBERT retrieval with LLaMA-2 summarization.The manual baseline appends three manually selected PTKB statements to the resolved query and uses the top three passages to construct a response.Source: §2.4 · p. 3
2.5 PTKB Statement Relevance Assessment
- PTKB relevance was assessed by both organizers and NIST assessors, but NIST judged a smaller pool of turns selected for passage relevance.Organizers annotated every turn, with a third annotator resolving disagreements by majority vote.Source: §2.5 · p. 3
2.6 Passage Retrieval Assessment
- NIST assessors judged passages for 176 of 326 turns, using a pool of 26,159 passages and an average of 147 judgments per turn.Clarification turns and overly general utterances were among those left unassessed.Source: §2.6 · p. 3
2.7 Response Quality Assessment
- Response quality was assessed with GPT-4 for groundedness and naturalness on selected personalized turns, after filtering low-quality responses and using the teams’ indicated passages or the top five.The chosen prompting setup had the highest agreement with a subset of organizer labels.Source: §2.7 · p. 4
3 EVALUATION
The evaluation uses ranking metrics for PTKB statement selection and passage retrieval, and GPT-4 assessments for generated response groundedness and naturalness.
3 EVALUATION
- PTKB statement ranking is evaluated at turn and conversation levels using nDCG@3, P@3, Recall@5, and MRR..Source: §3 · p. 4
- Passage ranking is assessed by mean nDCG@5 across turns, with uniform turn weights, at ranking cutoffs 3 and 5 and varying conversation depths.Additional metrics include nDCG, nDCG@3, P@10, Recall, Recall@10, and mean average precision.Source: §3 · pp. 4–5
- Generated responses are assessed with GPT-4 for groundedness and naturalness, averaged across turns.Groundedness concerns attribution to the passages used to generate a response.Source: §3 · p. 5
4 PARTICIPANTS
Seven groups submitted 24 main-task runs, most using LLMs and following retrieve-then-generate or generate-retrieve-generate pipelines. Many retrieve-then-generate systems use multiple stages for PTKB selection, query rewriting, retrieval, and reranking.
4 PARTICIPANTS
- The main task received 24 run submissions from seven groups, alongside four organizer baselines: two automatic and two manual.Participants supplied run metadata and descriptions.Source: §4 · p. 5
4.1 Participant Runs
- Most participant runs used LLMs and followed either retrieve-then-generate or generate-retrieve-generate pipelines.A common multi-step retrieve-then-generate approach predicts relevant PTKB statements, rewrites or expands the query, retrieves passages, then reranks them with neural language models.Source: §4 · p. 5
- Some runs retrieved and reranked passages using the top two relevant PTKB statements, while others generated an initial response before selecting or summarizing passages.The described approaches include Pyserini retrieval with LLM reranking and Georgetown runs that use generated responses to identify passages.Source: §4.1 · p. 5
5.1 Passage Ranking
Automatic runs using generate-then-retrieve-then-generate (G→R→G) tended to outperform retrieve-then-generate (R→G) runs in passage ranking. Performance varied by topic and conversation depth, with PTKB-dependent turns usually lower-performing and the largest gap appearing early.
5.1.1 Overall results.
- G→R→G runs tended to outperform R→G models in passage ranking.The top four automatic submissions used G→R→G.Source: §5.1.1 · p. 9
5.1.2 Performance per dialogue.
- Topics 12-1 and 21-1 were easiest, while 19-1 and 20-2 were most difficult for passage ranking.The comparison covers runs with median-or-better average nDCG@5.Source: §5.1.2 · p. 9
5.1.3 Performance at different depths.
- Passage-ranking performance generally fell as conversation turns progressed, with some mid-conversation peaks; PTKB-dependent turns were lower-performing in most cases, and their largest gap was at the first turn.The authors identify predicting the right PTKB statements early as important, while noting PTKB dependence did not always lower performance.Source: §5.1.3 · p. 9, Figure 7 · p. 9
5.2 PTKB Provenance
PTKB statement-ranking results differed from passage-ranking results: G→R→G models were not the top runs, while zero-shot Llama achieved the best result under both assessment sets. Topic difficulty and turn-depth patterns also varied.
5.2.1 Overall results.
- Zero-shot Llama achieved the best PTKB statement-ranking result under both NIST and organizer assessments, while G→R→G models were not the top runs.The authors note that this contrasts with G→R→G success in passage ranking.Source: §5.2.1 · p. 9
5.2.2 Performance per dialogue.
- Topic 20-2 was most challenging for PTKB statement ranking, while 16-1 was among the easiest; the authors found no clear correlation with passage-retrieval performance.These topic-level comparisons use organizer assessments.Source: §5.2.2 · pp. 9–10, Table 5 · p. 10, Table 6 · p. 10
5.2.3 Performance at different depths.
- PTKB statement-ranking performance had high variance at greater conversation depths and peaked on average at turn 15, suggesting less depth dependence than passage ranking.The reported averages cover median-or-better submissions under organizer assessments.Source: §5.2.3 · p. 10, Figure 8 · p. 10
5.3 Response Evaluation
GPT-4-based models outperformed other models by a large margin in the response evaluation, but the authors caution that GPT-4 assessment may bias those results. They call for human assessments to provide an independent evaluation.
5.3 Response Evaluation
- GPT-4-based models outperformed other models by a large margin in the response evaluation.The evaluation assessed response groundedness and naturalness.Source: §5.3 · p. 10, Table 7 · p. 11
- Because GPT-4 also assessed the results, the authors warn that the apparent advantage may be biased and should not be taken at face value.They state that human assessments are required for an independent evaluation.Source: §5.3 · pp. 10–11
6 CONCLUSION
The first TREC iKAT edition developed resources for studying personalized conversational information seeking. Its results show G→R→G approaches outperforming R→G approaches in passage ranking and response generation, with the top four automatic submissions based on G→R→G.
6 CONCLUSION
- The first TREC iKAT edition developed resources for studying personalized conversational information seeking and added to the community’s understanding of the topic.The track focused on more personalized and complex conversations requiring advanced reasoning and use of personal knowledge graphs.Source: §6 · p. 11
- In passage ranking and response generation, R→G approaches were outperformed by G→R→G approaches, and all top four automatic submissions used G→R→G.The authors describe this as a shift toward generating answers from an LLM’s internal knowledge, grounding them through retrieval, and then generating the final response.Source: §6 · p. 11