Source-linked AI summary
MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations
Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, Marina Danilevsky
TL;DR
Multi-turn RAG lacks sufficient evaluation of questions that are unanswerable, underspecified, non-standalone, or require clarification. MTRAG-UN introduces a 666-task benchmark built from human-generated conversations and finds that retrieval and generation models continue to struggle with these challenges. Its scope is limited to English conversations in six closed domains created by a small annotator pool.
Problem
Existing multi-turn RAG evaluation did not separately cover several difficult cases, including unanswerable, underspecified, non-standalone questions and unclear responses.
Method
MTRAG-UN constructs 666 evaluation tasks from 666 human-generated conversations, with an average of 8 turns per conversation, and evaluates retrieval, generation, and the full RAG pipeline.
Results
Retrieval and generation models continue to struggle with unanswerable, underspecified, non-standalone, and unclear-response conversations, with performance lower than target answers and reference settings outperforming RAG settings.
Takeaways & Limitations
MTRAG-UN highlights ongoing challenges in multi-turn RAG and is released to encourage advances on these evaluation settings.
Takeaways & Limitations
The conversations are limited to English and 6 closed domains, were created by a small set of human annotators, and may contain biases toward those individuals.
Abstract
from arXiv · showhide
We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models. We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora. Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses. Our benchmark is available at https://github.com/IBM/mt-rag-benchmark
1 Introduction
MTRAG-UN targets multi-turn RAG situations where questions are unanswerable, underspecified, non-standalone, or require clarification. The benchmark addresses these challenges with new conversation types and evaluates retrieval, generation, and the full RAG pipeline.
- MTRAG-UN focuses on user goals that cannot be achieved through a single question-response exchange with an LLM.
- An underspecified fast-food question illustrates how plausible candidate entities can require the model to ask the user for clarification.
- Frontier models struggle with these tasks by answering from plausible but assumed interpretations, and the difficulties persist in both retrieval and generation.
- The benchmark introduces UNanswerable, UNderspecified, NONstandalone, and UNclear-response challenges in multi-turn RAG.
- 666 tasks evaluate Retrieval, Generation, and the full RAG pipeline across multi-turn conversations.
2 Benchmark Creation
MTRAG-UN constructs 666 multi-turn evaluation tasks from human-generated conversations across six document corpora, targeting answerability, ambiguity, standalone context, and clarification challenges.
- Data Sources: The benchmark contains human-generated conversations averaging 8 turns, with reference passages drawn from six document corpora.The six corpora include four inherited from MTRAG and new Banking and Telco corpora.
- Task Definitions: UNanswerable questions lack relevant retrieved passages, while UNderspecified questions lack enough information to determine a single clear intent.Annotators were instructed to include at least two unanswerable questions per conversation.
- Task Definitions: NONstandalone questions require prior conversational turns, whereas UNclear responses address clarification after users misunderstand or disagree with an earlier answer.Both categories target challenges specific to multi-turn interaction.
- Document Corpora: Banking and Telco add enterprise content to the benchmark, using roughly 1K crawled web pages per corpus with interconnected pages for complex conversations.The pages come from banking and telecommunications companies.
- Task Construction: 666 evaluation tasks are created by selecting one conversation turn and retaining the conversation up to that question.Tasks preferentially select challenging UN-turns, including underspecified questions.
3 Evaluation
Evaluation measures retrieval and generation under reference and RAG settings using answerability-aware metrics. Results show persistent multi-turn difficulty, especially when retrieval noise or ambiguous intent is involved.
- Retrieval: Retrieval experiments evaluate 468 answerable and partially answerable questions using lexical, sparse, dense, and query-rewrite systems.Query Rewrite outperforms the last turn, and Granite English R2 is compared with BGE-base 1.5.
- Retrieval: Banking and Telco achieve .32 and .39 R@5 respectively, below the .52 R@5 average for the other domains.Longer documents, denser links, and structurally similar pages likely increase retrieval difficulty.
- Generation Setup: Generation uses reference passages or the top 5 Elser passages retrieved with query rewrite, with an added instruction to request clarification for underspecified questions.The prompt also requires responses to remain grounded and under 150 words.
- Generation Results: Generation scores remain below target answers, reference performance consistently exceeds RAG, and GPT-OSS-120B achieves the best scores.The reference–RAG gap reflects the added difficulty of retrieval noise; larger models usually perform better within families.
- Generation Results: Models perform poorly on underspecified questions and clarification questions, while DeepSeek-V3 and GPT-OSS more often respond with IDK on unanswerables.Clarification performance is lower than follow-up performance, indicating difficulty with intent refinement and self-correction.
- Generation Results: Performance across Banking and Telco is largely comparable, while other domains trend lower because of the challenging FiQA corpus.The domain comparison is reported in Figure 3c.
4 Conclusion and Future Work
MTRAG-UN highlights ongoing challenges in multi-turn RAG and releases a 666-task benchmark to encourage further advances.
- Conclusion: The benchmark and its baseline results highlight existing and ongoing challenges in multi-turn RAG.The authors plan to release multilingual RAG conversations in future work.
Limitations
MTRAG-UN is limited to English and six closed domains, with conversations created by a small annotator pool and using Elser and Mixtral 8x7b for initial retrieval and generation.
- The benchmark covers only English and six closed domains.Its conversations were created by a small set of human annotators, which may introduce annotator-specific biases.
- The initial retrieval and generation process used the Elser retriever and Mixtral 8x7b generator.The paper identifies these components as another potential source of bias in the conversations.
- The authors suggest expanding the annotator pool and adding other languages to address these limitations.
A Stats and Metrics
The evaluation includes a clarification judge, whose agreement with human judgments was checked after replacing GPT-4o-mini with GPT-OSS-120B. The open-source judge produced slightly improved or consistent correlations.
- The clarification judge outputs “yes” for clarification responses and “no” otherwise.
- A clarification judge determines whether a response asks for additional information to resolve ambiguity or underspecification.
- GPT-OSS-120B was evaluated as a replacement for GPT-4o-mini as the judge through a repeated correlation analysis.The analysis examined agreement between each judge and human judgments.
- The open-source judge's correlation with human judgments improved slightly or remained consistent relative to the proprietary judge.
B Details on UNderspecified
For underspecified questions, MTRAG-UN uses reference passages and target responses that ask users for more information. Responses are organized by the number and openness of plausible interpretations.
- Figure 1 illustrates an underspecified final turn, reference passages, and a target response asking the user for more information.The example concerns a vague fast-food chain in the United States.
- For two or three plausible options, “hedging with answers” lists the options and briefly describes or answers each.
- For four to eight plausible options, “hedging over list” enumerates the options without additional explanation.
- For many or unbounded options, the open-domain response asks the user to disambiguate the entity type they mean.
- Table 5 categorizes the response types used for underspecified questions.
B.1 Stitching of the underspecified questions
Underspecified questions were added as final turns to existing human-annotated multi-turn conversations through controlled topic-based stitching.
- The stitching process attached a human-written underspecified final turn to an existing human-annotated multi-turn conversation.
- 75% of underspecified tasks used existing conversations on the same or a very similar topic, keeping the new turn contextually aligned.
- The second stitching condition used an existing conversation on a different topic.This condition tests a final underspecified question that is not topically aligned with the preceding conversation.
B.2 Validation
Underspecified questions underwent careful validation to ensure that conversational context did not accidentally clarify the final turn’s intent.
- Underspecified questions were carefully validated before inclusion.Validation included filtering, editing the final turn or reference response, and, most often, direct validation.
- Filtering removed cases where added conversational context accidentally clarified the last turn’s intent.
- Validation could involve editing either the last turn or its reference model response.