Source-linked AI summary
MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, Marina Danilevsky
TL;DR
Multi-turn RAG is under-evaluated despite challenges involving later, non-standalone, and unanswerable questions. The paper introduces and evaluates MTRAG, a human-generated benchmark spanning diverse domains and conversation properties. Its evaluations show that current RAG systems struggle on these tasks, while synthetic data and automated evaluation remain areas for further study.
Problem
Multi-turn RAG has been largely overlooked by benchmarks, despite presenting challenges not covered by single-turn evaluation.
Method
The paper constructs and evaluates MTRAG, a human-generated multi-turn benchmark across four domains, and explores synthetic data and automated evaluation paths.
Results
Human evaluation demonstrates that all evaluated models struggle on MTRAG, especially on unanswerable questions and later turns.
Takeaways & Limitations
MTRAG provides an end-to-end benchmark for analyzing retrieval and generation challenges in realistic multi-turn RAG conversations.
Takeaways & Limitations
Synthetic generation produced few successful unanswerable questions, while longer conversations increased repetition and hallucinated responses.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.
1 Introduction
MTRAG addresses the largely overlooked challenge of evaluating retrieval-augmented generation in multi-turn conversations. It introduces a human-generated, diverse benchmark and evaluates retrieval, generation, and automation strategies.
- Motivation: Multi-turn RAG has been largely overlooked by benchmarks despite introducing challenges beyond single-turn RAG.These challenges include changing relevant passages, repeated retrieval, and questions that rely on previous turns.
- Benchmark: MTRAG is a human-generated multi-turn benchmark spanning four domains, varied question types, multi-turn patterns, and answerability conditions.Annotators interacted with a live RAG agent, revised retrieved passages, and repaired responses to create natural conversations.
- Evaluation: The study evaluates lexical, sparse, and dense retrieval plus nine LLMs under multiple retrieval settings with human response evaluation.Retrieval is examined using last-turn and query-rewrite settings, while generation is tested with reference, reference-plus-RAG, and full-RAG inputs.
- Findings: Human evaluation shows that all evaluated models struggle, particularly on unanswerable questions and later conversation turns.The benchmark also explores automated metrics and introduces the synthetic MTRAG-S companion benchmark.
2 Related Work
Prior multi-turn conversation datasets do not generally evaluate an actively changing retrieval component. MTRAG instead evaluates end-to-end retrieval and generation over evolving conversations.
- Existing work: Earlier work covered extractive, long-form, open-domain QA, and multi-turn conversation datasets, but multi-turn RAG remained comparatively underdeveloped.MT-Bench is identified as a multi-turn conversation dataset, while surveys summarize related approaches and datasets.
- Existing work: Except for FaithDial, prior multi-turn RAG datasets generally keep retrieval fixed or use it only to find evidence for an existing conversation.Fixed retrieval can restrict a conversation to initial passages, while post hoc retrieval can create mismatches between answers and verifiability.
3 MTRAG Benchmark
MTRAG is designed as a diverse and challenging benchmark for realistic multi-turn RAG conversations. Its construction varies question content, answerability, domains, passage relevance, and response quality.
- Question dimensions: MTRAG includes factoid, comparison, explanation, keyword, and other question types, with one or more labels per question.
- Answerability: Conversations include answerable, partially answerable, unanswerable, and conversational statements based on the document corpora.
- Domains: The benchmark covers four domains and corresponding document corpora to test RAG across different document types.
- Response quality: Reference answers are designed for faithfulness, appropriateness, naturalness, and completeness, summarized as FANC.
- Challenge properties: Conversations average 16.9 unique relevant passages and 20.9 relevant passages overall, while annotators repair responses on 92% of turns.
4 Benchmark Creation
MTRAG is built from four document domains and human-authored multi-turn conversations created with an interactive RAG application. The resulting benchmark contains 110 conversations averaging 7.7 turns and 842 task-level evaluations.
- Corpus construction: MTRAG spans four domains: CLAPNQ, FiQA, Govt, and Cloud.CLAPNQ and FiQA are existing QA/IR corpora, while Govt and Cloud were assembled specifically for the benchmark.
- Corpus construction: Govt and Cloud contain interconnected pages to support conversations that touch several passages.The two newly assembled corpora were designed for diverse multi-passage conversations.
- Conversation creation: Annotators created conversations by writing questions, adjusting retrieved passages, editing responses, and enriching questions with dimensions.They used a custom chat application with an ELSERv1 retriever and Mixtral 8X7b Instruct generator, and were directed to write 6–9 or more turns.
- Quality review: Human evaluation found that reference responses were preferred by humans.This result is reported after the conversation creation and review process.
- Benchmark statistics: 110 conversations across the four domains average 7.7 turns and produce 842 task-level evaluations.Each task includes all preceding conversation turns together with the current user question.
5 Retrieval
MTRAG evaluates retrieval in RAG systems using lexical, dense, and sparse retrievers and reports Recall and nDCG at multiple cutoffs. Query rewriting improves retrieval broadly, while later-turn and non-standalone questions remain challenging.
- Evaluation setup: MTRAG evaluates lexical BM25, dense BGE-base 1.5, and sparse Elser retrieval models.These models are evaluated using the indexed corpora from benchmark creation.
- Evaluation setup: Retrieval is measured with Recall and nDCG at @1, 3, 5, and 10.The retrieval task uses reference passages and is computed only for answerable and partially answerable tasks with reference passages.
- Evaluation caveat: Elser may receive an advantage because it was used for retrieval during benchmark data creation.This potential bias is identified as a limitation of the evaluation.
- Query strategies: Using the full conversation or several turns under-performed, while querying with only the last user turn was most effective before rewriting.Longer conversation queries often caused retrievers to return the same passages repeatedly.
- Results: Query rewriting consistently outperforms using only the last turn without rewriting across all metrics and models.Elser also outperforms BM25 and BGE-base 1.5 in the reported retrieval results.
- Results: Retrieval performance is significantly lower for later turns than for the first turn, and non-standalone questions remain challenging despite query rewriting.The domain, turn position, and standalone status breakdown is reported for Elser with query rewriting.
6 Generation
The generation experiments evaluate nine autoregressive models across retrieval settings and answerability, turn, and domain dimensions. Results show broad difficulty, especially with noisier retrieval, unanswerable questions, and later turns.
- Experimental setup: Nine autoregressive models are evaluated across Reference, Reference + RAG, and RAG retrieval settings.The setup supplies questions, preceding turns, and five passages; Reference simulates a perfect retriever, while Reference + RAG supplements references with retrieved passages.
- Evaluation metrics: RBalg combines Bert-Recall, Bert-K-Precision, and Rouge-L to approximate completeness, faithfulness, and appropriateness.RBllm is a reference-based LLM judge, while RLF is a reference-less faithfulness judge.
- Answerability conditioning: An IDK judge with over 97% accuracy detects full or partial answers before conditioning RBalg, RBllm, and RLF scores on answerability.Answerability accuracy measures how accurately IDK predicts the answerability value.
- Scope: Only 10 conversational tasks are excluded from the experiments and left for future work.The exclusion reflects special evaluation handling for conversational questions.
- Overall results: All models score below the reference answer, and generation performance degrades as retrieval becomes noisier.GPT-4o and Llama 3.1 405B Instruct generally perform best, while Qwen 2.5 72B and Command-R+ are relatively competitive in noisy settings.
- Results by dimension: Performance generally drops on partially answerable and unanswerable questions, with unanswerables producing especially dramatic declines.Models also usually perform better on first-turn than subsequent-turn questions, while FiQA tends to yield lower results than other domains.
7 Human Evaluation
Human evaluation compares GPT-4o and Llama 3.1 405B Instruct with human reference answers across FANC properties and win-rate. References are strongly preferred, while models remain weaker on faithfulness and completeness.
- Evaluation design: Human evaluation covers 159 tasks from 20 conversations, comparing GPT-4o and Llama 3.1 405B Instruct with reference answers.Annotators rate Faithfulness, Appropriateness, Naturalness, and Completeness from 1 to 4 and provide pairwise win-rate comparisons.
- Results: Reference answers are exceedingly preferred over model responses and receive the highest score on most individual properties.The results support the quality of the human-generated references while indicating room for frontier-model improvement.
- Results: Models perform well on naturalness and appropriateness but struggle more with faithfulness and completeness.Both frontier models are equally preferred for answerable questions.
- Unanswerable questions: Llama 3.1 405B Instruct is least preferred on the few unanswerable questions because it continues answering them.The authors characterize these responses as hallucinations.
8 Automatic Evaluation
The paper investigates whether automated metrics reflect human judgments in multi-turn RAG. Faithfulness-focused metrics correlate well, while some metrics are poorly suited to non-standalone questions or favor machine-generated text.
- Motivation: Automated evaluation is explored because human evaluation does not scale easily for long-term model assessment.The analysis compares reference-based and reference-less algorithmic metrics and LLM judges against human win-rate.
- Metric correlation: Bert-K-Prec and RAGAS Faithfulness correlate well with human win-rate because they focus on faithfulness.The paper contrasts these metrics with alternatives that show weaker alignment.
- Metric limitations: Some metrics are unsuitable for multi-turn evaluation because non-standalone questions complicate their question-reconstruction procedure.RLMTB has very low correlation with win-rate and favors machine-generated text.
9 Synthetic Conversations
MTRAG-S extends an existing framework to generate synthetic multi-turn conversations over the same corpora as MTRAG. Synthetic data differs structurally from human data and reproduces some challenging properties poorly.
- Construction: MTRAG-S is a companion benchmark of synthetically generated conversations built to explore scalable alternatives to manual data creation.It extends a previously proposed framework for automatically generating multi-turn conversations over the same corpora.
- Comparison with human data: Synthetic conversations average 5.9 turns and 4.6 unique passages, compared with 7.7 turns and 16.9 unique passages for human conversations.Synthetic questions are longer, while synthetic answers are shorter.
- Generation challenges: Synthetic generation often fails to produce genuinely unanswerable questions, instead creating questions with at least partial answers.Increasing the number of turns also tends to produce repetitive user questions and more hallucinated agent responses.
- Evaluation: Models receive higher faithfulness scores on synthetic than human-generated data under both RLF and Bert-K-Prec.The authors give multiple possible explanations, including synthetic-data idiosyncrasies and automatic-metric reliability or bias.
10 Conclusions and Future Work
MTRAG is a comprehensive human-generated benchmark for evaluating the full multi-turn RAG pipeline across diverse domains and conversation dimensions. Its findings identify persistent challenges in retrieval, generation, evaluation, and scaling, motivating future extensions and improved methods.
- Benchmark and contribution: MTRAG contains 110 human-generated conversations averaging 7.7 turns, totaling 842 tasks across four domains.The benchmark is designed to test the full RAG pipeline and is publicly available.
- Conclusions and future work: The authors call for better reference-less automatic metrics, more synthetic conversations, and stronger retrieval and generation for longer, unanswerable, and non-standalone questions.They are also motivated to extend MTRAG with adversarial turns, additional domains, and multilingual conversations.
- Benchmark construction: Review rejected 16 of 126 conversations and repaired 264 responses, increasing the final number of repaired responses from 778 to 799.Rejections reflected insufficient turn-wise or topic-level coherence and passage diversity.
- Evaluation and methods: The evaluation framework adapts query rewriting and LLM judges to multi-turn RAG, incorporating retrieved passages and assessing faithfulness, appropriateness, and completeness.Generation prompts include the current question, preceding turns, passages, and instructions; the MT-Bench adaptation focuses on the last user turn.
- Benchmark and contribution: The benchmark varies question type, multi-turn type, answerability, passage diversity, and conversation repair.Questions include follow-ups and clarifications, while responses may be answerable, unanswerable, or partial.
F.3 IDK correction to the metrics
This section examines IDK-based correction and synthetic-data evaluation. The IDK judge is highly accurate, while synthetic conversations produce higher faithfulness scores than human-generated MTRAG data.
- IDK correction: Over 97% accuracy was achieved by the IDK judge when evaluated against human-evaluation results across model responses and reference answers.The judge determines whether a response corresponds to an I-Don’t-know answer and is applied to unanswerable questions.
- Synthetic benchmark: MTRAG-S generates synthetic conversations with a question-type classifier and Mixtral 8x22B Instruct, restricting conversations to a maximum of 8 turns.The framework extends prior automatic conversation-generation work and targets the question types used in MTRAG.
- Synthetic benchmark: Models receive higher faithfulness scores on synthetic MTRAG-S than on human-generated MTRAG under both RLF and BERT-K-Prec.The comparison is reported across models using either faithfulness metric.
- Additional analyses: The appendix reports detailed retrieval results for Elser by domain and generation results broken down by answerability, turns, and domains.Additional generation analyses also use RLF, RBllm, and RBalg.
I.2 Generation results by question type and multi-turn type
Generation performance is further analyzed by question type and multi-turn type in the Reference retrieval setting. The results use RBalg, RBllm, and RLF across these breakdowns.
- Question type: Generation results are broken down by question type using RBalg, RBllm, and RLF in the Reference retrieval setting.The analysis is presented in Figure 9.
- Multi-turn type: Generation results are broken down by multi-turn type using RBalg, RBllm, and RLF in the Reference retrieval setting.The analysis is presented in Figure 10.