PaperlayerView exact revision on arXiv

Source-linked AI summary

Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG

Pranav Handa, Ariful Azad

arXiv:2609.36700v1cs.CLcs.AI

TL;DR

Existing RAG evaluations largely use fully specified single-turn questions, leaving conversational delivery without a matched comparison. This study converts multi-hop QA questions into underspecified conversations and evaluates LLMs with retrieval systems at scale. Multi-turn interaction lowers answer quality and reliability, with failures arising in query reformulation or evidence synthesis.

  • Problem

    RAG is evaluated almost exclusively on single-turn, fully specified questions, so existing benchmarks cannot quantify answer-quality loss when the same information is revealed gradually.

    Source: §1 · p. 1

  • Method

    The study transforms multi-hop QA questions into conversations and evaluates ten LLMs with retrieval systems across 1.5 million simulated conversations.

    Source: §1 · p. 2

  • Results

    11.7% average answer-quality loss accompanies multi-turn delivery, with degradation also increasing unreliability and arising from query reformulation or failed evidence synthesis.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Single-turn results alone do not establish conversational robustness, and some retrieval systems recover much of the gap when earlier-turn information is repeated.

    Source: §1 · p. 2, §6 · pp. 9–10

Abstract

from arXiv · show

When conversing with large language models (LLMs), users often begin with a simple question and build towards a multi-hop question through follow-up turns. Retrieval-augmented generation (RAG) and its graph-based variant (GraphRAG) have become the dominant approaches for grounding LLM responses in external evidence, yet both are evaluated almost exclusively on single-turn, fully specified queries. We systematically investigate this evaluation mismatch through a large-scale simulation study. Building on prior work on multi-turn LLM evaluation, we transform questions from multi-hop question answering (QA) benchmarks into underspecified conversations and evaluate ten LLM assistants with eight retrieval systems across 1.5 million simulated conversations. Our findings reveal that multi-turn interaction causes widespread performance degradation, incurring relative performance drops of up to 21% and increasing unreliability by 47%, making RAG systems simultaneously less accurate and less reliable. We identify two distinct failure modes behind this degradation. Systems are either lost in translation, where conversational rephrasing distorts the retrieval query, or lost in conversation, where retrieval succeeds but the LLM fails to synthesize evidence distributed across turns.

1 INTRODUCTION

This paper measures how RAG answer quality changes when multi-hop questions are delivered through conversation rather than upfront, addressing the lack of matched single-turn comparisons. Across large-scale simulations, multi-turn delivery reduces performance and reliability, through failures in query reformulation or evidence synthesis; a lightweight intervention partly recovers performance for some retrievers.

1 INTRODUCTION

  • Matched fully specified questions are missing from existing QA evaluations, while current conversational benchmarks cannot quantify degradation caused by conversational delivery.The study uses the original question as a single-turn reference and compares fully specified, concatenated-clue, and incrementally sharded conditions.Source: §1 · pp. 1–2
  • 11.7% average answer-quality loss, reaching 21% in the hardest-hit settings, accompanies two failure modes: distorted conversational queries and failure to synthesize retrieved evidence across turns.Query reformulation losses are especially pronounced for retrievers operating directly on query text or embeddings.Source: §1 · p. 2
  • 47% greater run-to-run unreliability shows that multi-turn interaction affects consistency as well as answer quality.The strongest runs remain relatively stable, and strong single-turn performance does not guarantee conversational robustness.Source: §1 · p. 2
  • A lightweight, training-free intervention that repeats earlier-turn information nearly erases the gap for a subset of retrievers sharing a preprocessing step.The reported full gain is specific to these retrieval systems and does not appear in the LLM-only setting.Source: §1 · p. 2
  • The study transforms 750 multi-hop QA questions from five benchmarks into manually reviewed conversations and evaluates ten LLMs with seven retrieval methods and a closed-book baseline.The broader evaluation comprises 1.5 million simulated conversations.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Prior work studies multi-turn performance and conversational retrieval, but benchmarks generally lack a matched fully specified version of the same question. Comparing matched single- and multi-turn tasks helps separate conversation effects from task difficulty.

2 BACKGROUND AND RELATED WORK

  • Existing conversational-RAG benchmarks evaluate systems only in multi-turn settings, making it difficult to separate conversation-related errors from underlying task difficulty.They do not pair each dialogue with a fully specified version of the same question.Source: §2 · p. 3
  • Conversational reformulation and evolving-dialogue benchmarks bring RAG evaluation closer to conversational use, but do not establish the penalty against matched single-turn questions.The cited work includes retrieval over evolving dialogues and rewriting context-dependent queries for retrieval.Source: §2 · p. 3
  • Earlier matched-question studies show that strong single-turn performance need not transfer to multi-turn settings, while Laban et al. (2026) report a 39% average drop for LLM-only generation tasks.Those studies do not include retrieval, where conversational delivery can also change which evidence is retrieved.Source: §2 · p. 3

3 CONSTRUCTING MULTI-TURN EVALUATIONS FROM MULTI-HOP QA

The evaluation turns multi-hop QA questions into conversations that reveal clues sequentially, while preserving the original question as a matched reference. It compares information-delivery conditions and retrieval policies within a simulated assistant interaction.

3 CONSTRUCTING MULTI-TURN EVALUATIONS FROM MULTI-HOP QA

  • Each evaluation starts with a fully specified multi-hop question, decomposes it into ordered shards, and reveals one additional clue at a time.The shards collectively preserve the original question’s information, while later clues can refer to entities introduced earlier.Source: §3 · p. 3

3.1 DATASETS

  • The dataset covers HotpotQA, 2WikiMultiHopQA, and MuSiQue at 2-, 3-, and 4-hop depths, with 150 reviewed questions per subset.The resulting 750 questions are evaluated against fixed 1,000-question retrieval corpora; deeper questions yield longer conversations.Source: §3.1 · pp. 3–4

3.2 FROM A FULL QUESTION TO A CONVERSATION

  • A GPT-4o-mini module segments each question into information units and rephrases them as conversational turns; shard sets are automatically verified and manually inspected.Shards follow compositional order and must preserve original entities and relations without adding facts or directly exposing intermediate answers.Source: §3.2 · pp. 4–5

3.3 SIMULATION TYPES

  • The simulation compares information-delivery conditions using the same underlying information, with FULL versus SHARDED as the central comparison and CONCAT separating formulation from turn-delivery effects.Closed-book CONCAT must retain at least 0.8 of FULL F1 to pass the information-preservation check; Figure 1 depicts retrieval and answering at a conversation turn.Source: Figure 1 · p. 4, §3.3 · p. 5

4 EXPERIMENTAL SETUP

The experiments compare retrieval systems and language models over shared evidence, while measuring answer quality and run-to-run variation. The setup spans passage, hierarchical, and graph retrieval, with a fixed evaluation protocol.

4.3 EVALUATION METRICS

  • Table 1 presents five information-delivery conditions, with each box representing one user turn.Table 2 reports FULL and SHARDED F1 averaged over ten LLMs, with shading indicating the direction and magnitude of change relative to FULL.Source: Table 1 · p. 5, Table 2 · p. 6

4.1 RETRIEVAL METHODS

  • Eight retrieval settings use a shared corpus: closed-book, BM25, dense RAG, RAPTOR, HippoRAG, HippoRAG2, ToG-2, and LightRAG.Graph retrievers use an LLM-based OpenIE graph built from the same passage corpus; methods retain their respective retrieval units and output rules.Source: §4.1 · p. 5

4.2 LANGUAGE MODELS

  • Ten LLMs span four model families and 4B to 70B+ parameters, including API and open-weight models.Prespecified API-model coverage and open-weight models provide broader coverage because evaluating API models across all combinations would be prohibitively expensive.Source: §4.2 · p. 5

4.3 EVALUATION METRICS

  • Each question is simulated five times to account for stochastic answer variation while keeping the larger model-by-retriever evaluation tractable.Answers are scored using normalized exact match and token-level F1, with conversation-level BEST defined as the maximum F1 over answer attempts.Source: §4.3 · p. 5

5 RESULTS

Across closed-book and retrieval settings, spreading multi-hop questions across turns lowers answer performance, with the size and source of the loss varying across systems. The analysis separates query reformulation from multi-turn delivery and finds that retrieving supporting passages does not ensure the LLM can use them across turns.

5.1 MULTI-TURN DEGRADATION IS PRESENT WITHOUT RETRIEVAL

  • 17%: closed-book F1 falls from 0.230 under FULL to 0.190 under SHARDED, averaged over ten LLMs and five datasets.For the largest model in each family, CONCAT retains 94.3% of FULL performance, and all five datasets satisfy the information-preservation criterion.Source: §5.1 · p. 6

5.2 RETRIEVAL DOES NOT ELIMINATE MULTI-TURN DEGRADATION

  • 11.7% and 16.3% relative losses: retrieval settings decline by 5.1 F1 points under HISTORY and 7.1 under CURRENT, respectively.Every retrieval method has a positive mean FULL–SHARDED gap under both query policies, despite retrieval improving answer quality over the closed-book setting.Source: §5.2 · p. 6
  • HippoRAG2 has the highest mean FULL score (50.6) but loses 6.8 F1 points under HISTORY and 9.9 under CURRENT; model-level robustness also varies.5.6 Luna loses 5.4 F1 points under HISTORY, while Gemma-4-31B loses 0.7; HISTORY improves mean SHARDED performance for six of seven methods, with BM25 the exception.Source: §5.2 · pp. 6–7

5.3 DECOMPOSING THE GAP: FORMULATION AND MULTI-TURN DELIVERY

  • BM25’s formulation loss is largest at 11.7 F1 points, while HippoRAG, LightRAG, and ToG-2 lose 1.4–2.8; repeated retrieval then partly reverses losses for some methods.BM25 recovers 5.2 F1 points over CONCAT, but every retriever ultimately has a net FULL–SHARDED degradation; SHARDED also falls 1.5 points below CONCAT without retrieval.Source: §5.3 · p. 8

5.4 RETRIEVAL COVERAGE ACROSS TURNS

  • Mean Recall@5 falls from 56.7 under FULL to 49.2 under CONCAT, while HISTORY cumulative recall reaches 55.8, close to the FULL baseline.BM25’s recall rises from 18.6 under CONCAT to 35.3 across accumulated SHARDED retrievals.Source: §5.4 · p. 8
  • Under CURRENT, cumulative recall reaches 61.0 and exceeds FULL for every retriever, yet SHARDED answer performance remains lower.When FULL and SHARDED both retrieve all supporting passages, SHARDED still scores 11.6 F1 points lower under HISTORY.Source: §5.4 · pp. 8–9
  • 47%: HISTORY unreliability rises from 13.4 F1 points under FULL to 19.7 under SHARDED, while estimated aptitude falls by only 1.9 points.Unreliability is the difference between the 90th- and 10th-percentile F1 across repeated simulations; aptitude is the 90th-percentile F1.Source: §5.4 · p. 9
  • Conversation consolidation can fully recover single-turn performance for some retrievers, while semantic-judge evaluation confirms degradation across retrieval methods and query policies.Plan-on-Graph’s HISTORY gap nearly disappears to about 2 F1 points (6%), but it uses more tokens and LLM calls and has the second-lowest overall F1 among tested retrieval methods.Source: §5.4 · p. 9

6 CONCLUSION

The study concludes that RAG does not eliminate multi-turn degradation: systems can lose performance through query reformulation or fail to use retrieved evidence across turns. Its shard-construction procedure reveals question dependencies incrementally and manually reviews generated conversations.

6 CONCLUSION

  • RAG remains less accurate and less reliable across turns; some systems lose performance after reformulation, while others retrieve needed evidence but fail to use it across turns.The authors conclude that multi-turn evaluation is needed to demonstrate robustness rather than assume it from single-turn results.Source: §6 · pp. 9–10

A.1 SHARD CONSTRUCTION AND VALIDATION

  • The benchmark turns each fully specified question into an opening objective shard followed by successive shards, with follow-ups focused on unresolved relations.Separate prompt pairs are used for HotpotQA, 2WikiMultiHopQA, and MuSiQue; dependency order replaces order-insensitivity and maximal-sharding requirements.Source: §A.1 · p. 15
  • Accepted shards preserve the original question’s information and wording while withholding the final answer and revealing dependencies in order.Each follow-up introduces one substantive clue; no specific shard count is targeted.Source: §A.1 · p. 15
  • Every generated shard set was manually reviewed because semi-automatic construction can hallucinate entities, merge dependencies, or combine clues.Reviewers edited violations and removed questions with source errors or missing referents.Source: §A.1 · p. 15

A.2 USER SIMULATION AND RESPONSE CLASSIFICATION

The simulation reveals how shard-based conversations are constructed and classified, then shows that closed-book models differ in their sensitivity to reformulated questions.

A.2 USER SIMULATION AND RESPONSE CLASSIFICATION

  • 98.7% of conversations preserve the constructed dependency order, indicating that simulated shards largely retain the intended reasoning chain.The fixed GPT-4o-mini simulator selects and rewrites at most one remaining shard after each assistant response.Source: §A.2 · pp. 16–17
  • The information in each shard remains fixed while its wording varies; one census shard appeared in 11 distinct forms across 15 simulations.The opening intent was identical in all 15 conversations.Source: §A.2 · p. 17
  • Under HISTORY, retrieval-based SHARDED turns were classified as answer attempts in 44.2% of cases, clarification requests in 40.7%, and discussion in 9.0%.The corresponding CURRENT proportions were 43.8%, 40.9%, and 9.1%.Source: §A.2 · p. 17

A.3 CLOSED-BOOK COMPARISON

  • Llama-3.1-8B-Inst and Qwen3-8B retain below 80% of question information after CONCAT, while Gemma-4-31B-IT and Qwen3.6-27B retain 98% and 97%, respectively.The passage notes that smaller models are particularly vulnerable to surface-level rephrasing.Source: §A.3 · pp. 17–18

B.1 AGENT-STYLE CONSOLIDATION: RECAP AND SNOWBALL

RECAP and SNOWBALL restate previously revealed clues before retrieval, but how much performance they recover depends on whether the resulting query works well for the retriever.

B.1 AGENT-STYLE CONSOLIDATION: RECAP AND SNOWBALL

  • RECAP adds a final turn consolidating all revealed shards; SNOWBALL repeats all earlier shards at every turn.RECAP assumes the ending is known in advance, while SNOWBALL can be used without knowing when the conversation will end.Source: §B.1 · p. 19
  • Under HISTORY, repeated shards appear multiple times in the retrieval query, whereas CURRENT is treated as the standard consolidation test because each shard appears once.HISTORY is used as a complementary setting.Source: §B.1 · p. 19
  • 92–123% of the FULL–SHARDED gap is recovered by RECAP for HippoRAG, LightRAG, and ToG-2, versus 7–39% for BM25, Dense, RAPTOR, and HippoRAG2.The first group returns to approximately FULL performance; the qualitative division also holds under HISTORY.Source: Figure 5 · p. 19, §B.1 · p. 19, Figure 6 · p. 20
  • The recovery split tracks retrieval evidence: HippoRAG retrieves at least as much supporting evidence under RECAP as under FULL, while BM25 remains below CONCAT and its final SHARDED retrieval.The passage contrasts retrievers that transform queries into entities or keywords with those operating directly on query text or embeddings.Source: §B.1 · p. 20
  • Under CURRENT, SNOWBALL recovers 94–117% of the gap for HippoRAG, LightRAG, and ToG-2, and 38–49% for Dense, RAPTOR, and HippoRAG2; BM25 remains below SHARDED.Repeating earlier information helps when the resulting turn is an effective retrieval query.Source: §B.1 · p. 20

B.2 ADAPTIVE GRAPH PLANNING RETRIEVAL: PLAN-ON-GRAPH

Plan-on-graph uses adaptive LLM planning to explore a knowledge graph, but its multi-turn results depend strongly on how much question history is available and come with high retrieval cost.

B.2 ADAPTIVE GRAPH PLANNING RETRIEVAL: PLAN-ON-GRAPH

  • PoG decomposes a query into sub-objectives, explores the graph hop by hop, maintains a running memory, and can reflect or backtrack if evidence is insufficient.The adaptation is evaluated on eight open-weight models because of the planning loop’s per-conversation cost.Source: §B.2 · p. 21
  • Under HISTORY, PoG loses 2.0 F1 points (6.1%) from FULL to SHARDED, but under CURRENT it loses 10.0 points (29.9%), the largest relative decline among retrieval methods.On HotpotQA, SHARDED with HISTORY scores 50.4 versus 49.7 for FULL.Source: Figure 7 · p. 21, §B.2 · p. 21
  • PoG’s FULL score of 33.4 is level with BM25 (33.7) and LightRAG (33.5), and it ranks second to last under SHARDED with HISTORY.Its small HISTORY gap therefore starts from a low absolute performance level.Source: §B.2 · p. 21
  • Each FULL PoG retrieval requires 22.1 LLM planning calls versus 2.5 for ToG-2; over a SHARDED conversation, PoG makes 47.5 calls versus 6.9.PoG’s token overhead is 2.9× ToG-2’s, and its additional LLM usage does not translate into competitive answer quality.Source: §B.2 · p. 22

B.3 APTITUDE AND UNRELIABILITY

The analysis distinguishes high-end answer potential from run-to-run variability and finds that multi-turn retrieval chiefly worsens lower-scoring runs, while unreliability rises broadly across models and retrieval methods.

B.3 APTITUDE AND UNRELIABILITY

  • Aptitude is the 90th-percentile F1 across five simulations, and unreliability is the difference between the 90th- and 10th-percentile F1 scores.The percentiles are linearly interpolated, and the per-question values are descriptive estimates averaged across datasets and compared models or retrieval methods.Source: §B.3 · p. 22
  • With retrieval, the ceiling falls 1.9 points and the floor 8.2, widening unreliability from 13.4 to 19.7; without retrieval, ceiling and floor fall 4.0 and 3.8 points.The retrieval-augmented decline is concentrated in lower-scoring simulations, while the best outcomes remain largely intact.Source: Figure 8 · p. 22, §B.3 · p. 22
  • Under HISTORY, unreliability rises by 5.1–7.8 points for all ten LLMs, while aptitude falls for eight; Gemma-4-31B gains 3.1 aptitude points and Qwen3.6-27B changes by +0.3.Gemma-4-31B’s SHARDED unreliability is 15.0, nearly double its FULL value of 7.6, despite a mean F1 drop of only 0.7 points.Source: §B.3 · pp. 22–23, Table 8 · p. 23
  • Under CURRENT, mean unreliability reaches 19.9 points, and every model and retrieval method is more unreliable than under FULL with either query policy..Source: §B.3 · p. 23

B.4 ANSWERING UNDER COMPLETE SUPPORTING-SOURCE COVERAGE

Among conversations where both conditions retrieved every supporting passage, SHARDED answers scored lower than FULL answers, though the size of the decline varied across LLMs. Complete coverage was uncommon and became rarer as questions required more evidence.

B.4 ANSWERING UNDER COMPLETE SUPPORTING-SOURCE COVERAGE

  • Mean BEST F1 fell from 73.7 under FULL to 62.1 under SHARDED, a difference of 11.6 points (95% paired bootstrap CI [9.9–13.4]).Significant declines appeared on four of five datasets among the qualifying pairs.Source: §B.4 · p. 23, Table 9 · p. 24
  • Complete supporting-source coverage fell from 46.9% of HotpotQA pairs to 0.5% on MuSiQue-4hop as required evidence grew.Only 24.4% of the 225,000 eligible pairs qualified for the paired comparison.Source: §B.4 · p. 23
  • The FULL–SHARDED decline varied nearly fivefold across LLMs, from 4.0 F1 points for Gemma-4-31B to 19.0 for Qwen3-14B.The four smallest models lost 17–19 points and retained about three quarters of FULL performance, while the most robust retained over 90%.Source: §B.4 · p. 24, Figure 9 · p. 24
  • These paired results apply only when both conditions retrieve every supporting passage; outside that subset, differences in retrieved evidence may also contribute to the overall loss.Complete coverage does not eliminate distractor passages, so the assistant must still select and combine relevant evidence.Source: §B.4 · pp. 23–24

B.5 LLM-AS-A-JUDGE EVALUATION

The LLM-judge robustness check finds that the FULL–SHARDED gap is similar under semantic correctness and token F1. The appendix also presents a representative conversation and summarizes the retrieval families evaluated.

B.5 LLM-AS-A-JUDGE EVALUATION

  • The mean HISTORY gap changes from 4.1 points under token F1 to 3.8 under the judge, with per-configuration gaps strongly correlated under HISTORY (r = 0.94) and CURRENT (r = 0.96).No gap reverses from a significant decline to a significant improvement under the bootstrap intervals.Source: §B.5 · p. 25, Figure 10 · p. 25

C QUALITATIVE RETRIEVAL AND CONVERSATION EXAMPLE

D RETRIEVAL SYSTEMS AND CONFIGURATION SUMMARY

  • The evaluated retrieval families span lexical passage retrieval, dense passage retrieval, knowledge-graph traversal, and hierarchical retrieval.The implementation details are summarized separately in Table 11.Source: D RETRIEVAL SYSTEMS AND CONFIGURATION… · p. 27

D.1 RETRIEVAL SYSTEMS •

  • The systems include a closed-book control, BM25 lexical retrieval, Dense TextRAG using BGE-large embeddings, and RAPTOR's hierarchical passage tree.BM25 and Dense TextRAG each return five passages; RAPTOR can retrieve at different abstraction levels.Source: §D.1 · p. 27
  • The graph-based systems are HippoRAG, HippoRAG2, ToG-2, and LightRAG, combining graph structure with query linking, passage context, or language-model guidance.Their specific retrieval strategies differ, from Personalized PageRank propagation to graph connectivity and passage-guided exploration.Source: §D.1 · p. 27

D.2 IMPLEMENTATION DETAILS

The appendices document the benchmark's implementation and prompt scaffolding, from corpus and model references to shard construction, conversation simulation, answering, and evaluation.

D.2 IMPLEMENTATION DETAILS

  • Tables summarize retrieval interfaces and parent corpora, while the model appendix lists evaluated versions and access details.Open-weight models were served locally with vLLM on NVIDIA A100 GPUs; model settings varied in temperature, reasoning, and answer cap.Source: §D.2 · p. 27, Table 11 · p. 27, Table 12 · p. 27

D.5 RETRIEVER CONFIGURATIONS

  • Retriever-specific configurations define indexing and evidence selection over a shared corpus and common answer-generation interface.The listed settings include five-passage retrieval and, for Dense TextRAG, cosine similarity with BAAI/bge-large-en-v1.5 embeddings.Source: §D.5 · pp. 28–29

E.1 SHARD CONSTRUCTION

  • The shard-construction prompts segment each factual question into information units, then rephrase those units as a goal-first opener and conversational clue turns.The instructions preserve bridge relations, avoid adding information, and keep follow-ups as clues rather than solver sub-questions.Source: E PROMPTS · p. 30, §E.1 · pp. 30–31

E.2 SIMULATION SCAFFOLD

  • A shared GPT-4o-mini simulates users by revealing at most one unrevealed shard per turn and classifies assistant turns into seven response categories.The simulator is instructed to preserve the original goal and avoid leaking unrevealed clues.Source: §E.2 · pp. 32–34

E.3 ASSISTANT

  • The assistant prompt permits answering from relevant retrieved context or model knowledge and requires a final, concise Answer line whenever it gives an answer.The closed-book variant removes the retrieved-context instructions and block.Source: §E.3 · p. 34

E.4 EVALUATION

  • The evaluation appendix describes a binary judge that accepts equivalent names, spellings, dates, and number formats but rejects contradictory, ambiguous, or absent answers.The robustness analysis rescored extracted answers against reference answers and aliases.Source: §E.4 · p. 35

Related papers

Loading 2609.36700v1…