MHTS: Multi-Hop Tree Structure Framework for Generating Difficulty-Controllable QA Datasets for RAG Evaluation

1 Introduction

• Existing benchmarks often treat queries as equally difficult, while hop-count and evidence-count measures miss semantic complexity and logical integration.Difficulty also depends on how widely supporting evidence is dispersed across source chunks.Source: §1 · pp. 1–2
• A robust RAG benchmark should provide accurate, complete answers, cover diverse semantic domains, and reflect varying reasoning difficulty.The authors identify difficulty as underexplored in existing benchmark datasets.Source: §1 · p. 1
• MHTS uses answer-first generation across semantically diverse evidence and estimates difficulty from reasoning hops and query-to-chunk semantic distance.The framework synthesizes answers requiring multi-hop reasoning before generating corresponding questions.Source: §1 · p. 2
• The difficulty score captures reasoning hops and semantic distance and aligns closely with real RAG performance trends.The authors position it as a tool for dataset curation and system evaluation.Source: §1 · p. 2
• The dataset combines logically complete ground-truth answers with broad semantic coverage, including underrepresented regions of the document corpus.The authors embed claims, associated chunks, and multi-level summaries in a shared semantic space to assess coverage.Source: §1 · p. 2

2.1 Multi-Hop QA Datasets and Generation Methods

• Existing multi-hop QA datasets can permit shortcut solutions or lack structured difficulty control, while MultiHop-RAG lacks an explicit mechanism to model and control reasoning complexity.MHTS addresses this gap with a tree-based framework for fine-grained control over multi-hop query difficulty.Source: §2.1 · p. 2

2.2 Difficulty Adjustment in Evaluation Datasets

• Prior difficulty-adjustment approaches include compositional filtering, unanswerable contrasts, adversarial collection, and dynamic prediction, but do not provide fine-grained controllable difficulty representations.MHTS models and manipulates difficulty during dataset synthesis; Figure 1 outlines its chunking, tree construction, question generation, evidence tracing, and answer synthesis stages.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

2.3 Fine-Grained Reasoning Evaluation in RAG Systems

• Prior RAG evaluation work offers diagnostic metrics and LLM-based judging, but the authors say these post hoc approaches lack control over input complexity.MHTS complements them by encoding multi-hop reasoning difficulty for systematic, fine-grained evaluation.Source: §2.3 · p. 3

2.4 Benchmarking RAG systems’s Retrieval and Synthesis

• Existing benchmarks jointly measure retrieval and generation across knowledge-intensive tasks, yet most lack explicit control over multi-hop difficulty.MHTS is presented as enabling fine-grained difficulty modeling for more diagnostic RAG evaluation.Source: §2.4 · p. 3

3.1 Claim Extraction and Clustering

• Claims are extracted from semantically coherent chunks, embedded, and grouped by a Gaussian Mixture Model with threshold-based soft assignment that permits overlapping cluster membership.The extraction process decontextualizes claims and maps them back to their source chunks.Source: §3.1 · p. 4

3.2 Multi-Hop Claim Generation

• MHTS recursively clusters claims and composes them into increasingly complex multi-hop claims, allowing control over the reasoning depth of generated QA tasks.New claims can be fed back into the clustering and composition pipeline to build further reasoning layers.Source: §3.2 · p. 4

3.3 Question and Answer Generation

• Questions are generated from multi-hop claims, then answers are expanded using the top three question-similar chunks and the top three claim-associated chunks, with overlap possible.The LLM reformulates or expands the claim into a contextualized answer grounded in the selected chunks.Source: §3.3 · pp. 4–5

3.4 Difficulty via Multi-hop

• Claims are decomposed into one-hop, conjunction, existence, and negation evidences; unmapped evidence is excluded as hallucination, while inferable intermediate suppositions are retained.The hop count is the number of decomposed evidence pieces; this verification checks grounding and inference depth.Source: §3.4 · p. 5
• Difficulty is scored as D = h − λs, where h is hop count, s is average question-to-supporting-chunk cosine similarity, and λ defaults to 1.When one evidence piece maps to multiple chunks, their similarities are averaged.Source: §3.4 · p. 5

4 Experiments

• The evaluation examines difficulty calibration, semantic coverage, and answer quality through win-rate analysis, embedding-based diversity measures, and qualitative comparison.For difficulty evaluation, GPT-4o judges proposed and RAG answers on Comprehensiveness, Diversity, and Empowerment; answer order is swapped to mitigate positional bias.Source: §4 · p. 5, §4.3 · p. 6

4.1.1 Dataset

• The QA dataset uses David Copperfield, divided into 505 paragraph-based chunks of 540–1,016 tokens each.The novel is selected for its character relationships and distributed event information, which support multi-hop questions.Source: §4.1.1 · p. 5

4.2 Baseline and Ground-Truth (GT) Context

• The RAG baseline retrieves ten chunks by cosine similarity and reranks them to three; the GT context uses up to six chunks, including mapped gold evidence and retrieved chunks.The GT context selects three gold chunks most similar to the question and adds three RAG-retrieved chunks, with fewer than six when they overlap.Source: §4.2 · p. 6

4.3 Fine-grained Difficulty

• GT answers consistently outperform naive RAG outputs across Comprehensiveness, Diversity, and Empowerment, with win rates increasing as difficulty rises.The reported difficulty–win-rate correlation is 0.99; hop-count-based win rates fluctuate, while MHTS difficulty splits improve consistently.Source: §4.3 · pp. 6–7, Figure 3 · p. 6

4.4 Diversity

• MHTS-generated (Q, A) pairs have a higher average embedding distance than RAPTOR nodes (1.23 vs. 1.12) and a higher eigenvalue sum (0.76 vs. 0.63).The authors interpret these measures as greater semantic dispersion and a richer, more varied embedding subspace.Source: §4.4 · p. 7, Table 1 · p. 7

4.5 Quality

• In the quality case study, naive RAG misses the mapped evidence chunk and gives a hallucinated answer, while the MHTS-grounded answer identifies Emily’s uncle.The example concerns the person accompanying Emily on the moving ship.Source: §4.5 · pp. 7–8, Table 2 · p. 8

5 Conclusion

• MHTS combines reasoning hops and semantic dispersion in its difficulty definition, and its answers achieve higher win rates as difficulty increases.The conclusion links this finding to the importance of fine-grained difficulty calibration for developing and evaluating advanced RAG systems.Source: §5 · p. 8

Limitation

• Evaluation is limited to a single literary corpus, David Copperfield.The authors note that this may not fully represent challenges in other domains.Source: Limitation · pp. 8–9
• MHTS relies on LLMs for claim extraction, clustering, and multi-hop synthesis.The authors warn that smaller or less capable models may reduce dataset quality and controllability, raising accessibility and robustness concerns in resource-constrained settings.Source: Limitation · p. 9

• Figure 4 shows MHTS claims as more semantically dispersed than RAPTOR nodes, suggesting broader topic coverage.Source: Figure 4 · p. 10

A.1 Diversity Visualization

• The comparison projects 5,527 MHTS claims sampled by K-means against node representations from a RAPTOR-style summarization hierarchy.The embeddings are projected into two dimensions using UMAP.Source: §A.1 · p. 10
• MHTS claims appear more dispersed than RAPTOR nodes, suggesting slightly higher topic-level diversity, though the difference is subtle.Source: §A.1 · p. 10
• Despite the dispersion difference, the overall semantic coverage of both datasets remains comparable.Source: §A.1 · p. 11

A.2 Additional Quality Experiments

• In the Doctors’ Commons example, naive RAG’s top-three retrieved chunks omit the relevant passage, while the MHTS-produced ground-truth answer identifies where the narrator loitered.Source: §A.2 · p. 11
• In the Friday-night infant example, naive RAG hallucinates “Miss Betsey” and an optimistic outlook, whereas the ground-truth answer identifies the nurse’s and wise women’s predictions of misfortune and seeing spirits.Source: §A.2 · p. 11
• The authors state that MHTS’s more comprehensive evidence mapping avoids context omissions that lead to RAG hallucinations and yields answers aligned with source facts.Source: §A.2 · p. 11

Naiive RAG

• After breakfast on the Heath, the narrator walks back to Doctors’ Commons and loiters there for half an hour until old Tiffey appears with his key.Source: Naiive RAG · p. 12
• In Table 3, naive RAG misses the relevant chunk among its top three retrieved chunks, producing an incorrect, hallucinated response.The table compares naive RAG with the MHTS ground-truth answer and shows the supporting evidence in the bottom row.Source: Table 3 · p. 12
• Table 4 compares naive RAG with the MHTS ground-truth answer and shows the question’s supporting evidence in the bottom row.Source: Table 4 · p. 13