PaperlayerView exact revision on arXiv

Source-linked AI summary

Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation

Lorenz Brehme, Adam Jatowt

arXiv:2608.21558v1cs.CLcs.AI

TL;DR

Domain-specific RAG evaluation requires custom multi-hop and unanswerable QA datasets because public benchmarks are tied to particular corpora and often omit abstention cases. TRIAD generates, validates, and context-labels such datasets from an existing vector database. Its datasets are suitable for comparative RAG evaluation, showing similar performance trends across setups, while limitations include restricted generalization and non-deterministic LLM-based experiments.

  • Problem

    Public multi-hop datasets are tied to specific domains, and many approaches omit unanswerable questions needed to evaluate hallucination and abstention on proprietary RAG corpora.

  • Method

    TRIAD generates multi-hop QA pairs, validates them in a feedback loop, and adds relevant and non-relevant context documents using an existing RAG vector database.

  • Results

    TRIAD-generated datasets show similar performance trends to HotpotQA and MuSiQue across different RAG setups, and human validation indicates that the questions are suitable for evaluation.

  • Takeaways & Limitations

    TRIAD supports comparative evaluation of domain-specific RAG systems and benchmarking hallucination through unanswerable questions.

  • Takeaways & Limitations

    Results may not generalize beyond the tested models and prompts, experiments are non-deterministic, and evaluation used relatively clean, mostly general-knowledge corpora.

Abstract

from arXiv · show

Recent advances in LLMs and the adoption of RAG systems in industry have created a need for domain-specific question-answer datasets that can assess RAG performance on proprietary data. Existing datasets, such as HotpotQA, challenge current RAG systems on Wikipedia-based knowledge, but they cannot be transferred directly to domain-specific settings. A comprehensive evaluation of RAG system quality requires both multi-hop queries and unanswerable questions. This paper introduces TRIAD, a three-stage automated dataset generation approach. First, it generates question--answer (QA) pairs for the domain-specific knowledge base of a RAG system. Second, a validator checks each QA-pair in a feedback loop. Third, the QA pairs are extended with relevance-labeled context documents for downstream evaluation. We evaluate this approach against the established MuSiQue and HotpotQA datasets. The results show that the generated dataset exhibits similar performance trends across different RAG setups, while human validation indicates that the questions are suitable for evaluating a domain-specific RAG system. The code used to generate the dataset and all validation results are available in our GitHub repository(https://github.com/lorenzbrehme/triad).

1 Introduction

Domain-specific RAG evaluation needs custom datasets containing multi-hop and unanswerable questions, because existing benchmarks are tied to specific public domains and often omit abstention cases. TRIAD addresses this need by generating and validating QA pairs from an existing RAG vector database, then extending them with labeled contexts for evaluation.

  • Evaluation gap: Existing multi-hop datasets such as HotpotQA and MuSiQue are restricted to specific domains and cannot fully test RAG systems on proprietary corpora.Manual dataset creation is labor-intensive and costly because it requires expert annotators.
  • Evaluation gap: Many question-generation approaches omit unanswerable questions, limiting evaluation of whether RAG systems hallucinate or correctly abstain.Unanswerable cases test behavior when the available context is insufficient.
  • TRIAD contribution: TRIAD automatically generates multi-hop QA datasets with labeled context documents and unanswerable questions from the vector database already used by a RAG system.The approach requires no complex preprocessing and targets domain-specific RAG corpora.
  • TRIAD contribution: TRIAD is manually validated and compared with HotpotQA and MuSiQue across seven RAG setups to assess question quality and performance trends.The supplied passage describes this as a validation of the generated datasets and pipeline.

2 Related Work

Existing multi-hop QA datasets provide relevance labels but are manually tailored to particular source corpora, limiting their direct use for domain-specific RAG evaluation. Prior automated methods often rely on costly preprocessing, motivating TRIAD’s simpler generation approach with irrelevant contexts for unanswerable questions.

  • Existing datasets: Multi-hop QA datasets include comparison and bridging questions, while HotpotQA and MuSiQue also associate questions with relevant and non-relevant context documents.These labels can support retriever evaluation.
  • Existing datasets: Because HotpotQA and MuSiQue are manually created for specific source corpora, they do not directly evaluate RAG knowledge on domain-specific corpora.Automatic LLM-based generation is presented as an alternative.
  • Prior automated methods: Many automated question-generation frameworks construct a knowledge graph during preprocessing to identify related entities and supporting documents.Another approach iteratively rewrites questions by adding documents to create multi-hop queries.
  • TRIAD motivation: TRIAD generates multi-hop QA sets without complex preprocessing and includes irrelevant contexts for unanswerable questions.The approach also defines question types intended to trigger different retriever behaviors.

3 TRIAD Methodology

TRIAD generates, validates, and extracts multi-hop QA examples from a RAG corpus. Its question types target parallel or sequential retrieval, while extraction adds labeled distractors, difficulty levels, and simulated unanswerable cases.

  • Pipeline: TRIAD has three stages: generation creates questions, validation checks them, and extraction pairs them with relevant and non-relevant documents.The resulting context list is labeled for downstream evaluation.
  • Generation: The generation stage retrieves similar documents, identifies a bridging topic, and prompts an LLM with the documents, question type, and target role to produce a question and answer.Each sample also includes relevant contexts and type and role labels.
  • Question types: TRIAD defines comparison, intersection, and composition multi-hop types, plus binary and temporal cross-cutting categories and a separate unanswerable category.The multi-hop types are associated with distinct retrieval patterns.
  • Question types: Parallel retrieval exposes all key entities for direct retrieval, whereas sequential retrieval requires identifying an intermediate entity before retrieving the next chunk.Composition questions use sequential retrieval, which a simple single-step RAG architecture does not support.
  • Validation: Validation uses LLM-as-a-judge checks plus a sufficient-context test to reject questions answerable from only one context document.Failed requirements trigger feedback, revision, and re-evaluation in a mandatory feedback loop.
  • Extraction: Extraction adds non-relevant contexts, counts relevant contexts to assign difficulty, and replaces retrieval output with only non-relevant contexts to test abstention.Fewer relevant contexts indicate higher difficulty, and the difficulty signal supports analysis of retriever performance by question type.

4 Experimental Setup

The experiments generate and validate TRIAD datasets, then compare them with HotpotQA and MuSiQue across varied retrieval, embedding, and language-model configurations. Evaluation covers answer quality, faithfulness, and retrieval of labeled contexts.

  • Dataset construction: TRIAD experiments use HotpotQA and MuSiQue context documents as source corpora and generate 600 questions for each corpus.The generation, validation, and extraction stages use separate Gemini models with a PGVector store.
  • Dataset construction: After validation, 478 questions remain for MuSiQue and 481 for HotpotQA.The generated sets initially sampled 200 questions for each of three multi-hop types, including 100 binary and 100 non-binary questions.
  • Validation: Human validation evaluates 200 questions sampled evenly from the two selected QA datasets across answerability, correctness, true multi-hop reasoning, and an additional metric.Answerability checks whether the answer is present in the provided documents.
  • RAG configurations: The RAG comparison varies retrieved chunks, embedding models, and LLMs across seven distinct setups.The setups use either 3 or 5 retrieved chunks, three embeddings, and two LLMs, with PGVector for all configurations.
  • RAG configurations: Each configuration is compared with a standard setup by varying one component at a time and checking whether TRIAD shows trends similar to established benchmarks.The standard setup uses BAAI/bge-smallen-v1.5, 5 retrieved chunks, and Gemini-2.5-Flash-Lite.
  • Metrics: Evaluation measures faithfulness, answer correctness, context precision, and context recall.The first two use RAGAS with GPT-5-nano, while the retrieval metrics use exact matching against dataset relevance labels.

5 Validation

TRIAD-generated QA sets were human-validated and tested across RAG configurations, question types, embedding models, LLMs, and unanswerable queries. The generated datasets generally reproduced benchmark performance trends while supporting domain-specific evaluation.

  • Validator Behavior: 703 HotpotQA and 723 MuSiQue questions produced acceptance rates of 68% and 66%, respectively.Manual checks found true rejection rates of 85% for HotpotQA and 90% for MuSiQue.
  • RAG-Evaluation: Across seven RAG setups, generated datasets exhibited performance trends similar to HotpotQA and MuSiQue.The evaluation varied embedding models, retrieved-chunk counts, LLMs, and question types.
  • RAG-Evaluation: Google’s Embedding-002 performed best and all-MiniLM-L6-v2 worst on context recall, with generated datasets showing larger retrieval differences than benchmarks.For HotpotQA, the recall gap exceeded 70%; for MuSiQue, it remained below 45%. Reducing retrieved chunks increased precision but decreased recall.
  • RAG-Evaluation: LLM choice changed faithfulness and answer correctness but not context precision or recall; GPT-5-nano achieved the best answer correctness.Faithfulness results were less consistent across benchmarks.
  • Question Types: Intersection questions achieved the highest answer correctness, while comparison questions had the lowest; non-binary questions were substantially more correct than binary variants.For faithfulness, comparison questions performed best, while intersection and composition questions scored lower.
  • Unanswerable Questions: Gemini-2.5-Flash-Lite and Gemma-4-31B-It detected insufficient context in more than 99% of unanswerable cases, versus 94% for GPT-5-nano.Failure to detect insufficient context led to hallucinated answers based on prior knowledge, even when those answers were correct.

6 Discussion

TRIAD supports domain-specific RAG evaluation with unanswerable questions, question types targeting retrieval strategies, and validation safeguards. Its generated datasets preserve comparative performance trends despite retrieval and potential generation-evaluation biases.

  • Unanswerable questions: TRIAD benchmarks hallucination by supplying only non-relevant contexts, where systems should abstain instead of answering.GPT-5-nano had the highest hallucination rate among tested models.
  • Question types: Different question types target distinct retrieval strategies, enabling finer-grained analysis than a single aggregate score.Practitioners can emphasize question types aligned with expected user needs.
  • Validation: 10% to 15% of questions were falsely rejected by the validator, reflecting a deliberate preference for stricter rejection criteria.The validator uses several checks and a feedback loop to retain high-quality QA pairs.
  • Retrieval difficulty: Generated questions may be easier to retrieve than human-created benchmark questions, producing higher absolute retrieval scores.The effect is especially evident for context precision and context recall because generation relies on semantically similar documents.
  • Comparative evaluation: Relative performance trends remain consistent across evaluated RAG setups despite differences in retrieval difficulty.This supports comparative evaluation, although generated datasets may not match established benchmarks in absolute difficulty.
  • Potential bias: Using the same vector database and similar retrievers for generation and evaluation may bias system performance toward comparable retrieval strategies.Comparisons across embedding models produced patterns similar to established benchmarks, suggesting generation-phase bias was not primary.

7 Conclusion

TRIAD generates domain-specific multi-hop datasets for RAG evaluation, including answerable and unanswerable questions with labeled contexts. The approach supports comparative evaluation, though its datasets remain weaker than human-created benchmarks and require broader future validation.

  • Contribution: TRIAD generates domain-specific multi-hop datasets containing QA pairs, relevant contexts, and distractor contexts for unanswerable-question evaluation.Distractors can also modify retriever outputs during evaluation.
  • Results: The generated dataset shows similar performance trends across RAG setups but remains weaker than human-created benchmarks such as HotpotQA.The paper presents this as support for comparative evaluation in domain-specific settings.
  • Future work: Future work includes multi-turn multi-hop questions and real-world evaluation with domain-expert validation.These extensions target realism and practical relevance.

8 Limitations

The study’s evidence is bounded by its experimental models, prompts, LLM-based evaluation, and relatively clean, mostly general-knowledge corpora. These constraints limit reproducibility and leave performance on noisier, more domain-specific industry data untested.

  • Experimental scope: Reported results are limited to the models and prompts used in the experiments and may not generalize to other settings.The study also notes that LLM non-determinism prevents fully exact replication.
  • Evaluation: Correctness and faithfulness metrics depend on the reliability of an LLM judge and the RAGAS framework.Other metrics used exact matching or human assessment.
  • Data scope: The approach was evaluated on relatively clean corpora and mostly general-knowledge data rather than noisier industry-specific data.Future work should test domains where the model has not been explicitly trained.

Appendix

The appendix organizes supplementary material into four sections covering Figure 2 results, feedback-loop examples, TRIAD-generated questions, and experimental prompts.

  • Section A: Section A presents a table corresponding to Figure 2 and detailed results for Section 5.4.
  • Sections B–C: Section B provides three examples illustrating the feedback loop, while Section C contains questions generated using TRIAD.
  • Section D: Section D includes all prompts used throughout the experiments.

A Results

The results section presents comparisons of embedding models and unanswerable-question scores, with baselines and relative improvements indicated in the tables.

  • Embedding Models: Table 5 compares embedding models using Gemini-2.5-Flash-Lite.The baseline is shown in bold.
  • Embedding Models: Arrows in Table 5 indicate improvements relative to the baseline.The table uses bold formatting to identify the baseline.
  • Unanswerable Questions: Table 6 compares scores for unanswerable questions.The comparison is linked to the unanswerable-question evaluation in Section 5.4.

B Example of Validation Feedback

The validation feedback rejects questions that are ambiguous, insufficiently grounded, or unclear for retrieval, and proposes more explicit reformulations.

  • Rejected Questions: A question is rejected when it is not standalone or compares an entity whose relevant information is missing.The breed example explicitly states that only the Tsigai origin is provided, so the comparison is unsupported.
  • Retrieval Clarity: Questions should name identifiable entities so the retriever can target the correct documents.The actress example is rejected because its description could refer to many actresses.
  • Multi-Hop Reasoning: High-quality questions require combining information from at least two chunks, with neither chunk answering the question alone.This necessity test is stated as a global requirement for multi-hop questions.
  • Question Requirements: Questions must be concise, uniquely answerable, fully grounded in the chunks, and avoid revealing the answer.The requirements also call for short factual answers and prohibit external knowledge or invented relationships.
  • Comparison Questions: Comparison questions retrieve two entities or fact sets from different chunks and compare them on a specified factual attribute.The comparison result must not be directly stated and must require combining the chunks.
  • Attribute Composition: Attribute composition questions use a two-step lookup: one chunk identifies a hidden bridge entity, and another provides the final answer.The bridge entity must not appear explicitly in the question and must be recoverable through reasoning.

C Examples of Questions

Table 7 provides example multi-hop question-and-answer samples from the generated dataset.

  • Examples: Table 7 presents example multi-hop QA samples from the dataset.The table is intended to illustrate generated question-and-answer examples.
  • Examples: The examples are organized as samples rather than as a reported quantitative comparison.The caption identifies the table as an example collection.
  • Examples: The table concerns the generated dataset’s multi-hop question-answer format.Its caption specifically refers to multi-hop QA samples.
Loading 2608.21558v1…