PaperlayerView exact revision on arXiv

Source-linked AI summary

The RAT: A Unified Bayesian Model for RAG Evaluation

Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak, Jan Deriu

arXiv:2608.24753v1cs.CLcs.AI

TL;DR

RAG evaluation needs to distinguish end-to-end task success from component behavior and policy adherence. The paper introduces a Bayesian model of retrieval, abstention, and answer correctness, applies it to 27 configurations, and finds that conditional analysis exposes behavioral differences hidden by marginal metrics. It also shows that retrieval annotations are more informative for policy-adherence estimation and extends the framework to calibrated automated judgments.

  • Problem

    RAG evaluation often overlooks dependencies among retrieval, abstention, and generation, while task success can conceal whether the generator followed the appropriate policy.

  • Method

    A Bayesian framework factorizes retrieval, abstention, and task success according to pipeline information flow, derives generator success deterministically, and models LLM-as-a-judge labels as calibrated noisy observations.

  • Results

    Across 27 configurations, conditional decomposition reveals behavioral differences hidden by marginal metrics; retrieval-success annotations are more informative than task-success annotations for estimating policy adherence.

  • Takeaways & Limitations

    The framework enables unified analysis of policy adherence and answer correctness while combining limited human judgments with automated assessments.

  • Takeaways & Limitations

    The framework assumes binary judgments and a single deterministic policy, so it does not accommodate continuous retrieval metrics or soft and alternative policies.

Abstract

from arXiv · show

Evaluating Retrieval-Augmented Generation (RAG) systems requires assessing not only end-to-end correctness but also how individual components interact and how errors propagate through the pipeline. We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness, factorized according to the pipeline's information flow. The model distinguishes task success. Whether the user received a correct answer (from generator success) and whether the generator behaved appropriately given the retrieval outcome. We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators, and show that the conditional decomposition reveals substantial behavioral differences between systems that appear equivalent under marginal metrics. We further analyze the annotation allocation problem, demonstrating that retrieval-success annotations are more informative than task-success annotations for estimating policy adherence, and provide an information-theoretic explanation for this asymmetry. Finally, we extend the model to incorporate LLM-as-a-judge annotations as calibrated noisy observations, enabling practitioners to combine limited human judgments with cheaper automated assessments within a unified probabilistic model.

1 The Introduction

RAG evaluation must examine component behavior and dependencies, not only end-to-end correctness. The paper introduces a Bayesian framework that distinguishes task success from generator policy adherence and supports joint analysis of retrieval, abstention, and answer correctness.

  • End-to-end evaluation alone is insufficient because RAG components interact and errors can propagate through the pipeline.The paper argues that components should be evaluated both independently and within the complete pipeline.
  • Task success measures whether users receive correct answers, whereas generator success measures whether the system answers or abstains appropriately given retrieval.These quantities can diverge when a generator answers without adequate retrieval support or fails to abstain.
  • Systems with identical task success can exhibit substantially different behavior when retrieval fails, making policy adherence invisible to marginal metrics.A generator that never abstains may retain reasonable task success while systematically violating the desired policy.
  • The proposed Bayesian framework factorizes retrieval, abstention, and task success according to pipeline information flow and derives generator success from them.Bayesian inference propagates uncertainty and marginalizes unobserved variables when annotation costs differ.
  • The framework evaluates 27 configurations, studies annotation allocation, and incorporates calibrated LLM-as-a-judge observations alongside human labels.The contributions cover empirical behavioral comparisons, information-theoretic annotation analysis, and unified human-automated assessment.

2 The Related Work

Prior RAG evaluation commonly assesses retrieval, generation, or multiple pipeline dimensions separately. The paper positions its approach as a unified probabilistic framework that models dependencies between retrieval and generation outcomes.

  • RAG evaluation frameworks have commonly measured retrieval and generation components separately or used adversarial retrieval results.Retrieval evaluation may use standard information-retrieval scores when a gold standard exists.
  • Existing multi-dimensional frameworks evaluate aspects such as context relevance, faithfulness, and answer relevance without modeling statistical dependencies between retrieval and generation outcomes.The paper describes its contribution as integrating these dimensions into one probabilistic framework.
  • RAGAs and ARES follow the RAG Triad framing while treating evaluation dimensions independently.The related work also notes benchmarks covering broader pipeline choices, including chunking and reranking.

3 The Model

The model represents RAG behavior with binary retrieval, abstention, task-success, and generator-success variables linked by the pipeline’s information flow. A Bayesian formulation provides interpretable conditionals and supports uncertainty propagation with partially observed annotations.

  • The Variables: The model uses four binary variables: retrieval success R, abstention A, task success T, and generator success G.R indicates sufficient retrieved information, A indicates an explicit refusal, T indicates a correct final answer, and G indicates policy adherence.
  • The Conditionals: The joint distribution is factorized as P(R, A, T) = P(R) P(A | R) P(T | A, R), following information flow through the pipeline.This separates retrieval quality, retrieval-conditioned abstention, and answer correctness conditional on retrieval and answering.
  • The Generator Success: Generator success is deterministically defined as abstaining when retrieval fails or answering correctly when retrieval succeeds.This distinguishes appropriate behavior from merely producing a correct final output.
  • The Unified Model: Bayesian inference propagates uncertainty through primitive and derived quantities and accommodates partially observed data by marginalizing unobserved variables.The framework is designed for settings where some annotations are expensive while others are cheap.
  • The Conditionals: The model has five interpretable probabilities covering retrieval, abstention under retrieval failure or success, and supported or unsupported answer success.These parameters correspond directly to distinct behavioral aspects of RAG systems.

4 The Retrievers, The Generators, and The Tasks

The experiments use KILT tasks with a shared Wikipedia-based knowledge base, three retrieval strategies, and three language-model generators. The controlled setup supports comparison across 27 RAG configurations while imposing a deployment-relevant scope boundary.

  • The Tasks: The study uses KILT, a benchmark spanning 11 datasets across five task categories, including fact-checking and open-domain question answering.The datasets share a pre-processed Wikipedia snapshot to keep evaluation conditions consistent.
  • The Tasks: For each task, 10,000 queries were selected and combined into a shared knowledge base containing 1,720,160 paragraphs.The knowledge base was constructed from documents relevant to the selected queries.
  • The Retrievers: The evaluated retrievers are sparse BM25, dense embedding-based search with HNSW indexing, and hybrid retrieval using Reciprocal Rank Fusion.These strategies represent lexical, semantic, and combined retrieval approaches.
  • The shared knowledge base is more controlled than a fully open-domain corpus, so absolute retrieval-success rates may be optimistic for deployment.This is an explicit scope limitation of the experimental environment.
  • The Generators: The generator set includes Apertus 8B, Gemma3 12B, and Qwen3.5 9B.The models differ in architecture, parameter count, and development context as described in the paper.

5 The Experiments

Across 27 configurations, marginal task success often masks differences in retrieval-conditioned abstention and policy adherence. The experiments also show that annotation value depends on the target, while noisy automated judgments provide limited gains without strong calibration.

  • 5.1 The Marginals: Retrieval success is highest on FEV for Hybrid retrieval and on HQA for sparse retrieval, reflecting differences in dataset difficulty.FEV is easier because each sample has exactly one relevant document.
  • 5.1 The Marginals: Qwen3.5 achieves the highest policy adherence across datasets, while the three generators have broadly similar task-success rates.Task success is highest on FEV and lower on HQA and NQ, whose answers require exact matching to open-form references.
  • 5.2 The Conditionals: Conditional probabilities show that Gemma3 and Qwen3.5 abstain more when retrieval fails than when it succeeds, unlike Apertus.On HQA with hybrid retrieval, Qwen3.5 has P(A=1 | R=0) = 0.463 versus P(A=1 | R=1) = 0.036, while Apertus has 0.039 versus 0.005.
  • 5.2 The Conditionals: Similar task success can conceal sharply different policy adherence: on NQ with dense retrieval, the generators’ task success is nearly identical but their policy adherence differs substantially.Task success is 0.239, 0.242, and 0.243 for Apertus, Gemma3, and Qwen3.5, while policy adherence is 0.164, 0.414, and 0.496.
  • 5.3 The Sample Allocation Problem: Retrieval-focused annotations reduce estimation error more effectively for policy adherence, whereas task-focused annotations are more effective for task success.All-joint annotation is a robust default when the estimation target is unknown; the information-gain ranking predicts this empirical ordering.
  • 5.4 The Automated Judge: Automated judge annotations yield only marginal improvements when false-positive rates are substantial, so added noisy labels cannot substitute for modest human annotation.With 5,000 additional judge-annotated samples, P(G=1) credible-interval width decreases from 0.094 to 0.080 while MAE remains essentially unchanged.

6 The Conclusion

The paper presents a Bayesian framework that models retrieval, abstention, and task success according to the RAG pipeline’s information flow. Across 27 configurations, conditional analysis exposes policy-adherence differences hidden by marginal task-success metrics, while retrieval-focused annotations better estimate adherence.

  • The framework factorizes retrieval success, abstention, and task success according to the pipeline’s information flow.Generator success is derived deterministically from these variables.
  • 27 RAG configurations reveal sharply different policy adherence despite near-identical task success.The conditional decomposition makes this behavioral difference explicit when marginal metrics conceal it.
  • Retrieval-success annotations are more informative than task-success annotations for estimating policy adherence.Observing retrieval resolves generator success in three of four joint cells, whereas observing task success resolves only one.
  • High false-positive rates from an automated judge make even thousands of annotations yield only marginal gains over a small set of human judgments.This result underscores the importance of calibration quality when incorporating LLM-as-a-judge observations.
  • The probabilistic formulation supports reasoning about uncertainty, partial observability, and annotation efficiency within one model.The paper identifies continuous quality scales, deeper pipelines, and hierarchical modeling as natural extensions.

Limitations

The framework is limited by binary judgments, a fixed deterministic policy, restricted experimental diversity, a minimal single-turn pipeline, and exact-match abstention detection. These boundaries constrain how directly the model applies to richer retrieval metrics, alternative policies, production pipelines, conversations, and free-form responses.

  • The model assumes binary fail-versus-success judgments, rather than continuous retrieval metrics such as MAP and MRR.
  • Generator success follows one fixed policy: abstain if retrieval fails and answer correctly otherwise.The framework does not accommodate soft or application-dependent policy definitions.
  • Experiments cover three datasets and three relatively small open-weight generators with 8–12B parameters.The stated budget limitations narrow task and model diversity.
  • The framework models a minimal retrieve-then-generate pipeline and omits dependencies introduced by components such as reranking or query reformulation.Deeper pipelines require additional variables and a more complex dependency structure.
  • The evaluation is single-turn and does not model dialogue-history dependence or compounded errors in conversational RAG.
  • Exact matching detects abstention only in constrained outputs and misses hedged or indirect abstentions in free-form generation.A calibrated abstention classifier is suggested for such settings.

A Experimental Setup

The experimental retrieval configuration uses a fixed top-K setting for all retrievers.

  • All retrievers use top-K = 5.

A.3 Evaluation

Evaluation uses exact string matching for answers and abstentions, with task-specific prompts and constrained output formats. The appendix identifies the retrieval and generation/judge hyperparameter tables used to document this setup.

  • Generated answers are matched against ground truth without normalization, article stripping, or case adjustment.
  • Abstention detection uses exact matching for NOT_ENOUGH_INFO on FEVER and I DO NOT KNOW on HotpotQA and NQ.
  • Table 6 documents retrieval hyperparameters, while Table 7 documents generation and judge hyperparameters.
  • FEVER prompts restrict outputs to SUPPORTS, REFUTES, or NOT ENOUGH INFO.
  • HotpotQA and Natural Question use a constrained question-answering prompt requiring only an answer or I DO NOT KNOW.Responses are limited to a single word, name, entity, or number without explanation or punctuation.
  • The common user prompt supplies retrieved paragraphs and the original query, with retrieved content inserted through a templated block.

C Partial Retrieval Success

The paper extends binary retrieval evaluation to a ternary outcome—failure, partial retrieval, or success—and measures abstention and task success conditionally on it.

  • Retrieval is modeled as ternary: failure retrieves no relevant documents, partial retrieval retrieves some, and success retrieves all.
  • NQ has 0% partial retrievals, while Hybrid retrieval produces up to 453 partial FEV queries.
  • For HotpotQA, every query has two relevant documents by construction, making partial retrieval a relevant evaluation case.
  • The ternary model defines abstention and task-success probabilities conditionally for failure, partial, and successful retrieval.
  • Abstention decreases monotonically as retrieval quality increases, while task success increases.
  • Relevance annotations may not identify which documents are individually sufficient or jointly necessary.

E Extended Sample Allocation Plot

Figure 3 evaluates estimation error and posterior interval width for policy adherence and task success across annotation strategies, showing opposite allocation preferences for the two targets.

  • Figure 3 reports MAE and 95% credible interval width for policy adherence P(G=1) and task success P(T=1) across five annotation strategies.
  • For policy adherence, retrieval-focused strategies yield narrower intervals than task-focused strategies at every budget level.
  • At budget 500, All-R has CI width 0.105 for Qwen’s P(G=1), versus 0.132 for All-T.
  • For task success, the pattern reverses: at budget 500, All-T has CI width 0.075 versus 0.150 for All-R.
  • Coverage remains close to the nominal 95% level across strategies and budgets, indicating well-calibrated posterior intervals.
  • The all-joint strategy consistently produces among the narrowest intervals for both quantities and is a robust default when the target is unknown.

F Information Gain Calculations

The information-gain analysis explains why retrieval annotations are more informative than task annotations for estimating policy adherence, while showing when joint labels remain valuable.

  • Policy adherence is determined directly in three of four (R, A) cells; only correct retrieval with answering leaves G dependent on T.
  • Base Uncertainty: Given only abstention, Qwen has base uncertainty 0.526 and Apertus 0.490 about policy adherence.
  • All-Joint Strategy: All-joint observations resolve G completely and therefore provide the maximum achievable information gain per sample.
  • All-R Strategy: Observing retrieval yields information gains of 0.345 for Qwen and 0.284 for Apertus, eliminating most uncertainty about G.
  • All-T Strategy: Observing task success yields gains of 0.154 for Qwen and 0.200 for Apertus, because two cells remain unresolved.
  • Mixed Strategies: Half-Joint-R yields 0.436 for Qwen and 0.387 for Apertus, while Half-Joint-T yields 0.340 and 0.345, respectively.
  • Summary: The information-theoretic ranking reliably predicts which partial observation is more informative, but substitution for joint labels depends on constraining ambiguous-cell parameters.
  • Empirical Deviations: For Apertus, θT + = 0.519 has near-maximal entropy, making additional joint samples valuable; Qwen’s θT + = 0.678 is easier to estimate.
Loading 2608.24753v1…