PaperlayerView exact revision on arXiv

Source-linked AI summary

CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation

Siddharth Jain, Venkat Narayan Vedam

arXiv:2604.05467v1cs.IRcs.CLcs.LG

TL;DR

Existing RAG evaluation does not directly measure the operational utility of individual retrieved evidence items during system behavior. CUE-R uses controlled evidence perturbations and shallow observable traces to measure multiple utility effects, finding consistent harm from removal and replacement, non-neutral redundancy, and interacting multi-hop supports.

  • Problem

    Existing RAG evaluation does not directly target the operational utility of individual retrieved evidence items beyond final-answer, citation, or answer-level attribution measures.

  • Method

    CUE-R perturbs evidence items with REMOVE, REPLACE, and DUPLICATE, then measures correctness, proxy grounding faithfulness, confidence error, and trace divergence using shallow observable traces.

  • Results

    REMOVE and REPLACE strongly harm correctness, grounding, and confidence alignment while causing large trace divergence; DUPLICATE is milder on correctness but can alter traces and confidence error.

  • Takeaways & Limitations

    Trace-sensitive intervention evaluation exposes evidence effects that answer-only metrics do not cleanly separate.

  • Takeaways & Limitations

    CUE-R measures observable intervention sensitivity rather than strong causal contribution or hidden internal reasoning, and its current instantiation is limited to single-shot RAG.

Abstract

from arXiv · show

As language models shift from single-shot answer generation toward multi-step reasoning that retrieves and consumes evidence mid-inference, evaluating the role of individual retrieved items becomes more important. Existing RAG evaluation typically targets final-answer quality, citation faithfulness, or answer-level attribution, but none of these directly targets the intervention-based, per-evidence-item utility view we study here. We introduce CUE-R, a lightweight intervention-based framework for measuring per-evidence-item operational utility in single-shot RAG using shallow observable retrieval-use traces. CUE-R perturbs individual evidence items via REMOVE, REPLACE, and DUPLICATE operators, then measures changes along three utility axes (correctness, proxy-based grounding faithfulness, and confidence error) plus a trace-divergence signal. We also outline an operational evidence-role taxonomy for interpreting intervention outcomes. Experiments on HotpotQA and 2WikiMultihopQA with Qwen-3 8B and GPT-5.2 reveal a consistent pattern: REMOVE and REPLACE substantially harm correctness and grounding while producing large trace shifts, whereas DUPLICATE is often answer-redundant yet not fully behaviorally neutral. A zero-retrieval control confirms that these effects arise from degradation of meaningful retrieval. A two-support ablation further shows that multi-hop evidence items can interact non-additively: removing both supports harms performance far more than either single removal. Our results suggest that answer-only evaluation misses important evidence effects and that intervention-based utility analysis is a practical complement for RAG evaluation.

1 Introduction

CUE-R addresses the gap between answer-level RAG evaluation and the need to characterize how individual retrieved items affect observable behavior and utility. It uses controlled evidence interventions and multi-axis measurements to reveal harmful, redundant, and interacting evidence effects.

  • CUE-R targets the operational role of individual retrieved evidence items, which final-answer quality, trace faithfulness, and answer-level attribution do not fully capture.
  • The framework perturbs one evidence item at a time with REMOVE, REPLACE, or DUPLICATE and measures correctness, proxy grounding faithfulness, confidence error, and trace divergence.It is designed to detect behavioral changes even when the final answer remains unchanged.
  • A zero-retrieval control indicates that intervention effects reflect degradation of meaningful retrieval rather than arbitrary prompt instability.
  • Across two datasets and two model families, REMOVE and REPLACE are consistently harmful, whereas DUPLICATE is often answer-redundant but not behaviorally neutral.The harmful interventions can substantially affect correctness and grounding while producing large trace shifts.
  • Two-support ablations show non-additive multi-hop dependence: removing both supports causes far larger degradation than either single removal.This joint dependence is missed by single-item interventions alone.
  • An operational evidence-role taxonomy interprets intervention outcomes without claiming to recover hidden internal reasoning.CUE-R remains scoped to shallow observable traces in a single-shot RAG pipeline.

2 Related Work

CUE-R extends RAG evaluation beyond final-answer attribution, citation assessment, and generic trace evaluation by diagnosing how individual retrieved items affect observable retrieval-use traces and utility. Its positioning emphasizes controlled perturbations, multi-axis outcomes, and multi-item interactions.

  • 2.1 Answer-Level Attribution in RAG: Prior attribution work links retrieved evidence to final answers, but similarity-based attribution may be plausibly explanatory rather than causal.
  • 2.1 Answer-Level Attribution in RAG: CUE-R differs by analyzing observable retrieval-use traces and decomposing evidence effects across correctness, grounding faithfulness, confidence alignment, and evidence roles.
  • 2.2 Trace Faithfulness: Trace-faithfulness research evaluates whether reasoning appears grounded, while CUE-R asks what utility a specific retrieved item contributed after system use.A perturbation may leave correctness unchanged while altering trace behavior or confidence alignment.
  • 2.3 Grounded Agent Evaluation: Grounded agent benchmarks assess trajectories broadly, whereas CUE-R focuses on retrieved evidence items and their counterfactual role in RAG reasoning.
  • 2.4 RAG Robustness: Robustness studies examine performance under misleading or conflicting evidence; CUE-R additionally diagnoses how perturbations change trace behavior and utility.This distinguishes distractive, answer-redundant, and confidence-distorting evidence.
  • 2.6 Positioning of CUE-R: CUE-R is specifically an intervention-based diagnostic framework for individual evidence utility, not a citation evaluator, faithfulness classifier, hallucination detector, or general agent trajectory evaluator.Its positioning includes marginal necessity and multi-item interaction effects over shallow observable traces.

3 Problem Formulation

CUE-R formulates evidence evaluation as an intervention problem over observable retrieval-use traces, measuring how perturbing one retrieved item changes utility and behavior. Its multi-axis analysis supports operational role labels without claiming hidden causal mechanisms.

  • 3.1 Setup: The system exposes observable traces containing actions, states, final answers, and confidence signals during inference.In the shallow setting, states track active queries, selected evidence, support status, candidate answers, and step-level confidence.
  • 3.1 Setup: The formulation deliberately analyzes observable behavior under evidence perturbation rather than hidden internal reasoning.
  • 3.2 Evidence Intervention Problem: CUE-R asks what role each retrieved evidence item played in producing the observed trace and final outcome.This differs from standard evaluation focused only on final-answer correctness.
  • 3.2 Evidence Intervention Problem: REMOVE, REPLACE, and DUPLICATE respectively delete an item, substitute a non-supporting alternative, or duplicate it.Each operator produces a counterfactual trace for comparison with the original run.
  • 3.3 Multi-Axis Utility: Utility is decomposed into correctness, proxy-based grounding faithfulness, and confidence error rather than a single performance scalar.This captures cases where answers remain unchanged but grounding or confidence worsens.
  • 3.4 Trace Divergence: Trace divergence separately measures path-level behavioral change because utility deltas can miss different traces with the same answer.The simplified single-shot proxy combines evidence-set divergence, answer change, and confidence change.
  • 3.5 Evidence-Role Taxonomy: Intervention outcomes assign evidence roles such as constructive, corrective, redundant, distractive, and confidence-distorting.Corrective roles are difficult to detect from shallow traces because recovery from unsupported intermediate states is not directly observable.
  • 3.5 Evidence-Role Taxonomy: These role labels are operational diagnoses derived from utility changes and trace divergence, not claims about internal causality.The framework outputs per-item intervention diagnostics from the question, retrieved evidence set, and observed trace.

4 Method

CUE-R evaluates retrieved evidence by perturbing individual items and comparing the resulting observable traces and downstream utility. The method combines controlled operators, multi-axis metrics, and an operational evidence-role taxonomy in a single-shot RAG pipeline.

  • Framework Overview: CUE-R measures how perturbing retrieved evidence changes observable retrieval-use traces and downstream utility.It targets externally observable retrieval calls, selected evidence, candidate answers, verification status, and final outputs.
  • Framework Overview: Figure 1 routes a question and corpus through BM25 retrieval before generating original and counterfactual traces.The counterfactuals result from perturbing a target evidence item with three operators.
  • Trace Representation: In the experimental single-shot setting, the standardized trace collapses to retrieval, evidence selection, and answer generation.The schema can extend to agentic retrieve–select–verify loops, although those loops are not used here.
  • Interventions: REMOVE deletes the target, REPLACE substitutes a related non-supporting passage, and DUPLICATE adds a second copy.These operators test evidence necessity, robustness to misleading substitution, and sensitivity to redundancy.
  • Metrics: CUE-R measures correctness, grounding, and confidence error, while trace divergence captures behavioral changes that utility deltas may miss.Two runs can produce the same answer while following different paths.
  • Evidence Roles: Evidence items receive constructive, corrective, redundant, distractive, or confidence-distorting interpretations from observed intervention outcomes.The taxonomy is operational rather than mechanistic.
  • Experimental Pipeline: Experiments use HotpotQA and 2WikiMultihopQA in a single-shot RAG pipeline with BM25 retrieval and logged model-use traces.The setup retrieves five chunks, prompts answering from provided context, logs identifiers, answers, confidence, and rationale, then reruns four conditions.
  • Metrics: Correctness uses strict and soft matching, grounding uses title overlap with gold supports, and confidence error compares self-reported confidence with binary correctness.The grounding measure is a proxy based on used chunk identifiers and supporting titles.

5 Experimental Setup

The experiments test whether intervention types produce distinct, trace-sensitive evidence effects across datasets, models, and multi-hop interactions. They combine controlled perturbation settings with correctness, grounding, confidence, and trace metrics.

  • Research Questions: The study asks whether perturbations yield distinct utility profiles, expose effects missed by answer-only metrics, replicate across datasets and models, and reveal non-additive interactions.
  • Datasets: HotpotQA supplies distractor-setting examples for primary Qwen-3 8B analysis, zero-retrieval control, and GPT-5.2 replication, while 2WikiMultihopQA supports cross-dataset replication.The datasets use 200, 200, and 100 HotpotQA examples for the listed analyses, plus 100 2WikiMultihopQA examples.
  • Models: Qwen-3 8B is the primary local model, and GPT-5.2 provides a lightweight cross-family replication on HotpotQA.Both experiments use temperature 0.
  • Retrieval Setup: BM25 ranks stable passage chunks and provides the top k = 5 chunks to the model under a minimal retrieval setup.
  • Intervention Setup: Target evidence selection prioritizes model-used chunks matching gold support titles, then falls back through model-used, support-matching, and highest-ranked chunks.
  • Intervention Setup: Each example is rerun under ORIGINAL, REMOVE, REPLACE, and DUPLICATE conditions.
  • Replacement Robustness: The replacement-hardness sweep compares random, question-similar, and target-similar non-supporting chunks.The three modes are Easy, Medium, and Hard, respectively.
  • Controls: A zero-retrieval control removes retrieved context so the model answers from parametric knowledge alone.

6 Results

CUE-R interventions produce distinct multi-axis utility profiles: REMOVE and REPLACE substantially degrade answer quality, grounding, confidence alignment, and traces, while DUPLICATE often preserves correctness but can still alter behavior. These effects generalize across datasets and models, and joint-support removal reveals non-additive evidence dependence.

  • 6.3 Statistical Support for Intervention Effects: DUPLICATE preserves correctness while still producing measurable trace and grounding effects.Correctness remains 0.585, but trace divergence is 0.074 and paired analysis finds significant trace and grounding effects despite null correctness and F1 deltas.
  • 6.4 Zero-Retrieval Control: A zero-retrieval control confirms that intervention effects reflect degradation of meaningful evidence rather than arbitrary prompt instability.On HotpotQA, correctness falls from 0.58 with retrieval to 0.22 without retrieval, while F1 falls from 0.629 to 0.270 and grounding reaches 0.0.
  • Cross-Dataset and Model Results: The intervention ordering original > duplicate ≫ remove ≥ replace appears on both HotpotQA and 2WikiMultihopQA.On 2WikiMultihopQA, correctness is 0.540 originally, 0.510 with duplication, 0.390 after removal, and 0.370 after replacement; GPT-5.2 shows the same qualitative separation.
  • 6.8 Two-Support Synergy Ablation (Q5): Removing both multi-hop supports causes a mean F1 drop of 0.493, compared with 0.205 and 0.186 for individual removals.Joint removal harmed performance more than the worse single removal in 19.6% of eligible examples, while 13.7% showed failure only after both supports were removed.
  • Interpretation: Trace-sensitive evaluation reveals evidence effects that answer-only metrics can miss.Examples include confidence distortion under duplication and answer-preserving but trace-divergent interventions.

7 Limitations

CUE-R’s scope is constrained by its intervention design, observable single-shot traces, proxy metrics, limited intervention set, and experimental scale. These choices support reproducibility but limit causal interpretation and generalization.

  • Scope and interpretation: Prompt-level perturbations measure interventional sensitivity rather than causal contribution in the strongest counterfactual sense.Perturbations simultaneously change prompt length, context distribution, and attention allocation.
  • Scope and interpretation: CUE-R evaluates logged behavior rather than recovering hidden internal reasoning or the model’s true internal mechanism.Its current experiments use shallow single-shot traces, with deeper agentic workflows left for future work.
  • Measurement limits: The grounding score is a coarse title-overlap proxy that may miss correct information within chunks and useful evidence absent from gold titles.Sentence-level attribution would provide a finer-grained grounding measure.
  • Measurement limits: Self-reported confidence is noisy, so confidence error is a lightweight instance-level proxy rather than distributional calibration analysis.The metric relies on the model’s self-reported confidence.
  • Design limits: The study uses only REMOVE, REPLACE, and DUPLICATE, while systematic combinatorial interventions remain future work.The two-support ablation provides initial evidence of non-additive interaction, but not exhaustive pairwise analysis.
  • Generalization limits: Validation covers two datasets and two model families with 100–200 examples per condition, leaving broader generalization to larger studies.The retrieval backbone is BM25 with top-k passage selection rather than a state-of-the-art dense retriever.
  • Design limits: Target selection uses a priority heuristic, and alternative policies could produce different intervention profiles.The heuristic favors model-used and support-aligned chunks.
  • Interpretation: The evidence-role taxonomy is an operational diagnostic framework, not an ontology of model cognition.Its categories should be interpreted as forensic tools for analyzing behavior.

8 Conclusion

CUE-R frames per-item RAG utility as an intervention problem combining observable retrieval-use traces with multi-axis utility measures. Across datasets and models, removals and replacements sharply impair correctness and grounding with large trace shifts, while duplication is milder but behaviorally non-neutral.

  • Conclusion: CUE-R combines observable retrieval-use traces, three perturbation operators, and multi-axis utility analysis to expose effects missed by answer-only metrics.The framework asks what operational utility each retrieved evidence item contributes after system use.
  • Conclusion: Across HotpotQA and 2WikiMultihopQA with Qwen-3 8B and GPT-5.2, REMOVE and REPLACE sharply reduce correctness and grounding while inducing large trace divergence.The reported experiments use 100–200 examples per condition.
  • Conclusion: DUPLICATE interventions are milder on answer quality but still measurably alter confidence error and trace behavior.This pattern indicates that answer redundancy does not guarantee behavioral neutrality.
  • Conclusion: CUE-R is a practical diagnostic tool for identifying retrieved items that are useful, redundant, or behaviorally destabilizing.The authors present its taxonomy as forensic value rather than an account of model cognition.
  • Conclusion: The current instantiation is limited to shallow single-shot traces, proxy-based grounding, and self-reported confidence.Future work should address deeper traces, stronger grounding metrics, sentence-level interventions, and dense retrievers.

B Metric Definitions

The paper defines correctness and answer F1 using normalized answer comparisons, while grounding and trace divergence are separately defined metrics. Confidence error measures absolute deviation between self-reported confidence and correctness.

  • Answer metrics: Soft correctness equals 1 when normalized answers match, yes/no values match, numeric values are equivalent, or token-level F1 is at least 0.8.The definition compares predicted answer y with gold answer y∗.
  • Answer metrics: Answer F1 is computed as token-level F1 after normalization using precision and recall over common tokens.The supplied definition expresses F1 through P and R.
  • Grounding: Grounding Score is defined in Equation (11).The supplied passage names the metric but does not provide its equation.
  • Confidence: Confidence Error is CE = |c − ⊮[is_correct]|, using self-reported confidence c clamped to [0, 1].It is an instance-level absolute error, not a distributional calibration metric.
  • Trace metrics: Trace Divergence is defined in Equation (9), including Jaccard distance between used chunk-identifier sets.For sets E and E′, dJaccard(E, E′) = 1 − |E∩E′|/|E∪E′|.

C Duplicate Position Sensitivity

The duplicate-position experiment compares front, after-original, and end insertion of copied chunks. Duplication remains largely answer-preserving, but earlier placements produce larger trace shifts, implicating positional salience.

  • Experimental setup: The experiment tests duplicate insertion at the front, immediately after the original, and at the end, the default position.It uses 60 HotpotQA examples with Qwen-3 8B.
  • Results: Across all placements, duplication remains largely answer-preserving while consistently producing non-zero evidence-use divergence.The table caption states the same comparison across the three positions.
  • Results: Front and after-original duplication produce larger trace shifts than end duplication, suggesting a role for positional salience beyond redundancy alone.The reported correctness changes are at most 5% for front, 6.7% for after-original, and 1.7% for end.

E Reproducibility Details

This section identifies Table 14 as the location of the experiments’ reproducibility information and key experimental parameters.

  • Table 14 summarizes the key experimental parameters for reproducibility.
  • Table 14 provides reproducibility details for all experiments.
  • The reproducibility information is presented in a table dedicated to experimental details.
Loading 2604.05467v1…