PaperlayerView exact revision on arXiv

Source-linked AI summary

EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models

Suryadeep Singh Deswal

arXiv:2609.17081v1cs.CL

TL;DR

Answer accuracy can miss whether grounded models use the right evidence or respond appropriately to missing or conflicting evidence. EviScope evaluates the same questions under counterfactual evidence changes, and its paired metrics reveal model-dependent behavior, including weaker QCS for explicit gates on two local models. The benchmark also finds high Gemini joint success alongside continued conflict-blind answering, while its scope and validation leave limits on generalization and fine-grained support checks.

  • Problem

    Answer-only evaluation can obscure unsupported answers, wrong sources, ignored evidence, and answers given despite insufficient or conflicting evidence, although these errors require different remedies.

    Source: §1 · p. 1

  • Method

    EviScope pairs the same question with counterfactual evidence variants and evaluates grounding with conventional and quartet-level paired metrics.

    Source: §1 · p. 1, §2 · p. 2, §4 · p. 2

  • Results

    The evidence-action gate lowers QCS versus vanilla RAG on both local models, while Gemini reaches 0.944 joint success under both prompts and still answers 5% of conflict cases.

    Source: §6 · p. 3

  • Takeaways & Limitations

    Paired evidence transitions distinguish unsupported answering, conflict blindness, and wrong non-answer actions rather than scoring answers and citations in isolation.

    Source: §8 · p. 4

    EviScope-v1.1 is compact and SQuAD-derived, omits source authority, recency, and multi-document synthesis, and its model and prompt sample limits generalization.

    Source: §9 · p. 4

Abstract

from arXiv · show

Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or produced when evidence is insufficient or contradictory. We introduce EviScope, a paired counterfactual benchmark that holds the question fixed while adding, removing, distracting, or contradicting its evidence. EviScope-v1.1 contains 40 four-condition quartets with repaired counterfactual claims and span-level support labels for automatic evaluation. Across 960 gold-blind generations from Qwen2.5-7B, Llama 3.1 8B, and Gemini 3.5 Flash, paired metrics expose model-dependent grounding behavior that answer accuracy hides. On two local open models, an explicit evidence-action gate underperforms vanilla RAG on QCS: 0.15 vs. 0.50 for Qwen and 0.10 vs. 0.375 for Llama. Gemini reaches 0.944 joint success under both prompts, yet still answers 5% of conflict cases after contradiction insertion. EviScope therefore distinguishes unsupported answering, conflict blindness, and wrong non-answer actions rather than scoring answers alone.

1 Introduction

Answer accuracy alone can hide whether a grounded model used the right evidence or answered despite insufficient or conflicting evidence. EviScope tests evidence sensitivity by changing the evidence while holding the question fixed.

1 Introduction

  • Answer-only evaluation can collapse correct answers from memory, irrelevant citations, ignored evidence, and answers despite missing or conflicting evidence into the same score.These error types require different remedies.Source: §1 · p. 1
  • EviScope holds each question fixed while adding, removing, distracting, or contradicting its evidence context.The expected behavior is to answer with support, remain stable under irrelevant passages, abstain when support is removed, and flag inserted conflict.Source: §1 · p. 1
  • EviScope-v1.1 is a benchmark/evaluator package, not a new model, with 40 SQuAD questions in sufficient, noisy, insufficient, and conflicting variants.Its evaluator includes conventional grounding measures and paired quartet-level metrics.Source: §1 · p. 1

2 Related Work and Positioning

EviScope positions paired evidence interventions as a way to assess how the same question is handled when support is added, removed, distracted, or contradicted. Its framing connects evidence sufficiency and source-focused evaluation with paired transition metrics.

2 Related Work and Positioning

  • EviScope adds paired interventions to evidence-centric evaluation, testing how the same question changes when support is added, removed, distracted, or contradicted.Related work includes citation and factuality evaluation as well as hallucination recognition.Source: §2 · p. 1
  • The related work also covers methods for improving retrieval use, routing, compression, or reliance on context, with EviScope testing faithfulness alongside cost.The cited systems include Self-RAG, RA-DIT, Adaptive-RAG, RECOMP, and context-aware decoding.Source: §2 · p. 1
  • Compared with broader input-perturbation evaluations, EviScope links support, distractors, removal, and contradiction to paired transition metrics in one grounded-evidence quartet.Its positioning table defines “Suff.” as explicit evidence-sufficiency labels and “Paired” as evaluating the same question under counterfactual evidence variants.Source: §2 · p. 2, Table 1 · p. 2

3 Benchmark Construction

EviScope-v1.1 builds four evidence conditions around each of 40 public SQuAD question-answer-context records. Its repaired conflicts and external review support the benchmark’s primary action and evidence-status labels.

3 Benchmark Construction

  • EVISCOPE-V1.1 starts from 40 public SQuAD question-answer-context records, capped at three base examples per source title for topical diversity, and creates four context variants per question, yielding 160 examples.Source: §3 · p. 2
  • Version 1.1 freezes v1 as legacy and repairs every conflict passage to be grammatical, answer the question with a same-type alternative, contradict support, exclude the original answer, and contain no conflict cue.Repairs modify the answer-bearing claim where possible; strict lints and consistency checks cover all 40 quartets.Source: §3 · p. 2
  • Two external annotators reviewed all 160 blinded examples, with 0.994 agreement on evidence status and expected behavior and 1.000 on conflict presence.Agreement on secondary answer-valid and citation-valid QA notes was lower at 0.744; those fields are excluded from primary labels and retained as audit notes.Source: §3 · p. 2

4 Metrics

EviScope combines deterministic answer, source, support, abstention, conflict, and cost measures with paired metrics over evidence variants. Its flagship quartet consistency score checks whether behavior succeeds under all four conditions.

4 Metrics

  • The evaluator measures answer and source accuracy, span support, unsupported citations, abstention, conflict handling, hallucination under expected non-answer behavior, and token, call, and latency costs.It uses normalized answer matching, exact document-ID citation checks, span containment, and no LLM judge.Source: §4 · p. 2
  • Quartet consistency requires success across sufficient, noisy, insufficient, and conflicting variants, with success defined respectively as supported answering, abstention, or conflict flagging.The paired metrics also include noise robustness, conflict sensitivity, and grounded utility per call.Source: §4 · p. 2
  • Paired policy comparisons use 10,000-sample paired bootstrap intervals with seed 42 and an exact two-sided McNemar test over quartet success.Table 3 reports the main EviScope-v1.1 results and identifies answer accuracy on answerable rows and conflict-blind answering after contradiction insertion.Source: §4 · p. 3, Table 3 · p. 3

5 Reference Systems

The study compares vanilla RAG with an evidence-action gate across three models under matched inputs and output conditions. The gate is a diagnostic prompt baseline, not a proposed solution.

5 Reference Systems

  • The evidence-action gate classifies context as answerable, insufficient, or conflicting before selecting an output action.Vanilla RAG answers from supplied documents with citations and may abstain or flag conflict.Source: §5 · p. 3
  • Both policies receive only the question and document IDs and text, use temperature zero, and return the same structured schema.The prompts differ only in whether they explicitly ask the model to classify evidence state before choosing an action.Source: §5 · p. 3
  • The comparison covers Qwen2.5 7B, Llama 3.1 8B, and Gemini 3.5 Flash, with 160 examples per model-policy pair.The final matrix contains 960 logical generations and zero unresolved parse failures.Source: §5 · p. 3

6 Results

Paired evidence metrics show that the evidence-action gate's effects vary by model: it lowers QCS on both local models but edges up on Gemini. Gemini nonetheless answers some conflict cases, while the local models exhibit different failure patterns.

6 Results

  • Qwen's gate reduced evidence removal sensitivity from 0.750 to 0.150 and QCS from 0.500 to 0.150, despite equal 0.738 answer accuracy.It correctly abstained on only 8 of 40 insufficient variants; most remaining cases were mislabeled as conflict.Source: §6 · p. 3
  • Llama's gate raised answer accuracy to 0.850 from 0.700 but lowered conflict sensitivity to 0.175 from 0.550 and QCS to 0.100 from 0.375.It answered 77.5% of conflicting variants.Source: §6 · p. 3
  • Both Gemini policies reached 0.944 joint success, 0.900 noise robustness, and 0.875 conflict sensitivity, yet answered 5% of conflict cases.The gate's QCS was 0.875 versus 0.850 for vanilla; the paired test was inconclusive with only one discordant quartet.Source: §6 · p. 3
  • Gate-minus-vanilla QCS was −0.35 for Qwen, −0.275 for Llama, and +0.025 for Gemini, indicating model-dependent effects.The paired QCS-discordant quartets favored vanilla for Qwen and Llama, while Gemini had one discordant quartet favoring the gate.Source: §6 · p. 3
  • Both local models use one call per example, while Gemini averaged 1.81 calls for vanilla and 2.09 for the gate.Call-normalized utility is reported as an engineering cost measure, not a model-quality metric.Source: §6 · pp. 3–4

7 Qualitative Errors

The qualitative examples show that similar answer-level outcomes can conceal distinct evidence-handling failures. Models may persist with an answer after contradiction, misroute evidence removal, or answer despite an explicit conflicting source.

7 Qualitative Errors

  • Qwen vanilla repeats its supported Beyonce popularity claim after a contradictory source is inserted.The example illustrates failure to change behavior when the evidence changes.Source: §7 · p. 4
  • For the Sichuan earthquake year, Qwen's gate switches to conflict after support is removed, although no incompatible answer remains.The same example contrasts with Llama's gate, which answers “2008” despite a conflicting document asserting “2010.”Source: §7 · p. 4
  • Both Gemini prompts answer 5% of conflict variants, while merged abstention and conflict-flag metrics would hide these distinct errors.The examples distinguish wrong non-answer actions from answering when evidence conflicts.Source: §7 · p. 4

8 Conclusion

EviScope evaluates whether grounded models respond appropriately as evidence changes. Its results show model-dependent effects from explicit routing and residual conflict blindness, motivating paired evidence-transition evaluation.

8 Conclusion

  • EviScope evaluates grounded models by testing whether their responses change appropriately as evidence changes.The conclusion draws on results across Qwen2.5 7B, Llama 3.1 8B, and Gemini 3.5 Flash.Source: §8 · p. 4
  • Explicit routing can degrade QCS on local models, while Gemini reaches high joint success but still exhibits conflict blindness.The conclusion characterizes these as prompt-level negative results and a stronger-model ceiling.Source: §8 · p. 4
  • Grounded evaluation should report paired evidence transitions and distinguish wrong non-answer actions rather than score answers and citations in isolation.Source: §8 · p. 4

9 Limitations

EviScope-v1.1 is compact and SQuAD-derived, and its counterfactual conflicts omit source authority, recency, and multi-document synthesis. Validation supports categorical labels but leaves fine-grained source/span validation limited, and the model study has limited breadth.

9 Limitations

  • EviScope-v1.1 is compact and SQuAD-derived, and its conflicts omit source authority, recency, and multi-document synthesis.All conflicts require flagging, although realistic systems may resolve some using metadata.Source: §9 · p. 4
  • External validation covered the action, evidence-status, and conflict-presence labels, but completed sheets did not collect explicit support-span or support-document aliases.The paper therefore describes fine-grained source/span validation as limited.Source: §9 · p. 4
  • Model-level generalization is limited by two local open-weight models, one proprietary API model, and two prompts.The authors nevertheless state that the results demonstrate why insufficient and conflicting evidence require separate actions and paired metrics.Source: §9 · p. 4

Ethics and Responsible Research

  • Two external annotators worked independently using a fixed codebook; the annotators volunteered and gave consent.The work received no external funding, and the author declared no conflicts of interest.Source: Ethics and Responsible Research · p. 4

A Additional Qualitative Examples

The appendix provides an example quartet, representative paired failure cases, evidence-state decision rules, and human-validation fields. The validation procedure used independently completed, blinded sheets.

Related papers

Loading 2609.17081v1…