PaperlayerView exact revision on arXiv

Source-linked AI summary

Relevant Is Not Warranted: Evidence-Force Calibration for Cited RAG

Pin Qian, Su Wang, Xiaoyuan Wang, Yihang Chen, Wenxuan Xu, Qiaolin Yu, Shuhuai Lin, Sipeng Zhang, Junxian You, Xinpeng Wei

arXiv:2605.28044v1cs.AI

TL;DR

Cited RAG can present a real, relevant source as warrant for wording stronger than the evidence supports. The paper introduces FORCEBENCH, which holds evidence fixed and contrasts calibrated claims with localized force-raised variants across five force axes. On the fixed evaluation set, explicit warrant-strength prompting lowers aggregate model-judge MVR from 47.2% to 24.5%, but residual violations remain.

  • Problem

    Cited RAG evaluation can treat visible, relevant citations as grounding even when they under-warrant the attached wording.

  • Method

    FORCEBENCH holds one cited passage fixed and pairs an evidence-calibrated claim with a localized force-raised variant across five operational force axes.

  • Results

    Aggregate model-judge MVR falls from 47.2% with generic support prompting to 24.5% with explicit warrant-strength prompting on the fixed locality-filtered evaluation set.

  • Takeaways & Limitations

    MVR and force sensitivity can supplement conventional support accuracy as plug-in diagnostics for whether citation evaluators respect evidence-force ordering.

  • Takeaways & Limitations

    ForceBench isolates single English claim-citation pairs and excludes multi-hop synthesis, multiple-source aggregation, source authority, citation recency, placement, and answer-level usefulness.

Abstract

from arXiv · show

Cited RAG evaluation often treats visible sources as a grounding signal, but a real, topically relevant citation can still under-warrant the attached wording. We study this diagnostic failure as citation laundering: a related source is presented as warrant for an over-strong claim. We introduce FORCEBENCH, a contrastive stress test for evidence-force calibration. Each item holds a cited passage fixed and pairs an evidence-calibrated claim with a localized force-raised variant across five operational axes: relation, modality, scope, temporal validity, and numeric specificity. A calibrated evaluator should score the evidence-calibrated claim higher. Headline experiments use a fixed, locality-filtered 198-pair evaluation set. A citation-presence sanity check is uninformative by design; token and entity overlap still violate monotonicity on 32.8--36.4% of pairs. Across four reported model judges, standard generic support prompting is insufficient for this force-calibration stress test (aggregate MVR 47.2%), while explicit warrant-strength prompting lowers MVR to 24.5% but remains imperfect. We release the benchmark, prompts, outputs, and plug-in pipeline so citation evaluators can report monotonicity violation rate and force sensitivity alongside conventional support metrics.

1 Introduction

Citation laundering occurs when a real, topically relevant citation is used to support wording stronger than the cited evidence warrants. ForceBench turns this failure into a contrastive monotonicity test across five force axes.

  • 1 Introduction: Examples raise association to causation, subgroup findings to general claims, uncertainty to certainty, or ranges to exact values.These shifts can pass citation-presence and surface-overlap checks despite exceeding the evidence.
  • 1 Introduction: ForceBench holds cited evidence fixed while contrasting an evidence-calibrated claim with a minimally stronger overclaim across relation, modality, scope, temporal, and numeric axes.A calibrated evaluator should score the warranted claim higher.
  • 1 Introduction: A useful evaluator must preserve citation-warranted wording while rejecting only the added force needed for ranking, filtering, and repair.Generic support judgments can treat weaker and stronger variants equally when the evidence supports only the weaker wording.
  • 1 Introduction: Citation laundering is a claim-evidence mismatch distinct from missing citations, irrelevant retrieval, and ordinary unsupported generation.The source exists and is related, but the citation under-warrants the claim’s force.
  • 1 Introduction: The paper defines monotonicity violation rate and force sensitivity and evaluates deterministic baselines plus four deployed model judges.Standard generic citation-support prompting is insufficient, whereas explicit warrant-strength prompting helps but remains imperfect.

2 Evidence-Force Calibration

Evidence-force calibration asks whether a claim stays within the warrant licensed by its cited passage. ForceBench operationalizes this as a localized increase in one structured force dimension while holding the core proposition fixed.

  • 2 Evidence-Force Calibration: ForceBench asks whether evidence warrants a claim at the same level of relation, modality, scope, temporal validity, and numeric specificity.This is stricter than asking only whether cited evidence supports the topic or claim generally.
  • 2 Evidence-Force Calibration: Claim force is modeled as a structured categorical tuple, while evidence force captures the warrant licensed by the cited passage, including limitations and context.A claim is calibrated when it does not exceed the evidence on any included axis.
  • 2 Evidence-Force Calibration: ForceBench constructs pairs that preserve the same core proposition and differ by a localized force increase on one primary axis.Many claims are incomparable, so the benchmark excludes contrasts that combine multiple shifts.
  • 2 Evidence-Force Calibration: The five axes cover shifts from association to causation, possibility to certainty, subgroup to general scope, dated evidence to timeless claims, and ranges to exact values.The axes are operational and intentionally non-exhaustive.
  • 2 Evidence-Force Calibration: ForceBench isolates local warrant gaps rather than fabricated quotations, wrong-entity claims, missing links, or broad unsupported additions.Its diagnostic property is that support should decrease when claim force increases while evidence remains fixed.

3 ForceBench

ForceBench is a locally annotated, adjudicated benchmark built from cited claim-evidence contexts with calibrated and force-raised variants. Conservative quality control yields a fixed locality-filtered evaluation set for testing evidence-force ordering.

  • 3 ForceBench: Each ForceBench item contains a cited passage, decisive evidence span, calibrated claim, localized force-raised claim, primary axis, severity, and repair.The calibrated wording is judged licensed by the passage, while the stronger variant exceeds it on one primary axis.
  • 3 ForceBench: The benchmark samples cited QA and attribution contexts whose passages expose local boundaries in relation, modality, scope, temporal status, or numeric specificity.Sources include AttributionBench-derived rows, AttributedQA, GaRAGe, and additional ExpertQA examples.
  • 3 ForceBench: Every candidate is independently annotated by two research assistants, reconciled through adjudication, and checked for evidence-span and locality quality.The protocol asks what wording the displayed citation licenses and tests a stronger variant against the same passage.
  • 3 ForceBench: Admissible pairs preserve the cited passage, answer context, main entity, event, and topical relation while differing primarily in one force dimension.Candidates are rejected when the calibrated claim is unsupported, the stronger claim is supported, or the contrast introduces unrelated additions.
  • 3 ForceBench: 229 adjudicated pairs remain in the benchmark pool, while all headline results use a fixed 198-pair locality-filtered evaluation set.Per-axis retained counts range from 39 to 51 after capping surplus candidates and removing nonlocal rows.

4 Metrics

The paper evaluates whether support scores preserve the warranted ordering between evidence-calibrated and force-raised claims. It uses monotonicity violation rate and force sensitivity, with categorical judge outputs mapped to numeric scores.

  • A monotonicity violation occurs when a force-raised claim receives support greater than or equal to its evidence-calibrated counterpart.
  • MVR measures whether an evaluator preserves strict ordering, treating ties as failures to prefer the warranted wording.The strict inequality makes ties violations for ranking, filtering, or repair.
  • Force sensitivity measures the average score margin between the evidence-calibrated and force-raised claims, with higher values preferred.Both metrics are needed because larger margins can coexist with identical discrete labels.
  • Categorical judgments are converted to scores, assigning 1 to fully supported or force-calibrated labels, 0.5 to partially supported labels, and 0 to several negative labels.Unparsable or refused responses count as violations with margin 0.

5 Experiments

The experiments test whether citation evaluators preserve the preference for evidence-calibrated claims over localized force-raised variants. Force-aware prompting improves aggregate ordering, while topical-overlap signals and generic support prompting remain insufficient.

  • Baselines: Citation presence cannot distinguish paired claims, yielding 100% MVR by construction.
  • Baselines: The axis-aware lexical prototype is the strongest deterministic baseline, with MVR 0.232, but still fails when force is implicit.
  • Main Results: Across four judges, generic support prompting produces MVR 0.472 and force sensitivity 0.333 on the 198-pair evaluation set.
  • Main Results: Explicitly force-aware prompting lowers MVR to 0.245 and raises force sensitivity to 0.754 on the same model-pair units.
  • Prompt-Rubric Ablation: Both force prompts improve over generic support prompting for all four judges, whereas the dummy-axis control often worsens MVR.
  • Force-Axis Analysis: Relation shifts are comparatively visible, while numeric, temporal, and modality shifts are harder because they require attending to ranges, dates, hedges, or uncertainty.

6 Discussion

Evidence-force calibration requires evaluators to preserve citation-warranted wording while rejecting only localized increases in claim force. The fixed stress test shows force-aware prompting improves ordering but leaves residual violations.

  • Evidence-force calibration keeps the warranted claim while rejecting only the extra force.
  • Figure 4 shows relation shifts are comparatively visible, while numeric, temporal, and modality shifts are harder under ordinary support prompting.
  • The hand-aligned diagnostic heuristic still violates monotonicity on 35.0% of numeric and 43.6% of temporal examples.
  • The contrastive design holds evidence fixed and tests whether stronger claims receive lower support, reducing retrieval quality and external truth as alternative explanations.

7 Related Work

ForceBench extends citation evaluation from topical support to whether fixed evidence warrants claim force across a local contrast. It complements broader attribution, factuality, and citation-quality benchmarks.

  • ForceBench complements attribution and citation benchmarks that evaluate source verification, citation quality, or broader citation diagnostics.
  • ForceBench fixes the cited passage and measures local monotonicity under a controlled force increase.
  • Factuality, hallucination, NLI, and fact-verification work provide adjacent foundations for truthfulness, support, contradiction, neutrality, and uncertainty distinctions.
  • Its claim-citation pair unit tests whether the displayed citation licenses stronger wording.

8 Conclusion

ForceBench targets evidence warrant as a missing layer in cited RAG evaluation by holding evidence fixed and increasing claim force locally. On its fixed evaluation set, explicit warrant-strength prompting substantially improves monotonicity but does not eliminate violations.

  • 32.8–36.4% of overlap-baseline pairs violate monotonicity, while generic-support model judges reach 47.2% aggregate MVR.
  • Explicit warrant-strength prompting reduces aggregate model MVR to 24.5%, but residual failures remain.
  • Citation-presence checks should be supplemented with warrant-sensitive verification using MVR and FS as plug-in stress-test metrics.

Limitations

ForceBench is a targeted diagnostic whose results depend on its fixed evaluation design, prompts, and system measurements. Its scope excludes several broader citation and answer-level properties.

  • ForceBench uses a compact, targeted stress-test set, so its MVR numbers are not prevalence estimates.
  • Prompt gains reflect the selected evaluation set and explicit warrant-strength task framing.
  • Absolute and relative MVR values can change with model drift, decoding changes, and prompt-target alignment.
  • The benchmark isolates single English claim-citation pairs and excludes multi-hop synthesis, source authority, citation recency, placement, and answer-level usefulness.

Ethical Considerations

ForceBench is intended for citation-warrant evaluation, not prevalence estimation or judgments about real-world truth. It isolates whether cited evidence warrants the force of an attached claim, including in sensitive domains.

  • ForceBench identifies cases where cited evidence under-warrants the force of a generated claim.
  • The benchmark holds the source fixed and tests whether support decreases when only claim force becomes stronger.
  • ForceBench isolates a local citation failure that can survive relevant retrieval and plausible citation placement.
  • The resource assumes the cited passage is available and evaluates whether the attached claim is calibrated to that passage’s warrant.
  • ForceBench operationalizes evidential proportionality for generated cited answers through local contrastive pairs under one fixed citation.

B Dataset and Annotation Documentation

The benchmark is constructed through sampling, double annotation, adjudication, and locality filtering, with accepted pairs differing locally in claim force along defined axes. Its examples and checks enforce evidence-only judgments and conservative inclusion.

  • Sampling and candidate drafting: Contexts are retained only when the cited passage shows a visible local boundary across relation, modality, scope, temporal status, or numeric specificity.
  • Candidate accounting: 433 candidate rows entered annotation, 283 were accepted after adjudication and quality control, and surplus rows from overrepresented axes were capped before reporting.
  • Human review: All candidate rows were independently double-annotated before adjudication, with pre-adjudication agreement of Cohen’s κ = 0.78.
  • Annotation protocol: Annotators used only the displayed cited passage and ignored whether the claim might be true according to outside knowledge.
  • Annotation protocol: Each annotation row contains context, cited passage, calibrated and force-raised claims, and candidate metadata, with release-facing fields corrected during review.
  • Acceptance criteria: A row is accepted only when the calibrated claim is supported, the stronger claim is under-warranted, and the contrast is local along one dominant axis.
  • Reject criteria: Reject criteria exclude unsupported calibrated claims, supported stronger claims, changed targets, multiple unclear force shifts, unrelated additions, outside-knowledge judgments, and indecisive evidence spans.
  • Axis and severity definitions: The five axes distinguish shifts in relation, modality, scope, temporal status, and numeric specificity, while severity records the magnitude of the mismatch.
Loading 2605.28044v1…