Source-linked AI summary
Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval
Pasha Abrishamchian, Eugene Yang, Andrew Yates, Nazli Goharian, Ophir Frieder
TL;DR
Coverage benchmarks leave open whether retrievers surface corpus evidence that contradicts pretrained expectations. CoverageBench-Negate adds negated nuggets and edited supporting documents to CoverageBench; BM25 distinguishes original from edited near-twin evidence better than dense Qwen3 and Jina, which stay near chance.
Problem
Existing coverage benchmarks mostly evaluate familiar facts, leaving open whether retrievers find corpus evidence that conflicts with pretrained knowledge.
Source: §1 · p. 1Method
The paper negates each nugget and edits its supporting documents to support the negation while preserving context, creating paired evidence in CoverageBench-Negate.
Source: §1 · p. 1Results
BM25 distinguishes original from edited near-twin evidence better than dense Qwen3 and Jina, which stay near chance.
Source: §4.2 · p. 4Takeaways & Limitations
CoverageBench-Negate provides a diagnostic for corpus faithfulness under controlled contradiction and an initial characterization separating lexical and dense retrievers.
Source: §5 · p. 4The benchmark assumes negated claims conflict with parametric knowledge without verifying that conflict directly, and does not analyze whether retained links differ systematically from excluded ones.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Coverage-evaluation benchmarks quantify whether retrieved evidence covers the different pieces of information needed to satisfy an information need. Existing benchmarks mostly consider factual claims that may already be known to pretrained retrieval components, leaving open whether retrievers find corpus evidence when it contradicts those priors. We study how retrievers behave when their parametric knowledge and the information in a corpus disagree. To do so, we develop a process to augment a nugget-based coverage benchmark with controlled contradictions. For each nugget, we create a negated counterpart and edit its supporting documents to support that negation while preserving the context. Applying this process to CoverageBench, a benchmark measuring information coverage, we create CoverageBench-Negate, spanning six datasets and roughly 63,000 original-edited evidence pairs (one per nugget-document link). We attach rich construction metadata to every pair and additionally provide out-of-family LLM judgments over all pairs and a human audit of a sample. In baseline experiments, BM25 separates original from edited near-twin evidence substantially better than dense retrievers (Qwen3 and Jina), which stay near chance. We release the benchmark on Hugging Face Datasets (nuggets, documents, pairs) together with the construction materials, including the prompts used to create it.
1 Introduction
Coverage evaluation asks whether retrieved documents collectively contain the distinct information needed to answer a query, not merely whether each document is relevant. This paper introduces CoverageBench-Negate to test whether retrievers surface corpus evidence that contradicts pretrained expectations.
1 Introduction
- Relevant documents can still leave an answer incomplete when they repeat the same point or omit an important claim.Coverage therefore concerns a complete set of distinct evidence, not relevance alone.Source: §1 · p. 1
- Existing coverage benchmarks mostly use familiar facts, leaving open whether retrievers surface corpus evidence that conflicts with pretrained knowledge.The question matters because dense retrievers, rerankers, and LLM-based query expansion can inherit knowledge from pretraining.Source: §1 · p. 1
- CoverageBench-Negate keeps topics fixed, negates each nugget, and edits supporting documents to support that negation while preserving surrounding context.This paired design isolates claim polarity from topic or entity shifts.Source: §1 · p. 1
- BM25 distinguishes original from edited near-twin evidence better than dense Qwen3 and Jina, which stay near chance.The benchmark and validations are released with construction metadata for filtering, audit, and reuse.Source: §1 · p. 2
2 Background
Coverage-based retrieval evaluation measures whether a retrieved set covers distinct information needed for an answer. CoverageBench-Negate extends this setting to ask whether retrievers find corpus-supported claims that conflict with parametric knowledge.
2 Background
- Information coverage measures how many distinct pieces of information a retrieved set covers.Nugget-based evaluations use atomic claims to assess answer completeness and grounding.Source: §2 · p. 2
- CoverageBench standardizes topics, nuggets, relevance labels, collections, and baseline rankings across retrieval settings.Retrieval-side coverage metrics were shown to correlate strongly with information coverage in downstream RAG responses [19].Source: §2 · p. 2
- CRUMQs tests reliance on parametric knowledge when evidence is absent, while CoverageBench-Negate tests coverage of corpus-supported claims that contradict it.The two settings probe complementary conditions.Source: §2 · p. 2
- Dense retrievers may treat a claim and its negation as similar because they share a topic, despite the changed meaning.Controlled pairs keep context fixed while changing the supported claim, testing whether retrieval responds to that change.Source: §2 · p. 2
3 Benchmark Construction
CoverageBench-Negate adds negated nuggets and edited supporting documents to six CoverageBench datasets while retaining the original topics, nuggets, and documents. Its link-level pairs carry construction metadata and receive independent LLM judgments and a human audit.
3 Benchmark Construction
- The released benchmark contains 63,372 link-level pairs spanning every topic and 2,975 nuggets, from 116,842 original support links.It reuses 284 topics and retains original nuggets and links; Fair Ranking 2022 is omitted because its nuggets are fairness facets rather than factual claims.Source: §3 · pp. 2–3
3.1 Negated Nugget Construction
- Each original nugget receives a negated counterpart checked for meaning reversal, naturalness, entity preservation, and mutual exclusivity.Embedding checks label topicality, difference, and collisions; these checks label quality rather than filtering nuggets.Source: §3.1 · p. 3
3.2 Edited Document Construction
- For each surviving nugget–document link, the editor rewrites the supporting span to support the negation and checks that context is preserved.The pipeline releases 63,372 pairs, or 54.2% of links; missing supporting spans account for 35.9% of links removed.Source: §3.2 · p. 3
3.3 Metadata
- Metadata records negation type, edit size, similarity measures, and a pass, review, or fail confidence signal for each pair.A pair passes when the edit is closer to the negated nugget than the original and remains sufficiently similar to its source.Source: §3.3 · p. 3
3.4 Validation
- An out-of-family judge confirms all three pair criteria on 78% of pairs, while human ratings show stricter and more variable judgments on edit criteria.The human audit covers a stratified 120-pair sample; mutual exclusivity is recorded as a label because judging it is subjective.Source: §3.4 · p. 3
4 Baseline
The baseline evaluation measures augmented coverage and uses paired original–edited documents to diagnose retriever preferences. BM25 distinguishes originals from negation-supporting edits more reliably than the dense retrievers, particularly for small edits.
4 Baseline
- The baseline evaluates top-20 rankings with α-nDCG and Subtopic Recall, then analyzes paired score preferences on retriever-specific top-1,000 subsets.Pairs are stratified by edit size, from tiny (<2% change) to vast (>25% change); smaller edits retain more wording and context.Source: §4 · pp. 3–4, §4.2 · p. 4
4.1 Augmented Coverage Evaluation
- Across datasets, augmentation reduces coverage: BM25 drops 0.083 in α-nDCG@20 and 0.145 in StRecall@20, while dense retrievers drop 0.140–0.146 and 0.180–0.181, respectively.The relative α-nDCG drop remains larger for Qwen3 and Jina than BM25, while the relative StRecall gap nearly closes.Source: §4.1 · p. 4, Table 1 · p. 4
- On CAsT, α-nDCG remains approximately flat for all three retrievers because some edited documents add non-redundant gain, although StRecall falls for all three.The paper treats CAsT as a metric-specific exception, not evidence against the general degradation pattern.Source: §4.1 · p. 4
4.2 Paired Score-Gap Diagnostic
- BM25 prefers originals in about 84% of tiny-edit pairs, falling to about 61% as edits grow; Qwen3 (54–61%) and Jina (49–56%) stay near chance.Rank-level results show the same pattern: BM25 ranks originals above edits in 83% of tiny-edit pairs, falling to 57% for vast edits.Source: §4.2 · p. 4, Table 2 · p. 4
5 Conclusion
CoverageBench-Negate is presented as a diagnostic for coverage-oriented retrieval under controlled contradiction. The authors identify unverified parametric conflict and unexamined link-selection differences as limitations.
5 Conclusion
- CoverageBench-Negate tests whether first-stage retrievers cover corpus evidence that contradicts pretrained expectations.The authors characterize it as a useful diagnostic for coverage-oriented retrieval.Source: §5 · p. 4
- The premise that negated claims conflict with parametric knowledge is not directly verified, and retained links are not compared systematically with excluded links.Independent edits may also introduce higher-order contradictions; an independent out-of-family judge and human audit mitigate the LLM-based construction and checks.Source: §5 · p. 4
- Full per-bucket and per-dataset score-gap and rank-level numbers are released with the benchmark on Hugging Face.Source: §5 · p. 4
GenAI Usage Disclosure
The disclosure describes generative AI’s roles in constructing and validating CoverageBench-Negate, as well as in assisting with paper preparation.
GenAI Usage Disclosure
- Llama-3.3-70B-Instruct generates negated nuggets, localizes supporting spans, rewrites edited documents, and performs LLM-based quality checks.Source: GenAI Usage Disclosure · p. 5
- Qwen3-Embedding-8B produces embedding-based checks and confidence tiers, while Gemma-4-31B-it serves as the independent out-of-family validation judge.Source: GenAI Usage Disclosure · p. 5
- The authors also used generative AI tools for writing and editing, code debugging, and figure creation, and say they reviewed and take responsibility for the paper.Source: GenAI Usage Disclosure · p. 5