PaperlayerView exact revision on arXiv

Source-linked AI summary

AtomCite: Verification and Correction of Supplied Page-Level Citations in Multi-Page Documents

Chen Qian, Yimeng Wang, Yu Chen, Lingfei Wu, Andreas Stathopoulos

arXiv:2609.05802v1cs.CLcs.AIcs.IR

TL;DR

Existing benchmarks do not evaluate verification and correction of supplied page-level citations in document images. AtomCite checks claims against cited pages, and DocCite evaluates the task; across three model families, image grounding outperformed OCR-only conditions while repair substantially improved citation precision.

  • Problem

    No benchmark evaluates systems that verify and correct supplied page-level citations in multi-page document images, despite the need to check whether a cited page supports a claim.

    Source: §1 · p. 2, §3 · p. 2

  • Method

    AtomCite parses claim-page pairs, checks cited pages using OCR and images, and applies typed repair actions; DocCite evaluates the task with deterministic labels and a separate human-audited layer.

    Source: §1 · pp. 1–2, §4 · p. 4

  • Results

    Across three model families, image-aware verification beat every OCR-only condition; repair raised citation precision from 34.3% to 87–90% at ≈92% retention.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    Noisy automatic natural-error labels can bias verifier accuracy enough to reverse system rankings, motivating human-audited labels in DocCite.

    Source: §1 · p. 2, §7 · p. 7

    The documents are English business/legal images; other scripts, born-digital PDFs, and non-page granularities are out of scope.

    Source: §6 · p. 7

Abstract

from arXiv · show

Large language models answering questions over multi-page documents are expected to cite the supporting pages, yet supplied citations are sometimes inaccurate, and current evaluations score citations at generation time or against text passages: no existing benchmark evaluates whether a system can verify and correct a page-level citation already attached to an answer. We propose AtomCite, an agentic framework that parses an answer into claims, checks each claim against the image of its cited page, and applies a deterministic repair policy. To evaluate it, we introduce DocCite, to our knowledge the first benchmark for systems that verify and correct page-level citations in document images. Built on MP-DocVQA and DUDE, it combines 928 validated injected instances with 2,468 candidate natural errors harvested from frontier- and efficiency-tier models, of which a two-annotator audit confirms 1,909 as genuine errors. Primary labels are assigned deterministically, not by LLM judges, with the human audit as a separate validation layer. Across three model families (Gemini, Claude, and GPT), AtomCite reaches around 93% binary verification accuracy on the injected benchmark, significantly outperforming every OCR-only condition, including a compute-matched control, and exceeding every prior text-based baseline given the same OCR text. Its repair policy lifts citation precision on the injected mix from a constructed 34% to 87-90% while retaining over 90% of correct claims. AtomCite also transfers: with frozen prompts and zero training, it raises the hallucination-detection scores of two open 7-8B models on five public benchmarks above the same models prompted as direct judges. Finally, the audit shows that noise in automatic labels biases measured verifier accuracy and can reverse system rankings, so evaluations relying only on synthetic or automatic labels risk mismeasuring verification capability.

1 Introduction

Supplied page-level citations in multi-page document answers remain largely unevaluated, despite the risks of trusting incorrect citations. AtomCite verifies claims against cited-page text and images, while DocCite provides a benchmark for verification and correction.

1 Introduction

  • No existing benchmark evaluates whether a system can verify and correct page-level citations already attached to answers over multi-page documents.Earlier work scores citations against text passages, uses evidence pages as ground truth for answering, or evaluates citations at generation time.Source: §1 · p. 1
  • AtomCite parses claims with cited pages, checks them through OCR-text and image sub-agents, and applies typed actions to keep, repoint, or remove claims.The cited page determines the verdict; surrounding pages can propose corrections but cannot determine support.Source: §1 · p. 1
  • DocCite combines 928 validated injected instances with 2,468 natural-error candidates, with deterministic primary labels and a separate human-audited semantic layer.The audit confirms 1,909 natural candidates as genuine errors.Source: §1 · p. 2
  • The modality and architecture findings replicate across three model families and two OCR engines, and frozen prompts transfer to five hallucination benchmarks for two open 7–8B models.The transfer uses zero training.Source: §1 · p. 2
  • 93.7% binary accuracy with Gemini, 93.0% with Claude, and 93.0% with GPT; AtomCite beats every OCR-only condition, and correction raises precision from 34.3% to 87–90% while retaining about 92% of correct claims.The gap persists against compute-matched OCR controls; the reported natural-error audit also finds automatic-label noise can bias verifier comparisons.Source: §1 · p. 2

2 Related Work

Prior citation evaluation, multi-page document QA, and verification or correction systems address adjacent tasks, but do not center on post-hoc verification and repair of page-level citations in document images.

Answer attribution and citation evaluation

  • Answer-attribution work evaluates support from free-text passages or bibliographic records, while this paper targets supplied page citations over document images.Related work includes AIS, AutoAIS, ALCE, atomic-claim decomposition, citation-quality evaluators, and post-hoc text citation methods.Source: §2 · p. 2

Multi-page document QA

  • Multi-page QA benchmarks annotate evidence pages for answering, and image-based citation benchmarks evaluate or produce citations at generation time rather than verify supplied citations.The paper derives its testbeds from MP-DocVQA and DUDE.Source: §2 · p. 2

Verification and correction systems

  • Existing grounding detectors and correctors consume clean text spans, whereas this benchmark centers on verifying and correcting supplied page-level citations in multi-page document images.The paper adapts text-based systems as baselines and uses deterministic primary test labels rather than LLM judges.Source: §2 · p. 2

3 Testbeds and Ground Truth

DocCite evaluates whether a cited page supports an atomic claim and whether an unsupported citation can be localized or removed. Its testbeds pair deterministic page-localization labels with a separately reported human semantic audit.

Task

  • A citation instance is an atomic claim paired with a cited page; the primary decision is whether that page supports the claim.For correction, nonsupported claims are distinguished by whether support occurs elsewhere or nowhere, and supported-claim recall is also reported.Source: §3 · pp. 2–3

Two-layer ground truth

  • Primary labels use deterministic page-localization rather than LLM judgments, with a separate human calibration of semantic claim support.A document-wide scan expanded gold sets by 505 evidence pages across 2,058 QA pairs; the expansion was human-audited.Source: §3 · p. 3

DocCite-Nat: natural errors

  • The natural-error pool contains 2,468 candidate errors from six generator configurations; a two-annotator audit validates 1,909 errors, confirms 450 supported claims, and excludes 109 mixed items.Partial support counts as an error, and a seeded spot-check tests the candidates disputed by no verifier arm.Source: §3 · pp. 3–4

DocCite-Syn: validated injection

  • The injected test set retains 928 of 1,000 constructed instances after deterministic scans discard 32 accidentally supported cases and 40 with duplicate evidence.The scored set includes clean, wrong-page, partially supported, fabricated, and value-perturbed instances.Source: §3 · p. 4

4 The AtomCite Framework

AtomCite parses answers into claim–citation units, grounds each against the cited page through parallel OCR-text and image channels, and maps typed verdicts to conservative correction actions. Its design keeps nearby pages from determining citation verdicts and prevents uncertainty from silently deleting content.

Parse

  • AtomCite verifies each parsed claim whole as one atomic unit, with deterministic extraction of claims and citation markers into claim–page pairs.A finer decomposition variant is treated as an ablation rather than the default unit.Source: §4 · p. 4

Evidence

  • The evidence step supplies the cited page and a window of up to five pages, but only the cited page may determine the verdict.Window pages contribute correction proposals only, with the separation enforced by the type system.Source: §4 · p. 4

Ground and synthesize

  • Parallel text and image sub-agents check the cited page; failed checks can scan nearby pages only to propose a candidate supporting page.A hard channel conflict becomes inconsistent, while softer disagreements give the image channel authority on document cases.Source: §4 · p. 4

Correct

  • Typed correction keeps verified claims, repoints verified-nearby claims, removes false claims, and keeps inconsistent or abstaining claims with a flag.The reported precision change is paired with supported-claim retention to guard against deletion-heavy strategies.Source: §4 · p. 4

5 Experiments

The experiments compare image-aware verification with OCR, prompting, and text-baseline alternatives, then test natural-error auditing, correction, and transfer to public hallucination benchmarks.

5.1 Image-Aware vs. OCR-Only Verification

  • 2–6pp under MinerU and 5–9pp under corpus OCR separate AtomCite from OCR-only verification; the gap persists across model families and engines.The loss concentrates in supported-claim recall; Gemini’s image-based recall is 90.4%, versus at most 72.6% without the image.Source: Table 2 · p. 5, §5.1 · p. 5, §5.3 · p. 5

5.2 Architecture vs. Prompting

  • Every staged prompting arm trails AtomCite significantly; even the matched-compute sequential chain remains 2.5–6.2pp behind.Direct prompting’s high GPT and Claude supported-recall scores reflect lenient acceptance: it verifies more unsupported instances than AtomCite.Source: §5.2 · p. 5

5.4 Ablations

  • Decomposition lowers accuracy by 1.2–1.8pp, while replacing image-authoritative arbitration with a symmetric rule drops Gemini accuracy from 93.7% to 86.3%.Widening the correction window leaves verification accuracy unchanged but degrades repair precision; disabling the consistency check changes accuracy by only −0.1pp, not significantly.Source: §5.4 · p. 6

5.5 Text-Baseline Tier

  • Across families, image-aware verification beats text baselines; the strongest, same-backbone CiteEval-Auto, trails by 4.1–4.9pp.The same-family paired comparison favors AtomCite in every family, so the advantage does not reduce to backbone strength.Source: §5.5 · p. 6, Table 3 · p. 6

5.6 Natural Errors

  • On audited natural errors, AtomCite detects 97.3–98.6% at the lowest false-alarm rate, while flat OCR’s near-perfect detection comes with false alarms on nearly half of supported claims.The audit found 72.9% label noise among AtomCite-disputed instances versus 2.3% where only OCR or staged systems disagreed; raw automatic labels inverted the apparent system ranking.Source: §5.6 · p. 6, Table 4 · p. 6

5.7 Correction

  • 34.3% to 87–90%: the correction policy raises citation precision while retaining 91.8–92.5% of correct claims.On audited naturals, it acts on 97.2–98.6% of validated errors and retains 88.0–94.7% of confirmed-supported claims; repointing and removal address different failure types.Source: §5.7 · p. 7, Table 5 · p. 7

5.8 Transfer to Public Hallucination Benchmarks

  • With frozen prompts and no task-specific training, AtomCite improves on the direct judge in all ten evaluated cells across five hallucination benchmarks and two open 7–8B models.Results include 71.2 F1 on RAGTruth with Qwen and performance near reported bests on FaithBench and HalluMix.Source: §5.8 · p. 7, Table 6 · p. 7

6 Discussion, Limitations, and Ethics

Image-aware verification outperforms text-only conditions in the tested closed-world setting, but the authors limit the claim’s scope and identify deployment trade-offs.

6 Discussion, Limitations, and Ethics

  • Neither extra text passes nor sequential inference recovers the image-aware verifier’s advantage in the closed-world supplied-citation setting.The full verifier costs 2–3× layout OCR, creating a cost–accuracy choice for deployments.Source: §6 · p. 7
  • The evaluation covers English business and legal page images, not other scripts, born-digital PDFs, or non-page citation granularities.The authors also note page-localization labels are not entailment labels, gold-set expansion has finite recall, and T2/T3 baselines are adaptations.Source: §6 · p. 7
  • Because false removal can delete truthful content and wrong-page repair can give fabricated claims plausible support, high-stakes deployments should keep a human in the loop for removal.The policy uses typed, auditable actions and reports retention alongside precision.Source: §6 · p. 7

7 Conclusion

The paper presents AtomCite for verifying and repairing supplied page-level citations and DocCite for evaluating that task. Its conclusion highlights image-aware gains, correction, transfer, and the importance of human-audited labels.

7 Conclusion

  • Across three model families, page images beat OCR-only and staged conditions, while correction raises precision from 34.3% to 87–90% at approximately 92% retention.AtomCite also improves two open 7–8B backbones on five hallucination benchmarks; a two-annotator audit finds noisy automatic labels can reverse verifier rankings.Source: §7 · p. 7

A Testbed Construction Detail

The testbed combines controlled citation-error instances with naturally generated candidates, using document scans, frozen document pools, and a human audit to validate its construction. Its candidate-error prevalence varies by generator, with neither model tier uniformly cleaner.

A Testbed Construction Detail

  • Three disjoint, seeded document pools were fixed before scoring: injection, natural generation, and development.They contain 240, 150, and 30 documents, respectively, with no document shared across pools.Source: A Testbed Construction Detail · p. 10
  • Gold-set expansion added 505 evidence pages across 2,058 QA pairs after a document-wide fuzzy-match scan with frozen parameters.In the audited sample, 60.5% of added pages carried fully equivalent evidence under both annotators, illustrating the expansion's precision–recall trade-off.Source: A Testbed Construction Detail · p. 10
  • A random audit of 50 QA pairs found 89.5% annotator agreement on gold-set expansion decisions.The audited expansion sample contained 76 added pages.Source: A Testbed Construction Detail · p. 10
  • The 928 validated DocCite-Syn instances comprise 318 clean, 362 wrong-page, 113 partial, 82 fabricated, and 53 perturbed cases.Wrong-page shifts landing inside the expanded gold set are re-shifted; 251 of 362 wrong-page cases are within the ±2 correction window.Source: A Testbed Construction Detail · p. 10
  • Candidate-error prevalence spans 44.7–61.2% by generator, and neither the frontier nor efficiency tier is uniformly cleaner.GPT generators produce fewer answer-bearing cited claims than the other families.Source: A Testbed Construction Detail · p. 10
  • Table A1's pooled candidates were validated as genuine errors in 77.4% of cases, counting partial support as error.The two-annotator audit is distinct from the Layer-1 candidate-localization label.Source: Table A1 · p. 10

B Additional Results

Additional experiments test AtomCite across stronger OCR, natural-error labels, model tiers, correction settings, and external text data. They refine the trade-offs among verification accuracy, repair quality, localization, and cost.

Per-condition compute

  • The default claim-level (C) costs $0.024 per item versus $0.030 for (C-atom), while the decomposition variant costs 3.4× the layout-OCR condition.The image premium remains a deployment-dependent trade-off.Source: B Additional Results · p. 11, Table A2 · p. 11

Natural-error audit

  • Label noise affected 72.9% of (C)-disputed labels but 2.3% of OCR/chain-only disputed labels; audit-based orderings remain robust under a lenient reading.Scoring excluded verdicts as errors leaves 16 of 18 comparisons Holm-significant, with no ordering reversal; Gemini run-to-run accuracy varies by only 0.33pp.Source: B Additional Results · pp. 11–12

Correction on natural errors

  • On 450 human-confirmed supported claims, keep retention is 94.7% for Gemini and Claude and 88.0% for GPT; the policy acts on 97.2–98.6% of validated errors.Among fix-page proposals, 50.8/58.7/51.6% for Gemini/Claude/GPT fall inside the expanded gold set, a conservative lower bound.Source: B Additional Results · p. 12

OCR-engine robustness

  • +6.6 to +15.1pp supported recall from MinerU still leaves all nine paired gaps to (C) Holm-significant.Table A3 reports the MinerU replication; the stronger OCR engine improves every rung without closing the image-verification gap.Source: B Additional Results · pp. 12–13, Table A3 · p. 13

Correction-window sweep

  • Across ±1 to ±5 correction windows, binary accuracy stays at 91.9% while repair precision falls from 91.8% to 86.9%.The matched OCR control’s wrong-location recall is ≤1.4, whereas AtomCite configurations retain 50–58; localization is the capability the correction policy uses.Source: B Additional Results · p. 13, Table A4 · p. 13, Table A5 · p. 14

Why claim-level is the default

  • On multi-atom claims, strict conjunction makes (C-atom) trail claim-level (C) by 6.5–7.6pp, as atom-level grounding noise compounds.The decomposition variant supports fine-grained auditing, but claim-level verification is the accuracy-optimal default.Source: B Additional Results · p. 14

External text benchmark

  • On AttributionBench, the text-pair path reaches 70.2% accuracy under strict mapping, rising to 77.0% when partially verified claims count as attributable.The authors treat this single-configuration result as a sanity check, not evidence of generalization.Source: B Additional Results · p. 14

Model-tier 2×2

  • Under flagship claim-level (C), efficiency-tier models reach 93.4% (Gemini), 93.0% (Claude), and 93.5% (GPT), matching or exceeding frontier-tier results at roughly one-third the list price.Under (C), DUDE is harder than MP-DocVQA: 86.8–87.8% versus 94.7–95.4% binary accuracy, with family rankings unchanged.Source: B Additional Results · pp. 12, 14

C Transfer Suite: Protocols, Anchors, and Boundary

The transfer suite establishes baseline anchors and protocol constraints, while boundary tests place the framework’s domain at multi-claim long-form responses. The algorithm is described as a conservative, claim-level default that returns per-citation verdicts and a repaired citation layer.

Protocol details

  • Transfer runs use frozen prompts and greedy decoding, with nonzero temperature limited to bounded parse-retry rounds; all reported cells finish with zero unparsed outputs.Llama-3-8B reads long sources in 24k-character chunks because of its 8k context.Source: C Transfer Suite: Protocols, Anchors,… · p. 14

Fairness anchors

  • On RAGTruth, Qwen gains +21.7 F1 (95% CI [18.4, 25.0]); on TofuEval’s MediaSum main-topic slice, Qwen scores 66.5 against GPT-4’s 63.7.GPT-4 remains well ahead on marginal topics.Source: C Transfer Suite: Protocols, Anchors,… · p. 14
  • On VeriGray’s official sentence-level protocol, the direct arm scores 33.5 F1 against 35.9 for Lynx-8B and 35.1 for MiniCheck-7B.Both arms receive only the document and sentence, without surrounding context.Source: C Transfer Suite: Protocols, Anchors,… · p. 14

Boundary conditions

  • On VeriGray, the framework loses 4.7 F1 per sentence but gains 14.8 per response; boundary tests place its domain at multi-claim long-form responses.On pre-atomized LLM-AggreFact inputs, it scores −2.6 [−3.7, −1.5] against the direct arm; on TRIVIA+ and HaluBench pilots, F1 falls from 65.2 to 43.0 and accuracy from 75.3 to 57.6.Source: C Transfer Suite: Protocols, Anchors,… · p. 15

D Algorithm, Funnel, and Qualitative Cases

The qualitative cases contrast image-based rescue of OCR errors with a residual claim-level false alarm and a chain miss. The DocCite-Nat audit funnel reports no noise in the undisputed region’s Layer-1 labels.

Qualitative failure cases

  • The image rescues an OCR false alarm: a table cell supports the safflower claim despite lost column structure in OCR, while AtomCite verifies it.The passage identifies this as the dominant pattern behind flat OCR’s 44–49% false-alarm rate.Source: D Algorithm, Funnel, and Qualitative… · p. 15
  • GPT’s residual false-alarm class is document-scope claims anchored to one page, with a 12.3% rate versus 5.4–5.5% elsewhere.The cited first page supports the Youngstown State University claim through letterhead and context, but GPT judges it unsupported.Source: D Algorithm, Funnel, and Qualitative… · p. 15

DocCite-Nat construction funnel

  • The DocCite-Nat audit spot-check found 0% noise ([0, 7.3]) in the undisputed region, so its Layer-1 labels stand.Source: Figure A2 · p. 15

Qualitative failure cases

E Reproducibility and Artifacts

The paper pins model versions, decoding procedures, prompts, and randomization choices, and releases code, prompts, benchmark materials, and run ledgers. Preview-designated Gemini endpoints may be deprecated, but the call ledger records requests against the pinned identifiers.

Reproducibility

  • Experiments use pinned Gemini, Claude, and GPT model versions, with fixed seeds for randomized procedures and doc-cluster bootstrap comparisons using B=104.Prompts are frozen verbatim in the released repository.Source: E Reproducibility and Artifacts · p. 15

Artifacts

  • The release includes code, prompts, benchmark instances, labels, audit adjudications, annotation records, and per-run ledgers.Release terms follow the source-data licenses, including DUDE CC-BY and identifier-and-script release for MP-DocVQA.Source: E Reproducibility and Artifacts · p. 15

Endpoint traceability

Loading 2609.05802v1…