PaperlayerView exact revision on arXiv

Source-linked AI summary

CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

Dongsheng Ma, Jiayu Li, Zhengren Wang, Yijie Wang, Jiahao Kong, Weijun Zeng, Jutao Xiao, Jie Yang, Wentao Zhang, Bin Wang, Conghui He

arXiv:2605.12882v1cs.CLcs.CV

TL;DR

Doc-VQA evaluations focus on final answers without checking supporting evidence. CiteVQA requires element-level citations alongside answers and evaluates both jointly; across 20 mainstream MLLMs, Gemini-3.1-Pro-Preview’s SAA caps at 76.0 amid frequent incorrect citations.

  • Problem

    Doc-VQA evaluations focus on final-answer accuracy rather than precise evidence extraction, leaving the reliability of models’ document comprehension largely unverified.

    Source: §1 · p. 1

  • Method

    CiteVQA evaluates answers jointly with required element-level PDF bounding-box citations, using Strict Attributed Accuracy to measure citation fidelity.

    Source: §1 · pp. 2–3

  • Results

    76.0 is the SAA ceiling reported for Gemini-3.1-Pro-Preview; across 20 mainstream MLLMs, correct answers frequently accompany entirely wrong citations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CiteVQA establishes a rigorous standard for developing interpretable and reliable multimodal systems in high-stakes domains.

    Source: §6 · p. 10

    The MLLM-based automated curation pipeline imposes a significant computational-resource barrier to large-scale replication.

    Source: F Limitations & Potential Negative… · p. 32

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have significantly advanced document understanding, yet current Doc-VQA evaluations score only the final answer and leave the supporting evidence unchecked. This answer-only approach masks a critical failure mode: a model can land on the correct answer while grounding it in the wrong passage -- a critical risk in high-stakes domains like law, finance, and medicine, where every conclusion must be traceable to a specific source region. To address this, we introduce CiteVQA, a benchmark that requires models to return element-level bounding-box citations alongside each answer, evaluating both jointly. CiteVQA comprises 1,897 questions across 711 PDFs spanning seven domains and two languages, averaging 40.6 pages per document. To ensure fidelity and scalability, the ground-truth citations are generated by an automated pipeline-which identifies crucial evidence via masking ablation-and are subsequently validated through expert review. At the core of our evaluation is Strict Attributed Accuracy (SAA), which credits a prediction only when the answer and the cited region are both correct. Auditing 20 MLLMs reveals a pervasive Attribution Hallucination: models frequently produce the right answer while citing the wrong region. The strongest system (Gemini-3.1-Pro-Preview) achieves an SAA of only 76.0, and the strongest open-source MLLM reaches just 22.5. Ultimately, towards trustworthy document intelligence, CiteVQA exposes a reliability gap that answer-only evaluations overlook, providing the instrumentation needed to close it. Our repository is available at https://github.com/opendatalab/CiteVQA.

1 Introduction

CiteVQA evaluates document question answering by requiring both correct answers and element-level visual evidence citations. Audits reveal that models often answer correctly while citing the wrong region.

Introduction

  • Answer-only Doc-VQA evaluation overlooks evidence extraction, allowing models to guess from prior knowledge or give correct answers grounded in the wrong passage, risking untraceable errors in high-stakes domains.CiteVQA addresses this gap by evaluating answer accuracy and evidence faithfulness together.Source: §1 · p. 1
  • CiteVQA contains 1,897 questions from 711 PDFs across seven domains and requires element-level bounding-box citations so answers can be visually verified.It is designed for long-form, multi-domain, and cross-lingual scenarios, with document quantity and length intended to reflect real-world complexity.Source: §1 · p. 2
  • An automated annotation pipeline combines document-parsing models and MLLMs to scale fine-grained citation generation while addressing the cost and consistency limits of manual annotation.The resulting dataset is described as expert-validated.Source: §1 · pp. 2–3
  • SAA reaches only 76.0 for Gemini-3.1-Pro-Preview, while leading open-source MLLMs fail to surpass 25.0, exposing frequent correct answers grounded in wrong locations.The benchmark audits 20 mainstream MLLMs and identifies this behavior as Attribution Hallucination.Source: §1 · p. 2, Figure 1 · p. 2
  • Strict Attributed Accuracy (SAA) credits answers only when both the response and visual evidence are correct; Recall measures evidence coverage, while Relevance checks logical alignment.Together, these traceability metrics extend evaluation beyond final-answer accuracy.Source: §1 · p. 3

2 Related Work

Prior work advances document question answering, evidence-based reasoning, and document intelligence, but leaves precise visual evidence attribution insufficiently evaluated. CiteVQA addresses this gap with cross-page, element-level citations and joint answer-and-evidence evaluation.

Document Visual Question Answering

  • Earlier DocVQA benchmarks remain answer-centric or use page-level evidence, while bounding-box datasets lack consistent granularity and metrics; CiteVQA evaluates cross-page, multi-domain documents with element-level citations.Domain-specific chart tasks do not cover the diverse, layout-heavy challenges of real-world documents.Source: §2 · p. 3

Evidence-based Reasoning in LLMs

  • Evidence-based LLM work addresses hallucinations and source tracing in clinical, legal, and multimodal settings, but visually rich, long-form document grounding remains largely unexplored.The cited examples include clinical guideline alignment, legal-statute tracing, and interleaved image-text reasoning.Source: §2 · pp. 3–4

Document Intelligence Systems

  • Document systems evolved from page-level retrieval (Colpali [7], VisRAG [48], VDocRAG [36], M3DocRAG [5]) toward fine-grained retrieval and agentic navigation, but benchmarks still lack rigorous verification of reasoning paths and visual traceability.SimpleDoc [12] uses iterative summary-driven retrieval, while DocLens [57], DocDancer [52], and AgenticOCR [44] use tools to navigate from pages to localized elements.Source: §2 · p. 4

3 CiteVQA: A Benchmark for Faithful Evidence Attribution

CiteVQA is constructed through an automated pipeline that selects diverse documents, builds evidence-grounded question–answer pairs, verifies their quality, and analyzes the resulting benchmark’s scope and complexity.

3.1 Document Collection

  • From over 100 million PDFs, stratified sampling shortlisted approximately 250k candidates; two-stage MLLM classification yielded 711 documents across 7 domains and 30 sub-categories.The stages identify domain and language, then classify sub-categories within each domain.Source: §3.1 · p. 5

3.2 Question, Answer and Evidence Collection

  • The pipeline links documents by semantic alignment, extracts cross-page or cross-document evidence from parsed bounding boxes, then generates template-guided QA pairs from evidence packages.Authentic questions from open-source domain datasets inform the templates.Source: §3.2 · p. 5

3.3 Quality Control and Assessment

  • Candidate QA pairs are retained only if an MLLM answers from evidence screenshots; common-knowledge questions are removed, and masking each bounding box identifies crucial evidence when answers fail.Questions are paraphrased while preserving intent; expert evaluation and auxiliary training validation further assess benchmark quality.Source: §3.3 · p. 6

3.4 Dataset Overview and Analysis

  • 711 documents and 1,897 questions span 7 domains; questions average 2.57 evidence elements, nearly 30% non-textual, with evidence often crossing pages.Question scenarios include single-document, multi-document with one gold document, and multi-document with multiple gold documents.Source: Figure 4 · p. 6, §3.4 · p. 6

4 Evaluation

CiteVQA evaluates answer quality jointly with evidence grounding. Across tested MLLMs, strong answer scores often do not translate into comparably strong attributed accuracy.

4.1 Evaluation Metrics

  • SAA credits a sample only when Ans. ≥4 and either Rel. ≥4 or Rec. ≥0.6, requiring answer correctness plus high-quality grounding.Recall measures crucial-evidence localization at IoU@0.5; Rel. is an LLM-judged 0–5 support score, and Ans. is an LLM-judged 0–5 semantic-match score. Pagerecall, Precision, and F1 are also evaluated, with definitions and detailed results deferred to Appendix C.3 and C.4.Source: §4.1 · p. 7

4.3 Main Results

  • Overall SAA was 76.0 for Gemini-3.1-Pro-Preview versus 22.5 for strongest open-source Qwen3VL-235B; GPT-5.4 led Ans. at 87.1.Small-scale MLLMs often scored below 10.0 SAA.Source: Table 3 · p. 8, §4.3 · p. 9
  • GPT-5.4’s Ans. 87.1 but SAA 59.0, and Gemini-3-Flash’s 84.5 but 65.4, exemplify attribution hallucination and frequent localization failures even at IoU ≥0.5.Models often fail to localize crucial evidence or identify the correct page despite producing correct answers.Source: Table 3 · p. 8, §4.3 · pp. 8–9
  • Gemini-3.1-Pro’s Recall fell from 68.9 in Single-Doc tasks to 55.3 in Multi (N-Gold), which had the lowest SAA across scenarios.Answer accuracy remained relatively stable, while attribution became harder as document complexity increased.Source: Table 3 · p. 8, §4.3 · p. 9

4.2 Experimental Setup

  • The evaluation tested 20 proprietary and open-source MLLMs using sequential page screenshots, a unified prompt at temperature 1.0, and Qwen3-VL-235B-A22B as the primary automated judge.Input image resolution was adapted to each model’s context window capacity.Source: §4.2 · p. 8

5 Analysis & Discussion

CiteVQA analysis finds that question type, evidence quality, and search-space localization shape model performance. A case study shows that correct answers can still be grounded in blank or incomplete evidence crops.

5.1 Fine-grained Results

  • 82.6 SAA for Gemini-3.1-Pro-Preview on Quantitative Reasoning contrasts with Multimodal Parsing’s status as a major bottleneck.Multimodal Parsing requires locating document elements from descriptive cues and then parsing their content, challenging both evidence attribution and answer generation.Source: §5.1 · p. 9, Figure 5 · p. 9

5.2 Further Analysis of Evidence Attribution

  • After the 0–30-point Attribution Hallucination zone, Answer Accuracy tends to rise with Evidence Quality (max(Rel., Rec.)).The observed upward trend suggests precise evidence localization may support correct answering, rather than serving only as post-hoc justification.Source: §5.2 · p. 9, Figure 6 · p. 10
  • Restricting context to GT-Pages yielded gains up to +5.3%, while one Gold Document in multi-document settings produced a +13.4% gain for Qwen3-VL-8B.The ablations suggest that initial localization may be a CiteVQA bottleneck and that improved autonomous evidence-attribution recall could raise answering performance.Source: §5.2 · p. 10, Table 4 · p. 10

5.3 Case Study

  • Qwen3-VL-235B-A22B answered correctly (Ans.=1) but scored SAA=0 with blank or incomplete evidence crops, whereas Gemini-3.1-Pro-Preview scored SAA=1.Gemini’s first crop had a slight offset, but its second was nearly perfect; the example shows that a correct answer does not guarantee faithful attribution.Source: §5.3 · p. 10, Figure 7 · p. 10

6 Conclusion · Appendix

CiteVQA evaluates document intelligence by requiring element-level visual citations alongside answers, and its audit exposes Attribution Hallucination: correct answers paired with incorrect evidence.

6 Conclusion

  • CiteVQA requires models to provide element-level visual citations alongside answers to advance trustworthy document intelligence.The benchmark was built using an automated annotation pipeline and contains 1,897 questions from 711 diverse PDFs.Source: §6 · p. 10
  • The audit of top-tier models reveals Attribution Hallucination, in which correct answers are frequently paired with incorrect evidence.Source: §6 · p. 10
  • By exposing these hidden hallucinations, CiteVQA establishes a rigorous standard for developing interpretable and reliable multimodal systems in high-stakes domains.Source: §6 · p. 10

A Data Compliance & Ethics Statement … B.3 Details of Expert Evaluation

CiteVQA describes compliance and rights safeguards alongside methods for linking evidence across documents, distilling QA templates, and auditing automated outputs. Its pipeline is designed to support traceable citations, and experts sampled outputs for quality review.

A Data Compliance & Ethics Statement

  • The 707 benchmark PDFs come from Common Crawl under its Terms of Use and Robots Exclusion Protocol; distribution follows academic norms by sharing only public download links.The cited distribution paradigms include T5 [32], MMC4 [58], OBELICS, and CCpdf [40].Source: A Data Compliance & Ethics Statement · p. 15
  • The repository open-sources pipeline-generated metadata and bounding boxes, attributes PDFs in its metadata, and offers verified rights holders removal or updating of content.The authors say PDF usage logic follows large-scale open-source datasets such as LAION [33].Source: A Data Compliance & Ethics Statement · p. 15
  • CiteVQA aims to make evidence attribution transparent and auditable through an open-source, reproducible benchmark for studying attribution hallucination.The authors say the vision was inspired by OBELICS.Source: A Data Compliance & Ethics Statement · p. 15

B.1 Details of Multi-Document Linking

  • For multi-document linking, semantic profiles capture document type, thesis, and section units; normalized vectors retrieve the top 5 candidate documents by cosine similarity.This profile-based retrieval is intended to reduce semantic truncation from directly embedding long documents.Source: §B.1 · p. 16
  • An LLM matches anchor and candidate sections into at most 5 scored groups, each with 1–3 related segments, retaining high-scoring matches and returning none when no reliable match exists.The matching prompt reasons over document structure to identify logical bridges and outputs a rationale for each association.Source: §B.1 · p. 16
  • A bijective mapping links synthetic evidence boxes back to original PDF coordinates, preserving traceability between the assembled workspace and source pages.Matched pages are assembled into a synthetic document for QA pairs spanning at least two source documents.Source: §B.1 · p. 16

B.2 Details of Template Distillation

  • The template-distillation process draws on multi-domain open-source datasets and uses Gemini-3.1-Pro-Preview to extract four template categories for evidence-based QA synthesis.The templates guide the synthesis of logically rigorous QA pairs from specific evidence packages.Source: §B.2 · pp. 16–17

B.3 Details of Expert Evaluation

  • PhD-level experts audited 200 randomly sampled outputs for question difficulty, answer quality, and crucial evidence, with the paper reporting high-quality pipeline annotations.Experts used the models’ prompt templates; all human experts received task-based honoraria above the local minimum hourly wage.Source: §B.3 · p. 17, Table 7 · p. 17

B.4 Details of Auxiliary Training Validation · C Details & More Results of Experiments

The auxiliary training validation compares SFT using 3k CiteVQA-generated samples with 5k human-annotated samples and finds that CiteVQA nearly reaches the human-annotated data’s performance. The appendix also details the synthetic-data, sampling, training, and evaluation procedures.

B.4 Details of Auxiliary Training Validation

  • The comparison uses the same ViDoRe V3 PDFs to generate 3k CiteVQA samples and compares them with 5k human-annotated AgenticOCR [44] SFT samples.The validation is based on the ViDoRe V3 [19] corpus.Source: §B.4 · p. 17
  • On FinRAGBench-V, CiteVQA Pipeline (3k) performs comparably to or slightly better than Vidore Original (5k).The comparison uses the subset with bounding-box annotations.Source: Table 8 · p. 18
  • On held-out ViDoRe V3, Vidore Original shows a slight advantage over CiteVQA Pipeline.The paper’s overall characterization is that CiteVQA Pipeline nearly reaches the performance level of human-annotated data.Source: §B.4 · p. 17, Table 8 · p. 18
  • CiteVQA generates synthetic ViDoRe V3 data in the dataset format (I, Q, A, E), representing image, question, answer, and evidence bounding box, following AgenticOCR [44].Source: §B.4 · p. 18
  • Both data batches use an image_zoom_and_ocr_tool and Gemini-3-Pro-Preview trajectories filtered by an IoU threshold, yielding 3k and 5k high-quality samples.Source: §B.4 · p. 18
  • Training follows AgenticOCR’s protocol, masking user-prompt and tool-observation tokens from loss while training assistant-generated reasoning and tool-call tokens for 6 epochs at a 1 × 10−5 learning rate on 8 H200 GPUs.Source: §B.4 · p. 18
  • Evaluation uses approximately 200 bounding-box-annotated FinRAGBench-V [55] samples and approximately 400 manually annotated, held-out ViDoRe V3 samples disjoint from training.The evaluation metrics match those used in AgenticOCR, with results reported in Table 8.Source: §B.4 · p. 18

C.1 Details of Experimental Setup … C.5 More Fine-grained Results

Across the supplemental analyses, models struggle to locate and attribute evidence reliably, especially across multiple documents; performance also varies with resolution and document type. The appendix details the evaluation setup, judge validation, and additional localization metrics.

C.1 Details of Experimental Setup

  • Gemini models use the native File API; other models receive 150 DPI screenshots, kept at original resolution for long-context models and adaptively downscaled for standard-context models.Experiments use a unified prompt, a 4,096-token output cap, and 8× NVIDIA H200 GPUs for open-source model inference.Source: §C.1 · pp. 19–20
  • Halving pixels from 1024^2 to approximately 724^2 reduces Qwen3-VL-235B-A22B SAA from 22.5% to 11.8%, motivating 1024 × 1024 as the standard resolution.The passage reports that answer accuracy falls more moderately, while attribution measures—particularly recall and SAA—collapse sharply; higher resolution is constrained by token limits and position embeddings.Source: Table 10 · p. 19, §C.1 · p. 19

C.2 Analysis of Different Judges

  • Across 200 samples, Friedman-test p-values for Rel. and Ans. range from 0.14 to 0.53, showing no statistically significant difference between automated judges and human experts.The study compares human scores with those from Gemini-3-Flash-Preview and Qwen3-VL-235B-A22B.Source: Table 11 · p. 20, §C.2 · p. 20

C.3 More Evaluation Metrics

  • Precision penalizes redundant or irrelevant boxes against the full evidence set, while F1 is the harmonic mean of precision and recall for thorough, concise attribution.The evidence set includes crucial and auxiliary evidence.Source: §C.3 · pp. 20–21

C.4 More Results of Experiments

  • Gemini-3 models exceed 87% Overall Page. Recall, compared with 69.3% for GPT-5.2 and 57.8% for Qwen3-VL-235B-A22B.The low page-level recall of many models indicates that attribution failures often begin before fine-grained box localization: models fail to find the relevant page.Source: §C.4 · p. 21, Table 12 · p. 22
  • GPT-5.4 Page-level Recall falls from 88.5% in Single-Doc to 75.4% in N-Gold, while its F1-score drops from 29.6% to 20.6%.Qwen3-VL-235B-A22B Page-level Recall also drops from 64.4% to 50.5% in the multi-document setting.Source: §C.4 · p. 21, Table 12 · p. 22

C.5 More Fine-grained Results

  • Academic Tech reaches 85.0 SAA with Gemini-3.1-Pro-Preview, while Publishing & Media is hardest, with the highest SAA at 63.3.The passage attributes the domain gap to standardized academic layouts versus complex typography, nonlinear content, and image-text interleaving in publishing.Source: §C.5 · p. 21, Figure 8 · p. 22

D Case Study

The case studies show that correct answers can still be grounded in the wrong region, while weaker models may also fail to retrieve evidence and reason through multi-step attribution.

D Case Study

  • Gemini-3.1-Pro-Preview answers 5 and cites the correct table (SAA=1), while GPT-5.4 cites an incorrect pricing table despite the correct answer (Rec.=0; SAA=0).The cited table lists $145 vs. $170 for Year 1.Source: Figure 9 · p. 23
  • Gemini-2.5-Pro computes the widening gap as 0.26 eV and cites the evidence (SAA=1), while Qwen3-VL-8B extracts 0.34 and 0.54 eV, cites irrelevant segments, and scores Ans.=1 and SAA=0.The case demonstrates difficulty with retrieval and the logic required for multi-step attribution, even when evidence is explicitly stated.Source: Figure 10 · p. 24

E Prompt Templates

The prompt templates define how to assemble verifiable evidence bundles, cite document regions, generate question templates, and evaluate answer quality and citation relevance. They also specify a language-independent 0–5 answer-quality rubric and a concise two-line response format.

E.1 Prompts for CiteVQA Pipeline

  • Evidence bundles must span at least two pages and two element types, preserving complete table, figure, and cross-page context.The extraction prompt also directs the model to use OCR, page screenshots, and bounding boxes to identify and verify evidence.Source: §E.1 · p. 25
  • The extraction workflow searches for evidence cues, gathers surrounding elements, links related metrics across pages, and checks layout against screenshots and bounding boxes.It cautions against isolated elements, fragmented tables or figures, page-number dependencies, and summaries without a clear Q&A target.Source: §E.1 · p. 25
  • The pipeline requests at least 10 evidence bundles, each with a description and at least five relevant elements carrying type, content, bounding-box, page, and element-index fields.For tables and figures, the prompt requires associated titles, captions, legends, axis labels, and footnotes where applicable.Source: §E.1 · p. 25
  • A separate prompt converts each supplied question sample into a reusable English and Chinese template, a concrete example, and one of four question categories.Templates use placeholders such as [Entity], [Date], and [Metric], and the output must preserve each sample’s semantic consistency and ID.Source: §E.1 · p. 26
  • The rubric rates question difficulty, answer quality, and crucial evidence quality on 0–5 scales; difficulty ranges from simple fact retrieval to complex multi-step reasoning and cross-document comparison, while good-to-excellent answers are accurate and properly cited.Source: §E.1 · pp. 26–27

E.2 Prompts for CiteVQA Evaluation

  • Single-document inference requires citations to complete elements, separate bounding boxes for captions or footnotes, and multiple tags when an inference uses multiple evidence pieces.Coordinates use page-image-relative values from 0 to 1000, and page numbering starts at 1 while ignoring original page numbers.Source: §E.2 · pp. 28–29
  • Citation-relevance evaluation assesses truthfulness, sufficiency, localization accuracy, and alignment; a score of 5 requires a bounding box exactly covering cited text, with no extra content, skewing, or misalignment, and a perfect evidential match.Source: §E.2 · p. 30
  • Answer-correctness evaluation judges the generated response solely against the standard answer, without adding external knowledge.The evaluator returns reasoning and a score.Source: §E.2 · p. 31
  • Scores 0–2 range from off-topic or contradictory answers (0), through minimally relevant answers (1), to partial answers with missing key information or obvious factual errors (2).Evaluators are told to be strict and choose the lower score when uncertain.Source: §E.2 · p. 31
  • A score of 3 means the answer covers core facts but has gaps, lacks necessary details, or contains minor errors.The prompt says to reserve 3 for answers that are genuinely useful despite clear gaps.Source: §E.2 · p. 31
  • A score of 4 requires coverage of all key points with rigorous logic and near-complete accuracy; evaluators should reserve it for strong answers.Source: §E.2 · p. 31
  • A score of 5 requires a complete, accurate, well-structured answer that is not significantly more verbose than the standard answer, and is described as extremely difficult to earn.The prompt says not to award 5 unless the answer is perfect in every dimension.Source: §E.2 · p. 31
  • Evaluators must ignore citation-related phrasing, judge content without penalizing Chinese, English, or mixed-language answers, and award credit only for standard-answer facts, not extra details.Source: §E.2 · p. 31
  • The required response has two lines: a reason for the score followed by the score, with no additional content; the example wraps the score in a <qa_acc> tag.Source: §E.2 · p. 31

F Limitations & Potential Negative Impacts

CiteVQA prioritizes traceability and evaluation depth, but its scope, computational demands, and task-specific focus create limitations and potential negative impacts.

Limitations

  • CiteVQA may not capture specialized definitions of authoritative evidence, and its MLLM-based curation and fine-grained evaluation require substantial computational resources.The evaluation combines coordinate verification with fine-grained textual alignment, incurring more overhead than standard VQA tasks.Source: F Limitations & Potential Negative… · p. 32

Potential Negative Impacts

  • Over-optimizing for CiteVQA’s metrics and document distributions may reduce generalizability to real-world document structures absent from the dataset.The paper identifies this as a potential negative impact of the benchmark.Source: F Limitations & Potential Negative… · p. 32

Related papers

Loading 2605.12882v1…