PaperlayerView exact revision on arXiv

Source-linked AI summary

CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

Dongsheng Ma, Jiayu Li, Zhengren Wang, Yijie Wang, Jiahao Kong, Weijun Zeng, Jutao Xiao, Jie Yang, Wentao Zhang, Bin Wang, Conghui He

arXiv:2605.12882v1cs.CLcs.CV

TL;DR

Existing Doc-VQA benchmarks score answers without verifying their supporting evidence, leaving faithfulness untested in high-stakes document intelligence. CiteVQA addresses this with joint answer-and-element-level citation evaluation, revealing that correct answers often cite the wrong regions and that Gemini-3.1-Pro-Preview reaches only 76.0 SAA.

  • Problem

    Existing Doc-VQA evaluations emphasize final-answer accuracy while leaving the precise supporting evidence and its faithfulness unverified.

  • Method

    CiteVQA benchmarks joint answer and element-level bounding-box citation accuracy using an automated annotation pipeline and Strict Attributed Accuracy.

  • Results

    Across 20 MLLMs, correct answers frequently pair with incorrect evidence, while Gemini-3.1-Pro-Preview’s SAA caps at 76.0.

  • Takeaways & Limitations

    CiteVQA exposes attribution hallucination and establishes a standard for developing more traceable and reliable multimodal document systems.

  • Takeaways & Limitations

    The automated curation pipeline requires substantial computational resources, creating a barrier to large-scale replication.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have significantly advanced document understanding, yet current Doc-VQA evaluations score only the final answer and leave the supporting evidence unchecked. This answer-only approach masks a critical failure mode: a model can land on the correct answer while grounding it in the wrong passage -- a critical risk in high-stakes domains like law, finance, and medicine, where every conclusion must be traceable to a specific source region. To address this, we introduce CiteVQA, a benchmark that requires models to return element-level bounding-box citations alongside each answer, evaluating both jointly. CiteVQA comprises 1,897 questions across 711 PDFs spanning seven domains and two languages, averaging 40.6 pages per document. To ensure fidelity and scalability, the ground-truth citations are generated by an automated pipeline-which identifies crucial evidence via masking ablation-and are subsequently validated through expert review. At the core of our evaluation is Strict Attributed Accuracy (SAA), which credits a prediction only when the answer and the cited region are both correct. Auditing 20 MLLMs reveals a pervasive Attribution Hallucination: models frequently produce the right answer while citing the wrong region. The strongest system (Gemini-3.1-Pro-Preview) achieves an SAA of only 76.0, and the strongest open-source MLLM reaches just 22.5. Ultimately, towards trustworthy document intelligence, CiteVQA exposes a reliability gap that answer-only evaluations overlook, providing the instrumentation needed to close it. Our repository is available at https://github.com/opendatalab/CiteVQA.

1 Introduction

CiteVQA addresses the limitations of answer-only Doc-VQA by jointly evaluating answers and element-level visual evidence across long-form, multi-domain, cross-lingual documents. Its audit of 20 MLLMs reveals Attribution Hallucination, with correct answers frequently paired with incorrect citations.

  • Motivation: Answer-only Doc-VQA evaluation can reward correct answers grounded in the wrong passage, creating risks in legal, financial, and medical applications.This failure can also arise when models guess using pretrained background knowledge rather than document evidence.
  • Benchmark: CiteVQA contains 1,897 questions from 711 PDFs spanning seven domains, designed for long-form, multi-domain, and cross-lingual evaluation.The benchmark balances document quantity and length to better simulate real-world complexity.
  • Dataset Construction: An automated annotation pipeline combines document parsing models and MLLMs to generate fine-grained, consistent citation data at scale.The resulting dataset comprises 1,897 complex queries across 711 multi-page, multi-domain PDFs and is expert-validated.
  • Evaluation: Strict Attributed Accuracy (SAA) rewards predictions only when both the textual answer and visual evidence attribution are correct.Recall measures evidence coverage, while Relevance assesses logical alignment.
  • Findings: 76.0 SAA is the ceiling for Gemini-3.1-Pro-Preview, while leading open-source MLLMs fail to surpass 25.0, exposing pervasive Attribution Hallucination.The audited models often provide correct textual answers while citing entirely wrong locations.

2 Related Work

Related work has progressed from single-page DocVQA toward multi-page document reasoning, but existing evaluations still leave evidence grounding insufficiently assessed. Parallel efforts in evidence-based reasoning and document intelligence emphasize source tracing and finer-grained retrieval while exposing unresolved challenges.

  • Document Visual Question Answering: Early DocVQA benchmarks emphasized single-page comprehension and exact textual answer matching, while newer benchmarks extended evaluation to multi-page and full-document contexts.Examples include DocVQA, InfoVQA, OCR-VQA, MP-DocVQA, MMLongBench-Doc, and SlideVQA.
  • Evidence-based Reasoning in LLMs: Evidence-based reasoning addresses persistent LLM hallucination risks by enforcing clinical-guideline alignment, explicit legal source tracing, and multimodal evidence grounding.The cited efforts include Med-R2 and GAPS in medicine, CitaLaw in law, and MRAMG-bench for multimodal reasoning.
  • Document Visual Question Answering: Precise multimodal reasoning over scattered evidence in full documents remains a critical and largely unexplored challenge.This unresolved challenge connects the limitations of multi-page document understanding with the need for more precise evidence handling.
  • Document Intelligence Systems: Document intelligence systems commonly use page-level retrieval with page-wise chunks and multimodal vector search, but this approach falters when queries require precise element-level grounding.Colpali, VisRAG, VDocRAG, and M3DocRAG exemplify this macroscopic retrieval paradigm.

3 CiteVQA: A Benchmark for Faithful Evidence Attribution

CiteVQA is built through an automated pipeline that links documents, extracts spatially grounded evidence, generates and verifies QA pairs, and identifies crucial evidence by ablation. The resulting benchmark contains 711 documents and 1,897 questions spanning diverse domains, document settings, reasoning types, and multimodal evidence.

  • Document Selection: 711 documents were selected across 7 domains and 30 sub-categories through a fully automated filtering pipeline designed for reproducibility and scalability.The filtering began with over 100 million raw PDFs and approximately 250k preselected candidates.
  • Automated Benchmark Construction: CiteVQA uses an end-to-end automated pipeline that links documents, extracts evidence chains in fine-grained spatial contexts, and generates template-driven QA pairs.The pipeline aggregates cross-document evidence, navigates parsed bounding-box spaces, and distills questions from open-source datasets into constrained templates.
  • Verification and Evidence Validity: The verification process retains QA pairs only when a model can answer from their evidence screenshots, removes common-knowledge questions, and labels masked elements as crucial evidence when masking prevents correct answers.Expert evaluation and auxiliary training validation further assess benchmark reliability.
  • Data Statistics: 1,897 questions span 711 documents averaging 40.6 pages across 7 macro-domains, with single-document, single-gold-document multi-document, and multiple-gold-document multi-document settings.The three settings account for 52.0%, 25.7%, and 22.3% of questions, respectively.
  • Data Statistics: 2.57 evidence elements are required per task on average, with nearly 30% being non-textual and reasoning types ranging from Complex Synthesis to Multimodal Parsing.The benchmark also analyzes evidence locality and cross-page spans.

4 Evaluation

CiteVQA evaluates document intelligence by jointly measuring answer correctness and evidence grounding, with Strict Attributed Accuracy requiring both. Across 20 MLLMs, answer-only performance substantially exceeds faithful attribution, especially for open-source models and multi-document scenarios.

  • Evaluation Metrics: Strict Attributed Accuracy (SAA) credits a sample only when Ans.≥4 and either Rel.≥4 or Rec.≥0.6.Recall uses IoU@0.5 against crucial evidence, Relevance uses an LLM judge on a 0–5 scale, and Answer Correctness uses semantic matching on a 0–5 scale.
  • Evaluation Setup: 20 state-of-the-art MLLMs were evaluated on CiteVQA using sequential page screenshots, adapted image resolutions, a unified prompt, and temperature 1.0.Models included both leading proprietary and open-source systems, with automated evaluation employing Qwen3-V.
  • Attribution Hallucination: 87.1 and 84.5 answer scores for GPT-5.4 and Gemini-3-Flash fall to 59.0 and 65.4 in SAA, demonstrating pervasive Attribution Hallucination.Low Recall scores indicate frequent failures to localize crucial evidence or identify the correct page, even at IoU ≥0.5.
  • Performance Disparity across Model Tiers: 76.0 Overall SAA makes Gemini-3.1-Pro-Preview the leading closed-source result, while GPT-5.4 leads semantic answer correctness at 87.1.Qwen3VL-235B is identified as the strongest open-source model, but its reported result is truncated in the supplied passage.
  • Impact of Document Scenarios: 68.9 versus 55.3 Recall shows Gemini-3.1-Pro’s decline from Single-Doc to Multi (N-Gold) scenarios.Answer accuracy remains relatively stable across scenarios, whereas attribution becomes markedly harder in multi-document settings; Multi (N-Gold) consistently has the lowest SAA scores.

5 Analysis & Discussion

The analysis finds substantial variation across question types and a positive relationship between evidence quality and answer accuracy. Ablations and a case study further suggest that narrowing evidence search can improve performance, while correct answers may still lack faithful attribution.

  • Question Type Results: 82.6 SAA for Gemini-3.1-Pro-Preview on Quantitative Reasoning reflects models’ relative strength on objective numerical computations.Multimodal Parsing remains a major bottleneck because it requires locating specific document elements from descriptive cues.
  • Synergy between Attribution and Accuracy: Answer Accuracy tends to increase with Evidence Quality after the 0–30-point Attribution Hallucination zone.The reported relationship is a fluctuating upward trend, suggesting precise evidence localization may support reasoning rather than merely justify answers post hoc.
  • Evidence Attribution as a Potential Performance Driver: +5.3% gains were observed when restricting context to GT-Pages in ablation studies.Providing a single Gold Document in multi-document settings produced larger improvements, including +13.4% for Qwen3-VL-8B.
  • Case Study: Qwen3-VL-235B-A22 answered correctly with Ans.=1 but received SAA=0 because its evidence crops were blank or incomplete.The case study illustrates how linguistic performance can diverge from attribution accuracy.

6 Conclusion · Appendix

CiteVQA advances trustworthy document intelligence by requiring element-level visual citations alongside answers. Its benchmark construction and model audit expose Attribution Hallucination, in which correct answers are frequently paired with incorrect evidence.

  • 6 Conclusion: CiteVQA requires models to provide element-level visual citations alongside answers.This design targets trustworthy document intelligence.
  • 6 Conclusion: 1,897 questions from 711 diverse PDFs form CiteVQA’s large-scale dataset.The dataset was constructed using an automated annotation pipeline.
  • 6 Conclusion: An automated annotation pipeline supports CiteVQA’s dataset construction.The pipeline was used to derive questions from diverse PDFs.
  • 6 Conclusion: A systematic audit of top-tier models reveals Attribution Hallucination.The audit examines whether model answers are paired with correct evidence.
  • 6 Conclusion: Correct answers are frequently paired with incorrect evidence.This finding identifies a critical reliability problem in document intelligence.
  • Appendix: The benchmark links answer correctness with evidence attribution rather than evaluating answers alone.Its required visual citations make evidence correctness part of the task.

A Data Compliance & Ethics Statement … B.3 Details of Expert Evaluation

CiteVQA’s compliance framework uses Common Crawl PDFs under its terms and the Robots Exclusion Protocol, distributes only public download links, and supports reproducibility through open metadata and bounding-box coordinates. Its pipeline links documents and evidence spatially, distills templates for QA synthesis, and audits 200 outputs with PhD-level experts using standardized prompts.

  • A Data Compliance & Ethics Statement: 707 PDFs come from Common Crawl and are acquired under its Terms of Use and the Robots Exclusion Protocol.The acquisition process is described as respecting the intentions of original content distributors.
  • A Data Compliance & Ethics Statement: CiteVQA distributes only public download links, following academic data-distribution practices exemplified by T5, MMC4, OBELICS, and CCpdf.The workflow is framed as consistent with multimodal and document-intelligence norms.
  • A Data Compliance & Ethics Statement: The project open-sources structured metadata and bounding-box coordinates while keeping PDF usage logic consistent with large-scale datasets such as LAION.This combination is intended to support academic reproducibility while maintaining compliance in the content-distribution chain.
  • A Data Compliance & Ethics Statement: All Common Crawl PDFs are attributed in repository metadata, and verified rights-holder concerns trigger prompt cooperation on content removal or updating.The statement places the legal rights of original copyright holders at the center of the protection process.
  • B.1 Details of Multi-Document Linking: Document profiles capture document type, core thesis, and section units before encoder-based vector mapping supports semantic retrieval across long documents.The profiles provide global context beyond simple text snippets and support anchor-document retrieval.
  • B Details of CiteVQA Pipeline: An LLM performs chain-of-thought cross-document matching over section units to construct structured association groups and precise segment-level evidence chains.The matching reasons through document hierarchies to identify logical bridges between anchor and candidate documents.
  • B.1 Details of Multi-Document Linking: Synthetic documents assemble matched pages from diverse sources, while an element-level bijective mapping returns synthesized evidence boxes to original PDF coordinates.This spatial linkage preserves traceability from synthesized layouts to source pages and supports citation fidelity.
  • B.2 Details of Template Distillation: The pipeline distills multi-domain open-source problems into templates, uses Gemini-3.1-Pro-Preview to extract four core template categories, and guides QA synthesis from evidence packages.Table 5 covers five key domains, while the extracted categories are presented in Table 6.

B.4 Details of Auxiliary Training Validation · C Details & More Results of Experiments

The auxiliary validation tests whether CiteVQA-generated supervision can substitute for human annotations in AgenticOCR-style training. Using rejection-sampled evidence-grounded trajectories and evaluation on two held-out benchmarks, the CiteVQA pipeline nearly matches human-annotated training performance overall.

  • B.4 Details of Auxiliary Training Validation: CiteVQA-generated 3k samples nearly reach the performance of 5k human-annotated samples in AgenticOCR SFT training.The alignment experiment uses the same ViDoRe V3 PDFs and compares CiteVQA supervision with original AgenticOCR annotations.
  • B.4 Details of Auxiliary Training Validation: On FinRAGBench-V, CiteVQA Pipeline (3k) achieves comparable or slightly better performance than Vidore Original (5k).The comparison uses the subset with bounding-box annotations and reports Pageacc, Recallmin, and RecallEM.
  • B.4 Details of Auxiliary Training Validation: On ViDoRe V3, Vidore Original shows a slight advantage over CiteVQA Pipeline (3k).This evaluation uses the held-out test set of ViDoRe V3.
  • B.4 Details of Auxiliary Training Validation: The distillation pipeline generates synthetic (I, Q, A, E) data, representing Image, Question, Answer, and Evidence Bbox, from ViDoRe V3 PDFs.The procedure follows AgenticOCR’s SFT training-data distillation pipeline.
  • B.4 Details of Auxiliary Training Validation: Rejection sampling uses Gemini-3-Pro-Preview trajectories, an image_zoom_and_ocr_tool, and an IoU threshold to produce 3k and 5k high-quality samples.The tool enables zooming into image regions and obtaining OCR results before trajectory filtering.
  • B.4 Details of Auxiliary Training Validation: Training masks user-prompt and tool-observation tokens, while assistant-generated reasoning and tool-call tokens contribute to the loss.The protocol follows AgenticOCR, with a learning rate of 1 × 10−5 for 6 epochs on 8 H200 GPUs.
  • B.4 Details of Auxiliary Training Validation: Evaluation covers approximately 200 FinRAGBench-V bounding-box samples and approximately 400 manually annotated, held-out ViDoRe V3 samples disjoint from training.Both test sets use the AgenticOCR evaluation metrics, with final results reported in Table 8.

C.1 Details of Experimental Setup … C.5 More Fine-grained Results

The appendix details standardized document processing, validates automated judging, expands attribution metrics, and analyzes failures across document scenarios and types. Results show that lower resolution, multi-document complexity, and heterogeneous layouts sharply undermine evidence localization.

  • C.1 Details of Experimental Setup: Gemini models used the native File API, while other models received 150 DPI screenshots with adaptive downscaling for standard-context limits.Long-context models retained original screenshots; standard-context models were scaled according to family-specific constraints.
  • C.1 Details of Experimental Setup: Halving total pixels from 1024^2 to approximately 724^2 reduced SAA from 22.5% to 11.8%.Answer accuracy declined moderately, whereas Recall and SAA exhibited a sharp, non-linear collapse, motivating 1024 × 1024 as the standard resolution.
  • C.2 Analysis of Different Judges: Human-expert validation used 200 randomly selected samples and Friedman tests to compare expert scores with Gemini-3-Flash-Preview and Qwen3-VL-235B-A22B judges.Table 11 reports p-values above 0.05 across metrics, indicating no statistically significant judge differences.
  • C.3 More Evaluation Metrics: Supplementary metrics provide coarse page-level Recall, bounding-box Precision, and F1 to assess coverage, spatial accuracy, and the recall–precision trade-off.F1 is defined as the harmonic mean of Recall and Precision and summarizes whether evidence extraction is thorough and concise.
  • C.4 More Results of Experiments: Page-level Recall remains low for most models, indicating that attribution failures often reflect inability to navigate to the correct page rather than only missing the exact box.The Gemini-3 series achieves above 87% Overall Page. according to the reported analysis.
  • C.4 More Results of Experiments: GPT-5.4’s Page-level Recall drops from 88.5% in Single-Doc to 75.4% in N-Gold, while F1 falls from 29.6% to 20.6%.The passage characterizes Multi-Doc settings as especially difficult, with many open-source models reaching a performance ceiling.
  • C.5 More Fine-grained Results: Models peak at 85.0 in Academic Tech, while Publishing & Media is hardest, with the highest SAA reaching only 63.3.The analysis attributes the Academic Tech advantage to standardized layouts and logical rigor, while complex typography challenges Publishing & Media.

D Case Study

The case studies show that correct answers can conceal incorrect evidence attribution, while stronger models jointly retrieve, reason over, and cite the relevant document regions. Gemini-3.1-Pro-Preview achieves SAA=1 on the contract example, whereas GPT-5.4 receives Rec.=0 and SAA=0 despite the correct answer; Gemini-2.5-Pro similarly succeeds on the NbO2 calculation while Qwen3-VL-8B fails both answer and citation.

  • Case Study 1: The contract permits five additional one-year periods, with Technical Lead hourly rates rising from $170 in Year 1 to $194 in Year 5.The supplied ground-truth answer lists $176, $182, and $188 for Years 2–4; the Programmer rates are truncated in the passage.
  • Case Study 1: Gemini-3.1-Pro-Preview cites the correct Contract Optional Years table, while GPT-5.4 cites an incorrect pricing table despite Ans.=5.GPT-5.4 confuses $145 with $170 for Year 1, resulting in Rec.=0 and SAA=0; Gemini-3.1-Pro-Preview achieves SAA=1.
  • Case Study 2: Gemini-2.5-Pro correctly computes the NbO2 electronic energy-gap widening as 0.40 − 0.14 = 0.26 eV and cites the corresponding evidence segments.Qwen3-VL-8B extracts 0.34 and 0.54 eV, provides irrelevant citations, and receives Ans.=1 and SAA=0.

E Prompt Templates

The prompt templates operationalize CiteVQA’s evidence pipeline, dataset-question templating, annotation review, and single- versus multi-document inference. They enforce structured, element-level evidence selection with complete context and standardized bounding-box output.

  • E.1 Prompts for CiteVQA Pipeline: The PDF evidence-extraction prompt asks models to parse outlines, OCR blocks, bounding boxes, and page screenshots into verifiable evidence bundles for Q&A, analysis, calculation, and visual extraction.Bundles must span at least 2 pages and include at least 2 element types.
  • E.1 Prompts for CiteVQA Pipeline: Evidence bundles must preserve complete table, figure, and cross-page context, including titles, captions, legends, axis labels, footnotes, and continuation structure.The prompt also requires extracting surrounding relevant elements and using screenshots and bounding boxes to confirm type and layout.
  • E.1 Prompts for CiteVQA Pipeline: The extraction output is a list of at least 10 evidence bundles, each containing a description and element records with OCR content, bounding boxes, angles, page identifiers, and element indices.Each bundle requires at least 5 relevant elements, while the prompt rejects single-element, fragmented, or overly broad summaries.
  • E.1 Prompts for CiteVQA Pipeline: Dataset-question templating returns one strict-JSON item per sample, with bilingual reusable templates, concrete examples, placeholders, and one of four specified categories.The categories are Complex Synthesis, Factual Retrieval, Multimodal Parsing, and Quantitative Reasoning.
  • E.1 Prompts for CiteVQA Pipeline: Annotation evaluation scores Question Difficulty, Answer Quality, and Crucial Evidence Quality on 0-to-5 scales using criteria ranging from unanswerable questions or absent evidence to complex reasoning and precisely supporting citations.Strong evidence should come from authoritative sections and include necessary span-level references.

F Limitations & Potential Negative Impacts

CiteVQA’s limitations include domain-specific ambiguity in defining authoritative evidence, even across seven domains. Optimizing for the benchmark’s metrics and document distributions may also reduce generalizability to unrepresented real-world document structures.

  • Limitations: Seven domains do not eliminate domain-specific nuances in defining authoritative evidence for highly specialized vertical fields.The paper identifies further exploration of these specialized-domain nuances as necessary.
  • Potential Negative Impacts: Overfitting to CiteVQA’s metrics and document distributions may reduce generalizability to diverse real-world document structures absent from the dataset.This risk could arise from excessive optimization for the benchmark’s specific tasks.
Loading 2605.12882v1…