PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels

Zhuchenyang Liu, Yao Zhang, Yu Xiao

arXiv:2607.24651v1cs.CVcs.CLcs.IR

TL;DR

Visual document attribution often relies on coordinates, but models may cite wrong regions despite correct answers. This paper tests verbatim evidence quotes with retrieval and trains quote generation, substantially improving attribution while preserving answer quality.

  • Problem

    Visual document attribution commonly uses coordinate outputs, yet models can answer correctly while citing wrong regions, leaving whether this reflects capability limits or the interface unresolved.

  • Method

    The paper compares coordinate and verbatim-quote interfaces across six open VLMs, then trains an 8B backbone with region-label-free GRPO using retrieved-region judge rewards.

  • Results

    Evidence recall rises from at most 8.1 under coordinates to 25.9–46.9 under language, while hallucination falls from 82–97% to 39–65%, with little change in answer quality.

  • Takeaways & Limitations

    Verbatim quotes resolved by retrieval recover attribution across six open models, and GRPO further improves it without coordinates or costly region-level supervision.

  • Takeaways & Limitations

    The language interface depends on a layout parser, so its attribution ceiling is bounded by parser quality.

Abstract

from arXiv · show

Reliable visual document understanding requires a model to attribute each answer to the evidence regions that support it. Recent benchmarks and systems express this step through a coordinate interface: the model outputs the coordinates of bounding boxes that mark the evidence regions in the document. Under this interface, vision-language models often fail to identify the right regions even when the answer is correct, a failure known as Attribution Hallucination. We present a study that investigates whether this failure is partially limited by what the model can express through coordinates. On a verified bilingual CiteVQA subset, we compare the coordinate interface with a language interface in which the model outputs only text, quoting its evidence verbatim, and a multimodal retriever returns the location of each quote as a page region proposed by a layout parser (tables and figures are quoted through their captions or notes); the comparison is repeated over six open vision-language models. Compared with the coordinate interface, evidence recall rises from at most 8 points to between 26 and 47 and the hallucination rate roughly halves, with little change in answer quality. Building on this comparison, we use the same quote-and-retrieve pipeline as a training scaffold: because region-level evidence labels are expensive to collect for long documents, we introduce a GRPO recipe whose reward is a judge's reading of the gold answer and crops of the retrieved regions, training the model to quote better evidence without any region labels and raising an 8B backbone's strict attributed accuracy from 22.4 to 33.8. These findings indicate a practical path to improve attribution"without a coordinate interface and without costly region-level supervision.

1 Introduction

The introduction argues that attribution failures in visual document understanding may partly reflect the coordinate interface rather than absent grounding ability. A controlled quote-and-retrieve comparison improves evidence recall and reduces hallucination, while a region-label-free GRPO scaffold further improves attribution.

  • Motivation: Evidence attribution is necessary because answers in finance, law, and medicine must be verifiable, motivating benchmarks that jointly score answers and cited document elements.CiteVQA treats a paragraph, table, or figure on a page as an evidence element.
  • Problem: Under coordinate-based attribution, models generate numeric bounding-box tokens and often answer correctly while citing the wrong region, a failure termed Attribution Hallucination.Open models rarely exceed single-digit evidence recall under this interface.
  • Interface comparison: Evidence recall rises from at most 8.1 under coordinates to 25.9–46.9 under language evidence across six open models, with little change in answer quality.The comparison holds backbones, pages, questions, and scoring fixed while changing only evidence expression; ablations attribute recovery to location information expressed in generated language.
  • Interface comparison: Hallucination rate falls from 82–97 percent under coordinates to 39–65 percent under the language interface.The language condition quotes evidence text, while a parser and multimodal retriever resolve quotes to page regions.
  • Training scaffold: A region-label-free GRPO scaffold trains quoting with judged rewards from the gold answer and crops of retrieved regions, using data already present in QA sets.The recipe uses group-relative advantages and DAPO’s asymmetric clipping for stability.

2 Related Work

Related work spans text-based attribution, extractive document-answer localization, coordinate-based evidence localization in visual documents, and reinforcement learning for visual grounding. This paper builds on quote-based attribution and label-free grounding while contrasting with coordinate-only evidence systems.

  • Evidence attribution in text-based QA: Text-QA attribution uses verbatim quotes, formal citation evaluation, post-hoc retrieval, and automatically supervised fine-grained span citations.GopherCite, ALCE, RARR, and LongCite represent these approaches, respectively.
  • Answer localization in extractive Doc-VQA: Extractive Doc-VQA localizes answer spans as explanatory boxes through detected-region identifiers or OCR alignment.DLaVA tags detected text regions, while ARIAL maps generated answers back to OCR segments; the paper’s setting instead involves mostly abstractive synthesis over long documents.
  • Localizing evidence in visual documents: Visual-document evidence systems such as VISA, LAT, and Chain of Evidence express supporting evidence directly as coordinates.Their methods use box supervision, stepwise box prediction with reinforcement learning, or pixel-level boxes along reasoning chains.
  • Reinforcement learning for visual grounding: Visual-grounding reinforcement learning covers detection, referring, grounded reasoning, annotation-free closed loops, and multi-page documents.The paper’s reward instead evaluates natural-language quotes after retrieval, requires neither kind of label, and gates on answer correctness.

3 Method

The method replaces direct coordinate prediction with verbatim evidence quotes that are retrieved to document regions, while preserving element-level citation evaluation. It also enables region-label-free attribution training through a judge-based GRPO reward over answers and retrieved evidence crops.

  • Problem statement: The task outputs an answer and page-region citations, with element matches defined by IoU ≥ 0.5 and separate page- and element-level localization tasks.Ground truth identifies annotated evidence elements such as paragraphs, tables, and figures.
  • Language interface: The language interface generates a JSON answer with verbatim quotes, including sentence, table, caption, and note spans for textual and non-textual evidence.Unlike coordinate prediction, the model produces evidence in its native textual modality.
  • Quote-to-region retrieval: A layout parser converts pages into typed blocks, whose image crops are embedded multimodally; quotes are matched across the document by one-to-one Hungarian assignment while ignoring predicted page numbers.This retrieval design addresses non-textual evidence and prevents unreliable page predictions from making retrieval unrecoverable.
  • Attribution training: Training requires no region-level labels: GRPO optimizes a reward built from the question, gold answer, model answer, and retrieved citation crops.A vision-language judge scores answer correctness, evidence relevance, and evidence coverage on a 0–5 scale.
  • Attribution training: The multiplicative reward makes evidence scores matter only for correct answers, while summing relevance and coverage preserves within-group reward variance.Within-group normalization, asymmetric clipping, a KL penalty, and dynamic sampling complete the GRPO optimization procedure.

4 Experimental Setup

The experiments use a verified single-document CiteVQA subset, evaluate six instruction-tuned VLMs under a standardized full-document protocol, and report retrieval, judge-based quality, attribution, and citation-format metrics. Training uses LoRA on Qwen3-VL-8B with long-document questions and a separate reward judge.

  • Benchmark: 719 questions (72.9%) from 440 PDFs form the verified evaluation set, with a median document length of 34 pages and a maximum of 182.Questions were retained only when source PDFs and annotated evidence were verifiably available.
  • VLMs: Six instruction-tuned VLMs span four families, dense and MoE architectures, thinking and non-thinking variants, and sizes from 8B to 31B.The models are Qwen3-VL-8B, Qwen3-VL-30B-A3B, Qwen3.5-9B, Qwen3.5-27B, Gemma-3-12B, and Gemma-4-31B.
  • Implementation: All models process full documents with adaptive per-page resolution capped at one megapixel, using greedy decoding and 4096 or 1536 output tokens for thinking or non-thinking models, respectively.The setup matches CiteVQA’s protocol.
  • Metrics: Recall and precision measure evidence retrieval, while relevance, answer quality, Strict Attribution Accuracy, Attribution Hallucination, multi-threshold recall, and citation format rate cover quality and attribution.Recall uses necessary elements at IoU ≥0.5 on the same page; precision matches all annotated evidence, and judge-based scores use Gemini-3.5-Flash scaled by ×20 where applicable.
  • Training: Training applies LoRA to Qwen3-VL-8B’s language model with the vision tower frozen, using 1,584 LongDocURL questions and Qwen3.5-9B as the reward judge.The training set includes 83 held-out validation questions with no overlap with evaluation PDFs.

5 Results

The language interface substantially improves evidence attribution over coordinates while largely preserving answer quality, and its gains remain robust across overlap thresholds and parser choices. GRPO training further improves attribution metrics, though partly by producing more citations at lower precision.

  • Main interface comparison: 25.9–46.9% recall and 38.6–65.4% AH under language improve on coordinate-interface recall of at most 8.1% and AH of 82.1–97.0%.Answer scores reach 91.8 under coordinates, while the GRPO-trained 8B model achieves 51.3% recall and 33.8% SAA.
  • Main interface comparison: Answer scores stay within 2.3 points across interfaces for five of six backbones, while Qwen3-VL-8B drops 6.2 points under language.All recall, SAA, and AH gains are statistically significant.
  • Threshold robustness: 39.2% recall at IoU 0.7 under language versus 0.0% under coordinates for Qwen3-VL-8B demonstrates the language interface’s nearly flat threshold profile.Coordinate recall falls from 30.7% page-level to 5.8% at IoU 0.1, whereas language recall ranges from 42.3% at IoU 0.1 to 39.2% at IoU 0.7; the parser ceiling is 88.2% at IoU 0.5 and replicates with an independent parser.
  • Component ablations: 5.0–23.5% recall after snapping coordinate boxes remains below the quote pipeline, showing that matching retrieval granularity alone does not explain attribution recovery.Question-only retrieval trails the quote pipeline by 13–19 points for three models, while crop-embedding retrieval exceeds lexical and BM25 resolvers by 5–14 points.
  • GRPO training: 39.9 →51.3 box recall and 22.4 →33.8 SAA after GRPO training accompany AH falling from 41.2 to 28.4.Same-page recall rises from 63.2 to 73.8, but citations increase from 2.1 to 4.6 while precision declines from 37.6 to 26.5 and box F1 from 32.2 →28.8.

6 Discussion

The discussion argues that attribution failures primarily reflect limitations of the coordinate format rather than absent location ability, while showing that coordinate attribution remains difficult even as models scale. It also shows that models often express evidence verbally, and that quote-and-retrieve errors and extra citations require evaluation beyond annotation overlap.

  • Interface limitations: Retrieval-resolved quotes raise evidence recall by roughly an order of magnitude, indicating that the deficit lies in coordinate formatting rather than location ability.Threshold decomposition shows that coordinates preserve page identity but lose region geometry.
  • Interface limitations: 82.1% of correct answers are still mis-attributed by the best coordinate answerer, and scaling alone does not fix coordinate attribution.This pattern persists across four model families and a four-fold size range.
  • Format compliance: 33.1% of Qwen3-VL-30B-A3B responses contain parseable coordinate boxes, while non-citing outputs usually answer correctly and describe locations verbally.These descriptions include page numbers, section titles, and table names, revealing a mismatch between requested coordinate output and the model’s evidence expression.
  • Format compliance: 49.2% of Qwen3.5-9B responses contain parseable language citations, showing that format adherence varies by model rather than being inherent to either interface.The language interface therefore has its own compliance cost.
  • Retrieval failures: Three failure points affect language-interface evidence retrieval: parser coverage, omitted model quotes, and one-to-one assignment collisions.The parser ceiling is softer than IoU 0.5 suggests because many unreachable elements are granularity mismatches.
  • Evaluation beyond annotations: 27.4 is the trained model’s strict SAA score, up from 19.1 under a judge requiring Ans ≥4 ∧Rel ≥4, suggesting extra citations often support the answer.The improvement is statistically significant (McNemar p = 6 × 10−7).

7 Conclusion

Verbatim evidence quotes resolved by retrieval recover attribution across six open vision-language models from four families, while complementary GRPO-based training improves it further. Together, these findings indicate a practical path to improve attribution without coordinates or costly region-level supervision.

  • Conclusion: Quote-and-retrieve evidence recovers attribution across six open vision-language models from four families.Evidence is expressed as verbatim quotes and resolved by retrieval.
  • Conclusion: Complementary GRPO-based training improves attribution further without costly region-level supervision.The conclusion presents this training approach as complementary to quote-based retrieval.
  • Conclusion: The findings indicate a practical path to improve visual-document attribution without a coordinates interface.This addresses a failure previously read as a capability gap in open vision-language models.

Limitations

The study is limited to single-document attribution and does not evaluate heavily scanned collections. Its quote-based method relies on external parsing and retrieval for localization, while experiments use greedy single runs and a single RL seed.

  • Research scope: The study scopes its claims to single-document evidence attribution and does not study the retrieval dimension introduced by multi-document settings.
  • Methodology: The method does not teach models to localize independently: models write quotes, while a parser and retriever convert quotes into page regions.Retrieved regions can provide training labels, and the judge-based reward can apply to models that output regions.
  • Experiments: Evaluation is bounded by source-PDF availability, with retained documents skewed toward usable text layers and performance on heavily scanned collections untested.A verification filter also shrinks the benchmark.
  • Experiments: All results use single runs with greedy decoding, while the reinforcement-learning result uses a single seed on one backbone.Paired uncertainty estimates are reported in Appendix D.

A Implementation Details

The implementation uses adaptive full-document rendering, cached block-level retrieval, and a verified benchmark protocol. Region-label-free GRPO trains evidence quoting on windowed long-document QA data, with coverage and citation behavior driving observed reward gains.

  • Adaptive resolution: Every model receives all document pages at min(106, B/n) pixels, with B = 1.87 × 108; the one-megapixel cap applies in practice across the verified set.The longest verified document has 182 pages, and the setup matches CiteVQA’s standard adaptive scaling.
  • Retrieval implementation: Retrieval embeds MinerU semantic-block crops and quoted evidence with Qwen3-VL-Embedding-2B, using cached embeddings, cosine similarity, and Hungarian one-to-one assignment.Block crops are rendered at 150 dpi; the passage specifies a median of 388 blocks per document.
  • Training data construction: 1,584 training questions over 349 documents are constructed from LongDocURL using windows of up to 30 pages around annotated evidence, while annotations are excluded from reward computation.The evidence-page annotations place the rollout window but do not supervise the reward.
  • Training dynamics: Coverage rises from 1.4 to 2.5 during training, relevance gains about one point, and the model cites roughly one additional quote per response.Answer correctness changes least, while most recall gains come from evidence that was previously uncited.
  • Benchmark construction: The verified evaluation set requires resolvable PDFs, existing annotated evidence pages, and textual evidence verifiably present under each ground-truth box.These checks inspect only ground-truth annotations and never model outputs.

B Benchmark Filtering Funnel

Verification narrows the CiteVQA validation release from 987 questions to 719 valid questions by removing unresolved, mismatched, or unverifiable items. The retained benchmark includes document-format checks, while per-condition sample sizes vary because unparseable judge outputs are excluded.

  • Verification filtering: 719 valid questions remain from 987 after verification removes unresolved PDFs, out-of-file evidence pages, unfindable evidence text, and byte-unverifiable document versions.The removals total 117, 4, 131, and 16 questions, respectively.
  • Document-format assessment: 0.78 is the average substantive text-layer share across retained documents, measured to assess whether verification favors born-digital files.Substantive text means more than 200 extractable characters per page.
  • Condition-level evaluation: 681–719 is the per-condition sample-size range because responses lacking parseable score or relevance tags are excluded from the corresponding condition.Across 9,347 calls, 13 answer responses and 240 relevance responses lack parseable tags.

C Per-Language and Per-Type Results · D Statistical Tests · E Cross-Parser Check

The interface effect and training gain hold across languages and evidence types, while figures remain hardest and coordinate recall stays near zero for most backbones. Paired statistical tests and an independent Docling reparse support the robustness of the reported gains and threshold profile.

  • C Per-Language and Per-Type Results: Chinese questions score at least as well as English under the language interface, and the interface effect and training gain hold in both languages.These are reported as per-language results in Table 6.
  • C Per-Language and Per-Type Results: Tables are recovered nearly as well as running text, whereas figures remain the hardest evidence category under the language interface.Figures use caption-based anchoring; the evidence-type counts are 685 text, 271 table, and 71 figure elements.
  • C Per-Language and Per-Type Results: Coordinate-interface recall stays near zero for most backbones across evidence categories.This pattern is reported in the element-level comparison of text, table, and figure evidence.
  • D Statistical Tests: Differences in recall and hallucination rate are tested with paired bootstraps, while SAA uses an exact McNemar test on discordant pairs.The analysis uses 10,000 bootstrap resamples with a fixed seed and recomputes hallucination on questions answered correctly under both conditions.
  • D Statistical Tests: 27.4 versus 19.1: GRPO improves stricter SAA over its base model, with McNemar p = 6.0 × 10−7.The comparison has discordant pairs b = 99 and c = 40.
  • D Statistical Tests: 42.0 versus 50.0: the corresponding hallucination rate falls under the GRPO comparison.This decline accompanies the stricter SAA improvement over the base model.
  • E Cross-Parser Check: Docling re-parsing yields a ceiling of 100 and recall 3 to 6 points below the main results, consistent with finer segmentation.The cross-parser check re-parses all 440 documents with an independently developed parser and repeats judge-free geometric evaluation.
  • E Cross-Parser Check: 44.9 against 35.9: the training gain persists at IoU 0.5, while recall remains nearly flat from IoU 0.1 to 0.7 and backbone ordering is preserved.The unchanged profile indicates that the flat threshold pattern is not an artifact of one parser’s segmentation conventions.

F Credit-Assignment Controls · G Qualitative Examples

The controls show that coordinate snapping removes threshold decay but remains limited by same-page recall, while quote and answer readings provide complementary evidence signals. Qualitative examples illustrate coordinate misplacement, verbal location descriptions, and verbatim quotes resolved to annotated blocks.

  • F Credit-Assignment Controls: Snapping removes threshold decay, confirming raw coordinate collapse is a granularity phenomenon, but recall remains bounded by same-page recall and below the language interface.The control snaps each predicted coordinate box to the maximally overlapping semantic block, with a nearest-block fallback when needed.
  • F Credit-Assignment Controls: 18.9, 33.1, and 39.9 are recall at IoU 0.5 for question-only retrieval with k=1, k=3, and k=5 blocks, respectively.The corresponding same-page recalls are 34.9, 54.1, and 65.9; these fixed-budget figures are not directly comparable to language runs averaging 1.1–3.3 citations.
  • F Credit-Assignment Controls: Docling blocks reproduce the language-interface recall profile and training gain under an independent parser.The replication uses identical model quotes.
  • F Credit-Assignment Controls: Quote and q+a readings succeed on different questions, with per-question recalls correlating only 0.39 to 0.54 and differing on 30 percent of question–backbone pairs.Table 11’s oracle union measures the better reading per question to diagnose signal disjointness rather than define a deployable system.
  • F Credit-Assignment Controls: The resolver ablation compares token-F1 lexical matching and Okapi BM25 with the production multimodal resolver under identical quotes and one-to-one whole-document assignment.Exact matching requires the normalized quote to be a substring of one block; fuzzy matching requires token-F1 of at least 0.8.
  • G Qualitative Examples: Figure 6 shows a coordinate answer reaching the correct page but placing its box away from the evidence.The output is a real model result rather than a constructed illustration.
  • G Qualitative Examples: Figure 6 also shows a coordinate prompt answered with a verbal location description instead of a box, alongside verbatim evidence quoted and retrieved to the annotated block.The three panels correspond to the paper’s distinguished behaviors.

H Citation Precision and Residual Errors · I Selective Prediction on Citation Similarity · J Prompts

The appendix defines citation scoring and residual-error categories, shows that GRPO mainly improves evidence retrieval, and demonstrates similarity-based abstention for selective citation. It also specifies the coordinate and language interfaces and the strict judges used for answer accuracy, evidence relevance, and training rewards.

  • H Citation Precision and Residual Errors: Citations match annotated elements when they share a page and have IoU ≥ 0.5, with recall scored on necessary evidence and precision on all annotated evidence.The protocol also evaluates citations on the page index named by the model without correction.
  • H Citation Precision and Residual Errors: 26.5 precision and 28.8 F1 result from the GRPO model’s 4.6 cited regions per response, versus 37.6 precision and 32.2 F1 from its base model citing 2.1 regions.Precision makes recall interpretable because citing every parsed block would approach ceiling recall while falling below one percent precision.
  • H Citation Precision and Residual Errors: 13.9 percent of base-model elements are parser-unreachable, 48.5 percent are not retrieved, and 16.4 percent are rank-limited.Because 95.6 percent of parser-unreachable elements are contained within some block, many failures reflect granularity mismatch rather than missing document content.
  • H Citation Precision and Residual Errors: 10.4 of the GRPO model’s 11.9-point recovered-element gain comes from shrinking the not-retrieved bucket, while rank-limited errors barely change.The parser-unreachable share is constant by construction, and coverage scores and citations per response rise most during training.
  • I Selective Prediction on Citation Similarity: Citation hit rates increase from roughly 10 percent in the lowest similarity quintile to 16–47 percent in the highest across backbones.Keeping only the top fifth by similarity raises Qwen3.5-27B citation precision from 30.6 to 43.6 percent.
  • J.1 Coordinate-interface system prompt: The coordinate prompt requires complete element-level boxes, uses 1-based page numbers, and expresses image-relative coordinates on a 0–1000 scale.Tables and images with captions or footnotes require separate evidence boxes.
  • J.2 Language-interface system prompt / J.5 Training reward judge prompt: The language prompt requires verbatim element-level quotes and returns each quote with a 0-based page index in a JSON evidence list.The training reward judge separately scores answer correctness, evidence relevance, and evidence coverage on integer scales from 0 to 5 using the gold answer and retrieved crops.
Loading 2607.24651v1…