Source-linked AI summary
Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation
Peiyang Liu, Ziqiang Cui, Xi Wang, Di Liang, Wei Ye
TL;DR
Existing iRAG systems rely on text-level citations that are coarse and lose visual layout information needed for reasoning over rich documents. CoE reasons over document screenshots to produce pixel-level evidence boxes, achieving 80.4% evidence localization accuracy on Wiki-CoE and outperforming text-based baselines on SlideVQA.
Problem
Existing iRAG attribution methods provide coarse textual evidence and lack visual grounding for documents whose layout and graphical elements support reasoning.
Method
CoE performs retriever-agnostic visual attribution directly on document screenshots, generating bounding boxes for supporting text, tables, and diagrams.
Results
80.4% Loc-Acc on Wiki-CoE, with CoE significantly outperforming text-based baselines on SlideVQA.
Takeaways & Limitations
Visual grounding supports reasoning over complex documents where layout information is essential and enables pixel-level source attribution for interpretable iRAG.
Takeaways & Limitations
CoE struggles to localize implicit reasoning in inference questions, despite identifying the correct evidence documents.
Abstract
from arXiv · showhide
Iterative Retrieval-Augmented Generation (iRAG) has emerged as a powerful paradigm for answering complex multi-hop questions by progressively retrieving and reasoning over external documents. However, current systems predominantly operate on parsed text, which creates two critical bottlenecks: (1) \textit{Coarse-grained attribution}, where users are burdened with manually locating evidence within lengthy documents based on vague text-level citations; and (2) \textit{Visual semantic loss}, where the conversion of visually rich documents (e.g., slides, PDFs with charts) into text discards spatial logic and layout cues essential for reasoning. To bridge this gap, we present \textbf{Chain of Evidence (CoE)}, a retriever-agnostic visual attribution framework that leverages Vision-Language Models to reason directly over screenshots of retrieved document candidates. CoE eliminates format-specific parsing and outputs precise bounding boxes, visualizing the complete reasoning chain within the retrieved candidate set. We evaluate CoE on two distinct benchmarks: \textbf{Wiki-CoE}, a large-scale dataset of structured web pages derived from 2WikiMultiHopQA, and \textbf{SlideVQA}, a challenging dataset of presentation slides featuring complex diagrams and free-form layouts. Experiments demonstrate that fine-tuned Qwen3-VL-8B-Instruct achieves robust performance, significantly outperforming text-based baselines in scenarios requiring visual layout understanding, while establishing a retriever-agnostic solution for pixel-level interpretable iRAG. Our code is available at https://github.com/PeiYangLiu/CoE.git.
1 Introduction
The introduction identifies three attribution challenges in iterative RAG—coarse text citations, visual information loss, and opaque cross-document reasoning—and proposes Chain of Evidence (CoE), a screenshot-based framework that localizes evidence with bounding boxes. CoE is evaluated on Wiki-CoE and SlideVQA, achieving 80.4% evidence localization accuracy on Wiki-CoE and outperforming text-based baselines on SlideVQA.
- Motivation: iRAG improves retrieval-augmented answering for complex queries, but generation and verification remain disconnected in high-stakes domains.Users need to verify why an answer was generated, despite iRAG’s success on textual benchmarks.
- Challenges: Existing text-level citations force users to scan hundreds of pages to find claim-supporting evidence.This verification bottleneck imposes high cognitive load and undermines attribution utility in multi-hop scenarios.
- Challenges: OCR and text parsing destroy visual semantics in PDFs, slides, and web reports containing charts, diagrams, and complex layouts.Linearization can remove trends, causal flows, and hierarchical relationships encoded visually.
- Challenges: Current methods lack a unified mechanism to visualize how intermediate evidence guides subsequent document selection across an iRAG reasoning chain.This opacity prevents users from understanding the complete chain of evidence rather than only the final supporting item.
- Chain of Evidence: CoE operates directly on retrieved document screenshots and generates precise bounding boxes for pixel-level evidence attribution.The visual-first framework bypasses brittle, format-specific parsing pipelines.
- Evaluation: 80.4% evidence localization accuracy is achieved on Wiki-CoE, while CoE significantly outperforms text-based baselines on SlideVQA.Wiki-CoE contains 70,418 questions with bounding-box annotations, and SlideVQA tests complex visual reasoning in presentation slides.
2 Related Work
Related work advances retrieval for complex reasoning through iterative, multi-turn RAG, while attribution research improves the evaluation and training of evidence-supported generation. However, existing attribution approaches generally provide text-level citations that require manual evidence location.
- Foundational RAG and dense retrieval augment generation with retrieved passages but often struggle with complex queries requiring multi-step reasoning.
- Iterative RAG addresses complex reasoning by performing multi-turn retrieval, with recent work optimizing retrieval frequency and strategy.Jeong et al. proposed dynamically controlling retrieval frequency, while Zhang et al. introduced a retrieval-optimization approach.
- Trustworthy AI research treats verifiability, data integrity, and security as critical, motivating frameworks that assess whether generated content is supported by external evidence.Rashkin et al. established the Attributable to Identified Sources framework, followed by attribution objectives in task-specific QA and pretraining.
- Existing attribution approaches typically output text-level citations, forcing users to manually locate supporting evidence.
3 Wiki-CoE Dataset
Wiki-CoE extends 2WikiMultiHopQA into a large-scale benchmark for pixel-level visual evidence localization by preserving rendered Wikipedia pages and mapping supporting facts to screenshot bounding boxes. Strict filtering yields 70,418 multihop questions with entity-chain-disjoint train/test splits and 60,518 evidence screenshots.
- Motivation: Wiki-CoE addresses the lack of visual grounding in existing multi-hop QA datasets by providing bounding boxes for evidence localization in rendered documents.2WikiMultiHopQA supplies sentence-level supporting facts, but these do not directly capture layout, formatting, or visual elements.
- Design principles: The dataset preserves Wikipedia layouts, retains examples with box-mappable evidence chains, and prioritizes high-impact entities for scalable coverage.Preserved content includes tables, infoboxes, and images; entities are ranked by question association frequency.
- Annotation pipeline: The annotation pipeline matches 2WikiMultiHopQA supporting sentences to rendered text elements, generates screenshot-coordinate boxes, and clips and validates them against the frame.Rendered elements include paragraphs, list items, table cells, captions, and infobox-adjacent text; invalid or out-of-bounds regions are discarded.
- Dataset statistics: 70,418 multihop questions remain after quality filtering from a 76,000-page screenshot pool, split into train (35,210) and test (35,208) by entity chain.The cleaned benchmark references 60,518 unique evidence screenshots, with no entity chain shared across the two splits.
4 Methodology
CoE formulates visual evidence attribution as multimodal reasoning over document screenshots, preserving layout and graphical elements lost in parsed text. Given retrieved candidates, it jointly produces an answer and an ordered chain of evidence with selected images, bounding boxes, and hop-level reasoning.
- Problem formulation: CoE represents each document as a screenshot image, preserving its native layout, formatting, and graphical elements.This visual representation replaces the parsed-text document used in traditional text-based iRAG.
- Problem formulation: Given a multi-hop query and a top-k candidate set, CoE maps candidate screenshots to both an answer and a chain of evidence.The learned function Φ operates over the query and retrieved candidates and outputs a ∈ A and e ∈ E.
- Evidence representation: Each evidence hop identifies a pivotal document and locates its supporting content through one or more rectangular bounding boxes.The boxes B_t delineate evidence regions within the selected document d* at hop t.
- Retriever interface: CoE is retriever-agnostic, accepting generic top-k candidates from lexical, dense, hybrid, or visual retrievers without retriever-specific parameters.Experiments construct candidates from gold evidence documents plus plausible distractors, including same-deck distractors for SlideVQA and shuffled top-5 order.
- Chain generation: In one autoregressive pass, CoE orders the reasoning chain logically and outputs each hop’s image identifier, bounding boxes, and short evidence-seeking sub-question.The logical output order is independent of the candidates’ presentation order, whose images are labeled img_0, img_1, and so forth.
5 Experiment Setup
CoE is evaluated across structured web pages and visually complex presentation slides to test both multi-hop reasoning and spatially grounded evidence attribution. The protocol uses top-5 retrieved candidates, multiple accuracy dimensions, retriever-agnostic baselines, and Qwen3-VL models trained with a two-phase curriculum.
- Datasets: The evaluation spans Wiki-CoE for large-scale multi-hop reasoning on structured HTML Wikipedia pages and SlideVQA for complex visual understanding in free-form slides.SlideVQA contains 2,619 slide decks and approximately 52k images, with multi-hop questions requiring synthesis across slides.
- Datasets: SlideVQA tests layouts where diagrams, arrows, charts, and spatial arrangements carry information that brittle text extraction may miss.This contrasts with standard Wikipedia layouts and directly targets CoE’s visual-attribution motivation.
- Evaluation Metrics: Answer quality is measured with exact match, while Loc-Acc jointly requires the correct candidate image and an overlapping evidence box in the top-5 setting.A box matches when IoU ≥0.3 or its center falls inside the ground-truth evidence region.
- Evaluation Metrics: Chain-Acc checks correct visual-document selection at every hop and whether the ordered document chain matches the gold reasoning path, with joint chain metrics adding box correctness.All localization and chain checks are defined under the top-5 candidate setting.
- Baselines and Training: All methods receive the same top-5 candidates, and CoE is compared with text-based iRAG, text-attribution, and zero-shot vision-language baselines.The primary backbone is Qwen3-VL-8B-Instruct, with Qwen3-VL-4B-Instruct for scale analysis; training uses single-hop localization followed by multi-hop chain generation.
6 Experimental Results
CoE consistently improves evidence-chain and pixel-level localization quality, especially on visually complex layouts, while exposing limitations in implicit inference and longer retrieval chains. Ablations further show that task-specific supervision, high-resolution visual input, and spatial augmentation are central to reliable grounding, with manageable computational overhead.
- Overall Benchmark Results: On Wiki-CoE, CoE-8B achieves 82.3% EM, 94.4% Chain-Acc, and 80.4% Loc-Acc, while CoE-4B also surpasses general VLMs in attribution quality.CoE-4B reaches 89.7% Chain-Acc and 71.1% Loc-Acc, outperforming GPT-5 by 21.6 and 39.4 points, respectively.
- Overall Benchmark Results: On SlideVQA, CoE-8B nearly matches GPT-5 and Qwen3-VL-235B in EM but improves over GPT-5 by 32.1 Chain-Acc points and 26.9 Loc-Acc points.CoE-8B achieves 58.8% EM, 87.5% Chain-Acc, and 61.0% Loc-Acc, compared with GPT-5’s 58.5% EM, 55.4% Chain-Acc, and 34.1% Loc-Acc.
- Question-Type Analysis: CoE performs best when evidence is encoded in structured layouts, with Bridge-comparison and Comparison questions reaching 89.5% and 86.8% EM, respectively.Compositional questions achieve 99.4% Chain-Acc and 82.5% Loc-Acc, whereas Inference questions reach 99.5% Chain-Acc but only 30.5% EM and 38.4% Loc-Acc.
- Question-Type Analysis: Across Wiki-CoE, 4-hop questions reduce Chain-Acc to 88.2% from 96.4% for 2-hop questions, while Loc-Acc remains nearly unchanged at 80.5% versus 80.3%.The results indicate that reasoning depth mainly affects ordered document selection rather than final box grounding.
- Layout-Specific Analysis: CoE’s advantage is largest on diagram-heavy slides, where it achieves 56.5% EM versus 28.0% for OCR, a 28.5-point gap.On text-dominant slides, the gap is smaller at approximately 6%, while CoE reaches 59.0% EM on data charts and infographics.
- Ablation Studies: Removing single-hop training reduces Loc-Acc from 80.4% to 73.1% on Wiki-CoE and from 61.0% to 53.2% on SlideVQA, while reducing resolution to 512 × 512 causes larger Wiki-CoE drops.The low-resolution setting lowers Wiki-CoE EM, Chain-Acc, and Loc-Acc by 10.1, 13.4, and 16.0 points, respectively; text-only input trails by 26.0 and 22.3 points on Wiki-CoE and SlideVQA.
7 Conclusion
CoE shifts iterative retrieval-augmented generation from brittle text parsing to visual reasoning grounded in document screenshots. It addresses semantic layout loss and opaque source attribution, with evaluation on Wiki-CoE and SlideVQA.
- Conclusion: CoE introduces visual reasoning for multi-hop inference directly grounded in document screenshots.The framework is presented as a paradigm shift from brittle text parsing.
- Conclusion: CoE addresses the loss of semantic layout information and the opacity of source attribution in existing systems.
- Conclusion: CoE is evaluated on the newly constructed Wiki-CoE benchmark and the complex SlideVQA dataset.