PaperlayerView exact revision on arXiv

Source-linked AI summary

Navigating Sparse Evidence: Agentic Visual RAG via Explicit Context Selection and Consolidation

Yucheng Shen, Lingyong Yan, Jiulong Wu, Shuaiqiang Wang, Jianmin WU, Dawei Yin, Min Cao

arXiv:2609.15800v1cs.AI

TL;DR

Page-level VRAG must locate evidence that may be sparse or spread across pages, while agentic methods can leave supporting evidence implicit in exploration trajectories or textual memories. SCORE retains relevant observations and source pointers, then reloads and organizes original images for answering; it achieves state-of-the-art accuracy across three benchmarks at both 3B and 7B.

  • Problem

    Page-level VRAG must find evidence that may be localized within one page or scattered across pages, while existing agentic methods do not explicitly organize original visual evidence for final answering.

    Source: Introduction · p. 1

  • Method

    SCORE records relevant observations and source pointers in a ledger, reloads and organizes the referenced original images, and trains evidence organization with filtered cold-start distillation and evidence-aware reinforcement learning.

    Source: Introduction · p. 2, Training Pipeline · p. 4

  • Results

    SCORE achieves state-of-the-art accuracy across three VRAG benchmarks at both backbone sizes; on SlideVQA with Qwen2.5-VL-7B, accuracy is 77.16% versus 72.37% for the strongest prior baseline.

    Source: Introduction · p. 2, Main Results · p. 5

  • Takeaways & Limitations

    SCORE makes evidence use explicit and controllable by linking reasoning steps to their supporting images.

    Source: Conclusion · p. 7

    A case study shows that correct retrieval and evidence presence do not prevent chart-layout errors when binding the right brand logo to a year.

    Source: Case Study · p. 15

Abstract

from arXiv · show

Visual Retrieval-Augmented Generation (VRAG) empowers models to navigate and answer queries about visually rich documents by retrieving relevant page images as visual evidence and reasoning over their content. However, effectively utilizing this visual evidence is usually impeded by two main challenges. First, answer-relevant evidence is sparse and may be concentrated in a small region of one page or dispersed across multiple pages. Second, existing agentic methods often generate answers based on raw exploration trajectories or compressed textual memories rather than an explicitly organized set of supporting images, making answers susceptible to exploration noise and obscuring the evidence-backed reasoning trace. We argue that the bottleneck lies not only in evidence discovery but also in its preservation and organization before answer generation. We propose SCoRE (Selection and Consolidation for Robust Evidence), a unified agent loop for explicit evidence selection and consolidation. During exploration, SCoRE retains only query-relevant observations and their source pointers in a maintained textual ledger, preserving earlier evidence while keeping the visual context bounded. At termination, it reloads the referenced original images and consolidates the visual evidence for answering, arranging it into a logical sequence. This decouples final reasoning from exploratory trial-and-error while ensuring strict visual grounding via indexed claim-to-image linkages. To enable end-to-end optimization of this unified rollout, our training paradigm combines filtered cold-start trajectory distillation with evidence-aware reinforcement learning, whose reward promotes evidence coverage, consolidation compactness, and answer correctness.

Introduction

VRAG must locate evidence that may be localized or scattered across pages, while observations can remain ordered by exploration rather than answer logic. SCORE separates discovery from consolidation, retaining relevant observations and source pointers before reloading and organizing original images for grounded answering.

Visual Retrieval-Augmented Generation

  • OCR-based pipelines can lose layout, chart, and figure information, while VLMs enable methods to use raw visual inputs.The passage cites Zhang et al. 2025 for the limitations of OCR-based methods.Source: Introduction · p. 1

Introduction

  • Page-level retrieval makes evidence utilization difficult because relevant material may occupy a small region of one page or be scattered across pages.The agent must determine which pages and regions to inspect.Source: Introduction · p. 1

Agentic VRAG with Reinforcement Learning

  • Existing agentic methods improve search or maintain textual memories, but do not reconstruct and globally organize selected original images into a claim-linked visual chain for answering.The proposed distinction is that exploration discovers potential evidence while consolidation selects, restores, and organizes it.Source: Introduction · p. 2

Introduction

  • SCORE retains query-relevant observations with source pointers in a textual ledger, then reloads, filters, and orders the original images for answer generation in the same rollout.Indexed entries link claims to supporting evidence, separating exploratory trial and error from the final evidence chain.Source: Introduction · p. 2
  • Filtered teacher trajectories and an evidence-aware reinforcement-learning reward train evidence organization alongside answer correctness.Cold-start trajectories require full gold-page coverage and correct answers; the reward jointly promotes gold-page coverage, chain compactness, and correctness.Source: Introduction · p. 2
  • SCORE achieves state-of-the-art accuracy across three VRAG benchmarks with both 3B and 7B backbones, while ablations support complementary roles for selection and consolidation.Further analyses report robust retrieval and favorable efficiency.Source: Introduction · p. 2

Visual Retrieval-Augmented Generation

  • OCR-free VRAG methods use page images, but top-k retrieval can add irrelevant pages and cannot adapt retrieval based on prior observations.ColPali aligns text queries with image patches, while EVisRAG supplies retrieved page images to a VLM.Source: Visual Retrieval-Augmented Generation · p. 2

Agentic VRAG with Reinforcement Learning

  • Prior agentic VRAG work improves exploration, memory, or attribution, whereas SCORE makes evidence organization explicit within a unified agent loop.The passage contrasts this with multi-agent pipelines that can be difficult to optimize end-to-end and incur orchestration costs.Source: Agentic VRAG with Reinforcement Learning · p. 2

Method

SCORE retrieves and inspects page images from a flat corpus, retaining query-relevant observations with source pointers before reloading and organizing the original images for answering. Its training combines filtered teacher trajectories with reinforcement learning that rewards evidence coverage, compactness, and answer correctness.

Task Formulation

  • The task retrieves and aggregates visual evidence from a flat pool of page images, where relevant information may span pages and unrelated documents provide distractors.The corpus mixes pages from heterogeneous documents without document-level scoping.Source: Task Formulation · p. 3

SCORE Framework

  • At each turn, SCORE retains relevant observations and source pointers in a ledger while limiting raw visual history to a recent window, then consolidates reloaded images for answering.The agent can search for a page, inspect a bounding-box crop, or consolidate; the latter ends exploration and triggers answer generation.Source: SCORE Framework · pp. 3–4

Evidence Ledger: Selection and Consolidation

  • During consolidation, SCORE jointly reviews multiple reloaded images, keeps useful ledger entries, orders them logically, and links each answer claim to supporting visual evidence.With one image, it uses the terminal ledger directly without further selection or reordering.Source: Evidence Ledger: Selection and… · p. 4

Training Pipeline

  • Training filters teacher trajectories for correct answers and full gold-page coverage, then uses GRPO with a trajectory reward combining evidence coverage, ledger compactness, and answer correctness.The reward gates compactness and answer credit on full gold-page coverage; incomplete coverage instead incurs a penalty.Source: Training Pipeline · p. 4

Experiments

SCORE achieves the best accuracy across three VRAG benchmarks and both tested backbone sizes, with particularly strong gains on multi-hop and spatially complex evidence. Ablations and further analyses show how selection, consolidation, training, and backbone capacity shape its accuracy and efficiency.

Experimental Settings

  • Experiments use ViDoSeek, SlideVQA, and MMLongBench with all pages in a shared image pool, and evaluate answer accuracy using a binary LLM judge.Training uses 2,500 SlideVQA trajectories for cold-start and 1,600 queries for RL; baselines include vanilla and fine-tuned methods, with reproduced methods sharing the ColQwen2.5-v0.1 retrieval backbone.Source: Experimental Settings · p. 5

Main Results

  • On SlideVQA, ViDoSeek, and MMLongBench, SCORE raises 7B accuracy from 72.37% to 77.16%, 74.87% to 75.31%, and 28.45% to 31.52%, respectively.The same overall trend holds with the 3B backbone.Source: Main Results · p. 5
  • SCORE’s largest gains occur on multi-hop and complex-layout evidence: SlideVQA multi-hop accuracy is 62.26% vs. VISOR’s 53.62%, and 7B MMLongBench results lead on Text, Chart, Figure, and Layout.Selective ledger updates retain pages while bbox zoom-ins select regions, addressing uneven evidence distribution.Source: Main Results · pp. 5–6

Ablation Study

  • Both organization modules help, but their relative importance flips after fine-tuning: dropping consolidation costs −9.9 vs. −1.7 on Vanilla SlideVQA, while dropping relevance judgment costs −4.9 vs. −3.0 after fine-tuning.Removing both modules is consistently worst; training shifts denoising from consolidation toward relevance judgment.Source: Ablation Study · p. 6, Table 2 · p. 6

More Analysis

  • SCORE resolves a coverage–noise tension: it retains 0.16 non-gold pages and reaches 77.16 accuracy despite a marginal ledger-coverage drop, while high completeness alone need not ensure better performance.ViDoRAG’s 91.3 completeness comes with 10 retrieved images; VISOR retains 1.29 non-gold pages per query and may incur transcription errors.Source: More Analysis · p. 6, Table 3 · p. 6
  • SCORE averages 3.65 turns versus ViDoRAG’s 6.22 and VRAG-RL’s 4.36, while its two variants lie on SlideVQA’s accuracy–latency Pareto front.SCORE-direct reaches accuracy 70 at 4.97s; SCORE reaches 72 with average increases of 1.45s latency and 706 tokens, and uses 6.7× fewer tokens than ViDoRAG while scoring 13 points higher.Source: More Analysis · pp. 6–7, Figure 3 · p. 6, Table 4 · p. 7
  • Cold-start SFT supplies the dominant SlideVQA accuracy gain, from 55.17 to 75.62, while RL adds a smaller increase to 77.16.Ledger coverage rises from 64.0 to 76.6 at SFT and then to 78.4 with RL; multi-hop accuracy rises 25.2 points, from 36.3 to 61.6, at SFT.Source: More Analysis · p. 7, Table 5 · p. 7
  • In prompting-only tests across five backbones, SCORE improves every backbone, with gains peaking at 7B (+18.19) and narrowing on stronger models.At 3B, full SCORE scores 28.58 versus 42.26 for a ledger-only variant; training raises 7B SlideVQA accuracy to 77.16.Source: More Analysis · p. 7, Table 6 · p. 7

Conclusion

SCORE makes evidence use explicit and controllable by filtering observations during retrieval and reorganizing retained evidence before answering. Filtered cold-start distillation and an evidence-selection RL reward further teach evidence organization across benchmarks and backbones.

Conclusion

  • A single retrieve–reason loop filters observations at collection time, then re-selects and reorders retained evidence before answer generation.Each reasoning step links to its supporting image.Source: Conclusion · p. 7
  • The design controls answer-context noise through explicit evidence organization.Source: Conclusion · p. 7
  • Filtered cold-start distillation and an evidence-selection RL reward jointly teach evidence organization across benchmarks and backbones.Source: Conclusion · p. 7

Details of Search Engine

The search engine retrieves unseen pages in ranked order using cached page representations and the agent’s textual query.

Details of Search Engine

  • Page images are encoded offline into patch-level multivector representations and cached, so inference embeds only the query.Source: Details of Search Engine · p. 9
  • The query uses the same encoder and is scored against cached pages with the late-interaction MaxSim operator.Source: Details of Search Engine · p. 9
  • The environment returns the highest-ranked unseen page and signals when no top-k candidates remain unseen.This prevents repeated inspection of the same evidence and directs the agent toward its final answer when retrieval is exhausted.Source: Details of Search Engine · p. 9

Necessity of Crop-and-Zoom Tool.

Crop-and-zoom restores fine detail that becomes illegible when full pages are downsampled, while the agent-loop context design preserves relevant evidence and keeps visual context bounded. Action-specific prompts keep each step aligned with the query and its sub-goal.

Necessity of Crop-and-Zoom Tool.

  • A fixed max_pixels cap makes dense local content illegible, while crop-and-zoom rereads only the needed sub-region at high fidelity without increasing the global token budget.It trades one extra turn for the resolution the question requires.Source: Necessity of Crop-and-Zoom Tool · p. 9
  • The bbox maps displayed-image coordinates back to the original page, adds a fixed 28-pixel margin, clamps to page bounds, then crops and rescales the region.Malformed or degenerate coordinates prompt the model to reissue the action.Source: Necessity of Crop-and-Zoom Tool · p. 9
  • At full-page resolution the agent misread fine print as 2798, but a bbox zoom made the line legible and recovered the correct answer, 2796.The case attributes the error to perception, not reasoning; the evidence was already on the retrieved page.Source: Necessity of Crop-and-Zoom Tool · p. 9, Figure 4 · p. 9

Agent Loop Context

  • Multi-image trajectories can exhaust context and let early irrelevant images crowd out the query, so context is reconstructed each turn rather than accumulated as a transcript.The paired mechanisms are a persistent textual evidence ledger and a sliding window of raw visual turns.Source: Agent Loop Context · p. 9
  • The ledger stays visible while the raw-image window evicts older turns, bounding image context; W=2 preserves a retrieve-then-zoom pair.Query-relevant content from evicted images remains distilled in the ledger.Source: Agent Loop Context · pp. 9–10
  • Every returned observation restates the query and collected evidence, with action-specific prompts guiding relevance judgments, crop reading, consolidation, or answering.Consolidation prompts direct joint re-examination, noise filtering, and evidence ordering; answer prompts require responding from the consolidated evidence.Source: Agent Loop Context · p. 10

Evaluation Details

The evaluation uses a binary LLM judge for short factual answers and checks its consistency against an alternative judge. It covers three document benchmarks with distinct evidence regimes and question types.

Evaluation Details

  • Qwen-max-latest assigns binary semantic-correctness labels from the question, reference answer, and prediction; DeepSeek-V3.2 yields highly consistent SlideVQA scores, with only slight subtask-level differences.The authors report that results are not sensitive to judge choice.Source: Evaluation Details · p. 10, Table 7 · p. 10

Dataset Information

  • SlideVQA’s complete test split has 2,215 questions: 1,648 single-hop questions answerable from one slide and 567 multi-hop questions requiring evidence across slides.The questions come from presentation slides spanning real-world decks and topics.Source: Dataset Information · p. 10
  • ViDoSeek has 1,142 test questions, all answerable from one page, split into 645 extraction and 497 logic questions over a corpus of roughly 6,000 page images.Extraction locates and reads information; logic also requires inference or computation.Source: Dataset Information · p. 10
  • MMLongBench contributes 847 questions with verifiable reference answers, tagged across text, table, chart, figure, and layout content types.The content-type subsets overlap, so their counts exceed the evaluation-question total.Source: Dataset Information · p. 10

Baseline Implementation Details

The baseline implementations combine results quoted from original papers with reproduced methods evaluated under a shared retriever. Adaptations preserve each method’s main design while fitting document-page retrieval.

Baseline Implementation Details

  • Vanilla RAG and ReAct are single-pass and iterative retrieval baselines, respectively; Search-R1-VL adapts multi-turn RL-based reasoning to visual retrieval, while VRAG-RL adds crop-and-zoom with trajectory-level rewards.Search-R1-VL uses the same data, reward, and post-processing as VRAG-RL and starts from a cold-start checkpoint.Source: Baseline Implementation Details · p. 12
  • VISOR maintains a textual ledger of query-relevant observations, M3RAG uses specialized agents, and EVisRAG structures per-page evidence for multi-image understanding.VISOR and M3RAG results are quoted from their papers; EVisRAG uses its released 7B weights and retrieves the top three pages with ColQwen2.5v0.1.Source: Baseline Implementation Details · p. 12
  • MMSearch-R1’s text search is adapted to retrieve document-page images, while its image search redirects to text search; R1-Router retrieves top-five pages and is capped at three turns.ViDoRAG uses a planning, retrieval, and answering architecture, with top-10-page retrieval and Qwen2.5-VL-7B; these four baselines are reproduced using a shared retriever.Source: Baseline Implementation Details · p. 12
  • Experiments run on 8× NVIDIA A800 80GB GPUs; quoted baselines follow their original papers’ evaluation protocol, while M3RAG’s retrieval configuration differs from ours.The paper flags that difference alongside M3RAG’s result.Source: Baseline Implementation Details · pp. 10, 12

Training Hyperparameters

Cold-start SFT and RL use one hyperparameter configuration across the 7B and 3B backbones and across benchmarks. The authors suggest scaling learning rate and context length when adapting SCORE to substantially different VLM sizes.

Training Hyperparameters

Details of Cold-Start

Cold-start training uses filtered teacher rollouts to teach SCORE’s evidence-selection and consolidation behavior; a post-consolidation re-fetch module was dropped because it added little coverage and slightly reduced accuracy on current benchmarks.

Details of Cold-Start

  • Qwen3.5-122B-A10B generates SlideVQA teacher rollouts with search, bbox zoom-in, and consolidation actions, under fixed image-size and interaction-step budgets.If the teacher does not consolidate within the step budget, the environment forces consolidation over collected evidence.Source: Details of Cold-Start · p. 13
  • A rollout is retained only when an LLM judge accepts the answer and every annotated gold page appears in the ledger as relevant evidence.Samples without reference-page metadata are exempt from the coverage test; coverage is checked against kept ledger pages, not every page inspected.Source: Details of Cold-Start · p. 13
  • Surviving trajectories become multimodal SFT examples that supervise the full search, relevance-decision, consolidation, and answer-generation trace.SFT keeps all returned images in context, unlike inference and RL, which reconstruct context from the ledger plus a sliding window; assistant outputs alone receive loss.Source: Details of Cold-Start · pp. 13–14

On an Evidence Re-Fetch Module

  • The re-fetch proposal addresses pages discarded before later query constraints reveal their relevance, a risk illustrated by multi-hop questions with uncertain retrieval order.The proposed module lets the agent revisit previously dropped pages after consolidation.Source: On an Evidence Re-Fetch Module · p. 14
  • 78.4 → 78.5 ledger coverage and 77.16 → 76.79 accuracy show that re-fetch barely recovered pages and slightly lowered accuracy on SlideVQA.The authors attribute the small coverage gain to rare premature discards and the accuracy drop to noise from reopening pages lost through misperception.Source: Table 10 · p. 14, On an Evidence Re-Fetch Module · p. 14
  • The re-fetch finding is bounded to current benchmarks; longer horizons, more hops, or harder constraint ordering may make premature discards more frequent.The authors leave whether a well-designed re-fetch helps in those settings to future work.Source: On an Evidence Re-Fetch Module · p. 14

Main-Text Ablation Details

The ablations isolate relevance judgment and consolidation while holding the evidence-based training target constant, and separately examine the ledger and raw-image window as retrieval memory.

Training the Ablation Variants

  • Fine-tuned ablation variants are retrained from scratch under the same pipeline, retaining the sliding window and ledger while varying whether relevance judgment and consolidation are available.Variants are also reported under an untrained Vanilla setting; all reload ledger-referenced original images before answering.Source: Training the Ablation Variants · p. 14
  • The ablations remove relevance filtering, consolidation, or both, distinguishing unfiltered collection from the final reranking and ordering of retained images.Without consolidation, answers use reloaded images in collection order; without relevance judgment, every retrieved page enters the ledger.Source: Training the Ablation Variants · p. 14
  • Every fine-tuned variant keeps the same ledger-based coverage and compactness reward, so accuracy gaps reflect missing actions rather than changed training targets.The additional ablations are conducted on SlideVQA and report test accuracy.Source: Training the Ablation Variants · p. 14, Additional Ablations · p. 15

Additional Ablations

  • Removing the iterative textual ledger together with the sliding window lowers final accuracy because retrieval can drift and miss key images in multi-step cases.Table 11 evaluates removal of the iterative textual evidence ledger together with the sliding window.Source: Table 11 · p. 15, Additional Ablations · p. 15
  • W=2 gives the best sliding-window trade-off, preserving the common retrieve–zoom pair while leaving older evidence to the ledger.The study compares W=1, 2, and 3 with a setting that retains no raw image window; larger windows add visual tokens and irrelevant history.Source: Additional Ablations · p. 15

Prompt Template

The prompt template combines instructions, evidence context, action-specific prompts, consolidation reranking, and answer generation; case studies show both an ordered evidence chain and a chart-structure failure.

Prompt Template

  • SCORE’s prompt template includes system instructions, a textual evidence space, action-specific observation prompts, consolidation reranking, and a final-answer prompt.Figure 7 summarizes the templates used throughout the agent loop.Source: Prompt Template · p. 15, Figure 7 · p. 16
  • Blue template fields are environment-filled runtime variables, including the question, returned visual tokens, image filenames, and evidence entries.Figure 7 marks the runtime-variable fields in blue.Source: Prompt Template · p. 15, Figure 7 · p. 16

Case Study

  • In the good case, SCORE rejects an irrelevant slide and keeps the growth chart and financial page needed to answer the multi-hop question.Figure 8 depicts this relevance-filtering and consolidation example.Source: Case Study · p. 15, Figure 8 · p. 17
  • Consolidation orders the retained pages as the ranking evidence first and the 2011 financial page second, making the reasoning path inspectable.The final evidence table and answer expose that path without requiring inspection of the full retrieval history.Source: Case Study · p. 15, Figure 8 · p. 17
  • In the bad case, retrieval finds the relevant divestiture and acquisition slides, but the model binds the 2000 chart bar to Purina rather than PowerBar.Figure 9 shows the chart-structure error despite retrieval of the answer-bearing image.Source: Case Study · p. 15, Figure 9 · p. 18
  • The failure is chart-layout perception, not missing evidence or search: the model cannot correctly bind a brand logo to the year in a dense chart.The case marks a limit of selection and consolidation when fine-grained visual structure must be parsed.Source: Case Study · p. 15, Figure 9 · p. 18
Loading 2609.15800v1…