VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning

Introduction

• The framework addresses underused visual reasoning, inefficient retrieval from poorly articulated requirements, and instability in VLM multi-turn reasoning.The paper attributes these challenges to insufficient visual-specific perception and reasoning-token allocation, repetitive or meaningless interactions, and limits in VLM instruction-following and reasoning.Source: §1 · p. 2
• VRAG-RL trains VLMs for iterative reasoning and tool invocation with search engines, using automatic trajectory sampling, GRPO, and strategies designed to stabilize multi-turn training.The authors also re-annotate visually rich document datasets and develop a pipeline to scale data for RL and SFT.Source: §1 · p. 3
• Its visual-perception action space lets VLMs select, crop, and scale regions of interest to gather information progressively from coarse to fine.Focusing on information-dense image or chart regions is intended to activate vision-specific reasoning within limited context length.Source: §1 · pp. 2–3
• VRAG-RL combines retrieval-performance and model-based outcome rewards to encourage pertinent retrieval and bridge users’ original intentions with the retriever.The authors note that persistently retrieving irrelevant documents adds noise and hampers reasoning.Source: §1 · pp. 2–3
• VRAG-RL significantly outperforms strong baselines, achieving over 20% improvement on various benchmarks.Source: §1 · p. 3

2.1 Problem Formulation

• Given a query and a large image collection, the system must retrieve relevant images, reason over their complex visual content, and generate an answer.The images include flowcharts, charts, tables, and varied layouts from real-world documents such as slides and reports.Source: §2.1 · p. 3

2.2 Visual Perception Action Integration for Understanding Information-Dense Regions

• Visual-perception actions let the model select query-relevant or information-dense regions, then crop, zoom, and re-encode them for higher-resolution inspection.The crop-and-re-input strategy increases the density of vision tokens relative to encoding the full image within the encoder’s pixel limit.Source: §2.2 · pp. 3–5
• Before RL, multi-expert sampling scales up SFT trajectories: large models guide reasoning and tool choices, while smaller experts annotate regions and localize coordinates.When a visual-perception action is produced, grounding-specific experts use the large model’s thought to refine region coordinates before the cropped, re-encoded image is returned as an observation.Source: §2.2 · pp. 4–5

2.3 Fine-Grained Reward Function Tailored for Enhancing RAG Framework

• Retrieval efficiency reward uses a modified NDCG over retrieved images and predefined relevance scores, favoring early retrieval of relevant images and penalizing unnecessary context.The authors note that excessively long context can interfere when information is already sufficient.Source: §2.3 · p. 5
• The combined reward balances retrieval, model-evaluated answer correctness, and pattern consistency; pattern reward is usually zero after SFT but is set to 0.1 for RL cold starts.The evaluation model judges the generated answer against the query and golden answer; the three reward components are weighted to sum to one.Source: §2.3 · p. 6

2.4 Reinforcement Learning Framework with Iterative Reasoning

• In multi-turn rollouts, the model alternates search and visual-perception actions with image observations inserted as user content, matching its pre-training convention for image tokens.The interaction ends when the model emits an answer or reaches the iteration limit.Source: §2.4 · p. 6
• Training uses GRPO with group-sampled role-play trajectories to optimize retrieval-augmented reasoning under a KL-divergence penalty against a reference model.The policy’s rollout process combines the policy model with the external environment.Source: §2.4 · p. 7

3.1 Experimental Settings

• Experiments compare Vanilla RAG, ReAct, and Search-R1(-VL) on ViDoSeek, SlideVQA, and MMLongBench using binary model-based evaluation.SFT and RL use llama-factory and verl, respectively; training and inference use a search engine built from approximately 70k visual documents.Source: §3.1 · p. 7

3.2 Results

• On Qwen2.5-VL-7B-Instruct, VRAG-RL improves over Search-R1-VL (34.7 →57.1), and on Qwen2.5-VL-3B-Instruct, it improves from 21.3 →53.5.The paper reports improvements across ViDoSeek, SlideVQA, and MMLongBench, and says visual-based methods outperform OCR-based approaches while prompt-based Vanilla RAG and ReAct lag RL-based methods.Source: §3.2 · p. 8
• Removing any VRAG-RL module reduces accuracy, supporting the contributions of its RAG-specific reward and visual-perception action space.The ablations also indicate that relevant retrieval is a prerequisite for high-quality generation and that visual actions particularly help with high-density visual information.Source: §3.2 · p. 8, Table 2 · p. 8

3.3 Analysis

• More relevant information retrieved early supports focused contexts and higher-quality answers, while excessively long contexts can introduce noise.The paper reports that VRAG-RL retrieves relevant information more effectively than traditional prompt-based rewrite methods.Source: §3.3 · p. 8, Figure 4 · p. 8
• The visual-perception action space focuses on information-dense image regions and improves performance on visual tasks, particularly Layout, Chart, and Figure.Its dynamic-resolution strategy aims to improve perception while conserving computational resources; case studies show selective visual actions and identification of subtle image clues.Source: §3.3 · pp. 9–10, Figure 5 · p. 9
• With reinforcement learning, VRAG-RL reduces invalid actions and increases finish rates, supporting more effective multi-step reasoning over out-of-domain visual information.Invalid actions include format errors and hallucinations such as incorrect cropping or answering before retrieval; ineffective reasoning can also involve repeated searches.Source: §3.3 · p. 9, Table 3 · p. 9
• A model-based reward provides a more flexible and stable training signal than exact match or recall rewards for visual reasoning tasks.The paper says exact-match rewards can be too strict, while recall rewards can invite repetitive responses that destabilize training.Source: §3.3 · p. 9
• VRAG-RL’s multi-turn interactions can increase latency, but the paper describes the resulting accuracy trade-off as beneficial for visually rich retrieval and understanding.Unlike ReAct RAG, which is limited to answer and search actions and can repeat searches, VRAG-RL’s visual-perception space helps it extract answers after retrieval.Source: §3.3 · pp. 9–10, Figure 6 · p. 9

Vision-based Retrieval-augmented Generation

• Prior research includes text-based RAG agents, OCR-free retrieval aligning textual queries with images, and multimodal RAG agents; this work builds on these developments by adding visual-perception actions to visual-based RAG.Source: §4 · p. 10

Reinforcement Learning with Large Models

• RL has been used to strengthen LLM and VLM reasoning and train large-model agents, but frameworks tailored to multimodal iterative reasoning remain scarce; VRAG-RL targets this gap.Its framework enables VLMs to reason iteratively with visual-perception actions in complex multimodal retrieval-augmented reasoning tasks.Source: §4 · p. 10

• VRAG-RL is a reinforcement-learning framework that helps VLMs interact more effectively with search engines across visually rich information.Source: §5 · p. 10
• Extensive benchmark evaluations demonstrated advantages in visual information reasoning, retrieval, and understanding.Source: §5 · p. 10
• Future work will add actions that mimic human handling of complex information, helping models focus on deeper thinking.Source: §5 · p. 10
• The authors also plan to leverage more advanced models to reduce hallucinations and improve framework accuracy and reliability.Source: §5 · p. 10

A Model-Based Reward

• Qwen2.5-7B-Instruct evaluates each generated response against the input query and reference answer, returning a binary accuracy score of 0 or 1.It is deployed on 4 NVIDIA A100 GPUs for batch evaluation.Source: A Model-Based Reward · p. 15
• Compared with rule-based exact match or recall rewards used in prior work (Jin et al., 2025; Chen et al., 2025a), the model-based reward is more flexible and comprehensive.The paper reports higher training efficiency and better generalization to diverse datasets.Source: A Model-Based Reward · p. 15

B The implementation of the search engine

• The implementation uses separate vision-based and OCR-based retrieval pipelines to support VRAG-RL tasks.The vision retriever uses ColPali to align textual queries with images, while PP-OCR extracts text from images for textual retrieval.Source: B The implementation of the search… · p. 15
• Llama-Index provides indexing and querying for large-scale image datasets.The vision-based retriever is built on ColPali; the textual pipeline uses PP-OCR.Source: B The implementation of the search… · p. 15
• The search engine ran on a single NVIDIA A100 80G GPU, and batch querying further optimized retrieval speed.The paper describes this deployment as supporting efficient large-scale queries and suitability for real-time applications.Source: B The implementation of the search… · p. 15

C Reinforcement Learning Framework with GRPO

• GRPO uses the average reward of multiple sampled outputs as a baseline instead of a learned value function.The policy is optimized by maximizing an objective based on this approach.Source: C Reinforcement Learning Framework… · p. 15
• For each question, the rollout module samples trajectory groups from the reference policy by interacting with the external environment.The sampled group is {y1, y2, . . . , yG}.Source: C Reinforcement Learning Framework… · p. 15
• Each trajectory’s advantage is computed from the relative rewards of outputs within its sampled group.Source: C Reinforcement Learning Framework… · p. 15

D Expert Trajectories Collection

• Qwen-VLmax-latest was used for prompt-based expert-trajectory collection, with a React-based prompt to elicit complex reasoning.Source: D Expert Trajectories Collection · p. 15
• When image-region grounding was needed, Qwen2.5VL-72B performed it under the guidance of historical trajectories.Source: D Expert Trajectories Collection · p. 15
• Training trajectories were balanced across 2–6 reasoning steps and search and perception actions to support diverse multi-step reasoning and environment interactions.Source: D Expert Trajectories Collection · pp. 15–16

E Dataset Information

• The evaluation uses SlideVQA, ViDoSeek, and MMLongbench as its three visually rich document datasets.Source: E Dataset Information · p. 16
• SlideVQA (Tanaka et al., 2023) contains over 2,600 slide decks, more than 52,000 slide images, and 14,500 questions involving single-hop, multi-hop, and numerical reasoning.It includes annotated arithmetic expressions for numerical questions.Source: E Dataset Information · p. 16
• ViDoSeek (Wang et al., 2025a) evaluates retrieval-reason-answer over approximately 6,000 images with diverse text, chart, table, and layout content.Unlike VQA datasets focused on individual images or documents, its queries have unique answers across a collection.Source: E Dataset Information · p. 16
• MMLongbench (Ma et al., 2024) assesses VLM document understanding on long-context multimodal documents containing text, images, charts, tables, and layouts.Source: E Dataset Information · p. 16

F Compared Baselines

• Vanilla RAG baselines retrieve directly from the original question, using either text or images as the search corpus and inserting retrieved material into context.Text-based Vanilla RAG retrieves text; visual-based Vanilla RAG retrieves images.Source: F Compared Baselines · p. 16
• ReAct RAG (Yao et al., 2023) adds Chain-of-Thought prompting in a Thought-Action-Observation loop, with retrieval and inserted-information modalities distinguishing text and visual variants.Source: F Compared Baselines · p. 16
• Search-R1 (Jin et al., 2025) is a multi-turn reasoning RL method for text RAG, reproduced with multi-turn interactions and rule-based rewards.Source: F Compared Baselines · p. 16
• Search-R1-VL is a vision-based Search-R1 baseline implemented in the authors’ framework, trained from cold start on VRAG-RL’s dataset with the same reward and post-processing methods.Source: F Compared Baselines · p. 16

G Hyperparameters

• Training uses identical hyperparameters across models, and Table 4 lists key SFT hyperparameters.Detailed training settings are reported in Tables 4 and 5.Source: G Hyperparameters · p. 16, Table 4 · p. 17

H Case Study

• Table 5 lists key hyperparameters for RL.Source: Table 5 · p. 17
• VRAG-RL reflects on image retrieval and uses visual perception actions when needed to identify subtle clues and obtain higher-resolution information.The two cases illustrate challenges of retrieving relevant images and perceiving reference information at higher resolution; the model uses perception actions selectively rather than simply reproducing training patterns.Source: H Case Study · p. 17, Figure 7 · p. 18, Figure 8 · p. 19

I Prompts

• The paper presents its prompts and notes that some are taken from Search-R1 Jin et al. (2025).Source: I Prompts · p. 17

I.3 ReAct RAG Prompt

• ReAct RAG uses the same prompt as Search-R1, shown in Figure 10.Source: §I.3 · p. 17, Figure 10 · p. 20

I.5 Model-based Reward Prompt

• The appendix identifies separate prompts for VRAG-RL, Search-R1(-VL) and ReAct RAG, and Vanilla RAG.Figures 9, 10, and 11 label the respective prompts.Source: Figure 9 · p. 20, Figure 10 · p. 20, Figure 11 · p. 21
• The model-based reward prompt frames evaluation as an expert system for chatbot question answering, given the query, generated answer, and reference answer.Source: §I.5 · p. 21, Figure 12 · p. 21