Source-linked AI summary
VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning
Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang, Pengjun Xie, Fei Huang, Feng Zhao
TL;DR
Visually rich information remains difficult for RAG because text-based methods cannot represent visual content and existing vision-based methods may underuse visual perception and reasoning. VRAG-RL trains VLMs through iterative search-engine interaction, visual perception actions, and integrated retrieval and outcome rewards, achieving over 20% improvement on various benchmarks and showing significant advantages in visual information reasoning, retrieval, and understanding.
Problem
RAG methods struggle with visually rich information, while existing visual approaches insufficiently activate visual-specific perception and reasoning.
Method
VRAG-RL uses reinforcement learning with iterative VLM-search interaction, coarse-to-fine visual perception actions, and rewards combining retrieval performance with model-based outcomes.
Results
Over 20% improvement on various benchmarks is reported, alongside significant advantages in visual information reasoning, retrieval, and understanding.
Takeaways & Limitations
The framework provides a reinforcement-learning approach tailored to training VLMs to reason, retrieve, and understand visually rich information.
Takeaways & Limitations
Multi-turn interaction with external environments can increase latency compared with single-search vanilla RAG.
Abstract
from arXiv · showhide
Effectively retrieving, reasoning and understanding visually rich information remains a challenge for RAG methods. Traditional text-based methods cannot handle visual-related information. On the other hand, current vision-based RAG approaches are often limited by fixed pipelines and frequently struggle to reason effectively due to the insufficient activation of the fundamental capabilities of models. As RL has been proven to be beneficial for model reasoning, we introduce VRAG-RL, a novel RL framework tailored for complex reasoning across visually rich information. With this framework, VLMs interact with search engines, autonomously sampling single-turn or multi-turn reasoning trajectories with the help of visual perception tokens and undergoing continual optimization based on these samples. Our approach highlights key limitations of RL in RAG domains: (i) Prior Multi-modal RAG approaches tend to merely incorporate images into the context, leading to insufficient reasoning token allocation and neglecting visual-specific perception; and (ii) When models interact with search engines, their queries often fail to retrieve relevant information due to the inability to articulate requirements, thereby leading to suboptimal performance. To address these challenges, we define an action space tailored for visually rich inputs, with actions including cropping and scaling, allowing the model to gather information from a coarse-to-fine perspective. Furthermore, to bridge the gap between users' original inquiries and the retriever, we employ a simple yet effective reward that integrates query rewriting and retrieval performance with a model-based reward. Our VRAG-RL optimizes VLMs for RAG tasks using specially designed RL strategies, aligning the model with real-world applications. The code is available at https://github.com/Alibaba-NLP/VRAG.
1 Introduction
VRAG-RL is a reinforcement-learning framework for improving VLM reasoning and retrieval over visually rich information. It combines visual perception actions, iterative tool interaction, and RAG-specific rewards to address limitations of existing visual RAG methods.
- Motivation: Existing visual RAG methods often insert images into context without sufficiently activating visual-specific perception and reasoning capabilities.This can limit reasoning-token allocation when models must handle complex visual information.
- Framework: VRAG-RL models VLM interaction with search engines and visual perception as iterative reasoning and tool invocation, with GRPO supporting automatic trajectory sampling and training.The framework supports multi-turn interaction and uses sampling strategies, post-processing, and model-based rewards for training stability.
- Visual perception: Its visual perception action space lets VLMs select, crop, and scale information-dense regions, progressively extracting information from coarse to fine levels.These actions are designed to focus attention on relevant visual regions and activate vision-specific reasoning capabilities.
- Reward design: VRAG-RL integrates retrieval performance with model-based outcome reward to better connect users’ original inquiries with retriever behavior.The reward structure treats retrieval quality as part of the optimization target rather than relying only on final outcomes.
- Results: Over 20% improvement on various benchmarks demonstrates that VRAG-RL significantly outperforms strong baselines.The reported result comes from extensive experiments evaluating the proposed method.
2 VRAG-RL
VRAG-RL frames visually rich RAG as iterative VLM reasoning with search and visual perception actions. Its framework combines coarse-to-fine region processing with retrieval- and outcome-oriented rewards to improve information gathering and answer generation.
- Visual perception actions: The visual perception action space unifies search queries, answer summaries, and region-specific actions for coarse-to-fine information acquisition.The model can select information-dense or query-relevant regions and progressively inspect them in greater detail.
- Visual perception actions: Visual perception tokens select, crop, zoom, and re-encode regions of retrieved images before reinserting them into the context.This addresses the mismatch between compressed vision-encoder representations and the higher-resolution visual details in real-world documents.
- Framework formulation: VRAG-RL formulates multimodal RAG as iterative reasoning and tool invocation between a VLM and search engine.The rollout and training process supports automatic trajectory sampling, multi-turn interaction, and GRPO-based optimization.
- Rollout and training: The framework uses large models to guide reasoning and smaller expert models to localize regions of interest during trajectory construction.Expert-localized coordinates replace the original perception tokens, and the re-encoded image becomes the next observation.
- Reward design: VRAG-RL combines pattern, retrieval-efficiency, and model-based outcome rewards to optimize both retrieval behavior and generated answers.Its retrieval reward is based on modified NDCG, favoring earlier and more comprehensive retrieval of relevant images while discouraging unnecessarily long contexts.
3 Experiments
Experiments evaluate VRAG-RL against text-based, vision-based, prompt-based, and RL-based baselines across retrieval, visual understanding, reasoning, and efficiency analyses. Results show gains from visual perception actions, RAG-specific rewards, and reinforcement learning, alongside a latency trade-off.
- Main Results: VRAG-RL outperforms Search-R1-VL on Qwen2.5-VL-7B-Instruct (34.7 →57.1) and Qwen2.5-VL-3B-Instruct (21.3 →53.5).The paper also reports stronger performance than prompt-based baselines across various benchmarks and effectiveness and generalization across retrieval and reasoning tasks.
- Approach Ablations: Removing any key module causes a clear accuracy drop across three benchmarks, supporting the contributions of the RAG-specific reward and visual-perception action space.Ablations further associate the action space with improvements in high-density visual information and the reward model with relevant retrieval for high-quality generation.
- Visual Perception Analysis: Visual perception actions improve relative performance particularly on Layout, Chart, and Figure tasks while optimizing resource use through dynamic resolution.The action space focuses on information-dense regions instead of simply maximizing the original image resolution.
- Reinforcement Learning Analysis: Reinforcement learning reduces invalid actions and increases finish rate during iterative reasoning with visual perception actions.Invalid actions include pattern errors, hallucinated crops, and answering before retrieval; inefficient reasoning can also involve repeated meaningless searches.
- Reward Analysis: Model-based rewards provide a more flexible and stable training signal than exact-match or recall-based rewards for visual reasoning tasks.The paper describes exact match as too strict and recall as vulnerable to misjudgments and repetitive reward-hacking responses.
- Time Efficiency: Multi-turn interaction increases latency, but the paper reports a beneficial latency–accuracy trade-off because retrieved context supports higher-quality answers.Vanilla RAG performs a single search, whereas ReAct RAG can enter repetitive search loops; VRAG-RL uses visual perception to avoid ineffective searches.
4 Related Work
Prior work applies RAG and reinforcement learning to language and vision-language models, but visual RAG remains an active area for improving reasoning with search engines.
- Traditional text-based RAG methods use agents to interact with search engines for knowledge-intensive problems.
- Vision-language models have been integrated with search engines to extend RAG into the visual domain.
- Reinforcement learning has been used to enhance reasoning capabilities in large language and vision-language models.
5 Conclusion and Future Work
VRAG-RL is presented as a reinforcement learning framework for complex reasoning over visually rich information, with evaluations reporting advantages across visual reasoning, retrieval, and understanding benchmarks.
- VRAG-RL enables vision-language models to interact with search engines for improved reasoning and retrieval.
- Extensive benchmark evaluations report advantages in visual information reasoning, retrieval, and understanding.
- Future work will add human-like actions and use advanced models to reduce hallucinations and improve reliability.
A Model-Based Reward
The model-based reward evaluates generated responses using a separate instruction-tuned language model and produces a binary correctness signal.
- The reward model assesses generated-response quality and relevance using the input query, reference answer, and response.
- Qwen2.5-7B-Instruct serves as the reward model and outputs a binary value, 0 or 1, for response correctness.
B The implementation of the search engine
The search-engine implementation separates OCR-based and vision-based retrieval pipelines, using ColPali for image-text alignment and PP-OCR for text extraction.
- The implementation uses separate OCR-based and vision-based retrieval pipelines.
- The vision-based retriever uses ColPali to align textual queries with images.
- The textual retrieval pipeline uses PP-OCR to extract text from images.
C Reinforcement Learning Framework with GRPO
The framework applies GRPO to optimize VLM policies through sampled interaction trajectories, using relative rewards within groups as the training signal. Training data is balanced across multi-step trajectories and search or perception actions.
- GRPO uses the average reward of multiple sampled outputs as a baseline instead of a learned value function.
- For each question, rollout samples a group of trajectories from a reference policy while interacting with an external environment.
- The advantage is computed from the relative rewards of outputs within each sampled group.
- Data Collection: Expert trajectories are collected with Qwen-VLmax-latest using a ReAct-based prompt, with Qwen2.5VL-72B handling image-region grounding when required.
- Data Proportions: Training trajectories are balanced across 2-6 reasoning steps and across search and perception actions.
E Dataset Information
The evaluation uses three visually rich document datasets covering slide question answering, large-scale document retrieval and reasoning, and long-context multimodal understanding. The section also describes text-based and visual-based RAG baselines alongside ReAct RAG, Search-R1, and Search-R1-VL.
- Evaluation covers SlideVQA, ViDoSeek, and MMLongbench.
- SlideVQA: SlideVQA contains over 2,600 slide decks, more than 52,000 slide images, and 14,500 questions requiring single-hop, multi-hop, and numerical reasoning.
- ViDoSeek: ViDoSeek evaluates visually rich document retrieval-reason-answer tasks across approximately 6,000 images containing text, charts, tables, and layouts.
- MMLongbench: MMLongbench evaluates VLM document understanding on long-context multimodal documents with text, images, charts, tables, and layout structures.
- Compared Baselines: Vanilla RAG retrieves with the original question using either text or image corpora, while ReAct RAG adds a Thought-Action-Observation loop.
- Compared Baselines: Search-R1 applies multi-turn reasoning RL to text RAG, and Search-R1-VL is a vision-based reproduction using VRAG-RL's framework, reward, and post-processing methods.
G Hyperparameters
The paper reports training hyperparameters in Tables 4 and 5, using identical hyperparameters across different models.
- Training hyperparameters are reported in Tables 4 and 5.These tables provide the detailed settings used during training.
- The same hyperparameters are used for different models.
- Table 4 lists key hyperparameters for supervised fine-tuning.
H Case Study
The case studies illustrate VRAG-RL trajectories for retrieving relevant images and increasing perception resolution when visual evidence is difficult to identify. The appendix also records prompts for VRAG-RL and comparison systems.
- Case Study: The VRAG-RL trajectories target accurate retrieval of relevant images and higher-resolution perception of reference information.
- Case Study: In one case, the model demonstrates reflective capability and eventually identifies subtle clues in relevant images.
- Prompts: ReAct RAG uses the same prompt as Search-R1, while the appendix presents prompts for VRAG-RL, Vanilla RAG, and the reward model.
- Case Study: One example asks for the difference between Dugwells and Shallow Tubewells in an IWMI map from India's Energy Divide.
- Case Study: Another example asks which H2 antagonist has lower bioavailability than Famotidine after identifying a drug affecting CYP-450 isoenzymes and hepatic blood flow.