UniDoc-RL: Coarse-to-Fine Visual RAG with Hierarchical Actions and Dense Rewards

• Visual RAG must address retrieval errors, noisy dense images, and sparse rewards; existing approaches often handle these factors separately or omit intermediate-decision credit assignment.Prior retrieval methods use generic similarity scores (Yu et al., 2024; Wang et al., 2025b), while visual utilization is often passive (Yu et al., 2024; Jin et al., 2025) and RL rewards sparse (Jin et al., 2025; Narayan et al., 2025).Source: §1 · pp. 1–2
• Across three benchmarks, UniDoc-RL consistently outperforms state-of-the-art baselines, with gains up to 17.7% over prior RL-based methods.Source: §1 · p. 2
• Its active perception action uses crop and zoom operations to localize informative regions, reduce redundancy, and preserve high-resolution visual details.Source: §1 · p. 2
• Stage-specific dense multi-rewards provide training signals across decisions, while a curated dataset supplies high-quality reasoning trajectories with fine-grained action annotations.Source: §1 · p. 2
• UniDoc-RL jointly models retrieval, reranking, active visual perception, and reasoning as a single decision process.It uses external tools for coarse retrieval, then LVLM-driven precise selection to rerank candidates by fine-grained query alignment and filter document-level noise.Source: §1 · p. 2

Vision-based Retrieval-Augmented Generation

• UniDoc-RL introduces a “Search-Select-Perceive” action space to progressively filter noise and focus on critical visual evidence, extending visual RAG beyond embedding retrieval and prior perception-action RL.Earlier approaches include embedding-based systems such as ColPali and VisRAG, agentic frameworks, and VRAG-RL.Source: §2 · p. 3

Reinforcement Learning for Multimodal Reasoning

• To address sparse outcome rewards and credit assignment in multimodal reasoning, UniDoc-RL explicitly rewards retrieval relevance, selection accuracy, and cropping precision.The rewards supervise each pipeline stage and are jointly optimized.Source: §2 · p. 3

3.1 Problem Formulation

• Given a query and image corpus, the agent seeks relevant images, key visual evidence, and a correct answer through iterative thought, action, and observation until it has sufficient information.At each step, its policy acts on the query and interaction history, then receives an observation from the environment.Source: §3.1 · p. 3

3.2 Action Definition

• UniDoc-RL progressively narrows visual evidence through image search, LVLM-based semantic selection, and active localization, cropping, and zooming of query-relevant regions.Selection filters irrelevant candidates, while perception produces high-resolution, query-focused observations and removes redundant content.Source: §3.2 · pp. 3–4

3.3 Reward Function Tailored

• Dense, stage-specific rewards use NDCG for retrieval and pseudo-positive supervision from the top-ranked initial candidate when retrieved sets contain no ground-truth images.The pseudo-positive target provides discriminative feedback for selection when the standard reward would otherwise remain zero.Source: §3.3 · pp. 4–5
• Crop IoU, format-validity, and model-based answer-quality rewards join retrieval and selection rewards in a weighted total objective.The crop reward compares predicted boxes with ground-truth regions; the pattern reward checks constraints such as correct <search>, <select>, and <bbox> tags.Source: §3.3 · p. 5

3.4 Multi-Round Generation with Hierarchical Actions

• In multi-round interaction, UniDoc-RL appends retrieved or cropped observations to conversation history as User-role inputs, aligning visual input placement with the LVLM’s pretraining distribution.Each round generates and executes an action, adds the resulting observation to the history, and continues until answer generation or the step limit.Source: §3.4 · pp. 5–6

4.1 Training Data Synthesis

• The authors curate 12,621 SFT and 5,537 RL samples from public benchmarks using a multi-stage filtering pipeline.Qwen3-VL-235B synthesizes trajectories, while Mineru layout analysis detects elements and proposes candidate ROI bounding boxes for Visual Perception actions.Source: §4.1 · p. 6

4.2 Training Pipeline and Implementation Details

• Training cold-starts with SFT before GRPO fine-tuning using the multi-reward system.Experiments use full-parameter fine-tuning, a group size of 5, and 8 NVIDIA A100 80G GPUs; SFT runs on llama-factory and GRPO on verl.Source: §4.2 · p. 6

4.3 Evaluation Datasets and Baselines

• Evaluation covers ViDoSeek, SlideVQA, and MMLongBench, using binary answer correctness and overall accuracy.Baselines span Vanilla RAG (Faysse et al., 2024), ReAct (Yao et al., 2022), Search-R1(-VL) (Jin et al., 2025), and VRAG-RL (Wang et al., 2025b). SlideVQA and ViDoSeek emphasize reasoning, while MMLongBench emphasizes visual reference content.Source: §4.3 · pp. 6–7, Table 1 · p. 7

4.4 Main Results

• UniDoc-RL leads across benchmarks, beating VRAG-RL (Wang et al., 2025b) by 17.5% with 3B and 17.7% with 7B models.RL methods outperform supervised baselines across all datasets. The authors attribute gains to Precise Selection, Visual Perception, and multi-reward optimization; purely visual RAG also improves over OCR-based methods, which lose visual cues.Source: Table 1 · p. 7, §4.4 · p. 7

5.1 Impact of Hierarchical Actions.

• Precise Selection improves relevant-image recall, while Visual Perception adds gains through active cropping and IoU-based guidance for localizing key evidence.Source: §5.1 · p. 8

5.2 Ablation on Multi-Rewards.

• Selection yields the largest action gains on reasoning-heavy SlideVQA and ViDoSeek, whereas Visual Perception benefits fine-grained discrimination on MMLongBench; the corresponding rewards show the same task-specific pattern.Source: §5.1 · p. 8, §5.2 · p. 8
• The Full multi-reward framework improves performance across all three datasets over sparse outcome-based and pattern rewards, using step-wise supervision to optimize intermediate actions and prevent error propagation.The resulting reasoning paths are described as more robust.Source: §5.2 · p. 8

6.1 Selection Action Facilitates Better Retrieval Results.

• 85.0% vs. 79.7% on SlideVQA, 85.7% vs. 74.8% on ViDoSeek, and 52.7% vs. 48.9% on MMLongBench after adding selection.The selection action filters irrelevant candidates and provides clearer visual evidence for downstream reasoning.Source: §6.1 · p. 8, Figure 3 · p. 8

6.2 SFT Enables Tool Use, While RL Refines the Tool-Selection Behavior

• UniDoc-RL increased crop frequency across all datasets, reaching 13.3% on MMLongBench; SFT cropped only 2.4% on ViDoSeek, and RL matched the Teacher’s frequency on MMLongBench.The low SFT crop frequency suggests exposure bias and a tendency toward passive full-image reading.Source: §6.2 · p. 9, Figure 4 · p. 9

6.3 Case Study

• UniDoc-RL learned precise crops that tightly bound regions of interest, unlike SFT’s trivial crops of entire images or irrelevant backgrounds.The IoU-based perception reward and task-specific feedback helped optimize both whether to act and how to execute the crop.Source: §6.3 · p. 9, Figure 5 · p. 9
• In case studies, UniDoc-RL cropped a chart to identify a loss spike around 150k steps, but skipped cropping when a retrieved table was already legible.These examples illustrate an adaptive strategy for extracting fine-grained details when needed and directly answering when cropping adds no value.Source: §6.3 · p. 9

7 Conclusion

• UniDoc-RL unifies retrieval, reranking, active visual perception, and reasoning in an LVLM agent that progressively refines evidence from coarse retrieval to region cropping.Its hierarchical action space filters visual evidence, suppressing noise and focusing on key information.Source: §7 · p. 9
• A dense multi-reward mechanism provides stage-aware supervision to address sparse outcome-based rewards and enable end-to-end optimization via GRPO.Source: §7 · p. 9
• The authors publicly release a high-quality dataset of reasoning trajectories with fine-grained action annotations, and report consistent state-of-the-art gains across three benchmarks.Source: §7 · p. 9

A.1 Data Collection

• Over 2,600 slide decks, 52K images, and 14.5K QA pairs make up SlideVQA (Tanaka et al., 2023), which tests cross-slide single-hop, multi-hop, and numerical reasoning.Source: §A.1 · p. 13
• 3,276 documents across 72,880 pages and 6 languages form DoubleBench (Shen et al., 2025), with 5,168 single-hop and multi-hop queries across PDFs, scans, slides, and HTML.Source: §A.1 · p. 13
• Over 1,200 documents and 35,000 QA pairs across 16 languages make up VisRAG-Bench (Chen et al., 2025a), covering chart-, text-, and table-based questions.Source: §A.1 · p. 13
• DUDE (Landeghem et al., 2023) spans healthcare, legal, technical, and financial documents, including digitally born and scanned files with varied layouts and question types.Source: §A.1 · p. 13
• 229 real-world documents and 1,102 QA pairs make up DocBench (Zou et al., 2024), spanning five domains and text, multimodal, metadata, and unanswerable question types.Its QA pairs combine human annotation with GPT-4 synthesis.Source: §A.1 · p. 13

A.2 Automated Trajectory Synthesis.

• A proprietary LVLM teacher, such as Qwen3-VL-235B (Bai et al., 2025a), generates thought-action pairs from the trajectory history to supervise coordination among retrieval, selection, and perception.Source: §A.2 · p. 13
• After precise image selection, Mineru (Niu et al., 2025) proposes layout-based candidate regions; the teacher selects a crop when critical information is absent globally, and the crop becomes the next observation.Source: §A.2 · p. 13

Quality Filtering

• Qwen3-VL-235B checks each synthesized trajectory against the reference answer, and trajectories with incorrect final answers are discarded.This quality filter retains factually accurate reasoning paths for training.Source: §A.3 · p. 13

Difficulty-Aware Filtering

• The difficulty-aware filter retains teacher-correct samples that an intermediate SFT model, trained on quality-filtered SlideVQA data, answers incorrectly.This removes samples the partially trained model can already solve.Source: §A.3 · pp. 13–14

RL Data Curation

• For RL curation, five high-temperature SFT rollouts identify queries with successful relevant-document retrieval but at least one incorrect answer rollout.The method also prioritizes samples discarded during quality filtering as likely hard cases that may benefit from RL optimization.Source: §A.3 · p. 14

A.4 Data Statistics

• The final dataset contains 12,621 SFT samples and 5,537 RL samples, with composition by source dataset reported in Table 4.Source: §A.4 · p. 14

B Case Study

• Two cases show adaptive action use: cropping zooms into a loss chart to identify a spike around 150k steps, while a legible table is answered without cropping.The examples illustrate how UniDoc-RL uses Visual Perception for fine-grained details but skips cropping when the table is already legible.Source: B Case Study · p. 15

C Hyperparameters

• Detailed SFT and RL training hyperparameters are provided in Tables 5 and 6.Source: Table 5 · p. 15, Table 6 · p. 15

D Model-Based Reward

• Qwen2.5-72B-Instruct (Bai et al., 2025b) evaluates generated-response correctness against the query and reference answer, outputting a binary reward of 0 or 1.The reward-model prompt is shown in Figure 7.Source: D Model-Based Reward · p. 15, Figure 7 · p. 16

E Prompts

• Figure 8 presents the prompts used for training and testing.Source: E Prompts · p. 15, Figure 8 · p. 16
• Figures 9–11 provide data-synthesis prompts for the image search, precise selection, and visual perception actions.Source: E Prompts · p. 15, Figure 9 · p. 17, Figure 10 · p. 17