PaperlayerView exact revision on arXiv

Source-linked AI summary

Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation

Ruofan Hu, Shengyang Xu, Minjie Hong, Xiaoda Yang, Sashuai Zhou, Ke Lei, Tao Jin, Zhou Zhao

arXiv:2608.30163v1cs.IRcs.CV

TL;DR

Existing multimodal RAG systems are limited by multi-image reasoning challenges, closed-document assumptions, and redundant visual-token computation. The paper introduces DocLongRAG and Doc-REFRAG, which train on realistic retrieval contexts and selectively expand question-relevant visual chunks; across six benchmarks, Doc-REFRAG outperforms eleven baselines with lower latency.

  • Problem

    Existing multimodal RAG models struggle with heterogeneous multi-image retrieval contexts and incur computational overhead from irrelevant visual tokens.

  • Method

    DocLongRAG provides long, noisy retrieval contexts, while Doc-REFRAG compresses visual tokens into chunks and selectively expands relevant ones with a lightweight RL-based selector.

  • Results

    Across six benchmarks, Doc-REFRAG outperforms eleven baselines, achieving state-of-the-art accuracy with lower inference latency.

  • Takeaways & Limitations

    The approach supports efficient large-context multimodal RAG inference, especially in resource-constrained deployments.

  • Takeaways & Limitations

    The chunk size k is fixed during training and cannot adapt dynamically at inference time; documents exceeding 4,000 characters show approximately 5% lower ANLS.

Abstract

from arXiv · show

Real-world knowledge resides in multimodal documents, necessitating retrieval-augmented generation (RAG) for accurate question answering. However, existing multimodal RAG models are primarily designed for single-image or closed-document settings and exhibit limited accuracy in realistic multi-image scenarios. Moreover, processing numerous retrieved images incurs substantial computational overhead from irrelevant visual tokens. To address these challenges, we introduce DocLongRAG, a large-scale dataset of 343K question--answer pairs, each associated with an average of 37.4 retrieved images to reflect authentic RAG workflows. Building on this dataset, we propose Doc-REFRAG, a question-guided framework that compresses visual tokens into coarse chunks and selectively expands question-relevant ones via a lightweight RL-based selector. Experiments on six benchmarks show that Doc-REFRAG outperforms eleven strong baselines, achieving state-of-the-art accuracy with significantly lower inference latency. Our resources are available at https://github.com/Collab-Gen/Doc-REFRAG.

1 Introduction

Multimodal RAG must handle heterogeneous, noisy multi-image evidence while controlling visual-token overhead. DocLongRAG and Doc-REFRAG address these challenges through realistic retrieval training data and question-guided selective visual expansion.

  • Existing MLLMs struggle with multi-image reasoning, while retrieved visual tokens create substantial computational overhead.
  • Closed-document training does not reflect RAG inputs, which contain heterogeneous, redundant, and contradictory images from diverse documents.
  • Visual-compression methods that prioritize complex patches can retain question-irrelevant regions, leaving redundant tokens and high prefill latency.
  • 343,474 question–answer pairs grounded in 3.3M document images provide broad-format coverage, with each question associated with an average of 37.4 images.
  • Doc-REFRAG compresses visual tokens into chunks and selectively re-expands question-relevant chunks through a lightweight question-conditioned selector.
  • A three-stage training framework combines single-image reconstruction, multi-image continual pretraining, and RL-based selector training using answer accuracy as reward.
  • Across six benchmarks, Doc-REFRAG outperforms eleven existing approaches.

2 Related Work

Related multimodal document-RAG work addresses retrieval and visual-token compression, but commonly targets coherent document inputs rather than fragmented real-world retrieval contexts.

  • Multimodal document-RAG research focuses on retriever design while often using general-purpose MLLMs as generators.
  • Visual-token compression methods use importance, redundancy, or lightweight component-replacement strategies to reduce the cost of multi-image inputs.

3 DocLongRAG

DocLongRAG is designed to train multimodal models on long, noisy, retrieval-dependent contexts rather than isolated or coherent document images. It interleaves relevant images with hard negatives while preserving diverse formats and answer types.

  • Conventional VDU datasets provide limited context, whereas RAG requires reasoning across long, noisy, and conflicting retrieved inputs.
  • DocLongRAG simulates practical retrieval scale and noise to support training stability.
  • 3.1 Dataset Collection and Filtering: The corpus filters context-independent, online-search, unanswerable, and ethically problematic questions, reducing the training pool by 18.2%.
  • 3.2 Negative Document Extension: Relevant images are interleaved with hard-negative distractors to form extended input sequences under realistic retrieval noise.
  • 3.2 Negative Document Extension: For each relevant image, a document-aware retriever selects similar images outside the relevant set, while manual inspection finds an average of 2.47 retrieved images still relevant.
  • 3.2 Negative Document Extension: The resulting context is long, redundant, and sometimes conflicting, with constructed noise used for training and real-retriever noise reserved for evaluation.
  • 3.2 Negative Document Extension: Placing relevant images before hard negatives consistently improves training performance and establishes longer dependencies.
  • DocLongRAG aggregates diverse formats and question types, requires answers grounded in document images, and associates each question with an average of 37.4 images.

4 Doc-REFRAG

Doc-REFRAG compresses retrieved-image visual tokens into coarse, layout-aware chunks, then uses a question-conditioned RL selector to restore fine-grained tokens for relevant regions. Its training curriculum aligns chunk semantics, supports mixed multi-image sequences, and trains selection under accuracy rewards.

  • Model Architecture: Doc-REFRAG encodes each retrieved image into 324 visual tokens, groups them into non-overlapping chunks, and projects each chunk into one d-dimensional embedding.Chunking preserves localized page structure while reducing sequence length.
  • Model Architecture: Question-guided compression prioritizes semantic relevance over visual appearance because visual density correlates poorly with question relevance in RAG.The method addresses the question-agnostic behavior of existing compression approaches.
  • Model Architecture: Expanded and coarse visual representations are concatenated with question tokens and fed to a decoder-only language model to generate the answer.The architecture includes a document-tailored vision encoder, MLP projector, selector, and decoder-only language model.
  • Model Architecture: A lightweight RL-based selector dynamically expands selected chunks back to their original k constituent tokens, balancing efficiency with fine-grained semantic fidelity.The selector chooses exactly T = ⌊p·n·L⌋ chunks, with p controlling the expansion budget.
  • Model Training: Training proceeds through single-image reconstruction, multi-image continual pretraining, and RL-based selector training using answer accuracy as the reward.The curriculum progresses from semantic reconstruction to hybrid multi-image sequences and selector optimization.
  • Model Training: DocLongRAG training introduces heterogeneous, redundant, and contradictory contexts with sparse and imperfect expansion to approximate practical RAG deployment.Training expands 20% of chunks in earlier stages, then reduces expansion to 10% for DocLongRAG while activating random irrelevant chunks.
  • Model Training: Doc-REFRAG achieves lower inference latency than ColQwen with competitive accuracy.The selector is trained with GRPO-style policy optimization and clipped PPO updates while the decoder remains frozen for reward computation.

5 Experiments

Experiments evaluate Doc-REFRAG across six visual document understanding benchmarks under realistic multi-image retrieval settings, including accuracy, latency, selector strategies, and ablations. The results show strong accuracy, lower generation latency, effective chunk selection, and the importance of the training design and DocLongRAG.

  • Experimental Setup: Six benchmarks evaluate Doc-REFRAG against eleven multi-image MLLM baselines using retrieved image contexts.The evaluation retrieves the top-20 images per question, covering 93.4% of gold images on average.
  • Main Results: Doc-REFRAG with k = 3 outperforms general and specialized long-document MLLMs on noisy retrieval contexts.DocLongRAG training and question-guided compression address redundant, conflicting, or irrelevant content while reducing tokens.
  • Efficiency Analysis: 2.18 s lower TTIT than Docopilot accompanies a slight TTFT increase, while accuracy improves by over 18% versus token-compressed MLLMs at comparable token budgets.Within Doc-REFRAG, k = 3 provides a 2.57 s TTFT trade-off; k = 6 reduces TTFT to 1.95 s.
  • Selector Analysis: 57.5% average accuracy surpasses random selection at 38.8% and perplexity-based selection at 43.7%.ColQwen reaches 9.3 s latency, whereas the RL selector identifies relevant chunks in 3.7 s.
  • Quality Analysis: ANLS remains above 95% through 4,000 characters with 150 tokens per image, matching higher-token baselines.At 6,000 characters, performance stays within a 5% absolute gap from the baselines.
  • Ablations: Removing reconstruction supervision reduces accuracy from 55.1% to 19.7%, while shuffled multi-image training reduces it to 24.2%.Excluding DocLongRAG lowers average accuracy from 55.1% to 48.9%, and fine-tuning other MLLMs on it improves accuracy by 3.9%, 4.3%, and 3.9%.

6 Conclusion

The paper introduces Doc-REFRAG for efficient multimodal document RAG through question-guided visual compression, selective expansion, and three-stage training. Across six benchmarks, it reports state-of-the-art accuracy with lower latency than eleven baselines and validates transfer to document reranking.

  • Conclusion: Doc-REFRAG compresses visual tokens into chunks and selectively expands question-relevant details with an RL-based selector.Its three-stage training paradigm supports chunk representations and mixed multi-image processing.
  • Conclusion: Across six benchmarks, Doc-REFRAG outperforms eleven baselines with state-of-the-art accuracy and lower latency.The paper also validates its training methodology, DocLongRAG’s role, and transferability to document reranking.

Limitations

The paper identifies three limitations concerning fixed chunk granularity, coarse reward attribution, and errors in dense or structurally complex regions. It leaves adaptive chunk-size generalization and richer selector supervision for future work.

  • Limitations: Fixed chunk size cannot adapt at inference time, and performance degrades by approximately 5% ANLS beyond 4,000-character documents.Supporting multiple chunk sizes would require additional routing and separately trained selectors, which can undermine efficiency.
  • Limitations: The RL selector uses answer accuracy as its sole reward, without fine-grained attribution of individual chunk contributions.A richer reward signal or chunk-level supervision is suggested for improving precision in difficult regions.
  • Limitations: 64% of remaining errors concentrate on dense or structurally complex text regions.This distribution identifies the main error concentration reported for the selector.

Ethics Statement

DocLongRAG is built from public datasets and document corpora with filtering for unsuitable questions and ethically problematic content. Synthetic dense-text images contain no real user data, while deployment remains subject to errors and requires human oversight in high-stakes settings.

  • Dataset Ethics: A four-stage filtering pipeline removes context-independent, online-search-dependent, unanswerable, and ethically problematic instances.The excluded content includes personally identifiable information, offensive language, and sensitive imagery.
  • Dataset Ethics: Synthetic document images for dense-text evaluation are generated programmatically and contain no real user data.
  • Deployment Boundary: The system can still err on dense, overlapping, or contradictory evidence, so high-stakes deployment requires human oversight.The paper states that outputs should not be relied on solely in such contexts.

A Differentiated Contributions Over Related Compression Methods

Doc-REFRAG differs from prior visual-token compression methods through question-conditioned learned restoration, dynamic RL-trained budgets, and training on heterogeneous retrieval noise.

  • Query-conditioned compression: Unlike heuristic query-conditioned methods, Doc-REFRAG trains selection end-to-end against downstream answer accuracy.Prior methods use attention, instruction aggregation, or generated context tokens without downstream reward optimization.
  • Dynamic budget with RL training: Doc-REFRAG samples a dynamic budget p ∈[0.1, 0.3] during training and uses GRPO with clipped PPO updates to optimize selection.The frozen decoder’s answer accuracy serves as the reward signal.
  • RAG noise robustness: Doc-REFRAG trains on retrieval contexts containing heterogeneous, redundant, contradictory images and interleaved hard negatives.This differs from prior compression methods developed for coherent single-source documents or video frames.
  • Learned restoration vs. heuristic retrieval vs. condensation: Doc-REFRAG compresses all visual tokens into chunks, then selectively restores question-relevant regions through a learned policy.This learned restoration differs from heuristic retrieval and irreversible, question-agnostic condensation.
  • Dataset construction: DocLongRAG aggregates English data from 15 open-source datasets.The dataset construction spans multiple source datasets used to support the retrieval-oriented training setting.

C Evaluation of Hard Negative Relevance

The hard-negative evaluation examines relevance among retrieved images and exposes annotation limitations, positional effects, and the geometric supervision used for chunk selection.

  • Annotation limitations: The original MMDocIR annotation missed Fig. 5b, which also supports the answer and identifies a +20-point increase for the “Some college or more” subgroup.The example shows that an exclusively annotated target can omit an equally relevant candidate.
  • Human validation: 500 randomly selected questions were evaluated across top-20 non-ground-truth retrieved images by two expert annotators.Each candidate was rated for answer-supporting content on a 5-point Likert scale.
  • Human validation: Annotator agreement reached Krippendorff’s alpha of 0.78, indicating substantial agreement.The relevance labels distinguish explicit evidence, partial evidence, topical relatedness, superficial similarity, and clear irrelevance.
  • Relevance by rank: Average relevance scores were 4.62, 4.31, and 3.94 at ranks 1, 2, and 3, but remained below 3.27 from rank 4 onward.The results indicate that higher-ranked retrieved images more often contain meaningful supporting evidence.
  • Insertion strategy: Relevant images placed before hard negatives yielded better performance by establishing longer dependencies with the decoder.At inference, however, the selector re-evaluates chunks independently and can expand any image based on content relevance rather than position.
  • Geometric mapping: Coarse-to-fine pseudo-labels aggregate ColQwen question-to-patch similarities over geometrically mapped fine patches.The deterministic mapping preserves local spatial structure and provides supervision for chunk selection.

H Analysis of Chunk Size k

Chunk-size experiments identify k = 3 as the preferred balance between accuracy and efficiency, while broader analyses quantify budget, cost, and failure-mode trade-offs.

  • Chunk-size comparison: k = 3 achieves the highest average accuracy of 57.5% across six document-level benchmarks.It improves average accuracy by 2.0% over k = 2 while maintaining comparable inference latency.
  • Chunk-size comparison: k = 6 reduces visual tokens from ∼194 to ∼108 per image and TTFT from 2.57s to 1.95s, but lowers average accuracy by 5.2% versus k = 3.The degradation is especially evident on ChartQA and InfoVQA.
  • Expansion budget: p = 0.2 reaches 57.5% accuracy while maintaining low inference latency.Lower budgets provide insufficient context, whereas higher budgets add computational cost without significant improvement.
  • Expansion budget: At p = 0.2, the RL selector achieves 57.5% accuracy in 3.7 s, surpassing ColQwen’s 53.6% accuracy at 9.3 s.This corresponds to a 2.5× reduction in selection latency.
  • Training cost: LoRA in Stage 2 and a lightweight selector in Stage 3 keep the training footprint manageable on commodity RTX 3090 hardware.Stage 2 is the dominant training-cost bottleneck, whereas Stages 1 and 3 are lightweight.
  • Qualitative analysis: In a 100-question error analysis, 64% of failures involve dense or semantically complex text, 22% figures, and 14% structured data.These categories expose fidelity limits when selective chunking omits critical visual or numerical regions.
  • Efficiency validation: Doc-REFRAG’s TTFT acceleration reaches 2.46× with 50 retrieved images, while TTIT acceleration reaches at most 1.12×.Average accuracy with 50 retrieved images is 50.2%.

N Disentangling Architectural and Data Contributions

A factorial analysis separates the effects of DocLongRAG and the Doc-REFRAG architecture, showing that both contribute independently and interact beneficially.

  • Data contribution: Fine-tuning DocOwl2 on DocLongRAG alone raises average accuracy from 45.8% to 49.7%, a +3.9% data contribution.This measures the dataset effect without changing the architecture.
  • Architecture contribution: Replacing DocOwl2’s encoding with Doc-REFRAG’s architecture raises accuracy from 45.8% to 48.9%, a +3.1% architecture contribution.This measures the architectural effect without DocLongRAG.
  • Synergistic interaction: The full system reaches 55.1%, with a +2.3% synergistic interaction beyond the two individual effects.The interaction reflects the selector’s ability to exploit noisy-retrieval training signals from DocLongRAG.
  • Independent contributions: DocOwl2 fine-tuned on DocLongRAG reaches 49.7%, remaining 5.4 percentage points below Doc-REFRAG Full at 55.1%.Doc-REFRAG without DocLongRAG reaches 48.9%, also trailing the full system.
  • Random-data ablation: The naive-random variant achieves 35.4% average accuracy, 10.4 percentage points below DocOwl2 original at 45.8%.It uses approximately 37 randomly sampled images without hard-negative mining or quality filtering.
  • Random-data ablation: The full-versus-naive-random gap is 19.7 percentage points, validating the contribution of DocLongRAG’s construction pipeline beyond raw image quantity.The comparison holds architecture and other training choices constant.
Loading 2608.30163v1…