ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

António Loison; Quentin Macé; Antoine Edy; Victor Xing; Tom Balough; Gabriel Moreira; Bo Liu; Manuel Faysse; Céline Hudelot; Gautier Viaud

Source-linked AI summary

ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

António Loison, Quentin Macé, Antoine Edy, Victor Xing, Tom Balough, Gabriel Moreira, Bo Liu, Manuel Faysse, Céline Hudelot, Gautier Viaud

arXiv:2601.08620v2Jan 13, 2026cs.AIcs.CV

TL;DR

Existing RAG benchmarks only partially capture visual information, complex reasoning, multilingual evaluation, and source grounding. ViDoRe V3 addresses this gap with a human-annotated multimodal benchmark spanning 10 corpora, 26,000 pages, and 3,099 multilingual queries, and evaluates retrieval, generation, and grounding. The authors find that visual retrievers, late interaction, textual reranking, and visual contexts improve relevant pipeline outcomes, while current systems remain limited on difficult queries and fine-grained grounding.

Problem
Existing benchmarks only partially evaluate visually rich, large-corpus RAG because they often isolate retrieval, use extractive tasks, or lack multilingual and fine-grained grounding coverage.
Method
ViDoRe V3 combines human-in-the-loop annotation, VLM filtering, expert verification, and a two-axis taxonomy covering 7 query types and 3 query formats.
Results
Visual retrievers outperform textual ones, while late interaction and textual reranking provide substantial gains and visual context improves answer generation quality.
Takeaways & Limitations
ViDoRe V3 provides an end-to-end evaluation resource for multimodal RAG across retrieval, answer generation, and visual grounding.
Takeaways & Limitations
The benchmark uses English and French source documents and queries in 6 high-resource Western European languages, limiting its language coverage.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) pipelines must address challenges beyond simple single-document retrieval, such as interpreting visual elements (tables, charts, images), synthesizing information across documents, and providing accurate source grounding. Existing benchmarks fail to capture this complexity, often focusing on textual data, single-document comprehension, or evaluating retrieval and generation in isolation. We introduce ViDoRe v3, a comprehensive multimodal RAG benchmark featuring multi-type queries over visually rich document corpora. It covers 10 datasets across diverse professional domains, comprising ~26,000 document pages paired with 3,099 human-verified queries, each available in 6 languages. Through 12,000 hours of human annotation effort, we provide high-quality annotations for retrieval relevance, bounding box localization, and verified reference answers. Our evaluation of state-of-the-art RAG pipelines reveals that visual retrievers outperform textual ones, late-interaction models and textual reranking substantially improve performance, and hybrid or purely visual contexts enhance answer generation quality. However, current models still struggle with non-textual elements, open-ended queries, and fine-grained visual grounding. To encourage progress in addressing these challenges, the benchmark is released under a commercially permissive license at https://hf.co/vidore.

1 Introduction

Practical RAG must handle visually encoded information, cross-document reasoning, and precise source grounding. ViDoRe V3 addresses gaps in existing benchmarks by targeting complex, end-to-end evaluation on visually rich corpora.

Motivation: Visual elements, open-ended synthesis, and source-location grounding create requirements that single-document textual retrieval benchmarks often overlook.These requirements include interpreting tables, charts, and images; comparing scattered information; and localizing supporting evidence with bounding boxes.
Limitations of existing benchmarks: Existing benchmarks only partially evaluate large-corpus complexity, generation quality, grounding, complex reasoning, multilingual diversity, and fine-grained visual grounding.Some focus on single-page comprehension or retrieval alone, while others use short extractive answers.
ViDoRe V3: ViDoRe V3 is designed for complex and realistic end-to-end RAG evaluation on visually rich document corpora.The benchmark is introduced specifically to address the limitations identified in existing evaluations.

1. A Human Annotation Methodology for Realistic Queries

The benchmark uses human-in-the-loop annotation to produce diverse queries and fine-grained query-page annotations while reducing bias toward simple extractive tasks.

Annotation protocol: Annotators formulate queries without access to document content, capturing authentic search behavior and mitigating bias toward simple extractive queries.VLM filtering and human expert verification then support efficient, high-quality annotation at scale.

2. The ViDoRe V3 Benchmark

ViDoRe V3 is a multilingual benchmark built from industry-relevant document corpora, with large-scale query and page coverage and a commercially permissive release.

Benchmark construction: 10 industry-relevant document corpora yield 26,000 pages and 3,099 queries, each available in 6 languages.Two datasets are held out as a private test set to mitigate overfitting.

3. Comprehensive Evaluation and Insights

The evaluation measures retrieval, answer quality, and visual grounding across state-of-the-art multimodal RAG pipelines. Its granular annotations support analysis across modality, language, and pipeline configuration.

Evaluation scope: The benchmark evaluates retrieval accuracy by modality and language, answer quality across pipeline configurations, and visual grounding fidelity.These three axes correspond to retrieval, generation, and grounding in multimodal RAG.

2 Related Work

Existing visual-document benchmarks evaluate only parts of multimodal RAG, leaving gaps in multi-page retrieval, answer generation, grounding, and complex query reasoning.

Single-page VDU benchmarks overlook the multi-page context required by RAG.
Retrieval-centric visual benchmarks evaluate retrieval but do not jointly assess answer generation.
Multimodal datasets often use extractive short-answer tasks that do not exercise complex reasoning.
Existing multimodal evaluations also remain limited in multilingual diversity and fine-grained visual grounding.

3 Benchmark Creation

ViDoRe V3 is built through a human-in-the-loop process that combines diverse query generation, multimodal corpus coverage, VLM filtering, human verification, and grounded answer annotation. The resulting benchmark supports multilingual, multi-type evaluation across realistic document domains, while its annotations expose challenges in query difficulty and visual grounding.

Benchmark Creation: ViDoRe V3 uses a three-stage human-in-the-loop process covering document collection, query generation, and grounded query answering.The design targets pipeline-agnostic evaluation of retrieval, generation, and grounding.
Document Collection: The benchmark comprises 10 industry-relevant corpora spanning governmental, educational, and industry documents across multiple professional domains.The source collection focuses on English and French documents and includes domain-specific terminology and structures.
Query Generation: Its query taxonomy crosses 7 query types with 3 formats, separating users’ information needs from syntactic structure.Query types include open-ended, extractive, numerical, multi-hop, compare-contrast, boolean, and enumerative; formats include question, keyword, and instruction.
Query Generation: Queries are generated from local sections or cross-section summaries, with summaries incorporating extracted text and image descriptions across text, image, and table modalities.Synthetic and human query-writing procedures vary query type, format, length, and difficulty to promote diversity.
Annotation: VLM pre-filtering reduces candidate pages before human review, while multiple annotators and supervisors verify relevance, answers, and visual evidence.Annotators generate answers and delineate supporting bounding boxes labeled as Text, Table, Chart, Infographic, Image, Mixed, or Other.
Annotation: Average human bounding-box agreement is IoU 0.50 and F1 0.60 across all 10 datasets, reflecting subjectivity in annotation granularity.Annotators generally agreed on relevant content but differed in how tightly they bounded it.
Distribution: The final benchmark includes multilingual evaluation across 6 languages, with text prevalent but tables, charts, and infographics well represented.Extractive queries predominate, open-ended queries follow, and multi-hop queries were hardest to scale.

4 Experiments and Results

ViDoRe V3 evaluates retrieval, reranking, answer generation, and visual grounding across complex multimodal RAG settings. Visual retrieval generally leads at retrieval, textual reranking delivers the largest reranking gain, and hybrid or visual contexts improve difficult answer generation, while grounding and challenging queries remain difficult.

4.1 Retrieval: Visual retrievers outperform textual retrievers at matched parameter counts, while late-interaction methods score higher than dense methods.
4.1 Retrieval: Retrieval performance declines with query complexity, with Boolean and Numerical queries outperforming Open-ended and Multi-hop queries; Question formats generally outperform Instruction and Keyword formats.
4.1 Retrieval: Queries involving visual content are harder, Mixed content scores lowest, and performance declines as the number of annotated pages increases.The page-count relationship may be confounded because more complex query types tend to require more annotated pages.
4.1 Retrieval: Cross-lingual retrieval trails monolingual retrieval by 2–3 points.The reported comparison indicates that models need better adaptation to cross-lingual settings.
4.1 Retrieval: Textual reranking adds 13.2 NDCG@10 points and produces the highest overall retrieval performance, whereas visual reranking improves averages by only 0.2 points and degrades four datasets.
4.2 Final Answer Generation: Hybrid retrieval reaches 54.7% accuracy on hard queries, exceeding the strongest textual baseline at 52.1% and visual baseline at 54.5%.With Gemini 3 Pro fixed, image context also beats text context by 2.4 points for oracle retrieval and 2.8 points for ColEmbed-3B-v2.
4.2 Final Answer Generation: Even with oracle context, hard-query performance remains more than 10 percentage points below easy-query performance, exposing limits in multi-step reasoning and long-context synthesis.Model rankings can also change with query difficulty: GPT 5.2 leads Gemini 3 Pro on easy queries but trails it on hard queries.
Visual grounding lags human performance: Visual grounding remains far below human agreement: annotator F1 is 0.602, compared with 0.089 for Qwen3-VL-30B-A3B-Instruct and 0.065 for Gemini 3 Pro.On human-annotated pages, models annotate the same page only 16–17% of the time, while 26–27% receive no model annotation.

5 Conclusion

ViDoRe V3 is a human-annotated benchmark for cross-lingual retrieval, answer generation, and visual grounding on large industry-relevant document corpora. Its evaluations identify strong multimodal RAG capabilities alongside persistent challenges in visual interpretation, multi-page generation, and grounding.

ViDoRe V3 evaluates cross-lingual retrieval, final answer generation, and visual grounding on large industry-relevant document corpora.The benchmark uses human-annotated queries, relevant pages, bounding boxes, and reference answers.
Visual retrievers outperform textual ones, while late interaction and textual reranking yield substantial retrieval gains.The conclusion summarizes these as practical findings from state-of-the-art pipeline evaluations.
Visual context improves answer generation quality, but models still struggle with cross-lingual and open-ended queries requiring visual interpretation.The remaining difficulties extend to multi-page answer generation and accurate visual grounding.
ViDoRe V3 provides a framework for evaluating these limitations and supporting development of more robust document understanding models.

Limitations

The benchmark’s coverage is limited by its language selection, document distribution, and the subjectivity of annotations for complex reasoning and visual grounding.

The benchmark uses English and French source documents and queries in 6 high-resource Western European languages.Future iterations are intended to include more language families and non-Latin scripts.
Its publicly available long-form corpora do not represent private enterprise repositories or noisy short-form documents such as emails and support tickets.Scanned handwritten notes are also outside the represented source-document distribution.
Open-ended reasoning and visual-grounding annotations are inherently subjective, so valid retrieval paths and answer formulations may fall outside the annotated ground truths.

Ethical considerations

The benchmark addresses ethical and environmental considerations through controlled annotation practices, privacy-aware data selection, permissive licensing, and reported computational impacts.

Annotation was conducted by the benchmark creators and one external vendor selected after evaluation against protocol and ethical criteria.Annotators came from the vendor’s existing workforce and were recruited according to stated demographic requirements.
Documents were manually selected from open-license governmental, educational, and enterprise websites, and annotations were collected without private or personally identifiable information.The annotations are described as GDPR-compliant.
The benchmark uses a commercially permissive license to support broad research adoption while respecting original document creators’ intellectual property rights.
The benchmark is restricted to English and French sources and 6 high-resource Western European languages, which may favor systems optimized for those languages.The authors encourage evaluation using underrepresented language families and non-Latin scripts.
Creation required approximately 12,000 human-annotation hours, 3,000 NVIDIA H100 GPU hours, and an estimated 200 kg CO2e environmental impact.

Detailed Contributions

ViDoRe V3 combines a broad multimodal benchmark, human-centered annotation, and detailed evaluations across retrieval, generation, grounding, and query complexity. The analyses expose performance patterns and methodological considerations relevant to realistic RAG systems.

Benchmark scope: Figure 8 illustrates varied query types and visually rich document formats across multiple domains in ViDoRe V3.
Annotation: The annotation process uses trained annotators, pilots, quality control, and VLM pre-filtering to produce relevance, grounding, and answer annotations at scale.Annotator selection required relevant education, domain experience, language proficiency, and prior annotation experience.
Annotation: Pre-filtering creates prevalence imbalance, so the study reports both Krippendorff’s Alpha and Gwet’s AC2; overall scores are α = 0.469 and AC2 = 0.760.
Retrieval evaluation: Hybrid retrieval merges top-5 visual and textual results, and consistently outperforms single-modality baselines using unranked F1.
Performance analysis: Performance declines as annotated pages increase, with nearly uniform slopes of a ≈−0.024 across query types; open-ended and enumerative queries have the shallowest slopes.Their low performance at small page counts suggests difficulty synthesizing multiple relevant sources beyond page volume alone.

I Final answer evaluation

The final-answer evaluation uses an LLM judge in a pass@1 setup and examines both query difficulty and end-to-end pipeline stability. Results indicate robust evaluation signals alongside distinct visual-grounding failure modes and differences in box granularity.

Evaluation setup: Generated answers are evaluated in pass@1 by GPT 5.2, which compares each answer with the ground truth and assigns binary correctness.The evaluation includes Gemini 3 Pro, GPT-5, and Qwen3-VL-235B-A22 under specified reasoning settings.
End-to-End Pipeline Stability: 65.74 % average score with 0.94 % standard deviation and Krippendorff’s α of 0.80 indicate stable end-to-end evaluation across five independent runs.The stress test covered Industrial Maintenance in English and Finance in French, selected as the most challenging corpus in each language.
Easy/hard query filtering: 48.6 % of queries are classified as easy because at least one of six context-free LLMs answers them correctly.The remaining queries are labeled hard, and the panel spans multiple model families and capability levels.
Qualitative analysis: Gemini often makes off-by-one page-indexing errors, while Qwen3-VL produces larger boxes that more closely match human annotation patterns.Gemini tends to localize individual elements such as table cells or text lines; Qwen3-VL more often covers sections or paragraphs.
Annotation and evaluation prompts: Page-level annotation evaluates answerability relevance, content modalities, and evidence localization through bounding boxes.Annotators rate how relevant each page is to answering the query using the defined relevance criteria.
Annotation and evaluation prompts: The evaluation materials include prompts for query generation, page pre-filtering, answer merging, difficulty filtering, judging, answer generation, translation, and bounding-box prediction.These prompts are documented in Figures 20–27, while the judge criteria distinguish correct, partially correct, and incorrect answers.