Source-linked AI summary
ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval
Quentin Macé, António Loison, Manuel Faysse
TL;DR
ViDoRe Benchmark V2 addresses the saturation of ViDoRe V1 by introducing more realistic and challenging visual retrieval scenarios. It combines blind contextual, long and cross-document, and human-refined synthetic querying across diverse multilingual datasets, with results showing room for improvement and differences in generalization and multilingual capability.
Problem
ViDoRe V1 was approaching saturation, with top models exceeding 90 nDCG@5 and limited room to measure improvements in realistic retrieval scenarios.
Method
The benchmark uses blind contextual querying, long and cross-document queries, and hybrid synthetic query generation with extensive human refinement.
Results
ViDoRe v2 leaves substantial room for future improvements and reveals differences in model generalization, multilingual capability, scale, and computational cost.
Takeaways & Limitations
ViDoRe v2 provides a more discriminating evaluation resource for challenging visual retrieval models across realistic and multilingual settings.
Takeaways & Limitations
The evaluation procedure resized input images to a maximum height of 1200 pixels, producing slightly lower ViDoRe v1 performance than voyageAI reported.
Abstract
from arXiv · showhide
The ViDoRe Benchmark V1 was approaching saturation with top models exceeding 90% nDCG@5, limiting its ability to discern improvements. ViDoRe Benchmark V2 introduces realistic, challenging retrieval scenarios via blind contextual querying, long and cross-document queries, and a hybrid synthetic and human-in-the-loop query generation process. It comprises four diverse, multilingual datasets and provides clear evaluation instructions. Initial results demonstrate substantial room for advancement and highlight insights on model generalization and multilingual capability. This benchmark is designed as a living resource, inviting community contributions to maintain relevance through future evaluations.
Why a new benchmark?
ViDoRe Benchmark V2 was introduced because ViDoRe V1 was nearing saturation, limiting its ability to measure improvements and distinguish advanced visual retrieval models.
- Current state-of-the-art models surpass 90 nDCG@5, while some ViDoRe V1 tasks have become too easy to provide meaningful signal.The original ColPali model reported 81.3 nDCG@5, whereas current leaderboard models exceed 90.
- The benchmark’s saturation leaves limited room to measure improvements and understand model capabilities in realistic retrieval scenarios.The paper motivates a new benchmark specifically to challenge advanced visual retrieval models.
1 Motivating the Creation of ViDoRe Benchmark V2
ViDoRe Benchmark V2 addresses limitations in existing benchmarks that make them poor representations of realistic retrieval behavior, especially for contextual, cross-document, and synthetic-query settings.
- The benchmark aims to be difficult, diverse, meaningful, and reflective of real-world retrieval challenges.
- Existing benchmarks often use extractive queries, although real users rarely formulate queries from exact phrases in documents.
- Many benchmarks emphasize single-page retrieval and neglect complex multi-document or cross-document queries common in real-world applications.
- Purely synthetic query generation can produce irrelevant, trivial, or outlier queries, making costly human filtering necessary for quality control.
2 Design Decisions and Techniques Used
ViDoRe Benchmark V2 combines blind contextual querying, long and cross-document queries, and human refinement of synthetic queries to create more realistic and reliable retrieval evaluations.
- These features are presented as innovations for addressing the limitations of existing retrieval benchmarks.
- Blind contextual querying gives annotator models limited document information and filters irrelevant outputs to reduce extractive bias.Inputs may include summaries or metadata rather than full document content, better reproducing users’ limited corpus knowledge.
- Long-form and cross-document queries target retrieval situations involving comprehensive documents or multiple documents.
- A hybrid process generates queries synthetically and extensively refines them through human review to improve query quality and dataset reliability.The paper characterizes this human-in-the-loop process as intensive.
3 Dataset Selection for ViDoRe Benchmark V2
ViDoRe Benchmark V2 selects diverse, publicly available datasets with distinct visual complexity for realistic retrieval tasks, including multilingual versions with translated queries.
- The selected datasets are diverse, publicly available, challenging, and suited to realistic retrieval tasks.
- Each dataset has a multilingual version with queries translated into French, English, Spanish, and German.
- The multilingual design extends the benchmark’s applicability and challenge level.
4 Evaluating Models
ViDoRe Benchmark V2 can be evaluated through a CLI or custom retriever workflow, using specified model, collection, format, and split settings. The release also records dataset and evaluation caveats.
- Evaluation options: The evaluation offers two options: using the CLI or creating a custom retriever.Detailed custom-retriever instructions are provided in the benchmark repository.
- CLI evaluation: The CLI example evaluates a ColPali-type retriever on ViDoRe Benchmark V2.The example uses the colpali model class and the vidore/colpali-v1.3 model.
- Dataset caveat: The insurance dataset was removed after release for legal copyright reasons.This is a dataset availability caveat rather than an evaluation command setting.
- CLI evaluation: The CLI configuration uses BEIR format and the test split.These settings appear alongside the model and collection configuration.
- Evaluation options: The benchmark planned to transition evaluation of all models to the MTEB library.The current instructions refer users to the repository for other retrievers.
5 Results
Results show that ViDoRe V2 preserves broad model-ordering consistency with V1 while providing more room to distinguish models, including across multilingual capability, scale, and dataset quality. The evaluation also reveals computational trade-offs and weaker generalization for some models and smaller systems.
- Tabulated results: Evaluation results are reported as nDCG@5 across datasets, with the highest value in each column shown in bold.Table 2 organizes model performance by dataset.
- Overall comparison: ViDoRe V2 maintains strong correlation with V1 and consistent model rankings, while V1 shows saturation for top models.Figure 1 reports results using nDCG@5.
- Generalization: Certain models show signs of overfitting to the training distribution and reduced generalization to novel data.Examples include vidore/colSmol-256M, vidore/colSmol-500M, and Metric-AI/ColQwen2.5-3b-multilingual-v1.0.
- Evaluation caveat: 27 The adapted voyageAI evaluation produced slightly lower ViDoRe V1 performance than the values reported by voyageAI.The discrepancy likely results from resizing input images to a maximum height of 1200 pixels.
- Multilingual capability: Multilingual splits provide a more accurate assessment of nonenglish visual-retriever capabilities and reveal a significant gap between English-only and non-English-trained systems.The comparison concerns models trained exclusively in English using an English-only VLM versus models that are not.
- Model scale: Larger model scale is beneficial, but gme-qwen7B incurs significant computational cost and inference latency.Models under 1B parameters tend to lag, especially on previously unseen data distributions.
- Dataset quality: The human-labeled ESG human dataset separates model performances better than the synthetic datasets.This indicates slightly higher dataset quality and a more discriminating signal.
- Overall comparison: ViDoRe V2 leaves substantial room for future improvements, unlike V1, which was approaching performance saturation.The monolingual-task comparison contrasts the two benchmark versions directly.
6 Moving Forward
ViDoRe V2 is intended to remain a dynamic benchmark that expands with new tasks, datasets, metrics, multilingual coverage, and modalities.
- The benchmark is designed to grow through community contributions of new datasets and evaluation tasks.
- Future extensions may add confidence-estimation metrics, broader multilingual coverage, and new modalities such as audio and image querying.