PaperlayerView exact revision on arXiv

Source-linked AI summary

Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality

Xueguang Ma, Luyu Gao, Shengyao Zhuang, Jiaqi Samantha Zhan, Jamie Callan, Jimmy Lin

arXiv:2505.02466v1cs.IR

TL;DR

Retrieval research is expanding toward billion-scale models and multimodal tasks, while training memory, data organization, and inference efficiency remain challenges. Tevatron-v2 updates the toolkit with unified data handling and efficiency practices, and demonstrates retrieval across multilingual and multimodal settings. Its reported results include competitive effectiveness across multiple retrieval dimensions and cross-modal zero-shot effectiveness for text-trained variants.

  • Problem

    Billion-scale retrieval and multimodal retrieval are promising directions, but training them demands substantial GPU memory, existing text formats often poorly fit multimodal data, and deployment requires efficient inference.

    Source: §1 · p. 1

  • Method

    Tevatron-v2 updates the toolkit with unified multimodal data organization and LLM training and inference practices, and uses its datasets to train a unified dense retriever.

    Source: §1 · pp. 1–2

  • Results

    Tevatron’s unified retriever achieves competitive effectiveness across English multi-task, multilingual, and multimodal retrieval, while text-trained BGE variants show strong cross-modal zero-shot effectiveness on ViDoRe.

    Source: §3.2 · p. 4

  • Takeaways & Limitations

    The toolkit demonstrates a unified retriever’s versatility across multilingual text and image-document retrieval and is presented as a resource for future research.

    Source: §4 · p. 4

    When a query or document contains multiple text or image pieces, the format assumes they can be merged into a single representation.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Recent advancements in large language models (LLMs) have driven interest in billion-scale retrieval models with strong generalization across retrieval tasks and languages. Additionally, progress in large vision-language models has created new opportunities for multimodal retrieval. In response, we have updated the Tevatron toolkit, introducing a unified pipeline that enables researchers to explore retriever models at different scales, across multiple languages, and with various modalities. This demo paper highlights the toolkit's key features, bridging academia and industry by supporting efficient training, inference, and evaluation of neural retrievers. We showcase a unified dense retriever achieving strong multilingual and multimodal effectiveness, and conduct a cross-modality zero-shot study to demonstrate its research potential. Alongside, we release OmniEmbed, to the best of our knowledge, the first embedding model that unifies text, image document, video, and audio retrieval, serving as a baseline for future research.

1 Introduction

Billion-scale retrievers and multimodal models have opened new directions for retrieval, but researchers face barriers in training, data organization, and deployment efficiency. Tevatron-v2 updates the toolkit to address these challenges and support unified multilingual and multimodal retrieval research.

1 Introduction

  • Billion-scale retrievers generalize more strongly across tasks and languages than BERT-era counterparts, while vision-language models create new opportunities for multimodal retrieval.The paper frames unified retrieval across tasks, languages, and modalities as a promising direction.Source: §1 · p. 1
  • Researchers face GPU-memory demands for billion-scale training, data formats ill-suited to multimodal retrieval, and demand for more efficient inference in deployment.Source: §1 · p. 1
  • Tevatron-v2 updates the toolkit with unified multimodal data organization and LLM training and inference practices intended to improve memory and computational efficiency.It also converts open-source datasets to a unified format hosted in the Tevatron Hugging Face collection for ongoing maintenance.Source: §1 · p. 1
  • The toolkit supports training a unified dense retriever for multilingual and multimodal retrieval and analysis of cross-modality generalization.Source: §1 · p. 2

2 Tevatron-v2 Overview

Tevatron-v2 combines multimodal data management, memory-efficient training, faster inference, and an extensible framework for retrieval research. Its design supports varied data and modality combinations while offering options to manage compute and storage demands.

2.1 Unified Data Management

  • The unified data format decouples queries from corpus documents, storing candidate IDs and loading raw content dynamically to reduce storage requirements.It supports text-only, image-only, and mixed-modality data; when a query or document has multiple text or image pieces, they are assumed mergeable into one representation.Source: §2.1 · p. 2

2.2 GPU Memory Efficiency

  • 22 GB per GPU with CPU offloading (zero3-off) comes with a 44-hour training time; LoRA, ZeRO, and FlashAttention also make single-GPU fine-tuning feasible.Full fine-tuning without optimization runs out of memory; single-GPU LoRA, zero3, and FlashAttention use 69,324 MiB, with longer training time.Source: §2.2 · p. 2, Table 1 · p. 2

2.3 Inference Efficiency

  • vLLM encodes text and document images around 3 times faster than Transformers in the reported settings.The comparison uses an Llama3.1-8B retriever for text and a QWen2-VL-2B retriever with 784×784 image inputs for Wiki-SS.Source: §2.3 · p. 3, Figure 1 · p. 3
  • Matryoshka Representation Learning enables nested embeddings that can be truncated to smaller dimensions without significant effectiveness degradation.Users can specify target dimensions during training and adjust text embedding dimensionality at inference to meet storage or compute constraints.Source: §2.3 · p. 3

2.4 Comparison with Other Toolkits

  • Tevatron offers an extensible framework with modular data processing, training, and evaluation components for exploring new training paradigms, languages, and modalities.Source: §2.4 · p. 3

3 Experiment: Unified Dense Retriever

The experiment uses Tevatron-v2 to train unified retrievers on text and visual data, with video and audio added for Omni, and evaluates multilingual, multimodal, and zero-shot retrieval.

3.1.1 Training data. •

  • Training data include BGE-Training Data [1], WikiSS [14], PixMo-Docs [3], ColPali [4], MSR-VTT, and AudioCaps.The listed dataset sizes include 1.84 million BGE examples, 29.3k WikiSS samples, 1.75 million processed PixMo-Docs samples, and 127k ColPali query-image pairs; PixMo-Docs was filtered and hard negatives were mined.Source: §3.1.1 · pp. 3–4

3.2 Results

  • Tevatron-VL and Tevatron-Omni achieve competitive effectiveness across English multi-task, multilingual, and multimodal retrieval; Omni also supports video and audio embeddings.Table 2 compares multilingual and multimodal retrieval results; Table 3 covers Recall@1 on video and audio tasks.Source: Table 2 · p. 4, §3.2 · p. 4, Table 3 · p. 4

3.1.2 Hyperparameters.

  • Tevatron-VL uses Qwen2.5-VL-3B-Instruct for one epoch; Tevatron-Omni uses Qwen2.5Omni-7B [25] and adds video and audio data.Training uses eight NVIDIA H100 GPUs, batch size 128 queries, one positive and three hard negatives per query, and LoRA for memory efficiency.Source: §3.1.2 · p. 4

3.1.3 Evaluation.

  • Evaluation spans BEIR (13 English datasets) and MIRACL (18 languages), scored with nDCG@10, plus ViDoRe [4] table, chart, and screenshot retrieval at nDCG@5.Source: §3.1.3 · p. 4

3.1.4 Model Variants.

  • Tevatron-WikiSS trains only on Wikipedia screenshot retrieval data for in-modality zero-shot evaluation; Tevatron-BGE trains only on text retrieval data for cross-modality zero-shot evaluation.Both variants are evaluated on ViDoRe.Source: §3.1.4 · p. 4

3.2 Results

  • Text-only-trained Tevatron-BGE shows strong ViDoRe cross-modal zero-shot effectiveness, exceeding WikiSS in-modality zero-shot; the authors say relevancy fine-tuning is more critical than modality alignment.They suggest diverse text-only retrieval training may yield effective multimodal retrieval without multimodal training data.Source: §3.2 · p. 4

4 Conclusion

Tevatron-v2 is a unified, efficient toolkit for large-scale, multimodal, and multilingual retrieval research. Its demonstration trains a unified dense retriever for multilingual text and image-document retrieval.

  • Tevatron-v2 is a unified and efficient toolkit for research on large-scale, multimodal, and multilingual retrieval.Source: §4 · p. 4
  • The paper demonstrates a unified dense retriever that handles multilingual text and image-document retrieval.Source: §4 · p. 4
  • Tevatron-v2 is presented as a valuable tool for academia and industry, and is fully open-sourced.Source: §4 · p. 4

Related papers

Loading 2505.02466v1…