PaperlayerView exact revision on arXiv

Source-linked AI summary

MIEB: Massive Image Embedding Benchmark

Chenghao Xiao, Isaac Chung, Imene Kerboua, Jamie Stirling, Xin Zhang, Márton Kardos, Roman Solomatin, Noura Al Moubayed, Kenneth Enevoldsen, Niklas Muennighoff

arXiv:2504.10471v1cs.CVcs.CL

TL;DR

Image and multimodal embedding models lack standardized evaluation across their diverse capabilities. MIEB addresses this gap with a broad benchmark spanning 38 languages and 130 tasks, and finds fragmented performance across categories alongside strong correspondence between MIEB vision-encoder results and MLLM performance.

  • Problem

    Image and multimodal embedding models are evaluated with divergent task-specific protocols, leaving no standardized criteria for comparing image representations across capabilities.

    Source: §1 · p. 1

  • Method

    MIEB benchmarks image and image-text embeddings across 130 tasks in 38 languages, grouped into 8 categories and evaluated across 50 models.

    Source: §1 · p. 1, §8 · p. 8

  • Results

    No single method dominates all task categories, while Visual STS performance has over 99% correlation with corresponding MLLM performance on OCRBench and TextVQA.

    Source: §1 · pp. 1–2, §5 · p. 4

  • Takeaways & Limitations

    MIEB supports broader evaluation beyond classification and retrieval and provides a practical basis for selecting vision encoders for multimodal generative models.

    Source: §1 · p. 2, §8 · p. 8

    The benchmark emphasizes zero-shot evaluation, with linear probing as the special case where frozen embeddings support training a linear classifier.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Image representations are often evaluated through disjointed, task-specific protocols, leading to a fragmented understanding of model capabilities. For instance, it is unclear whether an image embedding model adept at clustering images is equally good at retrieving relevant images given a piece of text. We introduce the Massive Image Embedding Benchmark (MIEB) to evaluate the performance of image and image-text embedding models across the broadest spectrum to date. MIEB spans 38 languages across 130 individual tasks, which we group into 8 high-level categories. We benchmark 50 models across our benchmark, finding that no single method dominates across all task categories. We reveal hidden capabilities in advanced vision models such as their accurate visual representation of texts, and their yet limited capabilities in interleaved encodings and matching images and texts in the presence of confounders. We also show that the performance of vision encoders on MIEB correlates highly with their performance when used in multimodal large language models. Our code, dataset, and leaderboard are publicly available at https://github.com/embeddings-benchmark/mteb.

1. Introduction

MIEB addresses fragmented evaluation protocols by offering a unified benchmark for image and image-text embeddings across diverse capabilities. Its analysis finds fragmented model performance, while Visual STS performance strongly predicts corresponding MLLM performance.

Motivation

  • Divergent protocols spanning retrieval, classification, probing, fine-tuning, and MLLM proxies lack standardized criteria for assessing image representations.This fragmentation makes comparisons across capabilities difficult.Source: §1 · p. 1

Motivation and contribution

  • MIEB unifies evaluation of image and image-text embeddings across 130 tasks grouped into 8 categories, extending MTEB with fine-grained capabilities.The benchmark includes interleaved, multilingual, instruction-aware, compositionality, visual question answering, Visual STS, and document-understanding tasks.Source: §1 · p. 1

Findings

  • No method dominates all task categories, showing that current image embedding capabilities remain fragmented across the benchmark.The overall analysis compares performance across task categories rather than identifying a universal winner.Source: §1 · p. 1
  • Over 99% correlation links Visual STS performance with MLLM performance on OCRBench and TextVQA when both use the same vision encoder.This suggests MIEB results can help select vision encoders for MLLMs.Source: §1 · pp. 1–2

2. The MIEB Benchmark

MIEB evaluates image embeddings through diverse protocols and capabilities, emphasizing zero-shot, multimodal, and under-explored visual understanding tasks. It also provides MIEB-lite for more efficient evaluation.

Overview

  • MIEB organizes diverse embedding abilities into a unified framework spanning traditional tasks and under-explored capabilities such as document understanding and visual question answering.The taxonomy combines evaluation protocols with the abilities assessed.Source: §2.1 · pp. 2–3, Table 1 · p. 3
  • MIEB-lite selects 51 representative tasks from performance clusters to support efficient evaluation alongside the full 130-task benchmark.The lightweight version is intended to reduce evaluation cost while retaining representative coverage.Source: §2.1 · p. 2

Retrieval

  • Retrieval evaluates image-text and image-image similarity across modality combinations, multilinguality, and task instructions using nDCG@10 as the primary metric.Some tasks additionally use recall@1 or map@5.Source: §2.1 · p. 2

Specialized capabilities

  • Compositionality tests whether object, attribute, and spatial relations jointly align across images and texts, while vision-centric QA covers counting, spatial relations, and art perception.Compositionality mainly uses accuracy, and vision-centric QA is implemented as retrieval over query-specific options.Source: §2.1 · p. 3

Visual STS

  • Visual STS renders semantic-textual-similarity examples as images and compares embedding similarities with human annotations using dataset-level Spearman correlation.The protocol exposes visual encoders’ ability to understand relative text semantics and reveals OCR behavior in models such as CLIP.Source: §2.1 · p. 3

Design considerations

  • The benchmark emphasizes zero-shot evaluation with frozen embeddings, including few-shot linear probing where only a classifier is trained on top of unchanged representations.This design isolates information encoded in the embeddings rather than task-specific fine-tuning.Source: §2.1 · p. 2, §2.2 · p. 3

3. Models

MIEB evaluates self-supervised, language-supervised, MLLM-based, and multimodal API embedding models, with overlapping categories reflecting distinct training and modeling strategies.

Model categories

  • The benchmark covers three overlapping model categories, including self-supervised vision models, language-supervised image-text models, and MLLM-based embedding models.The categories are treated as broad groupings rather than mutually exclusive classes.Source: §3 · p. 4, §3.1 · p. 4, §3.2 · p. 4

Self-supervised models

  • Self-supervised models such as MOCO-v3 and DINO-v2 learn visual representations without predicting a fixed set of predefined object categories.DINO-v2 scales self-supervised learning to 142M images with similarity-based curation.Source: §3.1 · p. 4

Language-supervised models

  • Language-supervised models including CLIP, SigLIP, ALIGN, Jina-CLIP, DataComp-CLIP, Open-CLIP, and Eva-CLIP train on text-image relationships.VISTA combines a ViT encoder with a pretrained language model before CLIP-style training.Source: §3.2 · p. 4

MLLM-based models

  • MLLM-based embedding models include E5-V and VLM2Vec, which use multimodal backbones and paired image-text data or text-only contrastive fine-tuning.E5-V uses pretrained MLLMs and shows generalization to images and interleaved encodings.Source: §3.3 · p. 4

API models

  • The benchmark also evaluates Voyage, a multimodal API embedding model optimized for image search and business applications involving figure and table understanding.These capabilities make it a candidate for visual-text understanding tasks.Source: §3.3 · p. 4

4. Implementation Details

For interleaved inputs, MIEB uses each model’s native mixed-modality encoding when available and otherwise constructs an embedding by summing text and image embeddings.

Interleaved inputs

  • Models capable of mixed-modality inputs are evaluated using their original interleaved-input implementation.This applies, for example, to MLLM-based embedding models.Source: §4 · p. 4
  • For models without native interleaved encoding, MIEB defaults to summing text and image embeddings to obtain a combined representation.The fallback is used for CLIP-style models.Source: §4 · p. 4

5. Experimental Results

MIEB results show fragmented model capabilities: MLLM-based models lead overall and in visual-text and multilingual tasks, while CLIP-style models remain stronger on traditional image representation tasks. Performance varies by task granularity and modality, with no universal model dominating all categories.

  • No universal embedding model achieves the best performance across all MIEB task categories.Table 2 reports category-level results for the top 20 models across MIEB's 130 tasks.Source: §5 · p. 4, Table 2 · p. 5
  • MLLM-based models lead overall, especially on visual text understanding and multilingual tasks, but trail CLIP-style models on linear probing and zero-shot classification.MLLM-based models also struggle with fine-grained classification, including bird-species identification.Source: §5 · p. 4
  • CLIP-style models excel on linear probing, zero-shot classification, and retrieval, whereas they struggle with interleaved retrieval, visual text representations, and multilingual tasks without targeted optimization.Scaling model size, batch size, and dataset quality improves clustering, classification, and retrieval, but not universally.Source: §5 · p. 4, §5.3 · p. 6

Clustering

  • Fine-grained image distinctions remain difficult for MLLM-based models, while CLIP-style models provide more separated dog-breed clusters and dominate many clustering tasks.EVA-CLIP, DataComp-CLIP, and OpenCLIP checkpoints dominate most clustering tasks; E5-V shows less separation on ImageNet Dog15.Source: §5.2 · pp. 4–5, Figure 2 · p. 6

Multilingual Retrieval and Visual STS

  • E5-V achieves state-of-the-art multilingual retrieval and Visual STS performance, while strong multilingual text encoders and OCR capabilities support these results.Its multilingual retrieval performance is linked to LLaVA-Next initialization and light LoRA contrastive fine-tuning; Visual STS nearly matches regular STS under strong OCR.Source: §5.5 · p. 6, §5.6 · p. 6, Table 3 · p. 6

Document Understanding

  • E5-V is the best open-source model for document understanding, reaching an average nDCG@5 of 62.69 on 10 ViDoRe tasks.Voyage-multimodal-3 performs better but is closed-source.Source: §5.7 · p. 6

Compositionality

  • Compositionality and retrieval have the lowest scores, with WinoGround especially challenging because of image and textual confounders.The authors hypothesize that stronger reasoning and test-time scaling may improve compositionality performance.Source: §5.1 · p. 4, §5.8 · p. 7

Vision-centric QA

  • BLIP models place two entries among the top five vision-centric QA models, suggesting that including images during contrastive fine-tuning can benefit performance.This contrasts with training that excludes images during contrastive fine-tuning.Source: §5.9 · p. 7

6. Discussions

The discussion examines efficient evaluation choices, links vision-encoder performance to MLLM performance, and develops MIEB-lite by removing redundant tasks while preserving diagnostic value.

Evaluation protocol

  • The benchmark uses 16-shot linear probing by default because varying k preserves model rankings while keeping evaluation cheaper.The ablation evaluates k in {8,16,32,64,128,256} and motivates the modest default setting.Source: §6.1 · p. 7

MLLM predictability

  • MLLM-based evaluation is informative but impractical as a general benchmark because it requires large models and broad hyperparameter searches.This motivates predicting MLLM performance from cheaper vision-encoder evaluations.Source: §6.2 · p. 8
  • Visual STS performance strongly correlates with MLLM performance across 16 benchmarks, especially OCR and chart tasks.The study asks whether MLLM performance can be predicted from vision-encoder results and finds the pattern most pronounced for four OCR and chart tasks.Source: Figure 4 · p. 7, §6.2 · p. 8

MIEB-lite

  • MIEB-lite reduces evaluation cost by iteratively removing redundant tasks while preserving task-category coverage and inter-task correlations.Tasks are filtered using pairwise correlations, then clustered with UMAP+HDBSCAN to support balanced selection.Source: §6.3 · p. 8
  • 82.4% runtime reduction for E5-V accompanies near-preserved model ordering, with Spearman correlation 0.992 and Pearson correlation 0.986 between MIEB and MIEB-lite.Table 5 compares runtimes, while the reported correlations use overall average performance across 38 models.Source: §6.3 · p. 8, Table 5 · p. 8

7. Related Work

Prior image-embedding benchmarks emphasize narrow task scopes and protocols, while MIEB broadens both the abilities assessed and the evaluation procedures.

Benchmarks

  • Existing benchmarks focus mainly on classification and retrieval, leaving multimodal composition, social-media understanding, and multilingual evaluation underrepresented.The cited limitations concern both task diversity and the breadth of abilities measured.Source: §7 · p. 8

Protocol limitations

  • Linear probing and retrieval provide limited evidence about generalization, whereas fine-tuning is costly and risks overfitting.These protocol limitations motivate evaluating frozen embeddings through a broader suite of procedures.Source: §7 · p. 8
  • MIEB extends evaluation with retrieval, linear probing, zero-shot classification, pair-wise classification, and clustering across more diverse capabilities.Its protocol suite is designed to assess frozen embeddings without relying only on narrow prior settings.Source: §7 · p. 8

8. Conclusion

MIEB provides a large-scale benchmark for image and image-text embeddings, combining broad task coverage with model baselines and multimodal relevance.

Conclusion

  • MIEB covers 8 task categories, 130 tasks, and 38 languages, and benchmarks 50 models to provide baselines for future research.The benchmark emphasizes evaluation beyond classification and retrieval and examines relevance to multimodal generative models.Source: §8 · p. 8

A. Tasks overview

The task overview documents MIEB’s datasets, metrics, and task organization, including retrieval, clustering, classification, and visual-text evaluation.

Task documentation

  • The appendix records task size, language, metrics, and related details, while directing readers to the main taxonomy for capability-based categories.The appendix presents categories according to Abstask implementations, whereas Table 1 provides the capability-based taxonomy.Source: A. Tasks overview · p. 15

Task organization

  • Tables 6–8 organize retrieval, clustering, classification, visual STS, text-based multiple choice, and image-text pair classification tasks.Table 6 covers retrieval; Table 7 covers clustering and classification variants; Table 8 covers visual STS and related matching tasks.Source: A. Tasks overview · p. 15

B. Per Task Category Results

This section organizes per-task-category results across clustering, classification, retrieval, multilingual and document-oriented evaluation settings. The reported tables cover both category-level results and distinct evaluation variants such as coarse- versus fine-grained classification and English, cross-lingual, and multilingual STS.

Per-task categories

  • Per-category reporting covers clustering, vision-centric QA, multilingual retrieval, visual STS, document understanding, linear probing, zero-shot classification, compositionality, and retrieval.The section references dedicated results for each of these task families.Source: §B.1 · p. 15, §B.2 · p. 15, §B.3 · p. 15

Visual STS

  • Visual STS results are separated into English-only, cross-lingual, and multilingual evaluations.These results span 7 English STS tasks, 11 language pairs, and 10 languages.Source: §B.4 · p. 15

Classification

  • Classification results distinguish linear probing from zero-shot evaluation and coarse-grained from fine-grained tasks.The corresponding results are reported in separate tables for each evaluation protocol and task granularity.Source: §B.6 · p. 15, §B.7 · p. 15

Compositionality and retrieval

  • The section reports separate results for compositionality and retrieval rather than folding them into a single evaluation category.These task families are each assigned dedicated result tables.Source: §B.8 · p. 15, §B.9 · p. 15

C. Overall Results & First MIEB Leaderboard

The overall leaderboard aggregates per-task-category results in Table 22, while the surrounding material documents evaluated models, datasets, task examples, and category-specific result tables. The leaderboard includes only models evaluable on all tasks, with per-category tables providing details for excluded models.

Overall ranking

  • The first MIEB overall leaderboard aggregates results across task categories and ranks models by average performance on all tasks, including multilingual ones.Table 22 reports both English-only and all-language averages grouped by assessed category.Source: C. Overall Results & First MIEB… · p. 15, Table 22 · p. 30
  • 36 models appear in the first overall leaderboard because models unable to evaluate every task are excluded.Vision-only models such as Dino-2 and Moco-v3 are omitted from the overall table because they cannot test image-text tasks.Source: C. Overall Results & First MIEB… · p. 15
  • Excluded models are instead covered through per-task-category tables, preserving access to results beyond the overall ranking.The paper directs readers to category-specific tables for models absent from the overall table.Source: C. Overall Results & First MIEB… · p. 15

Task examples

  • Examples illustrate retrieval, vision-centric, compositionality, visual STS, document understanding, zero-shot classification, linear probing, and clustering tasks.The figures show representative queries or task instances for these evaluation settings.Source: §E.1 · p. 32, Figure 5 · p. 32, §E.2 · p. 32
Loading 2504.10471v1…