PaperlayerView exact revision on arXiv

Source-linked AI summary

MVEB: Massive Video Embedding Benchmark

Adnan El Assadi, Roman Solomatin, Isaac Chung, Chenghao Xiao, Deep Shah, Manan Dey, Shriya Sudhakar, Zacharie Bugaud, Wissam Siblini, Ayush Sunil Munot, Yashwanth Devavarapu, Rakshitha Ireddi, Michelle Yang, Márton Kardos, Niklas Muennighoff, Kenneth Enevoldsen

arXiv:2606.14958v1cs.CVcs.IRcs.LG

TL;DR

Existing video benchmarks isolate capabilities, obscuring general-purpose representation quality. MVEB evaluates video embeddings across 23 diverse tasks and finds that no single model dominates, while audio helps or hurts depending on annotation provenance.

  • Problem

    Existing video benchmarks typically test one capability, making general-purpose video representation quality difficult to assess across tasks.

  • Method

    MVEB constructs a unified 23-task benchmark from a 184-task pool, evaluates 33 models, and pairs video-only with audio+video evaluations.

  • Results

    No single model dominates: MLLM embeddings lead classification, clustering, pair classification, and QA, while multimodal binding leads retrieval and zero-shot classification; audio shows a six-point provenance-dependent gap.

  • Takeaways & Limitations

    Model strengths are task-dependent, and audio should be evaluated against dataset annotation provenance rather than assumed to improve video understanding uniformly.

  • Takeaways & Limitations

    Undisclosed training data make some MVEB scores uncertifiable as zero-shot because contamination audits are incomplete.

Abstract

from arXiv · show

We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering. We evaluate 33 models and find that no single model dominates: MLLM-based embeddings lead on classification, clustering, pair classification, and QA; multimodal binding leads on retrieval and zero-shot classification; generative MLLMs without contrastive adaptation collapse on cross-modal tasks. Paired video-only vs. audio+video evaluations show that audio's contribution depends on dataset annotation provenance: audio helps when labels were produced from both modalities and hurts when they were produced from visuals alone, a six-point gap consistent across model families. MVEB is derived from MVEB+, a 184-task pool, and is designed to maintain task diversity while reducing evaluation cost. It integrates into the MTEB ecosystem for unified evaluation across text, image, audio, and video. We release MVEB and all 184 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.

1 Introduction

MVEB addresses fragmented video-embedding evaluation with a zero-shot, 23-task benchmark spanning six task families and multiple modalities. Across 33 models, results show specialization across model paradigms, modality-dependent audio effects, and the importance of contrastive adaptation for cross-modal performance.

  • Motivation: Existing video benchmarks often test one capability, obscuring general-purpose representation quality and producing task-specific strengths that do not transfer across capabilities.The introduction contrasts action-recognition performance with retrieval performance to motivate unified evaluation.
  • Ecosystem integration: MVEB integrates into the MTEB ecosystem with task and model versioning to support unified evaluation, community contribution, and long-term maintenance.The benchmark is presented as the video sibling of related multilingual, image, and audio embedding benchmarks.
  • Benchmark design: MVEB is a zero-shot 23-task benchmark curated from the 184-task MVEB+ pool, covering classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering.MVEB also provides modality-restricted leaderboards from the same pool for models lacking full audio-video-text input support.
  • Findings: Audio’s contribution depends on annotation provenance, with paired video-only and video+audio evaluations measuring when audio improves video understanding.MVEB systematically constructs both variants for audio-bearing datasets, unlike prior video benchmarks described in the introduction.
  • Findings: Across 33 models, MLLM-based embeddings lead classification, clustering, pair classification, and QA, while multimodal binding leads retrieval and zero-shot classification.No single evaluated model dominates all task families.
  • Findings: Contrastive embedding adaptation is a near-prerequisite for cross-modal performance, and training-data alignment matters more than backbone scale on cross-modal tasks.Generative MLLMs used as embedders without contrastive adaptation collapse on cross-modal tasks.

2 Related Work

MVEB completes the MTEB family for video, extending a benchmark ecosystem that spans multilingual, image, and audio representation learning. It addresses limitations of prior video benchmarks through broader coverage and MTEB-style community maintenance.

  • Embedding Benchmarks: MVEB completes the MTEB family alongside multilingual, image, and audio extensions that provide broad, regularly maintained leaderboards with consistent protocols.The related benchmarks are MTEB, MMTEB, MIEB, and MAEB.
  • Video Representation Benchmarks: Prior video benchmarks cover only parts of MVEB’s task surface: VideoEval is visual-only, UVRB is retrieval-only, and LoVR contains 467 videos.VideoEval evaluates 20 models across 12 tasks, while UVRB covers 16 datasets.
  • Video Representation Benchmarks: MVEB differs from MMEB-V3 by treating audio+video as joint input rather than audio as a separate modality limited to cross-modal retrieval directions.This distinction concerns how multimodal inputs and retrieval directions are evaluated.
  • Video Representation Benchmarks: MVEB inherits MTEB’s community-driven contribution and continuous-update model instead of shipping as a static leaderboard.The paper presents this maintenance approach as avoiding stagnation affecting most prior video benchmarks.

3 Benchmark Construction

MVEB extends MTEB to video embedding evaluation through a curated 23-task benchmark selected from a 184-task pool. The curation substantially reduces runtime while preserving model-score and ranking agreement, and supports modality-restricted evaluation through additional leaderboards.

  • Benchmark framework: MVEB extends the MTEB ecosystem to video, inheriting standardized metrics, a minimal task/model interface, versioned artifacts, and community-driven maintenance.It evaluates embedding quality rather than transcription or generation.
  • Dataset selection: Dataset curation targets domain, task, and modality diversity while prioritizing established usage, clear licensing, and public availability.Covered modalities include video-only, video+text, and video+audio task variants.
  • Task selection: 23 tasks form the curated MVEB leaderboard, selected from the 184-task MVEB+ pool because exhaustive evaluation is prohibitively expensive.Selection applies principled filtering criteria intended to retain meaningful evaluation coverage.
  • Curation efficiency: 7–10× faster runtime accompanies Pearson r=0.996 score correlation and Spearman ρ=0.944 ranking correlation between MVEB and MVEB+.These results indicate that curation preserves relative model performance at a fraction of the runtime.
  • Modality-specific leaderboards: MVEB(text, video) provides 19 tasks and MVEB(video) provides 9 tasks, enabling evaluation of models without audio encoders and video-only encoders.Both leaderboards are drawn from the same task pool as MVEB.
  • Benchmark ranking: Model ranks use a Borda count over tasks, with both Borda rank and mean reported because Borda is not a continuous measure.The ranking procedure treats each task as a preference voter over models.

4 Results

MVEB shows no universally dominant model: category leadership is split across MLLM-based embeddings and multimodal binding, while text-video specialists lead when audio is removed. Contrastive adaptation is crucial for generative MLLM embeddings, and smaller specialized models can outperform larger general-purpose systems.

  • Overall leaderboard: No single model dominates: LCO-Embedding-Omni-7B ranks first by Borda count with a 57.6 mean, but category leadership is distributed across competing models.LCO-Embedding-Omni-7B leads QA and clustering; BidirLM-Omni-2.5B-Embedding leads classification at 61.2, eBind leads retrieval and zero-shot classification, and LCO-Embedding-Omni-3B leads pair classification.
  • Modality-restricted results: Text-video specialists lead the 19-task MVEB(text, video) scope, with Qwen3-VL-Embedding-8B and -2B ranking first and second at 60.9 and 58.1 mean.The Qwen3-VL family wins 5 of 6 task categories; QA remains with LCO-Embedding-Omni-3B at 32.8.
  • Model paradigms: Contrastive adaptation is a near-prerequisite: e5-omni-7B scores 55.0 versus 12.8 for the unadapted Qwen2.5-Omni-7B generative baseline.Qwen2.5-Omni-7B and 3B score 12.8 and 11.4 mean, far below same-scale MLLM-embed counterparts.
  • Model scale and specialization: Smaller specialized contrastive models punch above their weight: eBind, at 1.8B / 764M, ties for third by Borda and wins retrieval at 62.3 and zero-shot classification at 61.1.Perception Encoder audio-visual variants of 847M–2.2B also outscore comparable-size MLLM-embed models on retrieval and clustering.
  • Category difficulty and interpretation: Clustering and zero-shot classification have the lowest absolute scores: the strongest model reaches 27.3 on clustering and no model exceeds 61.1 on zero-shot classification.Clustering scores should be interpreted relatively because V-measure reflects label granularity and dataset noise and is not directly comparable to accuracy-based categories.

5 Analyses

Audio’s contribution is predicted by annotation provenance: it helps AV-grounded datasets but harms visually grounded ones. Evaluation performance rises sharply with early increases in sampled frames, while retrieval directions cluster into three capability groups rather than eight independent measures.

  • Audio contribution: Audio improves AV-grounded datasets by +0.016 on average but harms V-grounded datasets by −0.046, a six-point gap consistent across task types and model families.AV-grounded labels use audio and visual content; V-grounded labels use visuals alone.
  • Audio contribution: Audio effects vary more within paradigms than across them, with per-paradigm averages spanning −0.023 to +0.010 and multimodal binding consistently losing −0.023.MLLM-based embedders range from −0.039 at e5-omni-3B to +0.023 at BidirLM-Omni-2.5B.
  • Frame sampling: 43.7% relative improvement comes from moving from one frame to 8, while doubling from 32 to 64 adds only 2.2% absolute, making 32 frames a reasonable ceiling.Mean performance scales logarithmically with the number of sampled frames, although task-level scaling varies substantially.
  • Retrieval correlations: Retrieval directions cluster into three capability groups: text-target, video-target with audio in the query, and text-as-query.Within-group correlations are ρ = 0.96 for text-target, ρ ≥0.87 for video-target with audio in the query, and ρ ≥0.77 for text-as-query; the most decoupled pair is ρ = 0.38.

6 Conclusion

MVEB unifies evaluation of video embedding models through a 23-task benchmark drawn from a 184-task pool, with modality-restricted leaderboards. Across 33 models, no single approach dominates, while audio effects vary with annotation provenance and the benchmark is designed to evolve within MTEB.

  • Benchmark contribution: MVEB provides a unified 23-task benchmark drawn from the 184-task MVEB+ pool, with modality-restricted leaderboards for text-video and video-only encoders.It evaluates 33 models spanning major video-embedding paradigms.
  • Benchmark findings: Across 33 models and major video-embedding paradigms, no single approach dominates.The benchmark therefore distinguishes strengths across models rather than identifying one universally superior method.
  • Modality findings: Audio’s contribution depends on whether dataset annotations are AV-grounded or V-grounded.The paired video-only versus audio+video analysis shows that modality usefulness varies with annotation provenance.
  • Benchmark maintenance: MVEB is integrated into the MTEB ecosystem with task and model versioning plus a community contribution pipeline.These mechanisms allow the benchmark to co-evolve with the field rather than remain a fixed snapshot.

Limitations

MVEB’s limitations concern model coverage, evaluation budgets, dataset representation, training-data leakage, and annotation quality. The benchmark is a timely but incomplete snapshot whose scores and labels require contextual interpretation.

  • Model Coverage: The 33-checkpoint roster is a field snapshot, with additional publicly available models planned as inference APIs and versioned checkpoints become available.Models are added on a rolling basis through the MTEB registry.
  • Frame and Audio Budgets: Declared frame and audio sampling budgets can partly influence absolute scores, although frame-scaling analysis varies the test-time budget.Each model uses its declared sampling configuration.
  • Dataset Coverage: Despite 184 tasks, MVEB+ underrepresents low-resource languages, long-form video, and several fine-grained domains including scientific, medical, and sign-language video.Sports coverage beyond a few action-recognition datasets is also limited or absent.
  • Training Data Disclosure and Leakage: Web-curated sources common in video-caption pretraining create a real train-test contamination risk, while undisclosed training data makes the audit incomplete.Appendix E cross-references declared training data with MVEB task names and reports disclosed-zero-shot percentages.
  • Annotation Quality: Some source datasets contain annotation ambiguities, including multi-sentiment utterances forced into one label, surface-emotion labels for sarcasm, and apparent mislabels.These conventions are especially recurrent in emotion-recognition tasks, and Appendix L flags affected splits.

Ethical Considerations … D.8 Question Answering

MVEB documents its dataset provenance, modality scope, ethical limitations, task construction, and per-task evaluation across nested benchmarks and multiple task families. Results show complementary model strengths: MLLM-based embeddings lead several understanding tasks, while specialized models dominate selected retrieval, zero-shot, classification, and video-only settings.

  • Ethical Considerations: MVEB redistributes decoded frame samples and 16 kHz mono audio under source licenses without re-hosting original full-resolution videos, while inheriting English-language, US-centric, and demographic dataset biases.The authors encourage contributions expanding underrepresented languages, regions, and domains.
  • A.1 Audio–video annotation provenance: Audio helps on AV-grounded datasets and hurts on V-grounded datasets because paired variants measure ∆ = scoreva − scorev while labels originate from both modalities or visuals alone, respectively.The provenance distinction concerns annotation inputs, not whether source clips contain audio.
  • B Dataset Construction; B.1 Video Classification; B.2 Video-Centric QA; B.3 Video Retrieval: MVEB reuses publicly released clips across classification, video-centric QA, and retrieval, applying task-specific split, filtering, sampling, and audio-extraction procedures from upstream releases.Examples include approximately 4K AVE clips, approximately 1,200 Daily-Omni QA examples, and approximately 3,104 YouCook2 retrieval segments.
  • D.1 Modality-restricted leaderboards; D.2 MVEB+ aggregate: MVEB+ preserves MVEB’s relative model performance with Pearson r=0.996 and Spearman ρ=0.944, while modality-restricted leaderboards rank only models compatible with their input surfaces.These restricted tables accommodate missing audio or video inputs rather than defining parallel benchmarks.
  • D.6 Clustering; D.7 Pair Classification; D.8 Question Answering: LCO-Embedding-Omni-7B leads curated clustering at 27.3 v-measure, LCO-Embedding-Omni-3B leads pair classification at 80.7 max-AP, and LCO-Embedding-Omni-7B leads QA at 57.0.Clustering remains constrained by MELDEmotion AV, where no model exceeds 8.8; Perception Encoder audio-visual models score 27.8–30.6 on QA, behind MLLM-embed leaders.

E Data Contamination Analysis · F Extended Analyses on Temporal Context Scaling (# Frames) · F.1 Experimental Setup

The contamination analysis finds disclosed training overlap with MVEB for 15 of 33 models, reaching 65% in the worst case, while four models lack dataset-level disclosure. The temporal-context appendix evaluates uniformly sampled frame counts from 1 to 64 across seven long-clip tasks and five models under otherwise fixed MVEB conditions.

  • E Data Contamination Analysis: 15 of 33 evaluated models disclose training overlap with at least one MVEB task, with a worst-case overlap of 65%.Eleven models report no overlap, while seven do not disclose training data at dataset granularity.
  • E Data Contamination Analysis: Four models cannot be certified as zero-shot because they do not disclose training data at dataset granularity.Their Table 32 rows use NA for Zero-shot % and # Overlap rather than defaulting to 100% and 0.
  • F Extended Analyses on Temporal Context Scaling (# Frames): The temporal-context appendix expands the main-text frame-scaling analysis with task- and model-level breakdowns alongside its experimental setup.It documents the subsets, varied variable, held-fixed inputs, and detailed results underlying the aggregate plot.
  • F.1 Experimental Setup: The experiment sweeps uniformly sampled frame counts N ∈ {1, 8, 16, 32, 64} at test time for every selected task–model pair.Only the number of sampled video frames varies across runs.
  • F.1 Experimental Setup: All other model inputs and evaluation choices remain at their default MVEB settings, including audio handling, instructions, output layer, and task metrics.Metrics are nDCG@10 for retrieval, accuracy for classification and QA, and v-measure for clustering.
  • F.1 Experimental Setup: The task subset contains seven MVEB tasks whose average source clip exceeds 15 seconds, allowing multi-frame sampling room to matter.Table 33 lists the selected tasks.
  • F.1 Experimental Setup: The model subset contains five paradigms-spanning models ranging from 847M to 7B parameters, covering temporal attention and posthoc frame pooling.Fixed clip-length models such as X-CLIP and 16-frame PE-AV variants are excluded because their positional embeddings are tied to one budget.

F.2 Task-Level Performance Scaling … I Audio Contribution: Per-Dataset and Per-Model Breakdowns

Frame-count scaling varies substantially by task and architecture, while MTEB improves reproducibility through richer tracking, incremental evaluation, metadata extensions, and transparent reporting. The appendices further document per-model sampling, task correlations, parameter-performance relationships, score distributions, and audio-contribution analyses.

  • F.2 Task-Level Performance Scaling: Breakfast rises from 15.88 to 45.35 and VATEX T2VA from 38.24 to 76.03 as sampled frames increase from N=1 to N=64.OmniVideoBench QA and WorldSense1Min QA change only from 25.85 to 26.68 and 28.42 to 30.60, respectively.
  • F.3 Model-Level Performance Scaling: ebind-full reaches 45.62 at N=8 and only 46.68 at N=64, whereas omni-embed-nemotron-3b and pe-av-small reach 44.67 and 44.32 at N=64.MLLM-based and audio-visual contrastive models scale more smoothly through the frame-count sweep.
  • G Accessibility and Reproducibility: MTEB records complete model configurations and task parameters with each result, tracks package versions per dataset subset, and preserves reproducible result merging across versions.This reduces reruns when adding splits or languages while maintaining a reproducible traceback.
  • G Accessibility and Reproducibility: Incremental per-subset result writing limits crash recovery to interrupted subsets, while phase-level runtime tracking exposes evaluation wall-clock costs alongside quality metrics.The change improves evaluation efficiency and runtime transparency.
  • G Accessibility and Reproducibility: MODELMETA now records active inference parameters and required dependencies, improving efficiency comparisons and reproducibility for third-party models.Active parameters distinguish engaged inference parameters from total parameter count, while declared dependencies reduce registration friction.
  • G Accessibility and Reproducibility: A performance-over-time leaderboard visualization contextualizes new benchmark results against the historical progression of state-of-the-art model submissions.The visualization traces score trajectories across benchmarks over the submission history.
  • G.1 Per-model sampling configuration: Models retain their declared video and audio sampling configurations on every task instead of using a benchmark-wide sampling budget.The appendix points to §3.2 for the rationale behind honoring model-specific settings.
  • H Task correlation: MVEB diagnostics examine task correlations, task-level parameter-count relationships, per-task score distributions, and audio deltas, including per-dataset annotation-group breakdowns and per-model means across 48 paired task groups.These analyses support redundancy pruning, dataset coverage, and the expanded audio-contribution analysis; cross-modal retrieval alignment is also reported for datasets with both v2a and a2v variants.

J Cross-Modal Retrieval Direction Structure: Extended Analysis

Across eight cross-modal retrieval directions, rank correlations reveal three latent capability groups rather than one uniform retrieval skill. Within-group rankings are tightly aligned, while audio’s role differs sharply between joint targets and audio-conditioned queries, motivating grouped interpretation alongside per-direction reporting.

  • Three latent capability axes: 28 pairwise correlations resolve the eight retrieval directions into three latent capability groups.The analysis computes every pairwise Spearman correlation across the eight directions using 16 audio-capable models.
  • Three latent capability axes: 0.92 is the strongest correlation: T →V and T →V A differ only in whether the target includes audio.These text-query directions belong to the text-target group, whose correlations span 0.77–0.92.
  • Three latent capability axes: 0.94 and 0.87 correlations show that A → V, AT →V, and V →A measure essentially the same audio-video joint-encoder capability.The internal correlation range for this group is 0.87–0.94.
  • What this means for benchmarking: 0.38 and 0.39 are the lowest correlations, separating T →V A from A →V and AT →V across audio target-versus-query roles.Models performing well when audio is part of the target are not guaranteed to perform well when audio is part of the query.
  • What this means for benchmarking: MMEB-V3 cannot expose the joint-audio-target group because it includes only A →V and V →A, not T →V A.Its treatment of audio as a separate retrieval modality leaves audio as part of a joint retrieval target invisible.
  • Implication for the headline leaderboard: Three target-grouped scores can preserve rank information because within-group correlations exceed 0.85, while all eight directions remain available for transparency.The MVEB Retr column already mean-pools the eight directions into a single score aligned with the dominant capability axis.

K MVEB(text, video): Extended Analysis

MVEB(text, video) is led broadly by the Qwen3-VL-Embedding family, while remaining a text-video-only projection that excludes audio-conditioned directions and audio+video variants. Its task-pool relationship to MMEB-V2 enables comparison within MVEB’s shared evaluation framework.

  • Breadth of the Qwen3-VL family’s lead: Qwen3-VL-Embedding-8B and -2B rank first and second on MVEB(text, video), with means of 60.9 and 58.1.Qwen3-VL-Embedding-8B leads retrieval (69.2), classification (57.5), and zero-shot classification (59.2).
  • Breadth of the Qwen3-VL family’s lead: The observed Qwen3-VL lead is not attributed to a specific cause because checkpoints differ simultaneously in training recipe, training-data composition, and declared input modalities.These differences prevent isolating the source of the performance gap.
  • What MVEB(text, video) is not: MVEB(text, video) excludes audio-conditioned retrieval directions and audio+video task variants, so its rank does not establish generalization to audio-bearing tasks.The headline MVEB leaderboard remains the appropriate surface for audio-capable models.
  • Relation to MMEB-V2: MMEB-V2 evaluates 18 video tasks spanning retrieval, classification, VQA, grounding, and instance retrieval, whereas MVEB(text, video) uses the text-video projection of MVEB’s shared task pool.This shared pool supports comparing the same model on the same tasks with and without an audio pathway.

L Annotation Quality Notes

MVEB’s emotion-recognition splits are especially affected by known annotation-quality problems, so low scores there can reflect label noise rather than embedding limitations. The paper retains the original labels for comparability while outlining community mitigation paths.

  • Dataset-specific annotation issues: Known label noise primarily depresses MELD and RAVDESS scores, making their low task performance an imperfect measure of embedding quality.Figure 9 attributes the low emotion-recognition scores mainly to dataset label-quality issues rather than embedding-quality limits.
  • Dataset-specific annotation issues: MELD compresses multi-sentiment utterances into one emotion label and lets sitcom context cues override literal text content.Examples include simultaneous surprise, rebuke, and laughter, alongside laugh tracks, sarcastic delivery, and reaction shots.
  • Dataset-specific annotation issues: RAVDESS omits compound or transition emotions from its single-emotion labels and treats intensity as a separate axis.Its scripted performances come from 24 professional actors at two intensity levels.
  • Dataset-specific annotation issues: AVE-Dataset annotates only the dominant event when multiple events co-occur, introducing noise for whole-clip classification instead of bounded event spans.The dataset supplies class labels and bounding times, but clip-level tasks operate on the full clip.
  • Label policy and mitigation: The paper keeps as-released labels to preserve comparability and evaluate embeddings on standard imperfect surfaces, while proposing soft labels, agreement audits, and targeted re-annotation.The proposed directions include per-clip label distributions, fresh human checks on model-disagreement cases, and re-annotation of the most affected splits.
Loading 2606.14958v1…