PaperlayerView exact revision on arXiv

Source-linked AI summary

MAEB: Massive Audio Embedding Benchmark

Adnan El Assadi, Isaac Chung, Chenghao Xiao, Roman Solomatin, Animesh Jha, Rahul Chand, Silky Singh, Kaitlyn Wang, Ali Sartaz Khan, Marc Moussa Nasser, Sufen Fong, Pengfei He, Alan Xiao, Ayush Sunil Munot, Aditya Shrivastava, Artem Gazizov, Niklas Muennighoff, Kenneth Enevoldsen

arXiv:2602.16008v1cs.SDcs.AIcs.CLcs.LG

TL;DR

Audio embedding evaluation is fragmented across narrow task and domain benchmarks, limiting insight into transfer across applications. MAEB unifies 30 tasks across 100+ languages, and its evaluation finds task-specific strengths rather than a universally dominant model, with contrastive audio-text and speech-pretrained models favoring different task types.

  • Problem

    Existing audio benchmarks often focus on specific tasks or narrow domains, leaving limited insight into how embeddings transfer across applications.

    Source: §1 · p. 1

  • Method

    MAEB curates 30 tasks across speech, music, environmental sounds, and bioacoustics, selecting datasets for domain, task, linguistic diversity, and quality.

    Source: §2.1 · p. 2, §7 · p. 8

  • Results

    No model achieves universal performance: speech-pretrained models perform strongly on audio-only tasks but lack cross-modal evaluation, while contrastive audio-text models remain weak on multilingual speech tasks.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The evaluation supports treating audio embedding performance as task-dependent rather than assuming a universally strong model.

    Source: §7 · p. 8

    Coverage skews toward Western musical traditions and standard speech, while many underrepresented language families have limited representation across tasks.

    Source: §5 · p. 7

Abstract

from arXiv · show

We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasoning in 100+ languages. We evaluate 50+ models and find that no single model dominates across all tasks: contrastive audio-text models excel at environmental sound classification (e.g., ESC50) but score near random on multilingual speech tasks (e.g., SIB-FLEURS), while speech-pretrained models show the opposite pattern. Clustering remains challenging for all models, with even the best-performing model achieving only modest results. We observe that models excelling on acoustic understanding often perform poorly on linguistic tasks, and vice versa. We also show that the performance of audio encoders on MAEB correlates highly with their performance when used in audio large language models. MAEB is derived from MAEB+, a collection of 98 tasks. MAEB is designed to maintain task diversity while reducing evaluation cost, and it integrates into the MTEB ecosystem for unified evaluation across text, image, and audio modalities. We release MAEB and all 98 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.

1. Introduction

MAEB addresses fragmented audio-embedding evaluation with a broad, extensible benchmark spanning diverse audio tasks and domains. Its evaluation finds no universally dominant model and suggests encoder quality may relate to downstream Audio LLM performance.

Framework contribution

  • MAEB extends the MTEB-style evaluation framework to audio and is designed for community maintenance as audio representation learning advances.The framework builds on MTEB, MMTEB, and MIEB, which unified and expanded text and image embedding evaluation.Source: §1 · pp. 1–2

Benchmark scope

  • MAEB addresses fragmented evaluation by providing a unified benchmark spanning 30 tasks in 7 categories and covering speech, music, environmental sounds, and bioacoustics.Its scope includes multilingual understanding, long-form audio processing, cross-modal audio-text tasks, and fine-grained recognition.Source: §1 · p. 1

Efficiency and extensibility

  • A small audio-only model can be evaluated in 2 GPU hours, while MAEB(audio) offers 19 tasks and MAEB+ retains the full collection of 98 tasks.A modular architecture is intended to simplify adding models and datasets.Source: §1 · pp. 1–2

Evaluation findings

  • Across 53 evaluated models, no single model dominates across audio domains, and preliminary results suggest encoder quality correlates with Audio LLM performance (R2 = 0.86, n = 4).The introduction also identifies multilingual contexts and cross-modal understanding as areas where current models struggle.Source: §1 · p. 2

2. MAEB

MAEB extends MTEB’s unified evaluation framework to audio embeddings and selects a diverse, lower-cost task subset from MAEB+ while retaining broad evaluation coverage. It evaluates embeddings across classification, clustering, retrieval, pair classification, and reranking.

2. MAEB

  • MAEB integrates audio embedding evaluation into MTEB alongside text and image embeddings, with standardized protocols, extensible interfaces, reproducible artifacts and community maintenance.Its scope is embedding quality for downstream tasks, not transcription, generation or other capabilities outside representation learning.Source: §2 · p. 2

2.1. Benchmark Construction

  • Dataset curation spans domains, task types, languages and dialects, while task selection prioritizes validity, unique coverage, linguistic breadth, low redundancy and runtime efficiency.For redundancy removal, tasks with Spearman ρ > 0.8 to a retained task are removed, favoring broader coverage or lower runtime; quality and accessibility guide dataset inclusion.Source: §2.1 · pp. 2–3
  • MAEB reduces the 89-task MAEB(extended) set to 30 tasks, yielding a 2.2–3.3× speedup while preserving score correlation (Pearson r=0.981) and ranking correlation (Spearman ρ=0.912).Runtime varies by model type; runtime comparisons are measured in GPU hours on one NVIDIA A100 GPU.Source: Table 1 · p. 3, §2.1 · p. 3
  • The full unfiltered task collection is released as MAEB+, and the leaderboard reports both Borda-count ranks and mean scores.Borda count treats each task as a preference voter and offers scale invariance and robustness to outliers, but is not continuous.Source: §2.1 · p. 3

2.2. Tasks and Evaluation

  • Classification uses logistic-regression few-shot linear probing with 8 examples per class, while zero-shot classification matches embeddings directly to text prompts and measures accuracy.Zero-shot prompts convert class labels to text, such as “This is a sound of dog bark.”Source: §2.2 · p. 3
  • Clustering uses MiniBatchKMeans with k equal to the true label count and V-measure; retrieval ranks audio or text queries against audio documents by cosine similarity using CV Recall@5.Retrieval includes audio-to-audio, text-to-audio and audio-to-text scenarios.Source: §2.2 · p. 3
  • Pair classification predicts similarity between two audio inputs using average precision, while reranking evaluates pre-selected candidates with hard negatives using MAP@1000.Both methods compute similarity from embeddings; pair classification uses cosine similarity, and reranking tests fine-grained discrimination.Source: §2.2 · p. 3

3. Experimental Settings

The experimental setup evaluates 50+ audio encoders across distinct model families using consistent preprocessing and model-native embedding extraction.

3.1. Models

  • The benchmark evaluates 50+ encoders spanning audio-specific, sequence-to-sequence, contrastive audio-text, and adapted audio-language model families.Audio-specific models include self-supervised speech, transformer, CNN, and neural codec approaches.Source: §3.1 · pp. 3–4
  • The model-family taxonomy is approximate: LCO-Embedding (Xiao et al., 2025a) and Wav2Vec2/XLS-R (Babu et al., 2021) both use contrastive loss.Additional model details are provided in Appendix B.Source: §3.1 · p. 4

3.2. Implementation Details

  • Audio is truncated to at most 30 seconds, resampled to model-specific rates, and converted to mono when required.The stated rates are 16kHz for speech models, 48kHz for CLAP and MS-CLAP variants, and 24kHz for MuQ-MuLan and Encodec.Source: §3.2 · p. 4
  • Embedding extraction uses model-native pooling: temporal mean for transformers, global average for CNNs, specialized pooling as intended, and final-layer last-token pooling for audio-language models.Contrastive models use their audio encoder branches with L2 normalization for retrieval compatibility.Source: §3.2 · p. 4

4. Results

MAEB results show that model performance is specialized rather than universal: leaders vary across tasks and acoustic domains, multilingual performance remains uneven, and clustering is difficult. Encoder scores also show a preliminary positive relationship with Audio LLM performance.

4.1. Key Findings on Model Performance

  • Speech-trained models underperform on music, while music-focused CLAP variants struggle with speech, despite showing more balanced performance across categories.Whisper-medium achieves 51.7% on classification but 5.0% on clustering; CLAP variants have lower peak scores on speech-specific tasks.Source: §4.1 · pp. 4–5

4. Results

  • LCO-Embedding-Omni-7B ranks first overall by Borda count (52.2% average), while Qwen2-Audio-7B ranks first on audio-only tasks (50.8% average).Qwen2-Audio-7B ranks second overall with a 33.7% average; LCO-Embedding-Omni-7B scores 50.3% on cross-modal retrieval and 64.5% on zero-shot classification.Source: §4 · p. 4, Table 2 · p. 5
  • AST leads Music (71.6), Environmental (63.8), and Bioacoustics (45.2), while LCO leads Speech (68.2) and Qwen2-Audio leads Emotion (44.7).No domain leader reaches the dashed target of 80, and the domain-level results show no universal performer.Source: §4 · p. 4, Figure 2 · p. 6

4.1. Key Findings on Model Performance

  • Multilingual performance is resource-skewed, and cross-modal audio-text retrieval collapses across languages despite stronger audio-to-audio retrieval.On SIBFLEURS, high-resource European languages reach 40–60% accuracy while Umbundu, Yoruba, and Xhosa remain below 20%; on FLEURS, CLAP models score below 3% on most language pairs, often below 1% for audio-to-text and text-to-audio, compared with 50–99% audio-to-audio retrieval on JamAlt.Source: §4.1 · pp. 5–6
  • Acoustic-versus-linguistic trade-offs persist: CLAP-htsat-unfused scores 94.4% on gender identification versus 30.0% on language identification, while Whisper-medium scores 59.2% versus 99.4%.Audio-text task performance also varies: LCO-Embedding-Omni-7B and Qwen2-Audio-7B score 50.3% and 1.6%, respectively, on cross-modal retrieval, showing that similar training approaches do not ensure balanced performance.Source: §4.1 · pp. 5–6
  • Clustering is difficult across models: even the best, clap-htsat-fused, achieves only 22.7%, while LCO-Embedding-Omni-7B reaches 1.7%.Qwen2-Audio-7B scores 12.7% and Whisper-medium 5.0%, indicating a gap in unsupervised semantic organization.Source: §4.1 · p. 6

4.2. Correlation with Audio LLM Performance

  • Encoder embedding quality has a preliminary positive relationship with Audio LLM MMAU scores (R²=0.86, p=0.072, n=4), but the small sample warrants caution.The comparison uses four Audio LLMs and encoder scores from 26 MAEB+ classification tasks aligned with MMAU’s Speech, Music, and Sound domains.Source: Figure 3 · p. 7, §4.2 · p. 7

5. Limitations

MAEB has limitations in model and compute coverage, audio duration, dataset representation, task coverage, and ecological validity.

Technical Constraints

  • The evaluation covers 50+ models but only a subset of available models, while large models require substantial computational resources that limit accessibility.Whisper-large-v3 has 1.55B parameters and Wav2Vec2-XLS-R-2B has 2B parameters.Source: §5 · p. 7
  • Audio is limited to 30 seconds for memory management or to models’ shorter native limits, restricting evaluation of long-form content such as podcasts and lectures.Source: §5 · p. 7

Dataset Coverage Limitations

  • Despite covering 100+ languages, MAEB has gaps in underrepresented language families, with some languages appearing in only one dataset, and skews toward Western music and standard speech.Source: §5 · p. 7
  • MAEB’s 30 tasks across 7 categories omit audio-generation quality and real-time processing evaluation, while clean studio recordings limit ecological validity under noisy, reverberant, or compressed conditions.MAEB+ contains 98 tasks.Source: §5 · p. 7

6. Related Work

Prior embedding benchmarks established broad, standardized evaluation, but audio embedding evaluation remains limited in task coverage, multilingual support, and reproducibility. MAEB addresses these gaps within a maintained framework, while AudioBench and MMAU evaluate AudioLLMs rather than embedding models.

Text Embedding Benchmarks

  • MTEB evaluates text embeddings across 8 task families, 58 datasets, and 112 languages, motivating analogous audio benchmarking alongside MMTEB and MIEB.MTEB (Muennighoff et al., 2023) and later expansions toward multilingual text and image embeddings (Enevoldsen et al., 2025; Xiao et al., 2025b) illustrate the value of broad, maintained leaderboards with consistent protocols.Source: §6 · p. 7

Audio Representation Benchmarks

  • HEAR (Turian et al., 2022) evaluated 29 models on 19 downstream audio tasks, primarily testing pretrained features with simple classifiers such as MLPs.Its domains include speech recognition, music tagging, and environmental sound classification.Source: §6 · p. 7
  • Audio embedding evaluation has limited task coverage, fragmented zero-shot testing, insufficient multilingual support, and reproducibility challenges that MAEB addresses within a maintained framework.Prior work largely emphasizes classification, with retrieval and clustering neglected; zero-shot methods have been explored in isolation, and outdated datasets and inconsistent protocols hinder fair comparisons.Source: §6 · p. 8
  • AudioBench (Wang et al., 2024) and MMAU target AudioLLMs, not embedding models.AudioBench evaluates instruction-following across eight tasks using 26 datasets, while MMAU benchmarks multimodal reasoning across speech, sound, and music.Source: §6 · p. 8

7. Conclusion

MAEB evaluates audio embeddings across 30 tasks and 100+ languages, revealing that model strengths vary across task types and that clustering remains difficult. It also supports unified multimodal evaluation through MTEB and provides resources for community-driven progress.

7. Conclusion

  • MAEB comprises 30 tasks across 100+ languages, with baselines from 50+ models.Source: §7 · p. 8
  • In the MAEB evaluation, LCO-Embedding-Omni-7B ranks first overall, with strongest cross-modal retrieval (50.3%) and zero-shot classification (64.5%) averages; Qwen2-Audio-7B ranks second overall, first on audio-only tasks, and excels in reranking (80.8%).Source: §7 · p. 8
  • Speech-pretrained models perform strongly on audio-only tasks but cannot support cross-modal evaluation, while contrastive audio-text models enable cross-modal tasks but remain weak on multilingual speech.Source: §7 · p. 8
  • Clustering is universally challenging, with the best model at 22.7%, and acoustic strengths such as gender identification trade off with linguistic strengths such as language identification.Source: §7 · p. 8
  • LCO models achieve 50%+ accuracy in cross-modal multilingual retrieval across 100+ languages, while most other models remain below 2%; preliminary analysis across four Audio LLMs suggests MAEB encoder quality relates positively to downstream performance.Source: §7 · p. 8
  • MAEB integrates with MTEB for unified text, image, and audio evaluation, and its code, tasks, and leaderboards are released to support community-driven progress.Source: §7 · p. 8

Impact Statement

MAEB addresses the access and environmental costs of large benchmarks by reducing dataset sizes and reporting per-task CO2 measures.

Impact Statement

  • Large benchmarks can burden low-resource communities and the environment; MAEB reduces datasets to reasonable sizes and reports kilogram CO2 measures per task.The measures let users assess environmental benchmarking costs.Source: Impact Statement · p. 8

A. Tasks overview … B.4. Large Audio-Language Models

The appendix documents MAEB task characteristics and evaluation models, including the embedding extraction configurations used across several audio-model families.

A. Tasks overview

  • The appendix reports task size, language, and metrics in Tables 3–4 and domain distribution in Figure 5.Source: A. Tasks overview · p. 17

B. Overview of Models

B.1. Audio Encoders

B.2. Sequence-to-Sequence Models

  • Evaluated models include multilingual Whisper, MMS, SeamlessM4T (Communication et al., 2023), and SpeechT5 ASR (Ao et al., 2022), using encoder-derived retrieval embeddings.Source: §B.2 · p. 18

B.3. Contrastive Alignment Models

B.4. Large Audio-Language Models

  • Large audio-language models include Qwen2-Audio and two LCO-Embedding variants (Xiao et al., 2025a), with embeddings extracted by last-token pooling.Source: §B.4 · p. 18

C. Correlation Analysis Tasks

Figure 3’s correlation analysis uses 26 MAEB+ classification tasks grouped by domain to align with the MMAU benchmark.

C. Correlation Analysis Tasks

  • The 26-task subset covers speech, music, and sound classification, with speech contributing the largest group at 13 tasks.The speech tasks are SpeechCommands, FSDD, CommonLanguage (Age, Gender, Language), VoxPopuli (Accent, Gender, Language), VoxLingua107, LibriCount, VocalSound, VoxCelebSA, and SpokeNEnglish.Source: C. Correlation Analysis Tasks · p. 20
  • Music contributes five tasks, including genre, Beijing Opera, Mridingham, and NSynth classification.The subset includes GTZAN Genre, Beijing Opera, Mridingham (Stroke, Tonic), and NSynth.Source: C. Correlation Analysis Tasks · p. 20
  • Sound contributes eight tasks spanning environmental classification, acoustic scenes and context, gunshot triangulation, and general sound datasets.The subset includes ESC50, UrbanSound8k, TUT Acoustic Scenes, Ambient Acoustic Context, Gunshot Triangulation, AudioSet Mini, FSD50K, and FSD2019 Kaggle.Source: C. Correlation Analysis Tasks · p. 20

D. Domain Radar Chart Methodology

Figure 2 summarizes model performance across five acoustic domains using 94 MAEB+ tasks. It displays representative models that lead in at least one domain, highlighting domain specialists and generalists.

D. Domain Radar Chart Methodology

  • The chart covers 94 tasks: speech (44), music (13), environmental sounds (29), bioacoustics (2), and emotion (6).Tasks are assigned according to their primary audio content and intended application.Source: D. Domain Radar Chart Methodology · p. 20
  • For each model and domain, scores are the arithmetic mean of task main scores, with native 0–1 metrics mapped to a shared 0–100 scale.This aggregation gives different task types equal contribution to the domain average.Source: D. Domain Radar Chart Methodology · p. 20
  • The chart displays only representative models with the highest average score in at least one domain to show both specialists and generalists.Source: D. Domain Radar Chart Methodology · p. 20
  • Tasks unavailable to a model are omitted from its domain average rather than counted as zero, reflecting the quality of its existing capabilities.For example, audio-only encoders may not perform text-to-audio retrieval.Source: D. Domain Radar Chart Methodology · p. 21

E. Per Task Category Results … E.7. Reranking

Across task categories, no model leads everywhere: LCO-Embedding-Omni-7B posts strong averages and broad performance, while CLAP models excel in environmental sound tasks and other models lead on selected tasks.

E.2. Linear Probe Classification

  • 61.7%: Qwen2-Audio-7B achieves the highest linear-probe classification average, while LCO models excel on language and speaker tasks including MInDS14 (> 98%).Qwen2-Audio-7B leads across multiple emotion, music, and vocal-sound tasks; the AudioSet-finetuned model remains dominant on AmbientAcousticContext and BirdCLEF.Source: §E.2 · p. 21, Table 9 · p. 26

E.4. Clustering

  • 35.3%: larger_clap_music_and_speech achieves the highest clustering average, closely followed by clap-htsat-unfused at 35.0%.Contrastive objectives structure embeddings around semantic similarity, while ASR encoders and Audio-LLMs generally trail, with representations described as phonetically granular or generation-oriented.Source: §E.4 · p. 21, Table 18 · p. 35, Table 19 · p. 36

E.5. Pair Classification

  • 79.2%: LCO-Embedding-Omni-7B achieves the highest pair-classification score, compared with 59.9% for whisper-medium.Its performance across speech and mixed domains drives its superior average; CLAP models also show competence on environmental sound pairs.Source: §E.5 · pp. 21–22, Table 20 · p. 37

E.3. Multilabel Classification

  • LCO-Embedding-Omni-7B leads FSD2019Kaggle, while Qwen2-Audio-7B leads FSD50K in multilabel classification.This contrasts with earlier findings in which AudioSet-finetuned models were dominant; large-scale trained multimodal models perform better on these diverse acoustic tagging tasks.Source: §E.3 · p. 21, Table 21 · p. 38

E.1. Zero-Shot Classification

  • 76.2%: LCO-Embedding-Omni-7B achieves the highest overall zero-shot performance, while larger_clap_general tops ESC50 at 90.5%.LCO models score above 96% on SpeechCmd v0.01 and v0.02, whereas CLAP models generally underperform them on speech-specific tasks.Source: §E.1 · p. 21, Table 22 · p. 39

E.7. Reranking

  • 86.0%: LCO-Embedding-Omni-7B achieves the highest reranking average, while msclap-2023 leads on specific environmental reranking tasks.LCO tops VocalSound and UrbanSound8K; the results point to greater versatility across diverse reranking challenges than specialized models.Source: §E.7 · p. 22, Table 23 · p. 40

E.6. Retrieval

  • LCO-Embedding models lead speech-text retrieval, while larger_clap_general is stronger on environmental sound retrieval; UrbanSound8K is an exception favoring LCO.The reported speech-text tasks are CMU Arctic, EmoVDB, HiFiTTS, and LibriTTS; environmental tasks include AudioCaps, AudioSetStrong, and Clotho.Source: §E.6 · p. 22, Table 24 · p. 41

Related papers

Loading 2602.16008v1…