Source-linked AI summary
MAEB: Massive Audio Embedding Benchmark
Adnan El Assadi, Isaac Chung, Chenghao Xiao, Roman Solomatin, Animesh Jha, Rahul Chand, Silky Singh, Kaitlyn Wang, Ali Sartaz Khan, Marc Moussa Nasser, Sufen Fong, Pengfei He, Alan Xiao, Ayush Sunil Munot, Aditya Shrivastava, Artem Gazizov, Niklas Muennighoff, Kenneth Enevoldsen
TL;DR
Audio-embedding evaluation is fragmented across domains, tasks, and languages, leaving limited evidence about transfer and model capabilities. MAEB addresses this gap with a unified 30-task benchmark derived from 98 tasks and evaluated across 50+ models. Results show specialization rather than a universal model, severe clustering and cross-modal multilingual gaps, and a preliminary positive relationship between encoder and Audio LLM performance.
Problem
Existing benchmarks provide narrow task coverage and fragmented multilingual, zero-shot, retrieval, clustering, maintenance, and reproducibility support.
Method
MAEB selects 30 diverse tasks from MAEB+ using validity, coverage, linguistic breadth, redundancy, and efficiency criteria, integrating them into MTEB.
Results
No single model dominates: LCO-Embedding-Omni-7B ranks first overall, while speech-pretrained and contrastive audio-text models specialize in different capabilities; clustering remains difficult, with the best model at 22.7%.
Takeaways & Limitations
MAEB provides a unified way to compare audio embeddings across capabilities, and its scores may predict downstream Audio LLM performance.
Takeaways & Limitations
Long-form applicability is restricted by 30-second processing limits and computational demands, while coverage remains skewed toward Western music and standard speech patterns.
Abstract
from arXiv · showhide
We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasoning in 100+ languages. We evaluate 50+ models and find that no single model dominates across all tasks: contrastive audio-text models excel at environmental sound classification (e.g., ESC50) but score near random on multilingual speech tasks (e.g., SIB-FLEURS), while speech-pretrained models show the opposite pattern. Clustering remains challenging for all models, with even the best-performing model achieving only modest results. We observe that models excelling on acoustic understanding often perform poorly on linguistic tasks, and vice versa. We also show that the performance of audio encoders on MAEB correlates highly with their performance when used in audio large language models. MAEB is derived from MAEB+, a collection of 98 tasks. MAEB is designed to maintain task diversity while reducing evaluation cost, and it integrates into the MTEB ecosystem for unified evaluation across text, image, and audio modalities. We release MAEB and all 98 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.
1. Introduction
MAEB addresses fragmented audio-embedding evaluation with a unified benchmark spanning domains, languages, and task types. It provides broad coverage, efficient evaluation, extensibility, and evidence that model capabilities remain specialized.
- Motivation: Existing audio benchmarks emphasize narrow tasks or domains, limiting comparisons and insight into transfer across applications.The field also lacks unified maintenance and reproducibility infrastructure.
- Benchmark: MAEB provides a unified evaluation protocol for advancing toward universal audio embedding models.It extends the continually maintained MTEB-style framework from text and image embeddings to audio.
- Benchmark: MAEB spans 30 tasks across seven categories, including multilingual, long-form, and cross-modal audio-text evaluation.The benchmark covers diverse acoustic domains beyond traditional speech recognition.
- Efficiency and extensibility: MAEB evaluates a small audio-only model in 2 GPU hours while preserving coverage, and releases MAEB(audio), MAEB+, and a modular extension framework.MAEB(audio) contains 19 tasks, while MAEB+ contains the full 98-task collection.
- Findings: Evaluation of 53 models finds no universal leader, while preliminary evidence links MAEB encoder quality to Audio LLM performance with R2 = 0.86 across four models.Models excel in specific areas while underperforming in others.
- Findings: The benchmark identifies persistent weaknesses in multilingual and cross-modal understanding and provides a framework for standardizing evaluation and tracking progress.These are presented as directions for future research and community-wide evaluation.
2. MAEB
MAEB selects a diverse, lower-cost subset from MAEB+ and integrates it into MTEB with standardized, extensible evaluation procedures. Its tasks cover major audio-embedding applications using established metrics and probing protocols.
- Ecosystem integration: MAEB integrates into MTEB with standardized metrics, extensibility, reproducibility, and community-driven maintenance.The integration reuses validated implementations and versioned public artifacts.
- Dataset and task selection: MAEB selects datasets for domain, task, linguistic, quality, licensing, and accessibility diversity.Targeted domains include speech, music, environmental sounds, and bioacoustics.
- Dataset and task selection: MAEB filters MAEB+ using validity, unique coverage, linguistic breadth, redundancy removal, and runtime efficiency criteria.Redundant tasks are removed using pairwise model-ranking correlations.
- Benchmark construction: 2.2–3.3× speedup and Pearson r=0.981 score correlation show that MAEB reduces runtime while preserving agreement with MAEB(extended).Model-ranking correlation is Spearman ρ=0.912.
- Benchmark construction: MAEB+ remains available as the full unfiltered 98-task collection for comprehensive evaluation.The selected MAEB is the reduced benchmark, while MAEB+ preserves the complete task set.
- Tasks and evaluation: The benchmark evaluates classification, zero-shot classification, clustering, retrieval, pair classification, and reranking with task-specific metrics.Zero-shot classification matches audio embeddings directly to text prompts, while retrieval uses cosine similarity and CV Recall@5.
3. Experimental Settings
MAEB evaluates more than 50 audio encoders spanning audio-specific, sequence-to-sequence, contrastive alignment, and large audio-language model categories. Models use consistent preprocessing alongside architecture-native embedding extraction, but the categories are imperfect.
- Model coverage: The evaluation includes 50+ encoders organized into four broad development categories.The categories are audio encoders, sequence-to-sequence models, contrastive alignment models, and large audio-language models.
- Model categories: Audio-specific models include self-supervised speech, spectrogram-transformer, CNN, and neural-codec architectures.Examples include Wav2Vec2/XLS-R, AST, CNN14, YAMNet, VGGish, and Encodec.
- Model categories: Sequence-to-sequence models target speech recognition or translation, whereas contrastive alignment models learn joint audio-text embedding spaces.Representative models include Whisper, MMS, SeamlessM4T, CLAP, MSCLAP, and Wav2CLIP.
- Model categories: Large audio-language models are adapted for embeddings from generative multimodal LLMs using hidden states or contrastive refinement.Qwen2-Audio and LCO-Embedding are included in this category.
- Model categories: The four model categories are imperfect because models such as LCO-Embedding and Wav2Vec2/XLS-R can both use contrastive loss.Category labels therefore describe broad development groupings rather than mutually exclusive training objectives.
- Preprocessing: All models truncate audio to 30 seconds or less, resample to model-specific rates, and convert to mono when required.Shorter limits may be imposed by architecture or memory constraints.
- Embedding extraction: Embedding extraction uses architecture-native pooling, with L2 normalization for contrastive retrieval and last-token pooling for large audio-language models.Transformer models use temporal mean pooling and CNNs use global average pooling.
4. Results
MAEB reveals strong specialization across audio domains and task types rather than a universal encoder. Multilingual and clustering performance remain major weaknesses, while preliminary evidence links encoder quality with Audio LLM performance.
- Overall model performance: LCO-Embedding-Omni-7B ranks first overall at 52.2% average, while Qwen2-Audio-7B ranks first on audio-only tasks at 50.8%.LCO-Embedding-Omni-7B also leads cross-modal retrieval at 50.3% and zero-shot classification at 64.5%; Qwen2-Audio-7B leads reranking at 80.8%.
- Domain specialization: No single encoder achieves universal performance: LCO-Embedding-Omni-7B leads Speech at 68.2, AST leads Music at 71.6, Environmental at 63.8, and Bioacoustics at 45.2, while Qwen2-Audio leads Emotion at 44.7.The domain leaders differ, and the universal-performance target of 80 remains unmet in every category.
- Multilingual understanding: SeamlessM4T-v2-large achieves the best performance on 10 of 12 MInDS-14 languages, whereas CLAP variants score below 15% across all languages.The contrast reflects a trade-off between multilingual speech specialization and audio-text models’ environmental-audio strengths.
- Multilingual understanding: High-resource European languages reach 40–60% SIBFLEURS accuracy, while Umbundu, Yoruba, and Xhosa remain below 20% even for the best models.The benchmark spans more than 200 languages across several multilingual datasets, but performance remains biased toward high-resource languages.
- Acoustic–linguistic trade-offs: Audio and linguistic capabilities trade off: CLAP-htsat-unfused scores 94.4% on gender identification but 30.0% on language identification, while Whisper-medium scores 59.2% and 99.4%, respectively.Audio-only and audio-text performance also diverge, although LCO-Embedding-Omni-7B reaches 50.3% cross-modal retrieval and 64.5% zero-shot classification.
- Clustering: Clustering is difficult across models: clap-htsat-fused reaches only 22.7%, while LCO-Embedding-Omni-7B achieves 1.7% and Qwen2-Audio-7B 12.7%.Strong overall ranking does not imply strong semantic clustering performance.
5. Limitations
MAEB’s evaluation covers many models and tasks but remains constrained by model availability, audio-length limits, computational cost, dataset coverage, and missing or unrealistic task conditions.
- Technical constraints: The evaluation includes 50+ models but represents only a subset of available models.This limits how completely the benchmark characterizes the model landscape.
- Technical constraints: Audio-length handling limits models to their native settings or 30 seconds, restricting applicability to long-form podcasts and lectures.Large models also require substantial computational resources, limiting accessibility.
- Dataset coverage: Dataset coverage skews toward Western musical traditions and standard speech patterns, with underrepresented language families often appearing in only one dataset.Although MAEB spans 100+ languages, this distribution prevents comprehensive cross-task evaluation for some languages.
- Task and ecological coverage: The benchmark lacks audio-generation quality assessment and real-time processing evaluation across its 30 MAEB tasks and 98 MAEB+ tasks.Many tasks also use clean studio recordings that omit real-world noise, reverberation, and compression artifacts.
6. Related Work
Existing embedding benchmarks have advanced systematic evaluation, but audio evaluation remains fragmented across domains, tasks, and languages. MAEB addresses gaps in task coverage and unified comparison.
- MTEB evaluates text embeddings across 8 task families, 58 datasets, and 112 languages, while MMTEB and MIEB extend standardized evaluation to multilingual text and images.
- HEAR evaluated 29 audio models on 19 downstream tasks, primarily using pretrained features with simple multilayer perceptron classifiers.Its coverage included speech recognition, music tagging, and environmental sound classification.
- MAEB+ contains 70 English-language tasks, while zxx marks datasets with no linguistic content.
- Prior audio benchmarks often emphasize narrow domains or classification, leaving retrieval, clustering, and systematic zero-shot evaluation insufficiently covered.
7. Conclusion
MAEB evaluates audio embeddings across languages, tasks, and modalities, revealing specialization rather than universal performance. It also connects benchmark quality to downstream multimodal audio understanding and supports unified ecosystem-level evaluation.
- 30 tasks across 100+ languages are benchmarked with baselines from 50+ models.
- No single model dominates all domains: LCO-Embedding-Omni-7B ranks first overall, while Qwen2-Audio-7B ranks first on audio-only tasks.
- Speech-pretrained models perform strongly on audio-only tasks but cannot support cross-modal evaluation, whereas contrastive audio-text models remain weak on multilingual speech tasks.
- Clustering is universally difficult, with the best model reaching 22.7%, and acoustic-versus-linguistic strengths exhibit stark trade-offs.
- LCO models exceed 50% accuracy in cross-modal multilingual retrieval across 100+ languages, while most CLAP, Whisper, and ASR encoders remain below 2%.
- MAEB integrates with MTEB for unified text, image, and audio evaluation and releases code, tasks, and leaderboards for community-driven progress.
Impact Statement
Large benchmarks can burden low-resource communities and increase environmental costs. MAEB reduces dataset sizes and reports kilogram CO2 measures per task to make those costs assessable.
- Large benchmarks create barriers for low-resource communities and incur high environmental costs.
- MAEB reduces large datasets to reasonable sizes and includes kilogram CO2 measures per task for assessing benchmarking costs.
A. Tasks overview
The task overview lists evaluated models and their embedding extraction procedures across transformer, codec, speech, CNN, and audio-text model families. It also documents the benchmark’s model inventory and task metadata.
- Transformer-based Models: AST uses a vision-transformer architecture over mel-spectrograms and extracts 768-dimensional pooler-output embeddings for retrieval.
- Self-supervised Speech Models: Wav2Vec2 variants range from 95M to 2B parameters, with final-layer mean pooling; XLS-R extends pretraining to 128 languages using 436k hours of speech.
- Self-supervised Speech Models: WavLM evaluates seven variants including speaker verification and diarization models, with denoising pretraining supporting retrieval on real-world noisy audio.
- The evaluated inventory includes CNN, self-supervised speech, audio-text, multimodal, and large-language-model encoders, with model sizes listed in Table 5.
- Neural Codec Models: Encodec provides 128-dimensional continuous encoder embeddings before quantization, using temporal mean pooling for retrieval.
- CNN-based Models: Whisper evaluates five sizes from 39M to 1.55B parameters and extracts final encoder-layer embeddings with sequence mean pooling.
C. Correlation Analysis Tasks
The correlation analysis uses 26 MAEB+ classification tasks across speech, music, and sound domains. These tasks provide domain-aligned coverage for comparing model performance with Audio LLM results.
- 26 classification tasks are used for the correlation analysis, grouped by domain to align with the MMAU benchmark.
- Speech: 13 speech tasks cover command recognition, speaker and language attributes, counting, vocal sound, speaker verification, and English speech understanding.
- Music: 5 music tasks cover genre, Beijing Opera, stroke and tonic recognition, and instrument synthesis.
- Sound: 8 sound tasks cover environmental classification, acoustic scenes, gunshot triangulation, AudioSet, and FSD datasets.
D. Domain Radar Chart Methodology
The domain radar chart aggregates model performance across five acoustic domains using arithmetic means of available task scores. It displays representative specialists and generalists while preserving comparable scoring across task types.
- 94 MAEB+ tasks are assigned to domains based on their primary audio content and intended application.
- Metrics including Accuracy, v_measure, nDCG, and AP are aggregated on a shared 0–100 scale, giving task types equal contribution to domain averages.
- The domain breakdown includes 44 speech tasks, 13 music tasks, 29 environmental tasks, 2 bioacoustics tasks, and 6 emotion tasks.
- Only representative models with the highest average score in at least one domain are shown, highlighting specialists and generalists.
- Domain scores average the main scores of all available tasks assigned to each domain.Tasks that a model cannot perform are omitted rather than counted as zero.
E. Per Task Category Results
Results show strong specialization across task categories: multimodal LCO and Audio LLM models lead many speech and semantic tasks, whereas CLAP models retain advantages in environmental and clustering settings. No category yields a universal winner.
- Zero-shot classification: 76.2% is the highest overall zero-shot classification performance, achieved by LCO-Embedding-Omni-7B.LCO models lead speech commands and emotional speech, while CLAP models lead environmental sound tasks such as ESC50.
- Classification: 61.7% is the highest classification average, achieved by Qwen2-Audio-7B across emotion, music, and vocal sound tasks.LCO models dominate language and speaker tasks, while AudioSet-finetuned models retain leadership on selected environmental tasks.
- Multilabel classification: LCO-Embedding-Omni-7B leads FSD2019Kaggle multilabel classification, while Qwen2-Audio-7B leads FSD50K.
- Clustering: 35.3% is the highest clustering average, achieved by larger_clap_music_and_speech, narrowly ahead of clap-htsat-unfused at 35.0%.ASR encoders and Audio-LLMs generally trail contrastive models in clustering.
- Pair classification: 79.2% is the highest pair classification score, achieved by LCO-Embedding-Omni-7B versus 59.9% for whisper-medium.LCO’s performance spans speech and mixed domains, while CLAP models are competent on environmental sound pairs.
- Retrieval and reranking: 86.0% is the highest reranking average, achieved by LCO-Embedding-Omni-7B, while msclap-2023 leads selected environmental reranking tasks.Retrieval results similarly split by domain: LCO leads speech-text retrieval, whereas CLAP leads general acoustic-event retrieval, with UrbanSound8K as an exception.