Source-linked AI summary
HEAR: Holistic Evaluation of Audio Representations
Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W. Schuller, Christian J. Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse Engel, Justin Salamon, Philippe Esling, Pranay Manocha, Shinji Watanabe, Zeyu Jin, Yonatan Bisk
TL;DR
HEAR addresses the lack of broad evidence about which audio embeddings generalize across diverse downstream tasks without fine-tuning. It evaluates open, general-purpose submissions through a common benchmark and finds task-dependent patterns rather than one simple universal model, while leaving some tasks unexplained.
Problem
Audio representations are not commonly evaluated across broad audio problems, making it difficult to know which representation to use for a novel task.
Method
HEAR uses normalized scores across tasks to compare 29 models on diverse downstream evaluations, with one embedding representation used across tasks.
Results
Performance patterns are task-dependent: strong speech models lead speech-related tasks, CREPE-based models lead pitch tasks, and AudioSet-pretrained models perform best on broad semantic-object tagging.
Takeaways & Limitations
HEAR provides an open, reproducible basis for comparing general-purpose audio representations and studying how they transfer across domains.
Takeaways & Limitations
Some tasks have fragile, unpredictable model boundaries, and the benchmark offers no simple pretraining-data explanation for several difficult tasks.
Abstract
from arXiv · showhide
What audio embedding approach generalizes best to a wide range of downstream tasks across a variety of everyday domains without fine-tuning? The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. HEAR was launched as a NeurIPS 2021 shared challenge. In the spirit of shared exchange, each participant submitted an audio embedding model following a common API that is general-purpose, open-source, and freely available to use. Twenty-nine models by thirteen external teams were evaluated on nineteen diverse downstream tasks derived from sixteen datasets. Open evaluation code, submitted models and datasets are key contributions, enabling comprehensive and reproducible evaluation, as well as previously impossible longitudinal studies. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.
1. Introduction
Audio representations have received less broad evaluation than representations in language and vision, making it difficult to choose one for new audio tasks. HEAR addresses this gap with a common, reproducible benchmark spanning diverse tasks and domains.
- Broad evaluation is needed because good representations support machine learning and broad suites help prevent overfitting to common test sets.
- Audio representations are rarely evaluated across a broad range of problems, so choosing one for a novel task is difficult.
- HEAR encourages flexible audio representations, reveals how they generalize, and supports faster development for model developers and users.
- HEAR evaluates nineteen tasks across speech, environmental sound, and music, requiring one representation to solve them all.
- Evaluation uses classification and temporal onset-detection tasks, with embeddings passed to a simple generic predictor.
2. Background on representation learning
Audio representation learning maps raw or transformed audio into features useful for downstream discrimination. Approaches range from hand-crafted signal transforms to deep, supervised, self-supervised, multimodal, and cross-modal methods.
- A learned representation maps an input’s low-level form to a numerical vector that supports discriminative downstream tasks.
- A good representation should transfer across many tasks while requiring limited supervision.
- Audio inputs may be represented using log-Mel spectrograms, MFCCs, constant-Q transforms, scattering transforms, learned filterbanks, or deep architectures.
- Common architectures combine encoders with optional temporal modeling or generative decoders, using CNNs, RNNs, or Transformers.
- Training regimes include supervised learning, multitask supervision, self-supervision on unlabeled data, multimodal correspondence, and distillation from models in other modalities.
- Frozen intermediate outputs can serve as embeddings and improve over raw features without full fine-tuning.
3. HEAR: Holistic Evaluation of Audio Representations
HEAR is a reusable, open benchmark designed to test whether audio representations generalize across diverse domains, task types, and supervision conditions. It standardizes data and interfaces so models can be compared reproducibly without fine-tuning.
- HEAR aims to evaluate audio representations across everyday domains, audio phenomena, short and long time spans, and few-label settings.
- Unlike mainly speech-focused benchmarks, HEAR complements prior shared tasks by covering broader audio applications through reusable generic APIs.
- HEAR wraps existing models through a common API and supports scene-based classification and timestamp-based sound-event detection.
- Frozen embeddings keep downstream evaluation framework-independent, whereas fine-tuning can improve performance but increases training time.
- Timestamp-based tasks convert audio into framewise sequence labeling and postprocess predictions into events with start and end times.
- Datasets are standardized through common sample rates, fixed clip lengths, predefined splits, common packaging, and human-readable metadata.
- The shared challenge released open tasks including Speech Commands v2, NSynth Pitch, and DCASE 2016 Task 2.
4. Models evaluated
HEAR evaluates baseline and external-team models spanning speech, music, environmental audio, multimodal learning, and varied architectures and training regimes. The submissions differ in inputs, pretraining data, fusion strategies, and embedding construction.
- HEAR reports three strong baseline models and submissions from thirteen external teams.
- The evaluated models include CNNs, transformers, conformers, MobileNets, EfficientNets, and sequential gated MLPs operating on one- or two-dimensional audio representations.
- Pretraining spans speech, pitch tracking, AudioSet tagging, multimodal audio-video correspondence, self-supervised learning, music, and cross-modal CLIP distillation.
- The challenge encountered dependency conflicts across TensorFlow, CUDA, CuDNN, and PyPI packages despite requiring installable submissions.
- CP-JKU PaSST uses a spectrogram transformer pretrained on AudioSet, with variants concatenating longer windows or raw mel spectrograms for timestamp embeddings.
- NTU-GURA combines HuBERT, wav2vec2, and CREPE through averaging or concatenation, optionally fusing multiple model layers.
5. Results and Discussion
HEAR reveals task-specific strengths and uneven generalization across 29 models and 19 tasks, with no simple performance pattern spanning all audio domains. Results also show that specialized representations can dominate particular task families, while several difficult tasks remain poorly explained.
- Cross-task patterns: Normalized task scores and correlation analyses expose substantial differences among models and tasks, making a single aggregate score insufficient.Scores are standardized per task and Winsorized to limit the influence of extreme values during inter-task comparison.
- Pitch: Models incorporating CREPE embeddings perform best on the NSynth pitch and Maestro tasks because these tasks require pitch modeling.OpenL3 and wav2vec2 also represent pitch to some degree despite emphasizing semantic discrimination or speech.
- Broad-domain semantic-object tagging: CP-JKU PaSST models achieve a no-fine-tuning state of the art on FSD50K with a mean average precision of 0.641.FSD50K, ESC-50, and GTZAN results are strongly correlated, and smaller-task results predict performance on FSD50K.
- Vocals: FSD50K scores resemble Vocal Imitations and LibriCount, while strong speech and PaSST models lead Vocal Imitations and SERAB BYOL-S leads LibriCount.The passage associates these similarities with broad vocalizations and simultaneous event detection, respectively.
- Speech: Strong speech models perform best on speech-related tasks spanning environmental, paralinguistic, and semantic characteristics.These tasks include LibriCount, Vocal Imitations, CREMA-D emotion detection, VoxLingua107 language identification, and Speech Commands.
- Unresolved generalization: Several tasks defy neat categorization, including low-resource, non-Western music, and unusual sound tasks, leaving no simple pretraining-data explanation.The authors identify this as a negative result and describe existing model boundaries as fragile and unpredictable.
6. Conclusion
HEAR establishes an open, standardized basis for reproducible evaluation of audio representations across diverse domains and tasks. Its broad results expose difficult cases and leave open whether one general-purpose representation can match the holistic capabilities of human hearing.
- Openness and reproducibility: Open datasets, submissions, and evaluation code enable reproducible comparisons and previously impossible longitudinal studies.HEAR standardizes datasets and APIs while making models and evaluation resources broadly accessible.
- Benchmark scope: HEAR evaluates 29 models on 19 diverse tasks spanning speech, environmental sounds, music, and cross-domain datasets.The standardized suite supports comprehensive and reproducible evaluation across varied audio problems.
- Scientific contribution: HEAR exposes what the community does not yet know about which audio tasks and representations generalize predictably.The benchmark is intended to probe difficult cases rather than assume task difficulty in advance.
- Open question: Whether one general-purpose audio representation can perform as holistically as the human ear remains an open question.The conclusion presents this as HEAR’s central unresolved question.
Appendix A. Evaluation Tasks
HEAR evaluates frozen audio representations on 19 diverse tasks derived from 16 datasets, spanning speech, environmental sound, music, and novel task settings. The appendix defines each task’s data, prediction objective, splits, and evaluation measures.
- The benchmark comprises 19 tasks derived from 16 datasets, with tasks marked novel when they are not comparable to prior literature.
- Speech and pitch: Speech Commands evaluates known-command classification, while NSynth Pitch classifies instrumental sounds into 88 pitches using 5-hour and 50-hour variants.NSynth additionally reports pitch and chroma accuracy and is treated as classification rather than regression.
- Environmental sound: The benchmark covers environmental and audio-scene tasks including ESC-50 classification, FSD50K multilabel tagging, Beehive States classification, and DCASE office sound-event detection.FSD50K contains approximately 51,000 clips labeled across 200 AudioSet classes, while Beehive States uses 10-minute recordings.
- Additional tasks: Other tasks measure speaker counting, vocal-imitation matching, spoken-language identification, gunshot microphone-location classification, and speech emotion recognition.Gunshot Triangulation is the smallest HEAR dataset, with 88 clips across seven folds.
- Music: Music tasks include GTZAN Genre, GTZAN Music Speech, MAESTRO 5h transcription, Beijing Opera Percussion, and Mridangam Stroke and Tonic classification.MAESTRO transcription uses timestamp-based embeddings, with note-onset metrics defined in the original task literature.
Appendix B. Downstream training details
HEAR trains generic downstream predictors on frozen model embeddings and summarizes task and model relationships using normalized scores. The procedure supports reproducible model comparison through fixed evaluation choices, validation monitoring, and visual correlation analyses.
- Downstream predictors: For each task, a downstream MLP classifier is trained using a model’s frozen embeddings as input features.Scene-based multiclass tasks use a softmax output with cross-entropy loss, while multilabel tasks use sigmoid outputs with cross-entropy loss.
- Summary analyses: Figure 2 uses t-SNE to visualize task and model similarity from normalized scores, imputing missing values with sklearn’s multivariate IterativeImputer.
- Summary analyses: Figures 3 and 4 display the highest and lowest task-task or model-model correlations, sorting cells to minimize traveling-salesperson distance.The displayed correlations map [-1, +1] to distances [+2, 0].
- Training procedure: Validation scores, rather than loss, guide early stopping and model selection, with a maximum of 500 epochs and checks every three epochs.Timestamp-task validation uses sed_eval with median filtering and minimum event durations of 125 ms and 250 ms.