Source-linked AI summary
Audio Retrieval with Natural Language Queries: A Benchmark Study
A. Sophia Koepke, Andreea-Maria Oncescu, João F. Henriques, Zeynep Akata, Samuel Albanie
TL;DR
Natural-language audio retrieval has received limited attention, despite offering a way to search sound using detailed descriptions. This paper introduces three benchmarks and adapts video-retrieval models to audio; the authors report that multiple audio experts improve retrieval and that the adapted methods are relatively well-suited to the task, though room for improvement remains.
Problem
Natural-language audio retrieval has received limited attention, while unstructured audio is not readily searchable through the text-query interface used for other media.
Source: §I · p. 1Method
The paper introduces SoundDescs and benchmarks based on SoundDescs, AudioCaps, and Clotho, then adapts multimodal video-retrieval models for audio retrieval.
Source: §I · p. 2Results
The authors report that an ensemble of pretrained audio experts improves audio retrieval and that adapted video-retrieval methods are relatively well-suited, though there is room for improvement.
Source: §I · p. 2, §VI · p. 9Takeaways & Limitations
The authors hope the benchmarks will facilitate future audio-search development and make more produced media available for public use.
Source: §VI · p. 9SoundDescs descriptions are of mixed quality because they were obtained automatically from descriptions supplied with the data.
Source: §I · p. 2
Abstract
from arXiv · showhide
The objectives of this work are cross-modal text-audio and audio-text retrieval, in which the goal is to retrieve the audio content from a pool of candidates that best matches a given written description and vice versa. Text-audio retrieval enables users to search large databases through an intuitive interface: they simply issue free-form natural language descriptions of the sound they would like to hear. To study the tasks of text-audio and audio-text retrieval, which have received limited attention in the existing literature, we introduce three challenging new benchmarks. We first construct text-audio and audio-text retrieval benchmarks from the AudioCaps and Clotho audio captioning datasets. Additionally, we introduce the SoundDescs benchmark, which consists of paired audio and natural language descriptions for a diverse collection of sounds that are complementary to those found in AudioCaps and Clotho. We employ these three benchmarks to establish baselines for cross-modal text-audio and audio-text retrieval, where we demonstrate the benefits of pre-training on diverse audio tasks. We hope that our benchmarks will inspire further research into audio retrieval with free-form text queries. Code, audio features for all datasets used, and the SoundDescs dataset are publicly available at https://github.com/akoepke/audio-retrieval-benchmark.
I. INTRODUCTION
Free-form natural-language queries offer a way to search unstructured audio by detailed content rather than metadata or audio examples. The paper addresses limited audio-caption data diversity by introducing SoundDescs and three retrieval benchmarks, then adapting video-retrieval models for audio.
I. INTRODUCTION
- Free-form text queries can describe a temporal sequence of sound events, unlike a single class tag, and avoid the difficulty of providing an example audio query.Source: §I · p. 1
- SoundDescs addresses limited captioning-data size and audio diversity with varied sound descriptions and a relatively large vocabulary.The dataset was sourced from the BBC Sound Effects database.Source: §I · pp. 1–2
- The authors adapt video-retrieval frameworks and report that using an ensemble of pretrained audio experts improves audio retrieval.Source: §I · p. 2
- The paper introduces three natural-language audio-retrieval benchmarks using SoundDescs, AudioCaps, and Clotho, and provides adapted video-retrieval baselines.The authors describe these as, to their knowledge, the first public benchmarks for this task.Source: §I · p. 2
- This journal paper extends the Interspeech 2021 version with SoundDescs, dataset analysis, more extensive baseline ablations, and an additional retrieval architecture, the Multi-modal Transformer.Source: §I · p. 2
II. RELATED WORK
The related work spans audio recognition, captioning, and retrieval, alongside text-based video retrieval and cross-domain audio-text matching. The paper adapts expert-based video-retrieval architectures to audio retrieval with unconstrained natural-language descriptions.
Sound event recognition
- Sound event recognition assigns a semantic label to an audio segment, whereas this work focuses on retrieval with natural-language queries,,.Source: §II · p. 2
Audio captioning
- Audio captioning generates a natural-language sound description, and the paper uses AudioCaps and Clotho for cross-modal retrieval,,.Source: §II · p. 2
Audio-based and cross-domain retrieval
- Earlier audio-based retrieval includes query-by-example systems and methods that retrieve from associated text such as metadata or sound-event labels,,,, [48].Source: §II · p. 2
Text-based video retrieval
- Text-based video retrieval commonly embeds video and text jointly for efficient similarity computation; expert-based methods combine pretrained models specialized for different tasks [1],, [2],,,,.The paper adapts Mixture of Embedded Experts [2], Collaborative Experts, and Multi-Modal Transformer for audio retrieval.Source: §II · p. 2
Cross-domain audio retrieval
- Unlike sound retrieval using single-word tags [51] or onomatopoeias,, this work studies unconstrained text descriptions; retrieval avoids the expensive temporal annotations required for audio grounding, [63].Source: §II · p. 3
III. SOUNDDESCS DATASET
SOUNDDESCS adds 32,979 audio-description pairs from a diverse BBC sound-effects collection. Its duration range and descriptive vocabulary extend beyond those of AUDIOCAPS and CLOTHO, providing varied material for retrieval analysis.
III. SOUNDDESCS DATASET
- SOUNDDESCS contains 32,979 audio files with natural-language descriptions, sourced from BBC sound effects spanning 23 categories.The files include sounds from the BBC Radiophonic Workshop, the Blitz in London, BBC special effects, and the Natural History Unit archive.Source: §III · p. 3, A. Dataset collection · p. 3
A. Dataset collection
- The dataset proposes a random split of 70% training and 15% each for validation and testing.It includes 32,979 files with non-empty descriptions, out of 33,066 files on the BBC website.Source: A. Dataset collection · p. 3
B. Data analysis
- SOUNDDESCS spans a wider range of audio durations than AUDIOCAPS and CLOTHO, including 109 files longer than 10 minutes.Its total duration is 1,060 hours, compared with 135 hours for AUDIOCAPS; it has fewer captioned files, making it a more challenging retrieval benchmark.Source: B. Data analysis · pp. 3–4, Figure 2 · p. 3, Table I · p. 4
- For CLOTHO and SOUNDDESCS, which contain audio and descriptions with varied lengths, audio duration and description length show no strong correlation.Source: B. Data analysis · p. 4
- SOUNDDESCS descriptions have a larger vocabulary of nouns, verbs, and adjectives than AUDIOCAPS and CLOTHO, with almost 4,000 distinct nouns versus around 2,000 in each comparator.The authors link this noun diversity to a wide array of topics and the high proportion of nature sounds, whose descriptions often specify species names.Source: B. Data analysis · p. 4
C. Dataset examples
The examples visualize SOUNDDESCS description embeddings and report category clustering alongside specific, high-quality descriptions, including some that are not full sentences.
C. Dataset examples
- Averaged word2vec description vectors are embedded with t-SNE and color-coded by category; 500 samples from each of six categories are shown.The dataset contains 23 categories in total.Source: C. Dataset examples · p. 4
- The sampled descriptions cluster smoothly according to category in the t-SNE visualization.Source: C. Dataset examples · p. 4
- The descriptions are reported to be fairly specific and high quality, despite not always resembling full sentences.Source: C. Dataset examples · p. 4
D. Rights to use
SOUNDDESCS data is available for non-commercial, personal, or research use under the BBC RemArc licence, with code and URLs released for constructing the dataset and its proposed splits.
D. Rights to use
- The BBC RemArc licence permits SOUNDDESCS data use for non-commercial, personal, or research purposes.Source: D. Rights to use · pp. 4–5
- The authors released URLs and code for downloading and constructing SOUNDDESCS with the proposed train, validation, and test split.Source: D. Rights to use · p. 5
IV. METHODS, DATASETS, AND BENCHMARK
- The following section formulates text-audio retrieval as ranking audio by similarity to a written query and also considers retrieving text from audio queries.Source: §IV · p. 5
A. Methods
The methods learn shared audio-text embeddings with a contrastive ranking loss and adapt three video-retrieval architectures by replacing video experts with audio experts.
A. Methods
- The approach learns audio and text embedding functions that place matching pairs close and non-matching pairs farther apart in a shared space.Training minimizes a contrastive ranking loss based on cosine similarity.Source: A. Methods · p. 5
- MoEE and CE aggregate expert features over time with NetVLAD, while CE additionally uses collaborative gating to filter experts with an attention mask.Source: A. Methods · p. 5
- MMT refines expert embeddings through multi-headed self-attention across modalities and time steps, then computes similarity as a weighted sum across experts.Unlike MoEE and CE, it passes expert features directly into the transformer rather than temporally aggregating them first.Source: A. Methods · p. 5
- MoEE, CE, and MMT are adapted from text-video retrieval by retaining their text encoders and replacing video experts with audio experts.Source: A. Methods · p. 5
B. Datasets
The study uses three audio-centric datasets and two visual-centric datasets, with audio and text descriptions paired in different ways across them.
B. Datasets
- SoundDescs contains 23,085 training, 4,947 validation, and 4,947 test samples of BBC Sound Effects paired with free-form descriptions.Source: B. Datasets · p. 5
- AudioCaps uses AudioSet sounds; the filtered subset has 49,291 training, 428 validation, and 816 test samples.Source: B. Datasets · p. 6
- Clotho uses Freesound sounds described from audio alone; its descriptions exclude transcribed speech, and each sample has five descriptions.The study randomly split the dev set into 2,314 training and 579 validation samples; its evaluation set has 1,045 samples.Source: B. Datasets · p. 6
- ActivityNet-Captions and QueryD are visual-centric datasets whose descriptions focus primarily on video content.Source: B. Datasets · p. 6
C. Benchmark
The paper introduces SoundDescs and repurposes AudioCaps and Clotho as benchmarks for text-based audio retrieval.
C. Benchmark
- SoundDescs is introduced as a dataset for text-based audio retrieval.Source: C. Benchmark · p. 6
- AudioCaps and Clotho are repurposed from audio-captioning datasets into text-based audio retrieval benchmarks.Source: C. Benchmark · p. 6
- The approach follows precedents in vision-language research, where captioning datasets have become benchmarks for text-based video retrieval.Source: C. Benchmark · p. 6
V. EXPERIMENTS
The experiments establish baselines across three audio-retrieval benchmarks, compare audio and visual experts, and examine pre-training, query types, duration, and training scale.
Benchmark results
- MMT outperforms CE and MoEE on AudioCaps and Clotho in both retrieval directions; on SoundDescs, all three are comparable, with CE and MoEE slightly stronger.Source: Table II · p. 7, §V · p. 8
Audio-centric vs. visual-centric queries
- Performance is strongest on audio-centric SoundDescs and weakest on visual-centric ActivityNet-Captions.The experiments attribute weaker visual-centric results to query information not captured in audio; QueryD suggests audio-only video retrieval is possible at lower accuracy.Source: Table III · p. 7, §V · p. 7
Ablation study
- Audio experts significantly outperform visual experts on AudioCaps, while combining both expert types performs strongest overall.The strongest audio-only retrieval combines VGGish and VGGSound features, which the authors use in the remaining experiments.Source: Table IV · p. 7, §V · p. 7
Pre-training
- Pre-training on AudioCaps improves Clotho performance, while SoundDescs pre-training slightly boosts AudioCaps performance.SoundDescs pre-training harms Clotho performance; AudioCaps pre-training does not benefit SoundDescs, and Clotho pre-training yields negligible change on AudioCaps and SoundDescs.Source: Table V · p. 8, §V · p. 8
Speech features
- Speech features do not improve retrieval; AudioCaps captions and spoken words overlap little, with a METEOR score of 0.03.Source: §V · p. 8
Qualitative results
- Qualitative examples show semantically similar retrieved audio, while errors can occur between distinct sounds that sound alike.Examples include confusion between fire-engine and police sirens; some incorrect top-ranked results still sound reasonable.Source: §V · p. 8, Figure 6 · p. 8
Influence of audio segment duration
- Retrieval is slightly weaker for longer SoundDescs audio segments than for clips under 30 seconds, but remains solid beyond 120 seconds.Source: §V · p. 8, Figure 7 · p. 9
Influence of training scale
- SoundDescs retrieval performance increases monotonically as more training data is used.The authors report clear room for improvement through collecting additional training data.Source: §V · p. 9, Figure 7 · p. 9
VI. CONCLUSION
The paper introduces SoundDescs, proposes retrieval benchmarks using it alongside Clotho and AudioCaps, and reports baselines from adapted multimodal video retrieval methods.
VI. Conclusion
- The adapted methods are described as relatively well-suited to audio retrieval, though the underexplored task still has room for improvement.The authors hope the benchmarks will facilitate future audio search engines and broader public use of produced media.Source: §VI · p. 9
APPENDIX
The appendix adds description-level analyses of SoundDescs, including sentence lengths and distributions of unique nouns and verbs. It reports more descriptions without nouns and/or verbs than in AudioCaps and Clotho, with some of those counts explained by tagging and description content.
APPENDIX
- Because SOUNDDESCS descriptions can contain multiple sentences, the appendix also reports their sentence-length distribution.Source: APPENDIX · p. 9, Figure 8 · p. 9
- SOUNDDESCS has noticeably more descriptions with zero unique nouns and/or verbs than AudioCaps and Clotho.Some bird species are tagged as proper nouns rather than nouns; descriptions without verbs may be mistagged or may describe sounds without actions.Source: APPENDIX · p. 9, Figure 9 · p. 9, Figure 10 · p. 9
- Figures 9 and 10 show unique noun and verb counts per description, with zero indicating none were present.Source: Figure 9 · p. 9, Figure 10 · p. 9