Source-linked AI summary
Clotho: An Audio Captioning Dataset
Konstantinos Drossos, Samuel Lipping, Tuomas Virtanen
TL;DR
Audio captioning requires describing general audio content in free text, but existing datasets can constrain perceptual diversity and introduce contextual or domain biases. The paper presents Clotho, a diverse audio-caption dataset with a baseline method, and reports that the baseline began learning audio content but still needed tuning to express it properly. Clotho is designed to support training and evaluation through diverse captions and data splits that do not hamper those processes.
Problem
Audio captioning needs free-text descriptions of general audio content, while prior datasets can limit perceptual diversity through contextual information, domain specificity, and few captions per file.
Method
The paper constructs Clotho from Freesound audio with five audio-only crowdsourced captions per sample and evaluates it with a baseline encoder-attention-decoder method.
Results
The baseline began learning input-audio content, but its sentence structure and word order were incorrect and required further tuning.
Takeaways & Limitations
Clotho provides a freely available dataset designed to promote caption diversity, remove unique words and named entities, and avoid splits that hamper training or evaluation.
Abstract
from arXiv · showhide
Audio captioning is the novel task of general audio content description using free text. It is an intermodal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal. In this paper we present Clotho, a dataset for audio captioning consisting of 4981 audio samples of 15 to 30 seconds duration and 24 905 captions of eight to 20 words length, and a baseline method to provide initial results. Clotho is built with focus on audio content and caption diversity, and the splits of the data are not hampering the training or evaluation of methods. All sounds are from the Freesound platform, and captions are crowdsourced using Amazon Mechanical Turk and annotators from English speaking countries. Unique words, named entities, and speech transcription are removed with post-processing. Clotho is freely available online (https://zenodo.org/record/3490684).
1 Introduction
Audio captioning describes human-perceived information in general audio using free text rather than speech transcription. Clotho addresses limitations in prior datasets by emphasizing perceptual diversity, multiple captions, audio-only annotation, and broad non-domain-specific data.
- Audio captioning translates human-perceived information in general audio into free-text descriptions rather than speech transcripts.Relevant information can include sound events, acoustic scenes, source relationships, foreground and background distinctions, and object or environmental properties.
- Dataset design strongly influences what audio captioning methods can learn, while diverse captions capture perceptual differences and support more objective evaluation.Multiple captions for one audio signal help represent alternative perceptions of ambiguous sounds.
- Prior datasets introduce limitations through domain-specific content, translated or speech-focused annotations, visual context, and limited caption multiplicity.These factors can bias descriptions toward what annotators see, hear in speech, or know from contextual labels.
- Clotho annotation exposes annotators only to the audio signal, avoiding video or word-tag information during caption creation.This design targets descriptions of annotators’ perceived audio content rather than descriptions shaped by external context.
- Clotho contains 4981 audio samples and 24 905 captions, with five captions per 15-to-30-second sample and eight-to-20-word captions.The dataset is built from Freesound audio and captions collected through Amazon Mechanical Turk using an annotation protocol intended to promote diversity and reduce grammatical errors.
2 Creation of Clotho dataset
Clotho is constructed by filtering and balancing Freesound audio, collecting and sanitizing diverse captions, and designing splits that distribute vocabulary across development and evaluation data.
- Audio data collection and processing: 12,000 Freesound files are initially sampled using lossless audio, quality, duration, metadata, and content-tag criteria.The initial pool excludes files tagged as music, sound effects, or speech.
- Audio data collection and processing: Maximum-entropy sampling produces Xmed with the most uniform possible tag distribution and therefore the most diverse content.The ten most common tags include ambient, water, nature, birds, noise, rain, city, wind, metal, and people.
- Captions collection and processing: Each audio sample receives captions from multiple annotators through a three-step process of generation, correction or rephrasing, and accuracy-and-fluency scoring.The top Ncp captions are selected after sorting by accuracy and then fluency.
- Captions collection and processing: Post-processing removes speech transcription, named entities, and singleton words while retaining captions whose words occur at least twice in the corpus.Captions that cannot meet the minimum length or content requirements can lead to complete audio-sample removal.
- Data splitting: The dataset is split into non-overlapping development, evaluation, and testing sets of 60%-20%-20%, with vocabulary represented in development and at least one other split.The final split contains 2893 development, 1045 evaluation, and 1043 testing audio samples; the testing split is withheld.
3 Baseline method and evaluation
The baseline uses an attention-based encoder-decoder to generate captions from log mel-band audio features and evaluates them with standard translation metrics. It identifies some audio content, but sentence structure and word order remain incorrect.
- Baseline architecture: The baseline maps 64 log mel-band energy sequences to captions with an attention-based encoder-decoder.Its encoder uses three bidirectional GRU layers, followed by attention, a GRU decoder, and a softmax word classifier.
- Input representation: 64 log mel-band energies are extracted with a 46 ms Hamming window and 50% overlap before caption generation.
- Baseline architecture: Five captions per audio sample serve as target outputs during joint optimization of the encoder, attention mechanism, decoder, and classifier.Training uses 150 epochs, cross-entropy loss, and Adam optimization.
- Evaluation: BLEU_n, METEOR, CIDEr, and ROUGE_L compare generated captions with reference captions on the evaluation and testing splits.The metrics respectively assess n-gram precision, unigram precision-recall balance, n-gram similarity, and longest-common-subsequence similarity.
- Evaluation: The baseline identifies words related to audio content, but its sentence structure and word order are not yet correct.The paper suggests pre-calculated or jointly learned language models as ways to address these issues.
4 Conclusions
The paper introduces Clotho as a diverse audio captioning dataset and demonstrates its use with a baseline method. The baseline begins learning audio content, while further tuning is needed for proper expression.
- Dataset contribution: Clotho contains 4981 audio samples and five captions per file, totaling 24 905 captions.
- Dataset contribution: The dataset construction promotes caption diversity, removes singleton words and named entities, and provides splits designed not to hamper training or evaluation.
- Baseline findings: Baseline results indicate that the method learned input-audio content but requires more tuning to express that content properly.