Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering

I. INTRODUCTION

• Natural-language questions can target audio contents and express high-level information such as structure, repetitions, and event order.Answering them requires inferring signal information relevant to the question.Source: §I · p. 1
• Question answering datasets have largely covered visual, video, and textual modalities, with crowdsourcing used in those settings and in other audio-to-text tasks.Source: §I · p. 1
• Earlier AQA datasets CLEAR and DAQA used programmatically generated questions and answers, which the authors say may lack the diversity and challenges of real data.CLEAR uses fixed-length musical-note sequences; DAQA uses variable-length generic sound events.Source: §I · p. 1
• The authors present Clotho-AQA as the first crowdsourced AQA dataset, with six questions and three answers per question for each of 1,991 Clotho audio files.Question and answer annotators are different; question annotators receive only audio, while answer annotators receive the audio and question.Source: §I · p. 1
• The paper also presents multimodal binary and multi-class classifier baselines for yes/no and other single-word answers.Source: §I · p. 1

II. CREATION OF CLOTHO-AQA DATASET

• 1,991 audio files were selected at random from Clotho for the dataset.Source: §II · p. 1
• The clips are 15–30 seconds long and cover everyday environmental sounds, while excluding music, speech, and sound effects.The recordings come from Freesound.Source: §II · p. 2

A. Question and Answer Gathering

• AMT annotators had to meet task-approval and English-speaking-location criteria, and question annotators passed a custom qualification test.The criteria included at least 3,000 approved tasks and a 95% approval rating.Source: A. Question and Answer Gathering · p. 2
• For each audio, annotators create two sets of three questions: one yes-answerable, one no-answerable, and one with another single-word answer per set.The second set's annotators are asked to avoid questions similar to those in the first set.Source: A. Question and Answer Gathering · p. 2
• Manual screening checked question type, excluded speech-content questions, and removed questions that disclosed answers or addressed a listener directly.Source: A. Question and Answer Gathering · p. 2
• Three annotators answer each question after receiving its audio and text, producing 18 question-answer pairs per audio; answerers also report confidence.Source: A. Question and Answer Gathering · p. 2
• Because one-off answer words would occur in only one split, in-house annotators replaced them with suitable answers used for other audio files.They corrected typographical errors or chose a semantically close alternative; the process reduced unique answers from 1,889 to 830, with every final answer appearing at least twice.Source: A. Question and Answer Gathering · p. 2

B. Data Splitting

• The 60%-20%-20% split keeps each audio file in one split and places each answer in training and in validation or testing.Audio files are stratified by their sets of answer words using multilabel stratification.Source: B. Data Splitting · p. 2
• 1,174, 344, and 473 audio files form the training, validation, and test splits, with 830, 512, and 801 unique answers, respectively.The split was selected through a two-stage search: 2,000 60%-40% splits were generated, the 50 closest to ideal word-frequency allocation were further split, and the closest final split was selected.Source: B. Data Splitting · p. 2
• Yes and no are the most frequent answers across the final splits; frequent single-word answers include water, rain, one, two, three, and birds.Source: B. Data Splitting · p. 2

• Two baseline experiments demonstrate the use of Clotho-AQA for audio question answering.Source: §III · p. 3
• A binary classifier evaluates yes/no questions, which constitute two-thirds of the dataset.Source: §III · p. 3
• A multiclass classifier predicts among 828 single-word answers.Source: §III · p. 3

A. Baseline Methods

• Separate audio and question branches produce fixed-size representations that are concatenated and passed through fully connected layers for classification.The output is either yes/no or one of 828 single-word answers.Source: A. Baseline Methods · p. 3
• OpenL3 converts environmental-sound mel-spectrograms into 512-dimensional audio embeddings, which two bidirectional LSTM layers process to learn temporal relationships.The embeddings are extracted at 0.1-second hop size.Source: A. Baseline Methods · p. 3
• The audio representation is the final time step of the last bidirectional LSTM layer, yielding a fixed-size vector.Source: A. Baseline Methods · p. 3
• Questions are represented using pretrained FastText word vectors and processed by two bidirectional LSTM layers into a fixed-size representation.The binary classifier uses hidden size 128 and dropout 0.2; the multiclass classifier uses hidden size 512.Source: A. Baseline Methods · p. 3
• After concatenation, task-specific dense layers feed a classification layer with one binary output neuron or 828 softmax outputs.The binary dense layers have 256 and 128 neurons, while the multiclass layers have 1024 neurons each.Source: A. Baseline Methods · p. 3

B. Evaluation

• For binary classification, the Clotho-AQA splits contain 1,174 training, 344 validation, and 473 testing audio files, each with 12 yes/no question-answer pairs.Source: B. Evaluation · p. 3
• Binary evaluation uses 12 yes/no question-answer pairs per audio file, while multiclass evaluation uses six single-word-answer pairs per file.Source: B. Evaluation · p. 3
• The binary classifier is evaluated on unfiltered responses, unanimous responses, and majority-vote labels to examine annotator disagreements.Source: B. Evaluation · pp. 3–4
• Audio-only and question-only models are also trained to assess the information captured from each modality.Source: B. Evaluation · p. 4
• Models use Adam at a 0.001 learning rate for 100 epochs with cross-entropy loss, and the best validation-scoring model is tested.Source: B. Evaluation · p. 4

IV. RESULTS

• Question-only models perform as well as multimodal models for both binary yes/no and single-word multiclass classification.The binary model performs better when annotators answer unanimously than when they disagree.Source: §IV · p. 4, Table I · p. 4, Table II · p. 4
• The single-word classifier results indicate that the model is starting to learn relationships between multimodal data.Source: §IV · p. 4, Table II · p. 4
• The authors suggest strong language priors, a shallow baseline network, or poor pretrained audio embeddings as possible explanations for question-only parity.They state that future work will analyze the reasons and improve the data and model.Source: §IV · p. 4

• The question-only model produces equally good results as the multimodal input model.Source: §V · p. 4
• Clotho-AQA contains 1,991 Clotho audio files, each with six questions and three answers per question collected through AMT.Source: §V · p. 4
• Questions are answerable with a single word or yes/no, and answers come from three annotators per question.Source: §V · p. 4
• The authors post-process answers by replacing unique words with commonly occurring suitable answers and provide binary and multi-class baselines.Source: §V · p. 4
• The authors intend to analyze the model results and improve the data and model; they propose the dataset for developing AQA methods.Source: §V · p. 4