PaperlayerView exact revision on arXiv

Source-linked AI summary

SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval

Zineb Lahrichi, Marc Ferras, Gaël Richard, Geoffroy Peeters

arXiv:2609.02343v1cs.SDcs.CLcs.MMeess.AS

TL;DR

Existing audio-captioning datasets are limited by low semantic diversity, generic and acoustically shallow descriptions, and one-to-many auditory ambiguity. SonicCaps addresses these gaps with a multimodal, multi-stage captioning pipeline that generates diverse captions per audio, and training with those captions consistently improves retrieval and zero-shot classification while strengthening alignment with human judgments.

  • Problem

    Existing audio-captioning datasets have limited semantic diversity, generic descriptions lacking acoustic detail, and one-to-many mappings that do not reflect auditory ambiguity.

  • Method

    SonicCaps uses multimodal Qwen3-Omni recaptioning, prompt engineering, and few-shot generation to produce around 24 diverse captions per audio clip.

  • Results

    Diverse-caption training consistently improves audio retrieval and zero-shot classification across public and proprietary benchmarks, while higher-quality captions improve alignment with human judgments.

  • Takeaways & Limitations

    Enhancing and diversifying textual supervision strengthens cross-modal alignment and improves both retrieval performance and perceptual alignment in audio-text learning.

Abstract

from arXiv · show

Recent advances in audio-language modeling have been driven by large-scale audio captioning datasets. However, existing datasets remain limited by low semantic diversity, generic descriptions lacking acoustic details, and one-to-one audio-caption mappings that poorly reflect the inherent ambiguity of auditory perception. We introduce SonicCaps, a large-scale audio captioning dataset comprising ~15M captions paired with ~700k audio clips, generated using a multi-modal large language model (Qwen3-Omni) conditioned on both audio and text. To explicitly promote diversity, we generate around 24 captions per audio via structured prompt engineering and few- shot generation, spanning main descriptions, rephrased variants (verbosity, style) and semantic tags. Human evaluation shows that SonicCaps is rated significantly higher than existing captioning datasets, with fine-grained analyses indicating that our captions are perceived as more descriptive and precise, which strongly correlates with quality judgments. Finally, training CLAP models on SonicCaps with a multi-caption sampling strategy consistently improves audio retrieval and zero-shot classification, with stronger generalization across public and commercial benchmarks. We release both SonicCaps and two specialized CLAP models on hugging face: https://huggingface.co/datasets/Zineb/SonicCaps.

I. INTRODUCTION

SonicCaps addresses limited semantic diversity, generic and acoustically shallow captions, and one-to-many ambiguity by generating diverse, fine-grained captions from audio with multimodal prompting. The dataset and diverse-caption training strategy improve caption quality, retrieval, and zero-shot classification.

  • Motivation: Existing audio-language datasets suffer from limited semantic diversity, generic descriptions, and one-to-many audio-caption ambiguity.These limitations reflect both heterogeneous data distributions and the inherent ambiguity of auditory perception.
  • Dataset: SonicCaps contains ~15M captions paired with ~700k audio clips, generated through fidelity-focused recaptioning, diversity-focused recaptioning, and post-processing.The dataset is built with a multimodal large language model and includes only captions and corresponding audio identifiers in the released data.
  • Caption diversity: Around 24 captions per audio span multiple styles and levels of granularity through prompt engineering and few-shot generation.This strategy is designed to reflect the one-to-many nature of auditory perception.
  • Results: Diverse-caption sampling from SonicCaps consistently improves audio retrieval and zero-shot classification over existing baselines.The reported gains hold across public and proprietary benchmarks, indicating strong generalization.
  • Evaluation: A human evaluation framework compares captions pairwise across several qualitative dimensions and collects structured feedback for each rating.The evaluation is used to assess descriptive quality and perceptual detail.

II. RELATED WORKS

Prior audio-captioning datasets improved scale, annotation, and caption quality, but remained constrained by annotation cost, limited diversity, filtering losses, and relatively homogeneous source distributions. These gaps motivate captioning approaches that emphasize diversity alongside quality.

  • Human annotation: Human-written datasets such as AudioCaps and Clotho remain limited in scale and diversity because annotation is costly and variable across annotators.These constraints are especially pronounced for ambiguous or specialized sound events.
  • Automated collection: Automated datasets expanded scale, but some resources provide category labels rather than natural-language descriptions.VGGSound is presented as a large-scale audio-visual benchmark limited to classification categories.
  • Automated captioning: WavCaps uses ChatGPT-assisted captioning and filtering over heterogeneous sources, but filtering discards approximately half of the FreeSound data.Its text-conditioned generation can also propagate errors from low-quality raw captions.
  • Audio-aware captioning: Audio-aware methods such as AF-AudioSet and Auto-ACD produce informative captions and improve downstream performance through multimodal or multimodel processing.These approaches combine audio-language or audio-visual signals with filtering or LLM-based integration.
  • Open gap: Existing methods primarily emphasize caption quality or audio-data scaling rather than diversity and remain centered on relatively homogeneous distributions.The paper positions semantic diversity as an underdeveloped dimension of dataset design.

B. Human Perception based models

The paper situates its human-perception work within efforts to improve agreement between audio-text models and subjective judgments, while drawing on diverse audio sources with varied annotation quality. WavCaps provides a large-scale comparison point but retains filtering, detail, and diversity limitations.

  • Human perception alignment: Prior work seeks better alignment between CLAP-based similarity scores and subjective human judgments through evaluation protocols, comparative analyses, and preference-based training.The paper proposes training CLAP on fidelity-focused captions to improve perceptual alignment.
  • FreeSound: FreeSound contributes a large, diverse repository spanning music, environmental sounds, and synthesized effects, but its contributor descriptions and tags can be noisy.The source includes recordings under Creative Commons licenses and annotations that may be loosely related, ungrammatical, or incomplete.
  • AudioCaps: AudioCaps provides human-written descriptions for roughly 50k audio excerpts sampled from 527 AudioSet event categories.Each training example is paired with a single textual annotation.
  • BBC Sound Effects: BBC Sound Effects contains over 33k recordings across natural, urban, mechanical, and atmospheric sounds, with informative captions that may include grammatical inconsistencies and non-generic details.Examples of non-generic details include city names, landmarks, and dates.
  • WavCaps: WavCaps scales to over 400k audio-caption pairs but filters noisy conditioning data, discards approximately half of FreeSound, and often produces generic, weakly diverse descriptions.Its captions are generated from raw captions using unimodal text-to-text generation.

B. Multi-modal Audio Captioning

The proposed multimodal captioning pipeline uses three stages to produce four complementary caption types, beginning with fidelity-focused descriptions and extending them through diversity-oriented recaptioning. Qwen3-Omni is guided by explicit prompting intended to improve faithfulness and linguistic diversity.

  • Pipeline: The pipeline comprises three stages leading to four complementary caption types.The first category is fidelity-focused main captions, while the diversity-focused stage produces additional variants.
  • Fidelity-focused captions: Fidelity-focused main captions are factual, descriptive, and perceptually grounded descriptions intended to remain faithful to the audio.They form the first caption category in the pipeline.
  • Multimodal generation: Qwen/Qwen3-Omni-30B-A3B-Instruct generates captions from 16 kHz audio truncated to a maximum duration of 10 seconds.Generation uses stochastic decoding with temperature 0.6, top-p 0.95, top-k 20, and a maximum length of 30 tokens per caption.
  • Prompt engineering: Prompt refinement uses explicit dos and don’ts to discourage generic templates and improve faithfulness and linguistic diversity.The instructions encourage captions to begin with a verb in the -ing form, a noun, or an adjective.
  • Prompt engineering: The prompting instructions act as soft constraints rather than strict guarantees, although they empirically reduce undesired generation patterns.The authors report improvements in caption faithfulness under these instructions.

2) Diversity-focused recaptioning:

SonicCaps uses few-shot prompting conditioned on both audio and main captions to generate diverse rephrasings, concise queries, and semantic tags. Post-processing translates Chinese text and removes regenerated captions containing proper nouns.

  • Diversity-focused recaptioning: Few-shot prompting generates stylistically diverse rephrased captions for the same semantic concept.The rephrased setting targets approximately ten captions per audio with a length distribution comparable to main captions.
  • Diversity-focused recaptioning: Generation is conditioned jointly on the main caption and corresponding audio input.
  • Diversity-focused recaptioning: Approximately ten concise rephrased-short captions and three-word noun-focused tags are generated per sample.Rephrased-short captions resemble user queries, while tags emphasize semantic entities over actions.
  • Post-processing: Chinese captions are translated into English, while captions containing proper nouns are regenerated with those entities prohibited.
  • Post-processing: Existing automatic metrics remain limited because LAION-CLAP scores do not reliably reflect perceived caption quality.

C. Dataset overview

SonicCaps combines large scale with multiple caption types and increased linguistic diversity, then evaluates caption sampling and CLAP training under controlled retrieval experiments. The dataset retains residual redundancy for acoustically similar sounds, while its evaluation uses public and internal benchmarks.

  • Dataset overview: SonicCaps contains approximately 700k audio clips and 15M audio-caption pairs, with statistics reported for its fidelity-focused main subset.
  • Dataset overview: Residual redundancy remains among acoustically similar samples, suggesting limits in Qwen3-Omni's temporal and spectral resolution.
  • Dataset overview: SONICCAPS produces slightly longer captions than AudioCaps and WavCaps, with main captions longest and detail decreasing across rephrased, rephrased-short, and tags.
  • Dataset overview: Rephrased and rephrased-short expand vocabulary to 41K and 47K unique words versus 26K for main.
  • Dataset overview: Main captions reach 92.4% uniqueness, compared with 86.9% for rephrased and 52.2% for rephrased-short and tags.
  • Dataset overview: CLAP variants sample captions from baseline and SonicCaps subsets, using fixed encoders and independently optimized weights for each experiment.
  • Dataset overview: Retrieval evaluation reports R@5 and R@10 on AudioCaps validation data and three internal sound-effects datasets.
  • Dataset overview: Caption sampling perplexity measures effective caption exposure from source probabilities and candidate multiplicities, not semantic diversity.

B. Subjective Evaluation

The subjective evaluation compares captions from several datasets using pairwise judgments, MOS ratings, qualitative feedback, and confidence reports. Its criteria emphasize faithful, plausible, and richly detailed descriptions while accommodating multiple valid descriptions of one audio signal.

  • Subjective Evaluation: The perceptual study evaluates 60 samples drawn from FreeSound, AudioSet, AudioSet Strongly Labeled, and SonicCaps caption sources.
  • Subjective Evaluation: Participants compare caption pairs using completeness, correctness, plausibility, and qualitativeness criteria.
  • Subjective Evaluation: The evaluation prioritizes subjective plausibility because one audio signal can support multiple faithful linguistic descriptions.
  • Subjective Evaluation: Captions receive 1-to-5 MOS ratings alongside checkbox feedback, optional free-text comments, and confidence scores.
  • Subjective Evaluation: Table VI organizes retrieval and ablation results by caption-dataset composition and sampling probabilities across baseline and SonicCaps subsets.

C. Zero-shot Audio Classification

Zero-shot classification embeds audio and textual class labels in a shared space and predicts the class with highest cosine similarity. Evaluation compares SonicCaps-trained models with baselines on ESC-50, whose overlap with training data may inflate estimates.

  • Zero-shot Audio Classification: Zero-shot classification predicts the class whose text embedding has the highest cosine similarity with the audio embedding.
  • Zero-shot Audio Classification: Three SonicCaps-trained CLAP models are compared with LAION-CLAP and an AudioCaps-plus-WavCaps model using R@1 and R@5.
  • Zero-shot Audio Classification: ESC-50 contains 2k five-second clips across 50 classes, but overlaps with FreeSound-derived training data and may inflate performance estimates.

V. RESULTS

Retrieval improves as caption diversity increases, with mixed baseline behavior and the strongest results from combining complementary caption sources across the full SONICCAPS collection.

  • LAION-CLAP performs better on Commercial-Val, while OursAC+WC is stronger on several AudioCaps-Val retrieval settings.The authors attribute the discrepancy to LAION-630k's larger and more diverse training distribution, as a hypothesis.
  • Combining baseline and SONICCAPS main captions consistently outperforms the preceding baselines, showing complementary annotation sources are more effective together.Ours(3) and Ours(4) use sampling ratios of (0.8, 0.2) and (0.2, 0.8).
  • Adding rephrased-short, tags, and fine-grained rephrased captions improves retrieval individually and further improves it when combined.The experiments compare progressively richer mixtures of linguistic complexity and description granularity.
  • Ours(8) and Ours(9), using the complete SONICCAPS collection, achieve the best overall retrieval performance for equally weighted sources.
  • Caption diversity, rather than caption quality alone, strengthens cross-modal alignment without requiring additional audio data.The models use approximately the same amount of audio data as LAION-CLAP.
  • Caption sampling perplexity tracks effective captions per audio, and retrieval rises consistently across both evaluation datasets and T2A/A2T tasks.WavCaps and LAION-CLAP correspond to perplexity 1; the figure uses standard single-caption A2T evaluation on AudioCaps-Val.

B. Subjective Evaluation

Subjective evaluation favors SONICCAPS captions, which receive the highest preference scores and fewer negative observations, while CLAP–MOS agreement is assessed using pairwise and absolute formulations.

  • SONICCAPS achieves the highest preference scores, outperforming AudioCaps, raw FreeSound descriptions, and WavCaps captions.The study involved 25 participants and 375 assessments, with low-confidence and sparsely rated items excluded.
  • Approximately 50% of SONICCAPS captions receive no negative observations, compared with less than 25% for the other sources.Globally unrelated captions occur exclusively for WavCaps and FreeSound Raw, while AudioCaps has markedly fewer hallucinations.
  • Hallucinations and missing elements are the most penalizing observation types in mean opinion scores.When no observation is made, the average score approaches the maximum rating.
  • The CLAP–MOS analysis uses pairwise score differences as the primary agreement signal because the evaluation itself is pairwise.Absolute MOS values are reported for completeness but are not calibrated across annotators.
  • The study computes Spearman correlation between MOS and CLAP score differences, comparing LAION-CLAP with Ours(2).Caption pairs with zero MOS difference are excluded, and Ours(2) enables an in-domain comparison.

C. Zero-shot Audio classification

Zero-shot classification results vary by benchmark, but full SONICCAPS caption diversity delivers the strongest reported performance and improves transfer on the disjoint FoleyBench benchmark.

  • LAION-CLAP outperforms single-caption models on ESC-50, whereas all evaluated models outperform it on FoleyBench by at least 10% in R@5.The contrasting results indicate that benchmark overlap and architectural choices may influence the comparison.
  • Ours(1) reaches 68.9% R@1 on ESC-50, while Ours(2) reaches 9.66% versus 8.2% for Ours(1) on FoleyBench.The authors relate the ESC-50 result to overlapping FreeSound data and the FoleyBench result to cross-domain transfer.
  • Ours(8) and Ours(9), trained on the full SONICCAPS collection, obtain the best results on both benchmarks.Ours(9) reaches 25.6% R@5 on FoleyBench, compared with 18.4% for OursAC+WC and 9.1% for LAION-CLAP.
  • FoleyBench gains support stronger semantic representations and out-of-domain generalization from richer linguistic supervision.FoleyBench audio is entirely disjoint from the training data.

VI. CONCLUSION

SonicCaps is designed to improve audio–text learning by increasing caption diversity and fidelity. The results indicate that diverse textual supervision improves retrieval and zero-shot performance, while higher-quality captions improve alignment with human perceptual judgments.

  • Increasing caption diversity consistently improves cross-modal retrieval and zero-shot performance without scaling audio data.The gains outperform single-source and large-scale baselines.
  • Quality improvements alone yield only marginal retrieval gains, identifying diversity as the key performance driver.
  • Higher-quality captions enable CLAP models to align more closely with human perceptual judgments and subjective caption-quality evaluations.
Loading 2609.02343v1…