Source-linked AI summary
Do Audio-Language Models Understand Linguistic Variations?
Ramaneswaran Selvakumar, Sonal Kumar, Hemant Kumar Giri, Nishit Anand, Ashish Seth, Sreyan Ghosh, Dinesh Manocha
TL;DR
Audio-language models can struggle when natural-language audio queries vary in wording, leaving their robustness to linguistic variation an open question. The paper trains RobustCLAP with paraphrased caption views and reports improved retrieval across benchmarks, including gains of 0.8%–13%.
Problem
The paper investigates whether audio-language models generalize to linguistic variations in text queries, reporting retrieval-recall drops of 0.1%–16% across benchmarks.
Source: §1 · pp. 1–2Method
RobustCLAP fine-tunes pretrained CLAP with a multi-view contrastive objective that uses paraphrases as views of the same audio scene.
Source: §1 · p. 2, §3.2 · p. 3Results
Takeaways & Limitations
RobustCLAP improves retrieval on original and paraphrased benchmarks while retaining prior pretrained knowledge.
Source: §6 · p. 5Some LLM-generated paraphrases may be inaccurate despite diverse human-written examples and a correction mechanism.
Source: §8 · p. 5
Abstract
from arXiv · showhide
Open-vocabulary audio language models (ALMs), like Contrastive Language Audio Pretraining (CLAP), represent a promising new paradigm for audio-text retrieval using natural language queries. In this paper, for the first time, we perform controlled experiments on various benchmarks to show that existing ALMs struggle to generalize to linguistic variations in textual queries. To address this issue, we propose RobustCLAP, a novel and compute-efficient technique to learn audio-language representations agnostic to linguistic variations. Specifically, we reformulate the contrastive loss used in CLAP architectures by introducing a multi-view contrastive learning objective, where paraphrases are treated as different views of the same audio scene and use this for training. Our proposed approach improves the text-to-audio retrieval performance of CLAP by 0.8%-13% across benchmarks and enhances robustness to linguistic variation.
1 Introduction
Natural-language queries make audio retrieval intuitive, but it remained unclear whether audio-language models generalize across different wording. The paper evaluates this vulnerability and introduces RobustCLAP to improve retrieval under linguistic variation.
1 Introduction
- Existing ALMs show retrieval-recall drops of 0.1%–16% across benchmarks when captions are paraphrased.The introduction also reports drops of up to 16% from slight wording changes.Source: §1 · pp. 1–2
- The paper presents the first study evaluating ALM robustness for text-to-audio retrieval across five newly constructed benchmarks.The benchmarks use linguistically varied queries with similar intent.Source: §1 · p. 1
- RobustCLAP fine-tunes pretrained CLAP with a multi-view contrastive objective that aligns paraphrased captions with original captions and audio.Using only a fraction of the original pretraining data, it improves text-to-audio retrieval by 0.8%–13% on original and paraphrased benchmarks.Source: §1 · p. 2
2 Related Work
Prior work explored query variation in text and image retrieval, while audio retrieval received few such efforts. Synthetic data was also less explored for audio-language models, motivating a method that adapts existing CLAP models.
2.1 Retrieval With Query Variations
- Few studies addressed query variation in audio retrieval, despite its relevance to audio captioning and question answering.Prior work explored query variation in text and image retrieval, while ALM improvement efforts often focused on scaling or reasoning.Source: §2.1 · p. 2
2.2 Synthetic Data For Retrieval
- Unlike CompA, which trains ALMs from scratch using synthetic data, this approach adapts off-the-shelf CLAP with minimal additional training while preserving pretrained knowledge and capabilities.The authors describe synthetic data for improving ALMs as under-explored.Source: §2.2 · p. 2
3 Methodology
The methodology builds paraphrased retrieval benchmarks and trains CLAP with paraphrase views. Its benchmark construction uses LLM generation and correction, while its training objective progressively aligns varied captions with their audio and original text.
3.1 Paraphrased Audio Text Retrieval Benchmark
- Five retrieval benchmarks are extended with paraphrased captions: AudioCaps, Clotho, DCASE, Audioset Strong Labels, and SoundDesc.The original and paraphrased test sets are compared using Recall@10.Source: §3.1 · p. 2, Table 1 · p. 3
- LLMs generate paraphrases using human-written in-context examples, then assess accuracy and specificity and correct captions when needed.Human evaluators gave 100 sampled paraphrases an average quality score of 4.89/5 and preferred corrected captions 98% of the time in a 50-caption evaluation.Source: §3.1 · pp. 2–3
3.2 Improving CLAP With Paraphrases
- RobustCLAP uses two paraphrase views, first changing linguistic structure while retaining vocabulary, then changing both vocabulary and structure, to align text representations progressively.The method combines these paraphrase losses with the original CLAP loss to help prevent forgetting pretrained knowledge.Source: §3.2 · pp. 3–4
4 Experimental Setup
The experiments train RobustCLAP on augmented AudioCaps and Clotho data, then evaluate text-to-audio retrieval across five datasets using Recall@10 and established ALM baselines.
4 Experimental Setup
- Training combines AudioCaps and Clotho with two levels of paraphrased captions.The paper uses LAION-CLAP as the base model for RobustCLAP.Source: §4 · p. 4
- Text-to-audio retrieval is evaluated with Recall@10 on AudioCaps, Clotho, Audioset SL, SoundDesc, and DCASE.Source: §4 · p. 4
- The baselines are ML-ACT, MSCLAP-22, MSCLAP-23, CompA, and LAION-CLAP.Source: §4 · p. 4
5 Results And Analysis
RobustCLAP improves text-to-audio retrieval on original and paraphrased benchmarks, while analysis identifies where linguistic changes most affect retrieval and how CLAP errors arise. Zero-shot classification performance remains close to CLAP’s.
Quantitative Analysis
- RobustCLAP improves recall on original benchmarks by 0.8%–13% over its base model and paraphrased scores by 2%–12% over respective second-best models.Current ALMs show a 0.1%–16% recall drop on paraphrased captions versus original captions.Source: §5 · p. 4
- On ESC-50 and FSD50K, CLAP scores mAP@10 of 94.25 and 52.20, while RobustCLAP scores 94.07 and 52.81, respectively.The paper describes this as a negligible drop overall, indicating retained prior knowledge.Source: §5 · p. 4
- RobustCLAP also outperforms other audio-language models on paraphrased audio-to-text retrieval.The paper indicates these results in Table 7.Source: §5 · p. 4
Qualitative Analysis
- In 97% of sampled cases where CLAP failed and RobustCLAP succeeded, evaluators judged CLAP’s retrieved audio incorrect; 3% were judged correct.A small candidate set can contain multiple files with the same acoustic events, especially when only one or two events are present.Source: §5 · p. 4
- CLAP often favors explicitly mentioned events, misses some scene events, and fails to recognize event attributes or modifiers.These were the three common mistake patterns observed in the qualitative analysis.Source: §5 · p. 4
Impact Of Sound Event And Attributes
- Paraphrasing sound attributes drops Recall@1 by 3.8% (RobustCLAP reduces the decline to 0.4%); paraphrasing sound sources drops recall by up to 15% (reduced to 3%).Sound events have greater impact than attributes when linguistic structure is held fixed.Source: §5 · p. 4, Table 2 · p. 5
6 Conclusion
The paper finds that current audio-language models lack robustness to linguistic variation in natural-language inputs. It extends audio-text retrieval benchmarks with paraphrases and trains CLAP using a multi-view contrastive loss; RobustCLAP improves retrieval on original and paraphrased benchmarks while retaining prior knowledge.
6 Conclusion
- Current audio-language models lack robustness to linguistic variation in natural-language inputs.The paper extends several audio-text retrieval benchmarks with paraphrased captions generated through a two-step LLM-based process.Source: §6 · p. 5
- RobustCLAP fine-tunes CLAP models with a multi-view contrastive loss on a small set of paraphrased data.The resulting model improves retrieval on original and paraphrased benchmarks while retaining prior pre-trained knowledge.Source: §6 · p. 5
- The authors conclude that RobustCLAP improves retrieval recall on original and paraphrased benchmarks while retaining prior pre-trained knowledge.They hope the work will fuel further studies into audio-language-model robustness.Source: §6 · p. 5
8 Limitations and Future Work
The authors identify possible errors in LLM-generated paraphrases and note that future work could extend evaluation to other retrieval domains and longer audio. The appendix includes dataset, model, results, implementation, and prompt details.
8 Limitations and Future Work
- LLM-generated training and testing paraphrases may be inaccurate because of hallucinations, despite diverse human-written examples and a correction mechanism.This is listed as a limitation of the paraphrase-generation process.Source: §8 · p. 5
- The authors propose extending the work to music and speech retrieval and to longer audio segments.The current work primarily experiments with diverse audio benchmarks and relatively short segments.Source: §8 · p. 5
A Appendix
- The appendix lists dataset, model, additional-results, implementation, and prompt details; its dataset section describes datasets and points to sample information and paraphrase-prompt details.The dataset details include where dataset information and paraphrase-generation and correction prompts are provided.Source: A Appendix · p. 6
B.1 Benchmark Datasets
The benchmark datasets cover human-written captions, strong sound-event labels, and descriptions of everyday sounds and acoustic scenes. They vary in clip duration, annotation type, and source.
B.1 Benchmark Datasets
- AudioCaps has 46K ten-second AudioSet clips with human-written descriptions, while Clotho has 15–30-second Freesound clips captioned for environments, music, and activities.Both are described as widely used audio-retrieval benchmarks.Source: §B.1 · p. 7
- AudioSet Strong Labels contains over 2 million ten-second YouTube clips with sound-event labels, which the authors use to generate natural-language captions.Its labels include events such as dog barking, car engines, and crowd cheering.Source: §B.1 · p. 7
- SoundDesc pairs diverse everyday, natural-environment, and specific-event clips with detailed descriptions of sound, source, and context.Source: §B.1 · p. 7
- DCASE recordings cover environments such as streets, parks, and indoor settings, with sound-event or acoustic-scene labels.The dataset supports challenges in detecting and classifying acoustic scenes and events.Source: §B.1 · p. 7
- Table 3 overviews dataset audio-sample and caption counts for training and testing.Source: Table 3 · p. 7
B.2 Benchmark Paraphrase Evaluation
Human evaluation found the benchmark paraphrases high-quality and usually preferred corrected versions; RobustCLAP retained near-CLAP zero-shot classification scores.
B.2 Benchmark Paraphrase Evaluation
- Human evaluators rated generated paraphrases 4.89/5 on average and preferred corrected versions 98% of the time.Source: §B.2 · p. 7
C.1 Baseline Details
- The comparison includes ML-ACT, LAION-CLAP, LAION-CLAP Music, MS-CLAP 22/23, and CompA, whose training data and model designs differ.The baseline descriptions include differences in training datasets, encoders, and contrastive-training approaches.Source: §C.1 · pp. 7–8
D.1 Performance On Zero-Shot Audio Classification
- 94.25 (CLAP) vs 94.07 (RobustCLAP) mAP@10 on ESC-50; 52.20 vs 52.81 on FSD-50K, indicating negligible classification changes.The table reports zero-shot audio classification results in terms of mAP@10.Source: §D.1 · p. 8, Table 5 · p. 8
D.2 Error Analysis
Manual review found most CLAP retrievals in sampled failure cases were wrong, with recurring errors involving direct event matches, missed details, and sound attributes; bootstrap testing found RobustCLAP's improvement statistically significant.
D.2 Error Analysis
- 97% of the sampled CLAP retrievals that RobustCLAP corrected were judged incorrect by human evaluators.The remaining 3% were judged correct, which the authors relate to benchmark ambiguity when several clips contain the same sparse events.Source: §D.2 · p. 8
- CLAP can prioritize explicitly mentioned sound events even when the retrieved audio has a different overall meaning from the query.Examples include matching background music or gurgling and bubbling while other scene content differs.Source: §D.2 · p. 9
- CLAP may capture a scene's dominant context but miss queried events; in one urban example it missed vehicle hum and included birds and wind.Source: §D.2 · p. 9
- CLAP may fail to capture sound attributes used as modifiers; a retrieved clip had constant vehicle noise rather than the query's faint vehicle hums.Source: §D.2 · p. 9
D.3 Statistical Significance Test
- A bootstrap-based t-test found RobustCLAP's improvement over CLAP statistically significant.The method repeatedly samples test-set items with replacement, computes recall for each resample, and tests those recall values.Source: §D.3 · p. 9
E Additional Implementation Details
The appendix documents the model scale and infrastructure used for RobustCLAP, along with its software setup, paraphrase prompts, and qualitative risk checks.
E.1 Model Parameters
- Implemented ALMs use audio encoders with BERT-like text encoders and are typically under 300M parameters; LLaMA-3-70B generates paraphrases.Table 6 lists the ALMs and their sizes.Source: §E.1 · p. 9, Table 6 · p. 9
E.2 Compute Infrastructure
- RobustCLAP training takes around 2 hours on four NVIDIA A100 GPUs; inference requires one A100, while LLaMA-3-70B inference uses four.Source: §E.2 · pp. 9–10
E.3 Implementation Software And Packages:
- The implementation uses official model repositories and builds RobustCLAP on the LAION-CLAP repository and base models.The appendix also reports using vLLM for accelerated LLaMA-3-70B inference.Source: §E.3 · p. 10
E.4 Potential Risks:
- Although LLM paraphrases can be incorrect or toxic, qualitative analysis found no toxic outputs and consistently high-quality paraphrases.Source: §E.4 · p. 10
Paraphrase Generation Prompt
- Generation prompts request varied vocabulary and sentence structure while preserving caption meaning, including nuances and technical terms.One example paraphrases gunfire as shots while retaining the subsequent click and breaking glass.Source: Paraphrase Generation Prompt · p. 11
Prompt Paraphrase Correction
- Correction prompts ask the LLM to check whether paraphrases retain sound-event nuance and to revise wording when specificity is lost.For example, the prompt distinguishes a metallic clank from a generic collision.Source: Prompt Paraphrase Correction · p. 11