Source-linked AI summary
Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo, Yiwen Shao, Hao Zhang, Dong Yu
TL;DR
Audio-language pretraining lacks systematic evidence on whether it can produce transferable representations and how objectives behave across tasks and scales. The paper builds CaptionStew and evaluates contrastive and captioning approaches, finding competitive representations and distinct efficiency, scalability, and initialization patterns.
Problem
Audio-language models have limited evidence of general-purpose transfer, and objective behavior across heterogeneous tasks and scales remains poorly understood.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The empirical comparisons establish ALP as a practical, competitive approach for general-purpose audio representations and identify factors relevant to future progress.
Source: §1 · p. 2, §7 · p. 9CaptionStew combines sources with varied generation methods, including LLM-synthesized captions, without extensive quality control or human verification, which could affect model training.
Source: Limitation · p. 9
Abstract
from arXiv · showhide
Audio-language pretraining (ALP) holds promise for learning general-purpose audio representation, yet remains underexplored. Crucially, there is no consensus on whether audio-language models can build effective general-purpose audio encoders, nor a systematic understanding of how pretraining objectives behave across diverse tasks and scales. We identify three key barriers: limited scale of audio-text corpora, limited coverage of audio attributes in existing caption corpora, and lack of systematic exploration and evaluation. To fill this gap, we present the first principled empirical study of ALP. We first introduce CaptionStew, a 10.7M caption dataset aggregating open-source audio-text corpora across multiple domains and captioning focuses. We then conduct the first comprehensive evaluation comparing contrastive and captioning objectives for learning audio representation across speech, music, and environmental sound tasks. Our results not only demonstrate that ALP yields competitive, transferable representations, but reveal critical trade-offs: contrastive learning offers superior data efficiency, while captioning exhibits better scalability. Furthermore, we find that the benefits of supervised initialization often diminish at larger scales, challenging common practices. By grounding these claims in empirical evidence, we establish a viable pathway toward general-purpose audio representation learning, guiding future research.
1 Introduction
Audio representations remain specialized, leaving transfer across diverse tasks unresolved; audio-language pretraining offers a possible unified route, but its objectives and scaling behavior lack systematic evaluation. This study builds a larger, more diverse testbed and compares objectives and training factors, finding complementary trade-offs.
1 Introduction
- Audio methods often target narrow tasks, and systems strong at environmental sound classification may miss speaker identity or paralinguistic attributes, and vice versa.Robust transfer across diverse audio-processing tasks remains unresolved.Source: §1 · p. 1
- ALP uses natural-language captions as semantic supervision, but evidence that it supports general-purpose audio representations remains limited and key objective and transfer questions are unanswered.Existing audio-language models are largely evaluated on retrieval tasks.Source: §1 · pp. 1–2
- Prior progress is constrained by limited audio-text corpus scale, uneven coverage of audio attributes, and scarce comparisons of objectives across broad audio-understanding tasks.Captions often focus on sound events rather than speaker traits, musical properties, or acoustic environments.Source: §1 · p. 2
- The study aggregates diverse open-source captions into CaptionStew and comprehensively evaluates ALP across downstream tasks and protocols.The evaluation spans speech, music, and environmental audio domains.Source: §1 · p. 2
- Contrastive learning is more data-efficient, while captioning scales better; supervised-initialization gains diminish at larger scales and beyond audio event classification.The authors also identify limited lexical diversity as a possible constraint on performance scaling for some attributes.Source: §1 · p. 2
- The authors release data, training and evaluation code, and pretrained models to support further research.Source: §1 · p. 2
2 Related Work
Prior audio representation work includes supervised and self-supervised learning, while audio-language pretraining has mostly centered on contrastive objectives. Evaluation has also expanded toward complex audio understanding, and this work evaluates ALP across varied task types.
Audio Representation Learning
- Audio representation research includes supervised methods for specific target tasks and self-supervised approaches spanning speech, audio, and music.Source: §2 · p. 2
Audio–Language Pretraining
- Most prior ALP work focuses on contrastive objectives, alongside newer combinations of objectives and datasets ranging from human annotation to LLM augmentation.Source: §2 · p. 2
Universal Audio Understanding
- Audio evaluation has moved from task-specific benchmarks toward complex language-query systems; this work evaluates ALP on discriminative tasks, alignment, and open-form question answering.Source: §2 · p. 3
3 Audio-language Pretraining
ALP aligns audio representations with captions using either a contrastive or generative objective. The paper compares these paradigms, with contrastive learning matching paired embeddings and captioning training a decoder to generate descriptions.
3 Audio-language Pretraining
- ALP uses audio and text components to connect audio signals with captions, treating contrastive and captioning objectives as complementary discriminative and generative formulations.Source: §3 · p. 3
3.1 Contrastive Objective
- The contrastive objective applies symmetric InfoNCE to paired audio-text embeddings, bringing matched samples together and separating mismatched pairs.The loss encourages semantic organization in which similar content is grouped in embedding space.Source: §3.1 · p. 3
3.2 Captioning Objective
- Captioning generates textual descriptions from audio, with cross-attention providing frame-level supervision and token prediction supporting fine-grained attributes and relations.Source: §3.2 · p. 4
- Mixed autoregressive and parallel token prediction outperformed purely autoregressive decoding in a pilot transfer ablation and was used throughout the paper.Parallel prediction removes prior autoregressive context so token predictions depend on audio features.Source: §3.2 · p. 4
4 CaptionStew Dataset
CaptionStew addresses limited audio-caption scale and semantic coverage by aggregating open-source datasets with diverse domains, captioning methods, and descriptive styles. It provides a larger, broader testbed for studying ALP across objectives, tasks, and scales.
4 CaptionStew Dataset
- Single-pipeline caption datasets can exhibit stylistic bias, limited lexical diversity, and narrow coverage of audio characteristics.Source: §4 · p. 4
- CaptionStew aggregates open-source datasets covering sound events, expressive speech, and music, with captions from varied creation methods and descriptive granularities.Source: §4 · p. 4
- CaptionStew pairs audio samples with 10.7 million captions across 37,290 hours spanning speech, music, and environmental domains, substantially increasing scale and semantic coverage versus prior public audio caption resources.It provides a useful and reproducible testbed for studying audio–language pretraining across objectives, tasks, and training scales.Source: §4 · p. 5
5 Experimental Setup
The experiments pretrain audio encoders on CaptionStew from scratch or pretrained checkpoints, then assess frozen representations across discriminative, audio-language, and open-form question-answering tasks. Comparisons include a supervised audio-event classifier and domain-specialized self-supervised models.
5.1 Implementation Details
- Models use a Zipformer-M audio encoder and are trained either from scratch or initialized from pretrained checkpoints.Contrastive training uses a BERT-base text encoder, while captioning uses a BART-base text decoder.Source: §5.1 · p. 5
5.2 Evaluation Protocols and Datasets
- Frozen encoder representations are assessed through linear probing, audio-language alignment tasks, and open-form question answering across sound, speech, and music domains.The QA setup connects frozen encoders to an LLM through lightweight adaptors, training only the adaptor.Source: §5.2 · pp. 5–6
5.3 Baseline Methods
- The evaluation compares ALP encoders with Zipformer-AEC and domain-specialized SSL baselines BEATs, Wav2vec 2.0, and MERT.Zipformer-AEC is the primary baseline; the SSL models represent environmental/general audio, speech, and music, respectively.Source: §5.3 · p. 6
6 Experiment Results
Across evaluation tasks, contrastive and captioning objectives show complementary strengths, and ALP representations transfer across audio domains. Scaling and dataset analyses find that captioning scales better, while aggregation broadens semantic coverage more clearly than lexical diversity.
6 Experiment Results
- Contrastive learning outperforms captioning on linear probing, while captioning has slight advantages in open-form question answering; the best contrastive-init model transfers strongly across speaker, music, and retrieval tasks.The table reports evaluation results across tasks and protocols; the prose also notes competitiveness on audio-event classification and against domain-specialized SSL methods.Source: §6 · p. 6, Table 3 · p. 7
6.1 Data-Scaling Experiments
- Across nested CaptionStew scales, contrastive learning is more data-efficient, while captioning scales better and matches or surpasses it on language-involved tasks at 10M pairs.Sound event detection can degrade with more caption data, especially with AudioSet initialization; emotion recognition and instrument classification show weaker scaling gains.Source: §6.1 · pp. 6–8, Figure 2 · p. 8
6.2 Dataset Analysis
- CaptionStew combines sources with distinct descriptive perspectives and expands vocabulary, but aggregation improves semantic coverage more clearly than lexical diversity.The analysis reports 56,586 unique words for CaptionStew versus 4,060–27,906 for individual datasets; limited diversity may help explain weaker scaling for some tasks.Source: §6.2 · p. 8
7 Conclusions
The paper establishes a rigorous empirical baseline for general-purpose audio representation learning with audio–language pretraining. Its evaluation finds competitive representations across audio domains, complementary objective trade-offs, and diminishing value from supervised initialization at scale.
7 Conclusions
- Audio–language pretraining produces competitive representations across speech, music, and environmental sounds.The study compares contrastive and captioning objectives and finds complementary strengths in efficiency and scalability; standard supervised initialization may be unnecessary or detrimental at scale.Source: §7 · p. 9
- CaptionStew aggregates and harmonizes diverse datasets to address data scarcity and enable comparisons across objectives and data scales.The paper identifies limited lexical diversity in current caption datasets as a frontier for future improvement.Source: §7 · p. 9
Limitation
The study is primarily empirical and faces limitations in dataset quality, linguistic diversity, and experimental scale. Its results may not reflect performance at larger scales or with more sophisticated models and data curation.
Dataset Construction and Quality
- CaptionStew combines sources with varied caption-generation methods, including LLM-synthesized captions, without extensive quality control or human verification.The authors note that these choices may introduce systematic biases or artifacts; lexical diversity remains below that of mature image-text corpora despite prioritizing semantic diversity.Source: Limitation · p. 9
Limited Technical Novelty; Limited Model and Data Scalability
- The study combines existing techniques and makes empirical rather than fundamentally methodological contributions.Its experiments use 10M audio-text pairs and relatively modest models; the authors say larger-scale training may reveal different scaling dynamics, particularly for captioning.Source: Limitation · p. 9
Ethical Considerations
CaptionStew aggregates existing open-source datasets, relying on their providers’ compliance with consent, anonymization, and licensing requirements. The authors flag possible bias amplification and misuse risks, and do not claim suitability for high-stakes or safety-critical use without safeguards and validation.
Ethical Considerations
- The authors rely on source dataset providers’ compliance with consent, anonymization, and licensing requirements rather than collecting new data.They warn that aggregation may amplify source biases or artifacts, and that audio representations could facilitate surveillance, profiling, or misuse of speaker-related attributes.Source: Ethical Considerations · pp. 9–10
- Released models are intended for research, not high-stakes or safety-critical scenarios without safeguards and validation.The authors identify accessibility tools, audio search, and audio understanding as possible beneficial applications, alongside potential misuse.Source: Ethical Considerations · p. 10
A.1 Full Implementation details
The appendix details the training setup, Zipformer encoder, CaptionStew construction, evaluation protocols, and additional results. It also describes the use of language models during drafting and reports lexical-statistics and caption-source visualizations.
A.1 Full Implementation details
- Models use Zipformer-M audio encoders, with BERT-base text encoders for contrastive pretraining and BART-base decoders for captioning.Training is from scratch or from pretrained checkpoints; the audio encoder initialization uses a Zipformer audio event classifier trained on AudioSet.Source: §A.1 · p. 15
A.2 Zipformer Model
- Zipformer uses six Transformer stages at progressively decreasing and then increasing frame rates, with residual and upsampling connections; fused outputs yield 768-dimensional embeddings.The embeddings are used across downstream evaluation tasks, and preliminary experiments report competitive performance on environmental sound, music, and speaker-related tasks.Source: §A.2 · p. 16
A.3 Sourced Datasets for CaptionStew
- CaptionStew aggregates eight open-source caption datasets covering environmental sounds, music, and expressive speech, with varied captioning methods and descriptive styles.The authors filter clips longer than one minute and remove benchmark-overlapping samples; duplicate audio is consolidated while associated captions are merged.Source: §A.3 · p. 16
A.4 Evaluation Datasets
- The evaluation covers linear probing, audio-language alignment, and open-form question answering across the listed datasets and metrics.The appendix points to Table 7 for evaluation dataset and metric details.Source: §A.4 · p. 17, Table 7 · p. 18
A.5 Main Results (cont.)
- With attention pooling, captioning-scratch reaches 72.86% on speaker identification, compared with 46.67% with mean pooling.The performance gap between contrastive and captioning narrows with learned attention pooling, and the authors note that pooling choice can affect conclusions about objective effectiveness.Source: §A.5 · p. 18
A.6 Additional Results
- Contrastive-init achieves comparable or superior audio-text retrieval results to a state-of-the-art model, with particularly strong performance on speech and music retrieval.The comparison is against the retrieval model of Bai et al. (2025).Source: §A.6 · p. 18
A.7 The Use of Large Language Model
- Table 10 reports vocabulary size, average sentence length, and Distinct-n; Figure 4 visualizes caption sentence embeddings grouped by source.The authors report that large language models assisted with writing refinement and grammatical corrections, while technical content and conclusions remained their own.Source: §A.7 · p. 18, Table 10 · p. 19, Figure 4 · p. 20