Source-linked AI summary
Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment
Mustafa Talha İlerisoy, Hung Manh Pham, Mathias Funk, Mykola Pechenizkiy, Aaqib Saeed
TL;DR
Respiratory audio encoders capture diagnostic acoustic patterns but lack clinical semantic grounding for zero-shot inference, while paired audio-report data are scarce. REACH aligns pretrained audio and medical text encoders using LLM-synthesized reports, contrastive learning, and reconstruction regularization. Across 9 tasks on 6 datasets, it achieves 61.3% mean zero-shot AUC and 71.6% mean linear probing AUC using 43% of the full-scale baseline’s data.
Problem
Respiratory encoders lack clinical semantic grounding for zero-shot inference, and respiratory datasets do not provide paired audio-report data at scale.
Method
REACH projects pretrained audio and text encoders into a shared space, using LLM-synthesized reports as semantic anchors and combining sigmoid contrastive alignment with reconstruction regularization.
Results
61.3% mean zero-shot AUC surpasses CLAP (51.4%) and Qwen2 Audio 7B (54.9%), while mean linear probing AUC reaches 71.6% using 43% of the full-scale baseline’s training data.
Takeaways & Limitations
Targeted clinical semantic alignment can repurpose pretrained respiratory encoders for zero-shot classification without paired audio-report data.
Takeaways & Limitations
The primary data-matched comparison uses only the openly accessible subset, approximately 43% of the original OPERA training volume, rather than the full corpus.
Abstract
from arXiv · showhide
Self-supervised respiratory encoders lack semantic grounding in clinical domain needed for zero-shot inference, limiting their utility without task-specific labeled data. We propose a framework that aligns these encoders with medical terminology in a shared latent space turning them into a zero-shot-capable foundation model. To address paired data scarcity, we use a medical LLM to synthesize structured reports from metadata, creating dense semantic anchors for contrastive learning. Our training combines a sigmoid-based contrastive loss with encoder's native SSL objective and similarity-aware negative sampling to sharpen pathological boundaries. Across 9 tasks on 6 datasets, our method achieves a 61.3% mean zero-shot AUC, surpassing CLAP (51.4%) and Qwen2-Audio (54.9%) while reaching the highest linear probing AUC (71.6%) with only 43% of data used by full-scale baselines, showing that structured semantic alignment outperforms large-scale, general-purpose models in clinical diagnostics.
1. Introduction
REACH aligns pretrained respiratory audio encoders with clinical language to enable zero-shot classification without paired audio-report data. It combines LLM-generated reports, similarity-aware negatives, and structure-preserving contrastive training, outperforming larger or more data-intensive baselines across respiratory tasks.
- Motivation: Respiratory auscultation is widespread but subjective, and zero-shot generalization to novel pathologies remains largely unattained in respiratory audio modeling.
- Motivation: Self-supervised respiratory encoders capture fine-grained acoustic patterns, but their embedding spaces remain semantically opaque for clinical interpretation.
- Method: REACH uses a medical LLM to convert discrete metadata into structured reports, creating semantic anchors without paired audio-report datasets.
- Results: 61.3% mean zero-shot AUC surpasses CLAP (51.4%) and Qwen2-Audio (54.9%) across 9 tasks on 6 datasets.
- Results: 71.6% mean linear probing AUC is the highest reported, despite using only 43% of the full-scale baseline’s training data.
- Method: The framework jointly optimizes sigmoid contrastive alignment with the encoder’s native reconstruction loss and uses similarity-aware negative sampling to preserve acoustic structure.
2. Methodology
The methodology aligns pretrained respiratory audio and medical text encoders in a shared space, using synthetic clinical reports to overcome paired-data scarcity. It combines contrastive alignment, reconstruction regularization, and similarity-aware negatives for zero-shot classification.
- 2.1. Problem Formulation: Pretrained audio and text encoders are projected into a shared space where semantically corresponding respiratory recordings and clinical reports align.The audio and text spaces are independently learned, so projection heads establish their semantic correspondence.
- 2.1. Problem Formulation: Zero-shot classification assigns each recording to the clinical text anchor with the highest cosine similarity.The framework compares projected audio embeddings with class-specific prompts encoded in the shared space.
- 2.2. Report Synthesis: Discrete respiratory metadata is converted into standardized clinical reports by a medical-grade LLM, providing semantic anchors without paired audio-report datasets.The prompt restricts reports to supplied metadata and excludes follow-up recommendations; evaluation metadata is partitioned to prevent leakage.
- 2.3. Model Architecture: The modular framework keeps the medical text encoder frozen while optimizing the respiratory audio encoder and projection heads.This asymmetry moves the acoustic manifold toward a stable clinical text space and keeps inference prompts in the training semantic frame.
- 2.4. Alignment Objective: The training objective combines a SigLIP-based sigmoid contrastive loss with masked spectrogram reconstruction to align modalities while preserving acoustic feature geometry.The reconstruction MSE acts as a structural regularizer, while the contrastive term reshapes the space for cross-modal compatibility.
- 2.3. Similarity-Aware Negative Sampling: FAISS-based negative swapping retrieves semantically distant reports to make negatives more contrastive when random negatives are textually similar.The method applies negative swapping to 50% of audio samples during each training step.
3. Experimental Setup
The evaluation benchmarks representation quality, generalization, and cross-modal alignment across six respiratory datasets and nine tasks. It uses linear probing, kNN, and zero-shot AUC protocols, with baselines spanning unimodal, general-purpose audio-text, and audio-language models under data-parity controls.
- 3.1. Evaluation Benchmark: The benchmark covers 9 tasks from 6 publicly available respiratory sound datasets, including in-domain and out-of-domain settings.Five tasks are in-domain and four are out-of-domain; all are binary except T9, a five-class COPD severity task.
- 3.2. Evaluation Protocols: Evaluation uses linear probing, kNN, and zero-shot classification, reporting AUC (%) throughout.Linear probing averages five seeds, kNN evaluates embedding geometry directly, and zero-shot classification compares samples with clinical text prompts by cosine similarity.
- 3.3. Baselines: Baselines include unimodal encoders, CLAP as a general-purpose audio-text model, and Qwen2-Audio 7B and Audio-Flamingo-3 as audio-language decoders.The categories separate models evaluated with different protocols and test whether scale compensates for domain alignment.
- 3.4. Data Parity: The data-parity comparison retrains OPERA variants and the alignment framework on the openly accessible subset, comprising approximately 43% of the original training volume.This design ensures the compared methods use the same publicly accessible training subset.
4. Results
Across benchmark axes, REACH preserves and improves unimodal acoustic quality, reorganizes embeddings into clinical clusters, and enables stronger zero-shot transfer than general-purpose and large-scale baselines.
- Benchmark setup: Table 2 reports benchmark results as AUC percentages, distinguishing open-access-only training from full-corpus results and marking gains over OGT†.The table ranks models using data-matched variants trained on approximately 43% of the original corpus.
- Unimodal capability: 71.6 mean linear probing AUC surpasses OGT† by 6.2 points and the original OGT by 3.9 points.Improvements occur across all tasks, with the largest gains on T5 (+15.7), T7 (+8.5), T4 (+8.1), and T1 (+7.6).
- Latent space structure: 65.2 mean kNN AUC outperforms all baselines, including OGT† at 58.8, indicating improved embedding geometry without learned classification parameters.The strongest gain is on T5, while T8 is the only regression at −1.4, attributed to its 234-sample size.
- Zero-shot transfer: 61.3% mean zero-shot AUC exceeds CLAP at 51.4% and Qwen2 Audio 7B at 54.9%.Zero-shot classification uses cosine similarity to text anchors without labeled data; OPERA cannot operate in this setting because it is unimodal.
- Ablation analysis: Ablations show that components can preserve linear probing while severely degrading zero-shot AUC, demonstrating their importance for cross-modal alignment.Training from scratch yields 55.1 zero-shot AUC with 66.0 linear probing, while random negatives yield 53.5 zero-shot AUC with 70.6 linear probing.
5. Conclusion
REACH aligns pre-trained respiratory audio encoders with clinical text without paired audio-report data. The conclusion identifies semantic bridging as a sample-efficient alternative to scaling pre-training or building multimodal models from scratch.
- Conclusion: REACH aligns pre-trained respiratory audio encoders with clinical text representations, enabling zero-shot classification without paired audio-report data.The framework treats an off-the-shelf LLM as a metadata-to-report translator for contrastive alignment.
- Conclusion: The authors identify disconnection from clinical semantics, rather than acoustic representation quality, as the primary barrier to zero-shot respiratory diagnostics.Existing self-supervised encoders already capture acoustic diagnostic signals.
- Conclusion: REACH surpasses baselines trained on 57% more data and models with orders-of-magnitude more parameters.The conclusion presents targeted semantic bridging as more sample-efficient than scaling pre-training or building multimodal architectures from scratch.
- Conclusion: The proposed alignment principle may extend to clinical domains with capable unimodal encoders and structured metadata.This is stated as a candidate scope for generalization beyond respiratory audio.
6. Generative AI Use Disclosure
The authors state that generative AI was used only for language editing and polishing, while all technical work and conclusions were created by the authors.
- Generative AI Use Disclosure: Generative AI tools were used solely for language editing and polishing to improve manuscript clarity and readability.The disclosure excludes technical content, experimental design, analysis, and conclusions from generative-AI authorship.
- Generative AI Use Disclosure: The authors take full responsibility for the paper’s technical content, experimental design, analysis, and conclusions.