Source-linked AI summary
Why ML-based cough models do not generalize: a systematic cross-dataset evaluation for tuberculosis screening
Wensi Zhang, Tomas Teijeiro, Jérôme Thevenot, David Atienza
TL;DR
Whether cough models learn disease-related acoustics or data-collection artifacts remains unresolved, limiting confidence in their clinical utility. This study evaluates classical ML and DL classifiers across three independent TB-cough datasets and examines acquisition bias, device transfer, and a clinical-variable baseline. Both acoustic pipelines fail to generalize reliably despite moderate internal performance, whereas clinical variables transfer more consistently and device-diverse training improves robustness to unseen hardware.
Problem
It remains unclear whether cough-based TB models capture disease-related acoustics rather than artifacts of data collection, despite their promise for scalable screening.
Method
The study evaluates classical ML and DL cough classifiers across CODA, TBscreen, and Zambia, supplemented by acquisition-bias, device-generalization, and clinical-variable analyses.
Results
Both acoustic pipelines fail to generalize consistently across datasets, with external ROC-AUC frequently below 0.6 despite within-dataset ROC-AUC up to 0.755 ± 0.056.
Takeaways & Limitations
Strong within-dataset performance is insufficient for clinical readiness, while device-diverse training improves transfer to unseen hardware.
Takeaways & Limitations
Datasets differ in labeling criteria, acquisition protocols, devices, inclusion criteria, populations, and geographic origin, so these differences cannot be fully separated from reported generalization effects.
Abstract
from arXiv · showhide
Cough acoustics are promising for non-invasive tuberculosis (TB) screening, yet whether machine learning (ML) models capture disease-related acoustics or artifacts of data collection remains unresolved. We evaluated the cross-dataset generalizability of classical ML and deep learning (DL) cough-based TB classifiers across three independent datasets. Despite moderate within-dataset performance (ROC-AUC up to $0.755 \pm 0.056$), both pipelines fail to generalize, with external performance frequently below 0.6, indicating a possible limitation of the data. We further observed audio representations are organized by recording device and dataset rather than TB status, predicted TB probability tracks country-level prevalence in CODA, and device mismatch degrades transfer while device-diverse training improves it. Additionally, a clinical-variable baseline generalizes more consistently (ROC-AUC $0.655 - 0.711$), indicating acquisition-specific variability is a stronger driver of poor generalizability than population shift. High within-dataset performance is not enough. External validation is essential before cough-based TB models are clinically ready.
Introduction
Cough-based TB models show moderate internal performance but fail to generalize reliably across datasets. The analyses indicate that acquisition factors such as dataset, location, and recording device shape representations and predictions more strongly than TB status, while device-diverse training can improve transfer.
- Motivation: Public cough datasets contain unreliable labels, variable recording environments, heterogeneous devices, and uncontrolled comorbidities that complicate robust validation.These factors can differ across crowd-sourced or loosely controlled collections.
- Motivation: TB remains a major global health burden, motivating scalable cough-based screening tools for low-resource settings.Confirmatory diagnosis often requires sputum assays and imaging that are difficult to deploy consistently in high-burden settings.
- Study design: The study systematically evaluates classical ML and DL cough classifiers across CODA, TBscreen, and Zambia, adding acquisition-bias, device-generalization, and clinical-variable analyses.Models are trained on one dataset and evaluated on held-out datasets, with model selection performed within the source dataset.
- Implications: Strong within-dataset performance should not be interpreted as evidence that cough-based TB models are clinically ready.The study’s findings support external validation before clinical deployment.
- Cross-dataset generalization: 0.755 ± 0.056 was the strongest reported within-dataset ROC-AUC, yet external performance frequently fell below 0.6 for both acoustic pipelines.The classical pipeline also showed moderate internal ROC-AUC values, including 0.700 ± 0.053 on Zambia and 0.711 ± 0.099 on TBscreen passive cough.
- Acquisition-related bias: Acoustic representations clustered more strongly by dataset, recording device, and location than by TB status.The same pattern appeared in both CLAP embeddings and handcrafted time-frequency features, providing a mechanism for transfer failure when acquisition factors correlate with labels.
- Acquisition-related bias: In CODA, predicted TB probabilities tracked country-level prevalence, while device mismatch reduced transfer and device-diverse training improved performance on unseen hardware.The prevalence association was reduced but not eliminated under the conservative setting; in Zambia, training on three phones raised held-out-recorder ROC-AUC from 0.675, 0.655, or 0.725 individually to 0.732 together.
Discussion
Across independent datasets, cough models often fail to transfer because acquisition-related variation and shortcuts dominate disease-related acoustics. Clinical variables generalize more consistently, while external validation and better-balanced acquisition designs are needed before clinical deployment.
- Cross-dataset performance frequently falls below 0.6 despite moderate internal ROC-AUC, indicating data acquisition—not modeling approach—drives much of the domain shift.Internal performance reached 0.755 for DL and 0.711 for classical ML, but external performance often dropped below 0.6.
- Recording device, site, protocol, and population differences can overwhelm TB-related acoustic variation, producing source-valid shortcuts that fail after domain shifts.When acquisition factors correlate with TB status, models can use them directly as predictive shortcuts.
- Country-level prevalence tracking in CODA provides concrete evidence that predicted TB probabilities can reflect acquisition-linked population structure rather than disease acoustics.Balancing labels together with acquisition-defined subgroups attenuated this association.
- Device mismatch degrades transfer, whereas training across multiple devices improves unseen-device performance only when device diversity is introduced within a balanced study design.Confounding device type with location, prevalence, or recruitment can create additional shortcuts.
- Algorithmic domain-generalization methods gave no consistent external improvement because confounded domains and weak TB acoustic signals limit their ability to separate stable disease structure from acquisition variation.Enforcing domain invariance may discard useful information without improving the clinically relevant boundary.
- A simple clinical-variable model generalizes more consistently than acoustic models, suggesting audio-specific acquisition variability contributes more to poor transfer than population differences alone.Clinical variables may provide a stable baseline risk estimate while cough acoustics contribute complementary information.
Methods
The study evaluated classical ML and DL cough-based TB classifiers across three datasets, with additional analyses of clinical variables, acquisition bias, and device effects. Dataset subsets, cough types, preprocessing, feature representations, classifiers, and validation procedures were selected to support cross-dataset comparison.
- Study design: Three independently collected TB-cough datasets were analyzed using classical ML and DL pipelines, with a clinical-variable model as a performance reference.The datasets were CODA, TBscreen, and Zambia CIDRZ.
- Validation: Subject-level nested cross-validation used 5 outer folds, 4 inner folds, and two random repetitions to estimate within-dataset ROC-AUC.Subject-level splitting prevented recordings from the same participant entering both training and validation folds.
- Dataset preparation: The analysis used subgroup-balanced resampling and separated CODA subsets, TBscreen cough types, and study-specific accessible data before evaluation.TBscreen passive cough recordings were used for training, while forced cough recordings were reserved for validation.
- Deep learning pipeline: The DL pipeline converted cough recordings into spectrogram-based images and explored waveform and spectrogram augmentation alongside preprocessing choices.SpecAugment randomly masks time regions and frequency bands in spectrogram representations.
- Classical ML pipeline: Classical ML compared handcrafted acoustic descriptors and 1024-dimensional pretrained CLAP acoustic embeddings with logistic regression, random forests, and gradient boosting.Handcrafted features summarized frame-level time- and frequency-domain descriptors using means and standard deviations.
- Bias analysis: The CODA bias analysis could not isolate recording-device effects because country, protocol, clinical setting, prevalence, and device were entangled.The study therefore treated acquisition-related bias as a broader source of heterogeneity rather than attributing it solely to hardware.
Extended Data Figure 1 Classical ML performance across configurations
Extended Data Figure 1 compares classical ML ROC-AUC across source datasets, external testing datasets, classifiers, feature types, and RFE settings. Dataset pairings produced larger performance differences than modeling configuration choices, while within-dataset results varied across devices and locations in the accompanying analyses.
- Extended Data Figure 1: Training and testing dataset differences affected classical ML ROC-AUC more strongly than classifier, feature representation, or RFE choice.The heatmaps report subject-level ROC-AUC across all configuration combinations.
- Zambia: Performance was similar across Zambia recording devices and locations, indicating that its within-dataset result was not driven by one device or site.The accompanying analysis reports even performance across both dimensions.
- TBscreen: TBscreen passive-cough performance differed moderately across recording devices, indicating some within-dataset device sensitivity.The comparison concerns subject-level ROC-AUC for the DL pipeline.
- CODA: CODA performance variability was too large to support a meaningful conclusion from the device-stratified analysis.The CODA analysis was shown separately by recording device, with each device corresponding to a different country.
Extended Data Figure 5 Clinical-variable feature importance
Extended Data Figure 5 examines which clinical variables contributed to the clinical-variable baseline using logistic-regression coefficients and permutation importance. The analyses showed both stable associations across datasets and dataset-specific differences.
- Stable associations: Weight loss was positively associated with TB risk across all three datasets and contributed positively under permutation analysis.Night sweats also showed positive coefficients in the reported analysis.
- Feature importance: Clinical-variable feature importance was assessed using mean logistic-regression coefficients and permutation importance across training datasets.Permutation importance was measured as the mean decrease in ROC-AUC after randomly permuting each feature in the testing fold.
Extended Data Table 1 Study-specific dataset subsets
Extended Data Table 1 describes the filtered, study-specific dataset subsets used after cough extraction and validation controls. The analysis also assessed whether recursive feature elimination improved performance or feature stability.
- Study-specific subsets: The study analyzed selected subsets rather than the released datasets in their entirety, reporting subjects, cough events, subject-level TB+ rates, devices, and validation use.Reported counts reflect publicly accessible data and may differ from original dataset publications.
- Feature selection: Recursive feature elimination generally did not clearly improve mean ROC-AUC or stability across representations and classifiers.The main exception was CLAP embeddings combined with random forest classifiers.
- Feature selection: CLAP feature subsets showed greater overlap than random selection in the assessed random-forest setting, suggesting some stable embedding dimensions.Consistency was evaluated with Jaccard similarity across source datasets and cross-validation folds.
Supplementary B
Supplementary visualizations compare subgroup structure in handcrafted time-frequency features using MMD-based MDS. They complement the main CLAP analysis by examining device, location, dataset, and TB-label organization.
- MMD-based MDS visualizes pairwise subgroup distances in time/frequency-domain features.Subgroups are defined by recording device, location, and TB label.
- t-SNE plots compare CLAP embeddings and time-frequency features across dataset, recording-device, and TB-status groupings.Samples are colored by dataset and device or by dataset and TB status.
- Together, the plots confirm that acquisition-related structure complements the CLAP-based representation analysis.
Supplementary C
The supplementary discussion frames poor transfer as a domain-shift problem and examines whether acquisition-related structure dominates cough representations. It also motivates testing domain-generalization strategies.
- Domain shift is a recognized challenge across computer vision, speech processing, acoustic scene classification, and broader audio analysis.
- Handcrafted time-frequency features separate more strongly by recording device than by TB label, while dataset-domain separation is weaker than for CLAP embeddings.
- CLAP embeddings show stronger organization by dataset source and recording device than by TB status, indicating acquisition-related clustering in classical-model feature spaces.
- The study explores strategies including MixStyle, Mixup, LabelGrad, CrossGrad, domain-adversarial networks, and kernel-based approaches to reduce dependence on dataset-specific features.
Supplementary D
Model development uses nested subject-level cross-validation to separate within-dataset model selection from performance estimation. External datasets remain independent from development and tuning.
- Nested cross-validation partitions each source dataset at the subject level for within-dataset performance estimation.
- The inner loop selects models and hyperparameters within each outer training partition, while held-out outer folds estimate performance.
- External validation datasets are never used during development or tuning, keeping external validation fully independent.
Dataset summary and comparison
The study compares three TB-cough datasets with different collection settings and uses conventional time- and frequency-domain descriptors for fixed-length acoustic representations. The datasets differ in geography, participants, recording devices, and collection protocols.
- Dataset summary and comparison: CODA contains 733,756 cough sounds from 2,143 adults evaluated across seven countries using Android smartphones with country-varying phone models.
- Dataset summary and comparison: TBscreen was collected in Nairobi in a controlled setting from 149 participants with pulmonary TB and 46 controls using three recording devices.
- Dataset summary and comparison: The Zambia dataset was collected by CIDRZ at three clinical sites in Lusaka district from adults with TB-related symptoms, close TB contact, or newly diagnosed HIV.
- Feature construction: Handcrafted features use 0.02 s windows and 0.01 s steps, aggregating frame-level means and standard deviations into fixed-length recording representations.
- Feature construction: The descriptor set includes energy, entropy, spectral, MFCC, harmonic-ratio, fundamental-frequency, and chroma features.
- Representation structure: Supplementary t-SNE analysis shows strong organization by dataset source and recording device in time-frequency descriptors.
Supplementary G DL backbones
The DL pipeline evaluated multiple pretrained audio backbones, each paired with a task-specific head to predict TB status. Supplementary analyses describe nested cross-validation and show feature organization is stronger by dataset than by TB status.
- The DL pipeline evaluated several pretrained backbone models commonly used for audio representation learning.
- ResNet-18 and ResNet-34 use residual convolutional architectures applied to spectrogram inputs for audio classification.Their residual connections support optimization of moderately deep networks.
- VGGish produces compact embeddings from log mel-spectrogram patches using a network pretrained on large-scale audio data.
- OPERA is a pretrained foundation model designed to learn transferable acoustic representations from diverse biomedical and human sound datasets.
- CLAP aligns audio and text in a shared embedding space, while its pretrained audio encoder can serve as a general-purpose acoustic feature extractor.
- A task-specific classification head was appended to each backbone to predict TB status.
- Feature visualizations showed strong organization by dataset and weak organization by TB status.
- Nested cross-validation held out one outer fold for testing and used inner cross-validation within development folds for model-selection procedures.