Source-linked AI summary
The COUGHVID crowdsourcing dataset: A corpus for the study of large-scale cough analysis algorithms
Lara Orlandic, Tomas Teijeiro, David Atienza
TL;DR
COUGHVID addresses the absence of a validated cough database for scalable COVID-19 screening by constructing and publicly releasing a large crowdsourced corpus. The authors filter recordings automatically, add pulmonologist annotations, and validate sample provenance and labels; the resulting dataset supports cough-classification research while expert diagnostic agreement remains limited.
Problem
No validated cough-sound database was available for training machine-learning models for widespread COVID-19 screening.
Method
The authors collected crowdsourced recordings, filtered non-coughs with an open classifier, obtained expert pulmonologist annotations, and checked geographic provenance and label agreement.
Results
More than 20,000 recordings were released, including over 2,000 expert-labeled coughs with diagnoses, severity, anomalies, and broad demographic and geographic metadata.
Takeaways & Limitations
COUGHVID provides a public, richly annotated cough corpus for training models across COVID-19 and other cough-audio classification tasks.
Takeaways & Limitations
Expert agreement on cough diagnosis was poor, with Fleiss’ κ = 0.0031; additionally, binary labels were assigned for any reasonable suspicion.
Abstract
from arXiv · showhide
Cough audio signal classification has been successfully used to diagnose a variety of respiratory conditions, and there has been significant interest in leveraging Machine Learning (ML) to provide widespread COVID-19 screening. However, there is currently no validated database of cough sounds with which to train such ML models. The COUGHVID dataset provides over 20,000 crowdsourced cough recordings representing a wide range of subject ages, genders, geographic locations, and COVID-19 statuses. First, we filtered the dataset using our open-sourced cough detection algorithm. Second, experienced pulmonologists labeled more than 2,000 recordings to diagnose medical abnormalities present in the coughs, thereby contributing one of the largest expert-labeled cough datasets in existence that can be used for a plethora of cough audio classification tasks. Finally, we ensured that coughs labeled as symptomatic and COVID-19 originate from countries with high infection rates, and that their expert labels are consistent. As a result, the COUGHVID dataset contributes a wealth of cough recordings for training ML models to address the world's most urgent health crises.
Background & Summary
COUGHVID addresses the lack of a validated, scalable cough database for COVID-19 screening by providing a large public corpus with automated filtering, expert annotations, and validation procedures.
- COVID-19 screening requires inexpensive, rapid, scalable technology, while cough sounds offer a potential unobtrusive signal for respiratory-condition assessment.
- Existing datasets were limited by unavailable data, small sample sizes, manual filtering, or too few subjects for population-level generalization.
- COUGHVID contains more than 20,000 publicly available recordings, including 1,010 recordings whose contributors claimed COVID-19.
- Three expert pulmonologists annotated a subset to identify whether recordings realistically originated from COVID-19 patients.
- The dataset includes over 2,000 expert-labeled coughs with diagnoses, severity, audible anomalies, and metadata spanning demographics, health statuses, and geographic locations.
- Validity checks required COVID-19-labeled recordings to come from countries with prevalent infection and assessed agreement among expert diagnoses.
Methods
COUGHVID collected short smartphone cough recordings through a minimal web interface, filtered them with an open cough classifier, and selected recordings for expert annotation using stratified sampling.
- Data Collection: Recordings were collected from April 1 to September 10, 2020 through a web application designed around a one-recording, one-click workflow.
- Data Collection: The interface recorded microphone audio for up to 10 seconds and requested optional metadata including age, gender, condition, and geolocation.
- Database Cleaning: The cough detector supplied a cough probability in each record’s metadata so users could exclude non-cough sounds.
- Database Cleaning: The classifier used 121 cough and 94 non-cough examples, preprocessing, 68 audio features, and an XGB model tuned with cross-validation.
- Database Cleaning: 10.4% of recordings with cough_detected below 0.8 actually contained cough sounds, so those recordings were recommended only for robustness assessment.
- Expert Annotation: Three pulmonologists each reviewed 1,000 recordings across quality, cough type, audible abnormalities, diagnosis, and severity categories.
- Expert Annotation: Expert-label recordings were stratified by self-reported status after requiring at least 0.8 cough probability, with 15% reviewed by all three experts.
Data Records
Each COUGHVID recording is distributed as paired audio and JSON metadata files, with a compiled CSV provided for convenient record-level access.
- File Structure: Each recording contains an audio file and a same-named JSON metadata file.
- File Structure: Audio is stored in WEBM or OGG format using the Opus codec at a 48 kHz sampling frequency.
- Metadata: The JSON metadata includes context variables, self-reported information, and expert clinical labels, while geolocation precision was reduced for privacy.
- Compiled Metadata: metadata_compiled.csv contains 40 columns and one row per record, indexed by each recording’s UUID.
Technical Validation
COUGHVID’s technical validation examined demographic and geographic coverage, expert-label agreement, and patterns among expert-labeled COVID-19 coughs. The analyses support broad dataset coverage but reveal limited agreement on diagnosis.
- Demographic representativeness: 77% of subjects were healthy, 15.5% COVID-19 symptomatic, and 7.5% COVID-19 positive, with an average age of 34.4 years.The dataset included 65.5% male and 33.8% female subjects; 81.9% had no pre-existing respiratory conditions.
- Geographic representativeness: 94.4% of COVID-19-labeled recordings and 91.3% of symptomatic recordings came from countries exceeding 20 newly confirmed cases per 1 million people.The analysis used infection statistics from the 14 days before each recording was uploaded.
- Inter-Rater Reliability: Expert agreement was moderate for audible nasal congestion, fair for cough type, and slight for cough severity, nothing specific, wheezing, and dyspnea.Agreement was assessed using Fleiss’ Kappa on 150 recordings labeled by all three pulmonologists.
- Inter-Rater Reliability: Cough diagnosis showed poor expert agreement (KFleiss = 0.0031), with only 22 of 86 coughs labeled COVID-19 by at least one rater reaching majority consensus.Expert 3 labeled none of the common recordings as COVID-19, while Experts 1 and 2 showed slight agreement (KFleiss = 0.0774).
- Trends in Expert COVID-19 Cough Labeling: 87.3% of pooled expert-labeled COVID-19 coughs were dry and 86.2% were mild, while most lacked audible dyspnea, wheezing, stridor, choking, or nasal congestion.The pooled set contained 632 COVID-19-labeled cough records.
Private Set and Testing Protocol
COUGHVID reserves a private expert-labeled test set for reproducible, independently evaluated model assessment while keeping prediction targets and input restrictions broadly flexible.
- Private test set: A private test set was withheld from publication and randomly selected from recordings labeled by at least one expert.The set was created to support reproducibility of experiments using COUGHVID.
- Testing protocol: Researchers may request independent evaluation on the private test set after demonstrating promising public-dataset results through cross-validation.An independent evaluator obtains the performance measurements under the repository’s protocol.
- Prediction tasks: Every variable except datetime and cough_detected may serve as a prediction target, enabling tasks from cough identification and sound-quality assessment to condition, age, or gender prediction.Models may also use metadata such as subject age or gender as inputs, with minimal restrictions on input and output variables.
Author contributions statement
The authors divided responsibility across research design, signal processing, data collection, expert coordination, metadata compilation, analysis, and manuscript preparation.
- Research design: L.O. and T.T. devised the idea and research.
- Modeling and analysis: L.O. developed the signal-processing code and cough-detection model and analyzed user metadata and expert labels.
- Data collection and manuscript: T.T. managed the data-collection website, coordinated expert labeling, and compiled metadata, while all authors contributed to writing and editing.
Figures & Tables
The paper’s figures and tables document the dataset’s metadata, expert-label consistency, geographic COVID-19 context, and cough-diagnosis disagreements.
- Table 1 lists metadata variables captured in the dataset’s JSON files.The metadata includes context, self-reported information, and expert clinical assessments.
- Table 2 organizes variables supplied by expert annotators for clinical cough assessment.Expert annotation fields include quality, cough type, and diagnosis variables.
- Table 3 reports inter-expert label consistency across common cough recordings.Agreement ranges from moderate for audible nasal congestion to slight or fair for several other attributes.
- Figure 2 combines cumulative COVID-19 cases per 1 million population with GPS coordinates of received recordings.This visualization supports checking whether COVID-19-labeled recordings originated in areas with high infection rates.