Source-linked AI summary
COVID-19 and Computer Audition: An Overview on What Speech & Sound Analysis Could Contribute in the SARS-CoV-2 Corona Crisis
Björn W. Schuller, Dagmar M. Schuller, Kun Qian, Juan Liu, Huaiyuan Zheng, Xiao Li
TL;DR
The paper asks how computer audition might contribute to addressing COVID-19, where audio diagnosis is not established and short samples may not distinguish it from similar illnesses. It surveys existing speech and sound analysis capabilities, potential applications, and practical challenges, concluding that these capabilities offer potential for pre-diagnosis and monitoring.
Problem
The authors note that no study had investigated audio from COVID-19 patients against varied controls, and that short audio samples may not distinguish COVID-19 from influenza or a cold.
Source: §3.2 · p. 3Method
Results
The overview finds that multiple audio analysis tasks have already been demonstrated feasible and identifies potential applications in risk assessment, symptom-history diagnosis, and monitoring.
Source: §5 · p. 5Takeaways & Limitations
The authors identify symptom tracking over time as a promising basis for audio-based differentiation among COVID-19, influenza, and cold.
Source: §3.2 · p. 3Publicly available, annotated audio data from COVID-19 patients was unavailable for research, and the authors call for patient and matched-control data collection.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
At the time of writing, the world population is suffering from more than 10,000 registered COVID-19 disease epidemic induced deaths since the outbreak of the Corona virus more than three months ago now officially known as SARS-CoV-2. Since, tremendous efforts have been made worldwide to counter-steer and control the epidemic by now labelled as pandemic. In this contribution, we provide an overview on the potential for computer audition (CA), i.e., the usage of speech and sound analysis by artificial intelligence to help in this scenario. We first survey which types of related or contextually significant phenomena can be automatically assessed from speech or sound. These include the automatic recognition and monitoring of breathing, dry and wet coughing or sneezing sounds, speech under cold, eating behaviour, sleepiness, or pain to name but a few. Then, we consider potential use-cases for exploitation. These include risk assessment and diagnosis based on symptom histograms and their development over time, as well as monitoring of spread, social distancing and its effects, treatment and recovery, and patient wellbeing. We quickly guide further through challenges that need to be faced for real-life usage. We come to the conclusion that CA appears ready for implementation of (pre-)diagnosis and monitoring tools, and more generally provides rich and significant, yet so far untapped potential in the fight against COVID-19 spread.
1. Introduction
COVID-19 was declared a pandemic amid rapid spread and extensive social-distancing measures. The introduction presents computer audition as an underused AI resource whose speech and sound analysis could contribute to the crisis response.
1. Introduction
- At the time of writing, reported global cases numbered 245 484, with 10 031 deaths and 86 035 recovered; long-term effects after recovery remained unresearched.The introduction describes the spread as assumed to follow exponential growth and notes widespread shutdown measures intended to slow it.Source: §1 · p. 1
- AI had been used for COVID-19 spread prediction [1] and CT-image screening [2], while smartphone sensor signals were also proposed for diagnosis [4].Source: §1 · p. 1
- Computer audition applies computing to audio processing, including speech and sound analysis, machine listening, computational paralinguistics, and synthesis.The authors say this resource had not yet been used in the COVID-19 context despite its potential.Source: §1 · p. 1
- The overview first surveys CA-assessable phenomena, then potential use-cases, and finally technical, ethical, and societal challenges.Source: §1 · p. 1
2. Computer Audition: Related Phenomena
The section surveys speech and sound phenomena that computer audition has already targeted and that may be relevant to COVID-19 applications. These include symptom-related vocal and bodily sounds, speaker traits, and environmental audio cues.
2.1. Speech Analysis
- Speech analysis has targeted cold speech, breathing patterns, masks, emotion, infant sounds, eating, sleepiness, pain, age, gender, and health states.These tasks are presented as potentially relevant to symptom assessment and individual risk monitoring.Source: §2.1 · p. 2
- Speaker diarisation and counting may support social-distancing monitoring, while deception, depression, and fear recognition are relevant to contact questioning and wellbeing.Spoken-language processing can also use textual cues through automatic speech recognition and natural-language processing [23].Source: §2.1 · p. 2
2. Computer Audition: Related Phenomena
- Table 1 presents the interdependence of Computer Audition (CA) tasks and potential use-cases in the Corona crisis; health state includes asthma, head and neck cancer, or smoking habits that can be inferred from speech audio.Source: Table 1 · p. 2
2.2. Sound Analysis
- Sound analysis can recognize cough types, sneezes and throat sounds, and respiratory or lung sounds associated with pneumonia.The section also notes snoring, heart status, local proximity, 3D localization, diarisation, and audio separation or denoising as relevant capabilities [34].Source: §2.2 · pp. 2–3
3. Potential Use-Cases
The overview maps existing audio-analysis capabilities to COVID-19 use-cases, from risk assessment and diagnosis to monitoring spread, social-distancing effects, treatment, and recovery. It also considers audio generation as a way to support communication and render adaptive alarms.
3.1. Risk Assessment
- Voice traits such as age, gender, and health state could inform individual mortality-risk estimates, while speaker counts, locations, and masks could support real-time ambient warnings.Source: §3.1 · p. 3
3.2. Diagnosis
- The authors caution that short audio samples may not distinguish COVID-19 from influenza or colds, since relevant respiratory sounds may not differ significantly.Source: §3.2 · p. 3
- Longitudinal symptom histograms, especially symptom-onset patterns, are proposed to help distinguish COVID-19 from influenza and colds and support remote screening.The authors contrast this with uncertainty about diagnosis from short audio samples and describe sharing histograms or targeted audio examples with clinicians.Source: §3.2 · p. 3
3.3. Monitoring of Spread
- Audio analysis of telephone or other spoken conversations could detect speech under cold or other symptoms and, with GPS or call-origin data, help establish real-time spread maps.Source: §3.3 · p. 3
3.4. Monitoring of Social Distancing and Effects
- CA could monitor social-distancing effects through interaction, speaking time, emotion, depression, and suicidal-risk indicators, and assess public distancing through speaker location and symptoms.The proposal also includes identifying crowded or under-spaced settings and detecting mask use in public spaces.Source: §3.4 · pp. 3–4
3. Potential Use-Cases
- Table 2 offers an indicative qualitative comparison of COVID-19, cold, and influenza symptoms using ternary levels and symptom-onset gradients.Source: §3.2 · p. 3, Table 2 · p. 4
3.5. Monitoring of Treatment and Recovery
- During treatment and recovery, CA could track progress by updating symptom histograms and monitoring wellbeing, including emotion, eating habits, fatigue, and pain.Source: §3.5 · p. 4
3.6. Generation of Speech and Sound
- AI speech conversion and synthesis could fill cough-related gaps or enhance voices affected by pain or fatigue, while generated alarms could adapt to ambient sound.The alarm signals are proposed to be mnemonic and recognizable.Source: §3.6 · p. 4
4. Challenges
Real-world CA use faces epidemiological urgency alongside data, robustness, efficiency, trust, and ethical constraints. The overview identifies concrete requirements for data collection and sharing, deployment conditions, reliable outputs, and privacy.
4.1. Time
- Risk assessment, diagnosis, monitoring, and treatment are urgently needed amid projected spread and uncertain persistent immunity.The passage notes model calculations indicating heavy spread over the next 6–9 months and says enhanced social distancing might delay spread.Source: §4.1 · p. 4
4.2. Collecting COVID-19 Patient Data
- COVID-19 audio data are not publicly available for research, so diagnosed patients and matched controls need to be recorded and annotated.The proposed controls should have equal recording conditions and demographic characteristics.Source: §4.2 · p. 4
4.3. Model Sharing
- Federated machine learning could support data exchange by sharing personal model parameters rather than audio.The passage also describes users donating data in a safe and private manner.Source: §4.3 · p. 4
4.4. Real-world Audio Processing
- Real-world audio may contain noise, reverberation, transmission loss, and other disturbances, while systems should handle multilingual, multicultural, and local variability.Source: §4.4 · p. 4
4.5. Green Processing
- Mobile and large-scale telephone analysis make energy, transmission, memory, and model-update efficiency important.Source: §4.5 · p. 4
4.6. Trustability of Results
- Because CA methods are prone to error, their error probability needs estimation and communication, with explainable results an additional goal.The passage notes that explainable AI is a young discipline, though its method inventory is growing.Source: §4.6 · p. 5
4.7. Ethics
- Monitoring applications carry substantial ethical responsibility; privacy is crucial, while legal and societal implications remain outside this overview’s discussion.Source: §4.7 · p. 5
5. Concluding Remarks
The overview synthesizes feasible AI-based audio-analysis tasks, potential COVID-19 use cases, and real-life challenges. It concludes that CA has substantial untapped potential, while noting that further possibilities and insights from other disciplines remain valuable.
- The overview presents feasible AI-based audio tasks and use cases spanning individual risk assessment, symptom-histogram diagnosis, monitoring, and audio generation.Monitoring targets include spread, social distancing, treatment, and recovery.Source: §5 · p. 5
- The authors conclude that rapid exploitation of computer audition’s largely untapped potential matters in the COVID-19 response.Source: §5 · p. 5
- The short overview does not include all possible use cases, and the authors say further interdisciplinary insights would be valuable.Source: §5 · p. 5