Source-linked AI summary
From Sound to Symptom: Real-Time Respiratory Signal Understanding for Conversational Healthcare Agents
Tanmay Laud, Herprit Mahal, Subhabrata Mukherjee
TL;DR
Conversational agents currently discard clinically valuable cough signals as noise. HealthCUES adds streaming, dialogue-aware respiratory analysis with structured MLLM predictions; it achieves strong detection and external validation while remaining within conversational latency constraints.
Problem
Existing conversational AI systems discard clinically valuable cough signals instead of analyzing them for respiratory information.
Method
HealthCUES uses turn-aligned rolling-buffer audio, parallel structured Qwen3-Omni prediction tasks, and dialogue-aware gating for real-time respiratory monitoring.
Results
HealthCUES achieves 93% F1 for cough detection, 0.75 weighted-F1 for wet-versus-dry classification, 340ms latency, and 0.91 macro-F1 on AMI three-way validation.
Takeaways & Limitations
HealthCUES provides fine-grained respiratory analytics that support conversational agents in reasoning over paralinguistic health signals.
Takeaways & Limitations
Performance may degrade in high noise or heavily compressed codecs, and the current evaluation is English-only with a three-person user study.
Abstract
from arXiv · showhide
Cough events during live spoken conversations carry clinically valuable respiratory signals, yet existing dialogue systems treat them as acoustic noise to be discarded. We present HealthCUES (Clinical Understanding from Embodied Sounds), a streaming pipeline for paralinguistic respiratory monitoring in real-time conversational agents, a capability that, to the best of our knowledge, is absent from all prior systems. HealthCUES processes audio through a rolling buffer aligned with dialogue turn boundaries, enabling sub-second event detection without interrupting conversational flow. Beyond binary cough detection, the system provides fine-grained analytics: (i) differentiation between coughing and throat clearing, (ii) cough subtype classification (dry, wet, barking, whooping) with confidence scores, and (iii) temporal duration estimation with start-end boundaries. To prevent alert fatigue, HealthCUES introduces dialogue-aware gating mechanisms that modulate triggering based on conversational context. The system leverages Qwen3Omni, a multimodal large language model (MLLM), with constrained structured outputs, decomposing cough analysis into parallel prediction tasks for independent prompt optimization. Evaluation on 847 in-house conversational audio segments demonstrates 93\% F1 for cough detection, 0.75 weighted-F1 for wet/dry subtype classification, and average end-to-end latency of 340ms; external validation on the AMI meeting corpus confirms robust cough, throat-clearing, and speech separation in the presence of speech (0.91 macro-F1). A user study with licensed healthcare professionals confirms the clinical relevance of subtype information and the system's utility in telehealth workflows.
1 Introduction
HealthCUES addresses the gap between clinically valuable cough signals and conversational systems that discard them as noise. It targets real-time, fine-grained respiratory understanding without disrupting dialogue.
- Cough characteristics can distinguish productive from non-productive coughs and identify barking or whooping patterns.
- Real-time analysis must detect events within natural conversational pauses despite noise, codec degradation, and the need for subtype differentiation.The target latency is below 500ms, while repeated alerts require dialogue-aware modulation.
- HealthCUES is introduced as a streaming cough analysis pipeline for live conversational integration.
- The system provides turn-aligned streaming cough detection for real-time conversational agents.
- Fine-grained analytics add four-way subtype classification, confidence scores, and temporal duration boundaries.The authors state these capabilities are absent from prior cough detection systems to the best of their knowledge.
- Dialogue-aware gating uses cooldown, topic suppression, and cough-to-speech ratio monitoring to reduce alert fatigue.
2 Related Work
Prior work covers broad audio events, dedicated cough analysis, dialogue-related paralinguistic signals, and multimodal audio understanding, but HealthCUES combines these directions for conversational respiratory monitoring.
- Audio Event Detection and Cough Analysis: Existing cough systems analyze isolated recordings and provide binary detection without the streaming, conversational setting targeted here.
- Paralinguistic Signals in Dialogue: Dialogue research has used physiological, respiratory, and affective signals to adapt system behavior and evaluate task success or user satisfaction.
- Multimodal Foundation Models: HealthCUES builds on MLLM audio understanding by adding structured output constraints and dialogue-aware post-processing for real-time operation.
3 System Overview
HealthCUES buffers conversational audio around turn boundaries, performs parallel structured respiratory predictions, and gates agent actions using dialogue context. Its outputs include event state, cough subtype, duration, and action-oriented responses.
- Inputs and Streaming Context: A rolling buffer analyzes audio after ASR detects turn completion, using a default 2s look-back to capture events before speech onset.Turn-level processing matches the dialogue agent’s planning granularity while retaining mid-turn and pre-speech coughs.
- Multi-Stage Inference: Qwen3-Omni decomposes cough analysis into parallel structured prediction tasks, allowing independent prompt optimization and graceful degradation.
- Multi-Stage Inference: State detection distinguishes coughing, throat clearing, and none because these states warrant different conversational responses.
- Multi-Stage Inference: Subtype classification labels coughs as dry, wet, barking, or whooping, with confidence scores for each event.These are qualitative cough descriptors rather than standardized diagnostic categories.
- Multi-Stage Inference: Duration estimation returns start and end timestamps and supports short, medium, and long severity buckets.The constrained schema represents subtype confidence pairs and duration time ranges.
- Dialogue-Aware Gating: Dialogue-aware gating combines a 60-second cough-to-speech ratio, quiet periods, and topic suppression to determine whether detections trigger action.Urgent subtypes and prolonged episodes clear lower action thresholds than brief dry coughs.
- Output and Dialogue Integration: Representative outputs show subtype-sensitive thresholds and suppression of redundant follow-up.
4 Evaluation
HealthCUES was evaluated on in-house conversational audio, external meeting speech, subtype classification, and clinician-rated telehealth scenarios. Results show strong cough and respiratory-state detection, sub-500ms latency, and practical dialogue utility, while rarer subtype recognition remains difficult.
- External validation: 0.91 macro-F1 and 0.92 accuracy were achieved on 719 AMI segments containing conversational speech without domain adaptation.Per-class F1 was 0.86 for coughing, 0.94 for throat clearing, and 0.92 for none.
- Subtype evaluation: Dry cough detection reached F1 0.89, while wet cough detection reached F1 0.55 within the 0.75 weighted-F1 wet/dry classification result.Barking and whooping coughs were harder because they were acoustically subtler and less frequent.
- Capability comparison: HealthCUES was the only evaluated system providing streaming, throat-clearing separation, subtype classification, duration estimation, and dialogue-aware gating.The capability comparison defines streaming as turn-aligned rolling-buffer processing and duration as start–end estimation.
- User study: Three licensed nurses confirmed appropriate branching, natural check-in phrasing, and improved information-gathering efficiency in structured telehealth scenarios.They also identified inconsistent detection during overlapping agent speech and gating issues during prolonged episodes.
5 Conclusion
HealthCUES combines MLLM-based audio understanding with dialogue-aware gating to provide fine-grained respiratory analytics for real-time conversational agents. Evaluation supports strong detection, clinically relevant subtype classification, and sub-500ms latency compatible with natural dialogue flow.
- Conclusion: HealthCUES provides fine-grained respiratory analytics for real-time conversational agents through a streaming cough analysis pipeline.The system combines MLLM-based audio understanding with dialogue-aware gating.
- Conclusion: The paper reports strong detection accuracy, clinically relevant subtype classification, and sub-500ms latency compatible with natural dialogue flow.These capabilities support reasoning over paralinguistic health signals in conversational systems.
Ethical Considerations
HealthCUES is positioned as an informational aid that should augment rather than replace clinical judgment, while deployment addresses privacy and fairness risks. Audio is processed transiently, but demographic, condition, equipment, and backend-training differences may affect performance.
- Privacy: Audio is processed transiently without persistent storage, and no content is retained beyond the active session.
- Clinical Limitations: HealthCUES provides informational cough analytics and triggers follow-up questions rather than clinical recommendations.The system is intended to augment, not replace, clinical judgment.
- Bias and Fairness: Cough acoustic variation across demographics, health conditions, and recording equipment may produce performance disparities.The authors recommend ongoing monitoring across user populations.
- Bias and Fairness: The MLLM backend’s training-data composition may bias subtype classification for underrepresented groups.
Limitations
HealthCUES has several evaluation and deployment boundaries, including imperfect subtype ground truth, environmental and language constraints, server-side dependence, and limited user-study scale.
- Evaluation: Subtype ground truth is imperfect because labels involve clinical judgment, with moderate inter-annotator agreement (κ=0.67).
- Signal Conditions: Performance may degrade in high-noise environments (SNR <5dB) or with heavily compressed codecs absent from evaluation.Very soft coughs and coughs co-occurring with loud speech may be missed.
- Generalization: The current evaluation is English-only, so generalization to other languages and accents requires dedicated validation.
- Deployment: Server-based deployment makes end-to-end latency dependent on server compute and introduces MLLM cost and availability constraints.The system does not target edge or fully on-device operation.
- User Study: The user study included 3 participants, so larger studies with diverse populations would strengthen the evidence base.