Source-linked AI summary
SuperSenseDoctor: A Multimodal and Contactless Agent for Health Tracking
Xuwen Zhang, Zijian Lu, Yicheng Lei, Rui Qiu, Jiale Li, Yiping Zuo, Weibei Fan, Fu Xiao
TL;DR
Home health tracking needs contactless measurements that preserve uncertainty while supporting care action, since conventional fusion can hide cross-modal evidence and raw-stream language-model prompting is costly and difficult to audit. SuperSenseDoctor combines WiFi BFI, mmWave, and thermal sensing into a persistent state, then uses deterministic event screening and bounded, evidence-grounded Agent reasoning. Its prototype reports vital-sign and fall-recognition results alongside a high criterion-level Agent checklist pass rate.
Problem
Home health tracking needs contactless measurements that preserve enough uncertainty to support care action, because conventional fusion can hide cross-modal evidence and raw-stream language-model prompting is costly and difficult to audit.
Source: §1 · p. 1Method
SuperSenseDoctor combines WiFi BFI, mmWave radar, and thermal sensing into a persistent state, preserving modality evidence and quality events for deterministic screening and event-driven Agent reasoning.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2Results
Heart-rate MAE/RMSD were 1.994/3.142 bpm, respiratory-rate MAE/RMSD 0.197/0.263 bpm, fall-recognition accuracy 96.5%, and Agent checklist pass rate 96.7%.
Source: §1 · p. 1, §4.2 · p. 4Takeaways & Limitations
The prototype provides an inspectable contactless sensing-to-care-support path, with raw streams local and high-risk actions requiring authorized confirmation.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Population aging is increasing the need to monitor older adults safely and independently at home. However, cameras, wearables, and manual checks often introduce privacy, adherence, and attention burdens that hinder sustained health monitoring. This paper presents SuperSenseDoctor, a multimodal contactless agent architecture for long-term home health tracking. The system transforms WiFi, mmWave radar, and surface temperature into a persistent human health state. The system relies on fixed decision rules to conduct continuous daily monitoring and respond to pre-defined hazards. When abnormal signals appear, event-driven reasoning analyzes only standardized evidence to produce traceable care-support measures. In this manner, SuperSenseDoctor integrates sensing, temporal state, reasoning, and action into a unified and auditable loop. The calibrated multimodal pipeline achieves 1.994 bpm mean absolute error (MAE) and 3.142 bpm root mean square deviation (RMSD) for heart rate, 0.197 bpm MAE and 0.263 bpm RMSD for respiratory rate, and 96.5% fall-recognition accuracy. The evaluation also covers 2686 one-second states across 9 chronological intervals and reaches a 96.7% criterion-level Agent checklist pass rate. These results demonstrate the feasibility of a stateful contactless sensing-to-action architecture for long-term home health monitoring.
1 Introduction
Home health tracking needs contactless measurements that preserve uncertainty while supporting timely care action. SuperSenseDoctor connects multimodal sensing to an inspectable health state and evidence-grounded care-support actions.
1 Introduction
- Contactless measurements should remain separate enough to explain uncertainty and connected enough to support timely care action.The motivating scenario combines behavioral sensing, thermal trends, WiFi BFI, and mmWave radar as partial evidence.Source: §1 · p. 1
- Conventional fusion can hide cross-modal evidence, while sending raw streams to a language model is costly and difficult to audit.WiFi BFI and mmWave can disagree when posture, bedding, or line-of-sight conditions change.Source: §1 · p. 1
- SuperSenseDoctor preserves per-modality evidence and quality events so downstream components can distinguish possible health changes from unreliable sensing.WiFi BFI, mmWave radar, and thermal sensing feed a structured state and event-driven Agent collaboration.Source: §1 · p. 1
- The prototype reaches heart-rate MAE and RMSD of 1.994 and 3.142 bpm, respiratory-rate MAE and RMSD of 0.197 and 0.263 bpm, and 96.5% fall-recognition accuracy.It also achieves complete state coverage across 9 chronological intervals and a 96.7% criterion-level Agent checklist pass rate.Source: §1 · p. 1
2 SuperSenseDoctor
SuperSenseDoctor arbitrates complementary contactless observations into a persistent multimodal state, then uses event-gated Agents to turn standardized evidence into traceable care-support actions.
2.1 Multimodal State Fusion
- WiFi BFI provides wide-area motion and periodic-signal cues, while mmWave adds localized chest-motion and body-movement evidence.Adapters derive vital-sign candidates, assess falls from short-window motion and posture cues, and use thermal and behavioral streams for contextual evidence.Source: §2.1 · p. 2
- When reliable radio branches agree within tolerance, confidence-aware arbitration computes a weighted mean; when plausible branches conflict, it preserves branch evidence and emits a quality event.The conflict-preserving approach makes uncertainty available to downstream reasoning rather than hiding it in a conventional early-fusion estimate.Source: §2.1 · p. 2
- The temporal updater stores selected values and unresolved evidence in a unified state carrying vital signs, activity, temperature, confidence, posture, fall status, quality events, and provenance.Thermal and behavioral evidence remain contextual rather than being collapsed into vital-sign estimates.Source: §2.1 · p. 2
2.2 Event-Driven Agent Layer
- The Nurse Agent screens each state against thresholds, personal baselines, persistence, and sensing-quality flags; normal windows update history without invoking the LLM.Falls, baseline shifts, extreme respiratory values, modality conflicts, weak dual sensing, and NLOS degradation become typed HealthEvents; high-risk combinations take a reflex path.Source: §2.2 · pp. 2–3
- The Diagnosis Agent uses registered tools and a project-policy card to produce a typed triage decision, while the Report Agent links decisions to sensing evidence, uncertainty, action channels, and episode logs.Clinician review can refine personal baselines for later monitoring, feeding back into future Nurse Agent thresholds.Source: §2.2 · p. 3
3 Implementation
The prototype combines room-side radio and thermal sensors with a local hub and asynchronous software chain, while retaining local sensing records and a structured interface for model use.
3 Implementation
- The room-side prototype uses Comfast WU785AC for WiFi BFI, TI AWR1843 for radar, and MLX90640 for thermal input, with an Intel NUC hosting the local hub and Agent runtime.Figures 3 and 4 show representative fused heart-rate and respiratory-rate traces against their references.Source: §3 · p. 3, Figure 3 · p. 3, Figure 4 · p. 3
- An asynchronous Python event chain aligns sensor data into state objects, persists them to SQLite, and connects agents through an event bus and lightweight tool registry.FastAPI provides the dashboard, configured derived files support replay, and deterministic synthetic states isolate controlled Agent behavior from sensing metrics.Source: §3 · p. 3
- Raw and derived sensing records remain local; an external model receives minimized structured event context, and the Agent contract includes access control, retention policy, encryption, and resident consent.A local model can replace the external endpoint without changing the Agent contract.Source: §3 · p. 3
4 Evaluation
Evaluation tests sensing across multiple time intervals and whether the Agent converts standardized situations into policy-matched records and actions.
4.1 Multi-Time-Interval Sensing Validation
- Sensing validation covered 2686 one-second state rows across nine intervals, using a watch heart-rate reference, a respiration-belt reference, and labeled fall-event replays.The intervals comprise seven 300 s recordings, one 249 s recording, and one 337 s recording.Source: §4.1 · p. 3
- Heart-rate MAE fell 51.6% vs mmWave and 52.9% vs WiFi BFI; respiratory-rate MAE fell 87.9% vs mmWave and 92.7% vs WiFi BFI; fall recognition reached 96.5% accuracy.Table 1 reports vital-sign MAE and RMSD in bpm.Source: §4.1 · pp. 3–4, Table 1 · p. 3
4.2 Agent Scenario Evaluation
- The Agent benchmark tests 60 manually specified state objects and 24 hidden-answer triage scenarios with expected tiers, channels, evidence anchors, and care-support boundaries.Deterministic cases cover normal signals, deviations, falls, modality conflicts, low-confidence input, compound risk, and threshold boundaries.Source: §4.2 · p. 4
- The Agent suite passed 206 of 213 criteria (96.7%); deterministic checks ran at 0.56 ms P95 and model-involved triage at 6560.7 ms P95 latency.Event screening matched 58 of 60 cases, fusion arbitration 23 of 24, and triage returned the expected tier and channel in 22 of 24 scenarios.Source: §4.2 · p. 4
5 Future Outlook and Development
The prototype provides an inspectable care-support path, while planned development extends health-state tracking across routines and rooms and toward screening support.
5 Future Outlook and Development
- The current prototype exposes Nurse events, evidence, action tiers, delivery channels, and persisted EpisodeLogs; raw streams stay local and high-risk actions require authorized confirmation.External models receive compact structured events.Source: §5 · p. 4
- Personalized baselines could connect respiratory, cardiac, activity, sleep, and surface-temperature patterns across days and rooms, with the Nurse detecting routine changes and the Diagnosis Agent retrieving relevant episode history.Clinician review could refine baseline ranges, evidence priority, and follow-up policy.Source: §5 · p. 4
- The shared multimodal state, event gate, Agent tools, and episode log could support screening across cue-linked condition families, while clinical interpretation remains with qualified professionals.Mapped cues include respiratory, cardiac, gait, activity, sleep, and surface temperature; condition families include COPD exacerbation, sleep apnea, heart-failure burden, Parkinsonian movement change, cognitive decline, and acute respiratory infection.Source: §5 · p. 4