Source-linked AI summary
Evaluating Deep Multivariate Imputation Models on Wearable Device Data
Skye Goodman, Roussel Desmond Nzoyem, Leandro Junges, Peter Kissack, Yasser Qureshi, Amberly Brigden, Jeff Clark, Nawid Keshtmand
TL;DR
Wearable imputation benchmarks often use random-point masking despite sensor-coupled, temporally structured missingness. Using one participant’s Garmin smartwatch data, the paper introduces realistic block-mask evaluation and matched training; model rankings then depend on feature dynamics, gap severity, and whether error or distributional fidelity is measured.
Problem
Realistic evaluation of deep imputation methods remains unresolved because random-point holdout assumes missingness is independent across features and time, unlike wearable sensor failures.
Method
The paper mines contiguous missing-run templates, stratifies them by per-feature gap-length quantiles, injects co-missing block masks, and trains models on the same missingness distribution.
Results
No single model dominates: matched training lowers BRITS severe-bucket MAE by 43%, extended BRITS performs better on dynamic features, and SAITS has better distributional fidelity.
Takeaways & Limitations
Imputation model rankings strongly depend on evaluation design and feature characteristics, so a single model should not be deployed uniformly across wearable features.
Takeaways & Limitations
The study uses data from a single participant, and low overall MAE may not translate into improved epilepsy forecasting because deep models under-represent important distributional tails.
Abstract
from arXiv · showhide
Wearable device data enables continuous health monitoring, but suffers from structured missingness: features sharing a physical sensor drop out together. Deep imputation methods such as BRITS and SAITS have seen limited evaluation on multimodal physiological data under realistic missingness, and existing benchmarks use random-point holdout protocols that incorrectly assume missingness is independent across features and time. Using data from a person with epilepsy recorded on a Garmin smartwatch, we develop an evaluation protocol that mines contiguous missing-run templates from training data, stratifies them by per-feature gap-length quantiles, and injects them as block masks with preserved co-missingness structure. A matched training protocol exposing models to the same missingness distribution reduces BRITS's severe-gap MAE by 43%, demonstrating the potential benefit of the proposed evaluation and training protocol within this single-participant dataset. We further extend BRITS with time-of-day encoding and a circadian harmonic channel. No single model dominates: linear interpolation is optimal for slow-moving features over short gaps; extended BRITS achieves lower MAE on dynamic cardiac features in moderate and severe gaps; and SAITS better preserves the ground-truth distribution by Jensen-Shannon distance despite higher MAE. Ultimately, model rankings strongly depend on evaluation designs. By exposing how traditional evaluation methods obscure true model capabilities, our transferable protocol establishes critical steps towards developing better imputation strategies for future multi-sensor wearable datasets.
1 Introduction
Wearable imputation is important for clinical monitoring but is commonly evaluated with random-point masking that misrepresents sensor-coupled, temporally structured failures. This paper proposes realistic evaluation and matched training protocols and reports substantially different model behavior under them.
- Wearable missingness arises from sensor failures, poor skin contact, and connectivity dropouts, directly affecting downstream learning.
- Random-point holdout assumes missingness is independent across features and time, unlike sustained co-dropouts and highly variable gap lengths in wearable data.
- The study uses physiological signals from a Garmin smartwatch worn by one person with epilepsy to evaluate wearable imputation methods.
- The proposed evaluation mines contiguous missing-run templates, stratifies them by per-feature gap-length quantiles, and injects block masks preserving co-missingness.
- 43% lower severe-bucket MAE is reported for BRITS under matched training, while the extended model reduces severe-missingness heart-rate MAE by 11% and sleep MAE by 29%.
2 Multi-Sensor Wearable Dataset Exploration
The dataset contains multimodal Garmin smartwatch signals with strong sensor-coupled co-missingness, heterogeneous outage durations, and meaningful cross-feature relationships. These properties motivate feature-specific, physically realistic masking.
- Nine physiological time series were collected from a Garmin smartwatch for one participant with epilepsy and resampled onto a 60-second grid.
- The features derive mainly from PPG and accelerometer sensors, creating strong co-missingness across related channels.
- P(ibi missing | hr missing) = 1.00 and P(device stress missing | hr missing) ≈0.99, while accelerometer failure causes near-total channel dropout.
- Gap lengths differ substantially by feature: pulseOx has mostly single-timestep dropouts, whereas hr, ibi, sleep, and bodyBattery have longer outages.
- The data show clusters including near-deterministic hr/ibi correlation, sympathetic-activation relationships, and activity–recovery associations.
3 Methodology
The methodology mines realistic multivariate outage templates, uses them for severity-aware evaluation and matched training, and augments BRITS with time-of-day and circadian signals. The design explicitly addresses helper-feature collapse during severe co-missingness.
- Wearable-realistic masking: Missing runs are mined as multivariate templates recording which other features are simultaneously missing, then grouped into typical, moderate, and severe severity buckets.
- Wearable-realistic masking: Evaluation injects sampled templates as target blocks while preserving helper-feature co-missingness and requiring physically plausible placements.
- Severity: Severe gaps account for 60.7–88.5% of naturally missing timesteps across features, despite representing only the upper quartile of missing episodes.
- Matched training: The original BRITS objective rarely exposes the model to structured co-missingness, because most observed entries have nearly complete helper availability.
- Matched training: Matched training adds block-mask reconstruction for BRITS and substitutes block masks for random-point MIT masking in SAITS, with a severity curriculum.
- Temporal context: Time-of-day sine–cosine encoding and weekly refitted per-feature 24-hour harmonics provide auxiliary temporal context independent of sensor availability.
4 Experimental
The experiments compare BRITS, SAITS, and classical baselines using standardised-value MAE, physical-unit sMAPE, and distributional Jensen–Shannon distance. Results are averaged over matched seeds and sampled templates.
- The study compares BRITS and SAITS with last observation carried forward and linear interpolation under fixed training and evaluation settings.
- MAE is computed on standardised values, while sMAPE is computed in physical units after inverse-standardising.
- Jensen–Shannon distance measures distributional fidelity using a 50-bin discrete approximation on shared support.
- All results are averaged over 10 seeds using identical sampled templates across models for matched-pair comparison.
5 Results
Under wearable-realistic block masking, matched training substantially improves BRITS, while model strengths vary by feature dynamics and evaluation metric. BRITS-ext is strongest for dynamic cardiac signals, linear interpolation for slowly varying features and short gaps, and SAITS for distributional fidelity.
- Temporal context extension: BRITS-ext consistently improves cardiac and stress features across severities, with severe-gap gains of 11% for hr, 13% for ibi, and 29% for sleep.The extension adds time-of-day and circadian channels; sleep receives no dedicated circadian channel.
- Full quantitative comparison: Linear interpolation remains competitive for slowly varying features and short gaps, while the performance gap between deep models and univariate baselines widens with severity.LOCF is consistently the weakest baseline.
- Full quantitative comparison: BRITS-ext reduces severe-gap device-stress MAE from 0.583 with LI to 0.470, while its improvement over LI for hr grows from 7% to 21% across severity buckets.These gains occur where cross-modal relationships and dynamic temporal behavior can be exploited.
- Full quantitative comparison: For slowly varying bodyBattery, LI achieves MAE 0.012 versus 0.095 for BRITS-ext across short-to-moderate gaps; LI also remains better for severe pulseOx gaps.The findings support feature-specific model selection rather than one uniform imputation strategy.
- Matched training protocol: Matched wearable-realistic block training lowers BRITS average bucket MAE by 34% in typical, 45% in moderate, and 43% in severe gaps.SAITS changes less: typical performance is essentially unchanged, while moderate and severe MAE improve by 12% and 20%.
- Distributional fidelity: SAITS consistently achieves lower JSDist than BRITS-ext in moderate and severe buckets, despite sometimes having higher pointwise MAE.BRITS-ext is more accurate at individual timesteps, whereas SAITS better preserves the overall distributional shape.
6 Limitations and Future work
The study identifies limitations in distributional fidelity, participant scope, and downstream forecasting evaluation, while proposing feature-specific hybrid imputation as future work.
- Limitations: Both deep models under-represent clinically relevant heart-rate and inter-beat-interval tails, so low overall MAE may not improve epilepsy forecasting.The missing tails are where peri-ictal autonomic changes are expected to occur.
- Limitations: Evaluation on a single participant limits how far the observed results can be generalized because missingness patterns and physiological baselines vary across individuals.
- Limitations: End-to-end forecasting evaluation was outside scope because it would require precise seizure timing and substantial computational resources unavailable from the weekly self-reported diary.
- Future work: A hybrid framework could select interpolation or deep models per feature using signal variability and helper availability.Linear interpolation suits slowly varying features and short gaps, while deep models outperform baselines on more dynamic features.
7 Conclusion
The conclusion finds that wearable-realistic block masking and matched training substantially alter imputation results. It recommends evaluation designs and feature-specific deployment choices that reflect each signal’s missingness and dynamics.
- Conclusion: 43%: BRITS severe-bucket MAE improves when training uses the same missingness distribution encountered during evaluation.
- Conclusion: No single model dominates: linear interpolation suits slow-moving short-gap features, extended BRITS dynamic features, and SAITS distributional fidelity.
- Conclusion: Imputation model rankings depend strongly on evaluation design, making wearable-realistic protocols important for assessing model capabilities.
- Conclusion: The protocol can be adapted to multi-sensor wearable datasets when contiguous-run statistics can be mined.
A Related Work
Prior work includes efficient classical imputers and increasingly capable deep models, but BRITS and SAITS have mainly been evaluated with random-point masking that does not reflect smartwatch sensor failures.
- Classical methods: Classical baselines include mean/median imputation, LOCF, interpolation, Kalman filtering, MICE, and k-nearest neighbours, each with applicability limits.Linear interpolation fits slowly varying signals over short gaps, whereas Kalman filtering relies on linear-Gaussian dynamics.
- Deep learning methods: BRITS uses bidirectional processing, feature-wise regression, cross-feature correlations, and a forward–backward consistency loss.
- Deep learning methods: Diffusion-based FGTI offers strong generative performance but at substantial computational cost.
- The evaluation gap: Random-point holdout assumes fixed-proportion, largely feature-specific missingness, unlike wearable failures coupling features that share physical sensors.Neither BRITS nor SAITS had been evaluated on smartwatch data in the cited passage.
B Training Protocol Ablation
The ablation shows that matched training helps SAITS across several feature–severity combinations but can leave some features unchanged or worse, revealing schedule–feature interactions.
- SAITS feature-level results: SAITS improves cardiac-feature MAE across buckets and roughly halves sleep MAE in moderate and severe gaps.For severe gaps, hr changes from 0.693 to 0.533 and ibi from 0.638 to 0.480; sleep changes from 0.639 to 0.395.
- SAITS feature-level results: Under the original schedule, SAITS has the lowest deep-model MAE for pulseOx in typical gaps and device stress in typical and moderate gaps.The reported values are pulseOx typical 0.156 and device stress typical/moderate 0.217 and 0.216.
- Schedule interactions: PulseOx severe MAE is lower under the original schedule, while device stress reverses in severe gaps under the modified schedule.PulseOx is 0.464 versus 0.494, whereas device stress is 0.575 versus 0.487.
- Schedule interactions: PulseOx’s mostly single-timestep dropouts with near-full helper availability leave little for wearable-realistic training to fix.Device stress may instead reflect late curriculum weighting interacting with helper availability.