Source-linked AI summary
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement
Xin Liu, Josh Fromm, Shwetak Patel, Daniel McDuff
TL;DR
Remote telehealth needs objective cardiopulmonary measurements without patient contact. The paper introduces MTTS-CAN, an on-device video model combining temporal shifts, attention, and multi-task learning, and reports strong accuracy with real-time performance across benchmark datasets. Its broader use requires transparent, consent-based deployment because non-contact sensing can expose physiological information without a subject’s knowledge.
Problem
Video-based telehealth often cannot objectively assess physiological state, while accurate, efficient, and privacy-conscious non-contact cardiopulmonary measurement remains needed.
Method
MTTS-CAN uses temporal shift modules, attention, and shared representations to jointly estimate pulse and respiration from video on-device.
Results
The proposed models reduce errors by 25-50% across datasets relative to 2D-CAN, while MTTS-CAN requires 6 ms per frame for inference.
Takeaways & Limitations
The approach improves the reported accuracy and inference speed of on-device contactless cardiopulmonary measurement for real-time applications.
Takeaways & Limitations
Non-contact sensing can enable covert measurement of personal physiological information, so deployment must be transparent and consent-based.
Abstract
from arXiv · showhide
Telehealth and remote health monitoring have become increasingly important during the SARS-CoV-2 pandemic and it is widely expected that this will have a lasting impact on healthcare practices. These tools can help reduce the risk of exposing patients and medical staff to infection, make healthcare services more accessible, and allow providers to see more patients. However, objective measurement of vital signs is challenging without direct contact with a patient. We present a video-based and on-device optical cardiopulmonary vital sign measurement approach. It leverages a novel multi-task temporal shift convolutional attention network (MTTS-CAN) and enables real-time cardiovascular and respiratory measurements on mobile platforms. We evaluate our system on an Advanced RISC Machine (ARM) CPU and achieve state-of-the-art accuracy while running at over 150 frames per second which enables real-time applications. Systematic experimentation on large benchmark datasets reveals that our approach leads to substantial (20%-50%) reductions in error and generalizes well across datasets.
1 Introduction
Telehealth is expanding, but video visits lack objective physiological data, especially for cardiopulmonary symptoms. The paper proposes an accurate, efficient, privacy-conscious on-device approach for measuring vital signs from facial video.
- Telehealth appointments increased by more than 10x during the pandemic, with benefits including reduced infection exposure and improved access for remote patients.
- Video visits generally cannot objectively assess physiological state, leaving physicians without important data for remote decisions about cardiopulmonary symptoms.
- Facial cameras can estimate heart rate, respiration rate, and blood oxygenation from subtle physiological changes in light reflected from skin.
- Existing non-contact methods need improved accuracy, on-device efficiency, privacy protection, and high frame rates for practical remote monitoring.
- MTTS-CAN combines temporal shifts, attention, and multi-task learning to jointly estimate pulse and respiration while addressing privacy, portability, and precision.
- The paper evaluates the system on two large public datasets and reports more than 150 frames per second on-device latency across architectures.
3 Method
The method combines an optical reflection model with temporal shift, attention, and multi-task learning to estimate pulse and respiration efficiently from video. MTTS-CAN shares representations across related physiological processes while targeting real-time on-device operation.
- 3.1 Optical Model: The optical model represents camera RGB measurements through luminance, specular reflection, diffuse reflection, physiological changes, and sensor noise.It decomposes stationary and time-dependent components to distinguish physiological variation from non-physiological effects.
- 3.1 Optical Model: Pulse and respiration are modeled jointly because their signals are closely intertwined and can share redundant information.The paper notes respiratory sinus arrhythmias and related reflection changes as physiological links between the signals.
- 3.2 Architecture: The architecture targets spatial feature learning, temporal noise suppression, physiological relationship modeling, and real-time deployment.It maps raw RGB values to latent pulse and respiration representations while addressing head motion and illumination changes.
- 3.2 Architecture: TS-CAN replaces 3D convolutions with temporal shifts that exchange information across neighboring frames without adding parameters.The motion branch uses shifted tensor chunks, while the appearance branch processes one averaged frame and supplies attention information.
- 3.2 Architecture: An attention module focuses processing on physiological pixels and reduces noise amplification introduced by temporal shifting.The module is inserted into the temporal-shift design to improve target-signal emphasis amid nonuniform physiological signals.
- 3.2 Architecture: MTTS-CAN shares an intermediate representation between BVP and respiration estimation, shrinking the computational budget by approximately 50%.This multi-task variant replaces two independent networks with a jointly trained model.
4 Experiments
The experiments compare proposed architectures with signal-processing and neural-network baselines across public datasets and motion conditions. They measure pulse and respiration accuracy, signal quality, computation, and ARM-CPU latency.
- 4 Experiments: AFRL contains 300 videos from 25 participants recorded at 658x492 resolution and 120 fps, down-sampled to 30 fps for experiments.The dataset includes still, controlled head-motion, and random-orientation tasks across two backgrounds.
- 4 Experiments: The proposed experiment variants share a two-branch architecture with four convolutional layers per branch, pooling, and dropout.The architectures differ in their convolutional operations, including 2D and 3D convolutions.
- 4 Experiments: The evaluation computes MAE, RMSE, correlation, and signal-to-noise ratio for heart-rate, breathing-rate, BVP, and respiration estimates.Metrics are computed over windows spanning the number of frames in each video.
- 4 Experiments: The on-device evaluation measures latency on an ARM-based Firefly-RK3399 system using its CPU rather than its mobile GPU.The platform combines two Cortex-A72 cores and four Cortex-A53 cores, supporting evaluation across ARM-oriented deployments.
5 Results and Discussion
The proposed models improve accuracy and efficiency across within-dataset, cross-dataset, and on-device evaluations. MTTS-CAN is especially efficient while preserving strong measurement performance and enabling real-time processing.
- Comparison with the State-of-the-Art: 50% lower pulse MAE and 20% lower respiration MAE were achieved by Hybrid-CAN and 3D-CAN versus the baselines on AFRL.TS-CAN also reduced error by more than 43% for pulse and 20% for respiration compared with 2D-CAN.
- Multi-task Learning: MT models used half the computation and half as many parameters as separately running pulse and respiration models.The multi-task variants did not improve pulse or respiration error, but substantially improved inference efficiency on resource-constrained mobile platforms.
- Cross-Dataset Generalization: 25%-50% error reductions versus 2D-CAN were observed when proposed models trained on AFRL generalized to MMSE-HR videos.MTTS-CAN and MT-Hybrid-CAN also performed strongly, supporting shared representations between pulse and respiration.
- Computation Cost and Latency: 6 ms per frame made MTTS-CAN the fastest evaluated architecture, compared with 12 ms for TS-CAN.MTTS-CAN and TS-CAN were the fastest architectures evaluated in the on-device latency comparison.
- Applications of MTTS-CAN: MTTS-CAN's low latency and high accuracy could support contactless heart-rate-variability tracking and applications in mental health and personalized monitoring.The paper connects low latency with tracking subtle changes between consecutive heartbeats.
6 Broader Impact
Contactless camera-based vital-sign monitoring may broaden telehealth access, especially for people facing travel or mobility barriers. Its use also raises consent, misuse, and data-protection concerns that require transparent deployment.
- Potential Benefits: Non-contact vital-sign monitoring could improve access for rural populations and people who find in-person travel difficult.The paper identifies low-resource settings as a context where these needs may be particularly acute.
- Risks and Ethics: Physiological sensing without a subject's knowledge could enable intrusive uses by law enforcement or employers.The paper states that such applications would be illegal in many cases and recommends consent and no penalty for declining measurement.
- Privacy and Data Protection: On-device processing is motivated partly by reducing risks when videos containing health data are transferred.The paper also invokes HIPAA and the HIPAA Privacy Rule as standards for protecting sensitive patient data.
7 Conclusions
The paper presents MTTS-CAN as a multi-task temporal shift convolutional attention network for accurate and computationally efficient cardiovascular and pulmonary sensing.
- 7 Conclusions: MTTS-CAN improves state-of-the-art cardiovascular and pulmonary sensing in both accuracy and computational efficiency.The conclusion frames these as the two dimensions addressed by the proposed approach.
9 Example Waveforms
The example pulse and respiration waveforms compare three multi-task architectures with gold-standard contact measurements. MT-Hybrid-CAN and MTTS-CAN show the strongest pulse agreement, while MTTS-CAN shows the strongest respiration agreement.
- The examples compare pulse and respiration waveforms from MT-2DCAN, MT-Hybrid-CAN, and MTTS-CAN against contact-sensor measurements.
- MT-Hybrid-CAN and MTTS-CAN have the highest agreement with gold-standard pulse measurements.
- MTTS-CAN has the highest agreement with gold-standard respiration measurements among the compared models.
11 Pre-processing
The preprocessing pipeline downsamples video frames, computes normalized frame differences, and standardizes both normalized and raw inputs per video.
- Input images are downsampled to 36 × 36 before further processing.
- Normalized frames are computed between consecutive frames to reduce dependence on absolute brightness and skin appearance.The normalized frame uses (c(t + 1) − c(t))/(c(t + 1) + c(t)).
- Each video provides both a normalized frame and a raw frame, with both mean-subtracted and scaled to unit standard deviation.
12 Post-processing
Outputs are filtered and windowed consistently across methods before heart-rate, breathing-rate, and signal-quality metrics are computed against gold-standard measurements.
- A 2nd-order Butterworth filter is applied identically to all methods before performance evaluation.Cutoffs are 0.75–2.5 Hz for heart rate and 0.08–0.5 Hz for breathing rate.
- AFRL data use non-overlapping 30-second windows, while MMSE-HR uses each video's full frame sequence as one window.
- Evaluation reports MAE, RMSE, and Pearson correlation for heart-rate and breathing-rate estimates.The estimates are compared with gold-standard contact-sensor measurements.
- BVP and respiration signal quality is evaluated using signal-to-noise ratios based on the gold-standard heart-rate or respiration-rate frequency.
- The heart-rate and respiration-rate frequency metrics use binary templates centered on the gold-standard rate and its first harmonic.The templates cover ±6 BPM around the rate and ±12 BPM around its first harmonic.
13 Baseline Methods
The evaluation compares the proposed approach with supervised convolutional-attention and unsupervised signal-processing baselines, using standardized metrics on pulse and respiration datasets.
- The proposed approach is compared with supervised CAN and three unsupervised methods.
- Table 3 reports pulse and respiration measurement results on the AFRL and MMSE-HR datasets.
- The metrics comprise MAE, RMSE, Pearson correlation, and BVP signal-to-noise ratio.
- CHROM uses filtered RGB chrominance combinations, whereas ICA applies independent component analysis to normalized color signals.
- POS computes normalized RGB signals in moving windows and transforms them into signals used to construct the final BVP.Window outputs are combined with overlap-add processing.
14 Additional Experimental Results
The additional experimental results cover multi-task and non-multi-task architectures, with architecture details and pulse-respiration error measurements organized by motion task.
- Additional experimental results: Tables 3 and 4 report results for multi-task and non-multi-task architectures, including MT-3DCAN and MT-2DCAN.These architectures were omitted from the main paper because of space constraints.
- Architecture details: Figures 6–8 provide architecture details for TS-CAN, MTTS-CAN, Hybrid-CAN, MT-Hybrid-CAN, 3D-CAN, and MT-3DCAN.
- Motion-task evaluation: Table 4 reports pulse and respiration measurement MAE on the AFRL dataset by motion task.
15 Additional Figures of Results
The additional figures visualize estimated-versus-reference heart rates and error distributions across the AFRL and MMSE datasets, including participant-level AFRL errors.
- Reference comparisons: The figures compare estimated heart rates with ground-truth or gold-standard reference measurements and display their distributions across subjects.
- AFRL dataset: Figures 9 and 10 show estimated and gold-standard heart rates and corresponding Bland–Altman plots across AFRL subjects.The results use 30-second time windows.
- MMSE dataset: Figures 11 and 12 show estimated and gold-standard heart rates and corresponding Bland–Altman plots across MMSE subjects.Figure 11 notes that files with unreliable reference measurements were removed.
- Participant-level errors: Figure 13 presents participant-level error box plots for 2D-CAN and MTTS-CAN on the AFRL dataset.