Source-linked AI summary
EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement
Xin Liu, Brian L. Hill, Ziheng Jiang, Shwetak Patel, Daniel McDuff
TL;DR
Existing camera-based physiological measurement methods often require preprocessing that increases implementation and computational burdens. EfficientPhys instead uses raw video frames with convolutional or transformer-based architectures, achieving strong benchmark performance and improved efficiency while targeting simple on-device deployment. Its broader applications remain subject to clinical validation and regulatory approval.
Problem
Prior high-performing camera-based physiological measurement methods require preprocessing that can be computationally costly and difficult to implement or replicate.
Method
EfficientPhys uses raw video frames as input to preprocessing-free convolutional and visual-transformer neural architectures for camera-based vitals measurement.
Results
EfficientPhys models achieve strong performance across three public datasets; Efficient-C improves UBFC MAE by 45% over Efficient-T1 and is more than 7x faster.
Takeaways & Limitations
The preprocessing-free design reduces engineering and deployment burden while supporting efficient on-device camera-based vitals measurement.
Takeaways & Limitations
Broader healthcare applications require appropriate clinical validation and regulatory approval, and extensions to other video tasks are presented as potential applications.
Abstract
from arXiv · showhide
Camera-based physiological measurement is a growing field with neural models providing state-the-art-performance. Prior research have explored various "end-to-end" models; however these methods still require several preprocessing steps. These additional operations are often non-trivial to implement making replication and deployment difficult and can even have a higher computational budget than the "core" network itself. In this paper, we propose two novel and efficient neural models for camera-based physiological measurement called EfficientPhys that remove the need for face detection, segmentation, normalization, color space transformation or any other preprocessing steps. Using an input of raw video frames, our models achieve strong performance on three public datasets. We show that this is the case whether using a transformer or convolutional backbone. We further evaluate the latency of the proposed networks and show that our most light weight network also achieves a 33% improvement in efficiency.
1. Introduction
Camera-based physiological measurement uses reflected light to capture cardiac signals, but existing high-performing neural methods still depend on costly and difficult preprocessing. EfficientPhys addresses this with preprocessing-free architectures that take unprocessed video frames as input.
- Motivation: Camera-based physiological measurement captures cardiac signals from light reflected by the body, including BVP measured through PPG.Heart rate, respiration rate, and pulse transit times can be derived from PPG signals.
- Limitations of prior methods: State-of-the-art neural approaches require hand-crafted representations, facial landmarks, regions of interest, and color-space operations before inference.Examples include normalized difference and appearance frames, as well as MSTmaps built from facial landmarks and RGB/YUV values.
- Limitations of prior methods: Preprocessing can impose computational costs, restrict data-driven feature learning, and make real-time deployment and cross-platform replication difficult.These costs are especially problematic for privacy-sensitive, on-device sensing intended for low-end mobile devices.
- EfficientPhys: EfficientPhys takes unprocessed video frames without requiring accurate face cropping and is designed for high-frame-rate, cross-platform mobile use.The paper proposes both convolutional and visual-transformer architectures.
- Contributions: The paper contributes two preprocessing-free neural architectures, evaluations on three benchmark datasets, and on-device latency comparisons with neural and signal-processing methods.It presents the visual transformer as a first exploration of this architecture for camera-based physiological measurement.
2. Related Work
Prior camera-vitals methods range from signal-processing algorithms to neural architectures, but many neural approaches rely on increasingly complex preprocessing. EfficientPhys is positioned as a simpler preprocessing-free alternative, while also extending evaluation to visual transformers.
- Camera-based Vital Measurement: Traditional camera-vitals methods use signal separation or physically based projections to extract pulse signals from RGB images.Examples include ICA and Plane Orthogonal-to-Skin (POS).
- Camera-based Vital Measurement: Deep convolutional networks surpassed traditional signal-processing approaches, while MTTS-CAN introduced efficient tensor-shift and 2D-convolutional spatial-temporal modeling.DeepPhys established the neural-network performance advantage described in this related-work discussion.
- Preprocessing burden: DeepPhys and MTTS-CAN require difference-frame calculation and image normalization before neural inference.These steps are simpler than the MSTMaps pipeline described for Dual-GAN and related methods.
- Preprocessing burden: Dual-GAN’s MSTMaps pipeline combines face cropping, landmark extraction, ROI combinations, pooling, color channels, and temporal maps, increasing memory and computational demands.The resulting MSTMap has size (2n −1) × T × 6, where T is the frame count and n is the number of ROI regions.
- Visual Transformers: Vision transformers provide a second architectural direction for EfficientPhys beyond the field’s established convolutional approaches.The paper frames transformer-based modeling as a newer computer-vision development alongside convolutional networks.
3. Method
EfficientPhys replaces hand-crafted preprocessing with raw-frame neural architectures for camera-based physiological measurement. It provides convolutional and transformer designs that model spatial-temporal information while remaining simple to deploy.
- Convolution-based EfficientPhys: The convolutional EfficientPhys combines custom normalization, self-attention, tensor shifting, and 2D convolution in a one-branch network.Its single branch learns attention masks end-to-end instead of relying on a separate appearance branch.
- Motivation: Existing methods use costly preprocessing such as facial landmark detection, ROI extraction, color-space averaging, and STMap construction.Dual-GAN’s preprocessing module takes 275 ms per frame, adding deployment burden on mobile devices.
- Normalization Module: The normalization module computes consecutive-frame differences followed by batch normalization to model motion and reduce lighting and motion noise.Batch normalization addresses subtle pixel changes and scale variation that make raw difference frames difficult to learn from.
- Architectural comparison: Figure 2 presents convolution-based and transformer-based EfficientPhys architectures, with N denoting the number of input video frames.The transformer design combines tensor-shift and window-partition shift modules for spatial-temporal modeling.
- Transformer-based EfficientPhys: The transformer-based EfficientPhys adds tensor shifts before Swin transformer blocks to exchange information across time while retaining shifted-window spatial attention.The tensor-shift module advances one input chunk and delays another along the temporal axis.
4. Experiments
The experiments evaluate EfficientPhys on public benchmark datasets and measure implementation and inference behavior, including latency on a Raspberry Pi 4B edge device.
- Datasets: Training uses AFRL and a synthetic avatar video dataset to support robust and generalizable cross-dataset evaluation.AFRL contains 300 videos from 25 subjects and includes stationary tasks and progressively faster head motions.
- Datasets: EfficientPhys is evaluated on three benchmark datasets, including UBFC with 42 videos from 42 subjects recorded at 30 Hz.UBFC videos use uncompressed 8-bit RGB frames at 640x480 resolution, and all tasks are stationary.
- On-device evaluation: On-device efficiency is evaluated on a quad-core Cortex-A72 Raspberry Pi 4B using averaged inference latency over ten runs.The comparison includes EfficientPhys and TS-CAN, while STMaps preprocessing is evaluated separately because Dual-GAN lacks an open-source implementation.
5. Results and Discussion
EfficientPhys models achieve strong cross-dataset heart-rate performance while simplifying preprocessing and deployment. The convolutional model offers a favorable accuracy-latency trade-off, whereas reducing transformer depth improves speed but substantially harms accuracy.
- Cross-dataset evaluation: EfficientPhys-C and EfficientPhys-T1 outperform existing methods on UBFC and PURE, while neural methods outperform signal-processing methods on MMSE.Models were trained on AFRL + Synthetic and tested across UBFC, PURE, and MMSE for cross-dataset generalization.
- On-device latency: 40ms per frame: EfficientPhys-C requires no extra preprocessing computation, compared with 63ms for TS-CAN and 275ms for Dual-GAN preprocessing.The reported Dual-GAN figure covers preprocessing only; its estimation network would add further computational time.
- Convolution vs. Transformer: EfficientPhys-T2 matches EfficientPhys-C's inference time after reducing transformer depth, but has the poorest performance on all three benchmark datasets.The lightweight transformer’s errors increased by 48% of MAE on UBFC, 141% on PURE, and 15% on MMSE.
- Convolution vs. Transformer: EfficientPhys-C outperforms EfficientPhys-T1 by 45% of MAE in UBFC and is more than 7x faster in latency.EfficientPhys-C shows similar performance to EfficientPhys-T1 on MMSE and PURE.
- Ablation study: Removing self-attention increases EfficientPhys-C MAE by 14%, while removing the normalization module increases MAE by 753% for EfficientPhys-C and 420% for EfficientPhys-T1.Removing the transformer tensor shift module increases error by 300%, indicating its role in exchanging temporal information and dynamics.
- Deployment implications: The one-stop design reduces cross-platform engineering burden and can support replication and reproduction of results.The raw-frame design may also extend to other video-based applications, including blood-pressure measurement and video understanding.
6. Broader Impacts and Ethics Statement
The authors frame EfficientPhys as a potential tool for widening access to camera-based vitals measurement, especially in low-resource settings, while acknowledging risks of bias and unequal impact.
- EfficientPhys could eventually support real-time vitals measurement in telehealth, but only with appropriate clinical validation and regulatory approval.
- The authors identify low-resource settings as a context where EfficientPhys could have especially strong impact because healthcare access barriers are greater.
- The paper acknowledges that machine-learning systems can be biased and may propagate inequalities across populations.
7. Conclusion
EfficientPhys is presented as a simple, fast, accurate approach for contactless camera-based vitals measurement. Its one-stop design achieves strong performance with substantially lower computational power and addresses deployment challenges.
- EfficientPhys enables simple, fast, accurate camera-based contactless vitals measurement.
- The method achieves strong performance while using significantly less computational power.
- Its simple one-stop design addresses last-mile machine-learning deployment and may help reduce health disparities.