Source-linked AI summary
Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks
Zitong Yu, Xiaobai Li, Guoying Zhao
TL;DR
Average HR from facial rPPG is insufficient for applications requiring precise beat-to-beat information such as HRV analysis. The paper introduces PhysNet, an end-to-end spatiotemporal network that reconstructs rPPG signals from raw facial videos. Across two benchmark datasets, it reports superior HR and HRV performance and promising AF-detection and emotion-recognition results.
Problem
Average HR from facial videos does not provide the precise pulse-peak and IBI information needed for HRV analysis and applications such as AF detection.
Method
The paper proposes PhysNet, an end-to-end spatiotemporal network that reconstructs rPPG signals from raw facial videos while modeling temporal context.
Results
PhysNet achieves superior performance on both HR and HRV levels compared with state-of-the-art methods, with reconstructed signals also producing promising AF-detection and emotion-recognition results.
Takeaways & Limitations
Accurately recovered pulse peaks provide average HR, IBI, and HRV information for potential remote AF detection and emotion recognition.
Takeaways & Limitations
The network is trained with PPG ground truth but evaluated against ECG ground truth because PPG more closely resembles facial rPPG.
Abstract
from arXiv · showhide
Recent studies demonstrated that the average heart rate (HR) can be measured from facial videos based on non-contact remote photoplethysmography (rPPG). However for many medical applications (e.g., atrial fibrillation (AF) detection) knowing only the average HR is not sufficient, and measuring precise rPPG signals from face for heart rate variability (HRV) analysis is needed. Here we propose an rPPG measurement method, which is the first work to use deep spatio-temporal networks for reconstructing precise rPPG signals from raw facial videos. With the constraint of trend-consistency with ground truth pulse curves, our method is able to recover rPPG signals with accurate pulse peaks. Comprehensive experiments are conducted on two benchmark datasets, and results demonstrate that our method can achieve superior performance on both HR and HRV levels comparing to the state-of-the-art methods. We also achieve promising results of using reconstructed rPPG signals for AF detection and emotion recognition.
1 Introduction
The introduction argues that average HR alone is insufficient for medical applications requiring beat-to-beat information, motivating precise rPPG recovery from facial videos. The paper proposes an end-to-end spatiotemporal network that reconstructs pulse peaks and supports HRV-related applications.
- Average HR alone is insufficient for applications such as AF detection because detailed IBI and HRV information is also needed.Traditional ECG and PPG provide both average HR and inter-beat intervals, whereas many earlier rPPG methods focused on average HR.
- Earlier rPPG methods selected facial regions empirically and used handcrafted features or filters that may not generalize across data.
- Deep learning approaches commonly treated HR estimation as one-stage regression, omitted temporal context, or retained handcrafted preprocessing and post-processing.
- The proposed method uses spatiotemporal modeling on facial videos to locate individual heartbeat peaks accurately.
- The first end-to-end spatiotemporal network, PhysNet, measures rPPG signals from raw facial videos and achieves superior HR and HRV performance compared with state-of-the-art methods.The reconstructed signals were further demonstrated for AF detection and emotion recognition, with good generalization shown in a cross-data test.
2 Related Work
Related work progressed from handcrafted rPPG methods and average-HR estimation toward deep learning, but precise pulse-peak and HRV measurement remained challenging. The paper motivates spatiotemporal modeling while noting that its suitability for subtle facial color changes was previously unknown.
- Traditional video-based rPPG methods analyze facial-region or skin-pixel color changes but require complex prior knowledge for region or skin-pixel selection.
- These traditional methods can be difficult to deploy, may generalize poorly to new data, and often target average HR rather than individual pulse-peak accuracy.
- Recent deep learning methods estimate average HR or pulse signals but are not consistently end-to-end and may depend on handcrafted features.
- HRV features are computed from inter-beat intervals and provide finer-scale cardiac information than the total heartbeat count represented by HR.Common HRV features include LF, HF, and LF/HF; accurate pulse-peak timing is required to measure them.
- Spatiotemporal networks capture spatial and temporal context, but their suitability for rPPG remains uncertain because facial skin-color variation is extremely subtle.The paper evaluates spatiotemporal models and loss functions as a foundation for rPPG-specific network optimization.
3 Methodology
PhysNet reconstructs rPPG signals from raw facial videos using end-to-end spatio-temporal modeling, alternative temporal architectures, and a trend-consistency loss designed to preserve heartbeat peaks for HRV analysis.
- Network Architecture: PhysNet merges color-subspace projection and irrelevant-signal removal into one end-to-end operation for recovering rPPG signals.The method targets noise from illumination or motion while avoiding separate handcrafted preprocessing and postprocessing steps.
- Network Architecture: The network accepts T-frame RGB face images and outputs an rPPG signal of length T after spatio-temporal feature extraction and channel-wise projection.Latent manifolds are projected into signal space using a 1 × 1 × 1 channel-wise convolution.
- Network Architecture: PhysNet compares 3DCNN and RNN-based spatio-temporal models for extracting temporal context from facial videos.The 3DCNN variant uses 3 × 3 × 3 convolutions, while RNN variants propagate spatial features temporally with LSTM, bi-directional LSTM, or ConvLSTM modules.
- Network Architecture: The RNN alternatives use 2DCNNs for spatial features followed by recurrent temporal propagation, with ConvLSTM applying global average pooling afterward.The recurrent models use forward/backward information flows to improve temporal context features.
- Loss Function: Negative Pearson correlation is used to maximize trend similarity and minimize pulse-peak location errors between predicted rPPG and ground-truth signals.The loss compares predicted signals x with ground-truth PPG signals y over signal length T.
- Ground Truth: PPG is used for training because finger PPG resembles facial rPPG as a measure of peripheral blood-volume changes, whereas ECG measures electrical activity.ECG is used during testing for comparison with previous work.
4 Experiments
Experiments on OBF and MAHNOB-HCI evaluate loss functions, spatio-temporal architectures, clip lengths, cross-method performance, applications, visualization, and inference speed. PhysNet achieves strong HR and HRV results, supports AF detection and emotion recognition, generalizes across datasets, and enables fast end-to-end recovery.
- Experimental setup: The study trains PhysNet on OBF, evaluates HR and HRV measurement, tests AF detection, cross-tests on MAHNOB-HCI, and explores emotion recognition.OBF provides facial videos with synchronized physiological signals; MAHNOB-HCI is used for cross-dataset testing.
- Spatio-temporal networks: Temporal convolutions improve HR and HRV performance over 2DCNN, while the temporal encoder-decoder further improves 3DCNN results.The experiments compare 2DCNN, 3DCNN with and without encoder-decoder structure, and LSTM-based models.
- Clip length: For PhysNet-3DCNN, the best HR and HRV performance occur at T = 128 and T = 64, respectively.Clip-length behavior varies by architecture: 2DCNN benefits from longer inputs, whereas 3DCNN performance is more stable.
- Comparison with previous methods: PhysNet128-3DCNN-ED outperforms ROI_green, CHROM, and POS on both HR and HRV levels.The model is the best-performing configuration in the comparison with previous methods.
- Applications and deployment: PhysNet-derived HRV features support AF detection and promising emotion recognition, while cross-testing on MAHNOB-HCI achieves top-level average-HR performance.The method also recovers signals without preprocessing and processes a 30-second test video in 0.235s, or 3830 fps, on a Tesla P100 GPU.
Conclusion
The proposed end-to-end spatio-temporal framework recovers rPPG signals from raw facial videos with accurate pulse-peak timing. This supports average HR, IBI, and HRV measurements for potential remote AF detection and emotion recognition.
- PhysNet recovers rPPG signals from raw facial videos with accurate time locations for individual pulse peaks.
- Accurate pulse peaks enable measurement of average HR, IBI information, and HRV-level features.
- The recovered signals support potential applications including remote AF detection and emotion recognition.
6 Aknowledgement
The work acknowledges financial, institutional, and computational support from multiple organizations.
- The study was supported by the National Natural Science Foundation of China, Finnish funding programs, the Academy of Finland, and Infotech Oulu.
- CSC-IT Center for Science, Finland, provided computational resources.