Source-linked AI summary
Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling
Xuesong Niu, Zitong Yu, Hu Han, Xiaobai Li, Shiguang Shan, Guoying Zhao
TL;DR
Contact-free rPPG measurement is useful when contact sensors are inconvenient, but weak physiological signals are easily contaminated by motion, illumination, and device variation. The paper combines MSTmaps with cross-verified disentangling to isolate physiological features for joint HR and rPPG prediction, reporting state-of-the-art performance across multiple tasks and databases.
Problem
Contact-free rPPG measurement must handle weak physiological signals that are affected by head movements, lighting conditions, device variation, and non-physiological information in face-video representations.
Method
The method converts face videos into MSTmaps and uses pairwise inputs with separate physiological and non-physiological encoders trained by cross-verified disentangling.
Results
The method achieves state-of-the-art performance on multiple physiological measurement tasks and databases, including cross-database testing.
Takeaways & Limitations
The learned physiological features support joint average HR estimation and rPPG signal regression for RF measurement and HRV analysis.
Abstract
from arXiv · showhide
Remote physiological measurements, e.g., remote photoplethysmography (rPPG) based heart rate (HR), heart rate variability (HRV) and respiration frequency (RF) measuring, are playing more and more important roles under the application scenarios where contact measurement is inconvenient or impossible. Since the amplitude of the physiological signals is very small, they can be easily affected by head movements, lighting conditions, and sensor diversities. To address these challenges, we propose a cross-verified feature disentangling strategy to disentangle the physiological features with non-physiological representations, and then use the distilled physiological features for robust multi-task physiological measurements. We first transform the input face videos into a multi-scale spatial-temporal map (MSTmap), which can suppress the irrelevant background and noise features while retaining most of the temporal characteristics of the periodic physiological signals. Then we take pairwise MSTmaps as inputs to an autoencoder architecture with two encoders (one for physiological signals and the other for non-physiological information) and use a cross-verified scheme to obtain physiological features disentangled with the non-physiological features. The disentangled features are finally used for the joint prediction of multiple physiological signals like average HR values and rPPG signals. Comprehensive experiments on different large-scale public datasets of multiple physiological measurement tasks as well as the cross-database testing demonstrate the robustness of our approach.
1 Introduction
Remote physiological measurement with rPPG avoids the discomfort and inconvenience of contact sensors but remains vulnerable to weak-signal contamination from motion, illumination, and device variation. The proposed method addresses this by combining MSTmaps with cross-verified feature disentangling for multi-task measurement.
- rPPG enables contact-free measurement of HR, RF, and HRV from face videos captured by ordinary cameras.This avoids the discomfort and inconvenience associated with dedicated skin-contact devices.
- The small amplitude of blood-volume-related optical changes makes rPPG signals susceptible to subject movement and illumination conditions.
- Existing learned representations retain physiological and non-physiological information, allowing head movements, lighting variations, and device noise to affect learned features.
- The method compresses face videos into multi-scale spatial-temporal maps and trains an end-to-end multi-task network with cross-verified disentangling.MSTmaps better represent physiological information, while the disentangling strategy separates it from head movements, lighting conditions, and device variations.
- An autoencoder uses separate physiological and non-physiological encoders with pairwise MSTmaps to cross-verify the features each encoder should represent.Pseudo MSTmaps are generated by combining features from different original MSTmaps, guiding the encoders toward the intended separation.
- The approach reports state-of-the-art performance across multiple physiological measurement databases and cross-database testing.
2 Related Work
Prior remote physiological measurement methods use hand-crafted signal processing, learned face-video representations, or specialized network structures. The paper situates its approach within disentangled representation learning while avoiding adversarial training and prior attribute labels.
- Traditional rPPG methods extract signals from color channels or facial regions and use ICA, PCA, or matrix completion to improve signal-to-noise ratio.
- Deep-learning approaches estimate physiological signals from motion representations, spatial-temporal maps, cropped faces, or face videos using CNN, RNN, attention, 2D convolution, or 3D convolution.
- Existing methods primarily emphasize designing face-video representations and network structures, while their representations can also contain non-physiological information.
- Related disentanglement methods separate identity, pose, appearance, blur, content, or domain attributes for recognition, synthesis, editing, and deblurring.
- The proposed method differs by avoiding adversarial training and requiring no prior attribute labels for disentanglement.
3 Proposed Method
The proposed method represents face videos as MSTmaps, disentangles physiological from non-physiological features through cross-verified encoding, and jointly predicts HR and rPPG signals. Its design combines multi-scale facial information, pairwise feature recombination, and multi-task supervision.
- 3.1 Multi-scale Spatial Temporal Map: The method generates MSTmaps from local and global facial regions to represent heartbeat-related skin-color variations across spatial scales and time.It uses informative facial ROIs, their non-empty combinations, and RGB and YUV channels.
- 3.1 Multi-scale Spatial Temporal Map: Each MSTmap contains temporal sequences formed from average pixel values across 2^n −1 ROI combinations and six RGB/YUV color channels.For T-frame clips, the resulting map has size (2^n −1) × T × 6 after max-min normalization to [0,255].
- 3.2 Cross-verified Feature Disentangling: Pairwise MSTmaps are encoded by separate physiological and non-physiological encoders, whose features reconstruct the original maps and cross-generate pseudo MSTmaps.Pseudo maps combine physiological features from one input with non-physiological features from the other.
- 3.2 Cross-verified Feature Disentangling: Cross-verifying features from original and pseudo MSTmaps encourages each encoder to retain the information it is intended to represent.The CVD loss enforces disentanglement and produces more representative physiological features for measurement.
- 3.3 Multi-task Physiological Measurement: The physiological estimator jointly predicts average HR and rPPG signals using a two-head network with complementary supervision.Average HR uses an L1 loss, while rPPG uses Pearson-correlation-based supervision and an additional HR-related loss derived from the rPPG branch.
4 Experiments
Experiments evaluate the method across three public databases, multiple physiological tasks, intra- and cross-database settings, and ablations. The approach outperforms prior methods and benefits from MSTmaps, cross-verified disentangling, and joint HR/rPPG learning.
- Experimental Settings: The method is evaluated on VIPL-HR, OBF, and MMSE-HR using average HR, RF, HRV, cross-database, and ablation experiments.VIPL-HR and OBF support intra-database evaluations, while MMSE-HR is used for cross-database testing.
- Intra-database Testing: 7.92 bpm Std, 5.02 bpm MAE, 7.97 bpm RMSE, and 0.79 r were achieved for VIPL-HR average HR estimation, outperforming compared methods.For large head movements, RMSE was 7.44 bpm versus 9.4 bpm for RhythmNet.
- Intra-database Testing: The method followed rapidly decreasing HR trends after exercise and achieved the best OBF average HR performance among the compared methods.The short-time examples indicate robustness under significant HR changes.
- Intra-database Testing: The approach outperformed existing methods by a large margin on all RF and HRV evaluation metrics and accurately predicted interbeat intervals.Evaluated HRV features included LF, HF, and LF/HF on OBF.
- Cross-database Testing: Cross-database testing on MMSE-HR produced the best results on all evaluation metrics compared with state-of-the-art methods.The authors report good generalization to unknown scenarios without prior knowledge.
- Ablation Study: Joint rPPG and HR training reduced MAE from 6.39 to 5.55 bpm and RMSE from 10.24 to 9.03 bpm, while CVD further reduced RMSE to 7.97 bpm and MAE to 5.02 bpm.MSTmap also outperformed STmap in key comparisons, including MAE and Pearson correlation.
5 Conclusions
The paper presents an end-to-end multi-task network that uses cross-verified disentangling to reduce non-physiological influences in remote physiological measurement. It compresses face videos into MSTmaps and uses the learned physiological features for HR estimation and rPPG regression.
- The network applies cross-verified disentangling to reduce influences from head movements and lighting conditions.
- Face videos are compressed into multi-scale spatial-temporal maps to better represent physiological information.
- Learned physiological features support both average HR estimation and rPPG signal regression.
- The proposed method achieves state-of-the-art performance across multiple physiological measurement tasks and databases.
Supplementary Material
Figure 4 details the physiological and non-physiological encoders, decoder, and physiological estimator. It also defines the principal convolutional, pooling, and feature-map scaling components used in these architectures.
- The figure presents the physiological and non-physiological encoders Ep and En.
- The figure presents the decoder D and the physiological estimator.
- Convolution and transposed convolution layers are used in the detailed architectures.