Source-linked AI summary
DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks
Weixuan Chen, Daniel McDuff
TL;DR
Video physiological measurement needs accurate, non-contact recovery of heart and breathing rates despite motion and lighting variation. DeepPhys introduces an end-to-end convolutional attention network with a skin-reflection-based motion representation and appearance-guided attention. It outperformed prior state-of-the-art approaches across the evaluated RGB and infrared video datasets while enabling spatial-temporal visualization of physiological distributions.
Problem
Existing video-based physiological measurement methods were often complex, insufficiently evaluated under motion and lighting variation, or lacked an end-to-end approach for recovering vital signs.
Method
DeepPhys uses a skin-reflection-based normalized frame-difference representation and appearance-derived attention to guide motion learning in an end-to-end convolutional network.
Results
DeepPhys outperformed all compared prior state-of-the-art approaches across three RGB-video datasets and one infrared-video dataset.
Takeaways & Limitations
The model supports robust heart and breathing rate measurement under increasing head-rotation range and angular velocity while visualizing spatial-temporal physiological distributions.
Takeaways & Limitations
The linear relationship assumed by prior iPPG methods generally holds only when motion is small and the skin region is stationary under constant lighting.
Abstract
from arXiv · showhide
Non-contact video-based physiological measurement has many applications in health care and human-computer interaction. Practical applications require measurements to be accurate even in the presence of large head rotations. We propose the first end-to-end system for video-based measurement of heart and breathing rate using a deep convolutional network. The system features a new motion representation based on a skin reflection model and a new attention mechanism using appearance information to guide motion estimation, both of which enable robust measurement under heterogeneous lighting and major motions. Our approach significantly outperforms all current state-of-the-art methods on both RGB and infrared video datasets. Furthermore, it allows spatial-temporal distributions of physiological signals to be visualized via the attention mechanism.
1 Introduction
DeepPhys addresses limitations of complex, multi-stage physiological measurement methods with an end-to-end convolutional attention network for recovering heart and breathing rates from video. Its skin-reflection-based motion representation and appearance-guided attention target heterogeneous lighting, major motion, and spatial-temporal signal visualization.
- Motivation: Non-contact video measurement can unobtrusively recover and visualize vital signs using ubiquitous sensors such as webcams and smartphone cameras.Video analysis extracts physiological information from subtle color changes and body motions.
- Research gap: Earlier methods were computationally impractical or insufficiently evaluated under varied motion and lighting, while newer methods often require complex multi-stage pipelines.Common stages include face tracking, registration, skin segmentation, color transformation, decomposition, and filtering.
- Contribution: The approach significantly outperforms state-of-the-art methods and enables spatial-temporal visualization of physiological information in RGB and infrared videos.The figure description identifies heart rate and breathing rate recovery across both video modalities.
- Contribution: DeepPhys uses normalized frame differences based on a skin reflection model to capture physiological motion under heterogeneous illumination.The representation is designed to address limitations of optical flow and frame differences under changing illumination and skin contours.
- Contribution: Appearance information guides attention toward where and how physiological motion should be estimated.The network jointly uses appearance and motion information to learn physiological signal distributions.
- Contribution: DeepPhys is the first end-to-end method presented for recovering heart and breathing rates from videos.The system is described as a convolutional attention network for video-based physiological measurement.
2 Related Work
Related work spans color-based, signal-decomposition, supervised-learning, motion-analysis, and attention-based approaches to video physiological measurement. DeepPhys reverses the common attention direction by using appearance to guide motion representation learning.
- Physiological measurement: Video physiological measurement recovers signals from subtle skin-reflected light changes and small body motions, including BVP, cardiac, and respiratory signals.These approaches use color and motion information captured by cameras.
- Color-space transforms: Color-space methods combine RGB channels using assumptions about skin color or frequency-dependent blood-volume changes to improve PPG estimation.Examples include CHROM and PBV.
- Signal decomposition: Blind-source separation uses multiple color channels and spatial locations to improve robustness to illumination changes and estimate HRV and breathing rate.The cited work applies signal decomposition to webcam videos.
- Supervised learning: Supervised approaches such as template matching, support vectors, regression, and nearest neighbors remain limited by modest performance or dependence on decomposition quality.The limitation is tied specifically to methods relying on ICA or PCA.
- Motion analysis: Video learning must account for both spatial appearance and temporal dynamics, motivating implicit motion modeling alongside explicit motion analysis.Motion analysis has been studied through optical flow estimation and motion prediction.
- Attention mechanisms: Unlike prior attention methods that derive attention from motion to guide appearance learning, DeepPhys derives attention from appearance to guide motion learning.The paper presents this direction as novel to its knowledge.
3 Skin Reflection Model
DeepPhys models each skin pixel with a dichromatic reflection framework that combines illumination, specular and diffuse reflection, physiological pulsation, non-physiological motion, and sensor noise. Because physiological and non-physiological variations interact nonlinearly during substantial motion, the method uses machine learning rather than a simple linear signal model.
- Model assumptions: The model builds on Shafer’s dichromatic reflection model to represent both skin color changes and body motions in video.The framework is chosen over the Lambert-Beer law because it provides a better model for both phenomena.
- Reflection components: Each pixel’s RGB vector depends on luminance intensity, specular reflection, diffuse reflection, physiological pulsation, and camera quantization noise.Specular reflection is mirror-like, while diffuse reflection represents light absorption and scattering in skin tissues.
- Motion components: Non-physiological variation m(t) includes light-source flicker, head rotation, and facial expressions.These variations appear in the changing reflection terms alongside physiological variation.
- Time-varying model: The observed intensity variation is represented by I0 · Ψ(m(t), p(t)), where I0 is stationary luminance and Ψ captures variation involving motion and physiology.The interaction between physiological and non-physiological motions is described as generally complex and nonlinear.
- Model derivation: The expanded pixel model combines stationary skin reflection, varying specular reflection, pulsatile BVP, and sensor noise after substituting the reflection components.The paper then approximates the expression by neglecting products between varying terms because they are much smaller than stationary components.
- Model motivation: Recovering p(t) from C_k(t) becomes difficult when m(t) is large, because the linear relationship assumed by prior iPPG work can harm measurement performance.The paper therefore motivates a machine-learning model capable of capturing the more general relationship.
4 Approach
DeepPhys combines a skin-reflection-based normalized motion representation with an appearance-guided attention mechanism in an end-to-end CNN for recovering physiological signals.
- 4.1 Motion Representation: The normalized frame difference reduces dependence on stationary skin color and heterogeneous illumination while retaining spatial motion information.Frames are downsampled, normalized by temporal means, and clipped to reduce outliers before entering the learning model.
- 4.3 Attention Mechanism: The appearance and motion models are learned jointly to estimate physiological motion and detect appropriate regions of interest simultaneously.The shared spatial masks guide motion-feature processing through the network architecture.
- 4.2 Convolutional Neural Network: A VGG-style CNN estimates the physiological-signal derivative, then band-pass filtering and dominant-frequency analysis produce heart-rate or breathing-rate estimates.The model uses MSE against the gold-standard derivative as its training objective.
- 4.3 Attention Mechanism: The attention mechanism uses appearance features to infer soft spatial masks that suppress irrelevant motion and emphasize regions with stronger physiological signals.Appearance information is retained in a separate model because motion normalization removes it.
- 4.3 Attention Mechanism: Sigmoid activations followed by l1 normalization produce less extreme attention masks than softmax, which are multiplied with motion feature maps.The masked feature maps are passed to subsequent layers.
5 Datasets
The evaluation uses four video datasets spanning RGB and infrared imaging, varied participants, lighting, acquisition setups, and head-motion conditions.
- RGB Video I: RGB Video I includes 25 participants recorded at 120 fps with gold-standard physiological measurements.Tasks range from chin-rest recordings and small natural motions to 120-degree head sweeps at 10, 20, and 30 degrees/sec and random reorientation.
- Datasets: Four datasets are evaluated: RGB Video I, RGB Video II, MAHNOB-HCI, and Infrared Video.Figure 3 identifies representative frames and cropped input regions for all four datasets.
- RGB Video II: RGB Video II contains 18 participants recorded with an Intel RealSense camera at approximately 24 fps under ambient light.Three participants were eliminated because of high collection error.
- MAHNOB-HCI: MAHNOB-HCI contributes 527 thirty-second clips from 27 subjects recorded at 61 fps.Heart-rate ground truth was computed from the EXG2 ECG channel.
6 Results and Discussion
The experiments compare DeepPhys and related CNN variants across participant-dependent, participant-independent, task-independent, transfer-learning, and infrared settings. Results generally favor attention-guided learning, especially under motion and across datasets.
- Implementation: Adadelta optimization, dropout, and dataset-specific hyperparameters were used to train the models.Three dropout layers were inserted to reduce overfitting, while hidden units and training epochs were varied for model complexity and generalization.
- Participant-dependent Performance: CAN, motion-only CNN, and stacked CNN outperform prior methods for heart-rate measurement across tasks two through six.The advantage is strongest for tasks involving high-velocity head motions; task one leaves little room for improvement.
- Participant-dependent Performance: CAN shows the strongest average performance among the proposed approaches, particularly on task six where changing regions of interest favor attention.Breathing-rate results follow a similar pattern.
- Participant-independent Performance: Participant-independent results decline relative to participant-dependent evaluation, but stacked CNN and CAN still outperform prior heart-rate methods.For breathing rate, CAN improves on the comparator in five tasks and overall, whereas the other CNN variants are similar or inferior.
- Task-independent Performance: Models trained on higher-motion tasks generalize better across tasks for heart rate, while breathing-rate models usually perform best on their training task.The CAN model trained on task six achieves lower MAE and higher SNR than models trained and tested within individual tasks.
- Transfer learning: Without fine-tuning, CAN attains the lowest MAE against prior methods on RGB Video II and MAHNOB-HCI.Its SNR ranks second to Wang et al. on RGB Video II and Tulyakov et al. on MAHNOB-HCI.
- Infrared Video: In infrared video, CAN beats the previous best results and other learning-based methods without attention for both heart-rate and breathing-rate measurement.Evaluation used participant-independent five-fold cross-validation.
- Visualization of Attention Weights: Attention maps focus heart-rate estimation on the forehead, earlobe, and carotid arteries, while breathing-rate attention is more scattered.Frequent nasal attention suggests that subtle nasal flaring contributes to respiratory tracking.
7 Conclusions
The paper presents the first end-to-end network for non-contact heart- and breathing-rate measurement. Across RGB and infrared datasets, it outperforms compared state-of-the-art methods, especially with increasing head-rotation motion, and visualizes physiological signal distributions.
- Conclusions: DeepPhys is the first end-to-end network proposed for non-contact measurement of heart rate and breathing rate.Its convolutional attention network learns color and motion information while visualizing spatial-temporal physiological distributions.
- Conclusions: DeepPhys outperformed all compared state-of-the-art approaches across three RGB datasets and one infrared dataset.The strongest improvements occurred in tasks with increasing head-rotation range and angular velocity.
- Conclusions: Participant-dependent, participant-independent, and transfer-learning results indicate generalization across people, skin types, and illumination conditions.The authors attribute the performance improvement to the model's learned mapping between video color and motion information.