Source-linked AI summary
Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-to-end Deep Learning Solution with Video Enhancement
Zitong Yu, Wei Peng, Xiaobai Li, Xiaopeng Hong, Guoying Zhao
TL;DR
Contactless rPPG depends on subtle facial-video details that compression can destroy. The paper introduces an end-to-end enhancement-and-recovery pipeline, which achieves robust measurement on compressed videos and generalizes to unseen compressed data.
Problem
Video compression removes subtle facial information needed for rPPG, while prior approaches generally did not address robust measurement from highly compressed videos.
Method
The method cascades STVEN for spatio-temporal video enhancement with rPPGNet for rPPG signal recovery, using joint loss training to connect both tasks.
Results
The proposed method achieves superior performance on compressed videos and generalizes well to novel data containing only compressed videos.
Takeaways & Limitations
The framework supports robust rPPG measurement from highly compressed facial videos, with potential for real-world remote applications.
Abstract
from arXiv · showhide
Remote photoplethysmography (rPPG), which aims at measuring heart activities without any contact, has great potential in many applications (e.g., remote healthcare). Existing rPPG approaches rely on analyzing very fine details of facial videos, which are prone to be affected by video compression. Here we propose a two-stage, end-to-end method using hidden rPPG information enhancement and attention networks, which is the first attempt to counter video compression loss and recover rPPG signals from highly compressed videos. The method includes two parts: 1) a Spatio-Temporal Video Enhancement Network (STVEN) for video enhancement, and 2) an rPPG network (rPPGNet) for rPPG signal recovery. The rPPGNet can work on its own for robust rPPG measurement, and the STVEN network can be added and jointly trained to further boost the performance especially on highly compressed videos. Comprehensive experiments are performed on two benchmark datasets to show that, 1) the proposed method not only achieves superior performance on compressed videos with high-quality videos pair, 2) it also generalizes well on novel data with only compressed videos available, which implies the promising potential for real world applications.
1. Introduction
rPPG offers contactless heart-activity measurement, but compression removes subtle facial information and degrades signal-based heart-rate estimation. The paper proposes an end-to-end framework combining video enhancement with rPPG recovery for highly compressed facial videos.
- ECG and PPG measure average heart rate and detailed heart-rate variability, but contact sensors can be uncomfortable and inconvenient for long-term monitoring.
- Remote photoplethysmography measures heart activity without physical contact, addressing limitations of skin-contact monitoring.
- Most prior rPPG work did not account for compression, although commercial-camera videos are commonly compressed at different bitrates and codecs.
- Compression artifacts remove subtle facial changes, producing noisy rPPG curves and inaccurate peaks that can yield erroneous heart-rate measurements.
- The proposed two-stage, end-to-end method uses hidden rPPG information enhancement and attention networks to recover signals from highly compressed facial videos.
- The framework combines STVEN video enhancement with rPPGNet signal recovery, and the authors report superior compressed-video performance and cross-dataset generalization.
2. Related Work
Earlier rPPG methods used handcrafted processing or spatial networks with limited temporal modeling and unequal task integration. Compression was known to reduce accuracy, but existing enhancement methods were not designed specifically for rPPG, motivating STVEN and rPPGNet.
- Remote Photoplethysmography Measurement: Traditional methods selected facial regions manually or arbitrarily, potentially losing information, while all-skin-pixel methods treated skin locations as equally informative.
- Remote Photoplethysmography Measurement: Deep learning methods for average heart-rate estimation often used spatial 2D CNNs that failed to capture temporal features essential for rPPG measurement.
- Remote Photoplethysmography Measurement: Separating skin segmentation from rPPG recovery limited mutual feature sharing between the related tasks.
- Video Compression and Its Impact: Compression artifacts from quantization remove subtle changes relevant to rPPG, and prior studies consistently found reduced heart-rate-estimation accuracy.
- Video Compression and Its Impact: Existing compression-enhancement networks addressed general compression or other tasks, rather than rPPG measurement specifically.
- The paper fills this gap with a two-stage, end-to-end deep-learning method for rPPG measurement from highly compressed videos.
3. Methodology
The method combines STVEN video enhancement with rPPGNet signal recovery in a two-stage end-to-end framework. STVEN learns bitrate-aware video translation, while rPPGNet uses spatiotemporal features, skin-based attention, and partition constraints to recover rPPG signals.
- STVEN: STVEN enhances compressed video by learning mappings across compression-bitrate distributions and can also generate videos matching specified lower-quality distributions.Videos are grouped into bitrate buckets, with 0 representing the lowest compression rate and C the highest.
- STVEN: STVEN uses translation and compression reconstruction losses, combining MSE for lost video details with L1 loss for compression reconstruction.Its total loss is the sum of reconstruction and cycle losses.
- STVEN: The STVEN architecture uses two downsampling layers, two upsampling layers, and six spatio-temporal blocks within a spatial-temporal convolutional network.The architecture also uses 3D convolutions, transposed convolutions, instance normalization, batch normalization, ReLU, and spatial global average pooling.
- rPPGNet: rPPGNet combines skin segmentation, parameter-free skin-based attention, and partition constraints to emphasize relevant skin regions and learn concentrated rPPG features.Attention weights use skin confidence and rPPG feature maps, while partition losses independently supervise uniform spatiotemporal parts.
- rPPGNet: rPPGNet directly maps T-frame RGB face images to rPPG signals through an end-to-end spatiotemporal convolutional network.Its training uses negative Pearson correlation because accurate pulse peak locations matter more than pointwise signal intensity.
- Joint training: Joint training updates STVEN while keeping high-level task-model parameters fixed, using rPPG and feature-space consistency objectives to enhance videos for rPPG recovery.High-quality-video rPPG signals serve as softer targets for STVEN, and the joint loss combines rPPG, feature consistency, and STVEN losses.
4. Experiments
Experiments evaluate rPPGNet and the STVEN–rPPGNet system on OBF and MAHNOB-HCI, including highly compressed videos and cross-dataset generalization. Results show that compression degrades measurement, while STVEN and joint training improve robustness, including when high-quality target videos are unavailable.
- Experimental design: The experiments cover rPPGNet accuracy, compression effects, STVEN enhancement, and MAHNOB-HCI generalization using OBF and MAHNOB-HCI datasets.OBF provides high-quality videos and ECG, while MAHNOB-HCI contains compressed facial videos and physiological signals.
- Results on high-quality OBF: rPPGNet outperforms other methods on high-quality OBF videos for both averaged HR and HRV features.Ablations attribute gains to skin segmentation, partition constraints, and skin-based attention.
- Highly compressed OBF videos: Performance decreases as bitrate decreases across codecs, but rPPGNet outperforms other methods at the same compression condition in most cases, especially at 250 kb/s.The authors note that low-bitrate accuracy remains unsatisfactory.
- STVEN enhancement: STVEN improves video quality, achieving larger △PSNR than ARCNN and DnCNN.Enhanced videos have △PSNR values larger than zero, indicating quality improvement.
- STVEN–rPPGNet results: On x264-compressed videos, STVEN boosts rPPGNet whereas DnCNN does not, and joint training performs better than separate training.The authors associate this difference with STVEN’s spatiotemporal structure and DnCNN’s single-frame design.
- Generalization to MAHNOB-HCI: On MAHNOB-HCI, rPPGNet outperforms deep-learning baselines, while STVEN+rPPGNet surpasses all other methods under subject-independent evaluation.STVEN is trained on compressed OBF videos and combined with rPPGNet trained on MAHNOB-HCI, where high-quality videos are unavailable.
- Visualization and discussion: STVEN-enhanced inputs produce attention maps concentrated on forehead and cheek skin regions, consistent with stronger rPPG information in those regions.Recovered signals also show more accurate inter-beat intervals and smoother behavior alongside reduced PSNR fluctuation.
5. Conclusions and Future Work
The paper presents an end-to-end method for recovering rPPG signals from highly compressed videos by enhancing video before estimating heart-rate features. Experiments on two benchmark datasets support its effectiveness, while future work targets compression-aware enhancement and video-quality metrics.
- The method enhances compressed videos with STVEN before cascading rPPGNet to recover signals for HR and HRV measurement.
- Comprehensive experiments on two benchmark datasets verified the effectiveness of the proposed method.
- Future work will use compression-related metrics to constrain STVEN and develop a video-quality metric specialized for rPPG recovery.