Source-linked AI summary
VIPL-HR: A Multi-modal Database for Pulse Estimation from Less-constrained Face Video
Xuesong Niu, Hu Han, Shiguang Shan, Xilin Chen
TL;DR
Remote heart-rate estimation from face videos could avoid inconvenient contact monitors, but existing evidence is limited in less-constrained scenarios and large-scale datasets. The paper introduces the multimodal VIPL-HR database and RhythmNet, a deep estimator using spatial-temporal representations. It reports promising accuracy in within-database and cross-database testing, while identifying future work on adjacent measurements, alignment, and situation-specific analysis.
Problem
Existing remote HR methods are mainly evaluated in well-controlled scenarios, and limited large-scale databases constrain deep representation learning for less-constrained estimation.
Method
The paper constructs the multimodal VIPL-HR database and trains RhythmNet with spatial-temporal maps and a staged learning strategy for face-video HR estimation.
Results
The approach achieves promising HR accuracies in both within-database and cross-database testing, including an HRrmse of 8.94bpm and a Pearson correlation coefficient r of 0.73.
Takeaways & Limitations
VIPL-HR’s variations in illumination, movement, and acquisition device support learning an HR estimator with generalization ability to unseen scenarios.
Takeaways & Limitations
Future work includes modeling relations between adjacent sliding-window measurements, improving ROI alignment through 3D modeling, and analyzing methods across recording situations.
Abstract
from arXiv · showhide
Heart rate (HR) is an important physiological signal that reflects the physical and emotional activities of humans. Traditional HR measurements are mainly based on contact monitors, which are inconvenient and may cause discomfort for the subjects. Recently, methods have been proposed for remote HR estimation from face videos. However, most of the existing methods focus on well-controlled scenarios, their generalization ability into less-constrained scenarios are not known. At the same time, lacking large-scale databases has limited the use of deep representation learning methods in remote HR estimation. In this paper, we introduce a large-scale multi-modal HR database (named as VIPL-HR), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database also contains various variations such as head movements, illumination variations, and acquisition device changes. We also learn a deep HR estimator (named as RhythmNet) with the proposed spatial-temporal representation, which achieves promising results on both the public-domain and our VIPL-HR HR estimation databases. We would like to put the VIPL-HR database into the public domain.
1 Introduction
Remote heart-rate estimation from face videos could make physiological monitoring more convenient, but existing methods lack evaluation in less-constrained settings. The paper introduces VIPL-HR and RhythmNet to address database and representation-learning limitations.
- Motivation: HR reflects physical and emotional activities, while contact monitors are inconvenient and limit measurement scenarios.Remote estimation from face videos is presented as a more convenient alternative for applications including training, health monitoring, and nursing care.
- Research gap: Existing video-based methods use remote photoplethysmography or ballistocardiographic signals, but many are evaluated only on private databases.Private-database evaluation makes comparison between methods difficult, while available public databases are limited in scale and conditions.
- Contribution: VIPL-HR is a large-scale multimodal database covering illumination variations, head movements, and acquisition-device changes.Figure 1 compares public-domain databases with VIPL-HR across these three types of variation.
- Contribution: RhythmNet estimates heart rate from face videos using an informative spatial-temporal map and an effective training strategy.The approach is evaluated in both within-database and cross-database experiments.
2 Related Work
Prior remote HR-estimation research includes signal-separation, skin-model, data-driven, and motion-based approaches. The literature remains constrained by small or controlled databases, hand-crafted assumptions, and limited comparability across methods.
- Remote HR estimation: Remote HR estimation methods use blind signal separation, skin optical models, or data-driven representations of facial signals.Examples include ICA on RGB channels, chrominance features, matrix temporal representations, and time-frequency maps.
- Remote HR estimation: BCG-based methods infer pulse from subtle head motions, but voluntary subject movements can strongly influence their HR signals.These methods track facial key points and use PCA to derive a pulse signal from feature-point trajectories.
- Limitations of prior work: Existing approaches are often tested on small, well-controlled databases and commonly rely on step-by-step pipelines with hand-crafted features.The paper argues that large-scale data-driven methods are needed because HR-related visual information is weak in face images.
- Public-domain databases: Private-database evaluations make performance comparisons difficult, while public databases remain limited in subjects or recording situations.The cited public resources include MAHNOB-HCI, MMSE-HR, PURE, and PFF, among others.
3 The VIPL-HR Database
VIPL-HR is a large-scale, multi-modal database designed to evaluate remote heart-rate estimation under natural head movements, illumination changes, and diverse recording devices. It includes synchronized physiological measurements, multiple recording situations, and a compressed version intended to preserve HR signals while easing data sharing.
- Setup and data collection: The collection uses web, RGB-D, and smartphone cameras to represent device diversity in daily-life recording conditions.The setup also includes a pulse oximeter and filament lamp for synchronized measurements and illumination changes.
- Setup and data collection: Nine recording situations combine natural behavior, head movement, illumination changes, camera distance, and fixed or handheld smartphone capture.Subjects were encouraged to talk and look around, and recordings also considered HR changes after exercise.
- Database compression: MJPG compression and two-thirds image resolution were selected, producing a compressed VIPL-HR dataset of about 48GB.The baseline RMSE evaluations indicated that MJPG preserved HR signals well and two-thirds resolution caused little signal damage.
- Database overview: VIPL-HR contains 2,378 color videos and 752 NIR videos from 107 participants aged 22–41.Each video is approximately 30 seconds long at about 30 fps.
- Database statistics: Head-pose rotation amplitudes reached 92° in roll, 105° in pitch, and 104° in yaw across videos with head movement.Mean face-region gray-scale intensity ranged from 60 to 212, while ground-truth HR ranged from 47 bpm to 146 bpm.
4 Deeply Learned HR Estimator
RhythmNet estimates heart rate from less-constrained face videos using spatial-temporal maps and a deep CNN. The pipeline localizes and aligns facial regions, represents their temporal color information, and trains for robustness to missing data.
- Estimator overview: RhythmNet divides each input video into short clips and predicts one HR value per clip before averaging them for the video.A CNN maps spatial-temporal representations to clip-level HR estimates.
- Face processing: Face localization uses SeetaFace5 landmarks, moving-average stabilization, eye-based alignment, and skin segmentation over cheek and forehead regions.The selected ROI is intended to retain facial areas containing color changes associated with heart rhythms.
- Spatial-temporal representation: Spatial-temporal maps divide the aligned face into n blocks and concatenate average pixel values across T frames and C YUV channels.The resulting map has size n × T × C, with blocks placed directly into rows.
- Network and inference: The model uses ResNet18 to learn the mapping from spatial-temporal maps to HR values.Each face video is processed with a fixed sliding window to generate clip-level maps.
- Training strategy: Three-stage training proceeds from ImageNet pre-training to synthetic rhythm-map training and final fine-tuning on real face videos.Synthetic data supplies prior knowledge for mapping video sequences to HR values.
- Robustness training: Random temporal masking of spatial-temporal maps simulates face-detector failures and trains robustness to missing signal data.The failure condition is associated with short intervals during head movement or rotation.
5 Experiments
The experiments evaluate RhythmNet within VIPL-HR and across databases using multiple HR-error and correlation metrics. RhythmNet performs well on color videos and generalizes to MMSE-HR, with further improvement after fine-tuning.
- Evaluation Protocols: The evaluation uses VIPL-HR for within-database testing and MMSE-HR for cross-database evaluation, including direct testing and fine-tuning.Within-database testing uses participant-dependent five-fold cross-validation; cross-database testing trains on VIPL-HR before testing or fine-tuning on MMSE-HR.
- NIR Face Videos: RhythmNet is also evaluated on VIPL-HR NIR videos using one-channel spatial-temporal maps, with 497 videos included because detected faces were available.No color space transformation is used for the single-channel NIR videos.
- Cross-database Testing: 10.58 bpm is the HRrmse when the VIPL-HR-pre-trained RhythmNet is directly tested on MMSE-HR.This cross-database result is reported before fine-tuning on MMSE-HR.
- Cross-database Testing: 8.22 bpm is the HRrmse after fine-tuning the VIPL-HR-pre-trained model on MMSE-HR, outperforming the previous methods used for comparison.The experiments compare against Li2014, Haan2013, and Tulyakov2016.
6 Conclusion and Further Work
The paper presents VIPL-HR and RhythmNet for remote heart-rate estimation under less-constrained conditions. Future work targets richer signal representations, temporal relationships between adjacent measurements, improved ROI alignment, and broader analyses and applications.
- Conclusion: VIPL-HR is a multimodality database designed for remote heart-rate estimation under head movement, illumination change, and camera diversity.These variations represent less-constrained recording conditions.
- Conclusion: RhythmNet uses a CNN, a spatial-temporal map, and an effective training strategy for remote heart-rate estimation.The approach achieves promising accuracy in both within-database and cross-database testing.
- Further Work: Future work will investigate new HR signal representations and models that leverage relations between adjacent sliding-window measurements.The authors also plan to study face dense alignment via 3D modeling for improved ROI alignment.
- Further Work: The authors plan detailed analyses across recording situations and applications of the approach to face presentation attack detection.These plans use the VIPL-HR database to analyze individual methods under varied recording conditions.