Source-linked AI summary
A Reproducible Study on Remote Heart Rate Measurement
Guillaume Heusch, André Anjos, Sébastien Marcel
TL;DR
The paper addresses limited reproducibility in rPPG research, where published algorithms are often evaluated on private databases. It introduces a public database and evaluates three selected rPPG algorithms across datasets and acquisition conditions, finding that none is suitable for acceptable performance in realistic scenarios.
Problem
rPPG research lacks published evaluations that train and test algorithms on different datasets, limiting principled assessment of generalization to unseen data.
Method
The paper introduces a public database and conducts a thorough, unbiased evaluation of three selected rPPG algorithms across datasets and acquisition conditions.
Results
Only one selected rPPG algorithm has stable behaviour, while performance generally depends strongly on careful parameter optimization and acquisition conditions.
Takeaways & Limitations
Generalization across resolution and illumination conditions should be a high concern when assessing rPPG algorithms.
Takeaways & Limitations
None of the selected algorithms is suitable for acceptable performance when acquisition conditions are unknown a priori.
Abstract
from arXiv · showhide
This paper studies the problem of reproducible research in remote photoplethysmography (rPPG). Most of the work published in this domain is assessed on privately-owned databases, making it difficult to evaluate proposed algorithms in a standard and principled manner. As a consequence, we present a new, publicly available database containing a relatively large number of subjects recorded under two different lighting conditions. Also, three state-of-the-art rPPG algorithms from the literature were selected, implemented and released as open source free software. After a thorough, unbiased experimental evaluation in various settings, it is shown that none of the selected algorithms is precise enough to be used in a real-world scenario.
1. Introduction
Remote photoplethysmography infers heart rate from subtle blood-volume-related changes in facial color captured by ambient light and ordinary cameras. The paper situates this goal within growing efforts to estimate heart rate from video skin pixels.
- Motivation: rPPG measures blood-volume-related light variations using ambient light and widely available sensors such as webcams.PPG uses illumination to measure tissue-volume changes; rPPG seeks the same measurement without structured light.
- Motivation: Camera color signals, especially the green channel, contain subtle changes correlated with blood-volume variation.Earlier work examined average forehead color values and related their dominant frequency to heart rate.
- Related work: Researchers have since developed many approaches for inferring heart rate from video sequences containing skin pixels.
2. Publicly available databases
The paper introduces and describes COHFACE, a public rPPG database with synchronized video and physiological recordings from 40 subjects under studio and natural illumination. Compared with Mahnob HCI-Tagging, COHFACE offers more realistic acquisition variability for benchmarking.
- COHFACE dataset: COHFACE contains 160 videos and physiological signals collected from 40 healthy individuals under more realistic conditions than Mahnob HCI-Tagging.
- Data collection: The dataset synchronizes webcam video with contact photoplethysmography and respiration measurements used as physiological references.The physiological recordings provide pulse and respiration signals for benchmarking rPPG algorithms.
- Data collection: Each subject completed four approximately one-minute webcam sessions, with two studio-lighting and two natural-lighting sequences.Studio sessions used a spot with closed blinds; natural sessions turned off lights and opened the blinds.
- Data distribution: The data are distributed in accessible MP4/AVI video and HDF5 physiological-signal formats.The HDF5 files include subject metadata, illumination condition, pulse, and respiration signals.
- Comparison to Mahnob HCI-Tagging: COHFACE provides more realistic rPPG testing than Mahnob HCI-Tagging through increased subject and acquisition-condition variability, despite fewer video samples.
3. Selected Baseline Algorithms
Three rPPG approaches—CHROM, LiCVPR, and 2SR—were selected as baselines and described through their signal-extraction procedures. Their designs differ in chrominance projection, tracked-face processing, and skin-color subspace rotation.
- Selected baselines: Three approaches were selected to retrieve the pulse signal: CHROM, LiCVPR, and Spatial Subspace Rotation (2SR).
- CHROM: CHROM detects skin-colored pixels, projects mean skin color into a chrominance subspace, filters temporal signals, and combines its two dimensions.Its published results were reported to outperform ICA and PCA baselines.
- LiCVPR: LiCVPR tracks the lower face, corrects the mean-green signal for illumination, removes large variations, and detrends the result.Tracking uses facial keypoints and feature-based affine updates across frames.
- 2SR: 2SR derives pulse from the rotation angle of the skin-pixel subspace between consecutive RGB frames.The method analyzes eigenvectors of the skin-pixel correlation matrix and does not require bandpass post-processing according to its description.
4. Experimental assessment of selected baselines
The experimental assessment evaluates the selected algorithms on the Mahnob HCI-Tagging and COHFACE databases using RMSE and Pearson’s correlation coefficient. It first attempts to reproduce published results before assessing the approaches more broadly.
- Evaluation protocol: The evaluation reports results on both the Mahnob HCI-Tagging and COHFACE databases.
- Evaluation protocol: RMSE and Pearson’s correlation coefficient ρ are used as the performance metrics.
- Evaluation protocol: The assessment first attempts to reproduce published results and then examines the approaches in greater detail.
4.1. Baselines
The study establishes baseline results on 527 Manhob HCI-Tagging sequences, but the authors could not reproduce prior results and identify test-data parameter tuning as a source of bias.
- 527 sequences of 30 seconds from the Manhob HCI-Tagging database were used to report RMSE, correlation, and optimized free-parameter counts.The sequences span frames 306–2135, and Table 2 presents results for the baseline algorithms.
- The authors could not reproduce Li et al.’s reported RMSE of 7.62 and correlation coefficient of 0.82.They attribute the discrepancy to differences in tracking procedure and considered background area, because the shared source code excluded tracking.
- Despite the reproduction discrepancy, the evaluated algorithm still significantly outperformed the two other baselines.
- Tuning parameters directly on test data introduces severe bias because generalization to unseen data cannot then be established.The authors therefore devised strict experimental protocols for subsequent evaluations.
4.2. Unbiased performances in clean illumination conditions
Unbiased evaluation separates training and test subjects to assess generalization under clean illumination, revealing a training–test performance gap and strong sensitivity to dataset conditions for most algorithms.
- 4.2. Unbiased performances in clean illumination conditions: Training and test data must be separated to assess whether parameter choices generalize to unseen data and conditions.The authors note that parameters can work for particular data characteristics, such as skin tone, yet fail elsewhere.
- 4.2.1. Manhob HCI-Tagging: 19 subjects formed the Manhob training set and 10 remaining subjects formed the test set, with sessions assigned by subject identity.The partition contains 3490 sequences recorded across four sessions under constant acquisition conditions.
- 4.2.1. Manhob HCI-Tagging: Performance was generally better on the Manhob training set than on the test set.The authors associate the drop with varying sequence lengths, different optimal parameters, and increased variation across more sequences.
- 4.2.2. COHFACE: 24 COHFACE subjects formed the training set and 16 formed the test set under the studio-lighting protocol.The protocol uses well-lit faces and applies the same subject-based partitioning approach as Manhob HCI-Tagging.
- 4.2.2. COHFACE: On COHFACE, two of three algorithms had Pearson correlation coefficients close to zero or negative, while CHROM remained consistent across datasets.The authors relate this sensitivity to COHFACE’s lower resolution and framerate; CHROM performed better on the test set than on the training set.
4.3. Evaluation on umatched conditions
Cross-database and cross-illumination tests show that rPPG performance depends strongly on acquisition conditions and parameter tuning. Under realistic mismatches, the selected algorithms do not achieve acceptable performance.
- Motivation: Different acquisition conditions, such as changing rooms and lighting, require assessing rPPG algorithms without readjusting parameters for every environment.The evaluation emulates this realistic mobile-device scenario through experiments across settings.
- Cross-database testing: Cross-database testing compares models tuned on Mahnob HCI-Tagging with tests on COHFACE, and vice versa.The reverse direction tunes on COHFACE studio data and tests on Mahnob HCI-Tagging data.
- Cross-database testing: When tuned on Mahnob HCI-Tagging, algorithms perform better on COHFACE studio data than when tuned on COHFACE itself, although LiCVPR and 2SR remain unsatisfactory.The broader Mahnob training set may provide more sequences for parameter tuning.
- Cross-database testing: In the reverse experiment, performance is generally worse because the limited COHFACE studio training set cannot optimize parameters for Mahnob’s more generic test conditions.These results underline the approaches’ sensitivity to parameter choices.
- Illumination conditions: Performance generally drops when algorithms tuned on studio data are applied to the natural-lighting test set.The COHFACE database contains equal-sized natural-lighting and well-lit recordings, enabling mismatched illumination evaluation.
- Illumination conditions: The illumination mismatch indicates that careful parameter selection remains necessary and that none of the selected algorithms is suitable for realistic scenarios with unknown acquisition conditions.The conclusion concerns settings where illumination and other acquisition conditions are not known beforehand.
4.4. Importance of the ROI
The ROI used to extract skin pixels materially affects some algorithms but not others. CHROM remains stable across ROIs, whereas 2SR and LiCVPR show stronger dependence on the selected region and conditions.
- ROI comparison: The full face bounding box, a bottom-face mask, and a skin-color filter are compared as alternative skin-pixel ROIs.The comparison is conducted on the COHFACE test set.
- CHROM: CHROM performance remains stable and comparable to the studio-dataset result regardless of the ROI considered.The authors relate this stability to using mean color, which should vary little across regions.
- CHROM: With parameters tuned under known conditions, CHROM can cope with illumination variability across the evaluated ROIs.This observation accompanies its stable performance on the broader illumination setting.
- 2SR: 2SR performs better with the mask, consistent with its design assumption that selected skin pixels form a compact RGB color subspace.The result suggests that ROI selection strongly influences 2SR results.
- LiCVPR: LiCVPR does not achieve acceptable performance for any ROI and is sensitive to image-resolution and illumination variations.Its lack of acceptable performance persists across the tested skin-pixel regions.
5. Conclusion
The paper introduces a public two-illumination database and evaluates three open-source rPPG algorithms through reproducible protocols. Results show that only one algorithm is stable, while performance remains highly dependent on parameter optimization and generalization across resolution and illumination.
- Contribution: The study evaluates three state-of-the-art rPPG algorithms using a new public database containing 40 subjects recorded under two illumination conditions.The framework also uses different datasets and associated evaluation protocols.
- Findings: Only one evaluated rPPG algorithm shows stable behaviour, while overall performance depends strongly on careful parameter optimization.The experiments also identify generalization across resolution and illumination as a major evaluation concern.
- Reproducibility: The data, protocols, and algorithm implementations are released as open-source free software to support principled and unbiased comparisons.The authors aim to help standardize comparisons of remote heart-rate measurement algorithms.