Source-linked AI summary
rPPG-Toolbox: Deep Remote PPG Toolbox
Xin Liu, Girish Narayanswamy, Akshay Paruchuri, Xiaoyu Zhang, Jiankai Tang, Yuzhe Zhang, Soumyadip Sengupta, Shwetak Patel, Yuntao Wang, Daniel McDuff
TL;DR
rPPG research lacks standardized, reproducible code and comparisons, while neural models require substantial preprocessing, modeling, and postprocessing. The paper presents rPPG-Toolbox, an end-to-end framework with public datasets, supervised and unsupervised methods, and benchmarking tools. It also identifies current scope limits, including unsupported architectures, learning paradigms, and physiological applications beyond heart rate.
Problem
Standardization and reproducibility are limited because published code often omits neural-network training and papers may not use comparable data and procedures.
Method
The paper presents an end-to-end toolbox supporting public datasets, preprocessing, supervised and unsupervised models, training, inference, augmentation, and evaluation.
Results
The toolbox provides reproducible benchmark experiments across multiple training and test datasets for reference and comparison.
Takeaways & Limitations
rPPG-Toolbox provides shared implementations and evaluation tools for more consistent comparison and reproduction of camera-based physiological-measurement methods.
Takeaways & Limitations
The current toolbox does not support all recent neural architectures or diverse datasets, unsupervised and self-supervised learning paradigms, or applications beyond heart-rate calculation.
Abstract
from arXiv · showhide
Camera-based physiological measurement is a fast growing field of computer vision. Remote photoplethysmography (rPPG) utilizes imaging devices (e.g., cameras) to measure the peripheral blood volume pulse (BVP) via photoplethysmography, and enables cardiac measurement via webcams and smartphones. However, the task is non-trivial with important pre-processing, modeling, and post-processing steps required to obtain state-of-the-art results. Replication of results and benchmarking of new models is critical for scientific progress; however, as with many other applications of deep learning, reliable codebases are not easy to find or use. We present a comprehensive toolbox, rPPG-Toolbox, that contains unsupervised and supervised rPPG models with support for public benchmark datasets, data augmentation, and systematic evaluation: \url{https://github.com/ubicomplab/rPPG-Toolbox}
1 Introduction
Camera-based physiological sensing is promising but difficult to standardize and reproduce. The paper addresses this gap with an end-to-end toolbox spanning datasets, models, training, inference, and evaluation.
- Camera-based physiological sensing can make health-signal measurement more comfortable and accessible.
- Standardization is severely lacking, making it difficult to interpret results and identify optimal rPPG system designs.
- Existing code often omits neural-network training, hindering end-to-end reproducible experiments and extensions of prior work.
- The rPPG-Toolbox supports six public datasets, preprocessing, six neural architectures, six unsupervised methods, supervised and unsupervised evaluation pipelines, and advanced training and inference.
- The toolbox provides reproducible benchmarks intended to support rigorous and informative comparisons between methods.
2 Related Work
Prior rPPG toolboxes provide useful algorithm implementations, but they generally lack Python, public dataset loading, or neural-network training and evaluation. These gaps limit reproducibility for modern neural methods.
- Existing open-source rPPG toolboxes include iPhys-Toolbox, PPG-I Toolbox, and pyVHR.
- iPhys-Toolbox offers MATLAB implementations and standard baselines but lacks Python support, public dataset loaders, and neural-network training and evaluation.
- PPG-I Toolbox provides six unsupervised signal-separation models and four evaluation metrics but lacks public dataset loading and neural-network training and evaluation.
- pyVHR uses Python and supports numerous unsupervised methods, but provides only two neural networks for inference and none for training.
3 The rPPG-Toolbox
The rPPG-Toolbox standardizes camera-based physiological measurement through dataset preprocessing, diverse signal-processing and neural methods, and reproducible benchmarking across public datasets.
- 3.1 Datasets: The toolbox preprocesses six public datasets spanning varied motion, lighting, skin tones, backgrounds, and recording conditions.Supported datasets include UBFC-rPPG, PURE, SCAMPS, MMPD, BP4D+, and UBFC-Phys.
- 3.1 Datasets: Preprocessed videos are stored as raw and difference frames in [N, W, H, C] arrays, with PPG labels in [N, 1] arrays.Videos can be divided into customizable non-overlapping chunks and processed with multithreading.
- 3.2.1 Unsupervised Methods: Six unsupervised methods recover estimated PPG signals using linear algebra and traditional signal processing.The methods are Green, ICA, CHROM, POS, PBV, and LGI.
- 3.2.2 Supervised Neural Methods: The toolbox implements supervised neural architectures including DeepPhys, PhysNet, PhysFormer, and TS-CAN using PyTorch-based training pipelines.The implementations use architecture-specific inputs, losses, and training choices, including difference-normalized frames for PhysNet and PhysFormer.
- 3.3 Benchmark Experiments: Benchmark results cover UBFC-rPPG, PURE, UBFC-Phys, and MMPD, with supervised methods trained across UBFC-rPPG, PURE, and SCAMPS.The benchmark is intended to provide reproducible reference results for toolbox users.
4 Additional Features
The toolbox adds pseudo-label training, motion-augmented data, physiological multitasking, and extensive training, evaluation, and analysis utilities.
- 4.1 Weakly Supervised Training: Pseudo PPG labels are generated with POS, bandpass filtered from 0.70 to 3.0 Hz, and amplitude normalized using a Hilbert-signal envelope.This supports supervised training when high-fidelity synchronous PPG labels are unavailable.
- 4.1 Weakly Supervised Training: Table 3 evaluates supervised methods trained on the entire BP4D+ dataset using POS-derived pseudo training labels.
- 4.2 Motion Augmented Training: The toolbox supports synthetic, motion-augmented versions of UBFC-rPPG, PURE, SCAMPS, and UBFC-PHYS for exploring motion-diverse training data.Table 4 reports training on motion-augmented UBFC-rPPG and testing on unaugmented PURE.
- 4.3 Extending the rPPG-Toolbox for Physiological Multitasking: BigSmall is implemented for multitasking PPG, respiration, and facial action classification using BP4D+ action-unit data and pseudo PPG labels.Results use 3-fold cross-validation across 12 common action units.
- 4.4 Training, Evaluation and Analysis Features: Training and evaluation utilities include data inspection, metric visualization, Bland–Altman agreement plots, and motion analysis.Tracked metrics include training loss, validation loss, and learning rate.
5 Limitations
The current toolbox has explicit coverage limits in architectures, datasets, learning paradigms, and physiological applications.
- The toolbox does not support all recent neural architectures or diverse datasets, unsupervised and self-supervised learning, or applications beyond heart-rate calculation.Uncovered applications include heart-rate variability, blood pressure, and SpO2.
6 Broader Impacts
The toolbox’s accessibility benefits are accompanied by privacy, surveillance, bias, consent, and misuse concerns that require responsible application.
- Remote sensing can collect sensitive physiological data without direct interaction or, in some cases, the monitored person’s knowledge.
- Widespread deployment in public and workplace settings could enable passive physiological surveillance without explicit consent.
- Physiological profiling could contribute to discrimination when markers such as elevated heart rate are misinterpreted in consequential decisions.
- Accuracy variation across skin tone, lighting, and other factors creates a risk of systematic bias or marginalization.
- Responsible use must address informed consent and potential misuse by businesses or authoritarian regimes.
7 Conclusion
The paper presents rPPG-Toolbox as an end-to-end infrastructure for reproducible camera-based physiological measurement. It combines dataset preparation, supervised and unsupervised methods, training, evaluation, and augmentation support.
- 7 Conclusion: rPPG-Toolbox addresses difficult reproduction and comparison by providing preprocessing, learning, postprocessing, and evaluation code in one open toolbox.
- B.1 Background: rPPG extracts physiological parameters from pixel changes caused by blood-flow-modulated light reflected from or transmitted through the body.
- B.3 Algorithms: Typical video pipelines localize skin regions, aggregate pixels to reduce noise, combine frequency bands, and estimate pulse waveforms through signal decomposition.
- B.3 Algorithms: Deep learning and convolutional neural networks provide current state-of-the-art results by learning nonlinear relationships between skin pixels and physiological signals.
- 7 Conclusion: Camera-based sensing offers scalable, low-cost monitoring through ordinary webcams and smartphones, with potential relevance to telehealth and remote care.
D Investigation of PhysNet
The PhysNet investigation examines implementation choices, architecture trade-offs, dataset diversity, and performance saturation. It reports poor SCAMPS-to-PURE generalization and recommends matching architectures to computational constraints.
- Investigation of PhysNet: DiffNorm frames are recommended as PhysNet inputs because they make training easier to converge.
- Investigation of PhysNet: Poor results when training on SCAMPS and testing on PURE indicate a cross-dataset generalization problem for this setting.SCAMPS is synthetic and can encourage overfitting, particularly for PhysNet’s 3D-CNN architecture.
- Network Recommendation: 2D-CNN architectures are recommended for mobile or edge deployment when computational efficiency is important.
- Network Recommendation: 3D-CNN and Transformer architectures offer superior performance for applications with lenient resource constraints, but require more resources.
- Network Recommendation: Performance saturation across architectures suggests diminishing returns from additional complexity or depth.
- Dataset Diversity: Diverse datasets are important because motion, lighting, and underrepresentation of darker-skinned individuals challenge robustness and may introduce bias.
F.1 rPPG Metrics
The toolbox documents metrics for heart-rate estimation, signal quality, correlation, classification, and uncertainty, with definitions tied to predicted and ground-truth signals.
- Heart-rate metrics: MAE, RMSE, and MAPE quantify differences between predicted and ground-truth signal rates across N instances.The toolbox defines these metrics for predicted rate Rp and ground-truth rate Rg.
- Correlation: Pearson correlation measures agreement between predicted and ground-truth rates using centered values across N instances.The metric uses the correlation coefficient and sample averages.
- Signal quality: SNR compares spectral power around the first and second ground-truth heart-rate harmonics with power elsewhere.The toolbox restricts the analyzed spectrum to 0.75–2.5 Hz, or 45–150 beats/min.
- Uncertainty: Standard error reports estimate accuracy while accounting for the number of samples, with a separate Pearson-correlation calculation.For metrics other than Pearson correlation, the toolbox uses SE = σ/√n; Pearson correlation is handled separately.
- Multitask metrics: The toolbox adds F1, precision, and accuracy metrics for evaluating facial action units in multitasking experiments.F1 is the harmonic mean of recall and precision; precision and accuracy use TP, FP, TN, and FN.
G Additional Results
Additional results reiterate benchmark, pseudo-label, motion-augmentation, and multitasking evaluations while documenting metric interpretations and dataset-filtering details.
- Additional evaluation: Additional tables report RMSE, SNR, Pearson correlation, and standard errors alongside previously reported results.The authors note that supplementary results may differ slightly because they were generated on a different machine and toolbox version.
- Benchmark results: Benchmark tables evaluate PURE, UBFC-rPPG, UBFC-Phys, and MMPD using cross-dataset training with UBFC-rPPG, PURE, and SCAMPS.The table descriptions identify the evaluated datasets and cross-dataset training setup.
- Metric interpretation: The reported metrics include MAE, RMSE, MAPE, Pearson correlation, and SNR for heart-rate estimation, plus F1 for action-unit classification.The metric descriptions specify their interpretations for rate estimation, spectral comparison, and AU classification.
- Motion augmentation: Motion-augmentation results train on augmented UBFC-rPPG data and test on unaugmented PURE, UBFC-Phys, and MMPD data.The experiment uses an open-source motion-augmentation pipeline.
- Pseudo labels: Pseudo-label experiments train supervised methods on BP4D+ using POS-derived pseudo training labels.The experiment uses the entire BP4D+ dataset.
- Dataset handling: The toolbox supports dataset filtering through exclusion lists or task-selection lists in configuration files.These controls determine which videos or tasks enter an evaluation.
I Multitasking Training and Evaluation Details
The multitasking example implements BigSmall for PPG, respiration, and facial action, using subject-wise three-fold evaluation on the BP4D+ action-unit subset.
- Multitask model: BigSmall is implemented as a multitasking model for PPG, respiration, and facial action.The example demonstrates how the toolbox can be extended beyond rPPG-only modeling.
- Data subset: Training and evaluation use the BP4D+ subset containing action-unit labels and approximately 20 seconds from tasks T1, T6, T7, and T8 per subject.This defines the data included in the toolbox’s AU subset.
- Cross-validation: The evaluation uses three folds whose trials contain mutually exclusive subjects.The subject-wise fold assignments are explicitly listed for folds 1–3.
- Pre-processing: Pre-processing visualization helps inspect face detection, cropping, visible face regions, and ground-truth signals before training.These visual checks are presented as part of standardizing rPPG pre-processing.
J.2 Training Loss, Validation Loss, and Learning Rate Visualization
The toolbox documents training defaults and provides configurable evaluation and motion-analysis utilities to inspect model behavior and dataset characteristics.
- Training visualization: Most supervised configurations use a 0.009 learning rate with Adam or AdamW, MSE or Negative Pearson loss, and a 1cycle scheduler.BigSmall is an exception, using a constant learning rate of 0.001.
- Evaluation visualization: Bland–Altman plots can be enabled through an evaluation configuration parameter for additional analysis.The toolbox exposes further plot configuration options as it is refined.
- Motion visualization: OpenFace scripts convert datasets, analyze rigid and non-rigid head motion, and plot motion comparisons across datasets.The utilities support extracting, visualizing, and analyzing motion in rPPG videos.