PaperlayerView exact revision on arXiv

Source-linked AI summary

The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video

John Gideon, Simon Stent

arXiv:2111.09748v1cs.CVcs.HC

TL;DR

Reliable physiological estimation from video could support low-cost, noncontact monitoring, but rPPG signals are subtle and existing deep methods commonly rely on labeled data. The paper introduces contrastive self-supervised learning with frequency and temporal priors, plus learned saliency sampling, and reports comparable or better performance than supervised methods without annotation. It also uses saliency maps to provide interpretable evidence about the model’s spatial focus.

  • Problem

    rPPG seeks low-cost, noncontact cardiac monitoring from video, where subtle signals and nuisance variation make reliable estimation challenging and supervised training requires ground truth data.

  • Method

    The paper uses contrastive self-supervised learning based on signal frequency and temporal-smoothness assumptions, with a learned saliency sampler that directs PPG estimation and produces saliency maps.

  • Results

    The approach achieves comparable or better results than recent supervised deep-learning methods across four rPPG datasets without annotation, while saliency sampling provides interpretable output.

  • Takeaways & Limitations

    Fully self-supervised rPPG can support domain adaptation without first acquiring labeled data, while saliency outputs help assess whether the system uses sensible image regions.

  • Takeaways & Limitations

    The signal is assumed to have a visible manifestation and to dominate the visual signal within the target frequency range.

Abstract

from arXiv · show

The ability to reliably estimate physiological signals from video is a powerful tool in low-cost, pre-clinical health monitoring. In this work we propose a new approach to remote photoplethysmography (rPPG) - the measurement of blood volume changes from observations of a person's face or skin. Similar to current state-of-the-art methods for rPPG, we apply neural networks to learn deep representations with invariance to nuisance image variation. In contrast to such methods, we employ a fully self-supervised training approach, which has no reliance on expensive ground truth physiological training data. Our proposed method uses contrastive learning with a weak prior over the frequency and temporal smoothness of the target signal of interest. We evaluate our approach on four rPPG datasets, showing that comparable or better results can be achieved compared to recent supervised deep learning methods but without using any annotation. In addition, we incorporate a learned saliency resampling module into both our unsupervised approach and supervised baseline. We show that by allowing the model to learn where to sample the input image, we can reduce the need for hand-engineered features while providing some interpretability into the model's behavior and possible failure modes. We release code for our complete training and evaluation pipeline to encourage reproducible progress in this exciting new direction.

1. Introduction

Remote photoplethysmography offers noncontact, low-cost cardiac monitoring from video, but subtle signals are easily obscured by nuisance variation and typically require labeled data. This work instead uses contrastive self-supervision based on signal priors, with learned saliency providing an interpretable sampling mechanism.

  • Evaluation: The approach is evaluated against supervised methods on video datasets with physiological ground truth and can match or sometimes improve upon them without annotation cost.The figure caption describes this result while noting that predicted-versus-ground-truth phase offsets may reflect synchronization issues.
  • Motivation: Remote photoplethysmography measures blood-flow-related changes in reflected or transmitted light from a distance using a standard imaging device.Unlike contact PPG, it can support noncontact monitoring from webcam-quality video.
  • Motivation: Lighting changes, motion, facial appearance, and gestures can obscure the underlying PPG signal in video.These nuisance variations motivate learned representations that are robust to changing visual conditions.
  • Approach: The proposed approach trains a deep neural network to estimate PPG and track heart rate from face video entirely without ground truth training data.Its contrastive framework uses assumptions about signal frequency range, temporal smoothness, and visible manifestation.
  • Approach: The method adds a saliency-based sampling module that emphasizes relevant input regions and outputs interpretable maps of the discovered signal location.The reported example highlights the forehead and parts of the nose and cheeks.

2. Background and Related Work

Prior rPPG systems combine preprocessing, PPG extraction, and heart-rate estimation, increasingly using deep learning to handle nuisance variation and distribution shifts. This work applies contrastive learning to temporal frequency changes rather than image-domain augmentation and adds saliency sampling for interpretability.

  • Remote photoplethysmography: rPPG pipelines typically include preprocessing, PPG signal extraction, and heart-rate estimation stages.Earlier approaches often sought robust signals through image colors or selected regions.
  • Remote photoplethysmography: Deep-learning rPPG methods learn spatiotemporal representations intended to handle noise from lighting, motion, and changing facial appearance.Examples include HR-CNN, DeepPhys, and PhysNet, which use different convolutional and recurrent designs.
  • Domain shift: Training and deployment distributions can differ substantially, motivating approaches that adapt using unlabeled samples in new domains.Prior transductive meta-learning models domain shift, whereas this work allows self-supervised training within entirely new domains.
  • Contrastive learning: Unlike standard contrastive learning with image augmentations, this method avoids image-domain augmentation and contrasts video sequences resampled at different rates.The design reflects that the target signal is stronger temporally than within an individual image.
  • Interpretability: A saliency sampling layer shows which image regions the system uses and helps practitioners assess whether unlabeled training has converged to a sensible solution.Its sparsity and temporal-consistency terms favor spatially concentrated, smoothly changing sampling patterns.

3. Method

The method combines saliency-based spatial sampling with a PPG estimator and supervised or contrastive training losses. Contrastive training uses frequency-resampled views and spectral distances, while saliency regularization favors sparse, temporally smooth attention.

  • Saliency Sampler: The pipeline samples video clips, optionally warps them with a saliency sampler, and passes them to a PPG estimator.The sampler both exposes task-relevant image regions and spatially emphasizes them before estimation.
  • Saliency Sampler: The saliency objective favors spatially sparse maps and smooth changes between consecutive frames.Sparsity encourages low-entropy attention, while temporal consistency regularizes frame-to-frame changes.
  • PPG Estimator: The PPG estimator is a modified 3DCNN-based PhysNet that produces a one-dimensional PPG signal from video.The architecture uses spatio-temporal convolutions, adaptive spatial pooling, and a final one-dimensional convolution.
  • Loss Functions: Contrastive training creates anchor, positive, and negative PPG views using frequency resampling, then applies MVTL with PSD MSE distances.The views enforce similarity within short windows while separating frequency-resampled negative examples.
  • Training: Training samples ten-second clips, and heart rate is estimated deterministically from the maximum PSD frequency within the relevant range.Samples with irrelevant power ratio above 60% are redrawn during training.
  • Loss Functions: Supervised training applies a selected loss between the ground-truth and estimated PPG signals, while MCC maximizes correlation at the ideal temporal offset.MCC uses frequency-domain filtering over 40 to 250 bpm and is designed to tolerate random synchronization offsets.

4. Datasets

The evaluation uses publicly available rPPG datasets selected for RGB video, ground-truth PPG, and reproducibility. The experiments cover varied capture conditions, motion settings, and recording environments, with results averaged across repeated runs.

  • Dataset Selection: The study evaluates four publicly available rPPG datasets to avoid the replication barriers of proprietary or unavailable data.The selected data support both training and evaluation with freely available recordings.
  • Dataset Characteristics: The selected datasets contain RGB video with ground-truth PPG and span controlled motion, studio or natural lighting, compression, and driving scenarios.MR-NIRP-Car contributes RGB garage recordings from driving-related data, while PURE and COHFACE provide controlled or varied capture conditions.
  • Dataset Characteristics: MR-NIRP-Car is a publicly available driving dataset with pulse recordings from subjects captured in RGB and near-infrared video.The experiments use its RGB garage subset, which has minimal head motion and consistent lighting.
  • Evaluation Protocol: The experiments average results across 25 independent training runs and compare supervised and contrastive systems with and without saliency.Table 3 also compares mean, median, and dataset-specific strong baselines, with comparability caveats for MR-NIRP-Car and UBFC.
  • Evaluation Protocol: Across the datasets, heart rate generally changes by no more than 2.5 bpm within a ten-second window.This observation supports the short-window heart-rate stability assumption used in the experiments.

5. Experiments

Across four datasets, the contrastive system achieved competitive rPPG performance without ground-truth labels, while the saliency sampler generally changed performance little and exposed interpretable behavior and failure modes.

  • Dataset performance: On PURE, the contrastive method achieved similar performance to other systems without ground truth, with RMSE between 2.4 and 3.0 across systems.The dataset’s high-quality video, constrained environment, and minimal movement likely contributed to the close-to-ideal results.
  • Dataset performance: 4.6 RMSE: the contrastive system performed best on COHFACE without labels, while the supervised model achieved 7.8 RMSE.The contrastive result provides evidence of robustness to video compression; the supervised model still outperformed the strongest comparable baseline at 10.8 RMSE.
  • Dataset performance: 1.6 RMSE: the supervised system improved on the 2.9 RMSE baseline for MR-NIRP-Car, whereas the contrastive system achieved 4.1 RMSE without ground-truth labels.The baseline was knowledge-based and did not rely on training data, so the comparison is not entirely comparable.
  • Dataset performance: 4.9 RMSE and 4.6 RMSE: supervised and contrastive methods respectively improved on the 7.4 RMSE UBFC baseline.The baseline used the first two seconds of all samples for adaptation, so the comparison is not perfectly comparable.
  • Saliency sampler: The saliency sampler had no significant performance effect on PURE and changed performance by at most one point on the other three datasets.It was evaluated both quantitatively for rPPG performance and qualitatively for interpretability.
  • Saliency sampler: The sparsity regularizer reliably improved saliency-map interpretability without significantly harming performance, whereas higher regularization made training less stable.One of five training runs failed to converge at a higher regularization level.
  • Saliency sampler: A periodic nuisance signal exposed a spurious contrastive-model dependency, while supervised sampling removed the injected signal before PPG estimation.The experiment used a random flashing pixel block at 60–180 bpm in UBFC.

6. Discussion

The paper presents fully self-supervised contrastive rPPG estimation and a saliency sampler for interpretable behavior, while identifying larger unlabelled datasets and harder conditions as future directions.

  • Contributions: The paper presents contrastive cardiac-activity estimation from face video as a fully self-supervised approach with zero annotation cost.The authors state that this could enable domain adaptation without first acquiring labelled data.
  • Contributions: The authors introduce a supervised-training loss that is more robust to ground-truth synchronization error and yields improved performance.They also propose saliency sampling to provide interpretable output for checking whether the system behaves as expected.
  • Future directions: Larger unlabelled datasets from sources such as the internet may help improve heart-rate estimation in domains with more severe lighting changes and head motion.The authors also plan to explore learned saliency or attention mechanisms that more efficiently direct downstream PPG estimation while preserving raw pixels.
  • Architecture: The modified PhysNet-3DCNN architecture uses an encoder-decoder structure with 3D convolutions to represent patterns through time.Table 4 defines “s” as stride, “p” as padding, and “C” as the number of input channels.
  • Architecture: The saliency network follows a truncated ResNet-18 structure with pretrained ImageNet weights.Each BasicBlock uses convolutions, batch normalization, ReLU activations, and a residual addition.

A2. Model Architecture

The PPG estimator is a modified 3D-CNN PhysNet, and the saliency sampler uses a shallower architecture designed to detect facial parts.

  • Model architecture: The PPG estimator modifies PhysNet by replacing transposed convolutions with interpolation and convolution to reduce decoder aliasing artifacts.The saliency sampler uses the architecture described in prior work, with its saliency network replaced by a shallower model sufficient for detecting facial parts.

A3. Other Loss Functions/Metrics

The paper discusses Pearson correlation and signal-to-noise ratio as alternative rPPG losses or metrics, each relying on different signal-alignment or frequency assumptions.

  • Correlation metric: Pearson correlation is scale invariant but assumes perfect temporal synchronization between ground truth and observed data.Without synchronization, the network must learn a constant temporal offset.
  • Frequency-domain metric: Signal-to-noise ratio relaxes temporal alignment by comparing target-frequency power against other spectral power in the frequency domain.It assumes that all frequencies outside the target heart-rate bin should be zero.

A4. Dataset PPG Performance

The appendix compares supervised and contrastive systems using PPG-signal statistics across four datasets. Contrastive learning often matches or exceeds supervised performance on synchronization-robust metrics, while metric behavior differs by objective.

  • PC: Supervised training achieves the highest PC across the dataset results.The authors suggest this may reflect learning an aligned mapping to minimally offset ground truth.
  • MCC: Supervised training tends to produce higher MCC, likely because ground truth guides learning toward repeating sensor artifacts.The contrastive method learns the periodic signal visible in the input video without ground-truth labels.
  • Cross-dataset comparison: Contrastive systems often perform comparably or better than supervised systems on synchronization-robust PPG metrics across datasets.These comparisons do not require ground-truth synchronization for evaluation.
  • SNR: Contrastive learning achieves higher SNR across all datasets.SNR penalizes frequencies beyond heart rate, whereas MCC can reward repeating artifacts present in the sensor signal.

A5. Loss Function Robustness

The appendix tests supervised loss functions under increasing video–PPG desynchronization. MCC remains stable as offsets grow, whereas PC degrades and SNR performs poorly across offsets.

  • Experimental setup: The experiment evaluates supervised models trained with PC, MCC, or SNR against a mean-heart-rate baseline under randomized ground-truth shifts.Each training clip receives a uniformly sampled offset between −Omax and Omax.
  • Desynchronization robustness: MCC remains relatively stable as injected synchronization error increases, while PC performance quickly degrades.Without injected desynchronization, PC and MCC perform similarly, consistent with aligned or consistently offset ground truth.
  • Loss comparison: SNR has consistently poor performance at all tested offsets, indicating a weaker supervisory signal than correlation-based measures for rPPG.The experiment varies random offsets up to Omax between 0 and 16 seconds on COHFACE.
  • Conclusion: MCC is favored as the supervised loss function, particularly when synchronization issues may exist in the training data.The choice follows its relative stability under injected offsets.

A6. Dataset Statistics

The appendix examines dataset statistics, training-data protocol, baseline comparability, and saliency-regularization sensitivity. The results indicate benefits from larger contrastive training sets and comparable performance to supervised deep-learning methods, while regularization effects depend on settings.

  • Dataset statistics: Heart-rate variation over short time intervals is treated as a dataset-supported assumption underlying the contrastive approach.Across datasets, the authors report approximately 80%–100% certainty for the relevant short-interval stability pattern.
  • Training-data protocol: Excluding validation data worsens contrastive test RMSE by 0.7 bpm on average across datasets.The result supports the value of larger training sets for contrastive loss, while the data-size trade-off remains future work.
  • Additional baseline: The self-supervised baseline achieves 4.6 RMSE with a 300-frame window, compared with 1.8 and 4.7 RMSE reported for 400- and 256-frame windows in the added baseline.The comparison is qualified because datasets, pretraining, and input-window protocols differ.
  • Additional baseline: Contrastive models achieve performance comparable to supervised deep-learning methods across a range of datasets without annotations.The appendix explicitly frames this as a cross-dataset comparison with the Siamese CNN baseline.
  • Saliency regularization: Varying saliency sparsity and temporal regularization weights does not significantly impact UBFC model performance.Higher sparsity weights make the saliency map more visible, but the best parameters depend on dataset and resolution.
Loading 2111.09748v1…