Source-linked AI summary
Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner
Eugene Lee, Evan Chen, Chen-Yi Lee
TL;DR
Video-based rPPG enables contactless heart rate estimation but is sensitive to shifts in subjects, environments, and visual conditions, while deployment labels are unavailable. The paper addresses this with a transductive meta-learner that adapts using unlabeled samples, reporting state-of-the-art performance on MAHNOB-HCI and UBFC-rPPG.
Problem
rPPG heart rate estimation is sensitive to deployment distribution shifts, and labeled samples needed for conventional meta-learning adaptation are unobtainable during deployment.
Method
The paper uses transductive meta-learning with synthetic gradient generation and prototypical distance minimization to provide self-supervised adaptation gradients from unlabeled deployment data.
Results
The approach achieves state-of-the-art performance on the MAHNOB-HCI and UBFC-rPPG datasets.
Takeaways & Limitations
Transductive inference enables fast adaptation of rPPG estimation to data outside the training distribution during deployment.
Takeaways & Limitations
The prototypical distance minimization technique assumes that the statistical information modeled by the network does not vary too much over time.
Abstract
from arXiv · showhide
Remote heart rate estimation is the measurement of heart rate without any physical contact with the subject and is accomplished using remote photoplethysmography (rPPG) in this work. rPPG signals are usually collected using a video camera with a limitation of being sensitive to multiple contributing factors, e.g. variation in skin tone, lighting condition and facial structure. End-to-end supervised learning approach performs well when training data is abundant, covering a distribution that doesn't deviate too much from the distribution of testing data or during deployment. To cope with the unforeseeable distributional changes during deployment, we propose a transductive meta-learner that takes unlabeled samples during testing (deployment) for a self-supervised weight adjustment (also known as transductive inference), providing fast adaptation to the distributional changes. Using this approach, we achieve state-of-the-art performance on MAHNOB-HCI and UBFC-rPPG.
1 Introduction
Remote heart rate estimation with rPPG is useful when contact-based ECG and PPG are infeasible, but deployment conditions and subjects can differ from training data. The paper proposes transductive meta-learning for fast, self-supervised adaptation using unlabeled deployment samples.
- Motivation: rPPG enables remote heart rate and heart rate variation estimation from video when contact-based ECG or PPG is infeasible.These physiological measurements are relevant to remote health-care.
- Motivation: Prior deep-learning approaches generally use end-to-end supervised learning with a global model during inference.This corresponds to inductive inference in the paper’s terminology.
- Challenge: Changes in skin tone, facial structure, environment, and other deployment conditions can hinder remote heart rate estimation.Such changes create distribution shifts between training data and real-world deployment.
- Approach: The proposed transductive meta-learner performs fast deployment adaptation through self-supervised weight updates using unlabeled data.The approach addresses the absence of labeled samples during deployment.
- Contributions: The framework combines spatiotemporal meta-learning, synthetic gradients, prototypical distance minimization, and ordinal regression for rPPG estimation.It is evaluated on MAHNOB-HCI and UBFC-rPPG, where the authors report performance exceeding existing methods.
2 Related Work
Prior rPPG research progressed from webcam signal extraction and hand-designed channel methods to deep-learning estimators. The paper positions meta-learning as a way to adapt rapidly when deployment settings are outside the training distribution, despite its origins in labeled few-shot tasks.
- rPPG Background: Remote heart rate estimation derives cardiac pulse information from video-based rPPG rather than contact PPG sensors.rPPG is acquired from visible-spectrum light using a video camera.
- Classical Methods: Early studies extracted rPPG from webcam videos using ambient light, blind source separation, and channel-specific skin absorptivity.The green channel was reported to provide a high signal-to-noise ratio for PPG acquisition.
- Classical Methods: Methods such as CHROM, POS, and related models combine RGB information or skin-reflection assumptions to separate pulse signals from motion distortion.These approaches use hand-designed signal-processing or channel-combination strategies.
- Deep Learning Methods: DeepPhys, rPPGNet, RhythmNet, and PhysNet introduced CNN, attention, recurrent, compression-aware, and 3D architectures for rPPG estimation.The cited approaches remain primarily end-to-end deep-learning systems.
- Meta-Learning: Meta-learning is introduced to enable fast weight adaptation when deployment settings are not covered by the training distribution.Unlike typical few-shot classification, this setting lacks labeled deployment data and maintains separate adaptation and estimation parameters.
3 Methodology
Meta-rPPG frames remote heart-rate estimation as task-based regression over video streams and adapts during deployment without labeled data. Its architecture separates visual feature extraction from temporal rPPG estimation while using synthetic and prototypical gradients to update the feature extractor.
- Input and task formulation: Input face videos are landmark-masked, cropped into K × K frames, and paired with temporally aligned PPG sequences for rPPG prediction.The network outputs an rPPG sequence y ∈ R^T, with a small temporal offset attributed to carotid-radial pulse wave velocity.
- Architecture: During inference, only the feature extractor and rPPG estimator operate, while transductive learning updates only the feature-extractor parameters θ.The synthetic gradient generator is used during training and transductive adaptation, not for ordinary rPPG inference.
- Input and task formulation: Meta-learning samples disjoint video streams as tasks, splitting each sequence into V support frames for adaptation and W query frames for evaluation.This differs from typical few-shot classification because the task is regression over image sequences rather than classification of independently sampled still images.
- Architecture: The model contains a convolutional encoder, an LSTM-MLP rPPG estimator, and a synthetic gradient generator.The encoder extracts per-frame latent features, while the LSTM models temporal information and the MLP estimates rPPG through multitask ordinal regression.
- Transductive adaptation: Deployment adaptation addresses out-of-distribution inputs using synthetic gradients and prototypical-distance minimization to provide gradients to the convolutional encoder.The prototypical method is based on constraining visual representations while treating rPPG estimation as unaffected by visual-input variation.
- Transductive adaptation: The synthetic gradient generator learns to model the gradient of the ordinal rPPG loss with respect to the latent feature z.Because deployment data are unlabeled, adaptation uses the synthetic-gradient and prototypical-distance sources instead of gradients from the ordinal loss.
Minimizing Prototypical Distance.
The method models each video as a task and adapts its visual representation during deployment using unlabeled frames and prototypical information. Training separates adaptation from supervised learning, while ordinal regression represents the ordered rPPG target.
- Task prototypes: Each video sequence is treated as a separate task, with consecutive samples used to estimate a latent-variable prototype.The prototype is formed by averaging latent variables generated from consecutive video samples.
- Task prototypes: The global latent prototype is updated iteratively because Monte Carlo task sampling prevents computing the entire dataset mean in one step.The update uses a weighted average between the previous prototype and a newly sampled prototype, controlled by γ.
- Transductive adaptation: During deployment, the encoder adapts on V frames using synthetic-gradient and prototypical-distance gradients because ordinal-regression labels are unavailable.Adaptation runs for L steps on the same V frames; supervised learning uses the disjoint W frames during training.
- Training phases: The learning phase trains the encoder and rPPG estimator end-to-end, while the synthetic-gradient generator is updated during supervised training.This phase is present only during training because it requires labeled data.
- Training phases: The network is first pre-trained for R epochs so synthetic gradients and prototypes depend on rPPG estimation rather than random weights.The authors state that this pre-training avoids instability before adaptation is introduced.
- Ordinal rPPG estimation: Ordinal regression normalizes PPG segments to [0,1], quantizes them into 40 ordered segments, and formulates estimation as S binary classification problems.The ordering addresses temporal and magnitude discrepancies between rPPG and PPG signals.
4 Experiments
The experiments evaluate Meta-rPPG on MAHNOB-HCI and UBFC-rPPG, comparing five training and inference configurations. Transductive inference produces more focused feature activations and is evaluated through standard error and correlation metrics.
- Evaluation metrics: Performance is reported using standard deviation, mean absolute error, root mean square error, and Pearson correlation coefficient.These metrics are used for evaluation on the public datasets.
- Ablation study: The study evaluates five configurations, from an end-to-end inductive baseline to Meta-rPPG using both prototypical and synthetic transductive adaptation.The configurations separately test Meta-rPPG without adaptation, each transductive component alone, and both components together.
- Datasets: MAHNOB-HCI and UBFC-rPPG provide the public datasets used to evaluate average heart-rate measurement performance.The experiments report results for both datasets, with their results presented in Tables 2 and 3.
- Adaptation-step evaluation: Using L = 10 adaptation steps during evaluation gives the best results because the training procedure also uses L = 10.Figure 3 examines how the number of adaptation steps affects MAE.
- Feature visualization: Transductive inference yields higher-contrast activation maps covering larger facial regions that contribute to remote heart-rate estimation.The visualization compares four subjects and different training methods.
5 Conclusion
The paper introduces transductive inference into rPPG estimation through two mechanisms that provide gradients when labeled deployment data are unavailable. The resulting meta-learning approach achieves substantial improvements and state-of-the-art results on MAHNOB-HCI and UBFC-rPPG.
- Conclusion: The method introduces transductive inference into rPPG estimation using a synthetic gradient generator and a prototypical distance minimizer.These mechanisms provide gradients to the feature extractor when labeled data are unobtainable.
- Conclusion: The meta-learning formulation achieves substantial improvements and state-of-the-art results on the MAHNOB-HCI and UBFC-rPPG datasets.
A Performing Transductive Inference During Deployment
During deployment, the inference process adds adaptation before ordinary feed-forward inference. The first 2 seconds of video supply the data for transductive learning.
- Deployment procedure: The deployment algorithm uses the first 2 seconds of video for transductive adaptation before actual inference.After adaptation, the remaining inference process resembles typical feed-forward deep-neural-network inference.
B Performance Comparison Using Different Adaptation Steps
The study examines how the number of transductive adaptation steps affects performance across multiple metrics and datasets. It finds different step-count behavior for synthetic gradients and the prototypical distance minimizer.
- Evaluation: The adaptation-step study evaluates MAE, SD, RMSE, and Pearson correlation on MAHNOB-HCI and UBFC-rPPG.The results are reported in tables and comparison plots for both datasets.
- Adaptation-step behavior: Synthetic-gradient adaptation should use the same number of steps during inference as during training.This rule does not apply to the prototypical distance minimizer.
- Adaptation-step behavior: The prototypical distance minimizer is hypothesized to converge toward a value without harming performance as adaptation steps increase.Its purpose is to pull out-of-distribution samples toward the training-modeled rPPG distribution center.
C Does Joint Adaptation of both Feature Extractor and rPPG Estimator Give Better Results?
The study tests whether adapting only the feature extractor is preferable to jointly adapting it with the rPPG estimator. The empirical results support updating the feature extractor alone.
- C Does Joint Adaptation of both Feature Extractor and rPPG Estimator Give Better Results?: Updating the feature extractor alone is compared with jointly updating the feature extractor and rPPG estimator during testing.The comparison targets adaptation to a newly observed distribution.
- C Does Joint Adaptation of both Feature Extractor and rPPG Estimator Give Better Results?: The empirical results support the hypothesis that adapting the feature extractor is effective for rPPG estimation.
- C Does Joint Adaptation of both Feature Extractor and rPPG Estimator Give Better Results?: MAE and SD results are reported for MAHNOB-HCI under different adaptation steps, L.These metrics are used to examine adaptation performance.
D Visualization of Feature Activation Map Using Different Methods
The paper examines feature activation maps and adaptation-step effects across multiple heart-rate metrics and datasets. The visualization study emphasizes transductive inference, while the supplied tables and figures organize performance by adaptation steps, L.
- D Visualization of Feature Activation Map Using Different Methods: Feature activation maps compare the proposed methods with an end-to-end supervised baseline and include ablations of individual proposed methods.The study uses these maps to examine the importance of transductive inference for rPPG estimation.
- D Visualization of Feature Activation Map Using Different Methods: MAHNOB-HCI and UBFC-rPPG results are organized by adaptation steps, L, for MAE, SD, RMSE, and Pearson correlation coefficient, R.The supplied tables cover MAE, SD, RMSE, and R for both datasets.
- D Visualization of Feature Activation Map Using Different Methods: The UBFC-rPPG tables report RMSE and Pearson correlation coefficient, R, under different adaptation steps, L.
- D Visualization of Feature Activation Map Using Different Methods: Figures 5–8 show how the number of adaptation steps, L, affects MAE, SD, RMSE, and R.
- D Visualization of Feature Activation Map Using Different Methods: A separate MAHNOB-HCI table compares joint adaptation with updating only the feature extractor for average heart-rate measurement.
E Demonstration Using Video
The paper demonstrates deployment on MAHNOB-HCI videos from three subjects. The example compares preprocessing, feature activations, rPPG signals, power spectral density, and predicted versus ground-truth heart rate.
- E Demonstration Using Video: The algorithm is demonstrated during deployment on videos extracted from MAHNOB-HCI involving three subjects.
- E Demonstration Using Video: Across the demonstrated videos, transductive-inference feature activation maps are described as relatively consistent compared with an end-to-end model.
- E Demonstration Using Video: The example frame compares a pre-processed face, end-to-end feature maps, Meta-rPPG feature maps, and signal, spectral, and heart-rate plots.The plots include rPPG signal, power spectral density, predicted heart rate, and ground-truth heart rate in BPM.
- E Demonstration Using Video: For eight subjects, transductive inference produces higher-contrast activations covering a larger region of facial features contributing to rPPG estimation.