Source-linked AI summary
AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching
Zitong Yu, Xiaobai Li, Xuesong Niu, Jingang Shi, Guoying Zhao
TL;DR
Existing end-to-end rPPG methods are vulnerable to head movement and bad illumination, motivating a stronger remote HR measurement baseline. AutoHR combines neural architecture search, Temporal Difference Convolution, hybrid time/frequency supervision, and spatio-temporal augmentation; experiments report superior performance across intra- and cross-dataset testing.
Problem
Existing end-to-end facial-video rPPG methods are vulnerable to less-constrained scenarios such as head movement and bad illumination.
Method
AutoHR uses a neural-architecture-searched backbone with Temporal Difference Convolution, hybrid time- and frequency-domain loss, and spatio-temporal data augmentation.
Results
AutoHR achieves superior or on-par state-of-the-art performance on intra- and cross-dataset tests, supporting its role as a strong end-to-end baseline.
Takeaways & Limitations
AutoHR generalizes well in less-constrained scenarios and provides a strong end-to-end baseline for remote HR measurement.
Abstract
from arXiv · showhide
Remote photoplethysmography (rPPG), which aims at measuring heart activities without any contact, has great potential in many applications (e.g., remote healthcare). Existing end-to-end rPPG and heart rate (HR) measurement methods from facial videos are vulnerable to the less-constrained scenarios (e.g., with head movement and bad illumination). In this letter, we explore the reason why existing end-to-end networks perform poorly in challenging conditions and establish a strong end-to-end baseline (AutoHR) for remote HR measurement with neural architecture search (NAS). The proposed method includes three parts: 1) a powerful searched backbone with novel Temporal Difference Convolution (TDC), intending to capture intrinsic rPPG-aware clues between frames; 2) a hybrid loss function considering constraints from both time and frequency domains; and 3) spatio-temporal data augmentation strategies for better representation learning. Comprehensive experiments are performed on three benchmark datasets to show our superior performance on both intra- and cross-dataset testing.
I. INTRODUCTION
Remote HR measurement avoids the discomfort and inconvenience of attached sensors, but existing end-to-end facial-video methods are vulnerable to complex scenarios. AutoHR addresses this challenge by combining neural architecture search, temporal and frequency constraints, augmentation, and evaluation across datasets.
- Remote photoplethysmography measures heart activity without contact, addressing discomfort and inconvenience associated with attached ECG and PPG sensors.
- Existing end-to-end rPPG methods directly process facial frames but are easily influenced by head movement and varying illumination.
- The paper identifies network architecture, loss function, and data augmentation as key factors influencing end-to-end rPPG robustness and generalization.
- AutoHR uses a searched backbone with Temporal Difference Convolution, applying neural architecture search to remote HR measurement.
- AutoHR combines time- and frequency-domain constraints with spatio-temporal augmentation to learn rPPG-aware features and improve representation learning.
- In intra- and cross-dataset tests, AutoHR achieves superior or on-par state-of-the-art performance and serves as a strong end-to-end rPPG baseline.
II. METHODOLOGY
Temporal Difference Convolution augments 3D convolution with temporal-difference information while retaining original RGB input. It is designed to capture fine-grained temporal context for robust rPPG recovery.
- A. Temporal Difference Convolution: Temporal Difference Convolution uses original RGB frames and describes temporal differences at feature levels, avoiding information loss from normalized clipping.
- A. Temporal Difference Convolution: TDC aggregates temporal difference clues from neighboring local temporal regions in addition to spatial information at the current time.
- A. Temporal Difference Convolution: The parameter θ controls temporal-difference contribution, with θ=0 reducing TDC to vanilla 3D convolution.
- A. Temporal Difference Convolution: TDC can mimic temporally normalized frame differences and provide fine-grained temporal context for tracking local ROIs during rPPG recovery.
B. Backbone Search for Remote HR Measurement
AutoHR uses neural architecture search to discover a backbone for remote HR measurement, representing cells as directed acyclic graphs whose edges select among candidate operations. The search learns network and architecture parameters jointly, then derives a discrete architecture from the learned operation weights.
- Backbone search: NAS automatically discovers a backbone for remote HR measurement instead of relying on manually designed end-to-end architectures.The search is based on gradient-based NAS methods.
- Cell and operation spaces: Each searched cell is a DAG whose edges represent information flow through candidate operations weighted by architecture parameters.The cell contains network-layer nodes, and each edge applies a weighted relaxation of the operation space.
- Search space: The architecture search space combines a stem, head, four stacked blocks, two cells per block, and nine candidate operations including TDC.The network also includes three cascaded MaxPool layers after the first three blocks.
- Search configurations: The method compares shared-cell and varied-cell configurations, with either one architecture shared across blocks or separate architectures searched for the blocks.The operation space is also evaluated with and without TDC operators.
- Architecture derivation: Network parameters and architecture parameters are optimized through a bi-level training-and-validation objective before selecting the highest-weight operations and two incoming edges per intermediate node.The final discrete architecture excludes the none operation when choosing operations and edges.
C. Supervision in the Time and Frequency Domain
AutoHR combines time-domain signal-rhythm supervision with frequency-domain HR supervision. The overall objective balances these complementary constraints to guide learning of rPPG features.
- Motivation: Existing NegPearson and SNR losses constrain only the time or frequency domain, motivating supervision in both domains.The paper seeks fine-grained signal guidance in time and spectral-distribution guidance in frequency.
- Time-domain supervision: The time-domain loss uses the predicted rPPG signal and ground-truth PPG signal to provide signal-level supervision.The signal length is denoted by T.
- Frequency-domain supervision: The frequency loss treats HR estimation as classification after transforming the predicted rPPG signal into the frequency domain.The power spectral density of the prediction is compared with the ground-truth HR using cross-entropy.
- Combined objective: The overall loss is Loverall = λ · Ltime + Lfre, combining time-domain and frequency-domain supervision with λ as a balancing parameter.Lfre uses the predicted signal’s power spectral density and the ground-truth HR value.
D. Spatio-Temporal Data Augmentation
AutoHR uses spatio-temporal augmentation to address two challenges in remote HR measurement: ROI occlusion from head movement and imbalanced HR distributions.
- DA1: occlusion simulation: DA1 randomly erases or cuts out partial spatio-temporal tubes within short random clips to mimic ROI occlusion from head movement.The erased region is limited to less than 20% of spatial size and temporal length.
- DA2: HR balancing: DA2 temporally upsamples and downsamples clips to address the severely unbalanced, reversed-V-shaped HR distribution.
A. Datasets and Metrics
The evaluation uses three public datasets, with VIPL-HR and MAHNOB-HCI supporting subject-independent intra-dataset testing and MMSE-HR supporting cross-dataset testing. Average-HR performance is assessed with error, variability, and correlation metrics.
- Datasets and protocols: VIPL-HR and MAHNOB-HCI are evaluated with subject-independent 5-fold cross-validation, while MMSE-HR is used for cross-dataset testing.
- Metrics: Average HR is evaluated using standard deviation, mean absolute error, root mean square error, and Pearson’s correlation coefficient.
- VIPL-HR comparison: Table I reports intra-dataset results on VIPL-HR and distinguishes traditional, non-end-to-end, and end-to-end learning-based methods.Best and second-best results are marked separately.
B. Ablation Study
The ablation studies examine loss functions, temporal-difference weighting, NAS configuration, and augmentation strategies, while benchmark results assess AutoHR on challenging intra-dataset settings.
- Impact of Loss Functions: The lowest RMSE, 9.9 bpm, is obtained with the overall time-and-frequency loss.The frequency-domain loss reduces RMSE by 4.3 bpm compared with the time-domain loss alone.
- Impact of θ in TDC: TDC outperforms vanilla 3D convolution in most settings, with θ=0.2 and θ=1 achieving RMSE values of 9.07 and 9.1 bpm.The temporal-difference contribution is controlled by θ.
- Impact of NAS Configuration: NAS-searched backbones consistently outperform manually designed backbones, with shared-cell searches and TDC operators producing better-suited architectures.The passage attributes the shared-cell preference partly to search inefficiency and limited data.
- Impact of Data Augmentation: DA1 alone increases RMSE by 0.5 bpm, whereas DA1+DA2 improves performance by 5%.DA2 enriches samples with extreme heart-rate values.
- Intra-dataset Testing: On VIPL-HR, AutoHR is comparable to RhythmNet, while existing end-to-end methods have low Pearson correlation coefficients of r ≤0.2.RhythmNet requires strict preprocessing, whereas AutoHR learns intrinsic rPPG-aware features without preprocessing.
- Intra-dataset Testing: On MAHNOB-HCI, AutoHR achieves the lowest MAE at 3.78 bpm and performs on par with RhythmNet.The dataset includes high compression and spontaneous facial motions.
D. Cross-dataset Testing
Cross-dataset testing trains on VIPL-HR and tests directly on MMSE-HR, where AutoHR achieves the highest reported correlation and is described as a strong baseline.
- Cross-dataset Testing: AutoHR achieves the highest Pearson correlation, r=0.89, when trained on VIPL-HR and tested on unseen MMSE-HR videos.The result is reported across traditional, non-end-to-end, and end-to-end methods.
- Cross-dataset Testing: The cross-dataset result indicates that AutoHR’s predicted heart rates correlate highly with ground truth and that it learns domain-invariant intrinsic rPPG-aware features.The passage explicitly connects the correlation and feature interpretation to unseen-domain testing.
- Cross-dataset Testing: AutoHR generalizes well in less-constrained scenarios and is presented as a strong end-to-end baseline for the rPPG research community.The conclusion links this claim to exploration of architecture, loss, and augmentation factors.
APPENDIX A: DATASET DETAILS
The experiments use three public datasets with distinct recording scales and evaluation roles, alongside fixed face-cropping, clip-sampling, optimization, and neural-search settings.
- Dataset Details: VIPL-HR contains 3,130 videos from 107 subjects, with finger BVP sensors providing ground-truth heart rate and PPG signals.
- Dataset Details: MAHNOB-HCI contains 527 videos from 27 subjects and uses EXG2 signals as the ground-truth ECG signal.
- Dataset Details: Evaluation uses 30-second clips, while testing uniformly divides each video into three 10-second clips.The 30-second clip uses frames 306 to 2135.
- Implementation Details: Face regions are cropped with MTCNN from the first frame and fixed across following frames, with loss tradeoff λ=0.2.
- Searching Setting: Neural architecture search samples 3×128×112×112 clips, starts with eight channels, and searches for 12 epochs on VIPL-HR Fold-1.Architecture parameters use Adam with learning rate 6e-4 and weight decay 1e-3.
APPENDIX C: THE SEARCHED ARCHITECTURE
The searched AutoHR architecture favors temporal operations in early cell nodes and spatial operations later, while visualizations show more stable high-level tracking and correlated signals under head movement.
- The Searched Architecture: The searched cell favors temporal convolutions in intermediate nodes 0 and 1 and spatial convolution in node 2.The cited operators are conv 3x1x1, conv 5x1x1, and conv 1x5x5.
- Visualization: Under head movement, AutoHR’s high-level features remain robust in tracking particular regions, unlike PhysNet’s unstable high-level features.Both models focus more on forehead regions in low-level features, while both become chaotic when the head bows.
- Visualization: AutoHR’s predicted rPPG signals are highly correlated with ground-truth PPG signals during head movement, but become noisy during fast movement between 10 and 13 seconds.The passage identifies quick, large pose changes as a vulnerability of end-to-end methods.