PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching

Zitong Yu, Xiaobai Li, Xuesong Niu, Jingang Shi, Guoying Zhao

arXiv:2004.12292v1cs.CV

TL;DR

Existing end-to-end rPPG methods are vulnerable to head movement and bad illumination, motivating a stronger remote HR measurement baseline. AutoHR combines neural architecture search, Temporal Difference Convolution, hybrid time/frequency supervision, and spatio-temporal augmentation; experiments report superior performance across intra- and cross-dataset testing.

  • Problem

    Existing end-to-end facial-video rPPG methods are vulnerable to less-constrained scenarios such as head movement and bad illumination.

    Source: §I · p. 1

  • Method

    AutoHR uses a neural-architecture-searched backbone with Temporal Difference Convolution, hybrid time- and frequency-domain loss, and spatio-temporal data augmentation.

    Source: §I · p. 2, A. Temporal Difference Convolution · p. 2, C. Supervision in the Time and… · p. 3

  • Results

    AutoHR achieves superior or on-par state-of-the-art performance on intra- and cross-dataset tests, supporting its role as a strong end-to-end baseline.

    Source: §I · p. 2, §IV · p. 4

  • Takeaways & Limitations

    AutoHR generalizes well in less-constrained scenarios and provides a strong end-to-end baseline for remote HR measurement.

    Source: §I · p. 2, §IV · p. 4

Abstract

from arXiv · show

Remote photoplethysmography (rPPG), which aims at measuring heart activities without any contact, has great potential in many applications (e.g., remote healthcare). Existing end-to-end rPPG and heart rate (HR) measurement methods from facial videos are vulnerable to the less-constrained scenarios (e.g., with head movement and bad illumination). In this letter, we explore the reason why existing end-to-end networks perform poorly in challenging conditions and establish a strong end-to-end baseline (AutoHR) for remote HR measurement with neural architecture search (NAS). The proposed method includes three parts: 1) a powerful searched backbone with novel Temporal Difference Convolution (TDC), intending to capture intrinsic rPPG-aware clues between frames; 2) a hybrid loss function considering constraints from both time and frequency domains; and 3) spatio-temporal data augmentation strategies for better representation learning. Comprehensive experiments are performed on three benchmark datasets to show our superior performance on both intra- and cross-dataset testing.

I. INTRODUCTION

Remote HR measurement avoids the discomfort and inconvenience of attached sensors, but existing end-to-end facial-video methods are vulnerable to complex scenarios. AutoHR addresses this challenge by combining neural architecture search, temporal and frequency constraints, augmentation, and evaluation across datasets.

  • Remote photoplethysmography measures heart activity without contact, addressing discomfort and inconvenience associated with attached ECG and PPG sensors.Source: §I · p. 1
  • Existing end-to-end rPPG methods directly process facial frames but are easily influenced by head movement and varying illumination.Source: §I · p. 1
  • The paper identifies network architecture, loss function, and data augmentation as key factors influencing end-to-end rPPG robustness and generalization.Source: §I · p. 2
  • AutoHR uses a searched backbone with Temporal Difference Convolution, applying neural architecture search to remote HR measurement.Source: §I · p. 2
  • AutoHR combines time- and frequency-domain constraints with spatio-temporal augmentation to learn rPPG-aware features and improve representation learning.Source: §I · p. 2
  • In intra- and cross-dataset tests, AutoHR achieves superior or on-par state-of-the-art performance and serves as a strong end-to-end rPPG baseline.Source: §I · p. 2

II. METHODOLOGY

Temporal Difference Convolution augments 3D convolution with temporal-difference information while retaining original RGB input. It is designed to capture fine-grained temporal context for robust rPPG recovery.

A. Temporal Difference Convolution

B. Backbone Search for Remote HR Measurement

AutoHR uses neural architecture search to discover a backbone for remote HR measurement, representing cells as directed acyclic graphs whose edges select among candidate operations. The search learns network and architecture parameters jointly, then derives a discrete architecture from the learned operation weights.

Backbone search

  • NAS automatically discovers a backbone for remote HR measurement instead of relying on manually designed end-to-end architectures.The search is based on gradient-based NAS methods.Source: B. Backbone Search for Remote HR… · p. 2

Cell and operation spaces

  • Each searched cell is a DAG whose edges represent information flow through candidate operations weighted by architecture parameters.The cell contains network-layer nodes, and each edge applies a weighted relaxation of the operation space.Source: B. Backbone Search for Remote HR… · p. 2

Search space

  • The architecture search space combines a stem, head, four stacked blocks, two cells per block, and nine candidate operations including TDC.The network also includes three cascaded MaxPool layers after the first three blocks.Source: Figure 3 · p. 3

Search configurations

  • The method compares shared-cell and varied-cell configurations, with either one architecture shared across blocks or separate architectures searched for the blocks.The operation space is also evaluated with and without TDC operators.Source: B. Backbone Search for Remote HR… · p. 3

Architecture derivation

  • Network parameters and architecture parameters are optimized through a bi-level training-and-validation objective before selecting the highest-weight operations and two incoming edges per intermediate node.The final discrete architecture excludes the none operation when choosing operations and edges.Source: B. Backbone Search for Remote HR… · p. 3

C. Supervision in the Time and Frequency Domain

AutoHR combines time-domain signal-rhythm supervision with frequency-domain HR supervision. The overall objective balances these complementary constraints to guide learning of rPPG features.

Motivation

  • Existing NegPearson and SNR losses constrain only the time or frequency domain, motivating supervision in both domains.The paper seeks fine-grained signal guidance in time and spectral-distribution guidance in frequency.Source: C. Supervision in the Time and… · p. 3

Time-domain supervision

Frequency-domain supervision

  • The frequency loss treats HR estimation as classification after transforming the predicted rPPG signal into the frequency domain.The power spectral density of the prediction is compared with the ground-truth HR using cross-entropy.Source: C. Supervision in the Time and… · p. 3

Combined objective

  • The overall loss is Loverall = λ · Ltime + Lfre, combining time-domain and frequency-domain supervision with λ as a balancing parameter.Lfre uses the predicted signal’s power spectral density and the ground-truth HR value.Source: C. Supervision in the Time and… · p. 3

D. Spatio-Temporal Data Augmentation

AutoHR uses spatio-temporal augmentation to address two challenges in remote HR measurement: ROI occlusion from head movement and imbalanced HR distributions.

DA1: occlusion simulation

  • DA1 randomly erases or cuts out partial spatio-temporal tubes within short random clips to mimic ROI occlusion from head movement.The erased region is limited to less than 20% of spatial size and temporal length.Source: D. Spatio-Temporal Data Augmentation · p. 3

DA2: HR balancing

A. Datasets and Metrics

The evaluation uses three public datasets, with VIPL-HR and MAHNOB-HCI supporting subject-independent intra-dataset testing and MMSE-HR supporting cross-dataset testing. Average-HR performance is assessed with error, variability, and correlation metrics.

Datasets and protocols

  • VIPL-HR and MAHNOB-HCI are evaluated with subject-independent 5-fold cross-validation, while MMSE-HR is used for cross-dataset testing.Source: A. Datasets and Metrics · p. 3

Metrics

  • Average HR is evaluated using standard deviation, mean absolute error, root mean square error, and Pearson’s correlation coefficient.Source: A. Datasets and Metrics · p. 3

VIPL-HR comparison

  • Table I reports intra-dataset results on VIPL-HR and distinguishes traditional, non-end-to-end, and end-to-end learning-based methods.Best and second-best results are marked separately.Source: Table I · p. 4

B. Ablation Study

The ablation studies examine loss functions, temporal-difference weighting, NAS configuration, and augmentation strategies, while benchmark results assess AutoHR on challenging intra-dataset settings.

Impact of Loss Functions

  • The lowest RMSE, 9.9 bpm, is obtained with the overall time-and-frequency loss.The frequency-domain loss reduces RMSE by 4.3 bpm compared with the time-domain loss alone.Source: B. Ablation Study · p. 4

Impact of θ in TDC

  • TDC outperforms vanilla 3D convolution in most settings, with θ=0.2 and θ=1 achieving RMSE values of 9.07 and 9.1 bpm.The temporal-difference contribution is controlled by θ.Source: B. Ablation Study · p. 4

Impact of NAS Configuration

  • NAS-searched backbones consistently outperform manually designed backbones, with shared-cell searches and TDC operators producing better-suited architectures.The passage attributes the shared-cell preference partly to search inefficiency and limited data.Source: B. Ablation Study · p. 4

Impact of Data Augmentation

  • DA1 alone increases RMSE by 0.5 bpm, whereas DA1+DA2 improves performance by 5%.DA2 enriches samples with extreme heart-rate values.Source: B. Ablation Study · p. 4

Intra-dataset Testing

  • On VIPL-HR, AutoHR is comparable to RhythmNet, while existing end-to-end methods have low Pearson correlation coefficients of r ≤0.2.RhythmNet requires strict preprocessing, whereas AutoHR learns intrinsic rPPG-aware features without preprocessing.Source: C. Intra-dataset Testing · p. 4
  • On MAHNOB-HCI, AutoHR achieves the lowest MAE at 3.78 bpm and performs on par with RhythmNet.The dataset includes high compression and spontaneous facial motions.Source: C. Intra-dataset Testing · p. 4

D. Cross-dataset Testing

Cross-dataset testing trains on VIPL-HR and tests directly on MMSE-HR, where AutoHR achieves the highest reported correlation and is described as a strong baseline.

Cross-dataset Testing

  • AutoHR achieves the highest Pearson correlation, r=0.89, when trained on VIPL-HR and tested on unseen MMSE-HR videos.The result is reported across traditional, non-end-to-end, and end-to-end methods.Source: D. Cross-dataset Testing · p. 4
  • The cross-dataset result indicates that AutoHR’s predicted heart rates correlate highly with ground truth and that it learns domain-invariant intrinsic rPPG-aware features.The passage explicitly connects the correlation and feature interpretation to unseen-domain testing.Source: D. Cross-dataset Testing · p. 4
  • AutoHR generalizes well in less-constrained scenarios and is presented as a strong end-to-end baseline for the rPPG research community.The conclusion links this claim to exploration of architecture, loss, and augmentation factors.Source: §IV · p. 4

APPENDIX A: DATASET DETAILS

The experiments use three public datasets with distinct recording scales and evaluation roles, alongside fixed face-cropping, clip-sampling, optimization, and neural-search settings.

Dataset Details

  • VIPL-HR contains 3,130 videos from 107 subjects, with finger BVP sensors providing ground-truth heart rate and PPG signals.Source: Appendix A · p. 6
  • MAHNOB-HCI contains 527 videos from 27 subjects and uses EXG2 signals as the ground-truth ECG signal.Source: Appendix A · p. 6
  • Evaluation uses 30-second clips, while testing uniformly divides each video into three 10-second clips.The 30-second clip uses frames 306 to 2135.Source: Appendix A · p. 6, Appendix B · p. 6

Implementation Details

  • Face regions are cropped with MTCNN from the first frame and fixed across following frames, with loss tradeoff λ=0.2.Source: Appendix B · p. 6

Searching Setting

  • Neural architecture search samples 3×128×112×112 clips, starts with eight channels, and searches for 12 epochs on VIPL-HR Fold-1.Architecture parameters use Adam with learning rate 6e-4 and weight decay 1e-3.Source: Appendix B · p. 6

APPENDIX C: THE SEARCHED ARCHITECTURE

The searched AutoHR architecture favors temporal operations in early cell nodes and spatial operations later, while visualizations show more stable high-level tracking and correlated signals under head movement.

The Searched Architecture

  • The searched cell favors temporal convolutions in intermediate nodes 0 and 1 and spatial convolution in node 2.The cited operators are conv 3x1x1, conv 5x1x1, and conv 1x5x5.Source: Appendix C · p. 6

Visualization

  • Under head movement, AutoHR’s high-level features remain robust in tracking particular regions, unlike PhysNet’s unstable high-level features.Both models focus more on forehead regions in low-level features, while both become chaotic when the head bows.Source: Appendix D · p. 6
  • AutoHR’s predicted rPPG signals are highly correlated with ground-truth PPG signals during head movement, but become noisy during fast movement between 10 and 13 seconds.The passage identifies quick, large pose changes as a vulnerability of end-to-end methods.Source: Appendix D · p. 6, Figure 6 · p. 6

Related papers

Loading 2004.12292v1…