PaperlayerView exact revision on arXiv

Source-linked AI summary

EVM-CNN: Real-Time Contactless Heart Rate Estimation from Facial Video

Ying Qiu, Yang Liu, Juan Arteaga-Falconi, Haiwei Dong, Abdulmotaleb El Saddik

arXiv:2212.13843v1cs.CVcs.MM

TL;DR

Remote facial-video HR estimation is motivated by the inconvenience of contact-based monitoring and remains challenged by processing time, realistic motion, and dataset limitations. This paper combines spatial decomposition and temporal filtering with a CNN that regresses HR from feature images. On MMSE-HR, the approach outperforms comparison methods for average and short-time HR estimation, while its operation depends on successful face detection within supported pose ranges.

  • Problem

    Facial-video HR estimation must address motion, signal-processing time, and limited realistic datasets while avoiding contact with the skin.

  • Method

    The framework applies spatial decomposition and temporal filtering to facial video feature images, then uses a CNN to regress HR over one-second intervals.

  • Results

    The approach achieves higher accuracy than comparison methods for average and short-time HR estimation on the MMSE-HR dataset.

  • Takeaways & Limitations

    The results support contactless HR estimation from facial video under realistic conditions using direct CNN-based prediction from filtered feature images.

  • Takeaways & Limitations

    HR is not estimated when faces cannot be detected continuously within one second or fall outside the supported pose ranges.

Abstract

from arXiv · show

With the increase in health consciousness, noninvasive body monitoring has aroused interest among researchers. As one of the most important pieces of physiological information, researchers have remotely estimated the heart rate (HR) from facial videos in recent years. Although progress has been made over the past few years, there are still some limitations, like the processing time increasing with accuracy and the lack of comprehensive and challenging datasets for use and comparison. Recently, it was shown that HR information can be extracted from facial videos by spatial decomposition and temporal filtering. Inspired by this, a new framework is introduced in this paper to remotely estimate the HR under realistic conditions by combining spatial and temporal filtering and a convolutional neural network. Our proposed approach shows better performance compared with the benchmark on the MMSE-HR dataset in terms of both the average HR estimation and short-time HR estimation. High consistency in short-time HR estimation is observed between our method and the ground truth.

I. INTRODUCTION

Remote HR estimation seeks a noninvasive alternative to contact-based monitoring, but motion, illumination, signal-quality, dataset, and processing-time challenges remain. The paper combines spatial and temporal filtering with a CNN to estimate HR from facial videos under realistic conditions.

  • Contactless HR monitoring is pursued because skin-contact devices can be inconvenient and uncomfortable.
  • Facial-video HR estimation is affected by rigid and non-rigid motion, illumination variation, signal strength, and limited appropriate datasets.
  • Whole-face approaches can handle non-rigid motion but require more filtering as environmental noise increases, making instantaneous estimation difficult.
  • Partial-face selection can reduce filtering time, but reliable region selection becomes difficult when subjects move or make facial expressions.
  • Existing approaches commonly use power spectral density after processing the signal into a clear PPG waveform, increasing computing time.
  • The proposed framework extracts facial color changes through spatial decomposition and temporal filtering, then uses a CNN to regress average HR over a time interval.

II. RELATED WORK

Remote HR estimation has progressed from camera-based PPG extraction and filtering toward methods designed for motion and realistic conditions. The paper builds on EVM-related processing by combining spatial decomposition, temporal filtering, and CNN-based HR estimation.

  • Consumer cameras can remotely measure facial PPG, with the green channel generally containing the strongest PPG signal.
  • Eulerian Video Magnification amplifies facial blood-flow-related changes through spatial decomposition and temporal filtering.
  • Later methods addressed motion using chrominance processing, face tracking, reliable-region selection, and challenging datasets with movements and spontaneous expressions.
  • Despite improved accuracy and robustness, prior processing time remained too long for instantaneous HR estimation.
  • The proposed framework combines EVM-inspired spatial and temporal processing with deep learning to estimate HR instantaneously.

III. METHOD

The method detects and tracks a facial cheek ROI, then processes it through spatial and temporal filtering before CNN-based HR estimation. Facial landmarks define a stable skin region that excludes the eyes and mouth, while tracking maintains a fixed ROI size over time.

  • III. METHOD: The pipeline detects and tracks the face, extracts an ROI, computes a feature image, and estimates HR with a CNN.The stages are face processing, spatial-temporal feature extraction, and neural-network regression.
  • A. Face Detection and Tracking: Facial landmarks define the ROI automatically under movement, head rotation, and facial-expression conditions.The method detects 68 landmarks and uses 8 points to define the ROI rectangle.
  • A. Face Detection and Tracking: The ROI coordinates use landmark points to determine the rectangle’s top-left position, width, and height.The equations use points 13, 16, 40, 41, 46, 47, 50, and 52.
  • A. Face Detection and Tracking: Excluding the eye and mouth regions reduces effects from blinking and mouth movements while retaining a cheek region without non-facial pixels.The selected blue rectangle is designed to remain suitable despite head rotation.
  • A. Face Detection and Tracking: ROI tracking keeps the cheek rectangle’s size constant over a time interval, reducing the impact of rigid head motion.The tracked ROI supplies the fixed-pixel temporal input required by feature extraction.

B. Feature Extraction

Feature extraction combines spatial decomposition and temporal filtering to convert one-second ROI sequences into feature images containing blood-flow-related signals. The process reshapes low-frequency spatial information, concatenates it across frames, and filters each channel in the frequency domain.

  • B. Feature Extraction: Spatial decomposition and temporal filtering extract a feature image containing signal related to blood flow.The ROI sequence is decomposed into spatial frequency bands before temporal bandpass filtering.
  • B. Feature Extraction: The method processes consecutive one-second video intervals to support instantaneous HR estimation.The feature-extraction loop produces feature images repeatedly until the video ends.
  • B. Feature Extraction: A wider bandpass filter covers the typical human HR range of 45–240 bpm, corresponding to 0.75–4.0 Hz.Frequency components outside the selected band are set to zero before inverse transformation.
  • B. Feature Extraction: Each ROI frame is downsampled into spatial bands, and the lowest band is reshaped into one column until one frame-rate-sized set is collected.The columns are then concatenated into a new image.
  • B. Feature Extraction: The concatenated image is split into three channels, transformed with FFT by rows, masked to retain the frequency band of interest, and returned to the time domain with IFFT.The three filtered channels are merged into the feature image.

C. HR Estimation by CNN

The CNN treats each extracted feature image as a regression input and outputs the corresponding HR. Its depthwise separable convolution structure reduces computational burden, while normalization, pooling, dropout, and Euclidean loss support training.

  • C. HR Estimation by CNN: A regression CNN maps each feature image to its corresponding HR value.The input image is sized 25×25×3, and the network produces one HR output.
  • C. HR Estimation by CNN: Depthwise separable convolutions divide standard convolution into depthwise and 1×1 pointwise operations to reduce computation and model size.The architecture uses depthwise filtering per input channel followed by pointwise channel combination.
  • C. HR Estimation by CNN: The CNN applies batch normalization and ReLU after convolution, uses stride for downsampling, and averages the final feature map spatially.Average pooling reduces spatial information and computation before the fully connected layer.
  • C. HR Estimation by CNN: A dropout ratio of 0.6 is used to reduce overfitting and improve network generalization during training.The dropout layer randomly stops 60% of feature detectors from working during training.
  • C. HR Estimation by CNN: HR labels are normalized from 45 to 240 bpm into the range 0–1, and Euclidean distance measures prediction error against ground truth.The loss computes the sum of squared differences between label and predicted values.

IV. EXPERIMENTS

The experiments use the challenging MMSE-HR dataset and generate one HR-labeled feature image per second. Training and testing subsets have similar HR distributions, though most samples fall in the 65–95 bpm range.

  • A. Dataset: The MMSE-HR dataset contains 102 RGB videos from 40 participants, recorded at 25 fps for 30 seconds to 1 minute each.The dataset was collected for challenging HR estimation under realistic conditions.
  • A. Dataset: Each second is assigned the mean of its 1000 HR values as the label for the corresponding feature image.The complete dataset produces 5839 feature images.
  • A. Dataset: 730 feature images are randomly selected for testing, with the remaining images used for training and validation.The subsets are compared with the full dataset through their HR distributions.
  • A. Dataset: 73% of the total HR data lies between 65 bpm and 95 bpm.The total dataset has mean HR 84.28 and variance 355.9.
  • A. Dataset: The training-validation and testing subsets have similar distributions to the full dataset.Their means are 84.32 and 84.0, with variances 355.6 and 357.6, respectively.

B. Evaluation Metrics

The evaluation uses HR error—the difference between predicted and ground-truth HR—to assess bias, dispersion, outliers, percentage accuracy, and linear correlation.

  • HR error is defined as the difference between predicted HR and ground-truth HR.
  • Mean error summarizes the average measurement error across N measurements.
  • Standard deviation evaluates the dispersion of HR error, with lower values indicating errors closer to their mean.
  • RMSE measures prediction differences and is sensitive to outliers; lower values indicate fewer outliers in HR error.
  • MeRate measures prediction accuracy as a percentage, with lower values indicating predictions closer to ground truth.
  • Pearson’s correlation measures the linear correlation between predicted HR and ground-truth HR, where 1 indicates total positive correlation.

C. Experiment Design

Three experiments evaluate CNN-based HR estimation, its training behavior, comparisons on MMSE-HR, and short-time prediction under multiple window sizes.

  • Experiment 1: Experiment 1 evaluates the CNN on a testing set of 730 randomly selected feature images, with remaining images used for training and validation.
  • Experiment 2: Experiment 2 compares the proposed approach with existing methods for average HR estimation on the challenging MMSE-HR dataset.
  • Experiment 3: Experiment 3 evaluates instantaneous HR prediction against the comparison methods using sequences with strong HR variation and non-overlapping 4-, 6-, and 8-second windows.
  • Implementation: Experiments run on Windows 10 with C++ and Caffe, using a four-level Gaussian pyramid and a 0.75–4.0 Hz temporal bandpass filter.

A. Visualization of Short-Time HR Estimation

Visualization compares predicted and ground-truth short-time HR across challenge sequences and analyzes error distributions across HR frequency bands.

  • Figure 7: Figure 7 compares ground-truth and predicted HR over three challenge sequences using 4-second windows and representative facial-expression frames.
  • Figure 7: The predicted HR follows the ground-truth trend, correctly predicting 69% of HR changes.
  • Figure 7: Among falsely predicted changes, 76.9% differ from ground truth by less than 5 bpm.
  • Figure 8: Figure 8 distributes HR errors across nine sections, with the horizontal axis showing error range and the vertical axis showing sample proportion.
  • Figure 8: The mean HR error is -1.25 with variance 311.25, and 74.13% of test samples have absolute error within 15 bpm.
  • Figure 8: Higher errors concentrate in the less represented LF and HF bands, whereas only 8.5% of MF samples have lower accuracy.

C. Average HR Prediction on the MMSE-HR Dataset

On MMSE-HR, the proposed approach outperforms existing methods for average and short-time HR prediction, with performance improving as the time window grows.

  • Average HR Prediction: For average HR prediction, the proposed approach reduces SDe to 6.85 and RMSE to 6.95 compared with existing methods.
  • Average HR Prediction: Average HR prediction achieves a MeRate of 6.55% and Pearson correlation ρ of 0.98.
  • Average HR Prediction: Overall, the approach outperforms methods,, and across all reported average HR prediction metrics.
  • Short-Time HR Estimation: For every short-time window size, the approach reports lower SDe, RMSE, and MeRate and higher ρ than the compared methods.
  • Short-Time HR Estimation: As the window size grows, SDe, RMSE, and MeRate decrease while ρ increases, indicating steadily improved performance across metrics.

E. Run Time

The framework reports separate processing speeds for preprocessing and HR estimation, while its evaluation includes dataset comparisons and identifies remaining frequency-coverage limitations.

  • Runtime: 110 fps for preprocessing and 290 fps for HR estimation were measured on a conventional laptop.The first stage includes video capture, frame extraction, face detection, tracking, ROI cropping, and feature-image extraction; the second is the HR estimator.
  • Operational boundary: The method estimates HR only when faces are consecutively detected within 1 s and 68 landmarks are identified.Supported head rotations are yaw from -90° to 90°, roll from -20° to 20°, and pitch from -20° to 20°.
  • Evaluation: The proposed framework is evaluated on MAHNOB-HCI using 27 participants and 527 usable videos with ECG-derived ground truth.Thirty-second intervals were extracted from each sequence, and half the sequences were used for training.
  • Method: The framework directly estimates HR from a spatially decomposed and temporally filtered feature image, reducing computational complexity and processing time.This replaces a series of filters intended to clean the underlying signal before power-spectrum-based estimation.
  • Results: 74.13% of the testing data were well estimated, while the remainder showed large errors attributed to missing high- and low-frequency components.On MMSE-HR, comparisons covered both average and short-time HR estimation, with higher accuracy than other methods.
  • Limitations: The authors identify imbalanced HR-frequency coverage as a remaining limitation and propose a larger dataset with better-proportioned HR data.The low- and high-frequency portions each account for no more than 30% of the data.
Loading 2212.13843v1…