PaperlayerView exact revision on arXiv

Source-linked AI summary

MMPD: Multi-Domain Mobile Video Physiology Dataset

Jiankai Tang, Kequan Chen, Yuntao Wang, Yuanchun Shi, Shwetak Patel, Daniel McDuff, Xin Liu

arXiv:2302.03840v2cs.CV

TL;DR

Public rPPG datasets provide valuable benchmarks but underrepresent mobile-phone capture, diverse skin tones, realistic motion, and varied lighting. MMPD addresses these gaps with a large, multi-domain mobile-video dataset and evaluates established unsupervised and neural methods. The study finds that supervised generalization is limited on darker skin tones and challenging motion or lighting, whereas unsupervised methods generalize better.

  • Problem

    Existing rPPG datasets inadequately represent mobile cameras, diverse appearance, environments, and activities needed for realistic evaluation.

    Source: §I · p. 1

  • Method

    MMPD provides mobile-phone recordings across diverse skin tones, lighting conditions, activities, and labels, alongside evaluations of unsupervised and neural methods.

    Source: §I · p. 1, §V · p. 5

  • Results

    Supervised methods generalize poorly to skin types 4-6 and challenging motion or lighting, while unsupervised methods show better generalizability without training dependence.

    Source: C. Discussion · pp. 4–5

  • Takeaways & Limitations

    MMPD supports benchmarking and highlights the importance of matching training and testing data to real-world conditions for rPPG generalizability.

    Source: C. Discussion · p. 5, §V · p. 5

Abstract

from arXiv · show

Remote photoplethysmography (rPPG) is an attractive method for noninvasive, convenient and concomitant measurement of physiological vital signals. Public benchmark datasets have served a valuable role in the development of this technology and improvements in accuracy over recent years.However, there remain gaps in the public datasets.First, despite the ubiquity of cameras on mobile devices, there are few datasets recorded specifically with mobile phone cameras. Second, most datasets are relatively small and therefore are limited in diversity, both in appearance (e.g., skin tone), behaviors (e.g., motion) and environment (e.g., lighting conditions). In an effort to help the field advance, we present the Multi-domain Mobile Video Physiology Dataset (MMPD), comprising 11 hours of recordings from mobile phones of 33 subjects. The dataset is designed to capture videos with greater representation across skin tone, body motion, and lighting conditions. MMPD is comprehensive with eight descriptive labels and can be used in conjunction with the rPPG-toolbox. The reliability of the dataset is verified by mainstream unsupervised methods and neural methods. The GitHub repository of our dataset: https://github.com/THU-CS-PI/MMPD_rPPG_dataset.

I. INTRODUCTION

rPPG enables noninvasive cardiac-pulse measurement but remains sensitive to appearance, lighting, and motion, while existing datasets inadequately cover these dimensions. MMPD addresses these gaps with diverse mobile-phone recordings and evaluation of neural and unsupervised methods.

  • rPPG performance is brittle because skin tone, lighting, and motion alter or obscure the reflected-light pulse signal.Source: §I · p. 1
  • Existing rPPG datasets lack systematic coverage across appearance, environment, and activity, limiting evaluation of realistic conditions.Source: §I · p. 1
  • MMPD includes 33 subjects with Fitzpatrick skin types 3-6, four lighting conditions, four activities, and mobile-phone video capture.Source: §I · p. 1
  • MMPD evaluates state-of-the-art neural and unsupervised methods to support algorithms for complex realistic scenarios and camera-based physiological measurement bias.Source: §I · p. 1

II. RELATED WORKS

Prior rPPG datasets provide useful benchmarks but differ in duration, modality, motion, skin-tone diversity, and realism. The comparison motivates broader datasets while noting that synthetic data are not directly comparable.

  • Existing datasets span rPPG-specific and broader physiological or vision collections, with PURE, MAHNOB-HCI, BP4D, VIPL-HR, COHFACE, UBFC-RPPG, and others used in comparison.Source: §II · p. 1
  • Dataset size is computed by video-frame count because recording durations differ, while SCAMPS is not directly comparable because it is synthetic.Source: §II · p. 2
  • UBFC-RPPG uses webcam videos with pulse-oximeter ground truth but has limited diversity in represented skin tones and motions.Source: §II · p. 2
  • PURE offers varied motions but lacks variety in skin tones, real-world motion tasks, and lighting conditions.Source: §II · p. 2
  • SCAMPS supplies synthetic physiological and facial-action labels, but its rendered simplicity can lead models toward overfitting and poor performance.Source: §II · p. 2

III. DATASET

MMPD was designed to capture real-world diversity through cross-country recruitment, varied lighting, mobile-phone recording, and simultaneous gold-standard pulse measurement.

  • MMPD contains 660 one-minute videos recorded with a Samsung Galaxy S22 Ultra while gold-standard PPG was recorded simultaneously using an HKG-07C+ oximeter.Source: §III · p. 2
  • The collection recruited subjects from different countries and used varied lighting configurations to capture diversity and complexity found in real-world applications.Source: §III · p. 2

A. Data Collection

The data-collection protocol jointly records face video and finger PPG while varying illumination, motion, and post-exercise state. Its design isolates realistic factors affecting pulse extraction.

  • The protocol simultaneously collects face videos and finger PPG, with recordings made at a distance allowing touch-based measurement.Source: A. Data Collection · p. 2
  • Four lighting conditions span low LED, incandescent, high LED, and natural light, while four motion tasks range from stationary viewing to walking selfie capture.Source: A. Data Collection · p. 2
  • Strenuous exercises raise post-exercise heart rate before stationary recordings, followed by recovery breaks before subsequent experiments.Source: A. Data Collection · p. 2

B. Data Processing

The dataset is converted into accessible formats, synchronized with ground-truth PPG, and labeled to support analysis of factors affecting rPPG.

C. Data Samples

MMPD samples span skin types, lighting conditions, and activities to represent multiple recording domains.

  • MMPD includes Fitzpatrick skin types 3-6, four lighting conditions, and four activities including stationary, head rotation, talking, and walking.Source: C. Data Samples · p. 3

IV. RESULT AND DISCUSSION

The evaluation compares unsupervised methods across isolated skin-tone, motion, and lighting conditions using MMPD.

A. Unsupervised Signal Processing Methods

B. Supervised Deep Learning Methods

A pre-trained TS-CAN model is evaluated on MMPD to examine supervised performance across skin tone, motion, and lighting conditions.

B. Supervised Deep Learning Methods

C. Discussion

MMPD findings show that supervised-method generalizability declines on darker skin types and challenging motion or lighting, while unsupervised methods generalize better because they do not rely on training. The study also emphasizes data-selection and collection practices for improving rPPG performance.

  • Supervised-method generalizability is limited for skin types 4–6 and challenging motion or lighting, whereas unsupervised methods generalize better without training dependence.Performance varies with the similarity between training and test data.Source: C. Discussion · p. 4
  • Public rPPG datasets underrepresent darker skin types, motion, and varied lighting, limiting supervised-method generalizability to real-world conditions.Source: C. Discussion · p. 5
  • Raw camera capture, careful phone positioning, efficient video processing, face alignment, and frame padding can improve mobile-video data quality and synchronization.These practices address subtle facial changes, processing delays, and larger faces in mobile-phone videos.Source: C. Discussion · p. 5
  • Selecting training and testing data while accounting for real-world collection challenges is important for improving rPPG generalizability and accuracy.Source: C. Discussion · p. 5

V. CONCLUSIONS

MMPD is an over-11-hour mobile-phone video dataset spanning four skin tones, four motion conditions, and four lighting conditions. With eight descriptive labels and evaluations of rPPG methods, it addresses diversity limitations in mobile-video benchmarks and supports broader benchmarking.

  • MMPD contains over 11 hours of mobile-phone video covering four skin tones, four motion conditions, and four lighting conditions.Source: §V · p. 5
  • Eight descriptive labels and diverse real-world motion and lighting data target limitations in existing mobile-phone rPPG datasets, especially for darker skin types.Source: §V · p. 5
  • The dataset and its rPPG-method evaluation provide a step toward improving the accuracy and generalizability of the technology.Source: §V · p. 5
Loading 2302.03840v2…