Source-linked AI summary
PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
Zitong Yu, Yuming Shen, Jingang Shi, Hengshuang Zhao, Philip Torr, Guoying Zhao
TL;DR
rPPG enables contactless physiological measurement, but existing approaches have limited long-range spatio-temporal modeling and can be affected by realistic-scene interference. PhysFormer uses temporal difference transformers and frequency-domain supervision, achieving superior or on-par benchmark performance without large-scale pretraining.
Problem
Existing rPPG approaches have limited spatio-temporal receptive fields, while end-to-end methods can be affected by head movement, illumination, and unrelated features.
Method
PhysFormer is an end-to-end video transformer using temporal difference transformers, label distribution learning, and curriculum-guided dynamic frequency-domain supervision.
Results
PhysFormer achieves superior or on-par state-of-the-art performance on intra- and cross-dataset tests without pretraining on large-scale datasets.
Takeaways & Limitations
PhysFormer can be trained from scratch on rPPG datasets, supporting its use as a transformer baseline for the rPPG community.
Takeaways & Limitations
The attention visualization still shows unexpected responses that may introduce task-irrelevant noise and damage performance.
Abstract
from arXiv · showhide
Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose the PhysFormer, an end-to-end video transformer based architecture, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. Furthermore, we also propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. One highlight is that, unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community. The codes will be released at https://github.com/ZitongYu/PhysFormer.
1. Introduction
PhysFormer addresses contact-free physiological measurement from facial video by modeling long-range spatio-temporal rPPG relationships and adding frequency-domain supervision. It reports superior or comparable performance across intra- and cross-dataset tests without large-scale pretraining.
- Remote photoplethysmography measures heart activity without contact, addressing discomfort and inconvenience associated with attached ECG and PPG sensors.
- Earlier learning-based methods preprocess facial regions into signal representations, while end-to-end methods directly predict rPPG from video but are vulnerable to complex scenarios.
- Long-range attention can exploit similar quasi-periodic rPPG trajectories across facial regions and temporal locations for feature modeling and enhancement.
- PhysFormer uses cascaded temporal difference transformers to enhance rPPG features through global spatio-temporal attention guided by fine-grained temporal skin-color differences.
- Frequency-domain supervision combines label distribution learning and curriculum-learning-guided dynamic loss to improve feature learning and alleviate overfitting.
- PhysFormer achieves superior or comparable state-of-the-art performance on intra- and cross-dataset tests without pretraining on large-scale datasets such as ImageNet-21K.
2. Related Work
Related work covers traditional and deep-learning approaches for remote physiological measurement and the use of transformers in vision. Existing transformer-based video methods are often unsuitable for long-video signal regression, while related rPPG transformer methods use preprocessed representations.
- Remote physiological measurement: Traditional remote physiological measurement methods merge information across color channels or regions of interest and use signal decomposition to improve subtle rPPG recovery.
- Transformer for vision tasks: Vision transformers extend transformer-based sequence modeling from image patches to video tasks including action recognition, detection, super-resolution, inpainting, and animation.
- Transformer for vision tasks: Most existing video transformer methods are incompatible with long-video-sequence signal regression tasks exceeding 150 frames.
- PhysFormer framework: Figure 2 places temporal difference transformers between a tube tokenizer and an rPPG predictor head, with TD-MHSA and ST-FF enhancing global and local spatio-temporal representations.
3. Methodology
PhysFormer combines convolutional preprocessing, tube tokenization, temporal-difference attention, local spatio-temporal refinement, label-distribution supervision, and dynamically weighted temporal-frequency losses for rPPG measurement.
- PhysFormer architecture: PhysFormer begins with a shallow convolutional stem that extracts coarse local spatio-temporal features from an RGB facial video.The stem uses three convolutional blocks with kernels (1x5x5), (3x3x3), and (3x3x3), followed by batch normalization, ReLU, and max pooling.
- PhysFormer architecture: Tube tokenization partitions coarse features into non-overlapping spatio-temporal tokens, aggregates neighboring semantics, and reduces subsequent transformer computation.The token map uses targeted temporal and spatial tube sizes, with dimensions determined by the partitioning ratios.
- Temporal difference transformer: Temporal difference multi-head self-attention uses temporal difference convolution for query and key projections to capture fine-grained temporal color changes in subtle rPPG signals.The attention output concatenates all heads and applies a linear projection; its sparsity is controlled by τ, for which smaller values are used for quasi-periodic rPPG features.
- Temporal difference transformer: A depthwise 3D convolution inside the feed-forward module refines local inconsistencies and noisy features while complementing temporal-difference attention.The added convolution provides richer locality with slight additional computational cost.
- Supervision: Label distribution learning represents each video with adjacent heart-rate labels and trains against the predicted signal's power spectral density using KL divergence.The formulation uses L=139 integer heart-rate classes spanning [42, 180] bpm and constructs distributions around the ground-truth heart rate.
- Supervision: Dynamic supervision combines temporal loss with frequency-domain cross-entropy and label-distribution losses, increasing frequency weighting during training.This curriculum lets the model learn signal trends early and progressively stronger frequency knowledge later; α=0.1, β0=1.0, and η=5.0.
4. Experimental Evaluation
Experiments evaluate PhysFormer for HR, HRV, and RF estimation across four datasets, including intra- and cross-dataset settings. Results show strong performance, while ablations examine tokenization, attention modules, supervision, and model hyperparameters.
- Experimental Setup: Experiments cover HR estimation on four datasets and HRV and RF estimation on the OBF dataset using subject-exclusive evaluation protocols.Reported metrics include SD, MAE, RMSE, and Pearson correlation coefficient r.
- Intra-dataset Testing: PhysFormer achieves the lowest SD of 3.87 bpm and highest r of 0.87 on MAHNOB-HCI among the compared methods.The model is fine-tuned from VIPL-HR because of MAHNOB-HCI’s low illumination and highly compressed videos.
- Intra-dataset Testing: On OBF, PhysFormer outperforms traditional and end-to-end methods across HR, RF, and all HRV metrics, and also improves HR, RF, and LF/HF over CVD.These results support prediction beyond average HR, including respiration and HRV analysis.
- Cross-dataset Testing: On cross-dataset MMSE-HR testing, PhysFormer records SD 5.22 bpm, MAE 2.84 bpm, RMSE 5.36 bpm, and r 0.92, leading all compared methods.The model is trained on VIPL-HR and tested directly on MMSE-HR.
- Ablation Study: Ablations show that the stem, spatial attention, TD-MHSA, and ST-FF are important, while smaller attention temperature and four heads improve performance.Removing the stem increases RMSE by 3.06 bpm; vanilla MHSA/FF yields 10.43/8.27 bpm RMSE, and τ=√Dh yields 9.51 bpm RMSE.
- Ablation Study: Dynamic frequency supervision accelerates convergence and reduces RMSE, while combining label-distribution and frequency cross-entropy losses gives the best performance.Exponential increases in frequency-loss weight outperform linear increases, and label-distribution learning helps constrain frequency interference and propagate adjacent-label information.
- Attention Analysis: Attention visualizations reveal periodic or quasi-periodic responses but also unexpected noisy responses and coarse temporal attention with phase shifts.These observations identify remaining imperfections in the spatio-temporal attention maps.
5. Conclusions and Future Work
PhysFormer delivers strong benchmark performance for remote physiological measurement, but its current design remains costly for mobile deployment and motivates more efficient future architectures.
- PhysFormer achieves superior performance on benchmark datasets using temporal difference transformers and elaborate supervisions.
- The study identifies video-transformer physiological measurement as an early-stage research area with opportunities for more efficient architectures.
- 7.03 M parameters and 47.01 GFLOPs make PhysFormer unfriendly for mobile deployment.
- Future work includes more accurate and efficient spatio-temporal self-attention for long-sequence rPPG monitoring.