Source-linked AI summary
PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformer
Zitong Yu, Yuming Shen, Jingang Shi, Hengshuang Zhao, Yawen Cui, Jiehua Zhang, Philip Torr, Guoying Zhao
TL;DR
Facial rPPG seeks contactless physiological measurement, but existing methods face interference and limited long-range spatio-temporal modeling. The paper introduces PhysFormer and PhysFormer++, transformer architectures with temporal-difference modeling, SlowFast pathways, and frequency-domain supervision. The models achieve superior or on-par state-of-the-art performance across intra- and cross-dataset tests and can be trained from scratch on rPPG datasets.
Problem
Facial-video rPPG must recover physiological signals without contact, while existing end-to-end methods are vulnerable to complex scenarios and limited long-range spatio-temporal modeling.
Method
PhysFormer and PhysFormer++ use temporal-difference video transformers, with PhysFormer++ adding SlowFast periodic and cross-attention pathways plus frequency-domain supervision.
Results
The PhysFormer family achieves superior or on-par state-of-the-art performance on intra- and cross-dataset tests without large-scale pretraining.
Takeaways & Limitations
The PhysFormer family provides a transformer baseline for rPPG that can be trained from scratch on rPPG datasets while modeling long-range spatio-temporal features.
Takeaways & Limitations
The proposed PhysFormer family remains computationally costly for mobile-level rPPG applications, motivating lighter designs using quantization or binarization.
Abstract
from arXiv · showhide
Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose two end-to-end video transformer based architectures, namely PhysFormer and PhysFormer++, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. To better exploit the temporal contextual and periodic rPPG clues, we also extend the PhysFormer to the two-pathway SlowFast based PhysFormer++ with temporal difference periodic and cross-attention transformers. Furthermore, we propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and PhysFormer++ and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. Unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer family can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community.
1 Introduction
Facial rPPG offers contactless physiological measurement, but existing approaches struggle with interference and limited long-range spatio-temporal modeling. PhysFormer and PhysFormer++ address these issues with temporal-difference transformers, SlowFast pathways, and frequency-domain supervision.
- Contactless rPPG aims to measure heart activity remotely, avoiding the discomfort and inconvenience of attached ECG or PPG/BVP sensors.
- End-to-end facial-video methods can be strongly affected by head movement, illumination, and rPPG-unrelated features, causing large performance drops on realistic datasets.
- PhysFormer uses temporal-difference transformer blocks to enhance rPPG features through global spatio-temporal attention based on fine-grained temporal skin-color differences.
- PhysFormer++ adds a two-pathway SlowFast architecture with periodic and cross-attention to capture temporal-contextual and periodic rPPG clues efficiently.
- Label distribution learning and curriculum-learning-guided frequency-domain supervision complement weak temporal signals, reduce interference-induced overfitting, and encourage intrinsic rPPG-aware features.
- Across intra- and cross-dataset tests, the PhysFormer family achieves superior or on-par state-of-the-art performance without large-scale pretraining.
2 Related Work
rPPG research spans traditional signal processing, non-end-to-end feature-map methods, and direct end-to-end learning. PhysFormer extends end-to-end modeling with video transformers that capture long-range spatio-temporal rPPG features while training from scratch on rPPG datasets.
- 2.1 rPPG measurement: Traditional rPPG methods use handcrafted color, region, filtering, decomposition, and projection choices, but require empirical design knowledge and lack supervised adaptation to severe interference.
- 2.1 rPPG measurement: Non-end-to-end STmap methods learn from dense, less-irrelevant facial signals and can converge faster, but require explicit and exhaustive preprocessing.
- 2.1 rPPG measurement: End-to-end methods process facial sequences directly with spatial or spatio-temporal models, reducing dependence on handcrafted task-specific preprocessing.
- 2.2 Transformer for vision tasks: The PhysFormer framework combines a shallow stem, tube tokenizer, temporal-difference transformer blocks, and an rPPG predictor, with TD-MHSA and ST-FF enhancing global and local representations.
- 2.2 Transformer for vision tasks: PhysFormer and PhysFormer++ are end-to-end video transformers designed to capture long-range spatio-temporal attentional rPPG features directly from facial video.
3 Methodology
PhysFormer and PhysFormer++ combine temporal-difference transformer processing with local spatio-temporal refinement, while PhysFormer++ adds SlowFast pathways for periodic and cross-pathway modeling. Label-distribution and dynamically expanded frequency-domain supervision further constrain learning for rPPG estimation.
- PhysFormer: PhysFormer extracts coarse convolutional features, partitions them into tube tokens, processes them with temporal-difference transformer blocks, and projects refined features into a 1D rPPG signal.The predictor temporally upsamples, spatially averages, and projects the transformer output.
- PhysFormer: Temporal-difference self-attention uses convolutional query and key projections to capture fine-grained local temporal differences, with a smaller attention temperature chosen to produce sparser activation.The temperature controls sparsity, and the smaller setting is motivated by the quasi-periodicity of rPPG features.
- PhysFormer++: PhysFormer++ uses SlowFast pathways with temporal-difference periodic transformers in the slow pathway and cross-attention transformers linking fast and slow features.The fast pathway provides fine-grained temporal information, while lateral connections transfer complementary details to the slow pathway.
- PhysFormer++: TD-MHPSA augments temporal-difference self-attention with learnable rPPG-aware periodicity encoding and supervises periodic attention using peak-map autocorrelation.The peak map is derived from ground-truth BVP peaks and used in a binary cross-entropy periodic-attention loss.
- Supervision: Label distribution learning represents each video with adjacent heart-rate labels, while dynamic frequency-domain supervision gradually enlarges constraints to reduce overfitting.The label-distribution loss applies KL divergence between a Gaussian HR distribution and the predicted signal’s power spectral density.
4 Experimental Evaluation
Across intra- and cross-dataset evaluations, PhysFormer and PhysFormer++ provide strong rPPG measurement across multiple physiological signals and challenging conditions. Ablations show that temporal-difference attention, supervision, tokenization, and SlowFast interactions materially affect performance.
- HR estimation on VIPL-HR: PhysFormer and PhysFormer++ achieve better or on-par performance with state-of-the-art methods while training directly from facial videos without preprocessing.On VIPL-HR, the models learn intrinsic and periodic rPPG-aware features automatically; performance is also on par with Meta-rPPG without transductive adaptation.
- Visualization and discussion: PhysFormer++ tracks HR changes under exercise and correlates with ground-truth HR across 47–147 bpm, while producing accurate interbeat intervals.These observations support robustness for rapidly changing heart rates and HRV-related analysis.
- HR, HRV and RF estimation on OBF: The models outperform CVD for HR, RF, and HRV-related measurements on OBF, extending evaluation beyond average heart-rate estimation.The results support prediction of rPPG signals for respiration-frequency measurement and HRV analysis.
- Cross-dataset testing: PhysFormer and PhysFormer++ generalize well from VIPL-HR to unseen MMSE-HR domains, including differing skin tones and lighting conditions.The cross-dataset results also indicate domain-invariant intrinsic rPPG-aware features and more accurate physiological signals than EfficientPhys-T1.
- Ablation study: TD-MHSA and ST-FF are necessary for PhysFormer, whereas vanilla attention produces RMSE values of 10.43 and 8.27 bpm without them.The proposed TD-MHSA aggregates long-range spatio-temporal features using local physiological clues, unlike vanilla MHSA alone.
5 Conclusions
The paper proposes PhysFormer and PhysFormer++ as end-to-end video transformer architectures for remote physiological measurement. With temporal-difference transformers and elaborate supervision, the PhysFormer family achieves strong benchmark performance across intra- and cross-dataset evaluations.
- PhysFormer and PhysFormer++ are proposed as end-to-end video transformer architectures for remote physiological measurement.
- Temporal-difference transformers and elaborate supervision enable the PhysFormer family to achieve superior benchmark performance on intra- and cross-dataset testings.
- Comprehensive ablation studies and visualization analysis demonstrate the effectiveness of the proposed methods.
- Future work will explore more accurate and efficient spatio-temporal self-attention for long-sequence rPPG monitoring and applications beyond rPPG measurement.