PaperlayerView exact revision on arXiv

Source-linked AI summary

RhythmFormer: Extracting Patterned rPPG Signals based on Periodic Sparse Attention

Bochao Zou, Zizheng Guo, Jiansheng Chen, Junbao Zhuo, Weiran Huang, Huimin Ma

arXiv:2402.12788v3cs.CV

TL;DR

rPPG transformers have not clearly outperformed CNNs, partly because quadratic attention causes coarse-grained extraction. RhythmFormer introduces periodic sparse attention with pre-attention and a fusion stem, achieving state-of-the-art results across intra- and cross-dataset evaluations.

  • Problem

    Transformer superiority over CNNs for rPPG remains inconclusive, while quadratic scaling produces coarse-grained extraction that affects robustness and generalization.

    Source: §1 · pp. 2–3

  • Method

    RhythmFormer learns periodic patterns before conventional attention to filter irrelevant computations and uses a fusion stem to guide attention toward rPPG features.

    Source: §1 · pp. 3–4, §3.2 · p. 8

  • Results

    RhythmFormer achieves state-of-the-art performance in extensive intra-dataset and cross-dataset experiments.

    Source: §1 · p. 4, §5 · p. 28

  • Takeaways & Limitations

    Periodic sparse attention and the plug-and-play fusion stem provide a method for recognizing quasi-periodic patterns in weak rPPG signals.

    Source: §5 · p. 28

    The method has a limitation associated with additional computation, which can lower performance in some setting and may require alleviation.

    Source: §4.9 · pp. 27–28

Abstract

from arXiv · show

Remote photoplethysmography (rPPG) is a non-contact method for detecting physiological signals based on facial videos, holding high potential in various applications. Due to the periodicity nature of rPPG signals, the long-range dependency capturing capacity of the transformer was assumed to be advantageous for such signals. However, existing methods have not conclusively demonstrated the superior performance of transformers over traditional convolutional neural networks. This may be attributed to the quadratic scaling exhibited by transformer with sequence length, resulting in coarse-grained feature extraction, which in turn affects robustness and generalization. To address that, this paper proposes a periodic sparse attention mechanism based on temporal attention sparsity induced by periodicity. A pre-attention stage is introduced before the conventional attention mechanism. This stage learns periodic patterns to filter out a large number of irrelevant attention computations, thus enabling fine-grained feature extraction. Moreover, to address the issue of fine-grained features being more susceptible to noise interference, a fusion stem is proposed to effectively guide self-attention towards rPPG features. It can be easily integrated into existing methods to enhance their performance. Extensive experiments show that the proposed method achieves state-of-the-art performance in both intra-dataset and cross-dataset evaluations. The codes are available at https://github.com/zizheng-guo/RhythmFormer.

1. Introduction

rPPG signals are weak, periodic, and vulnerable to noise, while transformer methods have not clearly surpassed CNNs because quadratic attention encourages coarse-grained extraction. RhythmFormer addresses this with periodic sparse attention and a fusion stem, achieving state-of-the-art intra- and cross-dataset performance.

  • rPPG signals have low amplitude and are strongly affected by lighting, motion, and other artifacts, making accurate extraction difficult in complex environments.Source: §1 · p. 3
  • Transformers can model long-range spatial and temporal dependencies in quasi-periodic rPPG signals, but prior work has not convincingly shown superiority over CNNs.Source: §1 · p. 3
  • Quadratic scaling with sequence length and high-dimensional video inputs produce coarse tokens that mix rPPG-rich regions with noisy regions, lowering robustness.Source: §1 · p. 3
  • Periodic sparse attention learns temporal patterns before conventional attention, filters irrelevant computations, and enables finer-grained extraction across H, W, and T.Source: §1 · pp. 3–4
  • The fusion stem captures frame-level BVP wave variations and guides self-attention toward rPPG features, while remaining integrable into existing methods.Source: §1 · p. 4
  • RhythmFormer achieves state-of-the-art performance in both intra-dataset and cross-dataset evaluations.Source: §1 · p. 4

2. Related Work

Prior rPPG research progressed from signal processing and CNNs toward transformers, but transformer advantages remain constrained by quadratic complexity and coarse extraction. RhythmFormer instead uses periodicity-aware sparse attention to obtain finer-grained attention at lower computational cost.

2.1. Remote Physiological Estimation

  • Transformers capture long-range spatial and temporal correlations in BVP signals, yet their performance advantage over CNNs remains unestablished.Source: §2.1 · p. 5
  • Quadratic complexity in prior transformer approaches leads to coarse-grained features that underuse transformers’ dependency-capturing capacity.Source: §2.1 · p. 5
  • RhythmFormer proactively exploits periodicity through a pre-attention stage that filters irrelevant computations and enables finer-grained attention at lower computational cost.Source: §2.1 · p. 5

2.2. Sparse Attention Mechanism

  • Conventional attention requires pair-wise interaction across all spatial-temporal locations, creating substantial computational and memory burdens.Source: §2.2 · p. 5
  • RhythmFormer uses periodic sparse attention because rPPG temporal attention is sparse by periodicity, filtering redundant computations for fine-grained processing.Source: §2.2 · p. 6
  • Its hierarchical temporal periodic transformer contains three TPT blocks, two temporal downsampling modules, and periodic sparse attention with top-k pre-attention token selection.Source: §2.2 · p. 6

3. Methodology

This section presents RhythmFormer’s methodology, combining a fusion stem, patch embedding, hierarchical temporal periodic transformer, and rPPG prediction head to extract fine-grained periodic features. It also describes pooled pre-attention for selecting correlated spatiotemporal regions, refined self-attention, and training with temporal, frequency-domain, and heart-rate constraints.

3.1. The General Framework of RhythmFormer

  • The general framework applies fusion stem processing, non-overlapping patch embedding, hierarchical temporal periodic transformation, and an MLP prediction head that projects rPPG features into BVP waves.Patch embedding aggregates neighboring spatial semantics, reduces transformer computation, and yields tokens without explicit positional embeddings; relative positional cues are supplied implicitly by convolutional and local-context components.Source: §3.1 · p. 7

3.2. Fusion Stem

  • The fusion stem integrates adjacent-frame differences with raw frames to enhance rPPG features at low computational cost and guide self-attention toward subtle skin-color changes.It addresses noise and weak-signal interference by extracting deep features across adjacent five frames, producing more rPPG information and less noise than simple frame differences.Source: §3.2 · pp. 8–9

3.3. Hierarchical Temporal Periodic Transformer

  • The hierarchical temporal periodic transformer uses three TPT blocks with downsampling factors n=1, 2, and 3 to reinforce small-scale inter-frame rPPG features before modeling larger-scale periodic features and reducing noise.Each temporal downsampling module aggregates non-overlapping temporal information at granularity 2^n.Source: §3.3 · p. 10
  • The periodic sparse attention module adds a pre-attention stage before attention to allocate computational resources for fine-grained operations within the temporal periodic transformer.This module is presented as the first part of each TPT block, following temporal downsampling in the hierarchical design.Source: §3.3 · p. 10
  • The hierarchical temporal periodic transformer pools projected Q and K over a larger spatiotemporal receptive field, selects top-k high-correlation tokens, and applies refined attention with local context enhancement.TDC processes Q and K to capture fine-grained local temporal differences, while 3D average pooling filters low-correlation key-value pairs before attention over the selected regions.Source: §3.3 · p. 11

3.4. Model Training

  • Training uses a hybrid loss combining negative-Pearson temporal loss and frequency-domain cross-entropy as Loverall = 0.2 · LTime + LFreq.The frequency loss compares the predicted BVP spectrum with the heart rate derived from the ground-truth BVP wave.Source: §3.4 · p. 12
  • Checkpoint selection adds an HR constraint BHR, computed as KL(HRgt, HRpred), to the overall loss and selects the checkpoint minimizing Loverall + BHR.The predicted and ground-truth heart-rate distributions are modeled as normal distributions, with predicted mean derived from the maximum power-spectrum index.Source: §3.4 · p. 12

4. Experiment

The experiments evaluate RhythmFormer across five public datasets using standard heart-rate and signal-quality metrics, with shared implementation and targeted augmentation for imbalanced heart-rate distributions.

4.1. Dataset and Performance Metric

  • Five public datasets—PURE, UBFC-rPPG, COHFACE, VIPL-HR, and MMPD—were evaluated using MAE, RMSE, MAPE, Pearson correlation, and SNR.The datasets vary in size, frame rate, activity, lighting, and environmental complexity.Source: §4.1 · p. 13
  • MMPD and VIPL-HR provide more challenging evaluation settings because of larger sample sizes and more complex environments than PURE and UBFC-rPPG.Source: §4.1 · p. 13

4.2. Implementation Details

  • The study implemented its proposed and comparative methods with the rPPG toolbox, using facial regions cropped and resized to 128×128 pixels in the first frame, then fixed thereafter.Post-processing applied a second-order Butterworth filter with cutoff frequencies of 0.75 Hz and 2.5 Hz.Source: §4.2 · p. 13

4.3. Details of Data Augmentation

  • Random sampling augmentation addresses imbalanced heart-rate distributions by downsampling examples above 90 bpm and upsampling examples below 75 bpm.Random horizontal flips were also applied to all videos.Source: §4.3 · p. 14

4.4. Intra-dataset Evaluation

Intra-dataset evaluations show strong performance across diverse datasets, including state-of-the-art results on PURE, COHFACE, and MMPD, with robust HR agreement but identifiable challenging cases.

PURE and UBFC-rPPG

  • RhythmFormer outperformed state-of-the-art methods across all metrics on PURE and achieved comparable performance to state-of-the-art methods on UBFC-rPPG.Source: §4.4 · p. 15

COHFACE

  • An MAE of 1.17 was achieved on COHFACE, where the method outperformed previous approaches despite the dataset’s heavy compression.The fusion stem further improved performance for PhysFormer and related models using raw frames.Source: §4.4 · p. 15

VIPL-HR and MMPD

  • RhythmFormer achieved optimal performance on MMPD and comparable performance to state-of-the-art methods on VIPL-HR under more complex environmental conditions.On VIPL-HR, lower MAE but higher RMSE may reflect extremely challenging examples.Source: §4.4 · pp. 15, 17

MMPD correlation analysis

  • On MMPD, RhythmFormer’s estimated heart rates strongly correlated with ground truth, aligned generally with y = x, and mostly fell within the Bland–Altman confidence interval.Outliers were primarily distributed in the 100–110 bpm range, possibly due to distribution differences in the MMPD dataset.Source: §4.4 · p. 17

Waveform visualization

  • Time- and frequency-domain examples showed that predictions aligned with ground-truth trends, captured BVP peaks and variations, and lacked some detailed high-frequency information.The comparisons provide a direct visual assessment of noise and artifact removal.Source: §4.4 · p. 17

4.5. Cross-dataset Evaluation

Cross-dataset experiments evaluate models trained on PURE or UBFC-rPPG and tested on the other datasets, with results supporting strong generalization of RhythmFormer’s learned representations.

Cross-dataset evaluation

  • RhythmFormer showed superior capability in modeling domain-invariant rPPG features and generalizing across diverse datasets.The evaluation trained models on PURE and UBFC-rPPG individually before testing across datasets.Source: §4.5 · p. 19
  • Scatter and Bland–Altman plots further demonstrated strong correlation between predictions and ground-truth heart rates in cross-dataset evaluation.Outliers were predominantly located in value regions with the least training data.Source: §4.5 · p. 21

4.6. Ablation Study

Ablations attribute performance gains to the fusion stem, periodic sparse attention, and multi-temporal modeling, while identifying preferred temporal settings and limits from frame rate, input length, and loss design.

Impact of Fusion Stem

  • Applying the fusion stem to other methods achieved consistent performance, enhancing rPPG feature representation and improving signal-to-noise ratio; a 5:5 fusion ratio of raw frames and frame differences achieved optimal results.Source: §4.6 · pp. 22–23
  • The best fusion-stem configuration used two surrounding frames with a step of 1, while interpolated steps of 0.5 and 1.5 slightly reduced performance.Small frame-rate variations did not significantly affect performance.Source: §4.6 · p. 22

Impact of Periodic Sparse Attention

  • Periodic sparse attention achieved finer-grained context relevance computation than vanilla attention, improving modeling capability while reducing noise interference.Source: §4.6 · pp. 21, 23

Impact of Multi-temporal Scale

  • Multi-scale modeling captured periodic rPPG features more effectively than a single temporal scale and reduced noise, outliers, and the 95% confidence interval.Increasing temporal granularity outperformed decreasing granularity, while hierarchical extraction was more effective than concatenation for quasi-periodic rPPG signals.Source: §4.6 · p. 24
  • A single temporal scale using 4-frame granularity was superior to other settings, whereas global attention incurred substantial memory and computational costs with suboptimal performance.Source: §4.6 · p. 25

Impact of Input Length

  • An input length of 160 frames achieved optimal results, while longer inputs risked losing the face after first-frame detection and shorter inputs captured fewer inter-frame features.Source: §4.6 · p. 25

Impact of FPS on the COHFACE

  • Upsampling COHFACE from 20 to 30 fps worsened MAE, RMSE, and SNR but improved MAPE and Pearson correlation, indicating better variation capture with added noise.Source: §4.6 · pp. 25–26

Impact of Loss Term and Bias Term

  • Ablations identify temporal loss as the core loss component, with frequency-domain loss and heart-rate bias also contributing.Source: §4.6 · p. 26

4.7. Computational Cost

The section compares computational cost across implemented methods using parameter count, MACs, throughput, and peak GPU memory, with attention to inference duration. Transformer-based methods show variable throughput and increased memory usage during longer inference, unlike CNN-based methods.

4.7. Computational Cost

  • The computational-cost comparison reports parameter count, average MACs per frame, throughput per frame, and peak GPU memory per frame.All implemented methods were evaluated using 30-second inference at 128×128 resolution; “(10s)” denotes average results for 10-second inference.Source: §4.7 · p. 26, Table 14 · p. 27
  • Transformer-based methods’ GPU memory usage significantly increased during 10-second inference because of pairwise attention computation between tokens.CNN-based methods maintained unchanged average computational cost per frame, while transformer-based methods varied in throughput and memory usage.Source: §4.7 · p. 26
  • The throughput analysis is provided in Section 4.9.Source: §4.7 · p. 26

4.8. Visualization

The visualization examines pre-attention and refined attention for representative query tokens. Pre-attention concentrates on skin regions with similar temporal phases, while refined attention focuses on selected high-correlation regions for finer rPPG extraction.

4.8. Visualization

  • The visualization compares attention score maps and key responses from both the pre-attention and refined-attention stages.Source: §4.8 · p. 27
  • Refined attention uses the pre-attention top-k region to examine rPPG features with finer granularity.A fine-grained sub-token from the query is used for refined attention within the selected region.Source: §4.8 · p. 27
  • Pre-attention concentrates attention on skin regions with similar temporal phases, expressing the periodic nature of rPPG signals.The visualization uses a coarse-grained token as the pre-attention query and reports attention score maps and key responses.Source: §4.8 · p. 27
  • Concentrating refined attention on top-k regions reduces attention to irrelevant areas and supports detailed rPPG feature extraction with reduced noise interference.Source: §4.8 · p. 27

4.9. Limitation

The limitation analysis identifies implementation overhead in the pre-attention stage despite its low computational cost. Additional GPU kernel launches and memory transactions can reduce throughput, although engineering optimization may alleviate this issue.

4.9. Limitation

  • Although pre-attention operates at a coarse region level and adds little computation, it inevitably causes additional GPU kernel launches and memory transactions.Source: §4.9 · p. 27
  • The visualization presents coarse and fine-grained query tokens, with refined attention calculated within the top-k region selected by pre-attention.Source: Figure 11 · p. 28, §4.9 · p. 28
  • RhythmFormer has lower computational complexity than Physformer but lower throughput, likely because pre-attention introduces additional GPU overhead.The cited overhead includes extra GPU kernel launches and memory transactions.Source: §4.9 · pp. 27–28
  • The throughput issue could be alleviated through engineering efforts such as GPU kernel fusion.Source: §4.9 · p. 28

5. Conclusion

The paper proposes periodic sparse attention and a plug-and-play fusion stem for weak, quasi-periodic rPPG signals. RhythmFormer achieves state-of-the-art performance in extensive experiments, while future work considers broader fine-grained and multimodal periodic applications.

5. Conclusion

  • The proposed method explicitly leverages periodicity to design sparse attention for subtle and periodic rPPG signals.Source: §5 · p. 28
  • RhythmFormer combines periodic sparse attention with a plug-and-play fusion stem to recognize quasi-periodic patterns in weak rPPG signals.Source: §5 · p. 28
  • The method achieves state-of-the-art performance in extensive experiments and is envisioned as a baseline for diverse rPPG methods and applications.Source: §5 · p. 28
  • Future work may explore multimodal periodic sparse attention to align periodic patterns across RGB, radar, and radio-frequency modalities.Source: §5 · p. 28
  • The conclusion reports that periodic sparse attention has proven highly effective in the rPPG task.Source: §5 · p. 29
  • Future work includes testing periodic sparse attention for broader fine-grained or periodic video-understanding tasks.Source: §5 · p. 29

Related papers

Loading 2402.12788v3…