Explaining RhythmFormer: A Systematic XAI Analysis of Periodic Sparse Attention for Remote Photoplethysmography

• Interpretability matters because clinicians need to understand and trust rPPG model reasoning before these systems enter diagnostic workflows.Source: §1 · p. 2
• Existing rPPG XAI mainly uses individual visualizations without systematic cross-method comparison, quantitative faithfulness evaluation, or physiological validation.Source: §1 · p. 2
• The framework adapts four attribution methods—raw attention, rollout, flow, and Beyond Intuition—to RhythmFormer’s bi-level routing attention with sparse top-k selection.Source: §1 · p. 2
• The study introduces skin attention quantification to measure attribution overlap with physiologically relevant face and neck regions across TPT levels.Source: §1 · p. 2
• SaCo is adapted to rPPG regression to provide quantitative faithfulness assessment, using perturbation impacts related to prediction quality.Source: §1 · pp. 2–3
• The experiments test whether spatial focus on skin regions corresponds to waveform Pearson r and heart-rate prediction error on UBFC-rPPG.Source: §1 · pp. 2–3

2.1 XAI Methods

• Raw attention provides per-layer views, rollout aggregates transitive token relationships, and flow computes maximum flow through an attention graph.Source: §2.1 · p. 3
• Beyond Intuition combines attention rollout with integrated gradients, separating information flow from attention entries to which the model is sensitive.Source: §2.1 · p. 3
• The paper focuses on four attention-based variants—raw, rollout, flow, and Beyond Intuition—to address concerns about raw attention faithfulness.Source: §2.1 · p. 3

2.2 XAI in rPPG

• Prior rPPG XAI relies on raw attention, gradient saliency, or Grad-CAM, and can highlight non-physiological areas such as red clothing without quantitative faithfulness evaluation.Source: §2.2 · p. 3

2.3 Quantification in XAI—Faithfulness Evaluation

• Cumulative perturbation confounds individual pixel-group effects, whereas SaCo perturbs groups independently and measures pairwise consistency between salience ranking and prediction impact.Source: §2.3 · p. 3
• This work presents the first adaptation of SaCo to rPPG regression.Source: §2.3 · p. 3

3.1 RhythmFormer Overview

• RhythmFormer maps RGB facial video to an rPPG signal through a Fusion Stem, Patch Embedding, hierarchical TPT, and Predictor Head.Source: §3.1 · p. 4
• The hierarchical TPT contains three levels with two TPT blocks each, totaling six attention layers at temporal resolutions T/2^n.Source: §3.1 · p. 4
• Periodic sparse attention uses a two-stage mechanism in which temporal-difference-convolved queries and keys produce scores for top-k region selection.Source: §3.1 · p. 4
• After routing, gathered key-value pairs undergo standard scaled dot-product attention with softmax in the refined attention stage.Source: §3.1 · p. 4

V g + LCE(V ), (1)

• RhythmFormer’s six BRA layers use top-40 routing, while LCE adds a depthwise local positional bias to the value pathway.Top-40 retains approximately 6.3%, 6.3%, and 12.5% of candidate regions across the three TPT levels; the scaling factor follows standard scaled dot-product attention [36].Source: V g + LCE(V ), (1) · p. 4

3.2 Attention-Centered Attribution Methods

• The analysis extracts six attention layers and adapts raw attention, rollout, flow, and Beyond Intuition to sparse refined and coarse attention matrices.Sparse refined attention is reconstructed as a dense matrix with zeros at excluded positions; Beyond Intuition applies value-projection correction and gradient-supported masking only to refined attention, preserving excluded zeros and using a mean-intensity baseline.Source: §3.2 · pp. 5–7

3.3 Faithfulness Evaluation via SaCo

• SaCo measures explanation faithfulness by checking whether masking higher-saliency groups causes greater prediction impact; F=1 indicates perfect faithfulness, F=−1 full inversion, and |F| correlation [25].Source: §3.3 · pp. 7–8, §5.2 · p. 12
• SaCo faithfulness uses the MAE between original and mean-intensity-perturbed predicted rPPG waveforms, where larger MAE indicates stronger model reliance on the removed region.Using the prediction as reference preserves SaCo’s focus on model sensitivity rather than prediction accuracy.Source: §3.3 · p. 8

3.4 Skin Attention Quantification

• Skin coverage measures the fraction of normalized, upsampled attention mass overlapping BiSeNet-derived skin regions, with 1.0 meaning all attention is on skin and 0.0 none.Attention maps are aligned to 128×128 face images before computing the skin-mass ratio.Source: §3.4 · p. 8

3.5 Waveform-Level Signal Quality

• Waveform-level signal quality is measured by per-clip Pearson correlation between predicted and ground-truth rPPG waveforms, distinct from heart-rate-level correlation.The waveform metric evaluates signal reconstruction fidelity rather than agreement between scalar heart-rate estimates.Source: §3.5 · pp. 8–9

3.6 Heart-Rate-Level Error

• Clip-level heart-rate error is the absolute difference between FFT-derived dominant spectral peaks of predicted and ground-truth waveforms, aggregated as HR MAE across clips.For UBFC-rPPG, analysis uses 160 frames at 30 Hz, detrending, 0.75–2.5 Hz bandpass filtering, and Welch power spectral density.Source: §3.6 · p. 9

4.1 Experimental Setup

• The dataset contains 42 approximately one-minute videos at 640 × 480 resolution and 30 fps, with 30 videos for training and 12 for testing.Testing uses non-overlapping 160-frame clips, yielding 141 clips across the 12 test subjects.Source: §4.1 · p. 9
• UBFC-rPPG is used because controlled recording conditions reduce dataset-side confounders and enable direct comparison with RhythmFormer's published results.The evaluation focuses on establishing an XAI framework rather than benchmarking across datasets.Source: §4.1 · p. 9
• Training runs for 30 epochs with batch size 4, learning rate 9 × 10^-3, 160-frame clips, and the original RhythmFormer architecture and hyperparameters.The hybrid loss combines negative Pearson correlation in the time domain with cross-entropy over power spectral densities, alongside temporal resampling and horizontal-flip augmentation.Source: §4.1 · pp. 9–10
• The XAI evaluation extracts attention from all six BRA layers, uses Nsteps=20 Beyond Intuition integration steps, and evaluates SaCo with K=8 spatial groupings using the MAE impact metric.Source: §4.1 · p. 10

4.2 Training Reproduction

• Close agreement with the original paper confirms a faithful reproduction and a reliable checkpoint for subsequent XAI analysis.Small discrepancies are attributed to nondeterministic distributed training and checkpoint selection without an independent validation set.Source: §4.2 · p. 10

5.1 Qualitative Visualization

• All methods highlight facial skin qualitatively, while rollout and Beyond Intuition concentrate more on cheeks and forehead and raw attention and flow spread more broadly.Refined raw attention becomes more skin-concentrated after top-k sparsification, whereas cumulative rollout and flow can restore attribution to positions excluded by individual layers.Source: §5.1 · pp. 10–12

5.2 Skin Coverage Analysis

• Skin coverage increases for refined raw attention by Δ=+0.22, but decreases for refined rollout and flow by Δ=−0.12 and −0.17, respectively.All reported differences are significant at p < 0.001; Beyond Intuition has significantly higher refined-level coverage than all attention-based methods but greater inter-subject variability.Source: §5.2 · p. 12
• On UBFC-rPPG with SaCo faithfulness F, K=8, Beyond Intuition achieves significantly higher refined-level skin coverage than all attention-based methods (p < 0.001) and highest overall faithfulness.Source: §5.2 · p. 12, Table 2 · p. 12, §5.4 · p. 13

5.3 Rollout Reconstruction of Top-k-Masked Connections

• Across 141 clips, six-layer rollout restores every top-k-excluded entry to a non-zero value, with recovered entries contributing approximately 48% of total off-diagonal rollout mass.Within a single TPT level, rollout recovers 98–100% of the first block’s top-k entries; recovered entries have median value 0.0106, only 1.8× smaller than always-retained entries.Source: §5.3 · p. 13

5.5 Clip-Level Correlation Analysis

• Skin coverage correlates significantly with waveform fidelity for rollout (ρ=0.31) and Beyond Intuition (ρ=0.24), but MAE–SaCo correlations are nonsignificant for every method.Coverage–MAE correlations follow the expected negative direction but are weak; on UBFC-rPPG, saturated HR accuracy masks the downstream influence of spatial attention, which may strengthen in noisier datasets.Source: §5.5 · pp. 14–16

5.6 Artifact Ablation Case Study

• The results also include a clip-level case study of one low-SaCo outlier for artifact ablation.The supplied passage identifies the case study as a result-level analysis but does not provide its ablation outcome.Source: §5.6 · p. 16
• The ablated clip contained six eye-rubbing hand-occlusion frames, representing 3.75% of the 160-frame input.The frames were replaced with stable neighboring frames from the same clip before rerunning the full forward pass and XAI pipeline.Source: §5.6 · pp. 16–17
• Across all four attribution methods, both SaCo faithfulness and skin coverage improved after the artifact intervention, with Beyond Intuition showing the largest SaCo gain.Table 3 reports the before-and-after per-method values, using the same BiSeNet skin mask for both conditions.Source: Table 3 · p. 17, §5.6 · p. 17
• BI SaCo rose from F = −0.53 to 0.76 after replacing six hand-occlusion frames, while all four methods recovered more skin-aligned attribution distributions.The intervention changed only the six frames, preserving the 160-frame input length and temporal context.Source: §5.6 · pp. 16–17, Figure 8 · p. 18

Mechanism: how bi-level routing shapes attribution

• 48% of rollout’s off-diagonal mass comes from recovered multi-hop connections that top-k layers had explicitly removed.This leakage lowers refined rollout coverage by 0.12 and produces the same refined-versus-pre-attention drop for flow, by 0.17.Source: Mechanism: how bi-level routing shapes… · p. 18
• Rollout and flow remain comparable to raw attention on SaCo faithfulness—0.85 and 0.84 versus 0.85—despite their off-skin leakage.Their cumulative path-based attributions can still identify regions the model genuinely relies on, even when those regions extend beyond the skin mask.Source: Mechanism: how bi-level routing shapes… · p. 18
• 0.83 median skin coverage is achieved by Beyond Intuition after value-projection correction and gradient masking, improving over vanilla rollout’s 0.57 across all 141 clips.The correction step raises coverage from 0.57 to 0.68, and the integrated-gradient product raises it further to 0.83.Source: Mechanism: how bi-level routing shapes… · pp. 18–19, Table 4 · p. 19

Spatial attention does not uniformly track prediction quality

• Skin coverage and SaCo faithfulness do not uniformly track waveform or heart-rate prediction quality, even when predictions are accurate.The analyses indicate that attribution consistency under perturbation is distinct from predictive accuracy and that equally accurate clips can use different spatial strategies.Source: Spatial attention does not uniformly… · p. 19
• SaCo can remain high for poorly predicted clips or low for accurately predicted clips because it measures reliance alignment rather than prediction correctness.A faithful attribution may identify regions genuinely used by a model whose regions lack strong rPPG signal, while low faithfulness can accompany distributed attention.Source: Spatial attention does not uniformly… · p. 19

Outlier analysis: artifact ablation

• After six hand-artefact frames are replaced, all four attribution methods recover toward F ≈0.8 in the illustrative outlier.Their agreement in the direction of change suggests consistent SaCo behavior across attribution families, while skin coverage is less responsive.Source: Outlier analysis: artifact ablation · p. 20
• The outlier ablation is illustrative rather than confirmatory because one clip cannot establish general attention-impact alignment.A systematic multi-outlier study would be needed to support the stronger claim.Source: Outlier analysis: artifact ablation · pp. 19–20
• The evaluation is limited to 12 UBFC-rPPG subjects under controlled conditions and RhythmFormer, so generalization of the leakage pattern remains open.UBFC’s near-zero heart-rate error and simple conditions cannot test spatial metrics in challenging regimes; cross-model replication is required.Source: Outlier analysis: artifact ablation · p. 20

Framework

• The framework quantitatively compares four representative XAI attribution methods for transformer-based rPPG models using skin coverage and SaCo faithfulness.Beyond Intuition is extended from classification to regression by using MAE between original and perturbed predicted rPPG waveforms as the SaCo impact signal.Source: §7 · pp. 20–21

Physiological validation

• Skin coverage shows a moderate positive correlation with waveform-level Pearson correlation and a weak negative trend with HR MAE, although both relationships are modest.The weaker MAE relationship may reflect UBFC-rPPG’s saturated heart-rate regime.Source: §7 · p. 21

Case study

• An artefact-ablation case study found SaCo recovering in the same direction across all four methods after offending frames were replaced.The result is suggestive for this clip but cannot rule out method-specific biases generally.Source: §7 · p. 21

Multi-hop leakage

• Under RhythmFormer’s top-k bi-level routing, multi-hop paths refill entries that individual attention layers explicitly zero out, challenging dense-attention assumptions.Positions filtered at one or more layers still accumulate roughly half of rollout’s off-diagonal attribution mass, and attention flow has the same path-based caveat.Source: §7 · p. 21

Results

• Beyond Intuition achieves the highest median skin coverage and SaCo among the four methods, while remaining robust to multi-hop leakage on UBFC-rPPG.Its higher inter-clip variability and tail risk mean rollout or flow may remain preferable when stable per-clip behavior matters.Source: §7 · p. 21

Limitations

• The conclusions remain bounded by UBFC-rPPG and require testing on more datasets, models, motion artefacts, and variable illumination conditions.The authors specifically identify the saturated HR regime as limiting interpretation of the MAE–skin-coverage relationship.Source: §7 · p. 21

• Claude was used for grammar and spelling checks.Source: Declaration on Generative AI · p. 22
• The authors also used Claude to generate code for figure-plotting scripts and assist with LaTeX formatting.Source: Declaration on Generative AI · p. 22
• The authors reviewed and edited the AI-assisted content and accepted full responsibility for the publication's content.Source: Declaration on Generative AI · p. 22