Cross-Dataset Transfer and Reliability of Explainable Artificial Intelligence for RhythmFormer Remote Photoplethysmography

Motivation

• rPPG estimates cardiac pulse from facial video, but weak skin-colour changes are confounded by illumination, camera response, and motion.Public datasets also vary in recording conditions and participant composition [2] [3] [5].Source: §1 · pp. 1–2

Dataset scope

• Dataset adaptation is central to evaluation because models trained on one dataset may not generalise to another, even when datasets appear similar [5] [6].The study therefore uses NCKU-rPPG’s eight varied scenarios alongside UBFC-rPPG.Source: §1 · p. 2

Study aim

• XAI methods are evaluated for whether their explanations provide insight into rPPG behaviour, transfer across datasets, and track model reliability.The study focuses on explanation faithfulness and reliability alongside model predictions.Source: §1 · p. 2

Experimental plan

• The experiments compare XAI methods on NCKU-rPPG Static level 3 and UBFC-rPPG, then relate median SaCo across eight scenarios to error, waveform correlation, and signal-to-noise ratio.Participant-independent splits and eight independently trained checkpoints support the comparisons.Source: §1 · p. 2

Scope and limitation

• The design supports descriptive transfer and reliability evidence, not claims of statistical significance, causality, or cross-dataset generalisation.The comparisons use conversion, participant-independent splitting, and independently trained checkpoints as their experimental basis.Source: §1 · p. 2

Dataset selection

• Complete facial video and synchronized waveforms made UBFC-rPPG the practical public reference, while NCKU-rPPG supplied the controlled multi-condition extension.Other datasets were limited by access agreements, unavailable downloads, missing original facial videos, synthetic participants, or unavailable acquisition routes.Source: §2 · pp. 2–3

XAI in rPPG

• Visual rPPG XAI studies report attention, gradient, Grad-CAM, or channel–spatial maps, but plausibility does not establish causal predictive relevance.Existing work does not quantitatively test whether attribution ordering agrees with spatial perturbation effects.Source: §2 · p. 3

Prior quantitative evaluation

• Beyond Intuition previously achieved the strongest median skin alignment and SaCo in a controlled UBFC-rPPG RhythmFormer evaluation, but only within one dataset and setting.Attention aggregation could restore excluded multi-hop paths, weakening rollout’s spatial specificity; the result motivates replication rather than a general method ranking.Source: §2 · p. 3

Study extension

• The present study extends this framework to eight independently trained RhythmFormer models across NCKU-rPPG illumination, speaking, head-rotation, and cycling scenarios.It tests whether observations transfer across recording conditions and models.Source: §2 · p. 3
• The analysis separates SaCo’s within-clip perturbation-faithfulness meaning from its unestablished association with reliability across independently trained models.Cross-scenario comparisons also examine whether previously observed multi-hop leakage persists under varied recording conditions.Source: §2 · p. 3

3.1 Dataset and experimental conditions

• The NCKU-rPPG source cohort contained 618 Camera 1 sessions from 78 participants, with usable sessions ranging from 75 to 77 across scenarios.Each participant contributed at most one session per scenario, so usable-participant and usable-session counts were identical.Source: §3.1 · pp. 3–4
• The facial-video recordings were paired with contact photoplethysmography and electrocardiography under controlled illumination and motion conditions.The analysis followed the experimental protocol established in the Nordling Lab and presented by Wang.Source: §3.1 · p. 4

3.1.1 Ethical statement

• The National Cheng Kung University Human Research Ethics Committee approved the experimental procedures under case numbers 106-262, 108-244, and 112-406.Data collection was conducted in accordance with the approved procedures and corresponding approval numbers.Source: §3.1.1 · p. 4

3.1.2 Acquisition system

• Camera 1 captured frontal facial video at 1920×1080 pixels and 50 frames per second, while ECG and finger PPG were recorded at 500 Hz and 125 Hz.Alignment files paired selected video frames with PPG measurements at the source video’s 50 Hz cadence.Source: §3.1.2 · p. 4
• The acquisition system used a Panasonic GX85 with an Olympus 14–150-mm lens, synchronization lighting cues, and a Philips IntelliVue MX400 for physiological references.Two FOTGA LED504 lights and a fluorescent ceiling light established illumination, checked beside the participant’s face with a lux meter.Source: §3.1.2 · p. 4

3.1.3 Experiment design

• Eight scenarios varied illumination, speaking, head rotation, and cycling to isolate local facial motion, pose changes, body motion, and changing heart rate.Static recordings used levels 1, 3, and 5; speaking and rotation used level 3; cycling repeated all three illumination levels.Source: §3.1.3 · p. 4
• The protocol comprised five two-minute seated recordings and three six-minute cycling recordings conducted under researcher-supervised cadence and resistance changes.Illumination targets were 40±50, 200±50, and 700 ± 50 lux for levels 1, 3, and 5, respectively.Source: §3.1.3 · p. 4

Alignment and cadence

• The converter retained the shorter decodable video or contiguous finite PPG sequence from each aligned start frame, preserving the original 50 Hz cadence.It required all expected source frames to be decodable and did not resample recordings to UBFC-rPPG’s 30 Hz rate.Source: §3.2 · p. 7

Resizing

• Each shifted square crop was extracted from the rotated source frame and resized from approximately 510 × 640 to 128 × 128 pixels.Participant-specific 90-degree corrections produced portrait-oriented frames before detection and extraction.Source: §3.2 · pp. 7–8

Face detection and cropping

• MTCNN selected the highest-confidence finite face candidate per frame, while FaceXFormer [25] estimated 68 landmarks to define each square crop.The crop was centered horizontally on landmark extrema and shifted upward by 8% of its side to include more forehead.Source: §3.2 · p. 7

Crop stabilization

• Crop centers and side lengths were interpolated and smoothed with five-frame median and 11-frame second-order Savitzky–Golay filters, requiring at least 0.80 raw detection.Leading and trailing gaps used nearest valid values, with shorter-recording adjustments to filter windows.Source: §3.2 · pp. 7–8

Output validation

• 607 accepted sessions met the minimum encode–decode temporal RGB correlation of 0.99 between source frames and decoded FFV1 outputs.Outputs below the threshold were deleted and processing stopped for that session.Source: §3.2 · p. 8

Participant split

• The converted dataset contained 607 sessions from 77 participants after one participant’s eight sessions were skipped because alignment CSV files were absent.Source: §3.3 · p. 8
• Participant-level splitting assigned 65 participants and 511 sessions to training and 12 participants and 96 sessions to testing before scenario filtering.Each scenario retained 63–65 training participants and all 12 test participants; no participant or session crossed partitions in any scenario.Source: §3.3 · p. 8

RhythmFormer preprocessing

• The loader read pre-cropped 128×128 videos without additional detection, tracking, expanded cropping, or resolution changes, and standardized video and PPG before segmentation.Video standardization used joint time–height–width–channel statistics, while PPG standardization used temporal statistics.Source: §3.3 · p. 8
• Each session was divided into non-overlapping 256-frame, 5.12-second clips, with terminal remainders discarded and clips with non-finite or near-zero-variance labels excluded.The clips approximated but did not exactly match the 5.33-second UBFC-rPPG training clips, and time was not resampled.Source: §3.3 · p. 8

• RhythmFormer maps RGB facial video to a one-dimensional rPPG waveform [9] using a Fusion Stem that combines raw frames and temporal frame differences.Source: §3.4 · p. 9
• Three hierarchical TPT stages contain two PSA blocks each, with stage-specific temporal downsampling and upsampling exposing periodic structure at multiple temporal scales.Source: §3.4 · p. 9
• After six PSA blocks, spatial averaging and a one-dimensional predictor produce one waveform value per input frame.Source: §3.4 · p. 9
• PSA uses coarse scores to route selected keys and values before refined attention, alongside a parallel local-context branch.Source: Figure 2 · p. 9
• For 256-frame, 50 Hz experiments, the model used top-k = 40 routing and no stochastic depth, with drop path rate 0.Source: §3.4 · p. 9

Scenario-specific training

• Eight independent models used only clips from their own scenario for training and were tested on corresponding held-out scenario clips.The shared training and inference configuration is reported in Table 3.Source: §3.5 · p. 9, Table 3 · p. 10
• The objective combined negative-Pearson waveform loss, frequency-domain heart-rate cross-entropy, and KL divergence between predicted and reference heart-rate distributions.These three components follow the published RhythmFormer objective.Source: §3.5 · p. 10
• Heart-rate-aware temporal augmentation was applied independently with probability 0.5, accelerating clips below 75 beats per minute and expanding intervals above 90 beats per minute.Clips from 75 to 90 beats per minute inclusive were left temporally unchanged.Source: §3.5 · p. 10
• Augmented labels with non-finite values or standard deviation no greater than 10^-8 were restored to their original video-label pairs to avoid undefined Pearson loss.The safeguard did not redraw augmentation or flip decisions, preserving the random sampling sequence.Source: §3.5 · p. 10
• Formal runs used AdamW with ϵ = 10^-8 and zero weight decay, without validation-loader model selection, and tested the final training epoch.No model was selected by validation performance.Source: §3.5 · p. 10

• The evaluation follows rPPG-Toolbox definitions for MAE, RMSE, MAPE, Pearson correlation, and SNR because the experiment and published RhythmFormer comparison use that convention [9].Estimates are participant-weighted, with clips averaged within each participant before participant means are averaged.Source: §3.6 · pp. 11–12
• One heart rate is estimated per 256-frame, 5.12-second clip without recording-level averaging, enabling comparison with the UBFC-rPPG reproduction.The choice addresses changing heart rates during cycling and matches UBFC-rPPG’s approximately 5.33-second clips.Source: §3.6 · p. 11
• 0.65–1.09 beats per minute is the median absolute difference between spectral-peak and systolic-peak reference estimates across stationary conditions, versus 1.81–2.47 during cycling.Cycling also produced mean absolute differences of 5.49–8.54 beats per minute, with 30 of 2,559 clips censored at the 150-beats-per-minute boundary.Source: §3.6 · p. 11
• Participant-specific lag fitting raised UBFC-rPPG’s participant-averaged waveform correlation from 0.785 to 0.828, while leaving heart-rate and SNR measures unchanged.The fitted lag measures waveform agreement rather than synchronization error and was two to seven times its null on NCKU-rPPG.Source: §3.6 · pp. 12–13
• 75.2 % of NCKU-rPPG clips have reference rates above 75 beats per minute, making the SNR’s second-harmonic term fall outside the applied bandpass in most clips.Because 256-frame periodograms resolve 11.7 beats per minute, the reported SNR is conservative rather than neutral at this clip length.Source: §3.6 · p. 13

Uncertainty

• Standard errors were computed over participant means, because clip-level independence would understate uncertainty by roughly the square root of clips per participant.This uncertainty represents between-participant variation for one trained seed, not numerical error or retraining variability.Source: §3.6 · p. 14

UBFC-rPPG reproduction

• UBFC-rPPG metrics were recomputed from archived waveforms using the identical estimator, and recomputed versus archived per-clip errors agreed within 0.03 beats per minute.The archived evidence supplied heart-rate error and waveform correlation but not reference heart rate or SNR, so those values were generated rather than read from the archive.Source: §3.6 · p. 14

Cross-dataset comparability

• UBFC-rPPG and NCKU-rPPG clips lasted 5.33 and 5.12 seconds, respectively, producing 4% different spectral resolving power and unavoidable Welch and padding differences.Matching either transform parameter would exceed the applicable cap or violate the required power-of-two padding.Source: §3.6 · p. 14
• About 1 decibel of padding difference favors UBFC-rPPG, so its reported SNR advantage is likely understated; heart-rate columns are unaffected.UBFC-rPPG padding differs from NCKU-rPPG because of clip-dependent periodogram lengths, whereas heart-rate estimates depend only on peak location.Source: §3.6 · p. 14

3.7 Explainable artificial intelligence attribution methods

• Raw attention, rollout, and attention flow collapsed temporal query and key axes before aggregation, producing 4 × 4 or 8 × 8 spatial maps without temporal rollout.Pre-attention maps had 4 × 4 spatial regions, whereas refined attention maps had 8 × 8 spatial tokens.Source: §3.7 · p. 15
• Four attribution methods were extracted from RhythmFormer’s PSA blocks: raw attention, attention rollout, attention flow, and Beyond Intuition.Raw attention used layer-local received attention; rollout multiplied residual-augmented matrices; attention flow computed maximum flow through a layered graph.Source: §3.7 · p. 15
• Beyond Intuition combined value-projection-corrected rollout with an integrated-gradient reasoning-feedback mask and retained T = 20 temporal structure until after attribution integration.Its integrated gradients used 20 steps from a standardised-space zero baseline corresponding to each clip’s mean-intensity state.Source: §3.7 · pp. 15–16
• Rotation attributions were recomputed in FP32 because mixed precision produced non-finite values, without changing the checkpoint, parameters, or training procedure.The change affected attribution generation only.Source: §3.7 · p. 16

Anatomical plausibility

• Skin coverage quantified anatomical plausibility using a BiSeNet face-skin and nose mask after attribution normalisation and upsampling to 128 × 128 pixels.It was explicitly not treated as a faithfulness measure.Source: §3.8 · p. 16

Perturbation faithfulness

• SaCo quantified perturbation faithfulness by ranking spatial positions by salience, dividing them into K = 8 equal groups, and masking each group independently.Masked pixels were replaced with zero in standardised space, equivalent to the per-clip per-channel mean intensity before standardisation.Source: §3.8 · p. 16
• SaCo compared each group’s summed salience with its perturbation impact, defined as waveform mean absolute error between original and masked clips.Beyond Intuition used an 8 × 8 refined attribution map, whereas the other methods used 4 × 4 source maps.Source: §3.8 · p. 16

Visualisation and aggregation

• Scenario-level visualisations averaged test clips, whereas violin plots averaged each clip across TPT1–TPT3 and scatter plots retained clip-level observations.The combined violin plots displayed UBFC-rPPG alongside eight NCKU-rPPG scenarios, while scatter plots used refined TPT3 coverage for attention-only methods.Source: §3.8 · pp. 16–17

3.9 Cross-dataset transfer and scenario-level reliability comparisons

• Cross-dataset comparisons retained differing participants, cameras, preprocessing, frame rates, and checkpoints, so they were not treated as controlled experiments.Static level 3 was compared with the published UBFC-rPPG evaluation of Chen and Nordling [22] because heart-rate accuracy was similar.Source: §3.9 · p. 17
• Within-participant Spearman correlations related skin coverage and SaCo to heart-rate MAE, waveform correlation, and SNR, with coefficients averaged through the Fisher transform.The analysis also included the relationship between SaCo and skin coverage, producing seven relationships per attribution method across eight NCKU-rPPG scenarios and UBFC-rPPG.Source: §3.9 · p. 17
• Pooling clips produced a spurious Static level 3 waveform Pearson–skin coverage coefficient of −0.41 with p < 10−12, whereas within-participant coefficients ranged from −0.16 to +0.13 and were unresolved.The within-participant estimator avoided measuring mostly between-person differences in heart rate, skin tone, and facial geometry.Source: §3.9 · p. 17
• At the scenario level, eight independently trained NCKU-rPPG conditions were summarized by median SaCo and related descriptively to MAE, waveform Pearson correlation, and SNR.Expected directions were negative with MAE and positive with waveform correlation and SNR, but no significance tests or causal interpretation were made.Source: §3.9 · p. 18

4.1 Performance across illumination and motion

• Static level 3 was the best static condition, with the lowest MAE, RMSE, and MAPE and the highest heart-rate correlation, waveform correlation, and SNR.Static level 1 ranked last and was the only static condition with negative mean SNR; level 5 ranked second on these measures.Source: §4.1 · p. 18
• Rotation degraded performance more than speaking, with heart-rate error about 1.4 times as large and SNR a further 6.3 dB lower.Both conditions were worse than the static level 3 model trained and tested at the same nominal illumination level.Source: §4.1 · p. 18
• Cycling was weakest across all eight NCKU-rPPG conditions, with mean absolute errors of 15 to 19 beats per minute and mean SNR from −8.9 to −11.9 dB.Bike level 5 was nominally best across six measures, while the worst cycling condition depended on the measure.Source: §4.1 · p. 18
• Across scenario-level values, MAE correlated with heart-rate correlation at ρ = −0.98 (p < .001), waveform correlation at ρ = −0.95 (p < .001), and SNR at ρ = −0.88 (p = .004).Waveform correlation also correlated with SNR at ρ = +0.98 (p < .001), but the two correlations were not interchangeable.Source: §4.1 · p. 19

4.2 Attribution summaries for Static level 3 and UBFC-rPPG

• Refinement was inert for attention-only skin coverage on Static level 3 but changed UBFC-rPPG coverage by 0.12 to 0.22 and reordered the methods.Across all nine conditions, refinement raised raw-attention coverage, lowered attention-flow coverage, and lowered attention-rollout coverage in eight conditions.Source: §4.2 · p. 20
• Within participants, neither skin coverage nor SaCo predicted how well the model performed on a given clip in Static level 3 or UBFC-rPPG.Only five of 28 relationships were directional and significant, while the remaining 23 were unresolved; examples included raw-attention MAE–coverage at ρ = −0.14 (p = .020) and Beyond Intuition SNR–SaCo at ρ = +0.19 (p = .044).Source: §4.2 · p. 20

4.3 Attribution summaries across the nine conditions

• Beyond Intuition had the highest median refined skin coverage in seven of nine conditions and the highest median SaCo in four clean-recording conditions.It ranked highest on both measures for Static level 3, Static level 5, Speak, and UBFC-rPPG, where it attributed more inside skin and ordered clips more faithfully than attention-only methods.Source: §4.3 · p. 23
• Beyond Intuition failed under the dimmest illumination: on Static level 1, 117 of 200 clips had negative SaCo and the median was −0.178.Negative SaCo otherwise occurred mainly in minority outlier clips, affecting 34 of 36 condition-method pairs; Bike level 1 also had 41% of clips below zero.Source: §4.3 · p. 23
• 28% of 40-lux SaCo values fell below zero, compared with 0.6% at 200 lux and 5.6% at 700 lux.The 40-lux condition had the lowest median refined coverage in seven of eight method–triple pairs and the lowest median SaCo in six.Source: §4.3 · p. 23
• Raw attention and attention rollout had SaCo correlation ρ = 0.97 but refined-coverage correlation ρ = 0.18, showing that similar attribution placement can yield different ordering faithfulness.Attention rollout and attention flow instead agreed at 0.96 in coverage and 0.56 in SaCo; Beyond Intuition was least similar in SaCo.Source: §4.3 · p. 23
• 25 of 252 within-condition correlations were directional, while 28 reached p < 0.05 against 13 expected by chance.No relationship was directional in more than six of 36 method–condition pairs, and four of seven were directional in fewer than four.Source: §4.3 · p. 24
• Within-participant correlations were near zero, whereas pooled coefficients reached 0.71 and between-participant coefficients reached 0.77, indicating participant differences dominated pooled associations.For waveform Pearson–skin coverage, the pooled coefficient was −0.41 but ranged from −0.16 to +0.13 within participants.Source: §4.3 · p. 27

4.4 Scenario-level SaCo and model reliability

• −0.43, +0.57, and +0.43 were Beyond Intuition’s skin-coverage correlations with MAE, waveform correlation, and SNR across scenarios, respectively.The three attention-only methods stayed within |ρ| = 0.21 on those relationships.Source: §4.4 · p. 28
• Attention-only SaCo moved opposite to reliability, rising with MAE at +0.31 to +0.33 and falling with waveform correlation and SNR at −0.17 to −0.50.Beyond Intuition’s SaCo coefficients stayed within 0.10 of zero, and the scenario comparison did not support SaCo as a consistent reliability tracker.Source: §4.4 · p. 28
• SaCo and skin coverage correlated positively for every method except raw attention, at +0.48 to +0.50 versus −0.33.This cross-method relationship did not establish SaCo as a reliability measure across the eight scenarios.Source: §4.4 · p. 28

5.1 Principal findings

• Within a single participant and condition, attribution measures were not related to heart-rate error, waveform correlation, or SNR; 186 of 252 coefficients had |ρ| below 0.10.These explainable artificial intelligence measures therefore carry information complementary to the three performance measures rather than serving as proxies for them.Source: §5.1 · p. 29
• Beyond Intuition was strongest overall, supplying 14 of 25 directional relationships, highest median refined skin coverage in seven of nine conditions, and highest median SaCo in four.Across scenarios, its coverage fell with heart-rate error at ρ = −0.43 and rose with waveform correlation and SNR at +0.57 and +0.43, whereas attention-only SaCo showed reverse associations.Source: §5.1 · p. 29

5.2 Model performance across NCKU-rPPG

• NCKU-rPPG performance was non-monotonic across static illumination, while motion substantially reduced robustness: cycling heart-rate error fell from 18.6 ± 5.5 to 16.1 ± 2.3 to 15.1 ± 3.5 beats per minute from 40 to 200 to 700 lux.Speaking and rotation reduced correlations and SNRs relative to Static level 3, and all cycling conditions had lower correlations and negative SNRs than static levels 3 and 5.Source: §5.2 · p. 30

5.3 Static level 3 versus UBFC-rPPG

• UBFC-rPPG was ahead of Static level 3 on every reported benchmark column, including heart-rate correlation 0.991 ± 0.041 versus 0.968 ± 0.079 and waveform correlation 0.828 ± 0.022 versus 0.626 ± 0.037.The waveform comparison depended on per-recording synchronization: Static level 3 was 0.28 at zero lag but 0.626 ± 0.037 after fitting a lag.Source: §5.3 · p. 31
• UBFC-rPPG had half the heart-rate error and higher SNR than Static level 3, but the evaluations remain descriptive because recordings, preprocessing, activities, checkpoints, and splits were uncontrolled.The reported SNR difference was 2.8 dB as a lower bound and about 4 dB after matching periodogram resolution; the waveform comparison removed recording lag.Source: §5.3 · p. 31

5.4 Static level 3 XAI versus UBFC-rPPG

• Beyond Intuition ranked highest on both evaluations, with median coverage 0.789 versus 0.826 and median SaCo 0.837 versus 0.917 on Static level 3 and UBFC-rPPG, respectively (Table 8) [22].Secondary rankings differed, so the repeated top rank is descriptive rather than proof that the full XAI ranking transfers.Source: §5.4 · p. 31
• SaCo showed substantial within-dataset heterogeneity: Static level 3 interquartile ranges were 0.228–0.280 with values 0.017–1.000, versus UBFC-rPPG ranges 0.135–0.201 and a −0.534 outlier (Figure 5).Replacing six hand-contaminated frames moved Beyond Intuition from −0.53 to 0.76 and brought all methods toward F ≈ 0.8, indicating clip-level sensitivity.Source: §5.4 · p. 32
• Within participants, neither skin coverage nor SaCo predicted clip-level performance: coverage, SaCo, heart-rate error, waveform correlation, and SNR moved independently.At scenario level, the eight-point analysis supports directions and magnitudes rather than established relationships because p < 0.05 occurs only at |ρ| = 0.74.Source: §5.4 · p. 33, §5.5 · p. 33

5.5 Do skin coverage and SaCo track reliability across scenarios?

• Beyond Intuition’s coverage tracked scenario-level reliability at ρ = −0.43 for heart-rate error and +0.57 and +0.43 for waveform correlation and SNR, respectively.Attention-only coverage stayed within |ρ| = 0.21 of zero, while their SaCo correlations reversed the predicted directions.Source: §5.5 · p. 33
• SaCo moved opposite to reliability for attention-only methods: +0.31 to +0.33 with error, −0.17 to −0.24 with waveform correlation, and −0.40 to −0.50 with SNR.SaCo remains a within-clip perturbation-faithfulness measure, not a surrogate for prediction accuracy or waveform quality across models.Source: §5.5 · p. 33
• Attribution location carries scenario-level information about model performance for a method whose maps are sufficiently concentrated, whereas attribution ordering does not; at 40 lux, median skin coverage and median SaCo are lowest, with 28% of SaCo values below zero.Source: §5.5 · p. 34

5.6 The evaluation window, and what the literature compares

• 3.34 beats per minute was the minimum combined window error at 7.68 s, versus 3.60 at 5.12 s, making the adopted clip 7% worse.On Static level 3, the spectral component fell from 2.62 to 0.84 beats per minute as windows grew from 5.12 to 61.44 s, while instantaneous-reference error rose from 3.42 to 4.27.Source: §5.6 · p. 34
• Cycling shifted the optimal window to 8.96–10.24 s, with errors three to five times higher and varying by no more than 17% across a twelvefold length range.The limitation was attributed to the reference rather than the window, because no window performed well under cycling.Source: §5.6 · p. 34
• 0.23 to 1.63 beats per minute was the change in mean absolute error across eight conditions when switching between the two commonest literature window choices.Only nine of 58 reviewed articles reported the transform length or window function, so error comparisons require the window, padding, and frame rate.Source: §5.6 · pp. 34–35

5.7 Reproducibility, limitations, and future work

• 0.67 beats per minute was the UBFC-rPPG median clip error, compared with 1.14 on the most favourable NCKU-rPPG condition, but the study still covered only two datasets.NCKU-rPPG deliberately varied illumination, speaking, head rotation, and cycling, whereas UBFC-rPPG used stationary, evenly lit subjects.Source: §5.7 · p. 35
• Training used 30 epochs once, a single random seed, separate models per condition, and final-epoch checkpoints without validation selection, so reported values estimate generalisation error rather than optimally tuned performance.Source: §5.7 · p. 35
• 601 of 3905 consecutive clip pairs changed by more than 10 beats per minute in the reference, making part of reported error reference movement rather than model error.This occurred in 2.5% of Static level 5 pairs versus 23.8% of Bike level 5 pairs; 30 of 2559 cycling test clips were also censored at the search-range ceiling.Source: §5.7 · pp. 35–36
• Using a 61.44 s window lowers mean absolute error in every condition—by 1.4 beats per minute on Static level 3 and 6.6 on Bike level 1—indicating larger errors with 5.12 s.Source: §5.7 · p. 36
• Skin coverage depends on the face parser and mask definition, while attribution methods differ in map resolution and temporal or cross-layer aggregation.Future evaluation should compare SaCo at a common spatial resolution and test mask and ranking stability across preprocessing choices.Source: §5.7 · p. 36
• Different clip counts and checkpoints mean ranking and correlation differences should not be treated as controlled dataset effects.Static level 3 summaries also used different experimental settings across datasets.Source: §5.7 · p. 36
• Future studies should repeat training across seeds, compare joint-condition with condition-specific models, recover matched participant identities, and test architectures beyond RhythmFormer.The authors identify broader model structures as the change most likely to extend the observations’ reach.Source: §5.7 · p. 36
• Matched preprocessing across NCKU-rPPG and UBFC-rPPG is needed for a direct cross-dataset transfer test, alongside direct evaluation of leakage and the multi-hop path mechanism.Additional experiments should test whether the multi-hop mechanism is consistent with the mechanism observed on UBFC-rPPG.Source: §5.7 · p. 36
• Future work should strengthen skin-mask and face-tracking pipelines with FaceXFormer and determine whether unreliable XAI results reflect training quality rather than limited interpretive value.Analyses of additional scenarios and integrated model-performance, waveform-quality, and XAI metrics may clarify their relationships.Source: §5.7 · p. 36

• 186 of 252 within-participant, within-condition coefficients had |ρ| < 0.10, showing that skin coverage and SaCo were complementary to performance rather than performance proxies.Twenty-eight coefficients reached p < 0.05, compared with 13 expected by chance.Source: §6 · p. 37
• Beyond Intuition supplied 14 of 25 directional relationships, had the highest median refined skin coverage in seven of nine conditions, and uniquely tracked all three performance measures.Its coverage followed heart-rate error, waveform correlation, and signal-to-noise ratio across eight scenarios.Source: §6 · p. 37
• Attribution methods were neither interchangeable nor independent: two methods ordered clips almost identically while agreeing on attribution locations at ρ = 0.18.The three attention-only methods warrant caution because their SaCo ran opposite to each performance measure.Source: §6 · p. 37
• Beyond Intuition failed only under the dimmest illumination, where median coverage fell to 0.180 and median SaCo to −0.178, whereas motion degraded estimates without this drop.Thus, attribution primarily reveals where the model looks, not the faithfulness of map ordering or model performance.Source: §6 · p. 37
• On NCKU-rPPG, static illumination rankings were non-monotonic, cycling rankings were monotonic on four of six measures, and no measure separated the three cycling conditions at clip level.Static level 3 combined heart-rate correlation 0.968 ± 0.079 with waveform correlation 0.626 ± 0.037, showing that neither illumination labels nor heart-rate error alone characterise robustness.Source: §6 · p. 37
• The conclusions are descriptive: one architecture, one seed, one final-epoch checkpoint, and short 5.12-second windows limit generalisation, motivating repeated seeds, more architectures, common-resolution evaluation, and controlled cross-dataset tests.Causal claims about illumination, generalisation, or clinical validity require these further experiments.Source: §6 · p. 37

Ethical approval

• The National Cheng Kung University Human Research Ethics Committee approved the procedures, and every participant gave written informed consent.Source: Declarations · p. 37

Generative artificial intelligence

• Claude was used for grammar, code generation, figure plotting, and LaTeX formatting, after which the authors reviewed and edited the content and retained responsibility.Source: Declarations · p. 38

Data availability

• UBFC-rPPG is public, whereas NCKU-rPPG is not publicly available and may be obtained from the corresponding author on reasonable request subject to approvals.Consented NCKU-rPPG recordings are planned for publication in a later article.Source: Declarations · p. 38

Code availability

• Conversion, training, evaluation, and attribution code is not publicly released but can be made available by the corresponding author on reasonable request.Source: Declarations · p. 38

Funding and competing interests

• The work was funded by Taiwan’s National Science and Technology Council, and the authors declared no known competing financial interests or personal relationships.Source: Declarations · p. 38

S1. Conversion contract and quality control

• The production conversion contract fixed source revision, preprocessing, FaceXFormer commit, checkpoint digest, codec, quality thresholds, identifiers, and expected skips before conversion.Nine sessions lacked alignment CSV files; one failed the raw detection-rate gate and one source video was corrupt.Source: S1. Conversion contract and quality… · p. 46

S2. Participant and session split

• 77 participants and 607 converted sessions formed the cohort, split into 65 participants and 511 sessions for training and 12 participants and 96 sessions for testing.The cohort had 77 participants because one participant had every session skipped; participant and session counts are equal within each scenario.Source: S2. Participant and session split · p. 46

S3. Clip retention

• 25,848 complete raw clips yielded 25,483 retained clips and 365 exclusions across eight scenarios.Exclusions required a non-finite or near-constant ground-truth PPG label; model error, predicted heart rate, correlation, and SNR did not determine inclusion.Source: S3. Clip retention · p. 47

S4. Low-variance label checks

• The 10^-6 low-variance gate excluded 285 training clips and 80 test clips before training, while augmentation fallback restored invalid random temporal draws without removing clips.The gate operated before training; fallback acted only on augmented labels and never removed a clip from the sample.Source: S4. Low-variance label checks · p. 49

S5. Training configuration

• All eight training runs shared the same code revision, participant manifest, and training settings, differing only in scenario, run identifier, cache, file-list, and output paths.A non-finite or low-variance augmented label restored the original video–label pair at a 10^-8 threshold.Source: S5. Training configuration · p. 50

S6. Explainable artificial intelligence results

• rHR = 0.968 ± 0.079 on Static level 3, yet its median clip had only loose waveform tracking, while Bike level 1 tracked hardly at all.Static level 3 had 1.14 beats per minute error and +3.06 dB SNR; Bike level 1 had 5.52 beats per minute error and −10.29 dB SNR.Source: S6. Explainable artificial… · p. 50
• At 61.44 s, mean absolute error decreases by 1.4 beats per minute on Static level 3 and 6.6 on Bike level 1, yet motion remains worse than static.The weakest static condition reaches 4.06 beats per minute versus 6.63 for the strongest motion condition, and motion-condition ordering changes with window length.Source: S6. Explainable artificial… · p. 51
• The reference photoplethysmogram disagrees with itself by a median of 0.65–1.09 beats per minute when stationary and a mean of 5.49–8.54 when cycling.Source: S6. Explainable artificial… · p. 54
• 290–700 milliseconds of fitted synchronization lag occurs across NCKU-rPPG conditions versus 0 milliseconds on UBFC-rPPG.The NCKU-rPPG correlation gain exceeds its mismatched-participant null by two to seven times, whereas UBFC-rPPG’s +0.043 gain is below its +0.110 null.Source: S6. Explainable artificial… · p. 54, Table 20 · p. 55
• Recomputing heart rate from evaluation-run waveforms shifts medians by 0.23 beats per minute on Static level 3 and 0.44 on Bike level 1, with larger cycling discrepancies.Single-clip shifts reach 17 and 67 beats per minute, while signal-to-noise-ratio medians move by 1.2 and 1.7 decibels.Source: S6. Explainable artificial… · p. 55
• 7.68 s minimizes Static level 3 error at 3.34 beats per minute, versus 3.60 at 5.12 s, because estimation error falls while representation error rises.The supplementary decomposition treats 7.68 seconds as a lower bound rather than a precise optimum, especially under cycling where the reference is motion-corrupted.Source: S6. Explainable artificial… · pp. 57–58
• The reported raw-attribution correlations are near zero across performance measures and SaCo–skin, with coefficients ranging from −0.07 to +0.10.These values are drawn from the participant-averaged Spearman summaries for the listed raw, rollout, and flow methods.Source: S6. Explainable artificial… · pp. 58–60, Table 22 · p. 59
• 25 of 252 coefficients were directional, with the largest reaching ρ = +0.32, so neither skin coverage nor SaCo predicts clip-level performance within participants.Beyond Intuition supplied 14 of the 25 directional coefficients, versus 3, 4, and 4 for the three attention-only methods.Source: S6. Explainable artificial… · p. 61
• Refinement raises raw-attention skin coverage in every NCKU-rPPG scenario, lowers attention-flow coverage in every scenario, and lowers rollout coverage in seven of eight, with the same directions on UBFC-rPPG.The NCKU-rPPG changes range from +0.004 to +0.141 for raw attention, −0.005 to −0.070 for attention flow, and the UBFC-rPPG changes are +0.224, −0.117, and −0.167.Source: S6. Explainable artificial… · p. 69
• Transformer depth affects coverage at opposite stages: pre-attention skin coverage rises with level for rollout in seven of eight scenarios and flow in five, whereas raw attention lacks a consistent pre-attention trend.Rollout and flow rise by as much as 0.30 from TPT1 to TPT3 on Bike level 1, while refinement compresses that gradient to at most 0.12; raw attention rises in four scenarios and falls in four before refinement.Source: S6. Explainable artificial… · p. 71
• At Static level 1, refinement is not resolved for any attention-only method; Figure 11 separates four methods sharply, with Beyond Intuition the only method having median SaCo below zero.Source: S6. Explainable artificial… · p. 72
• At Static level 5, refinement increases raw-attention coverage by 0.141 ± 0.019, decreases rollout by 0.026 ± 0.008, and decreases flow by 0.047 ± 0.010.Across methods, median SaCo spans 0.416–0.708 and median refined skin coverage spans 0.435–0.779.Source: S6. Explainable artificial… · p. 74
• Speaking has the highest median SaCo range, 0.824–0.863, despite heart-rate error of 8.9 ± 2.6 beats per minute being the fifth worst across eight conditions.Refinement raises raw-attention coverage by 0.086 ± 0.026, lowers flow by 0.066 ± 0.015, and leaves rollout’s −0.019 ± 0.019 change unresolved.Source: S6. Explainable artificial… · p. 76
• The supplementary distributions summarize clip-level SaCo and skin coverage, with coverage averaged across TPT1–TPT3 before plotting for speaking, rotation, and cycling conditions.These summaries accompany Figures 15, 17, 19, 21, and 23 and use complete test sets for the displayed attribution statistics.Source: Figure 15 · p. 77, Figure 17 · p. 79, Figure 19 · p. 81
• For rotation, attribution maps remain concentrated on the face, refinement raises raw-attention coverage by 0.067 ± 0.011, and lowers rollout and flow coverage by 0.082 ± 0.009 and 0.062 ± 0.013.Median SaCo values are 0.652–0.819, while rotation ranks second worst on SNR and fourth worst on heart-rate error.Source: S6. Explainable artificial… · p. 78, Figure 16 · p. 79, Figure 17 · p. 79
• Bike level 1 has the largest heart-rate error, 18.6 ± 5.5 beats per minute, while its median SaCo is lowest for Beyond Intuition at 0.201 and attention flow at 0.345.Refinement raises raw-attention coverage by 0.105 ± 0.015 and lowers attention-rollout and attention-flow coverage by 0.118 ± 0.019 and 0.062 ± 0.014, respectively.Source: S6. Explainable artificial… · p. 80, Figure 18 · p. 81, Figure 19 · p. 81
• Bike level 3 pairs high median SaCo values of 0.665–0.830 across methods with the worst SNR, −11.9 ± 2.0 dB, among eight conditions.Refinement is nearly inert for attention rollout at −0.003 ± 0.019, while raw attention changes by +0.036 ± 0.012 and attention flow by −0.039 ± 0.017.Source: S6. Explainable artificial… · p. 82, Figure 20 · p. 83
• Bike level 5 shows the largest refinement shifts: raw-attention coverage rises by +0.119 ± 0.009, while attention-rollout coverage falls by −0.128 ± 0.013.These changes are about 14 and 10 times their standard errors; median SaCo remains 0.680–0.767 despite heart-rate error of 15.1 ± 3.5 beats per minute.Source: S6. Explainable artificial… · p. 84, Figure 22 · p. 85

• 5 s to 60 s: reported evaluation windows span this range, while 30 s and 10 s are the two commonest choices, differing by a factor of three.The authors’ 5.12 s window is at the short end of the literature range, although four articles use 160-frame clips, or 5.33 s at 30 frames per second.Source: S7. Heart-rate estimation settings in… · p. 86
• 0.23 to 1.63 beats per minute: changing the window from 10.24 to 30.72 s alters mean absolute error across eight conditions, comparable to differences between methods.The evaluation fixes a 5.12 s window, with a Welch segment capped at 256 samples and a 45–150 beats-per-minute search band.Source: S7. Heart-rate estimation settings in… · p. 86
• 11 beats per minute: a 160-frame clip at 30 frames per second, transformed without zero-padding, resolves no better than this.Almost every article selects the largest spectral peak within a physiological band, most often 0.75–2.5 Hz, but only nine of 58 report transform length or window function.Source: S7. Heart-rate estimation settings in… · p. 86
• 50 of 58 articles process video at 30 Hz, whereas consecutive evaluation windows usually do not overlap; ten articles overlap them by one frame to 97% of the window.The authors’ recordings use 50 Hz, so they report both frame counts and durations because an equal-frame window is shorter in seconds than in most studies.Source: S7. Heart-rate estimation settings in… · p. 86
• 17 of 58 articles omit the heart-rate window, 23 omit overlap, and four omit the spectral method, leaving published mean absolute errors non-comparable for close to a third of the literature.The table records these four settings because they materially affect reported error and are often unrecoverable from method sections.Source: S7. Heart-rate estimation settings in… · pp. 86–87
• The literature collection examined every article published from 2020 onwards applying remote photoplethysmography and recorded video frame rate, window length, overlap, spectral settings, and heart-rate estimator.Table 25 provides one row for each article and defines the window as the span over which one heart rate is computed.Source: S7. Heart-rate estimation settings in… · p. 86, Table 25 · p. 88
• Most listed pipelines estimate heart rate from spectral peaks using Welch, fast Fourier, or related power-spectrum methods, while some derive rates from detected inter-beat intervals or network outputs.Reported alternatives include DiffPhys’s inter-beat-interval estimator, STGNet’s waveform-to-beats-per-minute mapping, and Lifelight’s fiducial-point counting.Source: S7. Heart-rate estimation settings in… · pp. 90–91
• Recent studies use diverse temporal settings, including 4.27-second, 5.33-second, 6-second, 9.5-second, 10-second, 20-second, 30-second, and 60-second windows, with overlap and step sizes often unstated.Examples include CbPPGGAN’s 128-frame clips, RhythmFormer’s 160 frames, pyMMER’s 6 seconds, and CAP-rPPG’s 30-second testing windows.Source: S7. Heart-rate estimation settings in… · pp. 90–92
• This work uses 5.12-second windows at 50 Hz, with Welch power spectral density after second-order Butterworth filtering over 0.75–2.5 Hz (45–150 beats per minute).The window is 256 frames, with longer durations swept to 61.44 seconds in Table 19 and the segment capped at 256 samples.Source: S7. Heart-rate estimation settings in… · p. 92