PaperlayerView exact revision on arXiv

Source-linked AI summary

CardiacMamba: Fair and Robust RGB-RF Fusion for Remote Heart Rate Estimation via State Space Modeling

Bo Zhao, Zheng Wu, Yiping Xie, Zitong YU

arXiv:2608.15831v1cs.CVcs.AI

TL;DR

RGB-only remote heart-rate estimation is limited by illumination, motion, and skin-tone-dependent optical reflectance. CardiacMamba fuses RGB and RF cardiac cues with state-space modeling, achieving 0.96 bpm MAE, 3.06 bpm RMSE, and a 0.26 bpm light-dark skin-tone MAE gap on EquiPleth.

  • Problem

    RGB-based remote heart-rate estimation remains limited by illumination changes, motion artifacts, and skin-tone-dependent optical reflectance, while effective RGB-RF fusion remains challenging.

  • Method

    CardiacMamba uses state-space RGB-RF fusion to integrate optical facial and RF cardiac cues, align their dynamics, and refine multimodal representations spectrally.

  • Results

    0.96 bpm MAE, 3.06 bpm RMSE, and 0.97 Pearson correlation are achieved on EquiPleth, with a 0.26 bpm light-dark skin-tone MAE gap.

  • Takeaways & Limitations

    CardiacMamba provides accurate remote heart-rate estimation with reduced observed skin-tone disparities and robustness under RGB degradation and RF-missing conditions.

  • Takeaways & Limitations

    Performance degrades substantially when RGB is unavailable, revealing limited RF-only fallback and asymmetric robustness.

Abstract

from arXiv · show

Remote photoplethysmography (rPPG) enables non-contact heart rate (HR) monitoring from facial videos, but RGB-only methods are vulnerable to illumination changes, motion artifacts, and skin-tone-dependent optical reflectance. We propose CardiacMamba, a fair and robust RGB-RF fusion framework that integrates optical facial cues and radio-frequency cardiac motion cues through state space modeling. CardiacMamba introduces a Temporal Difference Mamba Module (TDMM) to enhance subtle RF temporal variations, a bidirectional SSM-based interaction mechanism to align heterogeneous RGB-RF dynamics, and a Channel-wise Fast Fourier Transform (CFFT) module for channel-domain spectral refinement. On the EquiPleth dataset, CardiacMamba achieves state-of-the-art performance with 0.96 bpm MAE, 3.06 bpm RMSE, and 0.97 Pearson correlation, while reducing the observed light-dark skin-tone MAE gap to 0.26 bpm and maintaining robustness under RGB degradation and RF-missing conditions

1 Introduction

CardiacMamba addresses limitations of RGB-only remote heart-rate estimation by fusing optical facial cues with complementary RF cardiac-motion cues in a state-space framework. It targets heterogeneous temporal alignment, physiological spectral refinement, robustness, and fairness.

  • Motivation: RGB-based rPPG is degraded by illumination changes, motion artifacts, and reduced pulsatile contrast under darker skin tones.These factors reduce the signal-to-noise ratio of optical cardiac measurements.
  • Motivation: RF sensing is largely insensitive to ambient lighting and skin pigmentation but remains vulnerable to lower spatial resolution, body motion, and multipath interference.RF captures minute chest-wall vibrations through electromagnetic reflections, complementing RGB facial appearance cues.
  • Challenges: Existing RGB-RF fusion methods insufficiently model heterogeneous optical BVP and RF chest-motion dynamics, leaving temporal alignment, frequency interaction, and demographic disparities underexplored.Shallow feature concatenation and late-stage fusion are identified as inadequate strategies.
  • CardiacMamba: CardiacMamba uses TDMM to enhance subtle RF temporal variations, bidirectional SSM interaction to align RGB-RF dynamics, and CFFT for channel-domain spectral refinement.The framework is designed for fair and robust remote HR estimation.
  • Results: 0.96 bpm MAE is achieved by CardiacMamba on EquiPleth, demonstrating state-of-the-art accuracy in the reported experiments.The supplied contribution passage reports this MAE value for CardiacMamba.

2 Related Work

Prior rPPG research has progressed from handcrafted RGB methods to deep architectures, while RF sensing offers complementary robustness. Existing RGB-RF fusion remains limited by heterogeneous cardiac representations and insufficient temporal or frequency-domain interaction, motivating SSM-based fusion.

  • RGB-based rPPG: RGB-based rPPG evolved from PCA and ICA decomposition to CNN and Transformer architectures but remains constrained by optical sensing.These limitations motivate non-visual modalities for physiological measurement.
  • RF-based sensing: RF radar captures cardiac-induced chest displacement and is robust to illumination and skin pigmentation, but suffers from lower spatial resolution and body-motion interference.RF is therefore best used as a complement to RGB.
  • RGB-RF fusion: Prior RGB-RF fusion studies include camera and 77 GHz radar, but simple concatenation or late fusion inadequately addresses their heterogeneous cardiac signals.RGB measures optical blood-volume variations, whereas RF measures mechanical chest-wall motion.
  • RGB-RF fusion: Existing fusion methods often lack explicit cross-modal temporal alignment and frequency-domain interaction, limiting effective integration of RGB and RF dynamics.The modalities capture different manifestations of cardiac activity, requiring more than straightforward feature combination.
  • State-space modeling: State Space Models use structured state transitions for long sequences, while bidirectional Vision Mamba offers lower-cost modeling suited to dense video or radar sequences with weak quasi-periodic dynamics.These properties motivate building RGB-RF fusion on SSM-based modeling.

3 Methodology

CardiacMamba integrates RGB video and RF inputs through dual-level feature extraction, SSM-based temporal interaction, and frequency-domain fusion. Its modules enhance modality-specific physiological cues, align temporal representations, and perform bidirectional RGB-RF fusion for BVP reconstruction.

  • Overall architecture: CardiacMamba processes RGB video and RF inputs through three stages: dual-level feature extraction, SSM-based temporal interaction, and frequency-domain fusion.The overall architecture is organized around complementary RGB and RF observations.
  • Feature extraction and alignment: Modality-specific encoders extract physiological representations, with BDCF and SCFM enhancing RGB cues and TDMM plus RFAMs refining and aligning RF features.BDCF processes BVP-related temporal color variations, while SCFM aggregates informative spatial-channel responses; RFAM aligns RF temporal resolution with RGB.
  • SSM-based temporal interaction: Vision Mamba models long-range temporal dependencies in both modalities under a shared SSM-based dynamic prior, with residual connections and linear projections stabilizing refinement.The method adopts discretized Mamba state space modeling for sequence modeling.
  • RF temporal modeling: TDMM highlights weak RF cardiac variations through frame differencing and captures long-range dependencies with an SSM-based Mamba block.It addresses subtle chest-wall displacement signals obscured by static reflections and low-frequency body motion.
  • Frequency-domain fusion: CFFT refines channel-frequency spectra in both branches, after which lightweight cross-gating enables bidirectional multimodal fusion before BVP waveform reconstruction.CFFT retains the real part of the inverse transform, while each modality modulates the other through element-wise cross-gating.

4 Experiment

CardiacMamba delivers strong EquiPleth performance, skin-tone fairness, missing-modality robustness, and resilience to RGB degradation through complementary optical and RF representations. Ablations and qualitative analyses identify spectral refinement, temporal RF modeling, and multimodal cues as important contributors.

  • Overall Performance: 0.96 bpm MAE, 3.06 bpm RMSE, and 0.97 Pearson correlation establish CardiacMamba’s best overall EquiPleth performance among representative RGB-only, RF-based, and RGB-RF baselines.It reduces MAE and RMSE by 66.6% and 68.1% versus the best RGB-only baseline, and by 14.3% and 10.5% versus Vilesov et al..
  • Fairness: 0.26 bpm MAE gap gives CardiacMamba the smallest light-dark skin-tone disparity, with 1.28 bpm RMSE and 0.05 Pearson correlation gaps.The MAE gap is reduced by 61.2% versus Vilesov et al., while ICA and PhysNet have MAE gaps of 4.42 bpm and 2.22 bpm.
  • Missing-Modality Robustness: 1.20 bpm MAE under absent RF remains close to the 0.96 bpm full-modality result, while MAE improves by 82.4% over Vilesov et al. in that setting.With both modalities available, CardiacMamba reduces MAE and RMSE by 14.3% and 10.5% versus Vilesov et al..
  • Ablation and Representation Analysis: Removing CFFT raises MAE from 0.96 bpm to 4.92 bpm and lowers ρ from 0.97 to 0.77, while removing any component degrades performance.CFFT is identified as most critical, and TDMM contributes to RF temporal modeling; optical and RF branches encode complementary cardiac cues.
  • RGB Degradation Robustness: 2.15 bpm MAE and 0.88 correlation under Gaussian RGB noise outperform RGB-only degradation from 1.20 bpm to 8.54 bpm MAE and 0.41 correlation.The results support RF as a stable complementary signal when visual observations are corrupted.
  • Qualitative Validation: Predicted PPG follows the ground-truth periodic pattern with phase-consistent alignment, providing qualitative evidence of physiological waveform recovery and preserved beat-to-beat dynamics.Bland-Altman analysis also shows tighter clustering within confidence intervals than Vilesov et al..

5 Conclusion

CardiacMamba combines RGB and RF cues through TDMM, bidirectional SSM, and CFFT for robust and fair remote heart-rate estimation. It achieves state-of-the-art accuracy on EquiPleth, reduces observed skin-tone disparities, and remains robust under RGB degradation and RF-missing conditions.

  • 5 Conclusion: CardiacMamba integrates TDMM, bidirectional SSM, and CFFT within an RGB-RF fusion framework for remote heart-rate estimation.The framework is designed for robust and fair estimation.
  • 5 Conclusion: The framework achieves state-of-the-art accuracy on EquiPleth while reducing observed skin-tone-related disparities.These findings support its fairness and performance claims.
  • 5 Conclusion: CardiacMamba remains robust under RGB degradation and RF-missing conditions, while future work will improve RF-only fallback and test generalization across larger populations.The conclusion identifies limited RF-only fallback and broader-population validation as future directions.

A Datasets and Metrics

CardiacMamba is evaluated on the synchronized EquiPleth RGB-RF benchmark, which combines optical facial and mechanical chest-wall cardiac observations across diverse skin tones. Evaluation uses subject-independent splits and reports MAE, RMSE, and Pearson correlation for heart-rate estimation.

  • Datasets: EquiPleth contains recordings from 91 subjects, including 28 light-skin, 49 medium-skin, and 14 dark-skin subjects.
  • Datasets: Each subject completed six 30-second sessions captured by an RGB camera at 30 fps and a 77 GHz FMCW radar.
  • Datasets: RGB videos encode facial blood-volume-related appearance variations, while RF signals capture mechanical chest-wall motion.
  • Datasets: Subject-independent predefined training, validation, and testing folds prevent identity leakage and support crosssubject generalization.
  • Metrics: Heart-rate estimation uses MAE, RMSE, and Pearson correlation coefficient (ρ), with MAE and RMSE in bpm.Lower MAE and RMSE indicate better accuracy, while higher ρ indicates stronger agreement with ground-truth HR.

B Experimental Setup · B.1 Experimental Setup

The experimental setup preprocesses facial RGB frames and RF signals into spatial-temporal representations focused on torso-related cardiac displacement. Training uses a fixed 30-epoch Adam protocol with validation checkpoint selection and subject-independent baseline evaluation.

  • B.1 Experimental Setup: MTCNN detects facial regions before cropping each frame for RGB preprocessing.Detected regions are resized to 128 × 128, converted to floating-point tensors, and normalized by 255.
  • B.1 Experimental Setup: RGB facial crops are resized to 128 × 128 for model input.The resized crops are converted to floating-point tensors and normalized by 255.
  • B.1 Experimental Setup: RF preprocessing transforms raw IQ samples into range profiles using Discrete Fourier Transform (DFT).The profiles are stacked over time to form range-time representations.
  • B.1 Experimental Setup: The highest-average-energy range bin and a 25 cm neighboring window isolate torso-related cardiac displacement.This selection focuses the RF representation while suppressing unspecified interference.
  • B.1 Experimental Setup: 30 epochs of training use Adam on an NVIDIA RTX 4090 GPU with batch size 32.The initial learning rate is 3 × 10−4 and weight decay is 1 × 10−2.
  • B.1 Experimental Setup: The best checkpoint is selected on the validation set and evaluated on the held-out test set.Unless otherwise specified, learning-based baselines follow the same subject-independent protocol.

C State Space Model Preliminaries

State Space Models efficiently model long-range sequences by evolving latent states from continuous inputs to outputs. Their discretized recurrence and equivalent convolution support weak, quasi-periodic physiological dynamics across long RGB and RF sequences.

  • SSM formulation: State Space Models map continuous inputs x(t) ∈R to outputs y(t) ∈R through a latent state h(t) ∈R^N.This formulation is designed for efficient long-range sequence modeling through latent state evolution.
  • SSM formulation: Parameters A, B, C, and D represent state transition, input projection, output projection, and skip connection operations, respectively.These parameters define how inputs update the hidden state and produce outputs.
  • Discretization: Zero-Order Hold discretizes the continuous system using step size ∆, producing discrete parameters A and B for neural sequence modeling.The discretization is defined by A = exp(∆A) and B = (∆A)^−1 (exp(∆A) − I) ∆B.
  • Discrete dynamics: The resulting recurrence is h_k = Ah_{k−1} + Bx_k and y_k = Ch_k + Dx_k, and it can equivalently be computed as a structured convolution.This provides a recurrent and convolutional view of the same state-evolution process.
  • Physiological sequence modeling: The state-evolution structure efficiently models weak and quasi-periodic physiological dynamics over long RGB and RF sequences.This motivates using SSMs for physiological signal modeling in multimodal RGB-RF settings.

D Channel-wise Fast Fourier Transform Details … F.1 Loss Function

The paper details CFFT as a channel-domain spectral refinement module with learnable complex interactions, alongside temporal, spatial, and intermediate RF refinements. Training uses Negative Pearson Loss based on predicted–ground-truth waveform correlation.

  • D Channel-wise Fast Fourier Transform Details: CFFT performs Fourier spectral mixing across feature channels to enhance informative responses and suppress noisy or redundant channels.It models inter-channel dependencies in a transformed feature space rather than directly estimating physiological frequencies.
  • D Channel-wise Fast Fourier Transform Details: CFFT decomposes channel-transformed features and applies a learnable complex-valued linear transformation to their Fourier coefficients.The transformation uses separate real and imaginary parameters with a nonlinear activation.
  • D Channel-wise Fast Fourier Transform Details: The complex transformation jointly updates amplitude- and phase-related channel-frequency responses before inverse-transform reconstruction.The refined spectrum is converted back into the feature representation through the inverse transform.
  • E Additional Module Details: Additional module details combine temporal, spatial, and intermediate refinements to process RF features before channel-spectral refinement.The supplied details include TDMM temporal differences, SCFM spatial attention, and RFAM intermediate refinement.
  • E.1 Temporal Difference Computation in TDMM: TDMM constructs a five-frame temporal neighborhood and computes four adjacent feature differences for RF temporal processing.The differences are D−2, D−1, D1, and D2, followed by concatenation and temporal convolution aggregation.
  • E.2 Spatial Attention in SCFM: SCFM generates a spatial attention response with a lightweight 5 × 5 convolutional stem, then normalizes and applies it to the feature map.The attention response is defined as A = σ(Stem(Xfu)).
  • E.3 Intermediate Refinement in RFAM: RFAM extracts local temporal patterns with a 7 × 1 convolution and computes channel attention from average- and max-pooled temporal descriptors.The intermediate representation is batch-normalized after convolution.
  • E.4 Channel-Spectral Refinement in Overview: CFFT provides channel-spectral refinement by enhancing informative inter-channel dependencies and suppressing redundant responses in learned features.The module is described as spectral refinement rather than direct physiological-frequency estimation from raw temporal signals.
Loading 2608.15831v1…