PaperlayerView exact revision on arXiv

Source-linked AI summary

FLOW: Feature-Level Optimal Warping for Generalized Remote Physiological Measurement

Bo Zhao, Junzhe Cao, Dan Guo, Dongmin Huang, Wenjin Wang, Tao Tan, Yue Sun, Zitong YU

arXiv:2609.38913v1cs.CV

TL;DR

Domain shifts limit rPPG models on unseen environments, and existing approaches leave end-to-end domain generalization underexplored. FLOW combines temporal refinement, prototype-based optimal-transport alignment, and regularization; it reports state-of-the-art generalization across datasets and architectures.

  • Problem

    Domain shifts can cause rPPG performance drops on unseen environments, while end-to-end domain generalization remains underexplored.

    Source: §1 · p. 1

  • Method

    FLOW combines temporal refinement, prototype-based optimal-transport feature alignment, and source-consistency and identity-preservation regularization.

    Source: §1 · p. 2

  • Results

    FLOW achieves state-of-the-art generalization across diverse datasets and architectures; on BUAA-MIHR, MAE 2.23 and R 0.97 surpass MMD’s MAE 2.80 and R 0.95.

    Source: §4.1.1 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    FLOW learns domain-invariant yet physiologically consistent representations across diverse datasets and architectures.

    Source: §5 · p. 8

Abstract

from arXiv · show

Remote photoplethysmography (rPPG) enables non-contact physiological measurement but remains vulnerable to domain shifts from illumination, motion, and sensors. We propose \textbf{FLOW (Feature-Level Optimal Warping)}, an \emph{optimal transport--driven} framework for domain-generalized rPPG. FLOW integrates a \textbf{Temporal Refinement Module (TRM)} to stabilize temporal dynamics and a \textbf{Prototype-based Cross-Temporal Optimal Transport (PCOT)} module to achieve domain-invariant alignment via learnable prototypes.Beyond feature alignment, FLOW employs soft cross-temporal correspondence modeling that aligns temporal features in a flexible manner, allowing the model to respect and preserve the intrinsic rhythmic patterns of physiological signals. Moreover, the lightweight design of our modules allows seamless integration into existing end-to-end rPPG architectures without additional preprocessing. Two regularization terms further enforce source consistency and identity preservation. Theoretically, we derive a generalization bound under conditional optimal transport. Extensive experiments across four rPPG benchmarks show that FLOW achieves state-of-the-art cross-domain performance with lightweight design and strong physiological fidelity.

1. Introduction

End-to-end rPPG models face domain shifts that limit performance on unseen environments, while existing rPPG generalization work leaves end-to-end learning and principled feature alignment underexplored. FLOW uses optimal transport for feature alignment, combines temporal refinement with prototype-based alignment and regularization, and derives a conditional-OT generalization bound.

1. Introduction

  • Domain shifts in illumination, sensors, skin tones, or motion can sharply reduce rPPG performance, while labeled data from every deployment domain is impractical to collect.Domain generalization targets unseen domains without access to target data during training.Source: §1 · p. 1
  • Existing rPPG generalization methods often depend on preprocessing or single-source architectures, leaving end-to-end generalization and principled representation alignment underexplored.The paper also notes that domain generalization for end-to-end rPPG remains largely unexplored despite its growing use.Source: §1 · p. 1
  • FLOW is an optimal-transport-driven, architecture-agnostic framework that aligns feature distributions across multiple source domains in a shared latent space.Its feature-level warping module is designed for compatibility with various rPPG backbones.Source: §1 · pp. 1–2
  • TRM stabilizes temporal patterns, while PCOT softly aligns features to learnable prototypes and regularization encourages source consistency and identity preservation.The components aim to retain physiologically meaningful representations while reducing inter-domain discrepancies.Source: §1 · p. 2
  • A multi-source generalization bound under conditional optimal transport links alignment quality to prediction risk on unseen domains.The paper presents this as a theoretical explanation for robustness in cross-domain temporal regression.Source: §1 · p. 2

2. Related Work

The related work traces rPPG from signal-processing and STMap pipelines to end-to-end learning, whose cross-domain generalization remains challenging. It identifies gaps in rPPG domain generalization and in applying optimal transport to end-to-end temporal regression.

2.1. Remote Photoplethysmography (rPPG)

  • End-to-end rPPG models regress physiological signals from raw videos, but their generalization across domains remains a major challenge.Earlier approaches include signal processing and STMaps that convert pixel-level temporal signals into images for CNNs.Source: §2.1 · p. 2

2.2. Domain Generalization (DG)

  • Prior rPPG domain-generalization work mostly uses single-source settings or STMap preprocessing and lacks theoretical grounding for broader transfer.The paper states that no dedicated study had focused on domain generalization for end-to-end rPPG models.Source: §2.2 · p. 2

2.3. Optimal Transport for Domain Adaptation and Generalization

  • Existing optimal-transport domain-generalization methods largely target classification, while FLOW applies OT alignment to multi-source end-to-end rPPG.The cited related work describes OT as not yet explored for rPPG, especially as a plug-and-play module for end-to-end networks.Source: §2.3 · pp. 2–3

3. Methodology

FLOW first refines temporal representations, then aligns them to shared prototypes using soft optimal-transport correspondences. Its regularizers address cross-domain prototype consistency and preservation of original features and rhythms.

3. Methodology

  • FLOW refines intermediate features with TRM before PCOT aligns them to shared prototypes, with regularization objectives supporting temporal consistency and cross-domain coherence.The framework uses a shared backbone and is trained with OT alignment, source-consistency, identity-preservation, and task losses.Source: §3 · p. 3, Figure 1 · p. 4

3.1. Prototype-based Cross-Temporal Optimal Transport (PCOT)

  • PCOT softly matches temporal features to learnable prototypes using feature similarity and physiological coherence, then uses entropic OT to produce stable correspondences.The transport plan has normalized marginals; barycentric projection re-expresses temporal features in the prototype manifold, and a debiased Sinkhorn divergence is used as alignment loss.Source: §3.1 · p. 3

3.2. Temporal Refinement Module (TRM)

  • TRM converts heterogeneous feature tensors into a unified temporal form and refines local dynamics with lightweight depthwise-separable 1D convolutions.The module is described as a lowpass temporal filter that enhances phase stability and rhythmic coherence.Source: §3.2 · pp. 3–4

3.3. Regularization for Stable Cross-Domain Alignment

  • Source-consistency regularization encourages uniform prototype occupancy across domains, while identity preservation constrains the distance between original and aligned representations.The combined objectives are described as supporting global domain consistency and local physiological preservation without adversarial training.Source: §3.3 · pp. 4–5

4. Experiments

Experiments evaluate FLOW on four public rPPG datasets using MAE, RMSE, and Pearson correlation, across multi-source and limited-source domain-generalization settings. The results report cross-domain gains, transfer across backbones, and benefits from the model’s components and loss design.

4.1.1. Multi-source Domain Generalization Results

  • On BUAA-MIHR, FLOW reaches MAE 2.23 and R 0.97 versus MMD’s MAE 2.80 and R 0.95, while improving Pearson correlation by more than 0.4 over MMD on UBFC-rPPG and PURE.Traditional methods fluctuate across datasets, and PhysNet reaches R = -0.15 on PURE.Source: §4.1.1 · pp. 5–6

4.1.2. Limited-source Domain Generalization Results

  • With two source datasets, FLOW achieves MAE 9.06 and RMSE 14.23 on MMPD from PURE+UBFC-rPPG, and MAE 2.56 and R = 0.98 on BUAA-MIHR from PURE+MMPD.It maintains stable performance across source combinations, while CORAL and MMD show inconsistent gains.Source: §4.1.2 · pp. 6–7

4.1.3. Generalization of FLOW

  • Across source–target settings, FLOW yields the lowest MAE/RMSE among alignment methods on PhysFormer and improves over PhysFormer, MMD, and CORAL in reported examples.For PURE+MMPD targeting MMPD, FLOW reduces MAE from 14.86 to 10.38 and RMSE from 18.26 to 14.14; across four backbones, FLOW also consistently improves performance.Source: §4.1.3 · p. 7, Table 4 · p. 7

4.2.1. Ablation on Model Components

  • Removing TRM raises MAE from 2.23 to 3.12 on BUAA-MIHR and from 7.38 to 8.16 on MMPD, while removing PCOT yields MAE 4.67 and 10.24, respectively.The reported ablation attributes the largest degradation to removing PCOT.Source: §4.2.1 · p. 7, Table 5 · p. 7

4.2.2. Ablation on Loss Functions

  • Combining LOT, Lalign, and Lsc achieves the best reported MAE: 2.23 on BUAA-MIHR and 7.38 on MMPD.The terms address temporal prototype correspondence, cross-domain feature alignment, and semantic relationships among prototypes, respectively.Source: §4.2.2 · pp. 7–8, Table 6 · p. 8

4.2.3. Ablation on Backbone

  • Across four backbones and four cross-domain settings, FLOW consistently reduces prediction error, with substantial improvements reported for PhysFormer and EfficientPhys on BUAA-MIHR and MMPD.The paper describes the module as plug-and-play and not dependent on a specific architecture.Source: §4.2.3 · p. 8, Figure 2 · p. 8

4.2.4. Analysis of Structural Hyperparameters

  • MAE reaches 2.23 bpm at K = 64 prototypes and with three TRM aligner layers; further prototype increases slightly reduce gains, while deeper stacking yields marginal gains.The authors attribute the post-64 prototype decline as likely due to over-fragmentation weakening semantic compactness.Source: §4.2.4 · p. 8

5. Conclusion

FLOW combines temporal refinement and prototype-based optimal-transport alignment for domain-generalized physiological estimation. Experiments demonstrate state-of-the-art generalization across diverse datasets and architectures; future work will extend FLOW to multimodal sensing and real-world deployment.

5. Conclusion

  • FLOW combines the Temporal Refinement Module for temporal coherence with Prototype-based Cross-domain Optimal Transport for semantic alignment.The framework aims to learn domain-invariant yet physiologically consistent representations.Source: §5 · p. 8
  • Experiments demonstrate state-of-the-art generalization across diverse datasets and architectures.Source: §5 · p. 8
  • The authors plan to extend FLOW to multimodal sensing and real-world deployment.The stated aim is reliable physiological estimation under unconstrained conditions.Source: §5 · p. 8

FLOW: Optimal Transport-Driven Feature Warping for Generalized

The supplied passage identifies the topic as remote physiological measurement.

FLOW: Optimal Transport-Driven Feature Warping for Generalized

A. Introduction to the Datasets

The experiments use four rPPG datasets spanning varied recording conditions, movement scenarios, and subject characteristics. The BUAA-MIHR experiments exclude recordings below 10 lux because they require specialized enhancement beyond the study’s scope.

A. Introduction to the Datasets

  • MMPD contains 660 one-minute videos from 33 subjects, recorded under four lighting conditions during varied daily activities.The dataset includes diverse skin tones and gender distributions; videos are 30 fps at 320×240 resolution.Source: A. Introduction to the Datasets · p. 12
  • UBFC-rPPG contains 42 facial videos from 42 subjects recorded at 640×480 and 30 fps under natural and indoor lighting.Ground-truth signals are recorded with a CMS50E pulse oximeter at 60 Hz.Source: A. Introduction to the Datasets · p. 12
  • PURE comprises 60 videos from 10 subjects performing six head-movement scenarios under consistent indoor lighting.Videos are recorded at 30 fps, with synchronized physiological measurements sampled at 60 Hz.Source: A. Introduction to the Datasets · p. 12
  • BUAA-MIHR covers controlled illumination from below 10 lux to normal brightness, but experiments use only recordings at or above 10 lux.Extremely dim recordings are excluded because image degradation requires specialized enhancement beyond this study’s scope.Source: A. Introduction to the Datasets · p. 12

B. Experimental Settings

Evaluation uses four rPPG datasets and measures heart-rate prediction error and correlation. Implementation uses PhysLLM as the baseline, with fixed video preprocessing and specified training settings.

B. Experimental Settings

  • The evaluation uses MAE, RMSE, and Pearson’s correlation coefficient to assess predicted heart rates across four rPPG datasets.Lower MAE and RMSE indicate smaller errors, while higher R indicates stronger linear correlation with ground-truth heart rates.Source: B. Experimental Settings · p. 12
  • Video preprocessing crops the face using a bounding box from the first frame, resamples clips to 30 fps, and selects 128-frame chunks resized to 128 × 128 pixels.Source: B. Experimental Settings · p. 12
  • Experiments use PhysLLM as the baseline and train for 20 epochs with Adam, an initial learning rate of 1 × 10−4, weight decay of 5 × 10−5, and batch size 4.The experiments are implemented in PyTorch, primarily based on rPPG-Toolbox, and run on an NVIDIA H100 GPU.Source: B. Experimental Settings · p. 12

C. Theoretical Proof of the Generalization Bound

The proof develops conditional optimal transport (OT) as a way to connect task-aware cross-domain alignment with prediction risk on unseen domains.

C. Theoretical Proof of the Generalization Bound

  • The proof connects task-aware conditional OT alignment quality to prediction risk on an unseen target domain.The derivation is framed around the conditional OT geometry used in FLOW.Source: C. Theoretical Proof of the… · p. 12

Preliminaries

  • Each sample pairs a visual-temporal feature with continuous heart rate, and the task-driven cost accounts for feature distance and physiological consistency.The feature weighting matrix is learnable, while λhr and σ control the heart-rate kernel term’s influence and bandwidth.Source: Preliminaries · pp. 12–13
  • The source OT barycenter is a geometry-aware anchor, while practical Sinkhorn approximation and identity-preserving regularization contribute residuals ∆sink and ∆id.Both residuals are treated as small non-negative constants controlled by regularization strength and optimization accuracy.Source: Preliminaries · p. 13

Risk Discrepancy under Conditional OT

  • Assuming the conditional loss-risk function is Lc-Lipschitz, Lemma 1 bounds domain risk discrepancy through conditional OT distance, with ∆id accounting for regularization distortion.The bound applies to any distributions P and Q.Source: Risk Discrepancy under Conditional OT · p. 13

From Sources to Barycenter and Target

The proof transfers risk bounds through a source OT barycenter and compares conditional OT geometry with a Euclidean Wasserstein-1 bound.

From Sources to Barycenter and Target

Comparison with Traditional Geometric Bounds

  • The comparison is against a Euclidean Wasserstein-1 bound whose L1 is a Euclidean-metric Lipschitz constant and whose error term collects statistical and approximation errors.The conditional discrepancy comparison is subject to the stated boundedness and Lipschitz assumptions.Source: Comparison with Traditional Geometric… · p. 14
  • When 2rκ < 1, under standard boundedness and normalization assumptions, conditional OT yields a strictly tighter or comparable discrepancy term than the Euclidean Wasserstein-1 counterpart, up to (∆sink + ∆id + ∆est).Under these assumptions, both r and κ remain moderate constants.Source: Comparison with Traditional Geometric… · p. 14

D. Training and Inference Procedure of FLOW

FLOW’s training pipeline refines temporal features before prototype-based alignment, then uses the learned model to predict heart rate at inference.

Training

  • During training, the backbone and TRM produce refined features before PCOT forms soft cross-temporal correspondences to align features across domains.The algorithm uses multi-source videos, domain identifiers, and rPPG signals, with heart-rate labels obtained from FFT.Source: D. Training and Inference Procedure of… · pp. 14–15

Inference

  • At inference, FLOW applies the backbone and TRM, computes prototype correspondences, aggregates regression outputs, and predicts heart rate without domain labels or additional preprocessing.The procedure produces temporally coherent and domain-invariant representations.Source: D. Training and Inference Procedure of… · pp. 14–15

Training

Loading 2609.38913v1…