PhysVR: Vision-Language Model Guided Interference-aware Temporal Feature Refinement for Remote Physiological Measurement

Zixu Li; Jianjun Qian; Hang Shao; Daoheng Li; Lei Luo; Jian Yang

Source-linked AI summary

PhysVR: Vision-Language Model Guided Interference-aware Temporal Feature Refinement for Remote Physiological Measurement

Zixu Li, Jianjun Qian, Hang Shao, Daoheng Li, Lei Luo, Jian Yang

arXiv:2608.29663v1Aug 30, 2026cs.CV

TL;DR

rPPG estimation is vulnerable to visual interference, and existing methods rarely examine whether interference remains in learned temporal features before prediction. PhysVR combines physiological and VLM-derived visual evidence to refine those features before final estimation, consistently improving performance across five public benchmarks under intra- and cross-dataset evaluation.

Problem
Existing rPPG methods mainly suppress interference during feature learning, while residual interference in learned temporal features and its further suppression before prediction are rarely examined.
Method
PhysVR uses a physiological backbone, a frozen VLM visual-interference branch, temporal cross-attention, and context-guided shared and interference-specific refinement before final rPPG estimation.
Results
PhysVR consistently outperforms representative methods on five public benchmarks under both intra-dataset and cross-dataset evaluation protocols.
Takeaways & Limitations
Further temporal feature refinement guided by physiological reliability and visual interference evidence improves robust rPPG estimation under challenging visual conditions.

Abstract

from arXiv · show

Remote photoplethysmography (rPPG) enables contactless physiological measurement from facial videos, yet its subtle pulse-related variations are easily affected by illumination variation, head motion, facial blur, and region-of-interest instability. Existing methods mainly suppress interference during feature learning, while whether the learned temporal features remain affected by interference and how to further suppress such interference before rPPG estimation are rarely examined. To address this limitation, we propose PhysVR, a vision-language model guided interference-aware temporal feature refinement framework for rPPG estimation. Specifically, a physiological backbone produces global temporal features and a coarse rPPG prediction, from which signal-derived physiological reliability evidence is constructed from local temporal characteristics. In parallel, a frozen vision-language model processes sampled facial frames under an interference-oriented prompt, and an evidence head extracts visual interference evidence from the VLM output. Temporal cross-attention integrates the physiological and visual evidence with the global temporal features to construct interference-aware temporal context. Guided by this context, a shared temporal correction unit performs general refinement, while four interference-specific experts selectively suppress different interference through adaptive routing. The refined temporal features are then used for final rPPG estimation. Extensive experiments on five public benchmarks demonstrate that PhysVR consistently outperforms representative methods under both intra-dataset and cross-dataset evaluation protocols.

I. INTRODUCTION

rPPG is vulnerable to environmental and facial-video interference, while existing methods generally estimate signals directly from learned temporal features. PhysVR instead refines those features using physiological reliability and visual interference evidence before final estimation.

rPPG captures subtle facial skin-color variations for contactless physiological measurement, but illumination, motion, blur, and ROI instability can obscure pulse-related changes.
Most existing methods directly regress rPPG signals from learned spatio-temporal features after feature learning.
Whether learned temporal features remain interference-affected and require further suppression before prediction is rarely examined.
PhysVR further refines learned temporal features before final rPPG estimation using physiological reliability and VLM-derived visual interference evidence.
The proposed context-guided module combines shared temporal correction, four interference-specific experts, and adaptive routing to suppress interference selectively.
Experiments on five public benchmarks evaluate PhysVR under both intra-dataset and cross-dataset settings.

A. Remote Physiological Measurement

The physiological backbone converts frame differences into global temporal features and a coarse rPPG signal, then derives time-resolved reliability evidence from local signal characteristics. This design combines spatial-temporal modeling with fixed signal-processing descriptors to support later refinement.

Frame differencing suppresses temporally invariant facial appearance while emphasizing pulse-related and interference-related adjacent-frame variations.
The spatial encoder uses stacked Spatial-Blocks, while the temporal network contains two Depth-Blocks followed by two Spatial-Blocks to model temporal dependencies.
The backbone output H represents global temporal features used for rPPG estimation.
A signal prediction head maps H to a coarse rPPG signal ˆsc.
Four time-aligned descriptors characterize local temporal properties of the coarse signal, and their concatenation is projected into H's feature dimension.
Fixed temporal operators describe band-limited, slowly varying, difference, and deviation characteristics used to form time-resolved physiological reliability evidence R.

B. VLM-Based Visual Interference Perception

PhysVR uses a frozen VLM branch to characterize clip-level visual interference from sampled facial frames. A fixed interference-oriented prompt focuses this perception on visual factors relevant to rPPG estimation while limiting VLM processing cost.

The VLM branch extracts clip-level visual interference information from sampled facial frames because physiological reliability evidence does not explicitly characterize visual interference.
The fixed prompt directs the VLM toward illumination fluctuation, head motion, facial blur, and ROI instability.
Uniformly sampling N frames covers visual conditions across the clip while limiting VLM processing cost, with the first and last frames always selected.
The sampled frames and fixed prompt are processed jointly by a frozen Qwen2-VL-2B-Instruct model.
A trainable evidence head transforms the VLM hidden state into clip-level visual interference evidence ev in (−1, 1).

C. Context-Guided Temporal Feature Refinement

PhysVR constructs interference-aware temporal context by combining physiological and visual evidence with global temporal features, then applies shared and interference-specific refinements before final rPPG estimation.

Context construction: Temporal cross-attention uses global temporal features as queries to integrate time-resolved physiological evidence and clip-level visual evidence.The resulting context represents interference-aware temporal information for feature refinement.
Shared refinement: The shared temporal correction unit refines deviations common across different interference conditions using multi-scale temporal convolutions.It contains three residual correction blocks with kernel size 5 and dilation rates 1, 2, and 4.
Interference-specific refinement: Four experts separately address illumination variation, head motion, facial blur, and ROI instability.Their transformations include pointwise projections, temporal differences, local or dilated temporal convolutions, and longer-range temporal modeling.
Adaptive routing: A temporal router independently gates each expert at every temporal position, allowing multiple experts to activate simultaneously.Binary gates use a threshold of 0.5 and a straight-through estimator during training.
Output estimation: Activated expert corrections are combined with the shared correction, and a decoder maps the refined temporal features to the final rPPG signal.The decoder operates on refined features in R^T×d and outputs a signal in R^T.

D. Training Objective

PhysVR jointly supervises the final and coarse rPPG predictions, combining pointwise, correlation, and frequency-domain signal losses.

Coarse supervision: Coarse rPPG supervision preserves meaningful temporal information because physiological reliability evidence is derived from the coarse prediction.PhysVR therefore produces both coarse and final rPPG signals during training.
Signal loss: The signal loss combines point-wise MSE, Pearson correlation loss, and Smooth L1 distance between normalized physiological-band power spectra.The correlation term is defined as 1 − ρ(y, s).
Joint objective: The overall objective sums the final-signal loss with weighted coarse-signal supervision.The coefficient λc controls the contribution of the coarse prediction loss.

IV. EXPERIMENT

PhysVR is evaluated on five public rPPG benchmarks using intra-dataset and cross-dataset protocols for heart-rate estimation accuracy.

Evaluation setting: The evaluation covers BUAA-MIHR, VIPL-HR, NIRP-DRV, NIRP-IND, and MMPD.The study compares PhysVR with state-of-the-art methods through quantitative and qualitative analyses.
Evaluation protocols: Both intra-dataset and cross-dataset evaluations assess heart-rate estimation accuracy.The experiments also analyze the trade-off between estimation accuracy and computational cost.

A. Datasets

The five benchmarks cover low illumination, unconstrained recording, driving, indoor motion, and mobile conditions with synchronized physiological references.

BUAA-MIHR: BUAA-MIHR contains 165 RGB facial videos from 15 subjects recorded across 11 illumination levels from 1.0 to 100.0 lux.It is designed for low-illumination rPPG evaluation, with synchronized finger-clip PPG references.
VIPL-HR: VIPL-HR contains 3,130 facial videos from 107 subjects under variations in illumination, head motion, and acquisition devices.Each video lasts approximately 30 seconds and includes synchronized finger BVP references.
NIRP-DRV: NIRP-DRV contains 190 RGB and near-infrared recordings from 19 subjects in realistic driving environments.The recordings include diverse in-vehicle illumination and natural head movements with synchronized pulse-oximeter references.
NIRP-IND: NIRP-IND provides 15 RGB and near-infrared recordings from 8 subjects in an indoor setting with stationary and motion conditions.Synchronized pulse-oximeter measurements serve as references.
MMPD: MMPD contains 660 one-minute RGB facial videos from 33 subjects spanning skin types III–VI, four illumination conditions, and four activities.Activities include stationary, head rotation, talking, and walking, with synchronized finger PPG references.

B. Performance Metrics

PhysVR evaluates heart-rate estimation with MAE, RMSE, and Pearson’s correlation coefficient, where lower errors and higher correlation indicate better agreement with ground truth.

MAE, RMSE, and Pearson’s correlation coefficient (ρ) are the three metrics used for heart-rate estimation.
MAE and RMSE measure estimation error, while ρ measures agreement between estimated and ground-truth heart rates.

C. Implementation Details

The experiments use subject-disjoint facial-video splits, sampled VLM inputs, and both intra- and cross-dataset evaluations. Results report strong accuracy across benchmarks, qualitative agreement with reference signals, and a measured accuracy–complexity trade-off.

Implementation: PhysVR processes 160-frame aligned clips and sends eight uniformly sampled frames to a frozen Qwen2-VL-2B-Instruct model with an interference-oriented prompt.The remaining components are jointly optimized for 30 epochs using AdamW at a learning rate of 3 × 10−4.
Implementation: All dataset partitions are subject-disjoint, with VIPL-HR using five-fold evaluation and other datasets using specified training-to-testing splits.Cross-dataset evaluation uses NIRP → VIPL-HR, VIPL-HR → NIRP-IND, and VIPL-HR → NIRP-DRV without target-domain training or finetuning.
Intra-dataset Evaluation: PhysVR achieves the best overall intra-dataset performance across five benchmarks, including MAE 0.38 bpm and RMSE 0.85 bpm on NIRP-IND.On NIRP-DRV, it achieves MAE 2.56 bpm, RMSE 5.85 bpm, and correlation 0.87.
Qualitative Evaluation: On a representative VIPL-HR sample, PhysVR estimates 79.4 bpm versus a ground-truth 78.8 bpm, with an absolute error of 0.6 bpm and aligned dominant spectral peaks.On NIRP-DRV, its predictions are more concentrated around the identity line than competing methods.
Cross-dataset Evaluation: Cross-dataset evaluation shows PhysVR achieving the best overall performance across all three transfer settings.For NIRP → VIPL-HR, it obtains MAE 3.92 bpm, RMSE 7.23 bpm, and correlation 0.77.
Complexity Analysis: PhysVR achieves the lowest RMSE in the accuracy–complexity comparison, while processing only eight sampled frames with the frozen VLM.The reported MACs include the VLM forward computation, and restricted processing limits additional computational overhead.

F. Ablation Studies

The ablations show that PhysVR’s evidence-guided temporal refinement improves rPPG estimation, with physiological and visual evidence jointly providing the strongest gains. Visual evidence and expert routing also respond selectively to different interference conditions.

Temporal Response of the Interference-Aware Context: 0.627 to 0.695: context response increases from low- to high-error clips, indicating stronger guidance when coarse predictions are less reliable.The response tracks local coarse-prediction error and supports context-guided refinement.
Analysis of Visual Evidence and Interference-Specific Experts: The visual-evidence magnitude rises from the stable condition’s 34.6% percentile rank to 80.8% under large motion, while experts specialize by interference type.The illumination expert contributes 58.0% under dark conditions and 61.0% under bright conditions, whereas the motion expert contributes 47.0% under large motion.
Effectiveness of Temporal Feature Refinement: The local RMSE decreases from 0.58 to 0.46, and the dominant spectral peak shifts from 92.3 to 59.4 bpm toward the 62.1 bpm ground-truth HR after refinement.Final HR estimates are also closer to the identity line than coarse estimates.
Ablation of Physiological and Visual Evidence: Combining physiological and visual evidence yields the best ablation performance, reducing MAE and RMSE to 2.56 and 5.85 bpm from coarse values of 3.79 and 6.83 bpm.Each evidence source improves refinement individually, but their combination provides the largest improvement.
Ablation of the Context-Guided Refinement Module: Removing the refinement module raises MAE and RMSE from 3.59/5.68 bpm to 4.82/6.87 bpm, while removing experts or adaptive routing also degrades performance.The shared correction unit produces the largest component-level degradation, and each interference-specific expert contributes complementary refinement capability.
Effect of Visual Encoder, Prompt Design, and VLM Backbone: The Qwen2-VL-7B configuration achieves the best reported performance with MAE 3.47 bpm and RMSE 5.61 bpm, while larger-backbone gains remain marginal.SmolVLM2-2.2B reaches MAE 3.51 and RMSE 5.63 bpm under the same prompt.

V. CONCLUSION

PhysVR refines learned temporal features using time-resolved physiological reliability and clip-level visual interference evidence before final rPPG estimation. Experiments on five public benchmarks show consistent intra- and cross-dataset improvements, while frozen VLM processing adds computational overhead.

PhysVR combines time-resolved physiological reliability evidence with clip-level visual interference evidence to guide temporal feature refinement before final rPPG estimation.
Experiments on five public benchmarks demonstrate consistent improvements under both intra-dataset and cross-dataset settings.
The frozen VLM introduces additional computational overhead, motivating future work on knowledge distillation and more compact visual interference perception models.