PaperlayerView exact revision on arXiv

Source-linked AI summary

Tracking Is Not Sensing: Artifact-Aware Thermal Video Respiration Estimation in Head-Free Marmosets

Penghao Wang, Sen Yang, Fenghua Xie, He Wang, Xindong Song

arXiv:2610.33661v1cs.CV

TL;DR

Head-free thermal respiration estimation is challenging because its weak, localized cues are easily corrupted, and plausible tracking can fail to measure physiology. The paper uses learned temporal signals from thermal clips and introduces STAR to address motion-corrupted training observations. STAR improves head-free estimation, reaching 2.23 bpm MAE versus 20.76 bpm for tracked ROI analysis.

  • Problem

    Thermal respiration cues in head-free marmosets are weak, localized, and vulnerable to motion and pose changes, while contact sensors can interfere with head-free protocols.

    Source: §1 · pp. 1–2

  • Method

    The paper predicts respiratory waveforms from thermal clips and uses STAR, which locally imputes selected high-motion frames during training to attenuate thermal discontinuities.

    Source: §1 · p. 2, §3.2 · p. 3, §4.2 · p. 5

  • Results

    2.23 bpm MAE with STAR versus 20.76 bpm for tracked ROI analysis in head-free estimation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Under head-free motion, plausible nasal tracking can fail to preserve respiration, motivating learned recovery of respiratory dynamics from thermal clips.

    Source: §6 · p. 8

    The modest cohort and controlled recordings limit generalization, particularly across individuals.

    Source: §5.2 · p. 7, §6 · p. 8

Abstract

from arXiv · show

Continuous physiological monitoring is essential for linking behavior, internal state, and neural activity in neuroscience and translational research. In common marmosets, a key nonhuman primate model for social neuroscience, respiratory rate provides a basic readout of arousal, stress, and physiological regulation. Contact sensors can provide accurate measurements, but they add preparation burden and may interfere with head-free behavioral protocols. Thermal video offers a noncontact alternative by capturing breathing-related temperature changes around the nose and mouth. However, in head-free marmosets, these cues are weak, localized, and easily corrupted by head motion, pose changes, and fur. Here we show that this setting exposes a broader challenge in video-based physiological sensing: visually reliable tracking does not necessarily imply physiologically valid measurement. Using synchronized thermal video and contact respiration recordings, we show that marmoset respiration is recoverable from thermal video but strongly ROI-sensitive. Under head-free motion, visually plausible nasal tracking can still produce traces inconsistent with contact respiration, demonstrating that tracking is not sensing. We therefore estimate respiratory rate from learned temporal signals extracted from thermal clips and introduce Selective Temporal Attenuation Regularization (STAR). STAR selectively attenuates high-motion thermal discontinuities through local temporal imputation during training, regularizing models toward respiration-consistent dynamics while leaving inference unchanged on raw video. Across multiple deep-learning backbones, STAR improves head-free estimation, reaching 2.23 bpm MAE compared with 20.76 bpm for tracked ROI analysis.

1. Introduction

Thermal video could enable noncontact respiration monitoring in marmosets, but its localized, weak cues are vulnerable to motion and pose changes. The paper shows that plausible tracking can fail physiologically and motivates learned temporal signal extraction with STAR.

1. Introduction

  • Head-free marmoset respiration is difficult to estimate because its weak thermal cues are localized and easily corrupted by fur, pose changes, nostril orientation, and head motion.Contact sensors provide accurate reference signals but can add preparation burden and interfere with head-free protocols.Source: §1 · p. 1
  • A visually plausible nasal ROI can still yield a temperature trace dominated by motion rather than respiration, so successful tracking does not guarantee valid sensing.Conventional ROI methods average temperature over time and estimate rate from the dominant frequency, assuming the ROI stays aligned with a stable respiratory source.Source: §1 · pp. 1–2
  • Synchronized thermal video and contact recordings show that respiration is recoverable under fixed-head conditions but highly ROI-sensitive, while plausible tracking can fail under head-free motion.These findings motivate learned spatiotemporal signal extraction rather than relying on ROI-averaged tracking alone.Source: §1 · p. 2
  • STAR uses local temporal imputation to attenuate high-motion thermal discontinuities during training and improves head-free estimation across multiple backbones, reaching 2.23 bpm MAE versus 20.76 bpm for tracked ROI analysis.The models predict temporal signals from thermal clips to recover respiratory waveforms.Source: §1 · p. 2

2. Related Work

Prior work establishes video-based physiological signal recovery and thermal respiration monitoring, but head-free marmoset respiration remains underexplored. Small respiratory regions and substantial motion make ROI-based estimation challenging in this setting.

Video based physiological measurement

  • Video-based physiological measurement has moved from handcrafted projections, motion cues, and signal decomposition toward learned spatiotemporal models that recover physiological signals from images.Prior methods demonstrate that weak physiological information can be extracted from visual signals.Source: §2 · p. 2

Thermal video for respiratory monitoring

  • Thermal respiration methods exploit periodic nasal temperature changes, but much prior work uses controlled settings or larger subjects, limiting applicability to highly mobile marmosets.Classical approaches analyze a nasal ROI temperature trace, while recent work has also explored learned methods.Source: §2 · p. 2

Respiratory monitoring in head-free marmosets

  • Continuous thermal-video respiration estimation in marmosets with large motion variation remains underexplored, and small respiratory regions and pose changes challenge ROI-based estimation.Related marmoset work includes thermal imaging of nasal temperature variations associated with emotional state.Source: §2 · p. 2

3. From Localized Respiratory Cues to Tracking Failure

The section establishes that respiration can be measured from thermal video but that ROI localization and motion strongly affect validity. Under head-free motion, continuous-looking tracking does not ensure accurate respiratory estimation.

3.1. Thermal Video Respiratory Rate Estimation

  • The dataset contains 48 five-minute sessions from four male marmosets, with 32 head-free and 16 fixed-head sessions; evaluation retains 150 head-free and 63 fixed-head windows after excluding severely corrupted reference signals.Thermal frames and contact respiration were aligned using the same hardware-timed acquisition system.Source: §3.1 · pp. 2–3

3.2. Evaluation Protocol

  • The evaluation pairs synchronized thermal and contact recordings, aggregates clip predictions into 20-second windows, and assesses respiratory-rate error with MAE, MAPE, and Pearson correlation.The ROI frequency baseline assumes the ROI remains aligned with a respiratory source and its mean is dominated by respiratory variation.Source: §3.2 · p. 3

3.3. Fixed Head Feasibility and ROI Sensitivity

  • Fixed-head recordings show recoverable respiratory information, but estimates vary across ROI selections; the post-hoc oracle ROI improves performance over the manual ROI and is a nondeployable diagnostic upper bound.Respiratory-band energy is localized near the nostril, and the oracle ROI captures it more effectively.Source: §3.3 · pp. 3–4, Table 1 · p. 4

3.4. Tracking Availability Does Not Ensure Respiratory Accuracy

  • Even after manual verification of nasal coverage, head-free tracked-ROI respiration estimates remain poor, showing that visual continuity and coverage do not ensure physiological accuracy.Head motion changes facial orientation and thermal distribution, allowing motion-induced variation to dominate the ROI mean.Source: §3.4 · p. 4, Table 2 · p. 4

4. Method

The method predicts respiratory waveforms from thermal clips rather than relying on an ROI-averaged trace, then uses STAR to attenuate motion-corrupted observations during training while keeping inference on raw clips.

4.1. Thermal Respiratory Signal Recovery

  • TS-CAN, PhysNet, and PhysFormer predict continuous respiratory waveforms from thermal facial clips, replacing the ROI-averaged trace with learned spatiotemporal extraction.Respiratory rate uses the same dominant-frequency readout as the ROI baseline.Source: §4.1 · p. 4

4.2. Selective Temporal Attenuation Regularization

  • STAR addresses motion-corrupted training observations without changing the backbone, using local temporal imputation while leaving inference on raw thermal clips.The aim is to encourage reliance on respiration-consistent thermal dynamics.Source: §4.2 · p. 5
  • Adjacent thermal differences identify likely motion-corrupted frames, with an adaptive threshold selecting approximately the top 10% per clip by default.Adjacent differences locate frames for attenuation; they are not supplied as normalized difference inputs to the backbone.Source: §4.2 · p. 5
  • For selected frames, STAR uses neighboring observations to estimate a respiration-consistent signal under a smoothness assumption, while leaving low-motion and boundary frames unchanged.The imputation preserves clip length and alignment with the respiratory waveform label.Source: §4.2 · p. 5
  • Training stochastically exposes the model to raw or attenuated clips, while inference always uses raw thermal clips.The raw view preserves the inference input distribution; the attenuated view reduces local observation-label mismatch during training.Source: §4.2 · p. 6

5. Experiments

Experiments tested thermal respiratory recovery from head-fixed through head-free conditions and used a session-held-out individual-dependent protocol, with a harder individual-independent evaluation. Learned models outperformed tracked ROI analysis in head-free recordings, while ablations favored motion-guided local imputation over generic temporal operations.

5. Experiments

  • The main learned-model evaluation held out one recording session for testing and used disjoint sessions across training, validation, and test sets; a four-fold held-out-individual evaluation tested harder cross-individual generalization.Respiratory-rate predictions were aggregated over 20-second windows and estimated from the dominant spectral peak in the 0.8–2.5 Hz band.Source: §5 · p. 6

5.1. Head-fixed Respiratory Signal Recovery

  • PhysFormer achieved 0.82 bpm MAE and 0.97 Pearson correlation in head-fixed recordings, outperforming the manually selected nasal ROI frequency baseline.Grad-CAM responses concentrated around the nostril and nearby thermal structures, indicating that the model used local evidence across multiple regions rather than one fixed ROI average.Source: Table 3 · p. 6, §5.1 · p. 6

5.2. Head-Free Robust Respiration Estimation

  • Under individual-independent evaluation, learned-model performance degraded, and STAR did not fully resolve cross-individual domain shift.STAR clearly improved PhysNet but gave limited gains for PhysFormer; for TS-CAN, it slightly improved MAPE and correlation but not MAE.Source: §5.2 · p. 7
  • 20.76 bpm MAE for tracked ROI analysis contrasted with learned models below 4 bpm MAE and above 0.90 Pearson correlation for PhysFormer and PhysNet in individual-dependent evaluation.STAR consistently improved all evaluated backbones in this setting.Source: Table 4 · p. 7, §5.2 · p. 7

5.3. Ablation Study

  • Only motion-guided local temporal imputation improved the raw baseline in the B6 head-free ablation; random temporal operations and low-motion imputation did not improve it, while temporal filtering degraded performance.The results support STAR as a targeted intervention on likely motion-corrupted frames rather than generic temporal augmentation.Source: §5.3 · pp. 7–8, Table 5 · p. 8
  • Stochastic raw and attenuated-view exposure with ps = 0.50 performed best, while always-applied temporal filtering degraded sharply at ps = 1.00.The top 10% motion-mask setting gave the best balance between missing artifacts and overprocessing useful respiratory dynamics.Source: §5.3 · p. 8

6. Discussion and Conclusion

The discussion frames head-free tracking failure as motivation for learned waveform recovery and STAR, while noting that the study's modest, controlled cohort limits generalization, particularly across individuals.

6. Discussion and Conclusion

  • STAR uses local temporal imputation on selected high-motion frames during training to reduce motion-induced observation-label mismatch.The findings may also inform learning from perturbed visual evidence [30] and sensing affected by motion artifacts or cross-sensor misalignment [14].Source: §6 · p. 8
  • The modest cohort and controlled recordings limit generalization, particularly across individuals.Future work should expand the cohort and develop quality control for naturalistic motion, occlusion, camera-distance changes, and social interactions.Source: §6 · p. 8

Compliance with Ethical Standards

The animal procedures followed institutional and national care guidelines and received approval from Tsinghua University's IACUC.

Compliance with Ethical Standards

A. Data Collection and Screening Details

The appendix documents synchronized thermal and contact-respiration acquisition, the session collection, and screening of reference signals before evaluation.

A. Data Collection and Screening Details

B. Evaluation and Implementation Details

These implementation details describe diagnostic ROI analyses and the tracking, training, and evaluation protocols used to assess thermal respiration estimation.

B. Evaluation and Implementation Details

  • The oracle search normalizes candidate boxes by nostril distance, searches positions and scales, and selects the ROI minimizing contact-reference rate error; it is diagnostic, not deployable.Candidate ROIs are evaluated with the same frequency-domain pipeline used in the main evaluation.Source: B. Evaluation and Implementation Details · p. 12
  • 0.6 ND gives 0.59 bpm MAE, 0.83% MAPE, and Pearson ρ = 0.99; at 3.0 ND, MAE is 2.54 bpm and Pearson ρ = 0.85.The scale-sensitivity analysis finds stronger estimates from smaller nostril-adjacent ROIs, while broader ROIs dilute the respiratory signal.Source: Table 6 · p. 12, B. Evaluation and Implementation Details · p. 12
  • The RR power map assigns each candidate ROI a score based on respiratory-band power around the ground-truth rate; brighter regions indicate stronger power.The analysis computes mean thermal signals across ROI pixels and maps their respiratory-band power back to candidate ROI locations.Source: B. Evaluation and Implementation Details · p. 12
  • Tracking availability measures successful box returns rather than usable respiratory signal, and the main tracking analysis retains 141 manually verified CSRT windows.A window is visually available when ROI availability is at least 95%.Source: B. Evaluation and Implementation Details · p. 12
  • Learned models use synchronized contact-respiration supervision and nonoverlapping session splits; 50-frame clips feed models, with respiratory rates evaluated over 20-second windows.The clips correspond to 2 seconds at the 25 fps training sampling rate.Source: B. Evaluation and Implementation Details · pp. 12–13

C. Supplementary Agreement Analysis

The supplementary agreement analysis visualizes STAR-trained PhysNet predictions against reference respiratory rates and examines signed error across rates, complementing the main-text metrics with prediction-error distributions.

C. Supplementary Agreement Analysis

D. Motion Stratification Analysis

The analysis stratifies head-free windows by short-burst motion intensity and the fraction of frames affected by motion. Across both groupings, learned respiratory recovery maintains higher success rates than ROI frequency analysis, while tracking examples illustrate that visual continuity alone does not ensure an accurate respiratory trace.

D. Motion Stratification Analysis

E. Additional Tracking Examples

D. Motion Stratification Analysis

Related papers

Loading 2610.33661v1…