PaperlayerView exact revision on arXiv

Source-linked AI summary

Subject-Relative Micro-Motion and Sleep Dynamics for Near-Infrared Video Sleep Staging

Kunmin Jang, You Rim Choi, Hun Heo, Heonjun Lee, Suahn Bae, Dongik Park, Hyun-Woo Shin, Hyung-Sin Kim

arXiv:2609.05550v1cs.CVcs.AI

TL;DR

This work tests whether NIR video alone can provide sleep-stage evidence under PSG-defined labels, without reconstructing physiological proxies or using auxiliary physiological representations. ViNUSS combines subject-relative micro-motion learning with full-night dynamics modeling, achieving 0.80 accuracy and 0.78 macro-F1 for four-class staging on 475 overnight recordings.

  • Problem

    Contactless sleep staging under PSG-defined labels is motivated by limitations of existing monitoring, while NIR video methods often use video to recover physiological proxies rather than test its own stage evidence.

    Source: §1 · p. 2

  • Method

    ViNUSS combines spatially anchored pre-spatial micro-motion encoding, within-subject stage contrast, and two-scale modeling of within-epoch motion and full-night sleep dynamics.

    Source: §1 · p. 2

  • Results

    0.80 accuracy and 0.78 macro-F1 on four-class staging across 475 overnight NIR recordings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support NIR video as independently informative for PSG-defined staging, with visual evidence suggesting use of thoraco-abdominal periodicity and larger movements associated with arousals and position changes.

    Source: §1 · p. 2

    Fully video-based SleepVST reproduction was infeasible; the evaluated proxy-first oracle-input variants are not universal upper bounds for physiological-signal-based staging.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Near-infrared (NIR) video is a promising modality for contactless sleep monitoring, but recent video-based sleep staging methods often use it as a route to reconstructed respiratory/cardiac proxies or cross-modal physiological representations. We study video-only sleep staging under labels defined by polysomnography (PSG), where the model infers sleep stages from NIR video alone without explicit physiological proxy reconstruction or auxiliary physiological signal supervision. This tests whether NIR video itself can provide informative sleep-stage evidence, rather than only serving as an input for recovering physiological proxies. We propose ViNUSS (Video-Native Unmediated Sleep Staging), a framework that combines subject-relative micro-motion learning with full-night sleep dynamics modeling. Spatially anchored pre-spatial micro-motion encoding preserves localized temporal variation together with its spatial context. Within-subject stage contrast learns stage cues with respect to each subject's night-specific baseline. Two-scale sleep dynamics modeling captures within-epoch motion evolution and organizes epoch-level evidence into a coherent full-night sleep-stage trajectory. On 475 overnight NIR recordings (~3,250 hours), ViNUSS achieves 0.80 accuracy and 0.78 macro-F1 for four-class sleep staging. Interpretability analysis suggests attention to thoraco-abdominal periodic motion and gross body movements associated with arousals and position changes. These results support NIR video as an independently informative and complementary modality for PSG-defined sleep-stage estimation

1 Introduction

ViNUSS tests whether NIR video alone can provide informative evidence for PSG-defined sleep staging, without reconstructing physiological proxies or using auxiliary physiological representations. It combines subject-relative micro-motion learning with full-night dynamics and achieves 0.80 accuracy and 0.78 macro-F1 on four-class staging.

1 Introduction

  • Proxy-first and cross-modal methods use video to recover physiological signals or representations, but proxy extraction can be unreliable and may discard regional or transient motion.NIR video captures localized body motion in darkness, including respiration-related motion, limb movement, position changes, and brief arousals.Source: §1 · p. 2
  • PSG defines the labels, while the model uses NIR video alone, without explicit physiological proxy reconstruction or auxiliary physiological representation learning.The formulation tests whether video itself provides sleep-stage evidence rather than serving only to recover proxies or align with other physiological modalities.Source: §1 · p. 2
  • NIR sleep scenes are mostly static, with weak localized cues and subject-specific factors that can dominate; motion meaning also depends on its body location.The section motivates preserving localized temporal variation with spatial context, comparing stages against each subject’s night-specific baseline, and modeling dynamics across the night.Source: §1 · p. 2
  • ViNUSS combines spatially anchored pre-spatial micro-motion encoding, within-subject stage contrast, and two-scale modeling of epoch motion and full-night trajectories.The encoding scans each 30-second epoch before spatial aggregation; stage contrast uses the subject’s own visual and motion baseline.Source: §1 · p. 2
  • 0.80 accuracy and 0.78 macro-F1 on four-class staging across 475 overnight NIR recordings, with analysis suggesting attention to physiologically meaningful motion.The recordings total approximately 3,250 hours; identified cues include thoraco-abdominal periodicity and larger movements associated with arousals and position changes.Source: §1 · p. 2

2 Related Work

Related work spans clinical and contactless sleep staging, video-based proxy and cross-modal approaches, and general video-temporal models. The section positions NIR video as a source of localized evidence and motivates preserving weak motion cues while modeling full-night context.

2.1 Sleep Staging and Contactless Monitoring

  • PSG provides high-fidelity labels but requires contact sensors, overnight instrumentation, and expert scoring; contactless modalities avoid body-worn sensors but can capture limited aspects of sleep behavior.NIR video is relevant because it captures spatially localized body motion in darkness.Source: §2.1 · p. 3

2.2 Video-Based and Cross-Modal Sleep Staging

  • Video-based staging includes handcrafted motion features and proxy-first pipelines, whose physiological inputs are interpretable but can be unreliable under occlusion, position changes, far-field views, and face anonymization.Proxy compression can also discard regional motion patterns and transient movements not captured by predefined physiological signals.Source: §2.2 · p. 3
  • EEG-teacher distillation and video–EEG retrieval use physiological supervision; ViNUSS instead studies NIR video as an independently informative modality without EEG-derived features or explicit proxy reconstruction.PSG still defines the labels in ViNUSS.Source: §2.2 · p. 3

2.3 Video Understanding and Full-Night Temporal Modeling

  • Action-recognition models motivate motion-aware processing, but NIR sleep video requires preserving dense, low-amplitude temporal traces over a 30-second epoch before spatial abstraction.The related-work discussion contrasts these weak sleep cues with salient action cues targeted by general video methods.Source: §2.3 · p. 3
  • Sleep staging needs context beyond isolated epochs; ViNUSS summarizes motion within each 30-second epoch and models those representations across the subject’s full-night trajectory.The section identifies state-space sequence modeling as the means of modeling that trajectory.Source: §2.3 · p. 3

3 Method

ViNUSS stages PSG-labeled 30-second epochs from NIR video alone, without physiological signals as inputs or cross-modal supervision. It combines localized motion encoding, within-subject contrast, and full-night sequence modeling to address proxy and subject-specific limitations.

3.1 Design Motivation and ViNUSS Overview

  • Proxy reconstruction can miss localized evidence or introduce errors, while raw-video embeddings emphasize static structures and subject identity, motivating subject-relative, full-night modeling.The design preserves weak local micro-motion, compares stage evidence against each subject’s overnight baseline, and models the full-night trajectory.Source: §3.1 · pp. 4–5

3.2 Subject-Relative Micro-Motion Representation

  • PTE encodes pixel-wise temporal variation before spatial mixing, preserving localized traces; stacked filters target roughly 1–3-second sub-epoch intervals at 5 fps.Its learnable temporal responses are retained with spatial context before the backbone processes the enriched clip.Source: §3.2 · pp. 5–6
  • ISDL samples same-stage positives and different-stage negatives within one recording, making stage comparisons relative to the subject baseline; classification cross-entropy is combined with ISDL.The within-subject sampling reduces variation from cross-subject and acquisition differences.Source: §3.2 · p. 6

3.3 Full-Night Sleep Dynamics Modeling

  • SDM first condenses within-epoch motion, including periodicity and transient movement, then models epoch representations chronologically across the night before classification.It relates local motion summaries to the full-night trajectory to help calibrate subject-specific baselines and reduce abrupt epoch-wise fluctuations.Source: §3.3 · p. 6

4 Experiments

On KVSS, ViNUSS combines subject-relative micro-motion representation with full-night temporal modeling and achieves 0.80 accuracy and 0.78 macro-F1. The experiments examine component contributions, prediction trajectories, and the visual motion evidence associated with representative sleep states.

4.1 Experimental Setup

  • KVSS comprises 475 overnight NIR recordings (~3,250 hours); reproducing SleepVST's video pipeline was infeasible, so oracle-input variants were evaluated instead.The oracle variants use contact-sensor cardiac signals, with respiratory signals also included in the full-oracle setting; they are not universal upper bounds.Source: §4.1 · p. 7, §4.2 · p. 7

4.3 Ablation Study

  • With full SDM versus LongMamba only, Deep precision/recall rises from 0.63/0.56 to 0.67/0.68, while REM precision rises from 0.73 to 0.82 and recall falls from 0.87 to 0.83.Figure 4 compares predictions without SDM, with LongMamba only, and with full SDM.Source: Figure 4 · p. 8, §4.3 · pp. 8–9

4.2 Overall Performance

  • 0.80 accuracy, 0.68 κ, and 0.78 MF1 are achieved by ViNUSS on KVSS four-class staging.MF1 improves from 0.73 for the full-oracle SleepVST variant to 0.78.Source: §4.2 · p. 8

4.3 Ablation Study

  • On KVSS, adding PTE raises MF1 from 0.50 to 0.55, and adding ISDL raises it further to 0.56 while accuracy remains comparable.PTE preserves weak local temporal cues; ISDL improves stage-wise separability under subject-specific variation.Source: §4.3 · p. 8, Table 2 · p. 8
  • On the vanilla backbone, SDM raises MF1 from 0.50 to 0.67; with PTE and ISDL, MF1 reaches 0.78.With SDM held constant, adding PTE raises MF1 from 0.67 to 0.72, and adding ISDL raises it to 0.78.Source: §4.3 · p. 8, Table 2 · p. 8
  • Full-night modeling suppresses adjacent-stage oscillations and restores continuous Deep predictions when single-epoch visual evidence is weak.Figure 5(a) shows representative overnight prediction segments.Source: Figure 5 · p. 9, §4.3 · p. 9

4.4 Interpretation Analysis

  • In representative visualizations, stable Deep sleep draws saliency to thoraco-abdominal motion aligned with breathing, while arousals and position changes also draw attention to larger body movements.These patterns suggest the model uses motion evidence in NIR video without explicitly reconstructing predefined physiological proxies.Source: Figure 6 · p. 9, §4.4 · p. 9

4.3 Ablation Study

  • ISDL increases cosine similarity between consecutive epoch embeddings across stages, indicating smoother within-subject feature trajectories.Figure 5(b) presents the temporal-coherence analysis.Source: §4.3 · p. 9, Figure 5 · p. 9

5 Conclusion

ViNUSS combines subject-relative micro-motion learning with full-night dynamics modeling for PSG-defined staging from NIR video. Its results support NIR video as a complementary modality, while broader validation remains important.

5 Conclusion

  • On 475 overnight NIR recordings, ViNUSS achieves strong four-class staging performance, with ablations showing complementary benefits from its three components.The conclusion does not report specific accuracy or macro-F1 values.Source: §5 · p. 10
  • ViNUSS preserves localized temporal motion before spatial reduction, learns stage evidence relative to each subject’s overnight baseline, and organizes epoch evidence into full-night trajectories.Source: §5 · p. 10
  • Sleep Video EigenCAM suggests attention to thoraco-abdominal periodicity and broader body movements associated with arousals and position changes.Source: §5 · p. 10
  • The findings support NIR video as an independently informative, complementary modality for contactless PSG-defined staging; validation across more clinical sites and home environments remains important.The authors identify integration with physiological signals or wearables as a possible direction for more comprehensive monitoring.Source: §5 · p. 10

A.1 Dataset Composition

The KVSS Hospital A cohort combines synchronized PSG and far-field NIR recordings under varied clinical and recording conditions. The appendix details ViNUSS’s localized micro-motion encoding and two-scale modeling of within-epoch and full-night dynamics.

A.1 Dataset Composition

  • KVSS Hospital A pairs 475 overnight NIR recordings (~3,250 hours) with synchronized PSG and clinical metadata; the subject split was 335/70/70.Videos were captured at 640 × 480 and 5 fps from an approximately 3.5 m distant oblique viewpoint, with identifiable regions blurred.Source: §A.1 · p. 14

A.2 Demographic and Clinical Characteristics

  • The 475-person cohort included 353 males (74.3%), 122 females (25.7%), and 208 severe OSA cases (43.8%), spanning a broad age and severity range.BMI also varied substantially between individuals.Source: Figure A.2 · p. 14, §A.2 · p. 15

B.1.1 Subject-Relative Micro-Motion Representation.

  • PTE adds pixel-wise temporal responses before spatial mixing, then MoViNet-A2 features support epoch classification and ISDL while pre-GAP maps feed SDM.Its two branches cover local neighborhoods of up to 9 and 13 frames, respectively.Source: §B.1.1 · pp. 15–16, Figure B.1 · p. 15

B.1.2 Full-Night Sleep Dynamics Modeling.

  • ShortMamba summarizes motion within each 30-second epoch, while bidirectional LongMamba models epoch tokens across the night to predict all stages without post-hoc label smoothing.The full-night sequence uses positional embeddings, and Mamba enables linear-time modeling.Source: §B.1.2 · p. 16, Figure B.2 · p. 16

B.2 ViNUSS Training Details

The appendix specifies video preprocessing, contrastive representation training, and separate full-night dynamics training. It also defines the reported staging metrics and a per-subject accuracy summary for subgroup analyses.

B.2 ViNUSS Training Details

  • Videos were cropped around the bed, resized to 224 × 224, and segmented into aligned 30-second epochs of 150 frames, with last-frame padding for shorter epochs.Source: §B.2 · p. 16
  • The representation module combines epoch-level cross-entropy with ISDL, using same-subject, same-stage positives and same-subject, different-stage negatives.Training uses class-balanced anchor sampling, weighted sampling, augmentation, and AdamW at a learning rate of 10^-3.Source: §B.2 · p. 17
  • SDM is trained separately on saved pre-GAP features using individual full-night subject sequences and AdamW at a learning rate of 10^-4.Source: §B.2 · p. 17

B.3 Evaluation Metrics

  • Accuracy is the proportion of correctly classified epochs, macro-F1 equally averages class F1 scores, and Cohen’s κ measures agreement while accounting for chance.Equal class weighting makes macro-F1 useful under class imbalance.Source: §B.3 · p. 17
  • For subgroup robustness analyses, the authors average accuracy across subjects within each subgroup, giving every subject equal weight regardless of recording length.Source: §B.3 · p. 18

C SleepVST Re-implementation

The appendix describes SleepVST pretraining on contact-sensor datasets and its attempted transfer to KVSS. Waveform recovery challenges constrain a faithful video-only transfer, so oracle-input variants are also evaluated.

C SleepVST Re-implementation

  • SleepVST is pretrained for four-class staging on SHHS and MESA contact-sensor datasets using ECG-derived cardiac and thoracic-effort-derived breathing waveforms.The waveforms are encoded by convolutional layers and a Transformer over 30-second patches, with pretraining on 240-epoch windows.Source: C SleepVST Re-implementation · p. 18, §C.2 · p. 18

C.1 Datasets for SleepVST Training

  • SHHS provides overnight home PSG from more than 6,400 participants, while MESA’s Sleep Exam included overnight PSG for 2,237 participants.MESA also included wrist actigraphy and sleep questionnaires.Source: §C.1 · p. 18

C.3 Transfer to KVSS

  • The KVSS transfer attempt was constrained because blurred faces limit rPPG recovery and video-derived respiratory waveforms deviate from contact-belt signals under challenging recording conditions.Deviations are reported especially with bedding, position changes, weak motion, and non-respiratory body movement.Source: §C.3 · pp. 18–19
  • Because a faithful video-only SleepVST transfer could not be realized, the authors evaluated partial- and full-oracle inputs from contact sensors.These variants are framed as oracle-input versions of one proxy-first architecture, not universal upper bounds or practical video-only baselines.Source: §C.3 · p. 19
  • For both oracle variants, frozen SleepVST features are combined with DIS optical-flow features, then a Random Forest predicts four sleep classes.Source: §C.3 · p. 19

D.1 ViNUSS Results

Table D.1 presents ViNUSS class-wise performance in both five-class and four-class evaluations. The four-class evaluation merges N1 and N2 into Light and maps N3 to Deep using the same trained model.

D.1 ViNUSS Results

  • Table D.1 reports class-wise ViNUSS performance for both five-class and four-class evaluations.Source: Table D.1 · p. 19
  • For four-class evaluation, N1 and N2 are merged into Light, while N3 is treated as Deep.Source: §D.1 · p. 19
  • The five-class and four-class results come from the same trained model, with only the evaluation label mapping changed.Source: §D.1 · p. 19

D.2 Backbone Selection for Subject-Relative Micro-Motion Representation

The backbone comparison favors MoViNet-A2 for ViNUSS, while stratified test results indicate comparable performance across the represented recording conditions and subject characteristics.

D.2 Backbone Selection for Subject-Relative Micro-Motion Representation

  • MoViNet-A2 offers the best accuracy-efficiency tradeoff; MoViNet-A5 has over four times its parameters but does not improve accuracy and yields lower MF1.The comparison holds pre-spatial micro-motion encoding and within-subject stage contrast constant.Source: Table D.2 · p. 19, §D.2 · p. 20
  • Video Swin-Tiny performs worse than the MoViNet variants, consistent with the challenge of preserving weak, localized cues in NIR sleep video.The convolutional inductive bias of MoViNet is described as a more suitable local spatiotemporal prior for this setting.Source: §D.2 · p. 20

D.3 Stratified Performance across Recording Conditions and Subject Characteristics

  • Per-subject test accuracy remains comparable across the observed recording-condition and subject-characteristic subgroups, without clear systematic degradation.The analysis covers blanket occlusion, camera angle, room re-entry, light intensity, sex, age, BMI class, and AHI group.Source: Figure D.1 · p. 20, §D.3 · p. 20

D.4.1 Comparison with alternative sequence models.

Among inter-epoch-only sequence models, Mamba (Long only) performs strongest, and adding ShortMamba improves every class over LongMamba alone. Longer context also improves sleep-dynamics metrics, while examples show predictions following varied overnight trajectories, with errors concentrated near boundaries and short transients.

D.4.1 Comparison with alternative sequence models.

  • The full SDM improves every class over Mamba (Long only), especially Deep sleep, where F1 rises from 0.59 to 0.67.Among inter-epoch-only models, Mamba (Long only) leads with 0.78 accuracy, 0.74 MF1, and 0.65 κ; models share epoch-level inputs.Source: Table D.3 · p. 21, §D.4.1 · p. 21

D.4.2 Effect of temporal context length.

  • For the context-length experiment, each 30-second epoch is one timestamp, and the sequence model and training recipe are held fixed across window sizes.Source: §D.4.2 · p. 22
  • Accuracy increases from 0.72 at W = 10 to 0.78 with full-night context, while MF1 rises from 0.64 to 0.74 and κ from 0.53 to 0.65.Gains are largest up to W = 100; accuracy then begins to plateau, while MF1 and κ continue improving with longer context.Source: Table D.4 · p. 22, §D.4.2 · p. 22

D.5 Example Full-Night Test Predictions

  • Across example subjects with varied sleep architectures, predictions preserve longer stage structures, while most errors occur near stage boundaries or short transient segments.The examples include multi-cycle patterns, little or no Deep/REM sleep, prolonged wakefulness, and frequent Wake–Light alternations.Source: Figure D.2 · p. 22, §D.5 · p. 23

E Sleep Video EigenCAM (SVECAM)

SVECAM separates temporal and spatial channel weighting to visualize where and when ViNUSS attends within an epoch. Example visualizations highlight respiration-related thoraco-abdominal motion and transient larger movements.

E Sleep Video EigenCAM (SVECAM)

  • In the illustrated epoch, saliency concentrates around the thoraco-abdominal region and its temporal curve follows respiration-related periodic motion.The visualization suggests use of breathing-related visual motion without explicit respiratory-waveform reconstruction.Source: Figure E.1 · p. 23, E Sleep Video EigenCAM (SVECAM) · p. 24
  • SVECAM derives temporal and spatial channel weights separately, avoiding the direct EigenCAM approach of flattening temporal and spatial dimensions before SVD.Temporal weights capture co-variation at each spatial location, while spatial weights capture coherent activation within each frame.Source: E Sleep Video EigenCAM (SVECAM) · pp. 23–24
  • The final SVECAM saliency is a channel-weighted activation map followed by ReLU, highlighting temporally informative and spatially coherent activations.This preserves spatiotemporal structure useful for interpreting localized motion in fixed-camera NIR video.Source: E Sleep Video EigenCAM (SVECAM) · p. 24
  • When transient body movements occur, SVECAM highlights larger movements and returns to the thoraco-abdominal region during quieter intervals.The examples suggest use of both localized respiration-related and transient behavioral motion as staging evidence.Source: E Sleep Video EigenCAM (SVECAM) · p. 24
Loading 2609.05550v1…