PaperlayerView exact revision on arXiv

Source-linked AI summary

Freezing of Gait Prediction Under Spatial Occlusion: An IMU-Supervised Cross-Modal Distillation Approach

Chandan Biswas, Aryan Singh, Anabik Pal

arXiv:2609.09826v1cs.CV

TL;DR

FOG prediction needs both the robustness of IMUs and the deployability of video, but video suffers from self-occlusion during turning tasks and wearables constrain deployment. The paper distills a frozen kinematic teacher into a visual model with confidence-adaptive skeletal–pixel fusion. Experiments report lower classification entropy, an event-level F1 increase of 0.1854 over a generic early-fusion baseline, and precise prediction without continuous wearables.

  • Problem

    FOG video prediction can fail under severe turning-induced self-occlusion, while IMU-based assessment is difficult to deploy because lower-limb sensors must remain attached.

    Source: §I · p. 1

  • Method

    The framework distills invariant kinematic topology from an IMU oracle into a visual model that adaptively fuses skeletal and pixel representations using tracking confidence.

    Source: §I · p. 2, C. Probabilistic Dual-Stream Vision… · p. 3, D. Supervised Cross-Modal Subspace… · p. 4

  • Results

    The method reduces classification entropy and increases event-level F1 by 0.1854 versus the generic early-fusion baseline while using exclusively visual input during inference.

    Source: A. Quantitative Analysis · p. 6

  • Takeaways & Limitations

    The results support precise FOG detection from non-encoded video without continuous wearable components under the evaluated turning-in-place setting.

    Source: §VI · p. 8

    The training procedure assumes deterministic temporal synchronisation between hardware and visual signals, and future work targets settings with unknown alignment.

    Source: §VI · p. 8

Abstract

from arXiv · show

Parkinson's disease is a progressive neurodegenerative disorder characterised by gradual deterioration of movement control. Automated freezing-of-gait (FOG) detection supports the objective assessment of gait-related motor impairment. Two common approaches are used for FOG prediction: (i) analysing video recordings of the patient's movements and (ii) analysing data collected using inertial measurement unit (IMU) wearable sensors attached to the patient's lower limbs. Video-based approaches may suffer detection errors during continuous turning-in-place tasks because the lower limbs undergo substantial geometric self-occlusion, degrading pose-estimation accuracy. IMU-based approaches are generally less affected by visual occlusion; however, they are difficult to deploy outside clinical or laboratory settings, as the sensors must be attached securely and remain in place throughout the assessment. Motivated by this, we propose a cross-modal subspace distillation framework to mitigate the limitations of unimodal FOG detection by combining IMU accuracy with video-based practicality. We extract invariant latent topologies from a pre-trained kinematic oracle to structurally supervise a non-encoded visual architecture during training. To resolve periods of severe spatial occlusion, a dual-stream visual model probabilistically fuses skeletal graph nodes and continuous spatial pixels, dynamically shifting reliance to uninterrupted pixel boundaries as joint tracking confidence drops. Evaluated against a public, multi-modal sequence dataset of Parkinson's individuals executing continuous $360^\circ$ turns, empirical results demonstrate that applying sensory boundary topologies strictly mitigates tracking evaluation entropy. Our constrained optimisation confirms that highly precise FOG prediction bounds can be achieved over zero-wearable inference environments.

I. INTRODUCTION

FOG assessment is clinically important, but wearable IMUs are difficult to deploy and video methods can fail during turning-in-place because of lower-limb self-occlusion. The paper addresses this gap with cross-modal supervision and adaptive visual fusion.

  • Objective FOG assessment supports evaluation of motor-symptom progression, while prolonged FOG episodes correlate with higher fall and injury probability.Source: §I · p. 1
  • IMU-based spatial trajectory estimation provides reliable, high-precision FOG assessment but is uncomfortable to wear continuously outside controlled settings.Source: §I · p. 1
  • Video acquisition is easier to deploy, yet turning-in-place causes geometric self-occlusion that lowers joint-coordinate confidence and introduces signal noise.Source: §I · p. 1, Figure 1 · p. 1
  • A classifier without deterministic structural points fails to resolve subtle FOG kinematic characteristics across obscured visual regions.Source: §I · p. 1
  • The proposed pipeline transfers domain properties from one-dimensional spatial kinematics to unannotated spatio-temporal video subsets.Source: §I · p. 2
  • Adaptive fusion uses dynamic skeleton-coordinate confidence to address systematic lower-limb tracking corruption probabilistically.Source: §I · p. 2

B. Latent Space Alignment and Cross-modal Distillation

The framework trains a visual model to approximate an informative kinematic and clinical latent subspace while using only visual input during deployment. It builds on cross-modal distillation and structured representations to compensate for visual information loss.

B. Latent Space Alignment and Cross-modal Distillation

  • Cross-modal distillation transfers distributional properties across heterogeneous modalities to constrain the subordinate representation.Source: B. Latent Space Alignment and… · p. 2
  • Coordinate-based representations such as ST-GCN degrade when joint estimates fall below probabilistic noise bounds.Source: C. Coordinate Representation under… · p. 2
  • The proposed framework trains a vision-and-language model under supervision from an informative inertial kinematic subspace.Source: §III · pp. 2–3
  • Training uses synchronized visual frames, inertial measurements, and encoded clinical metadata, whereas client-side inference has access exclusively to the visual sequence.Source: A. Problem Formulation and… · p. 3

B. The Kinematic Teacher Oracle

The method uses a frozen inertial teacher and a probabilistic dual-stream visual model to transfer kinematic structure while handling occluded joints. Skeletal and pixel representations are fused according to pose-tracking confidence.

B. The Kinematic Teacher Oracle

C. Probabilistic Dual-Stream Vision Model

D. Supervised Cross-Modal Subspace Distillation

The framework supervises visual FOG representations with invariant kinematic and clinical subspaces, using supervised contrastive alignment to handle repeated FOG classes and optimize classification jointly.

Cross-modal supervision

  • The fused vision embedding zV is supervised by a kinematics oracle zI because video alone makes FOG micro-movement boundaries difficult to annotate.The text subspace zT models metadata features in parallel.Source: D. Supervised Cross-Modal Subspace… · p. 3

Supervised alignment

  • Unconditional contrastive alignment can falsely repel distinct sequence segments sharing the same FOG pathology.The method addresses this issue with a multitarget Supervised Contrastive Loss.Source: D. Supervised Cross-Modal Subspace… · p. 3
  • P(i) identifies batch instances whose inference classes match the anchor, allowing similarity alignment to incorporate the number of positives.The formulation uses a temperature-scaled similarity function.Source: D. Supervised Cross-Modal Subspace… · p. 3

Cross-modal supervision

Joint optimization

  • The classifier maps the distilled representation to FOG labels while optimization combines cross-entropy with supervised alignment constraints.The penalty weights λ and γ control the mutual-information constraints transferred from oracle distributions.Source: D. Supervised Cross-Modal Subspace… · p. 4

E. Algorithmic Optimisation Formulation

The optimization procedure trains a visual predictor with fixed auxiliary oracle representations, adaptive confidence-based processing, cross-modal losses, and zero-hardware inference constraints.

Optimization objective

  • The target visual parameter space Θ = {θrgb, θsk, ΘC} is designed to operate independently of inertial measurements during inference.The workflow aggregates the formulations into a batch-wise update process.Source: E. Algorithmic Optimisation Formulation · p. 4

Algorithmic workflow

  • Algorithm 1 processes synchronized multimodal batches while estimating sequence-topology bounds from invariant hardware-oracle constants.It iteratively regulates confidence vectors during parameter evaluation.Source: E. Algorithmic Optimisation Formulation · p. 4
  • The algorithm uses batch size M, training epochs T, learning rate η, temperature τ, and objective weights λ and γ as optimization inputs.It outputs optimized classification parameters Θ.Source: E. Algorithmic Optimisation Formulation · p. 4
  • Auxiliary inertial and text encoders are initialized as constant oracle mappings while the visual parameters are optimized over training epochs.The auxiliary vectors are specified to have zero gradients.Source: E. Algorithmic Optimisation Formulation · p. 4

Occlusion handling

  • During severe occlusion, the adaptive margin uses expected tracking confidence to regulate the representation calculation.The algorithm includes phases for visual encoding, cross-modal cost calculation, supervised target distillation, and weight optimization.Source: E. Algorithmic Optimisation Formulation · p. 4

Zero-hardware inference

  • The framework maps invariant spatial limits from physical hardware and metadata inputs during optimization, then evaluates zero-hardware inference constraints.The deployment formulation shifts representation toward optical derivatives when tracking validation limits fall under occlusion.Source: Figure 2 · p. 5

IV. EXPERIMENTAL SETUP

The evaluation framework specifies the dataset, model configurations, comparative baselines, and metrics used to assess FOG prediction under imbalanced event occurrence.

Evaluation framework

  • The experiments define a dataset, model configurations, comparative baselines, and a metric space tailored to imbalanced FOG events.The setup is intended to validate the cross-modal distillation methodology.Source: §IV · p. 5

A. Dataset and Protocol

The study evaluates synchronized video, IMU, and clinical metadata from Parkinson’s participants performing continuous turns across multiple temporal windows with overlapping segmentation.

Dataset and protocol

  • The public multimodal dataset contains 35 individuals with idiopathic Parkinson’s disease performing 360° turning-in-place sessions lasting two minutes.NFOG-Q and UPDRS-III assessments initialize the clinical metadata input xT.Source: A. Dataset and Protocol · p. 5

Temporal sampling

  • Video is sampled at 30 Hz and IMU data at 128 Hz across window durations Δt ∈ {1.0, 1.5, 2.0, 3.0} seconds.The windows use a 50% uniform sliding overlap.Source: A. Dataset and Protocol · p. 5
  • Each observation window contains a synchronized dimensional tuple of visual and inertial data.The tuple is formalized as the window representation wk(Δt).Source: A. Dataset and Protocol · p. 5

Labels and imbalance

  • A window receives target y(k) = 1 when any contained sequence overlaps an expert-annotated FOG instance.Changing the temporal boundary alters the class distribution because FOG occurrence is intermittent.Source: A. Dataset and Protocol · p. 5

Dataset parameters

  • Table I organizes discrete spatial and inertial sequence parameters across temporal windows using a 50% transition stride.The table conditions dataset parameter distributions on Δt.Source: Table I · p. 5

B. Implementation Details

The evaluation uses independent inertial, pixel, and skeletal embedding models, compares them with unimodal and fusion baselines, and reports imbalance-aware frame- and event-level metrics.

Model implementation

  • The inertial extractor is a frozen three-block 1D CNN projecting inputs to a 512-dimensional latent topology, while visual inputs use pixel and skeletal streams.The skeletal stream estimates 17 joints per frame and encodes their spatial dependencies with an ST-GCN.Source: B. Implementation Details · pp. 5–6

Baselines

  • The baselines comprise RGB-only, Skeleton-Only, Vision Early-Fusion, and a native IMU Oracle representing the attached-sensor upper bound.Skeleton-Only lacks a mechanism for recovering latent kinematics when coordinate confidence is low.Source: C. Baselines · p. 6

Metrics

  • Raw accuracy is discarded because class imbalance allows perpetual normal-walking predictions to appear spuriously strong.The evaluation instead uses MCC, positive-class F1, and AUPRC.Source: D. Evaluation Metrics and Parameters · p. 6

Evaluation protocol

  • Performance is measured at both frame level and event level, with event inference assessing detection over prolonged FOG bouts.Frame-level inference evaluates discrete windows, whereas event-level inference aggregates predictions across a ground-truth temporal event block.Source: D. Evaluation Metrics and Parameters · p. 6

V. RESULTS AND DISCUSSION

The experiments compare the proposed cross-modal distillation model with visual and inertial baselines using threshold-selected FOG metrics. The proposed configuration improves visual prediction and achieves the strongest event-level detection among the reported baselines.

Evaluation design

  • The evaluation compares clustering, frame-level and event-level performance, and feature-mapping stability under spatial occlusion.Thresholds are selected by maximizing positive-class F1 on the validation set.Source: §V · p. 6, A. Quantitative Analysis · p. 6

Oracle reference

  • AUPRC 0.7993 defines the IMU Oracle’s operational upper bound under constant sensor attachment.The Oracle uses the inertial extractor natively.Source: A. Quantitative Analysis · p. 6

Visual baselines

  • MCC = 0.3218 is reported for Skeleton-Only, whose missing target nodes compromise latent representation distances during occluded 360° turns.Early fusion only marginalizes prediction error without effectively bounding the FOG geometry.Source: A. Quantitative Analysis · p. 6

Cross-modal result

  • 0.1854 is the increase in event-level F1 over generic early fusion for the proposed cross-modal distillation configuration.The reported improvement is associated with reduced variance within positive sample margins while inference uses visual input only.Source: A. Quantitative Analysis · p. 6

Event-level performance

  • Event-F1 0.6943 is achieved by the proposed method, outperforming all baseline configurations in bout-level detection.Event-level scores are lower than frame-level scores because missed detections within a bout count as event-level failures.Source: A. Quantitative Analysis · p. 7

B. Sensitivity to Temporal Limits

Temporal-window and spatial-occlusion analyses examine how inference duration and geometric constraints affect the learned representations. The reported results favor a 1.5-second window and multimodal constraints, while showing that topology-only constraints can fragment responses under occlusion.

Temporal limits

  • Evaluation tests temporal windows of Δt ∈ {1.0, 1.5, 2.0, 3.0} seconds to assess sensitivity to observation duration.The analysis targets rapid trembling artifacts and broader gait poses across continuous bounds.Source: B. Sensitivity to Temporal Limits · p. 7
  • F1 = 0.6712 is obtained at Δt = 1.0s, where short windows can split transitional gait sequences and omit completed acceleration limits.At Δt = 3.0s, high-frequency 3−8 Hz signatures are diluted by longer walking sequences and prediction-boundary entropy increases.Source: B. Sensitivity to Temporal Limits · p. 7
  • Δt = 1.5s provides the reported optimal stabilization margin, preserving roughly 4 to 10 sequential FOG oscillation vectors.This setting is described as maximizing convergence for contrastive Oracle projection constraints.Source: B. Sensitivity to Temporal Limits · p. 7

Spatial occlusion

  • Dynamic 360° occlusion makes deterministic graph evaluation mathematically suboptimal, whereas expected coordinate continuity bounds sequences when optical joint overlap is low.The stated limiting behavior is limα→0 zV ≈ vrgb.Source: C. Visual Interpretability under… · p. 7

Spatial interpretability

  • Topology-only constraints can produce fragmented spatial responses and reduced localization accuracy in occluded regions with missing limb trajectories.Cross-modal constraints instead concentrate activations around anatomically relevant ankle and lower-limb regions.Source: C. Visual Interpretability under… · p. 7

Modal ablation

  • λ = 0 degrades the vision manifold toward occlusion-sensitive baseline bounds, while optimal cluster discrimination is reported near λ ≈ γ ≈ 1.0.Setting γ = 0 removes textual contextualization and patient-severity metadata from the evaluation constraints.Source: D. Modal Penalty Ablation and Manifold… · p. 7

Manifold topology

E. Computational Complexity for Edge Deployment

The deployment design reduces operational overhead by discarding training-only parameters and evaluating the visual model through spatial convolutions. Inference is reported as meeting zero-wearable goals, while the broader framework targets occlusion-robust FOG prediction but assumes deterministic training-time synchronisation.

  • Eliminating concurrent measurement synchronisation arrays reduces execution variables relative to multi-sensor data handling.Source: E. Computational Complexity for Edge… · p. 8
  • The inference environment discards the dual θI and θT parameters used during structural generation and evaluates XV through spatial convolutions.Source: E. Computational Complexity for Edge… · p. 8
  • 6.91 milliseconds is the reported estimation time under the stated timing limits.The timing measurement uses the inference setup described for Table IV.Source: E. Computational Complexity for Edge… · p. 8, Table IV · p. 8
  • The fused visual representation uses expected spatial coordinate confidence to mitigate noise propagation during lower-limb occlusion.Source: §VI · p. 8
  • The framework approaches the hardware-sensor apex boundary while targeting precise FOG detection without continuous wearable components.Source: §VI · p. 8
  • Training assumes deterministic temporal synchronisation between hardware and visual signals, which future work proposes to relax.Source: §VI · p. 8

Related papers

Loading 2609.09826v1…