PaperlayerView exact revision on arXiv

Source-linked AI summary

Freezing of Gait Prediction Under Spatial Occlusion: An IMU-Supervised Cross-Modal Distillation Approach

Chandan Biswas, Aryan Singh, Anabik Pal

arXiv:2609.09826v1cs.CV

TL;DR

FOG prediction needs both the robustness of IMUs and the deployability of video, but video suffers from self-occlusion during turning tasks and wearables constrain deployment. The paper distills a frozen kinematic teacher into a visual model with confidence-adaptive skeletal–pixel fusion. Experiments report lower classification entropy, an event-level F1 increase of 0.1854 over a generic early-fusion baseline, and precise prediction without continuous wearables.

  • Problem

    FOG video prediction can fail under severe turning-induced self-occlusion, while IMU-based assessment is difficult to deploy because lower-limb sensors must remain attached.

  • Method

    The framework distills invariant kinematic topology from an IMU oracle into a visual model that adaptively fuses skeletal and pixel representations using tracking confidence.

  • Results

    The method reduces classification entropy and increases event-level F1 by 0.1854 versus the generic early-fusion baseline while using exclusively visual input during inference.

  • Takeaways & Limitations

    The results support precise FOG detection from non-encoded video without continuous wearable components under the evaluated turning-in-place setting.

  • Takeaways & Limitations

    The training procedure assumes deterministic temporal synchronisation between hardware and visual signals, and future work targets settings with unknown alignment.

Abstract

from arXiv · show

Parkinson's disease is a progressive neurodegenerative disorder characterised by gradual deterioration of movement control. Automated freezing-of-gait (FOG) detection supports the objective assessment of gait-related motor impairment. Two common approaches are used for FOG prediction: (i) analysing video recordings of the patient's movements and (ii) analysing data collected using inertial measurement unit (IMU) wearable sensors attached to the patient's lower limbs. Video-based approaches may suffer detection errors during continuous turning-in-place tasks because the lower limbs undergo substantial geometric self-occlusion, degrading pose-estimation accuracy. IMU-based approaches are generally less affected by visual occlusion; however, they are difficult to deploy outside clinical or laboratory settings, as the sensors must be attached securely and remain in place throughout the assessment. Motivated by this, we propose a cross-modal subspace distillation framework to mitigate the limitations of unimodal FOG detection by combining IMU accuracy with video-based practicality. We extract invariant latent topologies from a pre-trained kinematic oracle to structurally supervise a non-encoded visual architecture during training. To resolve periods of severe spatial occlusion, a dual-stream visual model probabilistically fuses skeletal graph nodes and continuous spatial pixels, dynamically shifting reliance to uninterrupted pixel boundaries as joint tracking confidence drops. Evaluated against a public, multi-modal sequence dataset of Parkinson's individuals executing continuous $360^\circ$ turns, empirical results demonstrate that applying sensory boundary topologies strictly mitigates tracking evaluation entropy. Our constrained optimisation confirms that highly precise FOG prediction bounds can be achieved over zero-wearable inference environments.

I. INTRODUCTION

FOG assessment is clinically important, but wearable IMUs are difficult to deploy and video methods can fail during turning-in-place because of lower-limb self-occlusion. The paper addresses this gap with cross-modal supervision and adaptive visual fusion.

  • Objective FOG assessment supports evaluation of motor-symptom progression, while prolonged FOG episodes correlate with higher fall and injury probability.
  • IMU-based spatial trajectory estimation provides reliable, high-precision FOG assessment but is uncomfortable to wear continuously outside controlled settings.
  • Video acquisition is easier to deploy, yet turning-in-place causes geometric self-occlusion that lowers joint-coordinate confidence and introduces signal noise.
  • A classifier without deterministic structural points fails to resolve subtle FOG kinematic characteristics across obscured visual regions.
  • The proposed pipeline transfers domain properties from one-dimensional spatial kinematics to unannotated spatio-temporal video subsets.
  • Adaptive fusion uses dynamic skeleton-coordinate confidence to address systematic lower-limb tracking corruption probabilistically.

B. Latent Space Alignment and Cross-modal Distillation

The framework trains a visual model to approximate an informative kinematic and clinical latent subspace while using only visual input during deployment. It builds on cross-modal distillation and structured representations to compensate for visual information loss.

  • B. Latent Space Alignment and Cross-modal Distillation: Cross-modal distillation transfers distributional properties across heterogeneous modalities to constrain the subordinate representation.
  • B. Latent Space Alignment and Cross-modal Distillation: Coordinate-based representations such as ST-GCN degrade when joint estimates fall below probabilistic noise bounds.
  • B. Latent Space Alignment and Cross-modal Distillation: The proposed framework trains a vision-and-language model under supervision from an informative inertial kinematic subspace.
  • B. Latent Space Alignment and Cross-modal Distillation: Training uses synchronized visual frames, inertial measurements, and encoded clinical metadata, whereas client-side inference has access exclusively to the visual sequence.

B. The Kinematic Teacher Oracle

The method uses a frozen inertial teacher and a probabilistic dual-stream visual model to transfer kinematic structure while handling occluded joints. Skeletal and pixel representations are fused according to pose-tracking confidence.

  • B. The Kinematic Teacher Oracle: A pre-trained inertial network provides a discriminative kinematic representation that serves as the teacher reference.
  • C. Probabilistic Dual-Stream Vision Model: Visual self-occlusion makes deterministic skeletal-to-label mappings unreliable, motivating probabilistic visual feature extraction.
  • C. Probabilistic Dual-Stream Vision Model: The visual model combines dense-pixel and skeletal embeddings produced by a 3D convolutional network and a graph convolutional network.
  • C. Probabilistic Dual-Stream Vision Model: Skeletal outputs are modeled as noisy because pose tracking fails systematically under self-occlusion.
  • C. Probabilistic Dual-Stream Vision Model: The unified visual embedding is an expected-confidence-weighted mixture of a pixel stream resistant to missing nodes and a skeletal stream susceptible to occlusion.
  • C. Probabilistic Dual-Stream Vision Model: As tracking validity decreases, the learned fusion coefficient decays toward zero and shifts inference toward uninterrupted pixel topologies.

D. Supervised Cross-Modal Subspace Distillation

The framework supervises visual FOG representations with invariant kinematic and clinical subspaces, using supervised contrastive alignment to handle repeated FOG classes and optimize classification jointly.

  • Cross-modal supervision: The fused vision embedding zV is supervised by a kinematics oracle zI because video alone makes FOG micro-movement boundaries difficult to annotate.The text subspace zT models metadata features in parallel.
  • Supervised alignment: Unconditional contrastive alignment can falsely repel distinct sequence segments sharing the same FOG pathology.The method addresses this issue with a multitarget Supervised Contrastive Loss.
  • Supervised alignment: P(i) identifies batch instances whose inference classes match the anchor, allowing similarity alignment to incorporate the number of positives.The formulation uses a temperature-scaled similarity function.
  • Cross-modal supervision: The vision manifold is also aligned with the text component to represent varying clinical states without class collisions.The text component supplies a parallel latent-space constraint.
  • Joint optimization: The classifier maps the distilled representation to FOG labels while optimization combines cross-entropy with supervised alignment constraints.The penalty weights λ and γ control the mutual-information constraints transferred from oracle distributions.

E. Algorithmic Optimisation Formulation

The optimization procedure trains a visual predictor with fixed auxiliary oracle representations, adaptive confidence-based processing, cross-modal losses, and zero-hardware inference constraints.

  • Optimization objective: The target visual parameter space Θ = {θrgb, θsk, ΘC} is designed to operate independently of inertial measurements during inference.The workflow aggregates the formulations into a batch-wise update process.
  • Algorithmic workflow: Algorithm 1 processes synchronized multimodal batches while estimating sequence-topology bounds from invariant hardware-oracle constants.It iteratively regulates confidence vectors during parameter evaluation.
  • Algorithmic workflow: The algorithm uses batch size M, training epochs T, learning rate η, temperature τ, and objective weights λ and γ as optimization inputs.It outputs optimized classification parameters Θ.
  • Algorithmic workflow: Auxiliary inertial and text encoders are initialized as constant oracle mappings while the visual parameters are optimized over training epochs.The auxiliary vectors are specified to have zero gradients.
  • Occlusion handling: During severe occlusion, the adaptive margin uses expected tracking confidence to regulate the representation calculation.The algorithm includes phases for visual encoding, cross-modal cost calculation, supervised target distillation, and weight optimization.
  • Zero-hardware inference: The framework maps invariant spatial limits from physical hardware and metadata inputs during optimization, then evaluates zero-hardware inference constraints.The deployment formulation shifts representation toward optical derivatives when tracking validation limits fall under occlusion.

IV. EXPERIMENTAL SETUP

The evaluation framework specifies the dataset, model configurations, comparative baselines, and metrics used to assess FOG prediction under imbalanced event occurrence.

  • Evaluation framework: The experiments define a dataset, model configurations, comparative baselines, and a metric space tailored to imbalanced FOG events.The setup is intended to validate the cross-modal distillation methodology.

A. Dataset and Protocol

The study evaluates synchronized video, IMU, and clinical metadata from Parkinson’s participants performing continuous turns across multiple temporal windows with overlapping segmentation.

  • Dataset and protocol: The public multimodal dataset contains 35 individuals with idiopathic Parkinson’s disease performing 360° turning-in-place sessions lasting two minutes.NFOG-Q and UPDRS-III assessments initialize the clinical metadata input xT.
  • Temporal sampling: Video is sampled at 30 Hz and IMU data at 128 Hz across window durations Δt ∈ {1.0, 1.5, 2.0, 3.0} seconds.The windows use a 50% uniform sliding overlap.
  • Temporal sampling: Each observation window contains a synchronized dimensional tuple of visual and inertial data.The tuple is formalized as the window representation wk(Δt).
  • Labels and imbalance: A window receives target y(k) = 1 when any contained sequence overlaps an expert-annotated FOG instance.Changing the temporal boundary alters the class distribution because FOG occurrence is intermittent.
  • Dataset parameters: Table I organizes discrete spatial and inertial sequence parameters across temporal windows using a 50% transition stride.The table conditions dataset parameter distributions on Δt.

B. Implementation Details

The evaluation uses independent inertial, pixel, and skeletal embedding models, compares them with unimodal and fusion baselines, and reports imbalance-aware frame- and event-level metrics.

  • Model implementation: The inertial extractor is a frozen three-block 1D CNN projecting inputs to a 512-dimensional latent topology, while visual inputs use pixel and skeletal streams.The skeletal stream estimates 17 joints per frame and encodes their spatial dependencies with an ST-GCN.
  • Baselines: The baselines comprise RGB-only, Skeleton-Only, Vision Early-Fusion, and a native IMU Oracle representing the attached-sensor upper bound.Skeleton-Only lacks a mechanism for recovering latent kinematics when coordinate confidence is low.
  • Metrics: Raw accuracy is discarded because class imbalance allows perpetual normal-walking predictions to appear spuriously strong.The evaluation instead uses MCC, positive-class F1, and AUPRC.
  • Evaluation protocol: Performance is measured at both frame level and event level, with event inference assessing detection over prolonged FOG bouts.Frame-level inference evaluates discrete windows, whereas event-level inference aggregates predictions across a ground-truth temporal event block.

V. RESULTS AND DISCUSSION

The experiments compare the proposed cross-modal distillation model with visual and inertial baselines using threshold-selected FOG metrics. The proposed configuration improves visual prediction and achieves the strongest event-level detection among the reported baselines.

  • Evaluation design: The evaluation compares clustering, frame-level and event-level performance, and feature-mapping stability under spatial occlusion.Thresholds are selected by maximizing positive-class F1 on the validation set.
  • Oracle reference: AUPRC 0.7993 defines the IMU Oracle’s operational upper bound under constant sensor attachment.The Oracle uses the inertial extractor natively.
  • Visual baselines: MCC = 0.3218 is reported for Skeleton-Only, whose missing target nodes compromise latent representation distances during occluded 360° turns.Early fusion only marginalizes prediction error without effectively bounding the FOG geometry.
  • Cross-modal result: 0.1854 is the increase in event-level F1 over generic early fusion for the proposed cross-modal distillation configuration.The reported improvement is associated with reduced variance within positive sample margins while inference uses visual input only.
  • Event-level performance: Event-F1 0.6943 is achieved by the proposed method, outperforming all baseline configurations in bout-level detection.Event-level scores are lower than frame-level scores because missed detections within a bout count as event-level failures.

B. Sensitivity to Temporal Limits

Temporal-window and spatial-occlusion analyses examine how inference duration and geometric constraints affect the learned representations. The reported results favor a 1.5-second window and multimodal constraints, while showing that topology-only constraints can fragment responses under occlusion.

  • Temporal limits: Evaluation tests temporal windows of Δt ∈ {1.0, 1.5, 2.0, 3.0} seconds to assess sensitivity to observation duration.The analysis targets rapid trembling artifacts and broader gait poses across continuous bounds.
  • Temporal limits: F1 = 0.6712 is obtained at Δt = 1.0s, where short windows can split transitional gait sequences and omit completed acceleration limits.At Δt = 3.0s, high-frequency 3−8 Hz signatures are diluted by longer walking sequences and prediction-boundary entropy increases.
  • Temporal limits: Δt = 1.5s provides the reported optimal stabilization margin, preserving roughly 4 to 10 sequential FOG oscillation vectors.This setting is described as maximizing convergence for contrastive Oracle projection constraints.
  • Spatial occlusion: Dynamic 360° occlusion makes deterministic graph evaluation mathematically suboptimal, whereas expected coordinate continuity bounds sequences when optical joint overlap is low.The stated limiting behavior is limα→0 zV ≈ vrgb.
  • Spatial interpretability: Topology-only constraints can produce fragmented spatial responses and reduced localization accuracy in occluded regions with missing limb trajectories.Cross-modal constraints instead concentrate activations around anatomically relevant ankle and lower-limb regions.
  • Modal ablation: λ = 0 degrades the vision manifold toward occlusion-sensitive baseline bounds, while optimal cluster discrimination is reported near λ ≈ γ ≈ 1.0.Setting γ = 0 removes textual contextualization and patient-severity metadata from the evaluation constraints.
  • Manifold topology: λ = 1.0 and γ = 1.0 isolate independent positive feature mappings in the 512-dimensional sequence space and reduce latent-phase geometric ambiguity.Figure 3 uses t-SNE projections to visualize the constrained and unconstrained representations.

E. Computational Complexity for Edge Deployment

The deployment design reduces operational overhead by discarding training-only parameters and evaluating the visual model through spatial convolutions. Inference is reported as meeting zero-wearable goals, while the broader framework targets occlusion-robust FOG prediction but assumes deterministic training-time synchronisation.

  • Eliminating concurrent measurement synchronisation arrays reduces execution variables relative to multi-sensor data handling.
  • The inference environment discards the dual θI and θT parameters used during structural generation and evaluates XV through spatial convolutions.
  • 6.91 milliseconds is the reported estimation time under the stated timing limits.The timing measurement uses the inference setup described for Table IV.
  • The fused visual representation uses expected spatial coordinate confidence to mitigate noise propagation during lower-limb occlusion.
  • The framework approaches the hardware-sensor apex boundary while targeting precise FOG detection without continuous wearable components.
  • Training assumes deterministic temporal synchronisation between hardware and visual signals, which future work proposes to relax.
Loading 2609.09826v1…