PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervised Cross-Modal Feature Alignment for Zero-Wearable Freezing of Gait Detection in Parkinsonism

Aryan Singh, Chandan Biswas

arXiv:2609.08317v1cs.CV

TL;DR

Wearable IMUs support FoG detection but limit continuous deployment, while visual tracking is impaired by turning-related self-occlusion. The paper aligns a visual skeleton model with IMU and clinical-text oracles during training; its distilled representation reports 85.5% accuracy and 90.6% specificity.

  • Problem

    Wearable IMU attachment constrains continuous monitoring, while self-occlusion during turning degrades visual skeletal tracking for FoG detection.

    Source: §I · p. 1

  • Method

    A supervised cross-modal distillation framework trains a visual skeleton model to align with frozen IMU and clinical-text oracle spaces, retaining vision-only inference.

    Source: §I · p. 1

  • Results

    85.5% accuracy and 90.6% specificity are reported for the distilled representation.

    Source: §VII · p. 9

  • Takeaways & Limitations

    The visual student captures a substantial portion of inertial discriminative variance without requiring IMU measurements at deployment.

    Source: A. Overall Detection Performance and… · p. 8

    Evaluation uses a single 35-subject cohort, and the authors call for multicentre validation to assess broader generalization.

    Source: §VII · p. 9

Abstract

from arXiv · show

Objective assessment of Freezing of Gait (FoG) in Parkinson's disease (PD) relies predominantly on wearable Inertial Measurement Units (IMUs). While IMUs provide optimal kinematic precision, mandatory sensor attachment restricts continuous clinical deployment. Conversely, unobtrusive vision-based alternatives suffer substantial classification errors during turning-in-place tasks, where geometric self-occlusion degrades deterministic skeletal coordinates and obscures the high-frequency precursors required for FoG detection. To resolve these physical observation limits, we propose a supervised cross-modal subspace distillation framework. During optimisation, pre-trained kinematic data from IMU sensors and contextual clinical metadata act as oracles to guide a deployable visual architecture. By incorporating joint velocity and acceleration derivatives, utilising a confidence-based gating mechanism, the visual model mitigates some of the tracking errors during occlusion events. Empirical evaluations confirm this latent alignment transfers the predictive fidelity of hardware sensors directly into the visual representation, yielding $85.5\%$ accuracy, and $82.4\%$ balanced accuracy. All the while maintaining a vision only model at inference.

I. INTRODUCTION

FoG monitoring faces a trade-off between wearable IMU precision and the deployment constraints of attached sensors, while visual tracking degrades during turning-related self-occlusion. The paper positions its approach against wearable and vision-based detection methods and prior cross-modal transfer work.

I. INTRODUCTION

  • FoG is a transient inability to initiate or sustain stepping, often occurring during gait initiation or sharp turns and increasing fall risk,.Objective, continuous monitoring is identified as important for assessing disease progression and therapeutic efficacy.Source: §I · p. 1
  • Wearable IMUs provide high-precision detection,, but attachment and maintenance constrain long-term monitoring.Vision offers an unobtrusive alternative, yet turning-related self-occlusion degrades skeletal coordinates and obscures 3–8 Hz FoG variations,.Source: §I · p. 1

A. Kinematic and Wearable-Sensor FoG Detection

  • Wearable FoG methods evolved from freeze-index spectral analysis to machine learning, but sensor models can generalize poorly from induced laboratory sequences to real-world settings,,,.The freeze index compares power in the 3–8 Hz freeze band with the 0.5–3 Hz locomotor band.Source: A. Kinematic and Wearable-Sensor FoG… · p. 2

B. Vision-Based Gait Analysis and Cross-Modal Transfer

  • Skeleton-encoder inference used purely visual skeletal keypoints and achieved high specificity but moderate sensitivity; Kondo et al. found monocular-video 3D pose robust to camera-angle variance, yet occlusion degraded target keypoints, causing coordinate-extraction failures.Kumar et al. found covariance matrices of skeletal-joint trajectories in depth imagery to be highly discriminative motion cues.Source: B. Vision-Based Gait Analysis and… · p. 2

C. Public Benchmark Datasets

The section situates the approach in FoG benchmark data and three foundations: skeletal graph modelling, supervised contrastive learning, and privileged information.

C. Public Benchmark Datasets

  • The study’s dataset synchronizes 30 Hz lower-limb video with 128 Hz inertial measurements across 35 subjects during continuous turning-in-place tasks, including 1,611 seconds of annotated FoG episodes.Earlier public resources emphasized kinematic or physiological data, including DAPHNet and PhysioNet,.Source: C. Public Benchmark Datasets · p. 2

A. Spatial-Temporal Graph Convolutional Networks (ST-GCN)

  • ST-GCN models joints and their physical and temporal connections as a graph, applying spatial and temporal convolutions to represent motion.The graph representation focuses on joint kinematics rather than appearance or background, supporting the paper’s skeletal visual stream.Source: A. Spatial-Temporal Graph… · pp. 2–3

B. Supervised Contrastive Representation Learning (SupCon)

  • SupCon uses class labels to attract same-class batch instances, avoiding the false-negative repulsion that standard unsupervised contrastive learning can impose.This provides the objective for cross-modal alignment.Source: B. Supervised Contrastive… · p. 3

C. Learning Using Privileged Information (LUPI)

  • LUPI treats training-only high-fidelity modalities as expert oracles that regularize a deployable model, here using IMU and clinical text to constrain visual parameters.The framework uses those oracle boundaries to transfer physical and contextual structure into the visual representation.Source: C. Learning Using Privileged… · p. 3

IV. PROPOSED APPROACH

The approach uses synchronized video, inertial, clinical, and label data to train a visual encoder against frozen kinematic and contextual oracle spaces. At deployment, classification uses visual vectors alone.

IV. PROPOSED APPROACH

  • The framework trains a motion-augmented ST-GCN to project occluded video into latent spaces defined by an IMU oracle and a clinical text oracle, then classifies from visual vectors alone.The physical and textual streams are bypassed at inference.Source: §IV · p. 3

A. Data Instances and Problem Formulation

  • Each synchronized instance includes visual frames, inertial measurements, clinical attributes, and a binary FoG label, and training seeks to align visual embeddings with the informative oracle subspaces.The clinical attributes include examples such as age and UPDRS scores.Source: A. Data Instances and Problem… · p. 3

B. Kinematic and Contextual Oracles

  • A pretrained IMU network maps inertial inputs to a p-dimensional space, while the temporally invariant clinical-text embedding supplies a global inter-subject severity prior rather than within-session temporal supervision.Both oracle parameter sets remain frozen during multi-objective training.Source: B. Kinematic and Contextual Oracles · p. 3

C. Generative Prompting and Decoding Strategy

The contextual oracle converts structured clinical metadata into a fixed-length embedding, while the visual student uses confidence-filtered, motion-augmented skeleton sequences. Figure 1 shows the oracles used during training and the video-only inference branch.

C. Generative Prompting and Decoding Strategy

  • Clinical attributes such as age, gender, disease duration, medication dose, and severity scores are serialized into a standardized structured natural-language prompt.The prompt template is held constant across sessions.Source: C. Generative Prompting and Decoding… · p. 3
  • An example prompt describes a 56-year-old female Parkinson’s disease patient with 6 years’ disease duration, levodopa-equivalent daily dose 800 mg, MDS-UPDRS part III score 16, New Freezing of Gait Questionnaire score 20, and Hoehn and Yahr stage 3.Source: C. Generative Prompting and Decoding… · pp. 3–4
  • Figure 1 shows frozen clinical-text and IMU oracles guiding a visual branch through class-conditional contrastive alignment, with video-only classification at inference.The figure identifies 30 Hz video, 128 Hz IMU, and shared p-dimensional embeddings.Source: Figure 1 · p. 4
  • Frozen BioClinicalBERT processes the prompt, and pooled terminal-layer token states yield a p-dimensional embedding used as a clinical severity-prior oracle [24].The embedding is auxiliary supervision during SupCon distillation.Source: C. Generative Prompting and Decoding… · p. 4

D. Motion-Augmented Skeleton Student

  • The skeleton input drops tracking nodes below a confidence threshold, resamples surviving frames, and represents retained joint trajectories with velocity and acceleration derivatives.These steps address deterministic skeletal graph susceptibility to turning-related self-occlusion.Source: D. Motion-Augmented Skeleton Student · p. 4
  • An ST-GCN computes the deployable visual encoding from the confidence-filtered, motion-augmented skeleton sequence.The graph network operates over the topological joints of the extracted pose graph.Source: D. Motion-Augmented Skeleton Student · p. 4
  • Random planar rotation, scaling, mirroring, coordinate jitter, and joint dropout perturb the input during optimization to regularize against residual tracking noise from self-occlusion.The ST-GCN student parameters are trainable.Source: D. Motion-Augmented Skeleton Student · p. 4

E. Supervised Cross-Modal Subspace Distillation

The method uses class-conditional supervised contrastive alignment to guide visual representations toward kinematic and clinical oracle subspaces. A classifier then maps the distilled representation to target cluster labels.

E. Supervised Cross-Modal Subspace Distillation

  • Class-conditional alignment with the kinematic oracle pulls same-class representations together while avoiding false negatives from distinct sequences with equivalent target pathology.The matching index set P(i) identifies same-class examples in a batch; the temperature parameter τ controls the distribution.Source: E. Supervised Cross-Modal Subspace… · pp. 4–5
  • The visual representation is also aligned to the clinical-text subspace using the same objective with text embeddings substituted for inertial embeddings.Because text embeddings are temporally static, restricting alignment to matching classes avoids collapsing distinct FoG and normal-gait states for a subject.Source: E. Supervised Cross-Modal Subspace… · p. 5
  • The class-conditional clinical alignment conditions the dynamic visual subspace on a global severity prior, including baseline disease severity such as UPDRS-III.The passage describes this as encouraging visual clustering of dynamic FoG morphologies relative to cohort baseline severity.Source: E. Supervised Cross-Modal Subspace… · p. 5
  • A classifier matrix maps the distilled representation to target cluster labels, and the overall objective combines alignment with the oracle subspaces.The combination parameters λ and γ set the relative importance of alignment.Source: E. Supervised Cross-Modal Subspace… · p. 5

V. EXPERIMENTAL SETUP

The experiments assess whether a motion-augmented visual skeleton network guided by non-visual oracles during training can approximate sensor-based classification boundaries at inference. Evaluation uses a public multimodal FoG dataset and subject-disjoint, stratified five-fold partitions.

A. Dataset and Partitioning Protocol

  • The dataset contains 35 people with idiopathic Parkinson’s disease and 77 sessions, averaging 2.2 sessions per subject; six sessions without FoG were withheld by the original authors.Subjects were clinically evaluated at Hoehn and Yahr stages 2–4.Source: A. Dataset and Partitioning Protocol · p. 5
  • The protocol records lower-limb video at 30 Hz during two minutes of alternating 360° turns, with synchronized shank IMU data sampled at 128 Hz.The camera framing intentionally introduces the geometric self-occlusion examined in the study.Source: A. Dataset and Partitioning Protocol · p. 5
  • FoG episodes were annotated by two movement-disorder specialists, and the dataset contains 1,611 seconds of annotated FoG occurrences.The mean episode duration per subject was 3.0 seconds (±2.9 SD), while 36% of subjects had no FoG during the recorded task.Source: A. Dataset and Partitioning Protocol · p. 6
  • Subject-disjoint five-fold partitions stratify subjects by FoG prevalence, with seven subjects per fold and positive-class prevalence ranging from 16.9% to 41.8%.This fold imbalance motivates using metrics such as F-score and AUPRC rather than standard accuracy alone.Source: A. Dataset and Partitioning Protocol · p. 6, Table I · p. 6

B. Implementation Details and Optimisation Protocol

The workflow constructs separate kinematic, clinical-text, and visual latent spaces, with the visual representation intended for deployment. It uses frozen kinematic and text encoders to guide the visual ST-GCN during training.

B. Implementation Details and Optimisation Protocol

  • The latent spaces use a target projection dimension of 256, and optimization uses mini-batches of 128 with SupCon temperature 0.07.The combination constants λ and γ are selected by grid search; Adam uses an initial learning rate of 5 × 10^-4 and weight decay of 10^-4.Source: B. Implementation Details and… · pp. 6–7
  • A 1D CNN trained on inertial sequences with class-weighted binary cross-entropy provides a frozen kinematic oracle, while frozen BioClinicalBERT encodes clinical prompts.The deployable visual input uses pose graphs encoded by an ST-GCN.Source: B. Implementation Details and… · p. 6
  • The model is implemented in Python 3.12 and PyTorch 2.13.0, with optimization and inference run on an NVIDIA RTX A5500 GPU with 24 GB VRAM.The implementation uses CUDA 13.2 libraries for backend acceleration.Source: B. Implementation Details and… · p. 6

C. Baselines and Feature Configurations

The evaluation compares distilled vision with a video-only baseline, individual train-only oracles, and feature concatenations. The proposed visual representation uses cross-modal alignment during training and discards oracle modalities at inference.

C. Baselines and Feature Configurations

  • The video-only baseline trains the ST-GCN with class-weighted cross-entropy and no kinematic or clinical-text oracle involvement during training or inference.The classification head is an ℓ2-regularized logistic regression module across primary evaluations.Source: C. Baselines and Feature Configurations · p. 7
  • The proposed distilled vision representation is trained with cross-modal supervised contrastive alignment and uses only visual input at inference.This configuration is evaluated alongside the independently trained video-only baseline.Source: C. Baselines and Feature Configurations · p. 7
  • The kinematic oracle is evaluated independently as a train-only configuration and serves as the theoretical upper bound for kinematic detection.Source: C. Baselines and Feature Configurations · p. 7
  • The contextual oracle evaluates frozen clinical-text embeddings independently, while feature concatenation combines two or more frozen modalities before classification.Performance is based on pooled out-of-fold estimates from subject-disjoint five-fold cross-validation.Source: C. Baselines and Feature Configurations · p. 7

D. Evaluation Metrics

Because normal-gait windows greatly outnumber FoG windows, the section evaluates performance with metrics that account for class imbalance, including threshold-independent ROC-AUC.

D. Evaluation Metrics

  • Normal gait windows dominate the sample, so a majority-class classifier could achieve deceptively high accuracy while missing the minority FoG episodes.The text therefore treats standard accuracy alone as mathematically suboptimal for this imbalanced setting.Source: D. Evaluation Metrics · p. 7
  • The evaluation derives TP, TN, FP, and FN from a confusion matrix at a specified operational threshold.These quantities underpin the discrete sensitivity and specificity measures.Source: D. Evaluation Metrics · p. 7
  • Balanced accuracy is the unweighted mean of sensitivity and specificity, so poor performance on minority-class FoG directly lowers the score.Sensitivity measures correctly identified FoG events; specificity measures correctly classified normal gait.Source: D. Evaluation Metrics · p. 7
  • ROC-AUC is the primary continuous, threshold-independent metric, measuring how well the classifier ranks FoG instances above normal-gait instances across thresholds.It integrates true-positive rate against false-positive rate and is independent of class-distribution imbalance.Source: D. Evaluation Metrics · p. 7

VI. RESULTS AND DISCUSSION

Results use pooled out-of-fold probabilities from subject-disjoint five-fold validation, with a single global threshold across 35 unseen subjects. The distilled vision-only model is compared with prior single-camera work and privileged-oracle configurations, with gains and trade-offs across metrics.

VI. RESULTS AND DISCUSSION

  • Pooled out-of-fold probabilities enforce one global operational threshold across 35 unseen subjects, rather than separate thresholds for patient strata.The protocol uses subject-disjoint five-fold cross-validation; the authors state that this mimics clinical deployment and avoids patient-specific threshold calibration.Source: §VI · p. 7
  • The classification head is an ℓ2-regularised logistic-regression module unless otherwise stated.The feature configurations are defined elsewhere in the paper.Source: §VI · p. 7

A. Overall Detection Performance and Ablation

  • The distilled vision-only configuration improves accuracy by 3.1%, balanced accuracy by 1.1%, and specificity by 6.4% versus the undistilled vision baseline.The reported trade-off is lower sensitivity (78.3% to 74.2%) and ROC-AUC (88.5% to 84.9%) for vision-only inference.Source: A. Overall Detection Performance and… · p. 8
  • The fused representation zV,dist ⊕zI reaches 89.2% ROC-AUC versus 88.2% for baseline fusion, while zV,dist ⊕zI ⊕zT has the highest balanced accuracy, 84.7%, and 90.8% ROC-AUC.These configurations retain privileged streams at deployment, unlike the deployable vision-only configuration.Source: A. Overall Detection Performance and… · p. 8
  • 84.9% AUC for the distilled vision-only model approaches the kinematic oracle's 88.8%, while the contextual oracle alone reaches 54.7%.Figure 3 compares the vision-only model with the train-only oracles under the same logistic-regression head.Source: Figure 3 · p. 8, A. Overall Detection Performance and… · p. 8

B. Comparative Evaluation with Prior Formulations

  • 85.5% accuracy, 74.2% sensitivity, and 90.6% specificity are reported for the proposed single-camera model; versus Tian et al., accuracy rises 5.9 percentage points and sensitivity 23.1 percentage points.Specificity is 90.6% versus 89.1% for Tian et al.; the text cautions that dataset populations and evaluation protocols vary.Source: B. Comparative Evaluation with Prior… · p. 8

C. Robustness Across Classification Subspaces

Across classification heads, the distilled visual representation retains stable Average Precision near the IMU oracle, although linear ROC-AUC is lower than with non-linear heads. The conclusion reports 85.5% accuracy and 90.6% specificity, while noting cohort, task, and training-data constraints.

C. Robustness Across Classification Subspaces

  • Robustness tests vary the classification head while keeping encoder parameters, temporal window bounds, and normalisation scaling fixed.The alternatives include a class-weighted MLP and LightGBM, XGBoost, and CatBoost ensembles.Source: C. Robustness Across Classification… · p. 8
  • 84.9% ROC-AUC for ℓ2-LR rises to 91.0% for CatBoost and 90.5% for MLP; visual AP remains 82.6%–85.0% versus IMU 83.6%–86.2%.The authors report that AP remains stable across mapping functions and correlates closely with the privileged IMU oracle.Source: C. Robustness Across Classification… · p. 8, Table IV · p. 9, Figure 4 · p. 9

VII. CONCLUSION AND FUTURE WORK

  • The framework constrains an ST-GCN skeleton model to approximate pre-trained kinematic and contextual oracle latent topologies.The conclusion describes this as transferring hardware-level discriminative features into zero-wearable inference.Source: §VII · pp. 8–9
  • 85.5% accuracy and 90.6% specificity are reported for the distilled representation in the zero-wearable setting.The conclusion says it effectively bypasses geometric degradation of unconstrained skeletal tracking.Source: §VII · p. 9
  • Evaluation uses a single 35-subject cohort and discrete event detection, so broader pathological generalisation and temporal anticipation remain unvalidated.The paper identifies multicentre validation and continuous temporal forecasting as future work.Source: §VII · pp. 9–10
  • Inference is sensor-independent, but optimisation depends on privileged oracles and synchronised multimodal datasets.The authors propose asynchronous cross-modal distillation to relax the synchronisation requirement.Source: §VII · pp. 9–10
Loading 2609.08317v1…