PaperlayerView exact revision on arXiv

Source-linked AI summary

Kinematic Knowledge Maps for Pattern Alignment: Structured Latent Representational Learning in Multimodal Gait Analysis

Chen Dong, He Zonglin, Cheung Kenneth M. C

arXiv:2608.20969v1cs.CV

TL;DR

Weakly aligned clinical modalities and limited structured interpretability complicate multimodal gait-based AIS screening. ScoliDetect addresses this with a fixed-index kinematic knowledge map, temporal registration, and multimodal fusion, improving external discrimination while supporting factor-level explanations tied to gait phases and skeletal indices.

  • Problem

    Clinical multimodal learning must reconcile weakly aligned video, structured time series, and text, while AIS screening remains reliant on surface-deformity assessments with limited positive predictive value.

  • Method

    ScoliDetect aligns monocular gait video with a fixed-index kinematic knowledge map and template-based kinematic text using temporal registration, bidirectional cross-attention, and latent-bottleneck fusion.

  • Results

    Structured kinematic mediation and latent-bottleneck fusion improved external discrimination over unimodal and late-fusion baselines, with stable ranking across severity and curve-type subgroups.

  • Takeaways & Limitations

    KKM-based fusion provides a shared, auditable coordinate system for multimodal registration and factor-level explanations tied to specific kinematic phases and skeletal indices.

  • Takeaways & Limitations

    Internal data were dominated by a single hospital with many self-referred cases, limiting the study's internal-data representativeness.

Abstract

from arXiv · show

Multimodal clinical AI is limited by weakly aligned inputs and the absence of domain-specific interpretable representations, particularly when learning from dense video stream, structured time-series, and template-based kinematic text. Here we present ScoliDetect, an explainable framework for adolescent idiopathic scoliosis screening from monocular gait video, built around a kinematic knowledge map (KKM) and complementary template-based kinematic text derived from per-sequence pose statics. KKM is a fixed-index structured representation that encodes gait features across absolute motion, self-skeleton configuration and joint-joint signal correlation, providing anchor-referenced multimodal fusion and factor-level interpretation. We integrate video, KKM, and template-based kinematic text through bidirectional cross-attention with latent-bottleneck aggregation. In a multicenter cohort (n = 1,858 after exclusions), prespecified supervised ablations on an external screening cohort show that KKM-mediated multimodal fusion outperforms unimodal models and late concatenation. Under a staged training protocol, trimodal contrastive pretraining is applied after architecture selection as representation initialization, improving external ROC-AUC from 0.961 to 0.972. Furthermore, the structured nature of the KKM provides inherent, factor-level attributions mapped directly to specific kinematic phases and skeletal indices, offering verifiable interpretability. The results demonstrate that embedding explicit structural topologies into latent spaces significantly enhances both the generalization and explainability of multimodal pattern analysis systems.

I. INTRODUCTION

ScoliDetect addresses weak temporal and semantic alignment in multimodal gait analysis by using a structured kinematic representation to align video, time-series, and text. The framework evaluates supervised fusion and staged trimodal pretraining for AIS screening with explicit factor-level interpretability.

  • Motivation: Multimodal clinical AI struggles when raw video, derived time series, and text are poorly synchronized, allowing models to encode spurious acquisition context.These problems are especially acute for periodic biological motion, where gait phase, protocol, and operator variability affect cross-site generalization.
  • Motivation: AIS screening remains reliant on surface deformity, trained personnel, privacy-preserving facilities, and radiographic confirmation despite meaningful sensitivity and limited positive predictive value.Gait kinematics differ systematically between AIS and controls, motivating non-disrobing screening approaches.
  • Framework: ScoliDetect uses a fixed-index kinematic knowledge map and template-based kinematic text to provide structured, auditable coordinates for multimodal fusion.The KKM spans absolute motion, self-skeleton configuration, and joint–joint signal correlation, while sharing indices with text generation and factor-level readouts.
  • Framework: Video and KKM are fused with bidirectional cross-attention and Perceiver-style latent-bottleneck compression, while prespecified external-cohort ablations compare unimodal, fusion, and late-concatenation models.Trimodal contrastive pretraining is applied only after architecture selection as encoder initialization.
  • Interpretability: Gait cycles are peak-anchored before fusion, so video and KKM tokens refer to comparable phases and explanations can be read at domain-tagged factor indices.The reported attention and gradient readouts are treated as attributional correlates rather than causal claims.
  • Evaluation: The evaluation uses a multicenter observational cohort of n = 1,858 after exclusions with patient-level partitioning, external school-screening validation, prespecified ablations, and subject-level bootstrap inference.

B. Video, Pose, and Gait Pattern Analysis

ScoliDetect converts monocular gait video into aligned video, KKM, and template-based text representations for AIS screening. Its pipeline combines fixed-index kinematic factors, phase registration, index-biased video–KKM fusion, and text concatenation at the screening head.

  • Data construction: The KKM is constructed from hybrid pose-estimator outputs and used to filter and peak-anchor video frames before multimodal fusion.
  • Clinical and technical context: Existing scoliosis AI largely analyzes static anatomy, while gait-based models target dynamic asymmetries using video or skeleton representations.The paper positions AIS gait analysis as a test case for multimodal pattern analysis under misalignment.
  • Data construction: The representation stage performs pose estimation, scale-invariant joint normalization, fixed-index KKM construction, peak-anchored registration, and template-based text generation.The text is generated from per-sequence pose statistics.
  • Multimodal learning: Registered video, KKM, and text use modality-specific encoders; video and KKM undergo index-biased bidirectional cross-attention with a Perceiver-style latent bottleneck.The text embedding is concatenated with pooled video–KKM features at the screening head to form KVT.
  • Representation: Each sample contains a monocular video clip, an aligned KKM tensor with F = 238 predefined factors, and template-based kinematic text.Video and KKM share a normalized gait-phase axis after peak-anchored registration but retain different resolution.
  • Representation: KKM indices partition into absolute motion factors 0 – 63, self-skeleton configuration factors 64 – 171, and joint-joint signal correlation factors 172 – 237.

C. Pose Estimation and Target Selection

The pose pipeline combines bottom-up and top-down estimation to obtain robust, temporally consistent joint landmarks and select the target subject in multi-person scenes. It then normalizes coordinates around the hip midpoint using per-frame pose scaling while preserving pathological asymmetries.

  • Pose estimation: A hybrid bottom-up and top-down pose estimator improves robustness to occlusion and temporal consistency, respectively.Bottom-up detections are grouped into skeletons, while top-down estimation tracks joints per identity.
  • Target selection: In multi-person scenes, the target is selected as the candidate closest to the central screen by matching bottom-up skeleton proposals to a top-down reference.
  • Normalization: Raw 2D joint coordinates are re-centered on the hip midpoint, scaled by a torso-derived factor, and divided by maximum joint distance for scale-invariant trajectories.
  • Normalization: The per-frame pose scale uses torso size defined as the Euclidean distance between shoulder and hip centres, alongside the maximum joint distance from the origin.
  • Normalization: A 2.5 multiplier follows a BlazePose heuristic intended to prevent a single outlier joint from dominating scale when the subject is distant.
  • Normalization: Applying one scale factor to all joints preserves pathological asymmetries such as pelvic tilt and shoulder-height differences.The normalized trajectories feed all subsequent KKM feature computations.

E. Temporal Alignment and Kinematic Text

Gait sequences are temporally registered using the KKM trajectory, while template-based kinematic text summarizes per-sequence pose statistics with category-specific templates. The multimodal-learning design also includes dedicated encoders for video, KKM, and text.

  • Temporal Alignment: Stationary frames are removed, then sequences are aligned by motion onset and peak anchoring on the summed KKM trajectory.This registration makes video and KKM tokens refer to comparable gait phases before fusion or pretraining.
  • Kinematic Text: Template-based kinematic text summarizes mean and standard deviation of step width, trunk shift, and joint angles.Category-specific templates represent normal, single thoracic, single lumbar, and multi-curve patterns.
  • Modality-Specific Encoders: Video, KKM, and text are mapped through modality-specific encoders before multimodal fusion.The video branch uses ViViT, the KKM branch uses a Transformer, and the text branch uses a pretrained Sentence Transformer with an MLP projection.
  • Architecture Overview: Figure 3 summarizes trimodal contrastive pretraining, supervised fusion variants, and fixed-index KKM domains with top-k readouts.Its three panels cover alignment, supervised architecture choices, and structural explanation.

G. Multimodal Fusion Architecture

The fusion architecture aligns video and KKM tokens with index-biased bidirectional cross-attention, compresses them through a latent bottleneck, and adds text at the classification head. Temporal index tracks are normalized and converted into attention biases that favor colocated tokens.

  • Fusion Path: Video and KKM are fused through index-biased bidirectional cross-attention, while text is late-concatenated at the KVT classification head.Text does not enter the video–KKM cross-attention stack.
  • Temporal Indexing: Temporal index tracks are min–max normalized to [0, 1], and pairwise distances become additive attention biases for temporally colocated tokens.The bias is added to raw attention logits before softmax and preserves within-sequence ordering.
  • Latent Bottleneck: A learnable latent array updates by attending first to video tokens and then to KKM tokens as keys and values.The bottleneck uses M latent tokens with hidden size d and performs two cross-attention steps per layer.
  • Representation Aggregation: The bottleneck summary is mean-pooled over latent tokens, normalized, and added to separately pooled video and KKM vectors.The resulting modality vectors are combined with the text branch for classification.
  • Classification Head: The concatenated representation is processed by a two-layer MLP with normalization, nonlinearity, and dropout before producing the binary classification logit.The implementation identifies the nonlinearity as ReLU.

H. Optimization Objectives

Training uses staged trimodal contrastive pretraining after supervised architecture selection, followed by focal-loss fine-tuning with an optional video–KKM alignment term. Contrastive pretraining aligns all modality pairs in a shared embedding space.

  • Contrastive Pretraining: Trimodal contrastive pretraining aligns video, KKM, and text in a shared 512-dimensional space using symmetric InfoNCE losses on all modality pairs.This stage initializes encoders only after supervised ablations fix the fusion architecture.
  • Contrastive Objective: The total pretraining objective sums the symmetric contrastive losses for the video–KKM, video–text, and KKM–text modality pairs.The objective applies a learnable temperature to pairwise similarity matrices.
  • Supervised Fine-Tuning: Supervised fine-tuning uses focal loss for binary labels and class imbalance, with an optional symmetric video–KKM InfoNCE auxiliary term.The auxiliary term is weighted by λ in the combined objective.
  • Loss Parameters: The focal-loss parameters include α for class weighting and γ for focusing, while the auxiliary alignment coefficient controls multimodal alignment strength.The supplied formulation constrains α to [0, 1] and γ to nonnegative values.

I. Factor-level Interpretability

Interpretability is tied to predefined KKM factor indices and gait phases, with attributional readouts from temporal attention and attention-weighted gradients. The study evaluates these methods in a multicenter, participant-level partition with external screening data and bootstrap inference.

  • Factor-level Interpretability: Each KKM input dimension maps to a predefined factor index, enabling top-k readouts across motion, self-skeleton, and signal cross-correlation domains.Temporal attention and attention-times-gradient maps rank factors and gait phases for a fixed classifier.
  • Interpretive Scope: KKM attribution readouts indicate correlates of predictions rather than causal mechanisms.This qualification defines the interpretive scope of the attention-based explanations.
  • External Evaluation: External evaluation used DKC participants, while HKU-SZ supplied training and validation data and school cohorts formed additional screening datasets.PK controls had radiographic confirmation of Cobb ≤10° and contributed to the shared negative set.
  • Statistical Inference: ROC-AUC intervals and subgroup contrasts were estimated with participant-level bootstrap resampling using 2,000 replicates.Paired subgroup differences used identical bootstrap draws, with Holm adjustment for pairwise multiplicity.
  • Study Partitioning: 1,858 participants remained after exclusions from a multicenter recruitment and participant-level partitioning process.Figure 4 reports 1,974 enrolled participants and datasets 1–4 defined by analytic role.

E. Training Details

Training used staged optimization with fixed supervised architecture selection before trimodal pretraining initialization. On the external cohort, multimodal ablations showed progressive gains, with latent-bottleneck KVT and pretraining reaching the strongest reported AUC.

  • Optimization: Pretraining used batch size 64 for 200 epochs with early stopping, while supervised fine-tuning used batch size 32 and up to 100 epochs.The learning rate ranged from 1×10−4 to 1×10−6 during pretraining and from 8×10−6 to 1×10−7 during fine-tuning.
  • Optimization: The staged protocol trained task-specific heads first, progressively unfroze encoder blocks, and then jointly fine-tuned the full model.All encoders were initially frozen for 20 epochs; the final joint phase covered epochs 51–100.
  • External performance: 0.972 external ROC-AUC was achieved by the selected KVT model after trimodal encoder pretraining and supervised fine-tuning.The primary multimodal evidence came from prespecified supervised ablations rather than backbone choice.
  • Ablation results: External AUC increased from 0.784 for video-only to 0.961 for KVT, then to 0.972 with pretraining; latent-bottleneck fusion exceeded late concatenation by 0.005 AUC.Removing temporal alignment reduced performance to 0.943, while component ablations also supported auxiliary InfoNCE and bidirectional cross-attention.
  • Ablation results: External ROC curves evaluated modality/fusion ablation and component ablation on n = 454 cases, with AUC values reported in Tables II and III.The figure presents the two ablation families used to assess the multimodal design.

B. Backbone Robustness

Transformer backbones showed robust external performance, with the best ViViT configuration reaching 0.972 ROC-AUC. Severity-stratified performance remained similar across groups, while operating-point metrics quantified screening trade-offs.

  • Backbone comparison: 0.972 external AUC was achieved by ViViT with trimodal pretraining followed by supervised fine-tuning, while ViViT scratch reached 0.961.Transformer-based backbones consistently outperformed pretrained ResNet variants in internal validation.
  • Backbone comparison: The internal-to-external performance shift likely reflected cohort heterogeneity rather than backbone instability.External evaluation used more standardized school-screening acquisition, associated with a more stable gait phenotype distribution.
  • Severity subgroups: Severity subgroup ROC-AUC was 0.972 [0.952–0.989], with pairwise ΔAUC intervals including zero and Holm-adjusted p-values of 1.0.Mild, moderate, and severe groups were defined by Cobb-angle thresholds of 10°–25°, 25°–40°, and >40°.
  • Severity subgroups: At sensitivity ≥0.95, specificity ranged from 0.919 to 0.930, projected false positives were roughly 66–77 per 1,000 screened, and projected NPV was ≥0.997.Projected PPV ranged from about 0.38 to 0.42 across severity strata.
  • Interpretability: Figure 6 reports temporal attention, domain attribution by gait phase, and top-20 factors by subgroup for external-test interpretability.These views organize factor-level attribution across phases and subgroup analyses.

E. Input-level heterogeneity across datasets

KKM statistics differed between internal and external cohorts, with broader internal dispersion and more compact external geometry. This shift coincided with lower internal than external performance under cohort shift.

  • Cohort comparison: KKM comparisons used the internal HKU-SZ cohort with n = 820 and the external test cohort with n = 454 participant-level patches.The analysis examined element-wise mean differences, per-patch Frobenius norms, and joint PCA.
  • Cohort geometry: External KKM samples formed a compact PCA neighbourhood, whereas internal samples covered a wider region with isolated outliers.PC1 and PC2 explained 86.8% and 4.9% of variance, respectively.
  • Cohort geometry: Comparable ViViT-based models showed internal-to-external AUC values of 0.787 versus 0.961 under cohort shift.The broader internal dispersion aligned with lower internal validation AUC and was interpreted as consistent with referral and acquisition heterogeneity.

VI. DISCUSSION

ScoliDetect frames AIS screening as pattern analysis under weakly aligned multimodal inputs, using structured kinematic mediation and indexed explanations to improve external performance and auditability. The framework remains bounded by standardized capture, limited evidence for deployment effectiveness, and noncausal attribution semantics.

  • Standardized monocular capture was feasible for nine operators after brief training, supporting referral triage under protocol but not replacement of clinical examination or radiography.The intended use is assistive screening rather than standalone diagnosis.
  • Prespecified external-cohort ablations support KKM semantics, bidirectional cross-attention, latent-bottleneck aggregation, and related components as contributors to multimodal performance.Trimodal contrastive pretraining was applied only after architecture selection, as encoder initialization rather than as evidence for fusion design.
  • External ranking remained stable across severity and curve-type subgroups, while internal-external differences were associated with referral heterogeneity and standardized school-screening acquisition.The external cohort showed a more compact kinematic distribution than the more dispersed internal development cohort.
  • KKM-indexed attributions are auditable in biomechanical terms, but gradient-based rankings represent classifier correlates rather than causal effects of interventions on individual factors.Prospective reader or interventional studies are still needed to test effects on trust, audit, or failure detection.
  • The evidence is constrained by a hospital-heavy, self-referred internal cohort, rule-generated kinematic text, and attribution methods that do not support standalone referral decisions.Feasibility with briefly trained operators does not establish outcome effectiveness.
  • ScoliDetect aligns monocular gait video with a fixed-index KKM and template-based kinematic text, producing a shared coordinate system for multimodal fusion and factor-level explanations.The framework uses explicit temporal registration, index-biased video-KKM fusion, and late text concatenation at the screening head.
Loading 2608.20969v1…