Source-linked AI summary
TokenSTFormer: A Tokenized Spatial-temporal Attention Model for Holistic Motion Analysis in Adolescent Idiopathic Scoliosis Screening
Dong Chen, Kenneth M. C. Cheung
TL;DR
AIS screening requires approaches that reduce subjectivity, dependence on professional expertise, and scalability barriers while avoiding repeated X-ray exposure. This paper introduces the ScoliGait dataset and TokenSTFormer, which uses Spatial-Temporal Tokenization for gait-based holistic motion analysis. TokenSTFormer outperformed a Vision Transformer encoder, achieving 0.787 accuracy alongside reported sensitivity and specificity results.
Problem
AIS screening is constrained by subjective interpretation, professional-expertise requirements, limited scalability, and concerns about cumulative radiation exposure from repeated X-rays.
Method
The paper constructs a gait-video ScoliGait dataset with radiographic labels and uses a kinematic knowledge map with TokenSTFormer’s Spatial-Temporal Tokenization to represent holistic spatial-temporal motion.
Results
0.787 accuracy, 0.845 sensitivity, and 0.660 specificity were achieved by TokenSTFormer, outperforming the Vision Transformer encoder across all evaluation metrics.
Takeaways & Limitations
The study supports smartphone-based holistic motion analysis as a promising direction for scalable, cost-effective scoliosis screening and monitoring.
Abstract
from arXiv · showhide
Adolescent Idiopathic Scoliosis (AIS) is a prevalent spinal deformity in adolescents that, if left untreated, can result in severe health outcomes. Traditional screening methods are limited by subjective interpretation, reliance on professional expertise and low scalability. To address these challenges, we present ScoliGait dataset, which comprises 1,516 gait video clips paired with corresponding X-ray records. We also introduce TokenSTFormer, a novel model that tokenizes spatial and temporal semantics to enhance feature representation and convergence. Our model achieves state-of-the-art performance, surpassing vanilla Vision Transformer encoder across key metrics, including accuracy of 0.79. This study highlights the potential of leveraging holistic motion features derived from gait video and attention-based models for scalable, cost-effective AIS screening, paving the way for future clinical applications in scoliosis detection.
1 Introduction
AIS screening is important but current approaches face subjectivity, scalability, privacy, and preprocessing limitations. ScoliDetectTM addresses these challenges using gait videos, a kinematic knowledge map, and TokenSTFormer with Spatial-Temporal Tokenization.
- AIS screening is needed because untreated scoliosis can cause severe health consequences, while radiographic diagnosis raises concerns about cumulative radiation exposure.
- Traditional screening effectiveness varies and depends on subjective interpretation, participant characteristics, professional expertise, equipment costs, and privacy-sensitive procedures.
- Static back-asymmetry methods omit kinematic features, while existing gait methods require complex preprocessing pipelines that limit real-world practicality.
- ScoliDetectTM uses mobile-phone gait videos to construct a holistic-motion kinematic knowledge map for scalable and privacy-conscious AIS screening.
- TokenSTFormer incorporates Spatial-Temporal Tokenization to enhance feature representation and model convergence.
2 Dataset
ScoliGait combines mobile-phone gait videos with clinically grounded scoliosis labels from corresponding spinal X-rays. The dataset contains 1,516 clips collected from 758 participants under a standardized recording setup.
- 1,516 non-overlapping gait video clips were created from recordings of 758 participants for the ScoliGait dataset.Each clip captured 5 seconds of walking at 30Hz and 1080p resolution.
- ScoliGait pairs gait videos with corresponding spinal X-rays, which provide radiographic medical labels validated by senior medical doctors.
- The dataset’s scoliosis ground truth relies on radiographic measurements rather than traditional screening methods lacking sufficient evidence as reliable labels.
- Participants followed a standardized forward-and-backward walking protocol along a 4-meter path while a camera positioned 2.5 meters high captured shoulder-pelvic angles.
3 Methodology
The methodology constructs a holistic kinematic knowledge map from pose-estimated gait videos, then tokenizes its spatial and temporal semantics for TokenSTFormer processing. The section also defines classification metrics used to assess AIS screening performance.
- Kinematic Knowledge Map: 2D joint coordinates from gait videos are used to construct a kinematic knowledge map across motion, self-skeleton, and signal-correlation domains.The map represents overall gait patterns, skeletal structure, and motion relationships.
- Kinematic Knowledge Map: 238 features comprise the kinematic knowledge map: 140 in motion space, 32 in self-skeleton space, and 66 for signal correlation.Paired joint features use Euclidean distance, motion angles use trigonometric functions, and motion lagging uses signal cross-correlation.
- TokenSTFormer Architecture: TokenSTFormer follows a Vision Transformer architecture with residual blocks containing Multiheaded Self-Attention and Multi-Layer Perceptron components.Dense layers, LayerScale, and Stochastic Depth are also integrated into the model configuration.
- TokenSTFormer Architecture: Spatial-Temporal Tokenization converts the kinematic knowledge map into spatial and temporal tokens using 2D convolutions with column-size and row-size kernels.The two token types have output dimension d, are layer-normalized, and then concatenated.
- Training Objective: Binary cross entropy provides the main loss, while Mean Squared Error between temporal and spatial CLS tokens supplies an auxiliary loss.CLS tokens are applied separately to temporal and spatial embeddings.
- Evaluation Metrics: Accuracy, sensitivity, specificity, Positive Predictive Value, and Negative Predictive Value measure classification performance from positive and negative test outcomes.Accuracy uses all correctly classified samples, while sensitivity and specificity measure true-positive and true-negative identification, respectively.
4 Experiments
Experiments compare TokenSTFormer with a matched vanilla Vision Transformer and examine how its components and depth affect performance. TokenSTFormer outperforms the baseline across reported metrics, while five attention blocks gives the highest accuracy.
- Results: TokenSTFormer consistently outperformed the Vision Transformer encoder across all evaluation metrics.The comparison used the same hyperparameters and a 6 by 6 patch size for the baseline encoder.
- Results: 0.787 accuracy, 0.845 sensitivity, and 0.660 specificity were achieved by TokenSTFormer.Its predictive values were PV+ 0.845 and PV− 0.660.
- Ablation studies: LayerNorm and independent positional encoding play critical roles in SST modules for accuracy and robustness.Their effects were examined through ablation studies.
- Ablation studies: STT separates and normalizes temporal and spatial tokens to minimize their distance and reproject them into an enhanced semantic space.The cosine-similarity analysis found most points above the red dashed slope-1 line, indicating smaller similarity in the baseline model.
- Ablation studies: Five attention blocks produced the model’s highest accuracy among the evaluated layer configurations.This configuration was described as balancing model complexity and performance.
5 Conclusions
The study introduces ScoliDetectTM, an AI-assisted holistic motion analysis system using a smartphone camera for scoliosis screening. It leverages Spatial-Temporal Tokenization to learn robust spatial-temporal gait features for scalable and cost-effective screening and monitoring.
- 5 Conclusions: ScoliDetectTM uses smartphone-camera gait analysis for scalable and cost-effective scoliosis screening.The system is presented as an AI-assisted holistic motion analysis approach.
- 5 Conclusions: Spatial-Temporal Tokenization enables TokenSTFormer to learn robust spatial-temporal features from holistic motion.The conclusion frames this approach as a step toward accessible diagnostic tools for scoliosis screening and monitoring.