A Standardized Benchmark for Skeleton-Based Rehabilitation Assessment Using Deep Learning

1 Introduction

• Rehabilitation assessment evaluates movement quality and deviation from an ideal within the same activity class, unlike action recognition, which classifies motion types.For example, assessment may determine whether a squat is correctly performed and quantify its deviation from ideal motion.Source: §1 · p. 2
• Video-derived skeleton sequences offer a compact, interpretable motion representation that can be collected with accessible devices such as smartphones, webcams, or depth cameras.The paper contrasts these options with costly, intrusive, and often impractical motion-capture systems.Source: §1 · pp. 2–3
• The field lacks a unifying benchmark, and limited dataset coverage, inconsistent protocols, and unreleased implementations constrain comparison and reproducibility.Prior work has often relied on KIMORE and UI-PRMD, leaving performance across diverse populations, exercises, and real-world settings unclear.Source: §1 · p. 3
• The paper aggregates existing rehabilitation repositories into Rehab-Pile and proposes a unified framework for consistent comparison of deep-learning methods.The archive is intended to support analysis of existing and future methods.Source: §1 · pp. 3–4
• The benchmark covers classification and extrinsic regression, and the authors publicly release the aggregated datasets and re-implemented code.Classification uses 39 datasets and extrinsic regression uses 21.Source: §1 · p. 4

2 Definitions •

• A univariate time series is an ordered sequence of L timestamped data points, while a multivariate series contains M same-length univariate series.The classification dataset pairs each multivariate series with a discrete class-probability label; the extrinsic regression dataset pairs it with a continuous label.Source: §2 · p. 4
• A time-series classification dataset contains N multivariate series of M variables and length L, each labeled by a distribution over C classes.A time-series extrinsic regression dataset has the same series structure but associates each series with a continuous label.Source: §2 · p. 4
• A skeleton-based human motion sequence has L frames, each a D-dimensional skeleton of J joints, and can be represented by J·D variables over time.Source: §2 · p. 4

3 The Rehab Pile

• Rehab-Pile derives 60 exercise-level datasets from nine public repositories: 39 for classification and 21 for extrinsic regression.Each exercise is treated as a separate dataset because the paper says assessing multiple exercises with one model is not feasible.Source: §3 · p. 5

3.1.1 EHE REG

• The EHE repository contributes six regression datasets from daily nursing-home exercises, with labels indicating Alzheimer’s disease severity.The movements of 25 people were captured using a Kinect camera.Source: §3.1.1 · p. 5

3.1.2 KIMORE REG

• KIMORE contributes five regression datasets, using expert-assigned continuous scores from 0 to 100 for exercises performed by 71 people.The repository comprises 40 healthy people and 31 people with pathology.Source: §3.1.2 · p. 5

3.1.3 UI-PRMD REG

• UI-PRMD contributes ten regression datasets from segmented exercise sequences with continuous labels, although not all ten subjects perform every exercise.Source: §3.1.3 · pp. 5–6

3.2.1 IRDS CLF BN; 3.2.2 KERAAL CLF BN and KERAAL CLF MC

• IRDS contributes nine binary-classification datasets; KERAAL contributes six binary or multiclass datasets evaluated with healthy subjects for training and leave-one-subject-out patient testing.IRDS includes 14 healthy and 15 unhealthy participants, with some exercises performed seated, in wheelchairs, or using support.Source: §3.2.1 · p. 6, §3.2.2 · p. 7

3.2.3 KIMORE CLF BN

• KIMORE’s five classification datasets label exercise scores at or below 50 as incorrect and scores above 50 as correct.Source: §3.2.3 · p. 7

3.2.4 KINECAL CLF BN

• KINECAL retains four of 11 exercises because only six of 90 participants are clinically at risk, requiring at least five at-risk participants per retained exercise.Source: §3.2.4 · p. 7

3.2.5 SPHERE CLF BN; 3.2.6 UCDHE CLF BN and UCDHE CLF MC; 3.2.7 UI-PRMD CLF BN

• SPHERE WUS is reformulated as binary anomaly classification; UCDHE contributes four binary or multiclass datasets, and UI-PRMD adds ten binary datasets.For UCDHE, two exercises use labels for correct performance or error type, with binary variants merging error types; UI-PRMD’s additional binary labels indicate correct versus incorrect performance.Source: §3.2.5 · p. 7, §3.2.6 · p. 8, §3.2.7 · p. 8

4 Deep Learning for Rehabilitation Assessment

• The selected architectures include models originally developed for univariate or multivariate time-series classification and models designed for human-motion datasets.Source: §4 · p. 8
• The backbones range from convolutional and recurrent models to self-attention and graph-convolution approaches.Source: §4 · p. 8
• The section describes how each backbone processes temporal data and introduces definitions for its mathematical formulation.Source: §4 · p. 8

4.1.1 Convolution Operation on Temporal Data

• Temporal convolution detects patterns useful for distinguishing input categories, while dilation controls the filter’s view of the sequence.The section illustrates one-dimensional convolution with and without dilation.Source: §4.1.1 · pp. 8–9

4.1.2 Recurrent Neural Networks

• RNNs update a sequence representation using current inputs and past memory; LSTMs maintain hidden and cell states, while GRUs combine forget and input gates into an update gate.Both LSTM and GRU produce an output sequence over frames.Source: §4.1.2 · pp. 10–11

4.1.3 Self-Attention

• Self-attention relates positions across an input sequence through learned query, key, and value projections, with row-wise softmax weights determining attention between frames.An optional output projection adjusts the final representation’s dimensionality.Source: §4.1.3 · pp. 11–12

4.1.4 Graph Convolution Neural Networks

• Graph convolution updates node representations by aggregating information from neighboring nodes, using graph connectivity and learned transformations.The section describes temporal convolution as producing frame-level features for the graph operation.Source: §4.1.4 · p. 12

4.2 Deep Learning Architectures

• The study organizes architectures into convolution-, recurrent-, self-attention-, and graph-based families.The architecture section presents models individually within these four groups.Source: §4.2 · p. 13, Figure 4 · p. 13

4.2.1 Fully Convolutional Network (FCN)

• FCN uses three convolutional blocks and global average pooling before task-specific classification or regression outputs.Each block applies one-dimensional convolution, batch normalization, and ReLU; classification uses softmax, while regression predicts one continuous value.Source: §4.2.1 · p. 13

4.2.2 Hybrid Inception (H-Inception)

• H-Inception adapts time-series Inception with fixed, hand-crafted filters for increasing trends, decreasing trends, and peaks.The filters are integrated into the first layer and remain unchanged during training.Source: §4.2.2 · p. 14

4.2.3 Light Inception with boosTing tEchniques MultiVariate (LITEMV)

• LITEMV combines multiplexed depthwise-separable convolutions and fixed filters in its first block, then uses increasingly dilated convolutions in later blocks.The architecture uses three convolutional blocks and global average pooling before the downstream task layer.Source: §4.2.3 · p. 15

4.2.4 Disjoint Convolutional Neural Network (DisjointCNN)

• DisjointCNN uses four 1+1D convolution blocks, first extracting temporal patterns and then learning spatial relationships across dimensions.The final block is followed by local max pooling and global average pooling.Source: §4.2.4 · pp. 15–16

4.2.5–4.2.6 Recurrent Architectures

• ConvLSTM applies convolutional blocks before LSTM layers, while MotionGRU uses two GRU layers to summarize raw input sequences into a vector.ConvLSTM’s convolutional blocks use two-dimensional kernels of width 1; MotionGRU adds a fully connected layer to reduce dimensionality.Source: §4.2.5 · p. 16, §4.2.6 · p. 17

4.2.7–4.2.8 Transformer Architectures

• VanTran adapts a Transformer encoder with four attention blocks and sinusoidal absolute positional encoding; ConvTran pairs 1+1D embedding with time encoding and relative positional encoding.ConvTran uses one self-attention block without residual connections, unlike the four-block VanTran design.Source: §4.2.7 · p. 18, §4.2.8 · pp. 18–19

4.2.9 Spatio-Temporal Graph Convolution Network (STGCN)

• The enhanced STGCN replaces global average pooling with LSTM layers and adds a learnable attention matrix to weight skeleton joints.Its graph convolutions capture correlations between skeleton dimensions; the LSTM layers aggregate temporal information.Source: §4.2.9 · pp. 19–20

5.1.1 Data Resampling

• Sequences are Fourier-resampled to their dataset’s average length because samples share a frame rate and recording technology.This produces a common number of frames within each dataset.Source: §5.1.1 · p. 21

5.1.2 Train-Test Folds Creation

• Train-test folds separate subjects, and applicable test sets contain only unhealthy subjects.Datasets with at least ten subjects use five-fold cross-subject splits; smaller datasets use leave-one-subject-out splits, with the subject count determined by the applicable population.Source: §5.1.2 · p. 21

5.1.3 Data Normalization

• Min-max normalization is computed independently for each feature dimension, using training-set extrema for both training and test data.The paper states that this prevents test data from influencing training.Source: §5.1.3 · p. 22

5.1.4 Multivariate Time Series View

• Only STGCN uses the input as a skeleton sequence; other models flatten joint and feature dimensions into an MTS of shape (L, J · D).The temporal frame axis is retained.Source: §5.1.4 · p. 22

Classification

• Classification uses accuracy and balanced accuracy; balanced accuracy averages class-wise recall to account for class imbalance.The authors say this prevents misleadingly high scores when a model disproportionately predicts the majority class.Source: Classification · pp. 22–24

Regression

• Regression performance is measured with RMSE and MAE: MAE treats deviations equally, whereas RMSE is more sensitive to larger errors and outliers.The authors use both metrics for a more comprehensive view of model performance.Source: Regression · p. 24
• Training labels are normalized to 0.0–1.0, predictions are bounded to that range, and regression metrics are calculated after de-normalization.De-normalization multiplies predictions by the maximum value associated with the dataset in Table 1.Source: Regression · p. 24

5.2.2 Efficiency Evaluation

• Efficiency evaluation reports training time, inference time, trainable parameters, and FLOPs alongside performance metrics.The comparison aims to identify models that are both accurate and computationally efficient, including in resource-constrained environments.Source: §5.2.2 · pp. 24–25

5.3.1 Training Configuration and Computational Setup

• All models use a unified TensorFlow-Keras implementation and training configuration, with 1500 epochs, batch size 64, and a shared RTX 4090 machine.The setup also uses ReduceLROnPlateau scheduling and selects the checkpoint with the lowest training loss for test evaluation.Source: §5.3.1 · p. 25

5.3.2 Ensembling Strategy

• For each fold, five runs with different random initializations generate test predictions that are averaged before evaluation metrics are computed.Classification ensembles average predicted probability distributions, while regression ensembles average predicted values; final performance is averaged across folds.Source: §5.3.2 · pp. 25–26
• The ensemble setup reduces the effect of random initialization on predictions and minimizes bias from a specific train-test split by averaging performance across folds.The authors describe these as the setup’s two main purposes.Source: §5.3.2 · p. 26

6.1 Performance

• The Multi-Comparison Matrix ranks models by average performance across datasets and reports pairwise differences, win-tie-loss counts, and significance tests.The study uses two-sided Wilcoxon signed-rank tests, treating p-values below 0.05 as statistically significant.Source: §6.1 · pp. 26–27

6.1.1 Regression

• LITEMV statistically outperforms all competitors in regression, averaging 5.3497 MAE and 6.4286 RMSE across datasets.ConvTran ranks second; the authors suggest its convolutional embedding helps preserve informative features for this task.Source: §6.1.1 · pp. 27–28, Figure 16 · p. 28, Figure 17 · p. 29
• Regression rankings vary across architectures: MotionGRU outperforms ConvLSTM, while STGCN ranks sixth overall and records its wins over LITEMV on KIMORE and UIPRMD.LITEMV's depthwise separable convolutions capture temporal and spatial correlations that H-Inception lacks; the authors suggest STGCN's structural modeling suits those two repositories.Source: §6.1.1 · pp. 28–29

6.1.2 Classification

• LITEMV leads classification on average, but its advantage over VanTran is not statistically significant across the 39 datasets.VanTran's average performance is lower by 1.19% in accuracy and 2.86% in balanced accuracy.Source: §6.1.2 · pp. 29–30, Figure 18 · p. 30, Figure 19 · p. 30
• MotionGRU ranks second in balanced accuracy, while LITEMV's accuracy exceeds its balanced accuracy by nearly 5%.The authors report that the gap across models indicates difficulty handling class imbalance; FCN and ConvLSTM perform poorly, and STGCN performs worst, slightly above chance.Source: §6.1.2 · p. 31

6.2 Efficiency

• ConvTran trains fastest, while MotionGRU infers fastest; training-time variation is high for nearly all models, and inference-time spread is similar across models.The authors attribute the inference ranking partly to the average series length of 267 and distinguish feed-forward inference from gradient computation during training.Source: Figure 20 · p. 32, Figure 21 · p. 32, §6.2 · pp. 32–33
• LITEMV has the lowest FLOPs, while VanTran is most computationally expensive; VanTran's FLOPs scale quadratically with sequence length.H-Inception and VanTran show the least parameter-count dependence on dataset characteristics; LITEMV and ConvTran have similar parameter counts.Source: Figure 22 · p. 33, §6.2 · pp. 33–34, Figure 23 · p. 34

6.3 Finding The One Model To Rule Them All

• LITEMV offers the best overall trade-off between performance and efficiency, while VanTran has a similar average rank but is considerably less efficient.MotionGRU ranks closest to LITEMV on average, with slightly lower efficiency.Source: §6.3 · p. 34

• The work addresses limited standardization, inconsistent evaluation protocols, and scarce public source code in skeleton-based rehabilitation assessment.Source: §7 · p. 35
• The unified framework evaluates nine deep learning models across 60 rehabilitation assessment datasets: 39 classification and 21 extrinsic regression datasets.Source: §7 · p. 35
• LITEMV, particularly its ensemble form, consistently outperforms other models in both accuracy and efficiency.Source: §7 · p. 35
• The authors present the benchmark, publicly available source code, and datasets as a foundation for reproducible future research.Source: §7 · p. 35

• Regression-label distributions are shown for six EHE, five KIMORE, and ten UI-PRMD exercises.Source: Figure 25 · p. 44, Figure 26 · p. 45, Figure 27 · p. 46
• Classification-label distributions cover nine IRDS, three KERAAL, five KIMORE, four KINECAL, and one WUS exercise.Source: Figure 28 · p. 47, Figure 29 · p. 48, Figure 30 · p. 49
• Additional classification-label distributions cover two UCDHE and ten UI-PRMD exercises.Source: Figure 33 · p. 52, Figure 34 · p. 53