PhysioAI: Clinical Knowledge-Guided Semantic Supervision for Skeleton-Based Physiotherapy Action Recognition

1. Introduction

• Physiotherapy exercise classes have subtle kinematic distinctions, substantial within-class execution variability, and limited rehabilitation training data.The passage identifies KiMoRe and UI-PRMD as small relative to general action-recognition benchmarks.Source: §1 · p. 1
• Figure 1 illustrates variation both between healthy-control and motor-dysfunction movement examples and between correct and intentionally incorrect executions.Its plotted quantities are skeleton-derived proxies, not clinical movement-quality scores.Source: Figure 1 · p. 2
• Existing language-guided skeleton methods target general action recognition, leaving rehabilitation-specific semantic construction and supervision unresolved.Generic descriptions may omit kinematic cues that distinguish similar exercises, while limited data complicate learning stable representations and cross-modal mappings.Source: §1 · p. 2
• PhysioAI achieved the highest observed values across the reported endpoints on KiMoRe, UI-PRMD, and Hard-67.Hard-67 is a stress test derived from the training dynamics of an ST-GCN baseline.Source: §1 · p. 2
• PhysioAI combines structured Clinical Knowledge Dictionary anchors with uncertainty-weighted semantic alignment for skeleton-based exercise recognition and skeleton-only deployment.The paper evaluates the framework under subject-disjoint protocols on two public rehabilitation datasets and Hard-67.Source: §1 · p. 2

2.1. Physiotherapy Exercise Analysis

• Physiotherapy studies often emphasize quality assessment or score prediction, leaving recognition under exercise-execution variability less explored.PhysioAI addresses this gap by focusing on physiotherapy exercise recognition under such variability.Source: §2.1 · p. 3

2.2. Skeleton-Based Action Recognition

• Graph-based skeleton models capture spatial joint relations and temporal dynamics, but their rehabilitation applicability under limited data and participant-dependent variability remains unestablished.PhysioAI uses CTR-GCN rather than introducing another graph operator.Source: §2.2 · p. 3

2.3. Language-guided Skeleton Learning

• Language-guided methods align text and skeleton features or use language-derived structural priors, whereas PhysioAI derives supervision from structured physiotherapy exercise definitions.Examples include GAP, PURLS, LPSR, and LA-GCN.Source: §2.3 · p. 3

3.1. Framework Overview

• PhysioAI encodes CKD descriptions into fixed class-specific anchors and trains a CTR-GCN joint-stream encoder with classification and class-matched semantic alignment.Alignment begins at epoch 6 and the semantic components are discarded before inference, leaving the skeleton encoder and classifier.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Clinical Knowledge Dictionary

• The CKD contains 35 entries across five rehabilitation and functional-action domains, with structured movement descriptors and class-discriminating cues.Entries cover movement taxonomy, body region, plane, joint action, support, laterality, phases, and distinguishing cues.Source: §3.2 · p. 3
• CKD generation uses structured fields to produce six candidate discriminative sentences per action, then selects sentences using projected text-anchor geometry and format criteria.Candidates prioritize skeleton-observable cues such as moving region, plane, joint action, support pattern, laterality, and movement phases.Source: §3.2 · p. 4

3.3. CKD-Guided Semantic Supervision

• A fixed Gaussian projection maps normalized 512-dimensional CLIP embeddings into the 256-dimensional skeleton-feature space without trainable projection parameters.The resulting class-specific directions remain fixed; the projection adapts dimensions, while semantic alignment learns the relationship to skeleton features.Source: §3.3 · p. 5
• The alignment loss brings each skeleton feature toward the fixed semantic anchor for its labelled exercise class.Because the anchor is L2-normalized, the inner product is the cosine similarity between feature and anchor.Source: §3.3 · p. 5

3.4. Delayed Uncertainty-Weighted Optimisation

• Two learned global log-scale parameters weight classification and alignment objectives, with additive terms preventing unbounded suppression of either objective.Semantic alignment is inactive during epochs 1–5 and is activated at epoch 6, when uncertainty weighting controls its contribution.Source: §3.4 · p. 5

4.1. Datasets

• KiMoRe includes 78 participants, including 34 with motor dysfunctions; UI-PRMD covers ten movements by ten healthy participants, including deliberately non-optimal variants.Correct and deliberately incorrect executions share one UI-PRMD label because the task is exercise recognition, not movement-quality assessment.Source: §4.1 · p. 5

4.2. Implementation Details

• Backbones were trained from scratch for 80 epochs, and methods shared subject partitions, metrics, and checkpoint-selection rules for comparison.The standard setup used batch size 32 and Nesterov SGD; implementations followed released code or model specifications.Source: §4.2 · p. 6, §4.4 · p. 6

4.3. Hard-67 Stress-Test Protocol

• Hard-67 selects 67 low-confidence, low-variability samples from 400 KiMoRe samples using plain ST-GCN training dynamics; each was scored in its subject's test fold.Fold means and sample SDs are reported, with the earliest checkpoint attaining the highest held-out Overall accuracy selected for each sample.Source: §4.3 · p. 6

4.5. Hard-67 results

• On Hard-67, PhysioAI scored 94.64±7.36% versus GAP's 91.77±5.60%; all methods fell below parity with KiMoRe Overall, while PhysioAI's reduction was smallest.PhysioAI correctly classified 64/67 samples in the pooled endpoint, compared with 63/67 for MS-G3D and ProtoGCN.Source: §4.5 · p. 6, Figure 3 · p. 7

4.4. Evaluation and Comparison

• PhysioAI led every reported endpoint: 99.03±1.34% on KiMoRe Overall (+0.27 pp vs MS-G3D), with gains of 2.87 pp on Hard-67 and 1.33 pp on UI-PRMD Overall.The comparisons use subject-disjoint five-fold evaluation; Table 1 reports mean±SD values.Source: §4.4 · p. 6, Table 1 · p. 7

4.6. Ablation Studies

• Against CE-only training, full CKD improved KiMoRe Overall, Hard-67, and UI-PRMD Overall by 1.35, 9.82, and 1.59 pp, respectively.Biomechanics-only descriptions led KiMoRe and Hard-67, whereas full CKD led UI-PRMD; other description controls improved some endpoints but did not match full CKD on UI-PRMD.Source: §4.6 · p. 7, Table 2 · p. 8
• The fixed Gaussian projection gave the most balanced results across datasets; added learnable projection capacity provided no consistent advantage.A learned skeleton-to-CLIP linear mapping matched the fixed mapping on KiMoRe Overall and Hard-67 but was lower on UI-PRMD.Source: §4.6 · p. 8, Table 3 · p. 8
• Every alignment variant improved Hard-67 over classification-only training, but only delayed uncertainty weighting exceeded it on both Overall endpoints while matching the highest Hard-67 mean.The ablation compared fixed and uncertainty weighting with alignment activated at epoch 1 or epoch 6.Source: §4.6 · p. 8, Table 4 · p. 8

5. Limitations and Future Work

• Two relatively small public rehabilitation datasets cannot fully characterise variation across patient populations, functional abilities, or execution settings.Hard-67 was derived from one reference model’s training dynamics without clinician adjudication, so it is not a universal definition of clinically challenging movement or evidence of broad robustness.Source: §5 · p. 8
• Whether PhysioAI’s benefits generalise beyond the CTR-GCN implementation remains to be established across a broader range of skeleton architectures.The framework is designed as training-time semantic supervision and could be adapted to compatible encoders.Source: §5 · p. 8
• PhysioAI recognises prescribed exercise identity, but does not assess movement quality, therapeutic correctness, clinical outcomes, or performance across clinically defined movement difficulties.Future work will investigate clinician-guided hard-case datasets and multi-model difficulty criteria.Source: §5 · p. 8

6. Conclusion

• Across KiMoRe, UI-PRMD, and Hard-67, CKD-guided semantic supervision complements categorical labels, with the clearest observed advantages in more challenging settings.Ablations indicate that improvements depend on how semantic content is constructed and incorporated during training, not simply on adding text or an auxiliary alignment loss.Source: §6 · p. 8
• PhysioAI addresses skeleton-based physiotherapy action recognition under limited training data, fine-grained class distinctions, and substantial execution variability.The approach integrates structured rehabilitation knowledge into representation learning while retaining skeleton-only inference.Source: §6 · p. 8
• The reported ablations distinguish CKD semantic supervision from simply adding text or an auxiliary alignment loss.The improvements depend on how semantic content is constructed and incorporated during training.Source: §6 · p. 8

S1 CKD Construction and Sentence Selection

• The CKD construction section documents the procedure and fixed semantic assets used in the reported experiments.The catalogue, prompts, candidate sentences, mappings, objective, and selection process are described as fixed assets.Source: S1 CKD Construction and Sentence… · p. 11

S1.1 Frozen exercise catalogue

• The frozen catalogue contains 35 actions across five domains, with each entry recording its order, identifier, display name, domain, laterality, source hint, and brief kinematic definition.The complete pre-selection structured entries link each action to its catalogue position, selected candidate index, and final selected sentence.Source: S1.1 Frozen exercise catalogue · p. 11

S1.2 Two-stage language generation

• A two-stage language-generation process creates structured CKD entries, then six declarative candidate sentences per action prioritising movement region, plane, joint action, support pattern, laterality, and phases.The 210-candidate bank and the retained anchor sentences with selected indices are documented.Source: S1.2 Two-stage language generation · pp. 11–12, Table S1.1 · p. 11

S1.3 CLIP encoding and fixed projection

• Frozen CLIP encodes candidate sentences into normalised 512-dimensional embeddings, which a fixed seeded Gaussian projection maps to normalised 256-dimensional anchors kept unchanged during skeleton-model training.Dataset label mappings select five CKD embeddings for KiMoRe and ten for UI-PRMD.Source: S1.3 CLIP encoding and fixed projection · p. 12

S1.4 Text-geometry objective

• Candidate selection minimises a text-geometry objective that penalises similarity across all anchors, predefined confusable pairs, nearest neighbours, within-family pairs, and pairs above 0.90.A sentence-format penalty is also added when evaluating each candidate; the complete penalty definition is given in Table S1.2.Source: S1.4 Text-geometry objective · p. 12, Table S1.2 · p. 12

S1.5 Confusable pairs and action families

• The objective incorporates 15 predefined confusable action pairs and five action families, including trunk/pelvis, transfer, shoulder, gait/step, and weight/balance.The listed pairs include trunk rotation versus pelvis rotation and shoulder abduction versus scaption.Source: S1.4 Text-geometry objective · p. 12, S1.5 Confusable pairs and action… · pp. 12–13

S1.6 Greedy coordinate-descent selection

• Selection initializes each action with a minimum-penalty candidate among those below 0.5 format penalty (or all six if none), traverses the fixed catalogue order, and retains alternatives only when combined score is strictly lower.Source: S1.6 Greedy coordinate-descent selection · p. 13

S1.7 Deep Squat construction example

• For Deep Squat, six candidate sentences were generated, and candidate 3 was retained based on the objective for the complete 35-action anchor bank.Selection was based on the full bank rather than the individual sentence in isolation.Source: S1.7 Deep Squat construction example · p. 13

S2.1 Subject-disjoint five-fold evaluation

• Five subject-disjoint folds held out every sequence from listed subjects, with no subject appearing in both training and test partitions within a fold.The prepared datasets contained 400 KiMoRe sequences from 78 subjects and 1,326 UI-PRMD sequences from ten subjects.Source: S2.1 Subject-disjoint five-fold… · p. 14, Table S2.1 · p. 14

S2.2 Skeleton preparation and common loader

• Prepared archives were linearly resampled to 300 frames; the loader reshaped sequences to (3, T, 25, 2) and zero-padded KiMoRe’s unused second person slot, with no additional spatial normalization.ST-GCN used automatic zero-padding when required and random temporal selection during training; evaluation used leading model-length frames with zero-padding when required. Other common backbones used valid-frame crop-and-resize: training sampled crop proportions uniformly from [0.5, 1.0] with a minimum of 64 frames, and evaluation used the centered 95% interval.Source: S2.2 Skeleton preparation and common… · p. 14

S2.3 Training determinism and environments

• Training used fixed seeds, deterministic cuDNN settings, and a shared 80-epoch recipe, with documented software environments and a single RTX 3090 GPU.The learning rate warmed over five epochs and was multiplied by 0.1 at epochs 35, 55, and 75.Source: S2.3 Training determinism and… · p. 15, Table S2.3 · p. 15

S2.4 Checkpoint-selection convention

• Each fold’s held-out subjects were evaluated after every epoch, retaining the earliest checkpoint that attained the highest held-out Overall accuracy.Hard-67 membership, labels, and accuracy did not affect checkpoint selection.Source: S2.4 Checkpoint-selection convention · p. 15

S2.5 Metric definitions and aggregation

• Overall accuracy, macro-F1, and balanced accuracy were computed separately per fold, with class-level metrics averaged over classes represented in that fold.UI-PRMD fold 5 lacked a true label-9 sequence, so its macro-F1 and balanced accuracy averaged nine represented classes.Source: S2.5 Metric definitions and aggregation · pp. 15–16
• Reported means and SDs summarize five fold values, while pooled out-of-fold analyses concatenate mutually exclusive predictions so each sequence contributes once.The SDs reflect fold variation from one fixed training seed, not multi-seed training uncertainty; Hard-67 also reports pooled correct counts out of 67.Source: S2.5 Metric definitions and aggregation · p. 16

S3.1 Common comparison protocol

• All comparison methods used joint-coordinate inputs and were trained from scratch without pretrained action-recognition checkpoints, following the shared protocol unless an exception was documented.SkateFormer’s sampling-derived temporal indices were not treated as an additional sensing modality.Source: S3.1 Common comparison protocol · p. 16

S3.2 Final method configurations

• The graph-model baselines used differing temporal lengths and architectures, including multi-scale aggregation for MS-G3D, adaptive topology for AAGCN, and specified topology processing for BlockGCN.ST-GCN used 300-frame inputs; MS-G3D, PA-ResGCN, CTR-GCN, AAGCN, and BlockGCN used 64-frame inputs except MS-G3D’s 50 frames.Source: S3.2 Final method configurations · pp. 16–17
• SkateFormer used a small 64-frame configuration with reordered joints, while ProtoGCN used 100-frame clips and averaged ten test-time clips per sequence.Both followed documented model-specific settings within the controlled comparison recipe.Source: S3.2 Final method configurations · p. 17
• GAP trained a four-part CTR-GCN text-supervision setup, whereas PhysioAI used the same 64-frame CTR-GCN backbone with fixed CKD anchors and delayed uncertainty-weighted alignment.PhysioAI excluded semantic alignment during epochs 1–5 and discarded the semantic assets, projection, and weighting parameters at inference.Source: S3.2 Final method configurations · p. 17

S3.3 Ablation configurations

• The semantic-description ablation compared action-name-only, generic observable, biomechanics-focused, and full CKD descriptions while holding projection and delayed uncertainty weighting fixed.The full CKD sentence was the selected T3 description variant.Source: S3.3 Ablation configurations · p. 18
• Projection and alignment ablations varied fixed versus jointly optimised projections and compared weighting and onset schedules for the cosine-alignment objective.Delayed uncertainty weighting omitted both alignment loss and its log-scale term during epochs 1–5, activating them from epoch 6.Source: S3.3 Ablation configurations · p. 18, Table S3.2 · p. 18, Table S3.3 · p. 18

Fold-level results

• The detailed results comprise five fold values underlying the main comparison and ablation tables.The reported fold values are presented as percentages except where cells are counts; Hard-67 aggregates include fold mean and sample SD plus pooled correct counts.Source: S4 Detailed Ablation and Fold-Level… · p. 18, Table S4.1 · p. 19

Interpretation of reported values

• Bold and underlining identify descriptive best and second-best observed endpoint values, not statistically significant differences.Machine-readable files preserve full fold precision and integer counts used for Overall accuracy.Source: S4 Detailed Ablation and Fold-Level… · p. 18

Main comparison

• The main comparison tables separate KiMoRe Overall and Hard-67 results from UI-PRMD Overall, macro-F1, and balanced accuracy results.The supplied captions describe table scope but do not provide cell values.Source: Table S4.1 · p. 19, Table S4.2 · p. 19

Ablations

• The ablation tables report fold-level results for semantic descriptions, cross-modal projection, and alignment weighting and onset.Their captions specify Hard-67 count formatting and the text-versus-skeleton feature-space labels for projection results.Source: Table S4.3 · p. 20, Table S4.4 · p. 20, Table S4.5 · p. 20

S5.1 Reference probe and training dynamics

• A plain ST-GCN probe supplied training-partition true-class probabilities for each KiMoRe sample across four folds and 80 epochs, yielding 320 observations per sample.Neither held-out predictions nor outputs from PhysioAI or other comparison methods contributed to construction statistics.Source: S5.1 Reference probe and training… · p. 21

S5.2 Frozen selection rule

• The fixed selection rule retained low-confidence samples with low variability, targeting cases where the reference model remained consistently uncertain.Confidence was the mean probability and variability the population standard deviation of the same trajectory; the confidence quantile was 0.3319 and the variability threshold was the median.Source: S5.2 Frozen selection rule · p. 21, S5.1 Reference probe and training… · p. 21

S5.3 Manifest and distribution

• The fixed manifest records sample IDs, labels, exercise names, held-out folds, confidence, variability, and observation counts; the distribution is summarized by fold and exercise.Es5 corresponds to the KiMoRe exercise Squatting, and unequal fold sizes make unweighted fold-mean and pooled accuracy distinct summaries.Source: S5.3 Manifest and distribution · pp. 21–22, Table S5.2 · p. 22

S5.4 Held-out evaluation

• Each selected sample is evaluated in its subject’s test fold using the preselected Overall-accuracy checkpoint, without Hard-67 membership or scores guiding checkpoint selection.Hard-67 is a fixed ST-GCN-derived recognition stress test, not a measure of clinical movement difficulty, movement quality, therapeutic correctness, or robustness across all patient populations and settings.Source: S5.4 Held-out evaluation · p. 22

• The submitted ZIP contains the PDF, a top-level README.txt, and machine-readable assets indexed in the artifact materials.Source: S6 Artifact Index and Code Availability · p. 22
• Table S6.1 relates supplementary sections to submitted assets.Source: Table S6.1 · p. 22
• The README describes the files and aggregation procedure, while the CSV and JSON assets use UTF-8 encoding.Source: S6 Artifact Index and Code Availability · p. 22
• Full training and evaluation code will be released publicly upon acceptance.Source: S6 Artifact Index and Code Availability · p. 22