Source-linked AI summary
3rd Place Solution to Human Motion Challenges in Real-World and Clinical Settings (MoCha) @ECCV2026: Language-Aligned Motion Representations for Domain-Generalizable UPDRS-Gait Severity Estimation
Soojie Kim, Muhammad Munsif, Minkyung Kim, Seungryul Baek
TL;DR
The paper addresses UPDRS-gait severity estimation across heterogeneous clinical domains, where domain shifts can degrade models on unseen sites. It combines language-aligned Bi-GRU representation learning, pseudo labeling, and source-specific model merging, achieving 3rd place with 637K active inference parameters and no class-prior correction.
Problem
Clinical gait-motion distributions vary across subjects, cohorts, and acquisition sites, motivating representations that remain transferable to unseen domains.
Method
The framework trains language-aligned Bi-GRU motion representations, augments imbalanced data with GPT-5.5 pseudo labels, and merges independently adapted source-domain models.
Results
The method achieved 3rd place overall in the MoCha challenge and uses approximately 637K active parameters during inference.
Takeaways & Limitations
The final framework directly predicts from motion observations without class-prior correction, while consolidating complementary source-domain knowledge for unseen-site evaluation.
Takeaways & Limitations
The method assumes metadata, class-distribution priors, and test-set-level statistics are unavailable for unseen subjects or clinical sites.
Abstract
from arXiv · showhide
In this work, we introduce language-aligned motion representations for domain-generalizable UPDRS-Gait severity estimation, aiming to learn semantically structured motion features that generalize across heterogeneous clinical domains. We first learn motion representations using a Bi-GRU backbone that captures the temporal dynamics of SMPL sequences. Prior to model training, motion captions are generated offline using Qwen2.5-7B-Instruct. The backbone is then trained with both classification and text-alignment objectives to learn discriminative and semantically structured motion representations while accounting for the class imbalance present in the training data. We subsequently adapt the learned backbone independently to each source domain so that the model can capture domain-specific motion characteristics. The resulting source-specific models are then merged at the parameter level to consolidate complementary knowledge across source domains into a single domain-generalized model. To further mitigate class imbalance, we perform GPT-5.5-based pseudo labeling, and our final merged models for each site do not use any class-prior correction during inference. The resulting model is evaluated under the unseen-site setting of the MoCha Challenge, using Macro F1 as the primary evaluation metric. Our method achieves a macro-F1 of 0.57 on the hidden test set with only 637K active parameters at inference, ranking 3rd among 58 leaderboard entries in the MoCha 2026 Challenge. The challenge attracted 1,669 submissions from 112 participants and offered monetary prizes sponsored by Machine Medicine Technologies.
1 Introduction
The paper addresses cross-site degradation in clinical gait recognition by learning semantically meaningful motion features that transfer across heterogeneous domains. Its framework combines language-aligned representation learning, source-domain adaptation, and parameter-level merging while predicting from motion alone.
- Domain shifts across subjects, cohorts, and acquisition sites can cause models to rely on domain-specific patterns and degrade on unseen environments.
- Language-aligned motion representations are investigated to capture semantically meaningful gait characteristics while remaining transferable across clinical domains.
- A Bi-GRU backbone encodes temporal dynamics and combines classification with text alignment to capture discriminative motion patterns and semantic relationships between motion classes.
- Source-specific adaptation followed by parameter-level merging treats clinical domains as complementary sources of motion knowledge rather than interchangeable samples.
- The framework assumes metadata, class-distribution priors, and test-set statistics are unavailable, producing predictions directly from motion observations.
2 Methods
The method uses complementary Bi-GRU branches to learn motion representations from classification, reconstruction, and caption-alignment objectives, then augments imbalanced training data with GPT-5.5 pseudo labels. Independently adapted source models are merged at the parameter level, and their predictions are averaged.
- The framework has three stages: caption-aligned motion pretraining, GPT-5.5-based pseudo labeling, and source-specific fine-tuning with parameter-level model merging.
- Caption-aligned Bi-GRU motion encoder: Offline captions describe gait characteristics including speed, stride, foot movement, arm swing, limb mobility, and postural stability for text-aligned representation learning.
- Caption-aligned Bi-GRU motion encoder: Bi-GRU motion representation learning trains complementary branches with classification plus motion reconstruction or caption-based semantic alignment.
- Pseudo labeling: GPT-5.5 assigns pseudo UPDRS-gait labels to unlabeled CARE-PD motions using clinically relevant motion cues to mitigate class imbalance.
- Source-specific adaptation: Each pretrained branch is independently fine-tuned on four source datasets, allowing adaptation to domain characteristics from a common initialization.
- Source-specific adaptation: SVD-based parameter merging suppresses unusually large domain-specific components while preserving updates that are consistently useful across domains.
- Inference: The two merged branches produce four-class UPDRS-gait probability distributions that are averaged during inference.
3 Experiments
The system was evaluated on the hidden-site MoCha challenge and achieved third place while using constrained information and a lightweight inference path. Pseudo labeling and model merging support direct inference without class-prior correction, with approximately 637K active parameters.
- Challenge performance: The entry achieved 3rd place overall on the MoCha challenge leaderboard.
- Information dependencies: The final model uses neither test-subject grouping nor test-time adjustment and does not employ class-weighted training.
- Class imbalance: Pseudo-labeled data produce a more balanced training distribution, allowing the final model to omit post-hoc logit adjustment and class-prior correction.
- Efficiency: The proposed model uses approximately 637K active inference parameters, compared with approximately 17.66M for the Simple ReCon baseline.
- Efficiency: 27.7× fewer active parameters reduce model storage requirements and inference computation relative to the Simple ReCon baseline.
4 Conclusion
The framework combines language-aligned Bi-GRU pretraining, GPT-5.5 pseudo labeling, source-specific fine-tuning, and parameter-level merging to improve UPDRS-gait generalization under domain shifts and class imbalance.
- The framework addresses UPDRS-gait classification under multi-domain distribution shifts and severe class imbalance.
- It combines caption-aligned Bi-GRU motion pretraining, GPT-5.5-based pseudo labeling, source-specific fine-tuning, and parameter-level model merging.
- Pseudo labeling constructs a more balanced training set, enabling inference without class-prior correction in the proposed framework.
- The approach aims to learn robust gait representations and reduce source-specific bias for improved generalization to unseen clinical domains.