PaperlayerView exact revision on arXiv

Source-linked AI summary

MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching

Hao Yin, Paritosh Parmar, Lijun Gu, Lin Xu, Tianxiao Guo, Xiujin Liu, Tianyou Zheng, Yang Zhang, Weiwei Fu

arXiv:2608.26094v1cs.CVcs.AIcs.ETcs.HCcs.LG

TL;DR

Existing AQA systems mainly use visual inputs and monolithic action models, leaving physiological dynamics and fine-grained corrective reasoning underrepresented. MyoMechanix combines synchronized multimodal sensing with the FKG and CUBIST, and establishes AQA, VideoQA, and Video2EMG benchmarks. Experiments report improved performance, interpretability, and error attribution, while Video2EMG shows promise for video-based muscle-activity sensing.

  • Problem

    Existing AQA datasets and methods rely primarily on visual cues, overlook physiological dynamics, and lack fine-grained compositional reasoning for corrective feedback.

  • Method

    MyoMechanix combines synchronized multiview video, 3D pose, sEMG, and other physiological signals with expert annotations, the FKG, and CUBIST’s decomposition–analysis–recomposition framework.

  • Results

    Multimodal sensing and structured representations improve performance, interpretability, and error attribution; CUBIST achieves state-of-the-art results, while VideoQA and Video2EMG show promising capabilities.

  • Takeaways & Limitations

    MyoMechanix advances skilled activity understanding toward biomechanically grounded, multimodal, and compositional reasoning for fitness, rehabilitation, and healthcare.

  • Takeaways & Limitations

    Performance degrades under substantial distribution shifts across subjects and viewpoints, leaving action-quality disentanglement challenging.

Abstract

from arXiv · show

Existing action quality assessment (AQA) datasets and methods rely primarily on visual inputs such as RGB and pose, overlooking physiological dynamics such as muscle mechanics and often modeling actions as monolithic patterns. These limitations hinder fine-grained, biomechanically grounded feedback. We introduce MyoMechanix, a multimodal ecosystem for weight-loaded actions that aligns motion with muscle activity. Expert-annotated, it contains 7,500+ samples of 20 actions from 38 subjects, with synchronized multiview RGB video, 3D pose, sEMG, and additional physiological signals, forming the largest multimodal AQA benchmark to date. We further construct the Fitness Knowledge Graph (FKG), which organizes expert annotations into structured relationships among actions, phases, key steps, errors, and corrective feedback, enabling compositional scoring and interpretable assessment. Building on these representations, we develop CUBIST (Compositional Ontological Reasoning Engine), which performs decomposition-analysis-recomposition for fine-grained error attribution and feedback generation. We also establish MyoMechanix-AQA, MyoMechanix-VideoQA, and a novel MyoMechanix-Video2EMG task. Experiments show that multimodal sensing and structured representations improve performance, interpretability, and error attribution, with CUBIST achieving state-of-the-art results; VideoQA enhances language-grounded action understanding; and Video2EMG suggests video-based alternatives to costly EMG sensing. MyoMechanix advances skilled activity understanding toward biomechanically grounded, multimodal, and compositional reasoning for Physical AI applications in fitness, rehabilitation, healthcare, and machine learning. Project page: https://haoyin116.github.io/MyoMechanix/

1 Introduction

MyoMechanix addresses the limits of visually focused, monolithic action assessment by combining physiological sensing with structured, compositional representations. It introduces a multimodal ecosystem, the Fitness Knowledge Graph, CUBIST, and benchmarks for action quality, VideoQA, and Video2EMG.

  • Motivation: Existing AQA methods emphasize visual kinematics and treat actions monolithically, limiting access to physiological states and fine-grained error reasoning.These approaches struggle to capture muscle activation, fatigue, coordination strategies, and component-level causes of execution errors.
  • MyoMechanix ecosystem: MyoMechanix combines synchronized multiview video, motion capture, and sEMG for weight-loaded exercise assessment.The ecosystem contains over 7,500 samples across 20 exercise types and 38 subjects, with more than 40 hours of multimodal recordings.
  • Structured representation: The Fitness Knowledge Graph organizes actions into hierarchical phases and ordered key steps linked to errors and corrective feedback.This structure captures procedural organization and relationships between execution errors and their consequences.
  • Compositional reasoning: CUBIST decomposes actions, analyzes components with multimodal inputs, and recomposes results into interpretable assessments, error attribution, and corrective feedback.Its design moves beyond holistic scoring toward step-wise reasoning aligned with expert analysis.
  • Benchmarks and results: The ecosystem establishes MyoMechanix-AQA, MyoMechanix-VideoQA, and Video2EMG for structured assessment, language-grounded understanding, and visual inference of muscle activation.Experiments report improved performance, interpretability, and error attribution from multimodal sensing and structured representations, with CUBIST achieving state-of-the-art performance.
  • Implications: The work advances action understanding toward biomechanically grounded, multimodal, and compositional reasoning for fitness, rehabilitation, and smart healthcare applications.The stated scope is physically grounded AI for skilled activity understanding and coaching.

2 Related Work

Prior AQA, VideoQA, and EMG datasets and models provide limited physiological coverage, structured reasoning, corrective feedback, and weight-loaded exercise diversity. MyoMechanix addresses these gaps through expert-grounded multimodal data, the FKG, compositional scoring, CUBIST, and fine-grained VideoQA and Video2EMG tasks.

  • AQA datasets: Web-crawled and emerging controlled datasets provide limited ability-level diversity and commonly overlook implicit physiological states.MyoMechanix instead collects weight-training data in-house to obtain a more balanced skill-level representation.
  • AQA datasets: Existing fitness AQA datasets largely focus on self-loaded or low-load actions and cover too few exercises to represent weightlifting biomechanics.They often omit load magnitude, joint torque distribution, deep muscle activation, and complex neuromuscular demands.
  • Interpretability: Existing AQA annotations often provide only final scores or phase labels, lacking fine-grained breakdowns and intermediate reasoning steps.This limits interpretable modeling of how execution details contribute to assessments.
  • Structured assessment: MyoMechanix uses the FKG to convert expert knowledge into structured error annotations and a compositional scoring system based on error weights.The benchmark provides macroscopic scores, microscopic execution parameters, and interpretable intermediate reasoning annotations.
  • VideoQA: MyoMechanix-VideoQA supports language-conditioned reasoning about action phases, errors, muscle involvement, causal relations, and corrective strategies.Its graph-grounded question–answer pairs provide expert-grounded supervision for diagnosis and actionable feedback.
  • Video2EMG: Video2EMG estimates continuous muscle activation patterns from visual input using real sEMG as physiological supervision.The task offers a potential video-based surrogate for specialized EMG sensing rather than predicting only discrete muscle labels or binary activation states.

3.1 Motivation

MyoMechanix is designed as a multimodal, expert-annotated testbed for weight-loaded action understanding. Its synchronized physiological sensing and Fitness Knowledge Graph connect visible movement with muscle mechanics and structured corrective reasoning.

  • Dataset motivation: MyoMechanix addresses benchmark gaps through in-house acquisition, synchronized multiview recordings, physiological sensing, and expert annotations.Recorded signals include sEMG, heart rate, and respiratory rate, while experts provide comprehensive error labels and actionable feedback.
  • Dataset motivation: The dataset bridges explicit visual movement features with implicit muscle mechanics for biomechanically grounded Physical AI.Its scope centers on weight-training actions, whose external loads introduce greater biomechanical complexity and injury-related assessment challenges.
  • Fitness Knowledge Graph: The FKG encodes actions, action key steps, fine-grained error types, and prescriptive feedback to support structured and interpretable reasoning.It represents both procedural action structure and relationships between execution errors and corrective guidance.

3.2 MyoMechanix Design

MyoMechanix is designed for realistic, biomechanically challenging weight-loaded exercise assessment by combining diverse actions, subjects, viewpoints, and physiological signals. Its multimodal design addresses limitations of low-load, visually focused benchmarks.

  • Weight-loaded exercises increase joint torque, muscle activation, neuromuscular coordination demands, and injury risk compared with bodyweight movements.
  • The benchmark targets gaps in existing datasets, including limited load modeling, narrow exercise coverage, and insufficient diversity of realistic weight-loaded conditions.
  • MyoMechanix selects 20 representative compound actions spanning upper- and lower-body muscle groups and complex, injury-prone joints.
  • Four synchronized cinema cameras plus a smartphone viewpoint reduce occlusion and simulate typical user-recording scenarios.
  • The dataset combines multiview video, 3D motion capture, and physiological measurements including EMG, heart rate, and respiratory rate.
  • Participants span novice, amateur, and expert proficiency levels to represent varied movement patterns, error modes, and performance qualities.

3.3 Data Acquisition Protocol

The acquisition protocol prioritizes participant safety, controlled multimodal recording, and reduced external influence. Continuous repetitions also capture how action quality changes over time.

  • Subjects performed risk-prone weight-loaded actions at 80% of their one-repetition maximum with insurance, safety supervisors, emergency kits, and IRB review.
  • Staff securely attached customized vests and sEMG sensors, while physiological signals were monitored for internal exertion and fatigue-related responses.
  • Subjects entered individually, received only brief demonstrations, and performed without textual descriptions to minimize external influence.
  • Each subject completed 10 continuous repetitions per action to support modeling the temporal evolution of action quality.

3.4 Expert-Guided Data Annotation Framework

MyoMechanix uses expert-guided, biomechanically structured annotations that connect action phases and keysteps to errors, feedback, and transparent scores. The Fitness Knowledge Graph provides the organizing representation for this process.

  • The annotation framework integrates fitness guidelines, biomechanical phase decomposition, structured error taxonomies, the FKG, and compositional scoring.
  • Expert verification supports consistent annotations despite variation in fitness goals, training contexts, and coaching practices.
  • Actions are decomposed into Preparation, Concentric, and Eccentric phases, with finer-grained action keysteps inside each phase.
  • Error penalties are weighted hierarchically according to effects on joint safety, core stability, movement trajectory, force generation, posture, amplitude, and fine control.
  • Each exercise is represented through phases, ordered action keysteps, phase-conditioned errors, and corrective feedback within the Fitness Knowledge Graph.
  • Compositional scoring builds the overall quality score bottom-up from annotated error penalties, making deductions transparent and verifiable.

3.5 Benchmark Design

The benchmark suite extends structured exercise assessment into coaching, fine-grained VideoQA, and video-to-EMG prediction. Its protocols test both standard performance and generalization across subjects and viewpoints.

  • Benchmark tasks: MyoMechanix establishes three tasks: Action Quality Assessment and Coaching, Fine-Grained Video Question Answering, and Video-to-EMG Prediction.
  • MyoMechanix-AQA & Coaching: The AQA and Coaching benchmark covers quality estimation, error identification, score-deduction explanation, and rubric-grounded corrective feedback.
  • Evaluation protocols: Evaluation uses Vanilla, Cross-Subject, Cross-View, and Mix-View splits, with 60:20:20 training, validation, and test partitions.
  • MyoMechanix-VideoQA: MyoMechanix-VideoQA uses FKG-grounded questions spanning descriptive, relational, temporal, and causal-feedback reasoning.
  • MyoMechanix-VideoQA: VideoQA is designed as a language-grounded, query-conditioned platform for fine-grained action understanding rather than fixed assessment alone.
  • Video2EMG: Video2EMG predicts continuous sEMG from video using paired visual and physiological signals, targeting settings where wearable sensors are constrained.

3.6 Dataset Statistics

MyoMechanix contains 7,512 balanced action samples spanning 20 weight-loaded fitness actions and 38 subjects, with annotations designed for fine-grained reasoning and actionable coaching. Its textual annotations are detailed, linguistically demanding, and rich in causal explanations and corrective recommendations.

  • Dataset scale: 7,512 samples span 20 distinct weight-loaded fitness actions performed by 38 subjects across three ability levels.Per-action counts are balanced between 370 and 380 samples.
  • Textual statistics: FK Grade 13.78 and FRE 30.35 place the annotations at a relatively difficult reading level.The reported FK Grade approximately corresponds to first-year college reading in the United States.
  • Reasoning and actionability: Each sample contains an average of 1.85 explicit reasoning steps, with causal phrases linking movement patterns to functional consequences.The example “which reduces” connects an open grip to reduced control.
  • Reasoning and actionability: Each sample averages 8.41 actionable recommendations grounded in observed erroneous movement patterns.Examples include directive, corrective, and effect-oriented guidance such as keeping the head against the bench.
  • Reasoning and actionability: These causal and actionable annotations support models that identify errors, reason about consequences, and generate sample-grounded improvement feedback.The resulting benchmark is intended to evaluate fine-grained multimodal understanding and reasoning.

4 Our CUBIST Modeling Paradigm

CUBIST replaces monolithic action scoring with ontology-guided, phase-aware reasoning that decomposes actions, analyzes errors, and recomposes interpretable assessments and coaching feedback. Its architecture routes actions to expert pathways, processes multimodal temporal features, and addresses sparse motion-error detection with asymmetric optimization.

  • Modeling paradigm: CUBIST decomposes actions into components, analyzes each through multimodal modules, and recomposes results into assessments, error attribution, and corrective feedback.The paradigm is built on the Fitness Knowledge Graph and follows decomposition–analysis–recomposition.
  • Stage I—Action Perception-driven Expert Routing: Stage I routes each video to an action-specific expert pathway using a spatiotemporal encoder that also produces features for later error reasoning.Action classification is trained with cross-entropy loss and label smoothing.
  • Stage I—Action Perception-driven Expert Routing: The video backbone partitions inputs into tubelets with temporal size ts=2 and spatial patch size p=16, producing L = ⌊T/ts⌋×(H/p)2 tokens.The token embedding dimension is D.
  • Stage I—Action Perception-driven Expert Routing: A learnable-query attention pool aggregates spatiotemporal tokens into a global feature before an MLP produces action-category logits.The pooling uses multi-head attention, layer normalization, and squeezing of the query dimension.
  • Stage II—Phase-Aware Error Query Decoder: Stage II uses the Stage I action prior and ontology-driven reasoning to decode heterogeneous, phase-aware error types sequentially.A temporal adapter recovers temporal structure and models local dependencies with convolutional residual blocks.
  • Stage II—Phase-Aware Error Query Decoder: An asymmetric penalty suppresses the dominance of abundant easy negatives in sparse multi-label error detection, improving rare-error detection and generalization robustness.The mechanism uses a negative-sample probability margin and separate positive and negative focusing weights.
  • Cross-Action Shared Skill Modeling for Expert Adaptation: Cross-action shared skill modeling first learns transferable spatiotemporal representations across action categories, then adapts initialized expert models with progressive unfreezing.The adaptation initially freezes the backbone while training downstream modules.

5 Experiments

Experiments evaluate MyoMechanix through AQA, VideoQA, and Video2EMG, using unimodal, multimodal, multiview, and varied split protocols. Multimodal and structured representations improve assessment and interpretability, while CUBIST achieves the reported AQA state of the art and sEMG contributes physiological information beyond numerical accuracy.

  • Experimental scope: The experiments cover AQA, structured multimodal VideoQA, and the novel Video2EMG task for predicting muscle activation from visual observations.AQA includes standard and disjoint splits, while VideoQA targets diagnosis and corrective feedback.
  • Experimental scope: The AQA framework compares unimodal and multimodal models across Vanilla, Cross-Subject, Cross-View, and Mix-View protocols.The study includes vision, pose, and specialized EMG branches with synchronized multimodal cropping.
  • 5.1.4 Results: Table 5 reports improvement as modalities and views are incorporated, with the CUBIST paradigm achieving new SOTA performance.The table summarizes average performance across evaluation protocols.
  • 5.1.4 Results: ρ = 0.7829 and R −ℓ2 = 2.1542: CUBIST achieves new SOTA AQA performance, surpassing the strongest reported vision baseline DAE at ρ = 0.7431.The comparison is reported on the Vanilla split.
  • 5.1.4 Results: sEMG alone achieves ρ = 0.2046, while providing direct information about muscle activation, co-contraction, fatigue signatures, force imbalances, and compensatory recruitment.Its physiological information extends beyond visual and kinematic modalities even when standalone numerical performance is limited.

Interpretable Error Detection

The paper evaluates interpretable, compositional action-quality assessment using multimodal inputs and structured representations, with particular attention to generalization across subjects and views.

  • Multimodal Ablation: In single-view evaluation, tri-modal CUBIST reaches ρ = 0.8455, a 7.99% gain over unimodal CUBIST.The corresponding R −ℓ2 value is 1.0198.
  • Results: Multimodal fusion improves AQA performance by combining views, skeleton keypoints, and sEMG.Skeleton keypoints and multiple views increase ρ by 7.76% and 6.32%, while sEMG adds a stable 2.27% gain.
  • Results: CUBIST achieves state-of-the-art performance in both vision-only and multimodal evaluations.The results attribute this performance to FKG design, multimodal information, and the CUBIST paradigm.
  • Generalization: Under challenging cross-subject and cross-view splits, advanced models exhibit noticeable performance degradation.These protocols test whether models can disentangle action quality from subject- and viewpoint-specific variations.
  • Limitations: The benchmark leaves substantial opportunities for future work in model design, multimodal fusion, generalization, and interpretability.The paper identifies considerable room for improvement across all splits.

5.2 Video Question Answering

MyoMechanix-VideoQA tests whether vision-language models can produce query-conditioned, fine-grained biomechanical assessments. Fine-tuning improves language generation, but score prediction remains substantially weaker than specialized AQA systems.

  • Task and Evaluation: The VideoQA dataset evaluates action recognition, fine-grained error diagnosis, and feedback generation with vision-language models.Testing compares off-the-shelf, rules-based prompting, and supervised fine-tuning configurations.
  • Results: Prompting with domain-specific scoring rubrics yields moderate, consistent improvements across most metrics.The results suggest inference-time guidance helps align model outputs with task expectations.
  • Results: Supervised fine-tuning raises BLEU from 0.0415 to 0.1970 and CHRF++ from 39.5308 to 52.0005.These gains concern text-generation quality relative to prompt-only adaptation.
  • Results: Fine-tuning improves diagnostic text generation more than numerical score prediction.The R−ℓ2 error decreases by less than 2% across adaptation stages.
  • Qualitative Analysis: Current VLMs recognize action classes robustly but still struggle with kinematic nuances and fine-grained quality assessment.A qualitative example shows detailed biomechanical fault identification under fine-tuning, while prompt-only output is largely affirmative.
  • Limitations: VLM score prediction lags specialized AQA algorithms by up to 80%, despite supervised fine-tuning.The paper reports approximately 50% room for improvement in text generation.

5.3 Novel Technology—Video2EMG

Video2EMG estimates muscle-activation waveforms from ordinary video, offering a cross-modal alternative to costly wearable EMG sensing. Baselines show feasibility on the vanilla split, but waveform alignment degrades under subject and viewpoint shifts.

  • Task and Baselines: Video2EMG predicts internal physiological dynamics from visual input using end-to-end spatiotemporal visual regression models.The baseline designs use visual encoders and draw on time-series waveform analysis and emg2pose principles.
  • Metrics: The task is evaluated with MAE, RMSE, and Cross-Correlation Peak (CCP).CCP measures maximum normalized cross-correlation between target and predicted waveforms.
  • Results: The best vanilla-split configuration achieves MAE = 0.1465, RMSE = 0.1882, and CCP = 0.4345.These results follow 150 training epochs under three evaluation protocols.
  • Architectural Insights: Frozen ResNet-50 and ViT-S/16 backbones produce only a marginal performance gap in the tested setup.Both provide comparably mature high-level semantic features when used as frozen feature extractors.
  • Robustness Analysis: Baseline CCP reaches moderate waveform alignment on the vanilla split but declines noticeably under cross-subject and cross-view protocols.The decline indicates sensitivity to individual physiological variation and viewpoint shifts.
  • Limitations: Video-based muscle-level feedback is feasible, but Video2EMG remains highly challenging and far from solved.The paper estimates about 56% room for improvement based on CCP’s upper bound of 1.0.

6 Conclusion

MyoMechanix establishes a multimodal, biomechanically grounded foundation for compositional action quality assessment, combining synchronized physiological sensing, expert annotations, structured reasoning, and CUBIST. The ecosystem supports interpretable scoring, error attribution, and feedback generation for skilled activity understanding.

  • MyoMechanix ecosystem: MyoMechanix synchronizes multiview RGB video, 3D pose, sEMG, and complementary physiological signals for fine-grained action understanding.The ecosystem targets weight-loaded actions and provides expert annotations aligned with internal and external signals.
  • Structured reasoning: The Fitness Knowledge Graph encodes relationships among actions, phases, execution steps, errors, and corrective feedback.This structure provides a semantically grounded formulation of action quality assessment.
  • Compositional assessment: CUBIST decomposes actions into structured units, analyzes their quality, and recomposes them for holistic scoring, error attribution, and feedback generation.Its compositional reasoning framework connects intermediate analysis with final assessment.
  • Broader significance: The work establishes action understanding as multimodal, biomechanically informed, and compositionally interpretable.The authors identify applications in fitness, rehabilitation, healthcare, and representation learning.
  • Data availability: The full dataset and codebase are publicly released under the CC BY-NC-SA 4.0 license.The release is intended to support future research using the MyoMechanix ecosystem.
Loading 2608.26094v1…