PaperlayerView exact revision on arXiv

Source-linked AI summary

Encoder-Free Human Motion Understanding via Structured Motion Descriptions

Yao Zhang, Zhuchenyang Liu, Thomas Ploetz, Yu Xiao

arXiv:2604.21668v2cs.CV

TL;DR

Existing LLM-based motion understanding methods rely on learned encoders and cross-modal alignment, limiting direct use of language-model knowledge. The paper introduces deterministic Structured Motion Description, which converts motion into structured text for LoRA-adapted LLMs. SMD surpasses prior results on motion QA and captioning while supporting backbone portability and interpretable attention analysis.

  • Problem

    Existing LLM-based motion methods typically learn motion-language alignment through dedicated encoders, rather than directly exploiting pretrained language representations.

  • Method

    Structured Motion Description deterministically converts joint-position sequences into structured natural-language descriptions of joint angles, body-part movements, and global trajectory.

  • Results

    SMD surpasses prior state-of-the-art on both motion QA and captioning, including 66.7% accuracy on BABEL-QA and 90.1% on HuMMan-QA.

  • Takeaways & Limitations

    The same SMD text transfers across 8 LLMs from 6 model families with lightweight LoRA adapters and supports attention analysis over human-readable tokens.

  • Takeaways & Limitations

    SMD has longer inputs than VAE-based methods and targets 26 angles over 22 SMPL joints, potentially missing finer hand and finger articulation.

Abstract

from arXiv · show

The world knowledge and reasoning capabilities of text-based large language models (LLMs) are advancing rapidly, yet current approaches to human motion understanding, including motion question answering and captioning, have not fully exploited these capabilities. Existing LLM-based methods typically learn motion-language alignment through dedicated encoders that project motion features into the LLM's embedding space, remaining constrained by cross-modal representation and alignment. Inspired by biomechanical analysis, where joint angles and body-part kinematics have long served as a precise descriptive language for human movement, we propose \textbf{Structured Motion Description (SMD)}, a rule-based, deterministic approach that converts joint position sequences into structured natural language descriptions of joint angles, body part movements, and global trajectory. By representing motion as text, SMD enables LLMs to apply their pretrained knowledge of body parts, spatial directions, and movement semantics directly to motion reasoning, without requiring learned encoders or alignment modules. We show that this approach goes beyond state-of-the-art results on both motion question answering (66.7\% on BABEL-QA, 90.1\% on HuMMan-QA) and motion captioning (R@1 of 0.584, CIDEr of 53.16 on HumanML3D), surpassing all prior methods. SMD additionally offers practical benefits: the same text input works across different LLMs with only lightweight LoRA adaptation (validated on 8 LLMs from 6 model families), and its human-readable representation enables interpretable attention analysis over motion descriptions. Code, data, and pretrained LoRA adapters are available at https://yaozhang182.github.io/motion-smd/.

1 Introduction

Human motion understanding requires fine-grained reasoning over skeletal sequences for tasks including motion QA and captioning. SMD replaces learned motion embeddings with deterministic structured text that LLMs can process directly.

  • Motivation: Motion QA and captioning connect low-level skeletal data with fine-grained and high-level semantic understanding.QA reasons about body parts, directions, and actions, while captioning summarizes overall motion content.
  • Limitations of prior methods: Most LLM-based methods use learned motion encoders and alignment modules to project joint sequences into an LLM’s token space.These approaches may use discrete, continuous, or linearly projected motion representations.
  • Limitations of prior methods: Encoder-based pipelines require multi-stage paired motion-text training, depend on a specific LLM backbone, and produce non-human-readable motion tokens.These practical considerations constrain representation and alignment between motion and language.
  • Proposed approach: Structured Motion Description (SMD) deterministically converts joint-position sequences into structured natural-language text and feeds it directly to LLMs.The approach uses language about body parts, spatial directions, and movement semantics rather than learned motion embeddings.
  • Proposed approach: SMD draws on biomechanical joint angles and body-part kinematics to describe motion precisely in natural language.The motivation is illustrated by descriptions such as hip flexion increasing from 3° to 81°.
  • Reported benefits: With lightweight LoRA fine-tuning, SMD reports strong motion QA and captioning performance, portability across 8 LLMs from 6 families, and attention-based interpretability.The same text input is reused across backbones, with adapters of approximately 40M parameters and 2–8 GPU-hours on one H200.

2 Related Work

Prior work represents motion through numerical features, interpretable rule-based pose descriptions, or learned motion-language systems. SMD builds on biomechanical joint-angle representations while targeting motion QA and captioning with structured text.

  • Motion representation: HumanML3D commonly represents motion with high-dimensional numerical features including velocities, joint positions, rotations, and foot-contact labels.These expressive features require learned encoders to interface with LLMs.
  • Motion representation: PoseScript and PoseFix generate rule-based natural-language descriptions of static poses for retrieval and correction tasks.Their descriptions are interpretable but focus on static poses rather than the full motion-understanding setting addressed here.
  • Motion representation: Zhang et al. map biomechanical joint angles from skeleton sequences into structured pseudo-images for motion retrieval with vision transformers.SMD also uses joint-angle representations but converts them into text.
  • Motion understanding: Motion QA research includes task-specific neuro-symbolic and modular systems that reason over learned motion features.NSPose introduced BABEL-QA, while IMoRe is another representative motion-QA method.
  • Motion understanding: Motion captioning evolved from joint embedding methods toward LLM systems using VQ-VAE tokens, continuous VAE latents, and multi-granularity frameworks.These methods share learned motion representations aligned with language models.

3 Method

SMD converts skeleton sequences into structured text describing joint angles and global trajectory, then conditions LoRA-adapted LLMs for QA and captioning. The pipeline is deterministic, encoder-free, and organized around interpretable motion components.

  • Pipeline: Figure 2 presents two stages: deterministic conversion of skeleton sequences into SMD, followed by LoRA fine-tuning of a pretrained LLM.The method uses no learned motion encoder, VQ-VAE, or cross-modal alignment module.
  • SMD construction: fSMD maps a T-frame sequence of J 3D joint positions into a text string over the LLM vocabulary.For the SMPL-based representation, J = 22.
  • Joint angles: Joint-angle calculation produces scalar time series for biomechanical angles using bone vectors, anatomical planes, and joint-local coordinate frames.The resulting signed angles distinguish flexion from extension and follow kinematic chains rooted at a pelvis-based body frame.
  • Joint angles: The representation contains 26 biomechanical joint angles organized into 13 body-part groups with anatomically meaningful descriptions.The conventions support other skeleton formats when the relevant landmarks are available.
  • Global trajectory: Global trajectory complements joint angles by describing pelvis displacement along forward, lateral, and height axes plus body yaw.Smoothing and peak-valley segmentation suppress positional jitter while preserving motion intervals.
  • Temporal segmentation: Angle time series are segmented using a 5° minimum angular-change threshold and labeled as increases, decreases, holds, or repeated cycles.A hip-flexion example compresses 200 frames into three descriptive intervals while preserving temporal structure.
  • SMD assembly: SMD assembles metadata, global trajectory segments, and body-part-grouped joint-angle segments into one hierarchical text string.All 26 angles require approximately 4,000 tokens, while Top-3 active-joint selection uses approximately 1,000 input tokens.
  • Task formulation: QA and captioning are formulated as autoregressive text generation conditioned on SMD, with task-specific prompts and target responses.Training masks the prompt and computes loss only on target response tokens; LoRA adapters are trained with frozen base LLM weights.

4 Experiments and Analysis

Experiments evaluate SMD on motion QA and captioning, then analyze joint selection, trajectory encoding, parameter sensitivity, zero-shot behavior, and backbone portability. SMD achieves strong results across tasks and models, while trading shorter inputs for longer text sequences and slower inference.

  • Main Results: 66.7% accuracy on BABEL-QA and 90.1% on HuMMan-QA, surpassing IMoRe by 6.6 and 14.9 percentage points, respectively.Using the same Qwen2.5-7B backbone, MotionGPT3-Qwen reaches 50.1% on BABEL-QA and 22.0% on HuMMan-QA.
  • Main Results: SMD achieves the best HumanML3D captioning results on all metrics except R@1 and improves R-Precision at every level and MM-Distance over MotionGPT3.Relative gains include +17% on BLEU@4 and +31% on CIDEr; BERTScore increases from 35.23 to 45.58.
  • Ablation Studies: Top-3 joints already gives QA accuracy of 73.3%/91.0%, whereas captioning R@1 rises from 0.452 with trajectory-only input to 0.584 with All-26 joints.The ablation indicates that QA benefits from focusing on active joints, while open-ended captioning benefits from richer full-body descriptions.
  • Ablation Studies: Absolute trajectory performs best overall, particularly for retrieval, while removing trajectory decreases BABEL-QA accuracy by only 1.8 points and leaves captioning CIDEr comparable.The reported CIDEr values are 53.34 without trajectory and 53.16 with the compared representation.
  • Ablation Studies: SMD conversion parameters are stable: BABEL-QA varies by at most 4 points and captioning R@1 by at most 0.08 across tested settings.Some off-default settings edge out the defaults on R@1 or CIDEr, suggesting light per-task tuning may yield modest gains.
  • Backbone Portability: Zero-shot SMD reaches 35.6% on BABEL-QA and 31.7% on HuMMan-QA, while LoRA fine-tuning is needed for action-level captioning and concise output.Across 8 LLMs from 6 families, performance remains consistent: BABEL-QA exceeds 63%, HuMMan-QA exceeds 82%, R@1 ranges from 0.517–0.563, and CIDEr from 49.23–54.33.
  • Backbone Portability: SMD uses approximately 1,000–4,000 tokens rather than approximately 256 motion tokens, yielding roughly 1 sample/s inference on a single H200.The representation avoids a motion encoder and projection layer but incurs a longer input sequence and an associated inference-cost trade-off.
  • Interpretability: Attention heatmaps show selective focus on trajectory segments, cyclic joint-angle patterns, or right-shoulder and right-elbow segments relevant to each caption.The analysis aggregates attention weights over the SMD input across all layers of fine-tuned Qwen2.5-7B.

5 Conclusion

SMD converts human motion into structured text, allowing LLMs to perform motion understanding without learned encoders or cross-modal alignment. With lightweight LoRA fine-tuning, it achieves strong performance across tasks and models, while its fixed representation and long inputs impose practical limits.

  • SMD converts human motion into structured text without learned encoders or cross-modal alignment.
  • SMD surpasses prior state-of-the-art on motion QA and captioning with lightweight LoRA fine-tuning.It transfers across 8 LLMs from 6 families and provides attention-based interpretability over human-readable tokens.
  • Inference latency is the primary limitation because All-26 SMD uses approximately 4,000 tokens, about 15× longer than VAE-based inputs.Training cost is unaffected because training requires a single LoRA run.
  • The rule-based representation covers 26 biomechanical angles over 22 SMPL joints and may miss nuances such as hand and finger articulation.
  • Future work includes learned SMD generation, motion-adaptive segmentation thresholds, and applications to motion generation and editing.

A Method Details

SMD computes biomechanical joint angles and trajectory changes in local coordinate frames, then expresses them as structured, temporally segmented natural-language descriptions. The representation organizes 26 angles across 13 body-part groups and supports task-specific LLM prompts for QA and captioning.

  • Coordinate frames: The body-local frame uses the pelvis, left hip, and right hip to define the subject’s overall facing direction.
  • Coordinate frames: Joint-local frames express each angle relative to its parent segment along the kinematic chain.Examples include knee flexion in the hip frame and elbow flexion in the shoulder frame.
  • Angle representation: The representation contains 26 angles organized into 13 body-part groups, including pelvis, spine, neck, hips, knees, ankles, shoulders, and elbows.
  • Angle representation: Angles use biomechanical sign conventions, unwrap rotational quantities, and assume complete 3D SMPL-format joint positions.Missing or noisy joints require upstream imputation, and 2D inputs need additional processing.
  • Angle computation: Axial twist is measured with a secondary reference vector because a single bone vector cannot determine rotation around its longitudinal axis.
  • Temporal description: Smoothed angle and trajectory time series are segmented by peak and valley detection into increases and decreases, with repeats and holds represented in SMD text.The full example covers all 13 body-part groups, while Top-K retains joints with the largest peak-to-trough ranges.
  • LLM interface: Motion QA prompts combine instructions, SMD, questions, and answer options, whereas captioning prompts combine instructions with SMD and target a natural-language caption.Only target response tokens contribute to the training loss.

B.1 Open-vocabulary QA on BABEL-QA

The open-vocabulary evaluation removes candidate answer options at test time to assess whether SMD-based QA depends on seeing those options. The model retains strong accuracy and exceeds the reported MotionLLM result on BABEL-QA, while zero-shot captioning remains descriptive but sometimes overly generic.

  • Evaluation protocol: The open-vocabulary protocol is harder for baselines because MotionLLM, IMoRe, and NSPose were trained with candidate options visible.
  • Open-vocabulary QA: Removing candidate options costs the model 8.2 and 7.1 percentage points on BABEL-QA and HuMMan-QA, respectively.
  • Open-vocabulary QA: 65.1% open-vocabulary accuracy on BABEL-QA exceeds MotionLLM’s original 43.6% by more than 20 points.
  • Zero-shot behavior: Zero-shot walking captions identify lateral sway, torso rotation, and arm swinging but are more verbose than the concise reference action.
  • Zero-shot behavior: Zero-shot waltz captions recognize arm and leg movements but describe the action generically instead of identifying the waltz.

B.3 Feature distribution comparison: BABEL-QA vs HuMMan-QA

The feature-distribution analysis finds BABEL-QA nearly in-distribution for MotionGPT3’s VAE and HuMMan-QA only mildly shifted. This small shift contrasts with a much larger HuMMan-QA accuracy drop for the encoder-based baseline, motivating SMD’s source-robust text representation.

  • Evaluation method: The comparison measures per-dimension absolute mean, standard deviation, and Wasserstein-1 distance relative to HumanML3D’s VAE training distribution.
  • Distribution statistics: BABEL-QA is essentially in-distribution for the VAE, with overall |meanz| = 0.10, overall stdz = 0.98, and W1 = 0.14 to HumanML3D.
  • Distribution statistics: HuMMan-QA shows a mild shift, with |meanz| = 0.25, stdz = 0.68, and W1 = 0.36 to HumanML3D.Its W1 remains well below the W1 > 2 threshold for a meaningfully different distribution.
  • Interpretation: MotionGPT3-Qwen’s accuracy drops from 50.1% to 22.0% on HuMMan-QA despite the modest W1 = 0.36 input shift.
  • Interpretation: SMD is described as producing statistically equivalent text across the datasets, avoiding dependence on the VAE’s acquisition-source distribution.

C.1 Cross-skeleton generalization (NTU RGB+D 60)

SMD generalizes to NTU RGB+D 60 despite differences in sensor and skeleton topology, but zero-shot recognition remains limited by unseen action vocabulary and skeleton-only ambiguity. One epoch of LoRA fine-tuning substantially closes this gap.

  • Generalization setup: NTU RGB+D 60 tests SMD across a Kinect sensor and 25-joint topology distinct from the 22-joint SMPL representation.The mapping retains 19 direct counterparts, interpolates three SMPL joints, and drops four finger/thumb joints.
  • Zero-shot limitations: 20.5% zero-shot accuracy is modest, with strong performance on locomotor and gestural actions but failures on unseen everyday activities and object-dependent interactions.The zero-shot result is approximately twice the 10% chance level.
  • Zero-shot limitations: The pre-training corpus exposes the model to a narrow action vocabulary centered on locomotion, body-part gestures, and fitness.The authors identify broader multi-task fine-tuning across diverse activity datasets as a future direction.
  • Fine-tuned transfer: 83.8% top-1 accuracy follows one epoch of LoRA fine-tuning on NTU60 X-Sub, a 63-point increase over zero-shot.The SMD conversion pipeline, prompt template, and architecture remain unchanged; only the LoRA adapter weights are updated.

C.2 Fine-grained motion captioning

SMD is well matched to fine-grained motion-to-detailed-text captioning because its time-localized, per-joint descriptions directly encode body-part movements. On FineMotion, it outperforms MG-MotionLLM at both sequence and snippet levels.

  • Task and data: FineMotion pairs 0.5-second motion snippets with automatically generated descriptions of individual body-part movements.Evaluation is performed on detailed multi-snippet motion descriptions rather than free-form overall-action captions.
  • Evaluation: The m2dt protocol evaluates generated descriptions at both sequence and snippet levels using BLEU, ROUGE, and BERTScore.Sequence metrics score the complete document, while snippet metrics average scores over individually split temporal descriptions.
  • Evaluation: SMD retains its rule-based conversion while using a dedicated LoRA adapter trained on FineMotion, enabling a controlled comparison with MG-MotionLLM’s T5-Large configuration.The Qwen2.5-7B base, All-26 SMD conversion, and LoRA configuration are retained.
  • Results: SMD outperforms MG-MotionLLM on every metric at both evaluation levels.At sequence level, BLEU@1 improves by +5.22 points from 81.73 to 86.95, while BERTScore improves by +3.7; snippet-level BERTScore improves by +4.7.
  • Results: The snippet-level advantage indicates that SMD predictions remain semantically faithful when errors are penalized independently within each 0.5-second window.The authors attribute this fit to SMD’s body-part-and-time-indexed structure.

D.1 Datasets, baselines, and 10-option standardization

The evaluation standardizes motion QA to comparable option counts and assesses captioning with both linguistic and text-motion alignment metrics. Baselines span specialized motion models and encoder-based LLM approaches, including a controlled same-backbone replication.

  • QA datasets: BABEL-QA covers action, body-part, and direction questions, while HuMMan-QA follows the same format on the HuMMan dataset.The original benchmarks use varying option counts, complicating direct cross-method comparison.
  • QA protocol: The standardized QA protocol uses at most 10 options, sampling nine distractors plus the correct answer when original label sets are larger.Average option counts are 8.6 for BABEL-QA and 10.0 for HuMMan-QA.
  • Baselines: QA comparisons include retrained NSPose and IMoRe alongside MotionLLM’s original reported result, which is not directly comparable because it uses the original option format.IMoRe logits are masked to the candidate options before selecting the maximum.
  • Baselines: Captioning baselines include TM2T, MotionGPT, LaMP, MoTe, MotionGPT3, and MG-MotionLLM under the HumanML3D evaluation protocol.Prior baseline numbers are generally taken from MotionGPT3, with CIDEr recomputed on released checkpoints using a cleaned evaluation pipeline.
  • Baselines: MotionGPT3-Qwen provides a controlled same-backbone comparison by replacing SMD with a frozen 256-dimensional VAE latent projected into Qwen2.5-7B token embeddings.The projection uses a two-layer MLP before the LLM receives the motion representation.
  • Captioning metrics: Captioning evaluation combines linguistic metrics with text-motion alignment metrics based on a pretrained T2M evaluator.R-Precision measures retrieval success among 32 motions, whereas MM-Distance measures embedding-space Euclidean distance.

D.3 Training and inference cost comparison

SMD reduces training requirements by updating only lightweight LoRA adapters, but trades encoder compression for substantially longer input sequences and higher inference latency. Increasing projection-token capacity in the encoder baseline does not close the performance gap and can overfit.

  • Training cost: SMD retraining requires 2–8 GPU-hours for QA and 6–20 hours for captioning on a single H200, updating approximately 40M LoRA parameters while freezing the base LLM.The All-26 Qwen2.5-7B reference run accounts for the approximately 20 GPU-hour captioning figure.
  • Inference cost: SMD produces approximately 1,000 Top-3 tokens or 4,000 All-26 tokens, compared with approximately 256 tokens for encoder-based methods.The longer sequence is the principal inference trade-off identified for SMD.
  • Inference cost: Captioning latency is 915 ms/sample for Top-3 SMD and 1,154 ms/sample for All-26, versus 300 ms/sample for the same-backbone MotionGPT3-Qwen baseline.SMD uses approximately 15.5 GB of GPU memory, compared with 14.7 GB for MotionGPT3-Qwen.
  • Training cost: Encoder-based methods require multi-stage training involving hundreds of millions of parameters and tens of GPU-hours even for relatively small backbones.Their schedules include motion-encoder pretraining followed by projection or alignment training and LLM training.
  • Projection-token sweep: MotionGPT3-Qwen reaches only 47–50% BABEL-QA accuracy and 15–24% HuMMan-QA accuracy across projection-token configurations, below SMD’s 66.7% and 90.1%.Captioning performance degrades with 64 and 128 projection tokens because the larger projection layers overfit limited training data.

E Qualitative Analysis

Qualitative comparisons show that SMD produces more precise, body-part-grounded motion answers and captions than the baselines across the illustrated examples. Its interpretable biomechanical grounding also supports applications requiring body-part-level feedback.

  • Motion QA: SMD answers all four illustrated motion QA examples correctly across direction, body-part, and action questions.The examples span both BABEL-QA and HuMMan-QA, while both baselines select semantically related but incorrect options.
  • Motion QA: SMD’s grounded, body-part-indexed descriptions identify discriminative joints, whereas baselines confuse directions, body parts, or action categories.Reported baseline errors include reversing directions, selecting the wrong arm, and substituting related actions.
  • Motion Captioning: SMD captures multi-step, body-part-specific structure in all four illustrated HumanML3D captioning examples.Examples include sequential hand raises, repeated side steps, and throwing followed by catching.
  • Motion Captioning: Captioning baselines often produce generic descriptions, visually similar actions, or only coarse action labels without the discriminative body part.This contrasts with SMD captions that preserve the motion’s detailed sequence and body-part trace.
  • Practical Implications: Biomechanically grounded SMD representations are suited to applications needing interpretable, body-part-level feedback, including rehabilitation and sports settings.The paper specifically connects this grounding to virtual coaches, rehabilitation assistants, physical therapy tracking, and sports science.
Loading 2604.21668v2…