Source-linked AI summary
Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction
Yao Zhang, Zhuchenyang Liu, Yanlan He, Thomas Ploetz, Yu Xiao
TL;DR
Text-motion retrieval needs fine-grained, interpretable alignment, but global-embedding methods discard local correspondences. The paper combines joint-angle motion images with MaxSim late interaction and MLM regularization, outperforming state-of-the-art baselines on HumanML3D and KIT-ML while producing interpretable correspondences.
Problem
Global-embedding retrieval methods discard fine-grained local correspondences and provide limited interpretability for text-motion alignment.
Method
The framework maps translation-invariant joint-angle features into structured pseudo-images and matches text tokens to motion patches with MaxSim, using MLM regularization for contextual token embeddings.
Results
The approach consistently surpasses state-of-the-art text-motion retrieval baselines on HumanML3D and KIT-ML while producing interpretable correspondence maps.
Takeaways & Limitations
Token-patch correspondences align textual semantics with specific body joints and temporal phases, supporting transparent retrieval verification and downstream motion applications.
Takeaways & Limitations
Dense patch embeddings increase gallery storage substantially, from approximately 4 MB for global vectors to approximately 837 MB.
Abstract
from arXiv · showhide
Text-motion retrieval aims to learn a semantically aligned latent space between natural language descriptions and 3D human motion skeleton sequences, enabling bidirectional search across the two modalities. Most existing methods use a dual-encoder framework that compresses motion and text into global embeddings, discarding fine-grained local correspondences, and thus reducing accuracy. Additionally, these global-embedding methods offer limited interpretability of the retrieval results. To overcome these limitations, we propose an interpretable, joint-angle-based motion representation that maps joint-level local features into a structured pseudo-image, compatible with pre-trained Vision Transformers. For text-to-motion retrieval, we employ MaxSim, a token-wise late interaction mechanism, and enhance it with Masked Language Modeling regularization to foster robust, interpretable text-motion alignment. Extensive experiments on HumanML3D and KIT-ML show that our method outperforms state-of-the-art text-motion retrieval approaches while offering interpretable fine-grained correspondences between text and motion. The code is available in the supplementary material.
1 Introduction
Text-motion retrieval seeks semantic alignment between 3D skeleton sequences and language for bidirectional search, but global embeddings lose fine-grained correspondences. The paper addresses this with joint-angle motion images and token-patch late interaction for interpretable alignment.
- Text-motion retrieval establishes a shared semantic space between 3D human motion skeletons and natural-language descriptions for bidirectional retrieval.
- Global-embedding methods compress motion and text into single vectors, limiting fine-grained alignment and retrieval interpretability.
- Joint angles decouple local joint movements from global trajectories and are projected into a structured 224×224 pseudo-image for pre-trained Vision Transformers.
- MaxSim replaces global alignment with token-to-patch matching, retaining each text token’s maximum similarity across motion patches for interpretable correspondence maps.
- The combined framework consistently surpasses state-of-the-art text-motion retrieval baselines on HumanML3D and KIT-ML while aligning text with body joints and temporal phases.
2 Background and Related Work
Prior work represents motion as skeletal features or position-based pseudo-images and commonly uses dual encoders, while late interaction preserves element-level correspondences. The paper uses translation-invariant joint angles and structured motion regions to support fine-grained cross-modal matching.
- Existing motion representations include frame-wise skeletal features and image-like patches, but raw joint-position motion images can obscure fine-grained kinematic differences.
- Joint Coordinate Systems decompose joint motion into clinically meaningful angular components, providing anatomical interpretability and translation-invariant features.
- The paper constructs a Motion Image whose spatial regions encode distinct joints, enabling fine-grained token-to-patch alignment.
- TMR uses independent motion and text encoders that map inputs into a shared latent space for cosine-similarity retrieval.
- Late interaction retains per-token representations and aggregates each token’s best match, while allowing gallery items to be pre-encoded offline.
3 Methodology
The framework converts joint-angle motion features into a structured Motion Image, encodes motion and text densely, and aligns them through token-patch MaxSim with text-only MLM regularization.
- 3 Methodology: The three-stage pipeline constructs a Motion Image, performs dual-stream encoding and late interaction, and applies context-aware MLM regularization during training.The third step is training-only, while the other stages operate during training and inference.
- 3.1 Joint-angle-based Motion Representation: Each joint’s projected Degrees of Freedom occupies a 16-pixel horizontal band in a 224×224 Motion Image, creating a one-to-one joint-to-patch mapping.Learnable projections unify joints with varying one-to-three-dimensional DoFs before temporal stacking and padding.
- 3.2 Dual-Stream Architecture: A ViT and Transformer language model retain patch-level and token-level embeddings instead of collapsing either modality into a single global vector.The motion encoder retains the full patch sequence, while the text encoder uses content-token embeddings excluding [CLS] and [SEP].
- 3.1 Joint-angle-based Motion Representation: Joint angles provide translation-invariant, anatomically interpretable features that decouple local joint movements from global body trajectories.The representation uses 29 kinematic dimensions across 14 joints, computed through inverse kinematics and body-centric coordinate systems.
- 3.1 Joint-angle-based Motion Representation: The joint-angle Motion Image shows temporally localized, joint-specific activations, whereas the position-based image shows uniform drift that obscures kinematic differences.For the motion “a person walks slowly forward,” right-hip angles show periodic gait patterns while corresponding joint positions are dominated by trajectory drift.
- 3.3 Fine-Grained Late Interaction (MaxSim): MaxSim matches each text token with its most similar motion patch, and text-only MLM enriches token embeddings with sentence-level context for alignment.MLM is applied only to the text encoder because text tokens query motion patches in the unidirectional MaxSim design.
4 Experiments
Experiments evaluate bidirectional retrieval on HumanML3D and KIT-ML, compare against state-of-the-art methods, and analyze scaling, interpretability, ablations, and efficiency. Results show strong retrieval performance and localized token-to-patch alignment, with storage overhead mitigated by compression.
- Datasets and evaluation: HumanML3D contains 14,616 motion clips and 44,970 descriptions, while KIT-ML contains 3,911 motions and 6,278 descriptions.Experiments use standard train/validation/test splits at 20 FPS for HumanML3D and 12.5 FPS for KIT-ML.
- Datasets and evaluation: Evaluation ranks motions from text queries and texts from motions using R@5, R@10, and MedR, with higher recall and lower median rank preferred.R@1 and R@2 are omitted because near-duplicate interference causes high cross-seed variance.
- State-of-the-art comparison: 59.28% KIT-ML T2M R@10 and 7.00 MedR make MotionColBert the best reported base model, exceeding SGAR-L by 4.58 R@10 points.The base model matches or exceeds methods without LLM-based text augmentation on both benchmarks and remains competitive with augmented methods.
- Scaling to larger backbones: +4.28% HumanML3D T2M R@10 and MedR reduction from 14 to 11 show that MotionColBert-L benefits from larger backbones.MoPatch-L and SGAR-L show only marginal or negative gains over their base variants; the paper attributes the scaling advantage to semantically coherent patches and token-level interaction.
- Interpretability: MaxSim maps text tokens to relevant joints and temporal phases more sharply than position-based attention, which is diffuse.Joint-angle Motion Images assign each horizontal band to one joint, while position-based bands drift together with global trajectory and blur per-joint detail.
- Ablation study: Joint-angle representations improve HumanML3D T2M R@5 by +1.85% with MaxSim versus +0.75% with global matching.The ablation attributes the stronger gain to decoupling local joint movement from global trajectory and pairing structured features with fine-grained matching.
- Ablation study: Angle+MaxSim+MLM reaches 48.08 T2M R@10, while MLM raises Angle+MaxSim from 45.64 to 48.08 but lowers Angle+Global from 44.72 to 41.08.MLM improves every metric with MaxSim but is detrimental under global matching, consistent with token-level context being exploitable only by token-level matching.
- Efficiency analysis: MaxSim increases storage from approximately 4 MB to 837 MB, but query latency rises only from 3.14 ms to 4.10 ms on HumanML3D.Product Quantization provides 16× compression with at most 0.41% R@10 loss, while 32× binary hashing retains 43.10% T2M R@10.
5 Conclusion
The framework combines joint-angle motion representations, MaxSim late interaction, and MLM regularization for fine-grained text-motion retrieval. It achieves state-of-the-art retrieval performance and interpretable token-to-patch correspondences, while incurring storage and computational overhead from dense patch embeddings.
- The framework combines anatomically defined joint-angle representations with MaxSim late interaction and MLM regularization to support fine-grained retrieval.
- The method achieves state-of-the-art retrieval performance and interpretable token-to-patch correspondence maps.
- Dense patch embeddings improve accuracy and interpretability but increase storage and computational overhead for offline galleries.
- Future work will investigate approximate-nearest-neighbor search and vector quantization for scaling to massive motion repositories.