PaperlayerView exact revision on arXiv

Source-linked AI summary

A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models

Kaname Yokoyama, Norimichi Ukita

arXiv:2608.15984v1cs.CV

TL;DR

Monocular videos make the accurate 3D motions used by Motion Language Models difficult to obtain for real-world understanding. This paper adds a plug-and-play 2D interface to 3D-pretrained models, achieving comparable performance to 3D inputs and better performance than estimated 3D motions in the evaluated monocular setting.

  • Problem

    MoLMs rely on information-rich 3D motion, whereas monocular videos often provide observations from which equivalent 3D motions remain challenging to estimate.

  • Method

    A plug-and-play 2D motion encoder maps 2D motions into existing MoLMs’ latent space without modifying or fine-tuning the pretrained models.

  • Results

    Across multiple MoLMs, 2D inputs perform comparably to 3D inputs, while estimated 2D motions outperform estimated 3D motions in the evaluated monocular setting.

  • Takeaways & Limitations

    2D motion provides a practical interface for deploying 3D-pretrained MoLMs in real-world motion understanding from monocular videos.

  • Takeaways & Limitations

    The authors identify over-generalization of generated captions as an important practical-deployment issue for future work.

Abstract

from arXiv · show

Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.

1 Introduction

MoLMs rely on 3D motion inputs, but accurately and efficiently estimating the required 3D information from monocular videos is challenging. The paper introduces a plug-and-play 2D Motion Interface that lets 3D-pretrained MoLMs accept 2D motions without modifying or fine-tuning them.

  • Motivation: MoLMs commonly use 3D joint coordinates, rotations, and global positions, making monocular-video motion estimation difficult and potentially degrading understanding performance.The 3D motions are typically reconstructed from motion-capture systems or multi-view observations, whereas real-world inputs commonly come from monocular cameras.
  • Motivation: The paper investigates whether 2D motions derived from 2D poses can sufficiently approximate the motion representation required by 3D-pretrained MoLMs.The analysis specifically considers 2D features aligned to the 3D latent space and does not establish that 3D-specific information is generally unnecessary.
  • Method: The proposed plug-and-play 2D Motion Interface is a 2D motion encoder that enables existing MoLMs to directly accept 2D motion inputs.It requires neither modification nor fine-tuning of pretrained MoLMs and has a very low training cost.
  • Method: Because the 2D encoder shares the discrete tokenizer’s latent space, latent motion features from 2D and 3D inputs can be directly compared.This unified representation space supports experimental evaluation of whether MoLM motion representations can be approximated from 2D motions.
  • Evaluation: Experiments on multiple public datasets and MoLM baselines show that 2D motion inputs achieve performance comparable to 3D motion inputs.This result is presented as evidence for the practicality of using 2D inputs with 3D-pretrained MoLMs.

2 Related Work

Prior Motion Language Models unify motion and language through discrete motion tokens, but their reliance on 3D datasets limits deployment from monocular video. This work addresses the gap by aligning 2D motions with a frozen 3D-trained MoLM latent space without fine-tuning.

  • Motion Language Models: TM2T introduced languagelike discrete token sequences for 3D human motions, enabling autoregressive motion generation and captioning.MotionGPT and MG-MotionLLM extended this paradigm with pretrained language models and instruction tuning.
  • Motion Language Models: MoLMs have expanded to masked motion generation and unified language, audio, music, and motion modeling.MoMask improves controllability and generation quality, while M3 GPT and RapVerse combine multiple modalities and tasks.
  • 2D–3D Motion Gap: Most MoLMs rely on 3D datasets containing global positions and joint rotations, whereas monocular videos provide observations from which equivalent 3D motions remain difficult to estimate.This mismatch between training data and real-world observations obstructs practical MoLM deployment.
  • 2D–3D Motion Gap: Prior work bridges 2D observations and 3D motion through monocular or multi-view reconstruction, 3D-free learning, and representations shared across 2D and 3D.Free3D learns 3D motion from monocular 2D supervision using view consistency and physical plausibility regularizations.
  • 2D Motion Adaptation: The proposed approach aligns 2D motions with the latent space of frozen MoLMs, enabling motion-language understanding from 2D inputs without base-model fine-tuning.This directly explores adapting MoLMs trained on 3D motions to 2D inputs for real-world motion understanding.

3 Proposed Method

The proposed plug-and-play 2D Motion Interface lets existing 3D-pretrained MoLMs understand 2D motion by training a lightweight encoder to match the MoLM’s latent motion space. It converts monocular-observable motion into the pretrained model’s discrete tokens without modifying the MoLM.

  • 3 Proposed Method: The interface adds a 2D motion encoder that enables existing MoLMs to perform motion-understanding tasks directly from 2D motion inputs.The target MoLMs tokenize motion with a VQ-VAE before language-model processing.
  • 3 Proposed Method: During training, the pretrained MoLM remains frozen while only the 2D encoder is optimized with an L1 loss aligning 2D and 3D VQ-VAE latent features.The alignment matches features in the VQ-VAE’s continuous latent space.
  • 3 Proposed Method: During inference, 2D latent features are quantized with the pretrained VQ-VAE codebook into discrete tokens and passed to the language model with or without a task prompt.The resulting language-model output supports the motion-to-text setting used in the paper.
  • 3 Proposed Method: 2D motion is constructed by removing depth and joint rotations from 3D motion while retaining monocular-observable information.The pipeline reconstructs a 22-joint skeleton, randomizes yaw and pitch for viewpoint robustness, and projects it onto a 2D plane.

4 Experiments

The experiments evaluate the 2D motion encoder across motion-language datasets and compare it with 3D inputs and MoLMs trained from scratch on 2D motions. Analyses show that 2D inputs preserve performance through aligned latent representations and semantically related neighboring tokens.

  • Experimental comparisons: The method is compared with 3D motion inputs and MoLMs trained from scratch on 2D motions across motion understanding tasks.The evaluation covers multiple models and datasets, with motion captioning and motion-to-detailed-text comparisons.
  • Datasets: 14,616 motion sequences paired with 44,970 sequence-level text captions comprise HumanML3D, used for motion captioning; FineMotion provides finer-grained re-annotations.Experiments use HumanML3D and FineMotion for motion-language evaluation.
  • Experimental comparisons: The proposed encoder achieves performance comparable to 3D inputs across most text-motion alignment and language generation metrics while using monocular 2D motion.The reported alignment metrics include R-Precision and MM-Dist.
  • Analysis: The analysis compares latent features and discrete token sequences produced from corresponding 3D and 2D motions to explain preserved motion understanding performance.Both modalities are encoded, quantized with the 3D VQ-VAE codebook, and analyzed in a PCA space.
  • Analysis: 44.8% Top-1 agreement is observed between 2D- and 3D-derived token sequences, while random replacement causes −34.2% Avg Drop vs 3D and k = 9 causes only −1.4%.Small degradation under neighboring-token replacement supports similar semantics for neighboring VQ-VAE codebook tokens.

5 Evaluation on Monocular Real Videos

Evaluation on monocular real videos addresses noisy pose-estimated inputs by introducing a real-world motion dataset and a real-video adapter. Across motion-captioning experiments, adapted 2D inputs outperform 3D input settings on almost all metrics while requiring less computation than the compared 3D pipeline.

  • Real-world video dataset: The dataset contains 132 monocular RGB videos recorded by 10 participants, totaling 14,878 frames and 743.9 seconds.Participants imitated randomly selected HumanML3D test-set motion clips, with recordings spanning participants in their 20s to 50s.
  • Real-video adapter: The real-video adapter bridges orthographic training motions with perspective, pose-estimated observations by processing estimated 2D motions together with joint confidence scores.It is inserted before the 2D motion encoder and trained with pseudo-real pairs synthesized from AMASS motions rendered from random viewpoints and processed by ViTPose.
  • Comparison on estimated 2D/3D inputs: 2D Input w/ Areal outperforms all 3D Input settings on almost all metrics across all evaluated models.The comparison includes estimated 3D motions, estimated 2D motions, and adapted estimated 2D motions, with an A3D adapter also evaluated for estimated 3D inputs.
  • Comparison on estimated 2D/3D inputs: WHAM is substantially more computationally expensive than the combination of ViTPose and the real-video adapter used for 2D Input.The paper makes this comparison in the context of the monocular real-world video evaluation.
  • Limitations: Fine-grained root-trajectory information can be lost, and imitation errors may cause semantic mismatches between reference and performed motions.Qualitative reconstruction shows that an S-shaped root path may no longer be preserved.

6 Conclusion

The paper presents a plug-and-play 2D Motion Interface that lets existing VQ-VAE-based MoLMs process 2D motions without model modification, with performance comparable to 3D inputs. It also identifies over-generalized captions and limited motion vocabulary as directions for future improvement.

  • Conclusion: The interface maps 2D motions into existing VQ-VAE-based MoLMs’ continuous latent space without modifying the original models.This enables the models to accept 2D motion inputs directly.
  • Conclusion: Experiments across multiple MoLMs showed performance comparable to 3D motion inputs.
  • Conclusion: The authors constructed a monocular real-world video motion dataset and introduced a real-video adapter.Under the evaluated setting, estimated 2D motions were more effective than estimated 3D motions.
  • Future Work: Future work should address over-generalization in generated captions for practical deployment.This issue is discussed in Section 5.3.
  • Future Work: Further training MoLM language models on large-scale captioned human-motion video data could expand motion vocabulary and enable more fine-grained motion understanding.

Supplementary Material

The supplementary material adds training, implementation, qualitative, and analytical results omitted from the main paper because of space limitations.

  • Overview: It provides training and implementation details, along with additional experimental results excluded from the main paper due to space limitations.The material covers the 2D motion encoder and baseline-model implementations, qualitative motion results, and further encoder analyses.
  • Implementation Details: Section A presents additional implementation details for the 2D motion encoder and each baseline model.
  • Qualitative Results: Section B presents qualitative results for motion captioning and motion-to-detailed text.

A Additional Implementation Details. … B.1 Results of motion captioning

The appendix details the 2D encoder, normalization pipeline, baseline configurations, inference prompts, and qualitative captioning results. The proposed interface produces captions that remain semantically aligned with ground-truth descriptions on both simple and compound motions.

  • A.1 2D motion encoder E2D: The 2D motion encoder follows MotionGPT’s VQ-VAE encoder architecture and contains 9.6M parameters.Training lasts around 22 hours over 3000 epochs, with yaw sampled across 360° and pitch from 0° to 60°.
  • A.2 2D Motion Normalization: The 68-dimensional 2D feature X2D is normalized in two stages: per-clip scale normalization followed by dataset-level z-normalization.The dataset statistics are computed once on the HumanML3D training split and reused at evaluation.
  • A.2 2D Motion Normalization: Per-clip normalization removes translation and scale using a root-relative representation and a 99th-percentile scale across frames and joints.The percentile scale improves robustness to noisy real-video keypoints, while ϵ = 10^-8 guards against degenerate clips.
  • A.3 Baseline Models: For 3D-input comparisons, TM2T is modified to use the shared VQ-VAE architecture and retrained, while MotionGPT and MG-MotionLLM use official checkpoints.This setup isolates changes in input modality from model-specific VQ-VAE designs.
  • A.3 Baseline Models: For 2D-scratch comparisons, MG-MotionLLM and TM2T are retrained with 2D inputs, whereas MotionGPT’s result is omitted because training could not be stably reproduced.TM2T again uses the modified VQ-VAE architecture.
  • A.3 Baseline Models: The proposed method uses official MotionGPT and MG-MotionLLM checkpoints and the retrained modified-VQ-VAE checkpoint for TM2T.These checkpoint choices match the corresponding 3D-input settings.
  • A.4 Inference Details: For motion understanding, MG-MotionLLM and MotionGPT receive task prompts, using “Generate text:” for captioning and “Generate the motion script:” for motion-to-detailed text.The prompts accompany the discrete motion-token sequence fed into the language model.
  • B.1 Results of motion captioning: Qualitative HumanML3D examples show semantically consistent captions for simple motions and semantically aligned captions for a complex motion combining two primitives.The examples use MotionGPT as the baseline and complement the quantitative results reported in the main paper.

B.2 Results of motion-to-detailed text · C Additional analysis of 2D motion encoder

On FineMotion, the proposed method produces detailed motion captions that are semantically consistent with ground-truth captions. Qualitative examples show preservation of both coarse and fine-grained motion information.

  • B.2 Results of motion-to-detailed text: On FineMotion, captions generated with MG-MotionLLM as the baseline are semantically consistent with the ground-truth captions.The passage states that this qualitative consistency matches the quantitative results reported in the main paper.
  • B.2 Results of motion-to-detailed text: The qualitative results demonstrate that the method preserves coarse motion information as well as fine-grained motion details.This conclusion is explicitly drawn from the motion-to-detailed text examples.
  • C Additional analysis of 2D motion encoder: The provided material for the merged block contains no passage describing additional analysis of the 2D motion encoder.Accordingly, no encoder-analysis finding can be grounded in the supplied passages.
  • B.2 Results of motion-to-detailed text: The captions describe time-segmented movements of the right arm, left hand, head, and legs across 0.0–2.0 seconds.Examples include raising, lowering, extending, bending, and repositioning body parts within successive 0.5-second intervals.
  • B.2 Results of motion-to-detailed text: One example records coordinated upper-body movements, including raising the right arm and repositioning the left hand toward the thigh.The sequence also describes head turning and a motionless final interval.
  • B.2 Results of motion-to-detailed text: Another example captures lower-body and upper-body changes, including leg repositioning, knee bending, arm extension, and elbow movement.The described sequence spans four consecutive 0.5-second intervals.

C.1 3D reconstruction ability · C.2 Viewpoint Sensitivity via Multi-view Token Consistency

The analysis shows that 2D-derived tokens preserve some motion semantics but lose absolute 3D geometry, while view randomization yields substantial multi-view token consistency. Using MotionGPT’s VQ-VAE and 2D motion encoder, agreement reaches 63.6% for two viewpoints, 50.9% for three, and 39.7% for five.

  • C.1 3D reconstruction ability: 2D-derived discrete tokens tend to select neighboring tokens of 3D-derived tokens, causing limited semantic degradation.The comparison reconstructs 3D motions by decoding tokens derived from 3D and 2D inputs.
  • C.1 3D reconstruction ability: The reconstruction experiments use MotionGPT’s VQ-VAE, its corresponding 2D motion encoder, and the metrics FID, Diversity (Div), and ACCEL.These metrics are reported for evaluating 3D reconstruction quality.
  • C.1 3D reconstruction ability: MPJPE is substantially larger for 2D input than 3D input, indicating that 2D-derived tokens may lose absolute global 3D positions.MPJPE measures Euclidean distance between corresponding predicted and ground-truth joints in world coordinates.
  • C.1 3D reconstruction ability: The small PAMPJPE gap between 2D and 3D inputs indicates that 2D-derived tokens better preserve pose geometry after alignment.The supplied passage contrasts the substantially larger MPJPE gap with a small PAMPJPE gap.
  • C.2 Viewpoint Sensitivity via Multi-view Token Consistency: View randomization is introduced to encourage viewpoint-robust representations, evaluated by comparing tokens from multiple 2D projections of the same 3D motion.Different viewpoints are sampled from one 3D motion and encoded using the 2D motion encoder and VQ-VAE codebook.
  • C.2 Viewpoint Sensitivity via Multi-view Token Consistency: Multi-view consistency is measured as the averaged proportion of temporal positions whose tokens are identical across all sampled viewpoints.Under random token assignment from a codebook of size K, the expected consistency is (1/K)N−1.
  • C.2 Viewpoint Sensitivity via Multi-view Token Consistency: 63.6% agreement occurs for N = 2, 50.9% for N = 3, and 39.7% for N = 5 viewpoints.The experiments use MotionGPT’s VQ-VAE and corresponding 2D motion encoder; random assignment gives an expected agreement rate of 1/K ≃0.20% when N = 2.

D Details of the Monocular Real-World Video Motion Dataset … E.3 Real-video adapter for estimated 3D motions

The paper describes a monocular real-world video motion dataset validated against HumanML3D captions and details lightweight adapters for estimated 2D and 3D motions. The 2D adapter uses confidence-aware per-frame features, while the 3D adapter follows the same training procedure with 3D-specific inputs.

  • D Details of the Monocular Real-World Video Motion Dataset: The dataset additionally includes randomly extracted motion clips from the HumanML3D test set.This sampling procedure is specified for the dataset construction.
  • D Details of the Monocular Real-World Video Motion Dataset: Motion-X is excluded because its captions differ substantially from HumanML3D, creating a domain gap that prevents fair evaluation.Motion-X contains videos paired with estimated 2D and 3D motions, but its caption style may include emotional expressions.
  • D Details of the Monocular Real-World Video Motion Dataset: 86.4% of collected video-caption pairs were judged semantically consistent by three annotators, supporting the dataset’s evaluation validity.Annotators used binary yes/no judgments to assess whether each video matched its corresponding HumanML3D caption.
  • E Additional details of real-video adapter: The real-video adapter section supplies implementation details omitted from the main paper because of space limitations.The adapter is denoted Areal.
  • E.1 Datasets for Training: Training uses 3,262 HumanML3D training motions and 813 validation motions, rendering 10 viewpoints per motion and selecting the best validation checkpoint.AMASS motions are sampled according to the HumanML3D splits for training and validation.
  • E.2 Implementation Details: Areal is a residual MLP with LayerNorm, a two-layer 512-dimensional hidden MLP, GELU, and dropout p=0.1; estimated 2D motion and joint confidences are concatenated as inputs.The adapter operates on per-frame input features.
  • E.2 Implementation Details: Areal training uses AdamW for 150k iterations on one NVIDIA A100 40GB GPU, taking 1 hour and using 0.3M parameters.The optimizer settings include learning rate = 1×10−3, β = (0.9, 0.99), batch size = 64, weight decay = 1 × 10−4, and no scheduler.
  • E.3 Real-video adapter for estimated 3D motions: A3D is trained like Areal with TRACE- and WHAM-generated pseudo-real pairs, but takes 3D motion features with input dimensionality 263 and no confidence input.Both adapters use the same training objective, while A3D differs in its input representation.

E.4 Qualitative Evaluation · E.5 Ablation Study on Adapter Architecture

Qualitative results show that the real-video adapter makes MotionGPT’s captions more semantically aligned with ground truth and improves motion recognition under monocular pose-estimation noise. Adapter architectures perform comparably overall, with the MLP adapter selected as default for its strongest BLEU-1 and BERTScore results.

  • E.4 Qualitative Evaluation: With the adapter Areal, MotionGPT generates captions that are semantically closer to ground-truth captions across monocular real-world video examples.
  • E.4 Qualitative Evaluation: Without Areal, MotionGPT sometimes fails to recognize relatively simple motions or scenes with little occlusion.
  • E.4 Qualitative Evaluation: The adapter mitigates domain shift from pose-estimation noise and improves the reliability of motion tokens supplied to MotionGPT’s language model.
  • E.5 Ablation Study on Adapter Architecture: The architecture ablation compares Linear, Temporal Conv, and MLP adapters using MotionGPT under identical training and evaluation settings.
  • E.5 Ablation Study on Adapter Architecture: All tested adapter architectures achieve comparable performance, showing that real-video adapter effectiveness does not strongly depend on one architectural design.
  • E.5 Ablation Study on Adapter Architecture: The MLP adapter achieves the best BLEU-1 and BERTScore while remaining competitive on ROUGE-L and CIDEr.
  • E.5 Ablation Study on Adapter Architecture: The MLP adapter is therefore used as the default architecture.
Loading 2608.15984v1…