PaperlayerView exact revision on arXiv

Source-linked AI summary

A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models

Kaname Yokoyama, Norimichi Ukita

arXiv:2608.15984v1cs.CV

TL;DR

MoLMs commonly rely on 3D motion data, while real-world inputs are often monocular videos from which equivalent 3D motion is challenging to estimate. This paper adds a plug-and-play 2D interface to 3D-pretrained MoLMs; across multiple models, 2D inputs perform comparably to 3D inputs, and estimated 2D motions are more effective under the evaluated monocular pose-estimation setting.

  • Problem

    MoLMs commonly rely on rich 3D motion data, while real-world inputs are often monocular videos from which equivalent 3D motions remain challenging to estimate.

    Source: §2.1 · p. 3

  • Method

    A plug-and-play 2D motion encoder maps 2D motions into MoLMs’ latent space so pretrained models can accept them without modification or fine-tuning.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across multiple MoLMs, 2D inputs achieve performance comparable to 3D inputs, and estimated 2D motions are more effective than estimated 3D motions under the evaluated monocular pose-estimation setting.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    The results support 2D motion as a practical interface for MoLM deployment under the evaluated monocular pose-estimation setting.

    Source: §6 · p. 14

    A stated future direction is addressing over-generalization in generated captions.

    Source: §6 · p. 14

Abstract

from arXiv · show

Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.

1 Introduction

Because monocular 3D motion recovery is challenging, this paper introduces a plug-and-play interface for feeding 2D motions into 3D-pretrained MoLMs and evaluates its practical applicability. The analysis examines 2D features aligned to the 3D tokenizer’s latent space, rather than establishing that 3D-specific information is generally unnecessary.

1 Introduction

  • Monocular videos make accurate, efficient 3D motion estimation challenging because MoLM inputs include joint coordinates, rotations, and global positions; the resulting gap from training motions can degrade understanding.The paper motivates testing whether 2D poses can sufficiently approximate the representation required by MoLMs, while limiting that analysis to features aligned to the 3D latent space.Source: §1 · pp. 1–2
  • The 2D Motion Interface projects 2D motions into a pretrained MoLM’s VQ-VAE latent space without modifying or fine-tuning the base model, and requires low training cost.The shared latent space also enables direct comparison of latent features and discrete tokens derived from 2D and 3D motions.Source: §1 · pp. 2–3
  • Across multiple public datasets and MoLM baselines, 2D inputs achieve performance comparable to 3D inputs, with strong consistency in continuous latent and discrete token spaces.This supports the claim that 2D motions can approximate representations used by 3D-pretrained MoLMs for motion understanding.Source: §1 · p. 3
  • Under the evaluated monocular pose-estimation setting, 2D inputs achieve higher motion-understanding performance at lower computational cost than 3D motion estimation.The paper constructs a real-world video dataset pairing videos with estimated 2D and 3D motions and captions, and proposes a real-video adapter to address perspective-projection and pose-estimation-noise distribution shifts.Source: §1 · pp. 2–3

2 Related Work

Prior work established motion language modeling and explored 2D–3D alignment, but MoLMs remain mismatched with monocular real-world observations. This work addresses that gap by aligning 2D motions to a frozen 3D-pretrained MoLM without fine-tuning it.

2.1 Motion Language Models

  • TM2T introduced discrete motion token sequences for autoregressive generation and captioning, followed by pretrained-language-model approaches MotionGPT and MG-MotionLLM.These models bring motions and text into a unified language-modeling paradigm.Source: §2.1 · p. 3
  • Extensions include controllable masked-token generation in MoMask, multimodal modeling in M3 GPT and RapVerse, and social-interaction modeling in SocialGen and later studies.M3 GPT tokenizes additional modalities such as music or audio; RapVerse jointly generates vocals and whole-body motion.Source: §2.1 · p. 3
  • Most MoLM studies rely on 3D datasets such as HumanML3D, whereas real-world monocular video observations make estimating equivalent 3D motions challenging.This training-data and observation gap is a major obstacle to practical MoLM deployment for motion understanding.Source: §2.1 · p. 3

2.2 2D/3D Motion Representation and Cross-Modal Alignment

  • Prior 2D–3D work reconstructs 3D motion, learns it from monocular supervision, shares representations, or distills 3D features for 2D recognition.Examples include Free3D’s 3D-free regularizations, a shared 2D/3D codebook, and V-VIPE’s view-invariant embedding.Source: §2.2 · p. 4
  • This work aligns 2D motions to the latent space of a frozen 3D-trained MoLM, enabling motion-language understanding from 2D inputs without fine-tuning the base model.The approach adapts MoLMs for real-world motion understanding by aligning inputs to the existing latent space.Source: §2.2 · p. 4

3 Proposed Method

The method adds a 2D motion encoder that lets pretrained MoLMs process 2D inputs without modifying or fine-tuning the original models. It aligns 2D features with a MoLM’s 3D latent space, then uses the existing codebook and language model for inference.

3.1 Preliminaries: MoLMs

  • MoLMs represent motion as discrete VQ-VAE tokens processed by a language model, and this work focuses on motion-to-text tasks.The VQ-VAE encodes motion into latent features and quantizes them using its codebook; the language model processes the resulting sequence.Source: §3.1 · pp. 4–5

3 Proposed Method

  • The pretrained MoLM stays frozen while only the plug-and-play 2D motion encoder is trained; at inference, its features become motion tokens for existing understanding tasks.The encoder aligns 2D features with the continuous latent space of the MoLM’s VQ-VAE, without architectural modifications or fine-tuning.Source: §3 · p. 4, Figure 2 · p. 5

3.2 2D motion with View Randomization

  • The 68D 2D representation uses 13 COCO-format joints, formed by randomly yaw- and pitch-rotating a reconstructed 3D skeleton, projecting away depth, and deriving per-frame features.Orthographic projection avoids camera-parameter assumptions; the real-video adapter addresses the gap to perspective observations.Source: §3.2 · p. 6

3.3 2D motion encoder

  • The 2D motion encoder maps 2D sequences into latent features and is trained with an L1 feature-alignment loss against features from the MoLM’s VQ-VAE encoder.The alignment targets the continuous latent space used by the MoLM.Source: §3.3 · p. 6

3.4 Inference

  • At inference, the encoder’s 2D latent features are quantized with the VQ-VAE codebook and fed to the existing language model, optionally with a task prompt, to generate text.The resulting discrete sequence is denoted C2D.Source: Figure 2 · p. 5, §3.4 · p. 6

4 Experiments

Experiments compare the 2D motion encoder with 3D inputs and 2D-scratch models, then analyze latent-space alignment and robustness to token replacement.

4.1 Experimental Setup

  • HumanML3D supports motion captioning, while FineMotion provides fixed-interval snippets for detailed-text generation; the encoder uses 512-dimensional latents and is trained for 3,000 epochs.HumanML3D contains 14,616 sequences with 44,970 captions; FineMotion contains 420,968 snippets. Training uses AdamW at 1×10−4 with batch size 64 on one NVIDIA A100 40GB GPU.Source: §4.1 · p. 7

4.2 Results of Motion understanding Tasks

  • Across motion captioning and motion-to-detailed-text, the proposed encoder performs comparably to 3D inputs on many metrics and substantially reduces the gap versus 2D-scratch training.The detailed-text comparison uses MG-MotionLLM and reports language-generation metrics at both sequence and snippet levels.Source: Table 1 · p. 8, Table 2 · p. 8, §4.2 · p. 8

4.3 Analysis of 2D motion encoder

  • For MotionGPT, 2D and 3D latent-feature distributions largely overlap, with token agreement of 44.8% Top-1, 59.6% Top-2, and 67.1% Top-3.With codebook size K = 512, random-selection agreement is 1/K ≃0.20% Top-1 and 3/K ≃0.59% Top-3.Source: §4.3 · p. 9, Figure 3 · p. 9
  • Replacing 50% of tokens with k-nearest neighbors causes only −1.4% Avg Drop vs 3D at k = 9, whereas random replacement causes −34.2%.The findings support the explanation that neighboring codebook tokens have similar semantics, limiting degradation despite 44.8% Top-1 agreement between 2D- and 3D-derived tokens.Source: Table 3 · p. 9, §4.3 · p. 10

5 Evaluation on Monocular Real Videos

This section evaluates 2D and 3D motion inputs on a monocular real-world video dataset and introduces an adapter for real-video pose-estimation noise. With the adapter, 2D inputs outperform 3D input settings on almost all metrics across models.

5.1 Monocular real-world video motion dataset

  • The dataset contains 132 monocular videos from 10 participants, totaling 14,878 frames; 86.4% of video-caption pairs were judged semantically consistent.Participants imitated HumanML3D test-set clips recorded with an iPhone SE 2nd generation; camera viewpoints varied, and 3D and 2D poses were estimated from the videos.Source: §5.1 · p. 11

5.2 Noise reduction for real-world videos

  • The real-video adapter addresses perspective-projection and pose-estimation noise by transforming estimated 2D poses and joint confidences before the 2D motion encoder.It is trained on AMASS-derived pseudo-real pairs: randomized-view renders are processed by ViTPose, while the 2D encoder stays frozen and the adapter learns L1 feature alignment to 3D features.Source: §5.2 · pp. 11–12, Figure 4 · p. 11

5.3 Evaluation results

  • 2D Input w/ Areal outperforms all 3D Input settings on almost all metrics across models, and consistently beats 2D Input w/o Areal.WHAM requires 250.2 GFLOPs per frame, while ViTPose (Base) requires 17.2 GFLOPs; the comparison excludes human detection.Source: Table 4 · p. 13, §5.3 · p. 13
  • Adapter correction can lose fine-grained motion details, including an S-shaped walking trajectory, and imitation discrepancies may also cause semantic mismatches.MotionGPT and MG-MotionLLM show larger Avg Drop vs Ref 3D than TM2T; the passage suggests smoothing by the adapter may suppress detailed differences.Source: §5.3 · pp. 13–14

6 Conclusion

The paper introduces a plug-and-play interface that lets existing VQ-VAE-based MoLMs accept 2D motions, and reports comparable performance to 3D inputs alongside stronger results for estimated 2D motions in the evaluated monocular setting.

6 Conclusion

  • The 2D Motion Interface maps 2D motions into existing VQ-VAE-based MoLMs without modifying them, with performance comparable to 3D inputs and estimated 2D motions outperforming estimated 3D motions in the evaluated setting.The authors also constructed a monocular real-world video motion dataset and introduced a real-video adapter.Source: §6 · p. 14
  • Future work should address generated captions’ over-generalization and train MoLM language models on large-scale captioned human-motion video to expand motion vocabulary and enable finer-grained understanding.The first direction concerns practical deployment; the second concerns extending model capabilities.Source: §6 · p. 14

Supplementary Material

The supplementary material adds implementation and training details, qualitative results, encoder analyses, and documentation and evaluation of the real-video adapter and dataset.

Section A

Section B

Section C

Section D

Section E

  • Section E details real-video adapter training and implementation and provides an architecture ablation study and qualitative results.Source: Supplementary Material · p. 18

A Additional Implementation Details. … B Qualitative Results of motion-understanding tasks

The implementation uses a VQ-VAE-style 2D encoder, normalizes 2D motion to reduce dependence on subject scale and image position, and specifies baseline and inference procedures.

A.1 2D motion encoder E2D

  • The 2D encoder follows MotionGPT’s VQ-VAE encoder architecture, has 9.6M parameters, and trains for about 22 hours with randomized yaw and pitch views.Training uses 3,000 epochs, yaw angles across 360 degrees, and pitch angles from 0 to 60 degrees.Source: §A.1 · p. 18

A.2 2D Motion Normalization

  • The 68-dimensional feature is then z-normalized per dimension using HumanML3D training-split statistics, reused unchanged at evaluation to avoid test-set leakage.The normalization uses training-split mean and standard deviation.Source: §A.2 · pp. 18–19
  • Per-clip normalization removes root translation and scale, using a 99th-percentile scale robust to outlier joints, and makes the representation invariant to apparent size, camera distance, and image position.The scale is computed across all frames and 13 joints; orientation is scale-invariant, and root y position is offset by its first-frame value.Source: §A.2 · p. 18

A.3 Baseline Models

  • For 3D-input baselines, TM2T’s VQ-VAE was modified to match the other models and retrained, while MotionGPT and MG-MotionLLM used released checkpoints.This controls for model-specific VQ-VAE differences when comparing input modalities.Source: §A.3 · p. 19
  • For 2D-from-scratch baselines, MG-MotionLLM and TM2T were retrained, but MotionGPT was omitted because its training performance could not be stably reproduced.The MotionGPT reproducibility issue was also reported in the official repository issues and remained unresolved at the time of writing.Source: §A.3 · p. 19
  • For the proposed method, MotionGPT and MG-MotionLLM use released checkpoints, while TM2T uses the checkpoint retrained with the modified VQ-VAE.Source: §A.3 · p. 19

A.4 Inference Details

  • For motion-understanding inference, MG-MotionLLM and MotionGPT pair motion tokens with “Generate text:” for captioning or “Generate the motion script:” for motion-to-detailed-text.Source: §A.4 · p. 19

B.1 Results of motion captioning · B.2 Results of motion-to-detailed text · C Additional analysis of 2D motion encoder

Supplementary qualitative examples show that the proposed method’s captions remain semantically aligned with ground truth for both coarse captioning and snippet-level motion descriptions, complementing the paper’s benchmark results.

B.1 Results of motion captioning

  • The section provides qualitative examples alongside quantitative results reported in the main paper.The examples cover motion captioning and motion-to-detailed text.Source: §B.1 · p. 19
  • For simple HumanML3D motions, generated captions are semantically consistent with the ground-truth captions.The examples use MotionGPT as the baseline model.Source: §B.1 · pp. 19–20, Figure S1 · p. 20
  • For a more complex motion composed of two primitive motions, the generated caption remains semantically aligned with ground truth.This example also supports the conclusion that 2D input largely preserves motion-captioning performance.Source: §B.1 · p. 20, Figure S1 · p. 20
  • Across the captioning examples, the results support largely preserving motion-captioning performance when 2D motions are used as input.Source: §B.1 · p. 20

B.2 Results of motion-to-detailed text

  • On FineMotion, MG-MotionLLM-generated detailed captions in both examples are semantically consistent with their ground-truth captions.The examples are from the FineMotion test set.Source: §B.2 · p. 20, Figure S2 · p. 21
  • The motion-to-detailed text examples indicate that the method preserves fine-grained snippet-level information as well as coarse motion information.The outputs describe motions in successive time snippets.Source: §B.2 · pp. 20–21

C.1 3D reconstruction ability · C.2 Viewpoint Sensitivity via Multi-view Token Consistency

The 2D-derived tokens largely preserve pose structure and distribution-level motion quality despite greater absolute-position error, and token sequences and captions remain consistent across viewpoints.

C.1 3D reconstruction ability

  • Reconstruction quality is evaluated against ground-truth 3D motion using MPJPE and PAMPJPE, alongside FID, Diversity, and ACCEL.The test compares VQ-VAE reconstructions from discrete tokens derived from 2D and 3D motions.Source: §C.1 · pp. 21–22
  • 2D-input reconstructions have substantially larger MPJPE than 3D-input reconstructions, but only a small PAMPJPE gap and comparable FID, Diversity, and ACCEL.The comparison uses MotionGPT’s VQ-VAE and corresponding 2D motion encoder.Source: §C.1 · p. 22, Table S1 · p. 22
  • The small PAMPJPE gap indicates that 2D-derived tokens largely preserve pose structure, while their greater MPJPE reflects lost global position information.The passage attributes minimal semantic degradation to preserved pose structure rather than absolute global position.Source: §C.1 · pp. 21–22

C.2 Viewpoint Sensitivity via Multi-view Token Consistency

  • The viewpoint test encodes 2D motions of the same 3D motion from randomly sampled views and scores the share of temporal positions with identical tokens across all views.The score is averaged over test motions; random token assignment has expected consistency (1/K)^(N−1).Source: §C.2 · pp. 22–23
  • Agreement across viewpoints was 63.6% for N = 2, 50.9% for N = 3, and 39.7% for N = 5, far above the stated random-token expectation.For N = 2, the random-codebook expectation is 1/K ≃ 0.20%; the passage says it is squared for N = 3 and squared again for N = 5.Source: §C.2 · p. 23
  • Predicted captions also remain consistent when the same 3D motion is viewed from two viewpoints.Figure S3 gives qualitative examples of this viewpoint robustness.Source: Figure S3 · p. 22, §C.2 · p. 23

D Details of the Monocular Real-World Video Motion Dataset … E.3 Real-video adapter for estimated 3D motions

These sections describe how the real-world video dataset was assembled and verified, and provide training and implementation details for adapters using estimated 2D or 3D motion. They also explain why Motion-X was excluded from evaluation.

D Details of the Monocular Real-World Video Motion Dataset

  • The dataset construction randomly extracts motion clips from the HumanML3D test set.Source: D Details of the Monocular Real-World… · p. 23
  • Motion-X was excluded because its captions differ substantially in style from HumanML3D captions, creating a domain gap that hinders fair evaluation.Motion-X includes samples pairing videos with estimated 2D and 3D motions.Source: D Details of the Monocular Real-World… · p. 23
  • 86.4% of collected video-caption pairs were judged semantically consistent by three annotators, supporting use of the original HumanML3D captions for evaluation.The remaining inconsistencies may reflect imitation errors or mismatches between HumanML3D motions and captions.Source: D Details of the Monocular Real-World… · p. 23

E Additional details of real-video adapter

E.1 Datasets for Training

  • The adapter training split uses 3,262 HumanML3D training motions and 813 validation motions, rendering 10 viewpoints per motion.The best validation checkpoint is evaluated on the monocular real-world video motion dataset; motions are sampled from AMASS according to HumanML3D splits.Source: §E.1 · p. 24

E.2 Implementation Details

  • Areal is a residual MLP with LayerNorm, a two-layer 512-hidden-unit MLP, GELU, and dropout p=0.1, fed concatenated estimated 2D motion features and per-joint confidence scores.Source: §E.2 · p. 24
  • Areal has 0.3M parameters and is trained for 150k iterations with AdamW at learning rate 1×10−3, batch size 64, and weight decay 1 × 10−4.Training uses β = (0.9, 0.99), no learning-rate scheduler, and one NVIDIA A100 40GB GPU; it takes 1 hour.Source: §E.2 · p. 24

E.3 Real-video adapter for estimated 3D motions

  • A3D follows Areal’s training procedure and objective using pseudo-real training pairs generated with TRACE and WHAM instead of ViTPose.Unlike Areal, A3D takes 263-dimensional 3D motion features and does not use confidence inputs.Source: §E.3 · p. 24

E.4 Qualitative Evaluation · E.5 Ablation Study on Adapter Architecture

On real-world monocular videos, the adapter improves MotionGPT’s caption alignment with ground truth; across tested architectures, performance is comparable, with the MLP selected as default. The evaluation also attributes improved reliability to mitigating pose-estimation domain shift.

E.4 Qualitative Evaluation

  • With MotionGPT, captions generated using Areal are semantically closer to ground truth than captions generated without the adapter.The qualitative comparison uses monocular real-world video examples.Source: §E.4 · pp. 24–25, Figure S4 · p. 25
  • Without Areal, MotionGPT sometimes misrecognizes even relatively simple motions or scenes with little occlusion, producing captions that substantially deviate from ground truth.Source: §E.4 · pp. 24–25
  • The qualitative results indicate that Areal mitigates pose-estimation-noise domain shift and improves the reliability of motion tokens fed to MotionGPT.Source: §E.4 · p. 25

E.5 Ablation Study on Adapter Architecture

  • The architecture comparison evaluates Linear, Temporal Conv, and MLP adapters with MotionGPT under the same settings; all include residual connections.The MLP-based adapter is the one used in the main paper.Source: §E.5 · p. 25
  • The adapter-architecture comparison investigates whether Areal’s performance depends on its design, using MotionGPT as the baseline model.Source: §E.5 · p. 25
  • All tested adapter architectures perform comparably; the MLP leads on BLEU-1 and BERTScore while remaining competitive on ROUGE-L and CIDEr.The MLP is therefore used as the default architecture.Source: §E.5 · p. 25, Table S2 · p. 25

Related papers

Loading 2608.15984v1…