PaperlayerView exact revision on arXiv

Source-linked AI summary

AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism

Chongyang Zhong, Lei Hu, Zihao Zhang, Shihong Xia

arXiv:2309.00796v1cs.CV

TL;DR

Text-driven human motion generation must produce diverse, natural motions that match descriptions despite motion’s spatio-temporal complexity and difficult text-motion correspondences. AttT2M combines body-part attention for motion representation learning with global-local attention for cross-modal learning, then generates motion with a transformer. It outperforms prior work in qualitative and quantitative evaluations and supports fine-grained and action-to-motion generation.

  • Problem

    Text-driven motion generation must model motion’s complex spatial and temporal structure while learning local word-to-subsequence and global description-to-sequence correspondences.

    Source: §1 · pp. 1–2

  • Method

    AttT2M uses a body-part-attention spatio-temporal VQ-VAE for discrete motion representations and global-local attention to learn sentence-level and word-level text-motion relationships before transformer generation.

    Source: §1 · p. 2

  • Results

    AttT2M outperforms state-of-the-art methods in qualitative and quantitative evaluations on HumanML3D and KIT-ML, with generated motions matching descriptions while retaining diversity, realism, and naturalness.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The method also supports fine-grained motion generation and action-to-motion generation.

    Source: §1 · p. 2

    For very long, detailed test descriptions, generated motions have insufficient diversity, a limitation associated with few corresponding sequences in HumanML3D.

    Source: §5 · p. 9

Abstract

from arXiv · show

Generating 3D human motion based on textual descriptions has been a research focus in recent years. It requires the generated motion to be diverse, natural, and conform to the textual description. Due to the complex spatio-temporal nature of human motion and the difficulty in learning the cross-modal relationship between text and motion, text-driven motion generation is still a challenging problem. To address these issues, we propose \textbf{AttT2M}, a two-stage method with multi-perspective attention mechanism: \textbf{body-part attention} and \textbf{global-local motion-text attention}. The former focuses on the motion embedding perspective, which means introducing a body-part spatio-temporal encoder into VQ-VAE to learn a more expressive discrete latent space. The latter is from the cross-modal perspective, which is used to learn the sentence-level and word-level motion-text cross-modal relationship. The text-driven motion is finally generated with a generative transformer. Extensive experiments conducted on HumanML3D and KIT-ML demonstrate that our method outperforms the current state-of-the-art works in terms of qualitative and quantitative evaluation, and achieve fine-grained synthesis and action2motion. Our code is in https://github.com/ZcyMonkey/AttT2M

1. Introduction

Text-driven motion generation must handle both motion’s spatio-temporal structure and complex local and global text-motion correspondence. AttT2M addresses these challenges with a two-stage attention-based method and reports stronger qualitative and quantitative results than prior work, including fine-grained and action-to-motion generation.

1. Introduction

  • Text-driven motion synthesis is difficult because motion has irregular spatial structure and dynamic temporal characteristics, while text-motion correspondence is both local and global.Local words correspond to motion subsequences; the overall description corresponds to their order and connections.Source: §1 · pp. 1–2
  • AttT2M outperforms previous work in qualitative and quantitative evaluations on KIT-ML and HumanML3D, with realistic, natural, diverse motions that match descriptions.The method also supports fine-grained generation and action-to-motion generation.Source: §1 · p. 2
  • AttT2M uses body-part attention in a spatio-temporal VQ-VAE to produce a more expressive discrete motion representation.The motion encoder combines a spatial transformer and temporal convolution before VQ-VAE quantization.Source: §1 · p. 2
  • Global and Local Attention learns sentence-level and word-level motion-text correspondences for precise alignment between generated motion and text.The method uses motion-word cross-attention and motion-sentence conditional self-attention, followed by a generative transformer.Source: §1 · p. 2

2. Related Works

Related work spans motion synthesis under different conditions and several approaches to text-driven generation. AttT2M is positioned against methods that learn motion representations and methods that model text-motion relationships at different levels.

2. Related Works

  • Human motion synthesis research includes unconstrained generation, low-level control, motion-graph search and blending, and prediction from past motion sequences.The passage also identifies music and text as cross-modal conditions for motion generation.Source: §2 · pp. 2–3
  • Motion latent learning reduces the dimensionality of complex pose sequences; prior methods commonly emphasized temporal features, while spatial features were also found crucial.Autoencoders, VAEs, and VQ-VAEs are among the approaches used to learn motion representations.Source: §2 · p. 3
  • Prior text-driven methods either align text and motion in a shared representation, use rendered-body image-text learning, or condition motion generation on text.The conditional-generation group includes encoder-decoder and generative-model approaches, including discrete latent codes and diffusion models.Source: §2 · p. 3

3. Our Method

AttT2M frames text-driven motion as generating a motion sequence from a text description, separating motion embedding from text-motion relationship learning.

3.1. Method Overview

  • The framework first encodes motion into a discrete latent space, then learns text-motion correspondences and generates motion codes for decoding.The two stages are motion embedding and text-driven generation.Source: §3.1 · p. 4

3.2. The Motion Embedding Module

  • Body-part attention and temporal convolution encode motion’s spatial and temporal structure; VQ-VAE quantizes the resulting features into a discrete latent space.The encoder treats body parts as units for spatial feature extraction, while the temporal encoder produces spatio-temporal features.Source: §3.2 · p. 4
  • VQ-VAE maps spatio-temporal features to discrete motion representations using a learnable codebook, with reconstruction, velocity, embedding, and commitment loss terms.The passage identifies EMA and codebook reset as training choices.Source: §3.2 · p. 4

3.3. The Text-driven Motion Generation Module

  • Global and local attention model sentence-to-motion and word-to-motion correspondences, addressing both whole-sequence semantics and links between words and motion subsequences.The method uses local cross-attention for word-level features and conditional self-attention for sentence-level features.Source: §3.3 · pp. 4–5
  • A generative transformer predicts motion codes from the text-conditioned representation, and the motion decoder converts the generated code sequence into motion.At testing, generation starts from the sentence-level feature and proceeds through a motion-code sequence.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate AttT2M on KIT-ML and HumanML3D, assessing benchmark performance, qualitative generation, component ablations, attention behavior, and user preferences. The reported results include improved benchmark metrics, text-matched motion, fine-grained generation, and action-to-motion generation.

4.1. Dataset and Metrics

  • KIT-ML has 3,911 motions and 6,353 annotations; HumanML3D has 14,616 segments and 44,970 descriptions; evaluation uses FID, R-precision, MM-D, Div, and MM.The metrics assess distribution distance, text-motion consistency, feature distance, overall motion diversity, and diversity for the same text, respectively.Source: §4.1 · pp. 5–6

4.2. Quantitative Evaluation

  • HumanML3D results improve significantly on almost all metrics, with higher top-1, top-2, and top-3 R-Precision, and the lowest MM-D and FID.The table reports averages over 20 runs with 95% confidence intervals.Source: Table 1 · p. 6, §4.2 · p. 6

4.3. Qualitative Evaluation

  • Visualizations include diverse, consistent long motions such as flips, crouches, and crawls; versus MD and T2M-GPT, AttT2M better follows right-hand and quickness wording and clockwise-circle trajectories.The qualitative comparisons use skeleton sequences.Source: §4.3 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7
  • Replacing one word, such as “left” with “right” or “both,” or “put up” with “cross,” changes the generated action; action labels alone also generate motion without extra training.The reported fine-grained examples include raising different arms and crossing both arms.Source: Figure 6 · p. 7, §4.3 · p. 8, Figure 7 · p. 8

4.4. Ablation Study

  • Removing BPST lowered three R-Precision metrics and sharply raised FID; removing either GLA branch significantly lowered R-Precision, while removing both modules worsened all metrics.The ablations evaluate BPST, local attention, and global attention on HumanML3D.Source: §4.4 · p. 8, Table 3 · p. 8

4.5. Attention Visualization

  • Figure 8’s heatmaps show strongest word attention for action terms including “outstretched,” “raise,” and “returning,” while words tied to body parts and states also receive attention.The visualization presents motion-word cross-attention in 2D and 3D.Source: Figure 8 · p. 8, §4.5 · p. 9

4.6. User Perceptual Study

  • In a 30-participant perceptual study, “Ours” achieved the highest score and lowest standard deviation among six models generating motion from 30 HumanML3D test-set texts and 10 manual texts.Participants rated generated motion on a 1–6 scale, with 6 best, using a combination of motion realism and text-motion consistency.Source: §4.6 · p. 9

5. Limitations and Failure Cases

The paper reports limitations in motion diversity for very long, detailed descriptions, fine-grained generation without similar training examples, and out-of-distribution generation.

5. Limitations and Failure Cases

  • For very long, detailed test descriptions, generated motions can be too similar when HumanML3D contains few corresponding sequences.The authors report that the method can generate diverse motions generally, but diversity is insufficient for this case.Source: §5 · p. 9
  • Fine-grained edits may not match the text when the changed word represents motion absent from the dataset or without a similar example.Source: §5 · p. 9
  • For text outside the dataset distribution, the method cannot generate motion consistent with the description.Source: §5 · p. 9

6. Conclusion

AttT2M encodes motion into a discrete latent space, learns global and local text-motion relationships, and uses a generative transformer. The authors report better qualitative and quantitative results than state-of-the-art methods, alongside fine-grained synthesis and action-to-motion generation.

6. Conclusion

  • AttT2M combines body-part attention and temporal convolution with VQ-VAE to map spatio-temporal motion features into a discrete latent space.Source: §6 · p. 9
  • Global and local attention learn cross-modal relationships through sentence-level conditional self-attention and word-level cross-attention.Source: §6 · p. 9
  • A generative transformer produces motion, and experiments report better qualitative and quantitative results than state-of-the-art methods, plus fine-grained synthesis and action-to-motion generation.Source: §6 · p. 9

Related papers

Loading 2309.00796v1…