Source-linked AI summary
Action2Motion: Conditioned Generation of 3D Human Motions
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, Li Cheng
TL;DR
The paper addresses the problem of generating plausible, diverse 3D human motions from prescribed action types. It proposes a conditional temporal VAE using Lie algebra representations and evaluates it across three datasets, demonstrating effectiveness against comparison methods.
Problem
The paper studies how to generate diverse, natural 3D human motion sequences conditioned on prescribed action categories rather than recognizing actions.
Method
Action2motion combines a conditional temporal VAE with Lie algebraic motion representation and a learned prior conditioned on previous poses.
Results
Across three datasets, action2motion outperforms comparison methods on FID and attains values closest to real motions for diversity and multimodality.
Takeaways & Limitations
The approach provides a dedicated framework and benchmark suite for evaluating action-conditioned 3D human motion generation.
Abstract
from arXiv · showhide
Action recognition is a relatively established task, where givenan input sequence of human motion, the goal is to predict its ac-tion category. This paper, on the other hand, considers a relativelynew problem, which could be thought of as an inverse of actionrecognition: given a prescribed action type, we aim to generateplausible human motion sequences in 3D. Importantly, the set ofgenerated motions are expected to maintain itsdiversityto be ableto explore the entire action-conditioned motion space; meanwhile,each sampled sequence faithfully resembles anaturalhuman bodyarticulation dynamics. Motivated by these objectives, we followthe physics law of human kinematics by adopting the Lie Algebratheory to represent thenaturalhuman motions; we also propose atemporal Variational Auto-Encoder (VAE) that encourages adiversesampling of the motion space. A new 3D human motion dataset, HumanAct12, is also constructed. Empirical experiments overthree distinct human motion datasets (including ours) demonstratethe effectiveness of our approach.
1 INTRODUCTION
The paper introduces action-conditioned generation of diverse, natural 3D human motions, addressing challenges that existing 2D and deterministic approaches do not resolve. It proposes a Lie algebra-based temporal VAE and constructs HumanAct12, with experiments demonstrating effectiveness across three datasets.
- Motivation: Existing video-generation methods often produce distorted human appearances and motions, motivating explicit modeling of body poses and temporal articulations.The paper identifies human representation and temporal articulation as central challenges in automated generation.
- Approach: The proposed framework generates 3D motions without initial pose or motion conditions using a conditional temporal VAE with Lie algebra representation.Its learned prior uses previous poses to capture temporal dependencies across consecutive poses.
- Approach: Lie algebra representation mitigates trembling in joint-coordinate models, supports natural lifelike motions, and reaches equilibrium with one tenth as many training iterations.The comparison is specifically against training with joint-coordinate representation.
- Contributions: The paper contributes a new prescribed-action 3D motion-generation problem, a Lie algebra-based VAE, and the HumanAct12 dataset.It also re-annotates NTU-RGB-D because its existing pose annotations are severely inaccurate.
2 RELATED WORK
Prior work generates human dynamics from audio, text, images, or action categories, while pose representations range from joint coordinates to Lie-group structures. The paper positions action2motion as the first action-based 3D human motion-generation approach and motivates dedicated datasets for evaluation.
- Multimodal 3D Human Dynamics Generation: Prior multimodal methods generate human dynamics from audio, text, or images using learned correlations, autoencoders, RNNs, or GAN-based translation.Examples include speech-to-gesture, music-to-dance, and text-to-sign generation.
- Action-Based Human Motion Generation: Action-based human motion generation has mainly been studied in 2D, including a two-stage GAN that progressively generates 2D motion.The paper states that action2motion is the first work for action-based 3D human motion generation.
- Skeleton-Based Human Pose Representations: Human pose representations include joint coordinates, relative joint positions, pruned keypoints, body-part structures, joint angles, and Lie-group kinematic chains.Lie-group methods represent 3D skeletons as points in SO(3) or SE(3).
- 3D Human Motion Datasets: Existing 3D human-action datasets contain large numbers of poses and sequences but may have highly unbalanced categories that do not align with typical action types.This creates challenges for using them directly as action-generation benchmarks.
3 METHODOLOGY
Action2Motion generates action-conditioned 3D pose sequences using Lie algebra representations and a conditional temporal VAE. The model combines kinematic forward mapping with learned temporal priors and mixed training to produce diverse, temporally consistent motions.
- Given an action category, the framework generates a 3D pose sequence M = [p1, ..., pT] of length T.
- 3.1 Disentangled Lie Algebra Representation: Lie algebra represents skeletal rotations on SO(3), enabling optimization in a flat tangent space while preserving kinematic structure.The skeleton is modeled as a product of SO(3) groups, and Lie algebra provides a space for linear computations.
- 3.1 Disentangled Lie Algebra Representation: Forward kinematics transforms generated Lie algebra parameters and root translations into 3D joint coordinates.Bone lengths remain fixed during generation, preserving invariance and allowing controllable body scale.
- 3.2.2 Our Approach: At each timestep, the generator produces Lie algebra parameters and root translation before applying forward kinematics to obtain the current 3D pose.The input includes the action category, normalized time counter, and current or previous pose representation.
- 3.2 Temporal VAE: A recurrent temporal VAE uses an encoder for posterior inference, a learned prior conditioned on previous poses, and a generator for pose synthesis.The KL-divergence term keeps posterior and prior distributions close, while the likelihood term encourages generated samples to match real sequences.
- 3.2.2 Our Approach: Testing samples latent variables from the learned prior, while training uses a mixed strategy that alternates teacher forcing with generated previous poses.This strategy addresses the mismatch between training with ground-truth poses and testing with model-generated poses.
4 EXPERIMENTS
Experiments evaluate motion naturality, action correlation, diversity, and multimodality across adapted datasets and HumanAct12. Action2motion generally produces more realistic and varied motions than comparison methods, with Lie algebra improving motion quality and training efficiency.
- Datasets and metrics: The evaluation uses HumanAct12, re-annotated NTU-RGB-D, and reorganized CMU MoCap, with recognition accuracy, FID, diversity, and multimodality as metrics.FID is emphasized as the primary overall-quality indicator; diversity and multimodality are better when closer to real-motion values.
- Qualitative evaluation: Action2motion generates visually appealing motions that match prescribed action types while avoiding the collapse, jerking, freezing, and skeletal defects observed in baselines.The coordinate-based model without Lie algebra retains variation but can produce changing bone lengths and unnatural skeletons.
- Diversity and multimodality: Action2motion produces diverse motions, while Act-MoCoGAN shows low variation and Two-stage GAN produces motions that are not well recognizable.The authors associate variation maintenance with the temporal conditional VAE in sequential generation.
- Quantitative comparison: Action2motion clearly outperforms comparison methods on FID across the three datasets and achieves the highest recognition accuracy on HumanAct12 and CMU MoCap.The experiments are repeated 20 times and report 95% confidence intervals.
- Representation analysis: Lie algebra brings diversity and multimodality closer to real-motion values and requires one tenth as many training iterations as joint-coordinate representation to reach equilibrium.It also mitigates trembling and supports natural, lifelike motion generation.
- Subjective evaluation: In user studies, 64% of action2motion samples rank in the top two positions, and 58% are judged real, only 3% below the rate for real motions.These studies assess preference, action-category matching, and perceived realism of rendered SMPL motions.
5 CONCLUSION AND OUTLOOK
The paper frames Action2Motion as a new problem of generating diverse, natural 3D human motions from prescribed actions. It contributes dedicated benchmarks and reports extensive experiments demonstrating effectiveness against comparison methods.
- Action2Motion targets 3D human-motion generation grounded on prescribed actions, emphasizing both diversity and naturalness.
- The framework is evaluated using an in-house dataset and two adapted existing datasets that provide dedicated evaluation benchmarks.
- Extensive qualitative, quantitative, and subjective experiments demonstrate the approach’s effectiveness against comparison methods.
- Future work will investigate a wider range of human actions, including actions involving two or more people.
A.1 Neural Network Architecture
The architecture section presents the Action2Motion model architecture for HumanAct12 and notes that input dimensions vary with pose-vector dimensionality across datasets.
- For the other two datasets, the input-vector dimension may vary according to the pose-vector dimension.
- Table 3 presents the architecture of the Action2Motion model used on HumanAct12.
A.2 Implementation Details
The implementation uses a PyTorch recurrent architecture with specified layer sizes, latent noise dimensionality, optimization settings, and approximately 450,000 parameters.
- The model is implemented in PyTorch with encoder and decoder output sizes of 128 and 72, respectively.
- A two-layer GRU serves as the generator, while one-layer GRUs serve as the prior and posterior networks, all with 128 hidden units.
- The noise vector z has 30 dimensions, and the learned model contains about 450,000 parameters.
- Training uses Adam with a learning rate of 0.0002 and weight decay of 0.00001.
A.3 Exemplar Human Skeleton
The exemplar skeleton is organized into five kinematic chains covering the spine, arms, and legs, with joints and bones arranged along those chains.
- The exemplar skeleton contains 21 joints and 20 bones organized into five kinematic chains.
- The five chains are the spine, left arm, right arm, left leg, and right leg.
A.4 Training Process Comparison
The training comparison shows that Lie algebra representation reaches plausible, stable motion generation substantially faster than the model without Lie algebra.
- Figure 8 compares generated walk motions from models with and without Lie algebra across training iterations.
- With Lie algebra, the model is ready to generate plausible pose sequences within 5,000 iterations and reaches the comparison point at 40,000 iterations.
A.5 Design of User Study
The paper evaluates motion quality through user preference and fake-or-real surveys, while documenting the construction and scope of its motion datasets.
- Design of User Study: Two user studies assess generated motions: a preference survey and a fake-or-real discrimination survey.The studies are presented in Figure 9 and implemented in HTML.
- Design of User Study: In the preference survey, users compare six randomly ordered motion clips associated with an action type and select their preferred clip.
- Design of User Study: In the fake-or-real survey, users classify six generated and real motions for each action type without relying on direct clip-to-clip comparison.
- Dataset Details: The improved NTU-RGB-D dataset contains 3,902 motion clips selected from 13 action types after partial re-annotation.
- Dataset Details: HumanAct12 contains 1,061 motion clips across 12 action classes and 34 subclasses, with 3D positions and corresponding action-type annotations.Other modalities remain accessible through alignment with the original PHSPD dataset.