Source-linked AI summary
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction
Ye Yuan, Kris Kitani
TL;DR
Existing generative motion-prediction methods pay limited attention to producing diverse samples after training, and independent likelihood-based sampling does not ensure diversity. DLow learns mappings from one random variable to correlated latent codes, optimizes a diversity-promoting prior with KL constraints, and outperforms baselines in sample diversity and accuracy while supporting controllable motion generation.
Problem
Post-training sampling from deep generative models for human motion prediction is under-explored, and independent likelihood-based latent sampling does not ensure diverse or mode-covering samples.
Method
DLow maps one random variable through learnable functions to correlated latent codes, then optimizes a diversity-promoting prior with KL constraints.
Results
DLow outperforms state-of-the-art baselines in sample diversity and accuracy and can generate diverse motions with shared features such as similar leg motion.
Takeaways & Limitations
Flexible diversity-promoting priors extend pretrained generative models to controllable human motion prediction.
Abstract
from arXiv · showhide
Deep generative models are often used for human motion prediction as they are able to model multi-modal data distributions and characterize diverse human behavior. While much care has been taken into designing and learning deep generative models, how to efficiently produce diverse samples from a deep generative model after it has been trained is still an under-explored problem. To obtain samples from a pretrained generative model, most existing generative human motion prediction methods draw a set of independent Gaussian latent codes and convert them to motion samples. Clearly, this random sampling strategy is not guaranteed to produce diverse samples for two reasons: (1) The independent sampling cannot force the samples to be diverse; (2) The sampling is based solely on likelihood which may only produce samples that correspond to the major modes of the data distribution. To address these problems, we propose a novel sampling method, Diversifying Latent Flows (DLow), to produce a diverse set of samples from a pretrained deep generative model. Unlike random (independent) sampling, the proposed DLow sampling method samples a single random variable and then maps it with a set of learnable mapping functions to a set of correlated latent codes. The correlated latent codes are then decoded into a set of correlated samples. During training, DLow uses a diversity-promoting prior over samples as an objective to optimize the latent mappings to improve sample diversity. The design of the prior is highly flexible and can be customized to generate diverse motions with common features (e.g., similar leg motion but diverse upper-body motion). Our experiments demonstrate that DLow outperforms state-of-the-art baseline methods in terms of sample diversity and accuracy. Our code is released on the project page: https://www.ye-yuan.com/dlow.
1 Introduction
Human motion prediction must account for diverse, multi-modal futures, especially in safety-critical applications. DLow addresses underexplored sampling limitations by learning correlated latent codes and optimizing their diversity while preserving likelihood.
- Motivation: Future human motion is diverse and multi-modal, so safety-critical systems need diverse predictions rather than only the most likely motion.The paper illustrates this with pedestrians who may suddenly cross despite usually remaining in place.
- Problem: Most generative motion-prediction work emphasizes designing the generative model, while paying little attention to sampling from the pretrained model.Prior methods commonly use random sampling to produce motion sets.
- DLow: DLow maps one random variable through learnable functions into correlated latent codes, which are decoded into correlated motion samples.Using a common random factor enables modeling the joint sample distribution.
- DLow: DLow trains the mappings with a diversity-promoting prior and KL constraints to balance sample diversity against likelihood.The prior is based on pairwise sample distances, while the KL term supports likelihood.
- Results: Experiments show that DLow outperforms state-of-the-art baselines in sample diversity and accuracy.The paper also presents flexible prior designs for applications such as controllable motion prediction.
2 Related Work
Related work includes deterministic motion predictors and stochastic methods based on deep generative models. The paper also connects its sampling problem to broader diversity-driven inference methods.
- Human Motion Prediction: Deterministic human-motion predictors regress a single future motion from past poses or video frames.These methods can predict the most likely motion but do not model multi-modal future behavior.
- Human Motion Prediction: Stochastic human-motion prediction methods became more prominent with the development of deep generative models.The cited methods are presented as more closely related to DLow than deterministic approaches.
- Diverse Inference: Diverse inference has also been studied through M-Best MAP methods, multiple choice learning, and submodular function maximization.These approaches target diverse solution sets or diverse subsets in computer vision and machine learning.
3 Diversifying Latent Flows (DLow)
DLow addresses the under-explored problem of sampling diverse motions from a pretrained generative model by learning correlated latent mappings while keeping the generator fixed. Its objective promotes sample diversity while constraining latent distributions to preserve likelihood, and the framework supports structural diversity constraints.
- Motivation and overview: DLow targets sampling diverse outputs after generative-model training, rather than redesigning or retraining the pretrained generator.The method is presented as a sampling strategy for a fixed generative model and is applicable beyond the CVAE used in this work.
- Motivation and overview: Random independent latent sampling may produce similar motions and fail to cover other modes because it neither models sample repulsion nor explicitly promotes diversity.This creates low sample efficiency when many generated samples resemble one another.
- DLow sampling: DLow samples one shared random variable and maps it through learnable functions to correlated latent codes, which are decoded into correlated motion samples.The resulting joint latent and sample distributions are controlled by the mapping functions, enabling structural constraints on the sample set.
- Diversity-likelihood objective: A diversity-promoting prior minimizes cross entropy with the joint sample distribution, while KL constraints keep each latent distribution close to the Gaussian prior.The Lagrangian objective softens the constraints, allowing a trade-off between sample diversity and likelihood.
- Mapping design: Different rotations or reflections can map a shared Gaussian variable to distinct feasible latent points, improving diversity without compromising likelihood under hard constraints.Soft KL constraints enlarge the feasible mapping space and can further reduce the diversity-prior loss.
- Flexible diversity design: DLow’s correlated sampling framework allows structural diversity constraints, and its diversity-promoting prior can be designed for diverse motions with common features.The method therefore controls the joint sample distribution rather than relying on independent random draws.
4 Diverse Human Motion Prediction
Given past motion c, the goal is to generate diverse future motions X using a CVAE and DLow’s learned latent mappings. The diversity-promoting prior can also impose accuracy, controllability, and reference-motion constraints.
- Diverse Human Motion Prediction: Given past motion c, the task is to generate a diverse set of future motions X={x1,...,xK}.The past motion contains H time steps, while each future motion spans T time steps and V-dimensional 3D joint positions.
- Diverse Human Motion Prediction: A CVAE learns the future trajectory distribution pθ(x|c), using recurrent encoder and decoder networks with Gaussian prior, posterior, and likelihood distributions.The encoder produces posterior parameters from x and c, while the decoder reconstructs the future trajectory from z and c.
- Diversity Sampling with DLow: DLow optimizes learnable latent mappings after CVAE training, enabling correlated samples that can be distributed for both diversity and accuracy.Its main-experiment prior uses a weighting factor λr and Euclidean sample distance.
- Controllable Motion Prediction: The diversity-promoting prior can be redesigned to encourage diversity in selected motion subspaces while preserving similarity in others.For joints Js and Jd, the prior minimizes motion distance on Js and applies diversity promotion to Jd.
- Controllable Motion Prediction: A reference motion can serve as the first sample, with its latent code obtained through the CVAE encoder and used to construct the remaining sample set.DLow enforces the first mapping as an identity so the reference latent code remains covered by the sampling distribution.
5 Experiments
Experiments on Human3.6M and HumanEva-I evaluate DLow’s diversity, accuracy, multimodal coverage, qualitative behavior, trade-offs, and controllable prediction. DLow outperforms baselines while allowing flexible control over generated motion.
- Quantitative Results: DLow outperforms all baselines on Human3.6M and HumanEva-I for diversity, accuracy, and multimodal ground-truth coverage.Evaluation uses APD, ADE, FDE, MMADE, and MMFDE with sample set size K = 50.
- Ablation Study: DLow’s diversity and accuracy depend on both energy terms: removing reconstruction increases diversity but reduces accuracy, while removing diversity reduces both.Optimizing only LKL produces the worst results.
- Qualitative Results: DLow generates varied future motions covering the ground truth, whereas baselines mainly produce perturbations of one motion or nearly identical motions.This pattern appears on both Human3.6M and HumanEva-I qualitative comparisons.
- Diversity vs. Likelihood: Smaller β produces more diverse, less likely motions, while larger β favors less diverse samples near the generator’s major mode.Varying β therefore exposes a diversity–likelihood trade-off.
- Effect of varying ϵ: Changing ϵ produces different sample sets while preserving the main structure of each motion.This gives DLow multiple diverse sample sets, unlike DSF’s fixed diverse set.
- Controllable Motion Prediction: A flexible diversity-promoting prior enables similar leg motion with diverse upper-body motion, which random CVAE samples cannot enforce.The controllable experiment uses a reference motion and an energy function designed for this constraint.
6 Conclusion
The paper proposes DLow, a sampling strategy that uses learnable latent mappings and a diversity-promoting prior to generate diverse future human motions. Experiments show superior diverse-sample generation, and the flexible prior supports controllable motion prediction.
- Conclusion: DLow generates correlated future-motion samples through learnable latent mapping functions optimized with a diversity-promoting prior.The approach is presented as a sampling strategy for deep generative models.
- Conclusion: Experiments demonstrate superior performance in generating diverse motion samples.The conclusion reports this as the main experimental outcome.
- Conclusion: The flexible diversity-promoting prior enables applications such as controllable human motion prediction.The conclusion identifies controllable prediction as an example of the approach’s broader use.
A Additional Human3.6M Results
Additional Human3.6M results provide qualitative comparisons with baselines and further examples generated by DLow.
- The section compares DLow with baseline methods on Human3.6M using additional qualitative results.The comparison is presented in Fig. 8.
- It also presents additional DLow examples on Human3.6M.These examples are shown in Fig. 9.
A.1 Additional Comparison with Baselines on Human3.6M
Additional Human3.6M visualizations compare the ground-truth future with motion samples from each method and show further DLow examples across sequences.
- Figure 8 compares the start pose, ground-truth future end pose, and 10 sampled motion end poses for each method.
- Figure 9 shows additional DLow examples across different Human3.6M sequences using the start pose, ground-truth future end pose, and 10 sampled end poses.
B Additional Controllable Motion Prediction Results
The controllable-motion results show that DLow can preserve reference leg motion while varying upper-body motion, unlike random CVAE sampling.
- DLow constrains samples to have similar leg motion to the reference while maintaining diverse upper-body motion.The controllable results use Human3.6M and evaluate leg-motion distance alongside upper-body APD.
- Table 3 reports quantitative results for controllable motion prediction.
- The figure presents reference-conditioned comparisons between five DLow samples and five CVAE samples.
- CVAE random samples cannot enforce similar leg motion to the reference in this controllable setting.
C Metrics vs. Number of Samples K
Figure 11 plots metrics against the number of generated samples K for Human3.6M and HumanEva-I.
- The figure compares metrics as the sample count K changes on Human3.6M and HumanEva-I.Human3.6M is shown on the left and HumanEva-I on the right.
D Additional HumanEva-I Results
The paper adds qualitative HumanEva-I results, including baseline comparisons and further DLow examples across motion sequences.
- HumanEva-I provides qualitative evaluation on a much smaller dataset with less motion variation.
- Additional HumanEva-I comparisons show the start pose, ground-truth future end pose, and 10 motion-sample end poses for each method.
- Additional DLow examples on HumanEva-I present the start pose, ground-truth future end pose, and 10 sampled end poses for each sequence.