PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaussian-Mixture Latent Flow for Stochastic 3D Human Motion Prediction

Yue Ma, Frederick W. B. Li, Xiaohui Liang

arXiv:2608.21093v1cs.CV

TL;DR

Stochastic human motion prediction must model diverse, plausible futures and their uncertainty, but single-modal priors can entangle motion semantics. This paper uses a latent flow with a data-driven Gaussian mixture prior and reports state-of-the-art accuracy and plausibility on Human3.6M and AMASS.

  • Problem

    Stochastic human motion prediction requires diverse, plausible future motions with associated probability densities, while single-modal priors poorly capture complex multi-modal patterns.

  • Method

    The method combines a latent flow with a data-driven Gaussian mixture prior that adaptively models diverse motion patterns without additional annotations and supports tractable likelihood estimation.

  • Results

    The method achieves state-of-the-art performance across two large-scale datasets, including 8.5% and 13% FDE gains on Human3.6M and AMASS, respectively.

  • Takeaways & Limitations

    A learnable multi-modal prior better captures diverse motion patterns than standard Gaussian, fixed-mixture, or prior-free alternatives, while invertibility enables natural uncertainty quantification.

  • Takeaways & Limitations

    The prior may perform poorly on long-tailed data distributions, where its latent mixture can degenerate into a single-modal form.

Abstract

from arXiv · show

Stochastic human motion prediction aims to forecast future motion distributions. Although recent studies have achieved strong performance in terms of accuracy and diversity, they often overlook plausibility (e.g., resulting in physically unrealistic predictions) and uncertainty quantification, both of which are essential for real-world applications and downstream tasks. To address these issues, we propose a latent flow-based model equipped with a data-driven Gaussian mixture prior that more effectively disentangles diverse human behaviors than conventional single-modal priors. This prior is derived from patterns in the training data without requiring additional annotations. Furthermore, the fully invertible nature of our model enables natural uncertainty quantification through tractable likelihood computation. Experiments on the Human3.6M and AMASS datasets demonstrate that our approach achieves state-of-the-art performance in both accuracy and plausibility.

1. Introduction

Stochastic human motion prediction must model accuracy, diversity, plausibility, and uncertainty for real-world use. The proposed flow-based method combines a data-driven Gaussian mixture prior with invertibility to address these challenges.

  • Human motion prediction supports applications including autonomous driving, human–robot collaboration, and assistive robotics.
  • Stochastic HMP must forecast diverse, plausible motions while faithfully modeling predictive uncertainty.
  • Single-modal priors can sacrifice plausibility, producing physically infeasible or physiologically unnatural predictions.
  • The method uses a data-driven Gaussian mixture prior to disentangle diverse motion patterns and a fully invertible flow architecture for tractable likelihood computation.The design targets semantic confusion and enables uncertainty quantification.

2. Related Work

Prior stochastic HMP methods use generative models but retain limitations in multimodal representation, plausibility, and uncertainty estimation. Flow-based models offer invertibility, while this work applies a learned mixture prior to motion semantics.

  • Stochastic HMP methods include GANs, VAEs, and diffusion models that map simple priors toward future-motion distributions.
  • Single-modal assumptions persist in some methods, while intractable likelihoods hinder uncertainty estimation for safety-aware scenarios.
  • Existing uncertainty methods often rely on simplistic distributional assumptions or deterministic prediction, while Motron lacks constraints for realistic predictions.
  • Normalizing flows construct complex distributions through invertible transformations, with continuous variants representing the process using ODEs.
  • Most flow-based approaches use standard Gaussian bases, whereas this work learns a mixture distribution directly during training to capture motion semantics.

3. Preliminary

Flow matching models a probability path between two distributions with a velocity field governed by an ODE. Rectified Flow uses linear interpolation and trains the velocity field to match the resulting constant direction.

  • Flow matching defines a velocity field that establishes a probability path between distributions π0 and π1.
  • The ODE d x_t = v(x_t, t)d t evolves samples between endpoint distributions, with integration achieving the transformation.
  • Flow matching avoids time-consuming ODE integration during training by optimizing only the velocity field.
  • Rectified Flow defines the intermediate state as a linear interpolation between endpoint samples.The trajectory follows a straight-line direction with constant velocity.
  • The Rectified Flow objective minimizes the squared discrepancy between endpoint displacement and the predicted velocity over t ∈ [0,1].

4. Problem Formulation

The problem is to predict diverse future 3D human motions from observed frames together with their probability densities. The formulation represents uncertainty through reconstructed-sequence log-probabilities.

  • Given an observed sequence Xobs of T frames, the model predicts the subsequent N future frames.
  • The formulation distinguishes predicted future motion Ŷ, ground-truth future Y, and reconstructed full sequence X̂.
  • The reconstructed sequence has log-probability log p(X̂), which naturally quantifies predictive uncertainty.
  • A stochastic forecast is represented as a set of M diverse future predictions.

5. Methodology

The method constructs a data-driven Gaussian-mixture latent space and uses an invertible flow with neural-ODE dynamics to forecast diverse motion while supporting likelihood-based uncertainty quantification.

  • Gaussian-Mixture Latent Representation: An unsupervised EM procedure learns a Gaussian-mixture latent prior that disentangles distinct motion behaviors without auxiliary labels.The E-step computes component posteriors, while mixture parameters and flow parameters are optimized iteratively.
  • Gaussian-Mixture Latent Representation: The part-aware flow encodes and decodes motion through hierarchical body-part transformations while preserving joint-level semantic structure.The pipeline also applies DCT-based low-pass filtering to retain low-frequency temporal and channel components for smoother motion generation.
  • Inference and Decoding: During inference, the model pads observations, encodes them to Z0, evolves the latent state with an ODE solver, and decodes the resulting codes into future motion.The flow's forward and inverse processes allow one model to serve as both encoder and decoder.
  • Latent Flow Matching: Latent forecasting transports a noisy initial code pZ0 = N(Z0, I) toward the ground-truth latent code Z1 using a learned velocity field and linear conditional path.The velocity field is trained with a flow-matching objective based on the difference between Z1 and the sampled initial code.
  • Latent Flow Matching: A skeleton-aware transformer models the velocity field by tokenizing individual-joint trajectories and combining joint-wise attention with temporal conditioning on observed motion.Latent features are modulated according to the flow-matching time step through affine and scaling transformations within transformer blocks.

6. Experiments

Experiments on Human3.6M and AMASS evaluate accuracy, diversity, plausibility, qualitative realism, latent-prior design, and likelihood estimation. The method delivers strong accuracy and plausibility, while ablations and likelihood results support its learnable Gaussian-mixture prior and probabilistic formulation.

  • Quantitative Results: The method is evaluated on Human3.6M and AMASS using accuracy, diversity, and plausibility measures with Best-of-50 predictions.Human3.6M uses 25 observed and 100 predicted frames; AMASS uses 30 observed and 120 predicted frames.
  • Quantitative Results: 8.5% and 13% FDE gains are reported on Human3.6M and AMASS, respectively, while the method ranks among top performers on MMADE and MMFDE.On AMASS, only CoMusion surpasses the method for the multimodal accuracy metrics.
  • Qualitative and Plausibility Analysis: The method balances diversity and plausibility, avoiding unrealistic motions while retaining meaningful variation and competitive ground-truth distribution coverage.VAE-based methods achieve higher APD but exhibit worse FID and CMD scores; the proposed method shows slightly lower diversity than diffusion baselines.
  • Qualitative and Plausibility Analysis: Qualitative AMASS comparisons show more natural pose forecasts, fewer extreme joint bends, better alignment with observed-history semantics, and non-trivial variation.The figure visualizes the final frame of 10 randomly sampled predictions for each compared method.
  • Ablation Studies: A learnable mixture prior outperforms fixed-mixture, standard-normal, and no-prior variants, with single-modal or absent priors especially degrading plausibility.The ablation attributes the advantage to adapting to the data distribution and modeling multiple sub-distributions.
  • Ablation Studies: Removing past-history conditioning or using spatial tokenization degrades performance, while replacing N(Z0, I) with N(0, I) slightly increases diversity but significantly harms accuracy.Spatial tokenization substantially degrades both accuracy and plausibility relative to the comparison variant.
  • Likelihood Estimation: The Gaussian-mixture-prior model achieves the best log-likelihood on both datasets, supporting likelihood-based uncertainty quantification and modeling of complex motion distributions.It also surpasses the standard-normal, no-prior, and Motron comparisons.

7. Conclusion

The paper presents a latent flow model with a data-driven Gaussian mixture prior for stochastic human motion prediction, achieving state-of-the-art performance on two large-scale datasets. Its fully invertible design enables tractable likelihood estimation for uncertainty quantification, while the method remains limited on some tasks and long-tailed distributions.

  • The proposed model combines latent flows with a data-driven Gaussian mixture prior that adaptively captures diverse motion patterns.
  • The fully invertible model enables tractable likelihood estimation for natural uncertainty quantification.
  • The method achieves state-of-the-art performance on the Human3.6M and AMASS datasets.
  • The Gaussian mixture prior performs suboptimally when directly applied to classification and anomaly detection tasks.
  • The prior may degenerate toward a single-modal distribution on long-tailed data dominated by one mode.

I. Metric Calculation Details

The metric details define accuracy, diversity, plausibility, and latent-space analyses for stochastic motion prediction. Visualizations and initialization studies indicate that the multi-modal prior organizes motion patterns more coherently, while sufficiently many initial components preserve plausibility.

  • APD measures the average pairwise L2 distance among multiple predictions generated from the same observed history.
  • ADE measures the L2 distance between ground truth and the closest predicted future sample.
  • FDE measures the L2 distance between the ground-truth final pose and the closest predicted final pose.
  • MMADE and MMFDE aggregate closest-sample errors across multi-modal ground truths grouped by similar observed sequences.
  • CMD measures the difference between cumulative pseudo-data and predicted motion distributions using framewise displacement deviations.
  • Latent-space visualization: The multi-modal latent space forms partially overlapping clusters, whereas the single-modal space largely entangles sequences from different action labels.
  • Latent-space construction: With 16 or 32 initial components, accuracy, diversity, and plausibility remain nearly identical; reducing initialization to 8 or 1 significantly deteriorates plausibility.

IV. Analysis of the ODE Solver

The solver analysis examines Euler and Dopri5 integration for the flow model. Euler accuracy stabilizes beyond 100 steps, while Dopri5 attains comparable prediction quality at substantially lower computational cost.

  • Beyond 100 Euler integration steps, accuracy metrics remain stable and APD improves only marginally.
  • The solver comparison reports predictive performance and computational cost across Euler step counts from 1 to 200 and adaptive-step Dopri5.
  • Dopri5 produces prediction quality comparable to Euler while requiring significantly lower computational cost.

V. Analysis of Uncertainty Quantification

The uncertainty analysis evaluates whether likelihood-based rankings track prediction errors. Estimated probabilities correlate with lower ADE and FDE, with small ranking discrepancies and improvements over Motron while remaining close to ProbHMI.

  • Higher estimated probability ranks correlate strongly with lower ADE and FDE errors.
  • The analysis measures uncertainty quality using normalized area discrepancy between estimated and ideal error-rank curves.
  • ADE ranking discrepancies remain below 10% on both Human3.6M and AMASS, while FDE discrepancies are 13.7% and 14.4%, respectively.
  • The method reduces discrepancy by 25.6% on Human3.6M and 26.5% on AMASS compared with Motron, while remaining similar to ProbHMI.

VI. Additional Visualization Results

The method is compared qualitatively with recent stochastic predictors on Human3.6M and AMASS, including uncertainty-weighted visualizations. Additional deterministic experiments show competitive short-term performance and stronger long-term results, while the latent space offers limited benefit in that setting.

  • Qualitative comparisons: Comparisons with CoMusion, SLD, and SkeletonDiff evaluate accuracy, diversity, and plausibility on Human3.6M and AMASS.Each frame visualizes 10 stochastic forecasts.
  • Qualitative comparisons: Our approach produces more natural AMASS poses than baselines, avoiding unnaturally lifted legs and deviations from contextual cues.The cited examples compare against CoMusion and SkeletonDiff.
  • Uncertainty visualization: Uncertainty-weighted visualizations assign greater opacity to higher-probability predictions and concentrate high-opacity regions around the ground truth.Lower-opacity regions correspond to predictions with greater deviation.
  • Deterministic prediction: For deterministic prediction, the method is comparable to baselines at ≤400 ms and achieves the best results on Smoking, Sitting, and WalkingDog.These comparisons are reported in Table 7.
  • Deterministic prediction: At ≥560 ms, the method achieves the best performance on 7 of 15 action classes, while adding the latent space yields only marginal improvements.Competing methods achieve the best performance on at most 4 classes.
Loading 2608.21093v1…