Source-linked AI summary
Lifting Motion to the 3D World via 2D Diffusion
Jiaman Li, C. Karen Liu, Jiajun Wu
TL;DR
Existing 3D-motion methods typically require ground-truth 3D training data, which is difficult to collect for out-of-distribution motions and domains such as complex athletics or animals. MVLift progressively generates and optimizes consistent multi-view 2D pose sequences with diffusion models and geometric constraints, then uses synthetic data to train efficient multi-view generation. MVLift generalizes across human poses, human-object interactions, and animal poses and significantly outperforms existing methods on five datasets, including methods trained with 3D supervision.
Problem
Existing 3D-motion methods typically require ground-truth 3D training data, which is difficult to collect for out-of-distribution motions and domains such as complex athletics or animals.
Method
MVLift progressively generates and optimizes consistent multi-view 2D pose sequences with diffusion models and geometric constraints, then uses synthetic data to train efficient multi-view generation.
Results
MVLift generalizes across human poses, human-object interactions, and animal poses and significantly outperforms existing methods on five datasets, including methods trained with 3D supervision.
Takeaways & Limitations
2D-only training supports global 3D motion estimation in world coordinates without 3D supervision across human poses, human-object interactions, and animal poses.
Abstract
from arXiv · showhide
Estimating 3D motion from 2D observations is a long-standing research challenge. Prior work typically requires training on datasets containing ground truth 3D motions, limiting their applicability to activities well-represented in existing motion capture data. This dependency particularly hinders generalization to out-of-distribution scenarios or subjects where collecting 3D ground truth is challenging, such as complex athletic movements or animal motion. We introduce MVLift, a novel approach to predict global 3D motion -- including both joint rotations and root trajectories in the world coordinate system -- using only 2D pose sequences for training. Our multi-stage framework leverages 2D motion diffusion models to progressively generate consistent 2D pose sequences across multiple views, a key step in recovering accurate global 3D motion. MVLift generalizes across various domains, including human poses, human-object interactions, and animal poses. Despite not requiring 3D supervision, it outperforms prior work on five datasets, including those methods that require 3D supervision.
1. Introduction
MVLift targets global 3D motion from single-view 2D pose sequences without 3D training data, addressing limitations in out-of-distribution motions. It progressively establishes multi-view consistency through 2D diffusion and outperforms prior methods across five datasets.
- Existing approaches depend on 3D motion-capture data, limiting applicability to out-of-distribution motions and domains where 3D collection is difficult.The paper highlights complex athletic movements and animal motion as examples.
- MVLift progressively establishes multi-view consistency using 2D motion diffusion models, enabling 3D motion recovery without direct 3D supervision.The framework combines diffusion priors with geometric constraints across multiple stages.
- The method generalizes across human, animal, and interaction domains and significantly outperforms existing methods on five datasets, including 3D-supervised methods.
2. Related Work
Prior work commonly learns 3D pose or mesh motion from paired 2D–3D data, while video methods and multi-view generation address related reconstruction challenges. These approaches remain constrained by 3D-data availability for accurate global motion estimation.
- Human Pose Estimation from 2D Poses: Paired 2D–3D datasets support regressors based on fully connected, temporal convolutional, graph convolutional, and Transformer architectures.This paradigm maps 2D human joints to 3D poses but depends on paired data.
- Human Pose Estimation from Videos: Video-based methods estimate human mesh or motion from images and videos using optimization, stronger backbones, recurrent temporal modeling, and motion priors.
- Human Pose Estimation from Videos: WHAM [43] predicts world-coordinate 3D motion in one forward pass, but its global trajectory module is trained on ground-truth 3D datasets, constraining out-of-distribution root-trajectory accuracy.
- Multi-View Image Generation: Multi-view image-generation research has developed diffusion models that generate mutually consistent views, providing a related foundation for multi-view synthesis.
3. MVLift
MVLift lifts a single 2D motion sequence into global 3D by progressively strengthening multi-view consistency with geometric constraints, diffusion priors, optimization, and learned multi-view generation. Its pipeline moves from pairwise epipolar consistency to globally consistent synthetic data and efficient generation.
- Framework: MVLift combines 2D diffusion priors and geometric constraints to recover global 3D motion without ground-truth 3D training data.The goal is to estimate joint rotations and root trajectories in world coordinates from a single-view sequence.
- Stage 1: Line-Conditioned 2D Motion Diffusion Model: Stage 1 trains a line-conditioned diffusion model to generate 2D pose sequences whose joints align with simulated epipolar lines.The line conditions encode geometric relationships between camera views.
- Stages 3–4: Synthetic Data and Multi-View Generation: Stages 3 and 4 optimize 3D motion, reproject it into strictly consistent multi-view 2D sequences, and train a diffusion model for efficient simultaneous generation.The final model generates consistent views from a single 2D sequence input.
- Stage 1: Line-Conditioned 2D Motion Diffusion Model: The diffusion denoising step predicts a clean pose representation from a noisy sequence and line conditions using a neural network with fixed reverse-process variance.The denoising network uses a transformer architecture, with line conditions concatenated to noisy pose features.
- Stage 2: Multi-View 2D Motion Optimization: Stage 2 jointly optimizes unobserved-view 2D sequences with multi-view consistency and SDS objectives, improving beyond pairwise epipolar constraints.The consistency loss evaluates geometric relationships across view pairs, while SDS preserves the learned motion distribution.
4. Experiments
MVLift is evaluated across human, animal, and human-object motion datasets using 3D errors, reprojection quality, perceptual judgments, and ablations. It outperforms baselines across the reported lifting settings, including methods trained with 3D motion data.
- Datasets and Evaluation Metrics: The evaluation covers datasets with and without 3D motion, using root translation, MPJPE, PA-MPJPE, reprojection, FID, and object-motion metrics.AIST++ and OMOMO provide 3D motion, whereas Steezy, NicoleMove, and CatPlay provide only 2D motion.
- Human Pose Lifting: Participants preferred MVLift over every baseline across AIST++, Steezy, and NicoleMove; on AIST++, 39% preferred MVLift to ground truth and 11% noticed no difference.The study compared 15 generated sequences per approach using Amazon Mechanical Turk.
- Animal Pose Lifting: On CatPlay, MVLift outperforms all baselines and produces realistic 3D animal motion with natural root trajectories, unlike MAS and single-view SMPLify.MAS cannot generate root trajectories, while SMPLify produces unrealistic depth changes.
- Human-Object Interaction Lifting: On OMOMO, MVLift produces realistic human-object interactions, whereas SMPLify lacks realistic hand-object contact and generates abrupt object-depth changes.The method uses geometrically consistent generated multi-view 2D pose sequences for interaction optimization.
- Ablation Study: The final MVLift model estimates 3D motion more accurately than both earlier stages, while direct 3D optimization or unconditional SDS produces inaccurate or implausible motions.The ablations are evaluated on AIST++ against 3D ground truth.
5. Conclusion
MVLift estimates world-coordinate 3D motion from 2D motion sequences without 3D supervision by progressively enforcing multi-view consistency. It is demonstrated across human poses, human-object interactions, and animal poses.
- MVLift estimates world-coordinate 3D motion from 2D motion sequences without requiring 3D supervision.
- The framework combines 2D motion diffusion priors with geometric constraints to progressively establish cross-view consistency.
- MVLift uses optimized, strictly consistent multi-view sequences to train a final diffusion model for efficient multi-view generation.
- The approach is effective across human poses, human-object interactions, and animal poses.