Source-linked AI summary
Unsupervised 3D Pose Estimation with Geometric Self-Supervision
Ching-Hang Chen, Ambrish Tyagi, Amit Agrawal, Dylan Drover, Rohith MV, Stefan Stojanov, James M. Rehg
TL;DR
The paper tackles 3D human-pose recovery from 2D landmarks without 3D data, multi-view images, correspondences, or learned 3D priors. It uses geometric lift–reproject–lift self-supervision with a 2D discriminator and domain adapter, and reports improved Human3.6M performance over prior unsupervised methods. Temporal discrimination adds a further 7% improvement even for single-frame inference.
Problem
3D motion-capture data for supervised 2D-to-3D lifting is difficult, expensive, and limited, motivating unsupervised lifting from abundant 2D poses.
Method
A lifting network uses geometric self-consistency over random reprojections, a 2D pose discriminator, and an unsupervised adapter for cross-domain 2D joints.
Results
The approach improves reconstruction accuracy when geometric self-supervision is combined with a discriminator, while temporal discrimination adds 7% even for single-frame inference.
Takeaways & Limitations
2D pose data from multiple domains can be used for unsupervised 3D lifting through domain adaptation.
Takeaways & Limitations
The approach assumes complete joints and fixes skeleton distance to the camera because absolute metric depth cannot be recovered from a single view.
Abstract
from arXiv · showhide
We present an unsupervised learning approach to recover 3D human pose from 2D skeletal joints extracted from a single image. Our method does not require any multi-view image data, 3D skeletons, correspondences between 2D-3D points, or use previously learned 3D priors during training. A lifting network accepts 2D landmarks as inputs and generates a corresponding 3D skeleton estimate. During training, the recovered 3D skeleton is reprojected on random camera viewpoints to generate new "synthetic" 2D poses. By lifting the synthetic 2D poses back to 3D and re-projecting them in the original camera view, we can define self-consistency loss both in 3D and in 2D. The training can thus be self supervised by exploiting the geometric self-consistency of the lift-reproject-lift process. We show that self-consistency alone is not sufficient to generate realistic skeletons, however adding a 2D pose discriminator enables the lifter to output valid 3D poses. Additionally, to learn from 2D poses "in the wild", we train an unsupervised 2D domain adapter network to allow for an expansion of 2D data. This improves results and demonstrates the usefulness of 2D pose data for unsupervised 3D lifting. Results on Human3.6M dataset for 3D human pose estimation demonstrate that our approach improves upon the previous unsupervised methods by 30% and outperforms many weakly supervised approaches that explicitly use 3D data.
1. Introduction
The paper addresses unsupervised 3D human-pose lifting from 2D landmarks without 3D data, correspondences, multi-view images, or explicit 3D priors. It combines geometric self-consistency with adversarial feedback, domain adaptation, and temporal supervision to improve lifting from diverse 2D pose data.
- Motivation: 3D pose estimation is ill-posed, while obtaining 3D motion-capture data is difficult, expensive, and limited.The paper motivates learning from abundant 2D image and video pose data instead.
- Geometric self-supervision: The method exploits closure and viewpoint invariance: accurate 3D lifts should reproject to valid 2D poses and recover the same 3D skeleton across views.These properties provide the geometric basis for unsupervised lifting.
- Geometric self-supervision: Self-consistency is necessary but insufficient; combining its loss with a 2D pose discriminator improves lifting performance and encourages realistic projections.The discriminator checks whether randomly projected skeletons lie within the distribution of valid 2D poses.
- Domain adaptation: A 2D domain adapter maps source-domain joints toward a target domain without correspondences, enabling additional datasets to enrich target-domain viewpoints, poses, and articulations.The adapter supports using 2D poses from different domains.
- Temporal consistency: Adding a temporal discriminator improves performance by an additional 7%, even when inference uses a single frame.The discriminator classifies changes in 2D joints across subsequent frames as real or fake.
- Contribution: The approach learns to lift 2D joints using only samples of real 2D poses, without using 3D data in any form.This defines the paper’s unsupervised training setting.
2. Related Work
Prior work estimates 3D pose with paired 2D–3D supervision, unpaired 3D priors, multi-view data, or adversarial models requiring 3D information. The paper instead learns 3D lifting from 2D projections alone, without multi-view images or 3D inputs.
- Fully Supervised: Fully supervised methods learn 2D-to-3D mappings from paired 2D and 3D joint data, often incorporating dataset-specific camera and skeleton parameters.Examples include direct joint regression, distance-matrix regression, exemplar databases, and depth-ordering models.
- Weakly Supervised: Weakly supervised methods avoid paired correspondences but use unpaired 3D data to learn shape or pose priors.These priors include pose dictionaries, kinematic constraints, and low-dimensional representations learned from 3D annotations.
- Weakly Supervised: Drover et al. synthetically generated 12M 2D joints from ground-truth 3D points, augmenting 1.5M Human3.6M poses by almost ten times.Their approach oversamples camera variations to learn 3D priors.
- Unsupervised: Rhodin et al. require synchronized multi-view images and repeated observations of subjects, whereas this paper learns 3D pose from 2D projections alone.The cited unsupervised comparison reports 130.9 mm versus a 127.3 mm trivial baseline for another method.
- Learning Using Adversarial Loss: Earlier adversarial approaches discriminate real and fake 2D or 3D poses but still require 3D data or previously learned 3D pose models.This paper applies adversarial loss to randomly projected 2D poses instead.
3. Unsupervised 2D-3D Lifting
The method lifts normalized 2D poses into 3D, applies random camera transformations, and closes the cycle through reprojection and relifting. Self-consistency is combined with pose and temporal discriminators, while domain adaptation aligns diverse 2D inputs to the target representation.
- 3.1. Lifting Network: The lifting network G(x) maps each 2D pose to corresponding 3D joints while estimating depth offsets relative to a fixed camera-distance plane.Inputs are normalized, with the root joint at the origin and the mean head-to-root distance set to 1; absolute metric depth is unavailable from one view.
- 3.2. Random Projections: Generated 3D skeletons are randomly rotated, translated to camera distance c, and perspective-projected into synthetic 2D poses.Azimuth is sampled from [-π, π] and elevation from [-π/9, π/9].
- 3.3. Self-Supervision via Loop Closure: The synthetic 2D pose is lifted again, inverse-transformed, and reprojected so the original and reconstructed 2D and 3D skeletons can be compared.The same lifting network is used in both cycle directions, producing the geometric self-consistency losses L3D and L2D.
- 3.4. Discriminator for 2D Poses: A 2D pose discriminator distinguishes real poses from projected poses, providing feedback that helps the lifter learn plausible limb-length ratios and joint angles.The discriminator does not require the real pose to match the lifter’s training sample or any multiview correspondence.
- 3.5. Temporal Consistency During Training: An optional temporal discriminator classifies differences between adjacent 2D poses during training, improving single-frame lifting while preserving single-frame inference.The reported ablation attributes an additional 7% performance improvement to temporal discrimination.
- 3.6. Learning from 2D Poses in the Wild: A 2D adapter maps source-domain joints to target-domain semantics without correspondences, using distribution matching and a small-magnitude correction regularizer.Corrected poses are fed to both the lifting network and 2D discriminator during training; the adapter is an offline preprocessing step.
4. Experimental Evaluation
Experiments on Human3.6M and MPI-INF-3DHP show that the unsupervised approach improves over prior unsupervised lifting, while ablations identify discriminator, domain-adaptation, and temporal components as important. Qualitative tests show generalization to MPII and LSP, but failures remain with occlusions, viewing-direction ambiguities, and missing joints.
- Quantitative Results: 30% lower error than Rhodin et al. is achieved on Human3.6M, improving from 98.2mm to 68mm.
- Ablations: 55mm error is obtained after augmenting adversarial self-consistency training with adapted 2D poses from Kinetics.
- Ablations: 51mm error is obtained on Human3.6M when self-consistency, adversarial feedback, adapted Kinetics poses, and temporal cues are combined.Inference remains single-frame despite temporal-discriminator training.
- Ablations: Self-consistency alone can produce geometrically consistent but unrealistic skeletons, whereas discriminator feedback sharpens limb-ratio distributions toward real values.Symmetry aligns left/right limb distributions but does not ensure realistic ratios.
- Supervised Fine-tuning: With 5% randomly sampled Human3.6M 3D data for fine-tuning, the method reaches 37mm, comparable to a fully supervised method.
- Limitations: Failures on Human3.6M are mostly attributed to self-occlusions or flip ambiguities in viewing direction, and the approach cannot handle occluded or missing joints.On Kinetics, 17M skeletons with at least 10 joints yielded 9M complete 14-joint skeletons.
- Qualitative Results: The method successfully reconstructs poses on MPII and LSP without training on those datasets, demonstrating qualitative cross-dataset generalization.
5. Conclusions
The paper concludes that geometric self-supervision improves unsupervised 3D pose reconstruction when combined with a discriminator, while domain adaptation enables unsupervised use of data from different datasets.
- Geometric self-supervision improves 3D reconstruction accuracy when combined with a discriminator, but cannot generate realistic skeletons by itself.
- A domain adapter enables unsupervised use of data from different domains and datasets.
- The approach learns 3D skeletons from 2D joints without requiring 3D data in any form.