PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning of 3D Human Pose using Multi-view Geometry

Muhammed Kocabas, Salih Karagoz, Emre Akbas

arXiv:1903.02330v2cs.CV

TL;DR

Limited and costly 3D ground-truth data, along with extra supervision or camera parameters required by prior methods, motivate a different approach. EpipolarPose uses multi-view 2D poses and epipolar geometry to create 3D supervision without 3D ground truth or camera extrinsics, achieving state-of-the-art results among weakly/self-supervised methods on Human3.6M and MPI-INF-3DHP while introducing PSS for structural evaluation.

  • Problem

    3D pose estimation lacks abundant diverse ground-truth data, while prior weakly/self-supervised methods often require additional labels or camera extrinsics.

  • Method

    EpipolarPose estimates 2D poses from multi-view images and uses epipolar geometry to create 3D supervision for training a single-view 3D pose estimator without 3D ground truth or camera extrinsics.

  • Results

    EpipolarPose achieves state-of-the-art results on Human3.6M and MPI-INF-3DHP among weakly/self-supervised methods.

  • Takeaways & Limitations

    The method shows that multi-view images can provide self-supervision for 3D pose estimation without 3D ground truth or camera extrinsics, while PSS evaluates structural plausibility alongside localization metrics.

  • Takeaways & Limitations

    Related multi-view consistency methods may require a small amount of 3D ground-truth data to avoid degenerate pose solutions.

Abstract

from arXiv · show

Training accurate 3D human pose estimators requires large amount of 3D ground-truth data which is costly to collect. Various weakly or self supervised pose estimation methods have been proposed due to lack of 3D data. Nevertheless, these methods, in addition to 2D ground-truth poses, require either additional supervision in various forms (e.g. unpaired 3D ground truth data, a small subset of labels) or the camera parameters in multiview settings. To address these problems, we present EpipolarPose, a self-supervised learning method for 3D human pose estimation, which does not need any 3D ground-truth data or camera extrinsics. During training, EpipolarPose estimates 2D poses from multi-view images, and then, utilizes epipolar geometry to obtain a 3D pose and camera geometry which are subsequently used to train a 3D pose estimator. We demonstrate the effectiveness of our approach on standard benchmark datasets i.e. Human3.6M and MPI-INF-3DHP where we set the new state-of-the-art among weakly/self-supervised methods. Furthermore, we propose a new performance measure Pose Structure Score (PSS) which is a scale invariant, structure aware measure to evaluate the structural plausibility of a pose with respect to its ground truth. Code and pretrained models are available at https://github.com/mkocabas/EpipolarPose

1. Introduction

EpipolarPose addresses the scarcity and supervision requirements of 3D pose estimation by using multi-view 2D poses and epipolar geometry to create 3D supervision without 3D ground truth or camera extrinsics. The paper also introduces PSS to evaluate structural plausibility beyond joint-wise localization.

  • 3D pose datasets are costly to collect and often limited to laboratory settings, size, or diversity.
  • Existing weakly or self-supervised methods commonly require extra 3D labels, subsets of labels, or camera extrinsics beyond 2D ground-truth poses.
  • EpipolarPose uses multi-view 2D pose estimation and epipolar geometry to obtain 3D poses for training a 3D pose estimator without 3D supervision or camera extrinsics.
  • EpipolarPose sets the new state-of-the-art among weakly/self-supervised methods for 3D human pose estimation.
  • PSS is a scale invariant, structure-aware performance measure designed to capture structural pose errors that MPJPE and PCK may miss.
  • PSS assigns a score of 1 when predicted and ground-truth poses belong to the same learned pose cluster, and 0 otherwise.

2. Related Work

Prior 3D human-pose methods use single-view lifting, multi-view reconstruction, or weak/self-supervision, but commonly require 3D supervision, camera calibration, or other auxiliary supervision. EpipolarPose instead combines image features with epipolar geometry without 3D supervision or camera extrinsics.

  • Single-view methods: Single-view methods estimate 3D joints directly from images or lift estimated 2D poses into 3D space.Two-stage approaches commonly use state-of-the-art 2D pose estimators before 3D lifting.
  • Single-view methods: These single-view approaches require full 3D supervision or extra supervision such as ordinal depth.
  • Multi-view methods: Multi-view methods use multiple views during training and testing, often relying on calibrated cameras for triangulation or full 3D supervision.
  • Weakly/self-supervised methods: Weakly/self-supervised methods reduce reliance on 3D annotations but may require full camera calibration, 2D heatmaps, or a small amount of 3D ground truth.Rhodin et al. use multi-view consistency constraints but require 3D labels to avoid degenerate pose solutions.
  • Weakly/self-supervised methods: EpipolarPose uses image features and epipolar geometry, requiring neither 3D supervision nor camera extrinsics.The method is described as multi-view during training and monocular during inference.
  • Weakly/self-supervised methods: EpipolarPose reports 4.3 mm less error than Drover et al.'s method, another approach requiring neither 3D supervision nor camera extrinsics.

3. Models and Methods

EpipolarPose trains a single-view 3D pose estimator from synchronized multi-view images without requiring 3D supervision or camera extrinsics. It estimates 2D poses, recovers geometry through epipolar constraints and triangulation, and introduces PSS to assess structural plausibility.

  • EpipolarPose training: Training uses synchronized images from at least two cameras, with one trainable upper branch and one frozen 2D pose-estimation branch.The frozen branch produces 2D poses while the upper branch is optimized for 3D estimation.
  • EpipolarPose training: Volumetric heatmaps yield 3D poses through soft argmax over all dimensions and 2D poses through soft argmax over spatial dimensions.The heatmaps have spatial dimensions w and h and depth resolution d; J denotes the number of body joints.
  • Camera geometry and triangulation: Without camera extrinsics, corresponding 2D joints estimate the fundamental matrix, from which essential-matrix decomposition recovers camera geometry.RANSAC identifies the fundamental matrix, and the first camera is used as the coordinate-system center.
  • Camera geometry and triangulation: Triangulation converts synchronized corresponding 2D joints into 3D points, while multiple-camera settings use a vector median for the 3D position.Only joints unoccluded in both images are triangulated.
  • Training objective: The 3D branch minimizes smoothL1 error after projecting the triangulated pose into the corresponding camera frame.The projected triangulated pose provides the training target for the predicted camera-frame 3D pose.
  • Inference and refinement: Inference takes a single image and outputs a 3D pose from a volumetric heatmap, with an optional refinement unit that maps noisy detections to more reliable predictions.The refinement unit can be trained on unlabeled multiview footage and uses residual connections, normalization, leaky ReLU, dropout, and intermediate loss.
  • Pose Structure Score: PSS is a scale-invariant structural score that compares whether predicted and ground-truth poses belong to the same pose cluster.It complements MPJPE and PCK, which treat joints independently, and mPSS reports the percentage of structurally correct poses.

4. Experiments

Experiments evaluate EpipolarPose on Human3.6M and MPI-INF-3DHP using standard pose metrics and comparisons with supervised and weakly/self-supervised methods. Results show that triangulation quality, 2D keypoint quality, and refinement affect performance, while the self-supervised model achieves strong benchmark results without 3D ground-truth examples.

  • Datasets: Human3.6M contains 3.6 million images of 11 actors performing 15 activities across four camera views, with subjects 1, 5, 6, 7, and 8 used for training and subjects 9 and 11 for evaluation.Evaluation uses every 64th frame of the test set and reports average errors.
  • Datasets: MPI-INF-3DHP extends evaluation to indoor and outdoor scenes using five chest-height cameras and an official outdoor test set, without background augmentation.Results are reported using PCK and NPCK for consistency with prior work.
  • Triangulation: 4.3 mm MPJPE and 99% PSS result from triangulating ground-truth 2D keypoints with camera geometry on Human3.6M.The experiment identifies triangulation quality as a crucial factor in downstream performance.
  • Triangulation: An MPII-pretrained detector produces triangulations 11 mm better in MPJPE than Pavlakos et al.’s results.This comparison highlights the effect of the 2D keypoint source on triangulation accuracy.
  • Comparison to supervised methods: The self-supervised model performs well against fully 3D-supervised methods requiring abundant labeled data, achieving comparable results without 3D ground-truth examples.The authors characterize this as promising for 3D pose estimation.
  • Refinement: The refinement unit improves the self-supervised model’s performance by 20% and provides learned 3D depth cues that help 2D-to-3D lifting.The refinement unit is optional and is compared with a matching architecture for fairness.
  • Weakly/self-supervised comparison: On Human3.6M, the self-supervised model with or without refinement outperforms previous weakly/self-supervised methods by a large MPJPE margin.Training with ground-truth 2D triangulations versus MPII-pretrained keypoints differs by 21 mm, indicating that 2D keypoint quality is crucial.
  • Weakly/self-supervised comparison: The method narrows the full-supervision gap relative to Rhodin et al., with 12 and 9 mm gaps versus 15 and 18 mm for H36M and 3DHP, respectively.For subject-1 training, EpipolarPose improves by 14 mm on H36M and 4 mm on 3DHP, and yields comparable results to Rhodin et al.’s subject-1 setting.

5. Conclusion

EpipolarPose uses multi-view images and epipolar geometry to create self-supervision for a 3D pose estimator without 3D ground truth or camera extrinsics. It achieves state-of-the-art results among weakly/self-supervised methods and introduces PSS to assess structural plausibility.

  • Conclusion: EpipolarPose leverages 2D poses from multi-view images and epipolar geometry to self-supervise a 3D pose estimator without 3D ground-truth data or camera extrinsics.The approach uses multi-view information for training while targeting 3D pose estimation.
  • Conclusion: EpipolarPose achieves state-of-the-art results on Human3.6M and MPI-INF-3DHP among weakly/self-supervised methods and introduces PSS for structural plausibility evaluation.PSS complements localization-based metrics by evaluating pose structure relative to ground truth.
Loading 1903.02330v2…