PaperlayerView exact revision on arXiv

Source-linked AI summary

C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, Andrea Vedaldi

arXiv:1909.02533v2cs.CVcs.GR

TL;DR

Reconstructing deformable objects from unconstrained monocular views is challenging because viewpoint and shape change together, while multi-camera capture requires specialized hardware. C3DPO learns a monocular reconstruction network and a canonicalization network to factor these effects using 2D keypoints. It reports better performance than strong baselines across benchmarks with large non-rigid deformations.

  • Problem

    Reconstructing deformable objects from unconstrained monocular views is challenging because viewpoint and shape change together, while multi-camera capture requires expensive specialized hardware.

    Source: §1 · p. 1

  • Method

    C3DPO jointly learns a deep monocular reconstruction network and an auxiliary canonicalization network that regularizes factorization of viewpoint changes and internal deformation from 2D keypoints.

    Source: §1 · p. 2

  • Results

    C3DPO outperforms strong baselines on Human3.6M and Up3D for large within-category deformations and on PASCAL3D+ for large across-category deformations.

    Source: §5 · p. 9

  • Takeaways & Limitations

    C3DPO provides a deep-network alternative to matrix-factorization NR-SFM for learning 3D object-category models from unconstrained monocular views with 2D keypoints.

    Source: §5 · pp. 8–9

    Consistent factorization requires that no two reconstructions in X0 differ only by a viewpoint rotation.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

We propose C3DPO, a method for extracting 3D models of deformable objects from 2D keypoint annotations in unconstrained images. We do so by learning a deep network that reconstructs a 3D object from a single view at a time, accounting for partial occlusions, and explicitly factoring the effects of viewpoint changes and object deformations. In order to achieve this factorization, we introduce a novel regularization technique. We first show that the factorization is successful if, and only if, there exists a certain canonicalization function of the reconstructed shapes. Then, we learn the canonicalization function together with the reconstruction one, which constrains the result to be consistent. We demonstrate state-of-the-art reconstruction results for methods that do not use ground-truth 3D supervision for a number of benchmarks, including Up3D and PASCAL3D+. Source code has been made available at https://github.com/facebookresearch/c3dpo_nrsfm.

1. Introduction

C3DPO addresses deformable-object reconstruction from unconstrained monocular views with 2D keypoints, avoiding the specialized multi-camera capture often used for deforming objects. It learns a reconstruction network and an auxiliary canonicalization network to factor viewpoint from internal deformation, and reports superior results to methods without ground-truth 3D supervision.

1. Introduction

  • The method reconstructs deformable objects from unconstrained monocular views and 2D keypoint annotations, rather than relying on specialized multi-camera capture.Multi-camera systems can provide instantaneous reconstructions but require expensive hardware and do not themselves model deformation.Source: §1 · p. 1
  • C3DPO uses a deep network to model object shapes and deformations, trained jointly with 3D reconstruction from 2D keypoints rather than 3D supervision.The approach is motivated by the ability of deep learning to capture richer statistical models than traditional low-rank constraints.Source: §1 · p. 2
  • C3DPO reconstructs objects in a canonical frame that registers overall rigid motion and leaves internal object motion as residual variability.A jointly learned canonicalization network maps rigid-motion-equivalent reconstructions to a single canonical reconstruction, regularizing the factorization.Source: §1 · p. 2
  • On benchmarks including Human3.6M, PASCAL3D+, and Synthetic Up3D, C3DPO was compared with traditional NR-SFM baselines and deep-learning approaches, showing superior results for methods using no ground-truth 3D information.Source: §1 · p. 2

2. Related work

Prior work addresses deformable-object reconstruction through NR-SFM, category-specific shape models, and weakly supervised 3D pose estimation, with differing data and preprocessing requirements. C3DPO instead predicts canonical shape and viewpoint from a single image using 2D keypoints, with a self-supervised factorization constraint and support for occlusions and multiple categories.

NR-SFM

  • Traditional NR-SFM methods recover viewpoint and shape from multiple frames but face an under-constrained factorization problem and missing observations.Approaches use spatial, temporal, or spatio-temporal priors, among other constraints.Source: §2 · p. 2
  • Many NR-SFM approaches scale poorly, require correspondences across views, or make reconstruction of new test samples difficult or computationally expensive.The cited discussion notes reliable reconstruction limits of a few thousand images and hundreds of keypoints.Source: §2 · p. 2

Category specific 3D shapes

  • Category-specific methods include dolphin models, PASCAL VOC reconstructions by Vicente et al. [7], PASCAL3D+ reconstruction by Kar et al. [18], and image-driven rigid-category reconstruction [27].Kanazawa et al. [16] learn shape and texture for deformable objects; the described approaches rely on an initial SFM/NR-SFM step that can fail.Source: §2 · p. 2

Weakly supervised 3D human pose estimation

  • Weakly supervised human-pose methods use multi-view, ordinal-depth, unpaired 2D–3D, or video supervision, whereas C3DPO uses inexpensive 2D keypoint labels.Methods [21] require all keypoints visible in every frame; also uses camera intrinsics and 3D ground truth for synthetic views.Source: §2 · p. 3

Related work

  • C3DPO reports superior results to strong baselines across Human3.6M, Up3D, and PASCAL3D+, spanning large deformations within and across categories.Its stated contributions also include single-image recovery from 2D keypoints, a self-supervised factorization constraint, and handling occlusions and missing observations.Source: §2 · p. 3, Figure 2 · p. 3

3. Method

C3DPO factors viewpoint and deformation by reconstructing shapes in a canonical frame, with a learned mapping from visible 2D keypoints to viewpoint and pose. A jointly trained canonicalization network regularizes that factorization, while an additional loss enforces in-plane rotation invariance.

3.1. Structure from motion

  • SFM represents 2D views as camera-view matrices multiplied by a shared 3D structure, but this factorization has a mild reconstruction ambiguity.The method’s SFM setup assumes orthographic projection and removes translation by centering visible keypoints.Source: §3.1 · p. 3

3.2. Non-rigid structure from motion

  • NR-SFM allows structure to vary by view, using pose parameters and a shared shape basis to constrain the deformations.Its factorization also has mild ambiguities.Source: §3.2 · p. 4

3.3. Monocular motion and structure estimation

  • C3DPO replaces per-view matrix factorization with a feed-forward mapping from 2D keypoints and visibility flags to pose and camera parameters.The mapping is learned with a reprojection loss averaged over visible keypoints; Figure 3 reports more stable canonical shapes with Ψ as the input pose rotates around the camera y-axis.Source: §3.3 · p. 4, Figure 3 · p. 4

3.4. Consistent factorization via canonicalization

  • A canonicalization function exists if and only if the reconstruction set contains no distinct shapes related only by rotation; C3DPO uses this property to enforce consistent viewpoint-pose factorization.The method trains Ψ alongside Φ: Ψ receives a randomly rotated reconstruction and predicts pose parameters for a shape that should match the unrotated reconstruction.Source: §3.4 · p. 5
  • The combined reprojection and canonicalization losses train Φ and Ψ in parallel so rotated reconstructions are mapped back to their unrotated shapes.The canonicalization loss encourages a consistent decomposition of viewpoint and pose.Source: §3.4 · p. 5

3.5. In-plane rotation invariance

  • For a camera rotation around the optical axis, rotating the 2D keypoints by rz ∈ SO(2) requires reconstructions Φ(Y, v) = (α, θ) and Φ(rzY, v) = (α′, θ′) to yield the same 3D structure, α = α′.The resulting combined loss is ℓ2 + ℓ3; the comparable loss ranges are combined with equal weight.Source: §3.5 · p. 5

4. Experiments

The experiments evaluate C3DPO on benchmarks spanning synthetic human bodies, rigid-category objects with intra-class variation, and human poses, using quantitative metrics and qualitative comparisons. C3DPO outperforms the reported baselines on the quantitative benchmarks, with additional qualitative comparisons on bird shapes.

4.1. Datasets

  • The three benchmarks differ in keypoint density, object type, deformation, and intra-class variation; S-Up3D uses 8,515 meshes rendered in 30 views each and evaluates reconstruction on 79 representative SMPL vertices.PASCAL3D+ includes 12 rigid object categories, while Human3.6M uses either ground-truth 2D keypoints or Stacked Hourglass predictions.Source: §4.1 · pp. 5–6

4.2. Evaluation metrics

  • The evaluation reports MPJPE and ℓ1 stress, handling depth ambiguity by centering poses or point clouds and retaining the better result across original and depth-flipped predictions.Human3.6M poses are root-centered; PASCAL3D+ and S-Up3D point clouds are mean-depth-centered before evaluation.Source: §4.2 · p. 6

4.3. Baselines

  • Baselines include EM-SfM and GbNrSfM, Pose-GAN on Human3.6M, and CMR on PASCAL3D+; the ablations remove or replace losses and omit the canonicalization network.Pose-GAN cannot handle occluded keypoints, while CMR uses segmentation masks during training; EM-SfM and GbNrSfM are run directly on test sets after convergence.Source: §4.3 · p. 7

4.4. Technical details

  • The reconstruction and canonicalization networks each use six fully connected residual layers, and skip connections prevented convergence to a rigid average shape.For PASCAL3D+, padded multiclass keypoint annotations allow one C3DPO model to train across all 12 categories; camera translation is estimated from visible points on datasets with occlusions.Source: §4.4 · pp. 7–8

4.5. Results

  • CUB-200-2011 qualitative comparisons show C3DPO can model more flexible poses than CMR, with reconstructions viewed from the same angle and an angle offset by 90 degrees.The dataset has 11,788 images across 200 bird species, annotated with 15 semantic keypoints and visibility indicators, but no 3D keypoints.Source: §4.1 · p. 6, §4.5 · p. 8, Figure 6 · p. 8

5. Conclusions

C3DPO learns monocular 3D reconstruction from unconstrained views with 2D keypoints, factoring viewpoint changes from internal object deformations. Its canonicalization network constrains this ambiguous factorization toward consistency, and the method outperforms strong baselines on several benchmarks.

5. Conclusions

  • C3DPO learns a deep network for monocular 3D reconstruction from unconstrained views with 2D keypoints, factorizing viewpoint changes and internal object deformations.The approach differs from traditional NR-SFM solutions based on matrix factorization.Source: §5 · pp. 8–9
  • An auxiliary canonicalization network constrains the ambiguous factorization to be as consistent as possible.Source: §5 · p. 9
  • C3DPO outperforms strong baselines on Human3.6M and Up3D within-category benchmarks and PASCAL3D+ across categories.The benchmarks contain large non-rigid deformations within or across categories.Source: §5 · p. 9

C3PO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

The supplementary material outlines theoretical analyses of centering and ambiguities in SFM and NR-SFM, alongside additional architecture, robustness, qualitative-result, and failure-mode discussions. Its degree-of-freedom analysis gives conditions for uniquely solving the rigid and non-rigid formulations up to intrinsic ambiguity.

C3PO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

A. Theoretical analysis

A.1. Centering

  • Centering in orthographic SFM and NR-SFM preserves the equations when observations and structures are centered.The lemma states an if-and-only-if equivalence; its proof averages and removes the left- and right-hand sides.Source: §A.1 · p. 11

A.2. Degrees of freedom and ambiguities

  • SFM and NR-SFM matrix factorizations retain intrinsic ambiguities even with arbitrarily many observed points and views.Source: §A.2 · p. 11

A.2.1 Structure from motion

  • For SFM, at least two views and a full-rank point configuration leave only a 9-DoF ambiguity; setting the first view matrix to I2×3 fixes six degrees of freedom.Unique recovery up to the residual ambiguity requires 2NK ≥ 6N + 3K − 9; for two views, K ≥ 3 keypoints.Source: §A.2.1 · p. 11

A.2.2 Non-rigid structure from motion

  • For NR-SFM with shape-basis dimension D, unique recovery up to intrinsic ambiguity requires K ≥ 3 + D/2 keypoints.Once the basis is learned, single-view reconstruction is likewise solvable when K ≥ 3 + D/2.Source: §A.2.2 · p. 11

A.3. Proof of lemma 1

The lemma characterizes when a set of shapes admits a canonicalization function: each rotated shape must map back to its canonical representative. The supplementary figures also show qualitative reconstructions and the shared Ψ and Φ network architecture.

A.3. Proof of lemma 1

  • A shape set has the transversal property if and only if a canonicalization function Ψ maps every rotated structure back to its original structure.Source: §A.3 · p. 11
  • The proof constructs Ψ by mapping rotations of each shape to that shape, with the transversal property ensuring the mapping is well defined.The accompanying argument states that equivalent rotated inputs must yield the same canonical reconstruction.Source: §A.3 · pp. 11–12
  • Figure 7 shows 2D keypoint annotations above C3DPO reconstructions of all 6890 SMPL vertices from two viewpoints.Source: Figure 7 · p. 12
  • Figure 8 depicts Ψ and Φ with a shared trunk of six fully connected residual layers and differing input and output types.Source: Figure 8 · p. 12

B. Architecture of Ψ and Φ

The Ψ and Φ networks use residual blocks; the paper reports that this design prevents flattened-shape predictions compared with a simpler fully connected alternative.

B. Architecture of Ψ and Φ

C. Analysis of robustness

C3DPO’s robustness to noisy and occluded 2D keypoints improves with more input keypoints, though clean-input performance is slightly worse at 500 or more keypoints.

C. Analysis of robustness

  • The robustness experiment was designed to test C3DPO under noise in the input 2D keypoints.Source: C. Analysis of robustness · p. 12
  • The experiment adds Gaussian noise to 2D inputs and randomly occludes each point, varying the number of input keypoints across 79, 100, 500, and 1000.Evaluation uses the representative 79 vertices of S-Up3D-test.Source: C. Analysis of robustness · p. 12
  • More input keypoints improve robustness to noise, but performance without noise is slightly worse with 500 or more keypoints.The authors hypothesize that the optimizer then focuses less on reprojection losses for the 79 keypoints used in evaluation.Source: C. Analysis of robustness · p. 12

D. Additional qualitative results

Additional qualitative results show C3DPO reconstructions trained on all 6,890 SMPL vertices of S-Up3D; two comparison methods could not be run on this dataset because of scalability issues.

D. Additional qualitative results

E. C3DPO failure modes

C3DPO failures arise from keypoint-detection errors, unseen test poses, and strongly ambiguous 2D poses.

E. C3DPO failure modes

Related papers

Loading 1909.02533v2…