Source-linked AI summary
Unsupervised Learning of Shape and Pose with Differentiable Point Clouds
Eldar Insafutdinov, Alexey Dosovitskiy
TL;DR
The paper addresses learning 3D shape and camera pose from unlabeled 2D projections without ground-truth poses. It uses differentiable point clouds and distilled pose ensembles, successfully learning both with only a minor drop relative to pose-supervised training while predicting high-fidelity shapes.
Problem
Learning 3D shape and camera pose from 2D projections without ground-truth camera poses is challenging because shape and pose depend on each other and pose ambiguity creates local minima.
Method
The method jointly predicts point-cloud shapes and camera poses using reprojection loss, differentiable point-cloud rendering, and a diverse pose-predictor ensemble distilled into one student model.
Results
The method successfully predicts shape and pose with only a minor performance drop relative to ground-truth-pose training, while its best 128-resolution model decreases mean error by 30% versus the best baseline.
Takeaways & Limitations
Point clouds support high-fidelity shape learning from sufficiently high-resolution images and enable semantic correspondences between predicted objects.
Takeaways & Limitations
The projection method requires an explicit volume for occlusion reasoning, and extending learning to color images or videos would require explicit lighting reasoning.
Abstract
from arXiv · showhide
We address the problem of learning accurate 3D shape and camera pose from a collection of unlabeled category-specific images. We train a convolutional network to predict both the shape and the pose from a single image by minimizing the reprojection error: given several views of an object, the projections of the predicted shapes to the predicted camera poses should match the provided views. To deal with pose ambiguity, we introduce an ensemble of pose predictors which we then distill to a single "student" model. To allow for efficient learning of high-fidelity shapes, we represent the shapes by point clouds and devise a formulation allowing for differentiable projection of these. Our experiments show that the distilled ensemble of pose predictors learns to estimate the pose accurately, while the point cloud representation allows to predict detailed shape models. The supplementary video can be found at https://www.youtube.com/watch?v=LuIGovKeo60
1 Introduction
The paper learns high-fidelity 3D shapes and camera poses from 2D projections without ground-truth camera poses. It addresses pose ambiguity with an ensemble distilled to one model and uses differentiable point clouds for detailed shape learning.
- Motivation: Existing reprojection-based methods assume precisely known camera poses and typically predict low-resolution voxel volumes.These restrictions are described as practically and biologically unrealistic pose supervision and insufficient shape detail.
- Challenges: Jointly estimating shape and pose creates a chicken-and-egg problem because each prediction is needed to learn the other accurately.The paper reports that good shape estimates can nevertheless be learned with relatively noisy pose predictions.
- Pose ambiguity: Pose ambiguity can cause local minima in which the network predicts only one visually similar viewpoint, producing drastic pose errors.The proposed remedy is a diverse ensemble of pose predictors distilled into a single student model.
- Representation: The method represents shapes as point clouds and introduces a differentiable projection operator for learning them without explicit 3D supervision.The operator can generate silhouettes, color images, or depth maps from a point set and camera pose.
- Results: The approach predicts shape and pose from a single image and learns both using reprojection-based supervision from object views.Experiments report successful joint prediction, with only a minor performance drop relative to training with ground-truth camera poses.
2 Related Work
Prior work learns 3D structure from 2D projections using reprojection losses, while this paper combines unsupervised pose learning with a point-cloud representation. The related literature spans voxel, ray-based, generative, and alternative shape representations.
- Class-specific reconstruction: Earlier reconstruction methods use silhouettes, keypoints, or computer-vision techniques to recover class-specific deformable or volumetric shapes.The cited approaches range from small image collections to large-scale Pascal VOC data.
- Reprojection-based learning: Deep-learning methods commonly compare projections of predicted 3D shapes with ground-truth 2D projections to provide training signal.Examples learn from silhouettes, depth maps, color images, or semantic segmentation maps.
- Unsupervised pose learning: A concurrent unsupervised-pose method is voxel-based and uses reinforcement learning plus a GAN prior to handle pose-learning complications.In contrast, this paper uses point clouds, an ensemble for pose prediction, and no camera-pose prior.
- 3D representations: Voxel grids are computationally and memory intensive, motivating alternatives including octrees, meshes, multi-view depth maps, skeletons, and point clouds.The paper selects point clouds as less overcomplete than voxels while remaining more flexible than mesh- or skeleton-based representations.
3 Single-view Shape and Pose Estimation
The model predicts a point-cloud shape from one image and a camera pose from another, then trains through differentiable reprojection error. An ensemble of pose regressors addresses ambiguity, after which a student regressor is used at test time.
- Model overview: The network predicts a point cloud from one view and the camera pose from another view of the same object.Shape and pose predictors are convolutional networks that share most parameters and output fixed-length vectors.
- Reprojection training: A differentiable renderer projects the predicted point cloud from the predicted pose, and mean-squared error against the ground-truth view supplies the training loss.The procedure requires renderings for view pairs of the same object to match the provided views.
- Pose ambiguity: A single pose regressor can collapse to predicting all objects from the front or back because silhouettes may look similar across viewpoints.This local-minimum behavior causes inaccurate pose estimates.
- Ensemble training: The method replaces one pose regressor with an ensemble whose predictors specialize over subsets of possible poses.The ensemble is trained with a hindsight loss, selecting the pose candidate whose projection best matches the ground truth for the weight update.
- Distillation: The ensemble is distilled to a single regressor using its best model as teacher, and the distilled regressor estimates pose at test time.The student is trained with angular quaternion-rotation difference because standard MSE performs poorly for rotation regression.
4 Differentiable Point Clouds
Differentiable Point Clouds represent shapes as smooth, parameterized point sets and render them into views through camera transformation, discretization, occlusion reasoning, and projection. The formulation supports silhouettes, depth maps, and associated signals such as color while remaining differentiable.
- The renderer maps a point cloud and camera pose to a view, optionally projecting signals such as color attached to the points.
- Each point stores a 3D position, a size parameter describing its distribution, and an associated signal such as RGB color.Size parameters may be manually specified or learned jointly with point positions.
- Points are transformed using camera intrinsics and extrinsics, represented by a coordinate change that enables orthogonal projection along the third axis.
- Smooth per-point functions are summed into a clipped occupancy function and discretized on a D1×D2×D3 grid, with depth ordered along the third index.
- Differentiable ray tracing converts occupancies into termination probabilities so occluded points do not interfere with foreground signals before projection.
- The projected signal determines the output modality: silhouettes use occupancy, depth maps use normalized depth indices, and point-associated signals can produce color views.
- The basic point-to-volume implementation costs O(NV), while a second implementation is designed to scale more efficiently.Here N is the number of points and V is the volume of the grid.
5 Experiments
Experiments evaluate shape and pose prediction, colored point clouds, computational efficiency, and semantic correspondences. The point-cloud formulation improves shape fidelity and efficiency, while the pose ensemble improves joint shape-and-pose prediction.
- Experimental setup: Experiments use ShapeNet chairs, cars, and airplanes, with five rendered views per object and random camera azimuth and elevation.Camera azimuth is sampled from [0°, 360°) and elevation from [−20°, 40°].
- Experimental setup: Chamfer distance measures shape similarity, while pose evaluation uses accuracy within 30° and median angular error.Before evaluation, learned and dataset canonical poses are aligned using ICP on 20 validation models.
- Computational efficiency: The point-cloud method is more efficient at higher resolutions: training takes roughly 1 day at 64 resolution and 3 days at 128, while the voxel method takes 2.5 days at 64 and exceeds 12GB memory at 128.At resolution 32, training times are roughly comparable.
- Shape prediction with known pose: With known camera pose, the point-cloud method outperforms its voxel counterpart and benefits more from higher-resolution training samples; its best model reduces mean error by 30% versus the best baseline.The method also achieves a 50% improvement over EPCG, despite EPCG using depth supervision, while silhouette-trained EPCG has average error 8.20.
- Colored point clouds: Color supervision produces correctly colored point clouds despite lighting and shading variation, although complex textures can become blurred.The general color is preserved well, while fine details may be lost in some colored reconstructions.
- Shape representation and correspondences: Point clouds preserve fine details such as thin chair legs, and their fully connected outputs provide natural point-index correspondences across predicted instances.The correspondence hypothesis is empirically tested by transferring indexed point subsets selected from two chair templates to other instances.
- Shape and pose prediction: The full pose-ensemble model is 28% more accurate than MVC in shape prediction and only 10% less accurate than training with ground-truth pose.The naive model is 7% worse than MVC for shape despite poor pose prediction, while the full model improves pose and shape together.
6 Conclusion
The paper presents pose and shape learning from 2D projections using point clouds, and identifies limitations and future directions for projection, data, architectures, and optimization.
- The method learns 3D object pose and shape from 2D projections using a point cloud representation.
- The projection method requires an explicit volume for occlusion reasoning, although the authors view this as potentially relaxable with custom rendering.
- Applying the method to real-world data could use its lack of accurate ground-truth pose requirements, but color images and video require reasoning about lighting, shading, and backgrounds.
- More advanced point-cloud decoder architectures may improve the method’s efficiency and accuracy.
A.1 Network architecture
The network uses a convolutional encoder followed by shared fully connected layers and separate shape and pose branches, while point projection has basic and faster implementations with different efficiency trade-offs.
- Network architecture: A seven-layer convolutional encoder feeds two 1024-unit fully connected layers before separating into shape and pose prediction branches.
- Differentiable projection: The basic differentiable point-cloud implementation constructs an N×D1×D2×D3×3 coordinate tensor and computes a Gaussian for each point.
- Differentiable projection: The basic implementation permits independently changing point sizes, but its memory and computation requirements scale linearly with the number of points.
- Differentiable projection: A faster implementation composes trilinear interpolation with a convolution, reducing requirements while using the same kernel for all points.
A.3 Quantitative evaluation
The quantitative evaluation describes preprocessing for point-cloud comparisons, quaternion-based pose differences, and the metrics reported in the ablation table.
- Shape evaluation: Ground-truth meshes are converted using vertex densification, while voxel outputs use marching cubes followed by sampling roughly 10000 surface points.
- Pose evaluation: Pose evaluation measures angular difference between rotations represented by quaternions q1 and q2.
- Metrics: Table 3 reports Chamfer distance between normalized point clouds, multiplied by 100, together with precision and coverage.
B.1 Ablation study
The ablations examine point-cloud components, flexible Gaussian shapes, projection error across point counts, semantic correspondences, qualitative outputs, and latent-space interpolation.
- Ablation study: The full ablation method uses 8000 points, point dropout, a sigma schedule, and learned point scale; the evaluated techniques are useful, but performance is generally not very sensitive to them.
- Towards part-based models: More flexible per-point Gaussian shapes can represent whole object parts, such as a chair leg, with a single elongated Gaussian.
- Towards part-based models: Learned full covariance Gaussians outperform learned isotropic covariance, while training full covariance is computationally heavier in this implementation.
- Interpolation of shapes in the latent space: Linear interpolation in the shared latent space produces gradual shape transitions, while some five-legged-chair failures reflect limited dataset representation.
- Towards part-based models: Full learnable covariance can require roughly an order of magnitude fewer points than hand-tuned isotropic covariance to reach the same projection quality.