Source-linked AI summary
Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction
Shubham Tulsiani, Alexei A. Efros, Jitendra Malik
TL;DR
The paper addresses learning single-view 3D shape and pose without direct supervision, using multi-view observations from unknown poses. It enforces geometric consistency between independently predicted shape and pose across views, and reports competitive performance on ShapeNet while also applying the framework to online product images.
Problem
Existing single-view 3D methods rely on direct shape or pose supervision, motivating learning from more naturally available multi-view observations.
Method
The framework independently predicts shape and pose from different views of the same instance and enforces consistency with a depth or mask observation from the predicted viewpoint.
Results
The approach learns single-view shape and pose without direct supervision and achieves competitive performance relative to methods using stronger supervision.
Takeaways & Limitations
Multi-view observations from unknown poses can supervise single-view shape and pose prediction without requiring direct labels for either.
Takeaways & Limitations
The supervisory signal assumes validation images of unoccluded objects, leaving unknown occlusions as an unresolved challenge.
Abstract
from arXiv · showhide
We present a framework for learning single-view shape and pose prediction without using direct supervision for either. Our approach allows leveraging multi-view observations from unknown poses as supervisory signal during training. Our proposed training setup enforces geometric consistency between the independently predicted shape and pose from two views of the same instance. We consequently learn to predict shape in an emergent canonical (view-agnostic) frame along with a corresponding pose predictor. We show empirical and qualitative results using the ShapeNet dataset and observe encouragingly competitive performance to previous techniques which rely on stronger forms of supervision. We also demonstrate the applicability of our framework in a realistic setting which is beyond the scope of existing techniques: using a training dataset comprised of online product images where the underlying shape and pose are unknown.
1. Introduction
The paper asks whether single-view 3D shape and pose prediction can be learned from naturally available multi-view observations rather than direct shape or pose supervision. It proposes enforcing geometric consistency between independently predicted shape and pose across views.
- Classical multi-view reconstruction uses multiple views to recover instance-specific geometry but does not generalize to predicting a novel instance’s shape from one view.
- Single-view 3D inference requires learned knowledge of object geometry, but direct 3D supervision is costly, difficult to scale, and ecologically implausible.
- The learned models infer shape and pose from a single RGB input image at test time.
- The framework uses multi-view observations from unknown poses to learn both single-view shape and pose prediction without direct supervision for either.
- Training predicts shape from one image and pose from another image of the same instance, then enforces consistency with a depth or mask observation from the predicted viewpoint.
2. Related Work
Prior work reconstructs specific instances from many views, learns generative models from image observations, or trains single-view systems with direct supervision. The paper instead discriminatively learns single-view shape and pose estimation from multi-view observations.
- Structure-from-motion and related multi-view reconstruction methods recover instance-specific geometry and camera poses from many views of the same object or scene.
- Classical reconstruction methods cannot infer shape and pose from a single RGB image at test time, unlike the proposed method.
- Generative 3D modeling demonstrated that expressive models can be learned from image-based or multi-view observations, but single-view reconstruction applications were not conclusively established.
- Learning-based single-view depth methods use multi-view observations for supervision, with some approaches relaxing the assumption that relative pose is known.
- Initial CNN-based single-view reconstruction methods predicted voxel occupancy but required full 3D supervision during training.
3. Approach
The framework learns shape and pose predictors from sparse multi-view observations without direct shape or pose supervision. It enforces geometric consistency between independently predicted shape and pose, producing shape in an emergent canonical frame and enabling single-view inference after training.
- Problem Setup: Training uses multiple views of each object instance, with associated depth or mask observations but no direct shape or pose supervision.The dataset contains multiple instances from one category, each with a sparse set of views and depth/mask verification images.
- Geometric Consistency as Supervision: The shape predictor uses one image to infer an instance shape, while the pose predictor independently uses another image of the same instance to infer its pose.Independently obtaining the two predictions prevents either predictor from accessing the other image during inference.
- Geometric Consistency as Supervision: The method enforces that the predicted shape, rendered from the predicted pose, matches a depth or foreground-mask observation from the second view.This consistency relation supplies supervision despite unknown ground-truth shape and pose.
- Emergent Canonical Frame: Independent shape and pose prediction leads to an emergent canonical, view-independent shape frame, with predicted poses defined relative to that frame.The canonical orientation is arbitrary but shared across the learned predictors.
- Pose-differentiable Consistency Loss: The view consistency loss measures disagreement between a predicted voxel-occupancy shape viewed through a camera and a depth or mask image.The proposed formulation is differentiable with respect to both shape and camera pose, unlike the earlier formulation described in the passage.
- Pose-differentiable Consistency Loss: To make the loss pose-differentiable, the method samples 80 fixed depth points along each ray and obtains occupancies through trilinear sampling.Trilinear sampling is differentiable with respect to its arguments, allowing camera parameters to receive gradients.
- Learning: The training objective jointly minimizes consistency across shape predictions from images and verification views paired with independently predicted poses.For each instance, each shape prediction is compared with available verification images using the corresponding pose predictions.
4. Experiments
Experiments evaluate single-view shape and pose prediction under progressively weaker supervision on ShapeNet, then demonstrate predictions from online product images. The method remains competitive without direct shape or pose supervision, while revealing limitations from mask-only observations and unknown translation.
- Experimental Setup: Experiments use ShapeNet airplanes, cars, and chairs, with random 0.7/0.1/0.2 train, validation, and test splits.Training uses five images per instance from unknown views with depth/mask observations and varied azimuth, elevation, and lighting.
- Experimental Setup: The study compares 3D supervision, multi-view supervision with ground-truth pose, and multi-view training without pose supervision, with either known or unknown translation.These settings provide progressively weaker supervision and test the target scenario directly.
- Evaluation: Shape evaluation reports test-set mean IoU, while pose evaluation uses angular accuracy below 30 degrees and median angular error.Shape predictions are thresholded using an optimal validation-set threshold per CNN.
- Qualitative Results: Qualitative shape visualizations compare ground truth and supervised predictions with multi-view predictions under known and unknown rotation or translation.Meshes are rendered from a canonical pose, while rotation visualizations render ground-truth occupancies using predicted or supervised rotations.
- Pose Prior: A pose prior encourages diverse predictions and avoids local minima; an approximate prior also succeeds, whereas omitting the prior leads to a local optimum.An incorrect elevation prior was discarded midway through training, yet the network trained successfully.
- Results: Without direct shape or pose supervision, shape and pose performance is competitively close to stronger-supervision settings and degrades gracefully with less supervision.Mask supervision cannot recover chair concavities and unknown translation introduces scale ambiguities that lower empirical shape scores; online product images still yield meaningful predictions, though mask-only inputs confuse concavities and similar poses.
5. Discussion
The framework demonstrates single-view 3D prediction without direct shape or pose supervision, while identifying unresolved challenges for broader settings.
- The framework learns single-view 3D structure without direct supervision for shape or pose.
- Training relies on consistency with validation images of unoccluded objects.
- Unknown occlusions remain an unresolved challenge for the supervisory signal.
- Extending the approach to general scenes may require alternate 3D representations and handling object motion in dynamic scenes.
Appendix : Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction
The appendix identifies the paper's authors and their University of California, Berkeley affiliation.
- The paper is authored by Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik.
- The authors are affiliated with the University of California, Berkeley.
- The listed contact addresses use the eecs.berkeley.edu domain.
A1. Loss Formulation
The loss compares a predicted voxelized shape and camera with depth or foreground-mask observations through differentiable ray-based consistency costs.
- Notation: The shape is represented as voxel occupancy probabilities, while the camera uses intrinsic matrix K and extrinsics defined by translation t and rotation R.
- Per-pixel Error as Ray Consistency Cost: The view consistency loss sums differentiable per-pixel ray consistency costs over a verification image.
- Sampling Occupancies along a Ray: The method samples N = 80 points at predefined depth values along each pixel ray.
- Sampling Occupancies along a Ray: Trilinear sampling obtains occupancies at ray points and makes them differentiable with respect to both shape and camera.
- Probabilistic Ray Tracing: Probabilistic ray tracing models each ray as traveling forward or terminating at one sampled point, with an escape outcome at N + 1.
- Event Costs: Event costs compare predicted termination events with depth or foreground-mask observations, and the ray consistency cost is their expected cost.
A2. Online Product Images Dataset
The online product-image dataset uses chair images grouped by object instance, with approximate segmentation masks and manual filtering for image quality.
- The dataset uses chair images from the Stanford Online Products Dataset.
- Approximate foreground masks were obtained from a semantic segmentation system using the most likely chair label.
- Images were manually selected to retain reasonably accurate masks and unoccluded, untruncated instances.
- Training images and approximate masks are visualized with images from the same object grouped together.