Source-linked AI summary
Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction
Shubham Tulsiani, Alexei A. Efros, Jitendra Malik
TL;DR
Single-view 3D prediction has required direct shape supervision, while earlier multi-view consistency approaches still required pose supervision. This paper uses consistency between views from unknown poses to learn shape and pose prediction, achieving competitive performance against methods with stronger supervision.
Problem
Single-view 3D prediction has relied on full 3D supervision, while earlier multi-view consistency methods still required pose supervision.
Source: §1 · p. 1Method
The framework uses multi-view observations from unknown poses, enforcing consistency between independently predicted shape and pose and the corresponding novel-view observation.
Source: §1 · p. 2Results
The approach learns shape and pose prediction without direct supervision and achieves competitive performance on both tasks against methods using stronger supervision.
Source: §4.1.2 · pp. 7–8Takeaways & Limitations
The results indicate the feasibility of using natural forms of supervision for single-view 3D structure prediction.
Source: §5 · p. 8The framework relies on consistency with validation images of unoccluded objects and does not address unknown occlusions.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We present a framework for learning single-view shape and pose prediction without using direct supervision for either. Our approach allows leveraging multi-view observations from unknown poses as supervisory signal during training. Our proposed training setup enforces geometric consistency between the independently predicted shape and pose from two views of the same instance. We consequently learn to predict shape in an emergent canonical (view-agnostic) frame along with a corresponding pose predictor. We show empirical and qualitative results using the ShapeNet dataset and observe encouragingly competitive performance to previous techniques which rely on stronger forms of supervision. We also demonstrate the applicability of our framework in a realistic setting which is beyond the scope of existing techniques: using a training dataset comprised of online product images where the underlying shape and pose are unknown.
1. Introduction
The paper learns single-view shape and pose prediction from multi-view observations with unknown poses, without direct supervision for either. It does so by enforcing consistency between predictions from paired views and an observation from a novel view.
1. Introduction
- Humans use observations of objects from different perspectives as ecological supervision, while traditional multi-view reconstruction methods work on specific instances rather than predicting novel-instance shape from one view.The introduction motivates learning from multiple views as a more natural source of supervision.Source: §1 · p. 1
- Earlier learning-based single-view 3D methods [16] require known 3D shape for each training image, making their supervision tedious to acquire and difficult to scale.The paper presents multi-view supervision as an alternative to this requirement.Source: §1 · p. 1
- Recent methods using multi-view consistency to learn 3D shape prediction [25] [30] [35] still require object-pose supervision during training.This paper relaxes the need for both shape and pose supervision.Source: §1 · pp. 1–2
- The framework learns single-view shape and pose prediction from multi-view observations with unknown poses, relaxing the need for direct supervision of either.At test time, the learned models infer shape and pose from a single RGB image.Source: §1 · p. 2, Figure 2 · p. 2
- It predicts shape from one image and pose independently from another, then checks whether the shape explains a depth/mask observation from the second view at that pose.The overview depicts paired training views and a depth/mask verification image.Source: §1 · p. 2, Figure 2 · p. 2
2. Related Work
Prior work learned 3D structure from multiple views or used alternative supervision, but the paper distinguishes its approach by training single-view shape and pose predictors from multi-view observations without known poses or direct shape and pose supervision.
Structure from Motion and Multi-view Instance Reconstruction
- Classical reconstruction methods infer a specific instance’s 3D structure from many views, whereas this method predicts shape and pose from one RGB image at test time.SfM methods recover geometry and camera pose; multi-view reconstruction also uses range images, masks, or color images, and may incorporate depth or semantics.Source: §2 · p. 2
Generative 3D Modeling without 3D Supervision
- Earlier generative models learned 3D shape from image annotations or silhouettes, but their expressiveness or image-conditioned inference was limited; this work discriminatively trains single-view shape and pose systems.Prior models were restricted to deformations around a mean shape, did not examine image-conditioned inference, or demonstrated 3D inference only when shapes were known a priori.Source: §2 · pp. 2–3
Multi-view Supervision for Single-view Depth Prediction
Multi-view Supervised Single-view Reconstruction
- Prior multi-view supervised reconstruction methods used masks or other observations but required known training camera poses; this work relaxes that requirement and avoids direct shape supervision.Earlier CNN methods required full 3D supervision, and Zhu et al. relied on direct shape and pose supervision during pre-training.Source: §2 · p. 3
3. Approach
The approach learns shape and pose from multi-view images by requiring independently predicted geometry and viewpoint to explain another view of the same object. This consistency provides supervision without direct shape or pose labels and yields predictions in an emergent canonical frame.
3. Approach
- Training uses sparse multi-view images of same-category instances, with depth/mask verification and identity supervision but no direct shape or pose labels.Shape is represented as voxel occupancy probabilities, and pose as translation and rotation; experiments assume known camera intrinsics.Source: §3 · p. 3
3.1. Geometric Consistency as Supervision
- Independently predicting shape and pose from different views and verifying their geometric consistency encourages correct estimates in an emergent, view-independent canonical frame.The shape prediction is checked against a different, unknown view; the pose prediction must explain that view relative to the inferred shape.Source: §3.1 · pp. 3–4
3.2. Pose-differentiable Consistency Loss
- 80 depth samples per ray and differentiable trilinear occupancy lookup make the consistency loss differentiable with respect to shape and camera pose, extending Tulsiani et al. [30].The sampled 3D coordinates depend on camera parameters, and trilinear sampling makes the queried occupancy differentiable with respect to those parameters.Source: §3.2 · p. 4
3.3. Learning
- The shape and pose predictors are jointly trained so each predicted shape explains the same instance’s verification views under their predicted poses.Shape and pose are inferred from different images, preserving the independent-prediction setup that supports the emergent canonical frame.Source: §3.1 · p. 3, §3.3 · p. 4
3.4. Overcoming Local Minima
- To address pose-prediction local minima, training adds an adversarial pose prior and samples among eight pose estimates using their predicted probabilities.The prior uses uniform azimuth from 0 to 360 degrees and elevation from −20 to 40 degrees; REINFORCE supplies gradients for the probabilities.Source: §3.4 · p. 5
4. Experiments
The experiments evaluate shape and pose prediction from multi-view observations without direct supervision on ShapeNet and online product images. The ShapeNet results are competitive with stronger-supervision approaches, while the online-image results demonstrate meaningful predictions despite noisy masks and unknown shape and pose.
4.1.1 Experimental Setup
- ShapeNet experiments use airplanes, cars, and chairs, with five training views per model and settings ranging from known translation to unknown translation and rotation.Models are split into train/validation/test fractions of (0.7, 0.1, 0.2); rendered views vary in azimuth and elevation, with random lighting.Source: §4.1.1 · p. 5
- The study compares the target jointly trained, no-pose-supervision setting with controls using ground-truth 3D shape or ground-truth pose supervision.Separate models are trained for each object category.Source: §4.1.1 · p. 5
4.1.2 Results
- Shape prediction is encouragingly close to stronger-supervision results, and pose prediction is comparable to ground-truth-pose supervision, with often lower median errors.Mask-based consistency does not recover chair concavities, while depth can; mask supervision with unknown translation also shows a drop associated with scale ambiguity.Source: Figure 3 · p. 6, Figure 4 · p. 6, Table 1 · p. 7
- Predicted shape and pose use an unconstrained arbitrary frame, so evaluation aligns predictions to the canonical ShapeNet frame by maximizing voxel overlap.The alignment is searched using eight validation images per category; the learned prediction frames vary across classes.Source: Figure 5 · p. 6, §4.1.2 · p. 7
- A correct pose prior is not required: an approximate prior encourages diverse poses and avoids the local optimum observed without a prior.The experiment uses an incorrect elevation prior of [−40, 80] instead of [−20, 40], then removes its loss midway through training; training still succeeds.Source: §4.1.2 · p. 7, Figure 6 · p. 7
4.2. Learning from Online Product Images
- On online chair-product images, the method learns meaningful 3D structure and predicts shape and pose without direct supervision, despite noisy foreground masks.The filtered dataset contains N = 282 instances with Ni = 3.65 views on average; mask-only supervision leaves concavities unresolved and can confuse poses with similar silhouettes.Source: §4.2 · p. 8, Figure 7 · p. 8
5. Discussion
The framework learns single-view 3D structure without direct shape or pose supervision, indicating the feasibility of natural forms of supervision. It relies on consistency with unoccluded validation images, while unknown occlusions and general-scene structure remain challenges for future work.
5. Discussion
- The framework learns single-view 3D structure without direct supervision for shape or pose, indicating the feasibility of natural supervisory signals.Source: §5 · p. 8
- Its supervisory signal relies on consistency with validation images of unoccluded objects, leaving unknown occlusions as a challenge.Source: §5 · p. 8
- Extending the approach to general scenes may require alternate 3D representations and object motion to handle dynamic scenes.Source: §5 · p. 8
A1. Loss Formulation
The view-consistency loss sums per-pixel ray-consistency costs over depth or foreground-mask observations, using probabilistic ray termination events to make the loss differentiable with respect to shape and pose.
A1. Loss Formulation
- Shape is represented by voxel empty-space probabilities; 80 depth samples per ray use trilinear interpolation to obtain occupancies differentiable with respect to shape and camera.Camera parameters determine sample locations, allowing the sampled occupancies to vary with pose.Source: A1. Loss Formulation · p. 10
- The loss defines each pixel’s ray-consistency cost as the expected cost of probabilistic termination events, then sums these costs across pixels.Event costs compare termination depth with observed depth, or penalize events inconsistent with the foreground-mask value.Source: A1. Loss Formulation · pp. 10–11
- Unlike differentiable ray consistency [30], the formulation samples occupancies along rays with trilinear interpolation, making the loss differentiable with respect to pose.Yan et al. [35] use a similar sampling trick but restrict it to mask verification and do not leverage it to learn pose.Source: A1. Loss Formulation · p. 11
A2. Online Product Images Dataset
The online-products dataset uses chair images with manually selected instances, noisy predicted masks, and at least two valid views per training instance. Main-text visualizations use original dataset instances excluded from training for lacking enough valid views.
A2. Online Product Images Dataset
- Training used chair images from Stanford Online Products [29], with manually selected unoccluded, untruncated instances and reasonably accurate, though noisy, predicted masks; each instance had at least two valid views.Masks were predicted using the semantic segmentation system from Chen et al. [7]. Figure 8 shows the training images with approximate masks, grouped by object.Source: A2. Online Product Images Dataset · p. 11, Figure 8 · p. 12
- Main-text visualizations use original online product instances [29] that were excluded from training because they lacked enough valid views.Source: A2. Online Product Images Dataset · p. 11