PaperlayerView exact revision on arXiv

Source-linked AI summary

Shape and Viewpoint without Keypoints

Shubham Goel, Angjoo Kanazawa, Jitendra Malik

arXiv:2007.10982v1cs.CVcs.LGeess.IV

TL;DR

In multiple-instance single-view collections, each object instance has only one view, and keypoints are tedious to label across the thousands of categories we may want to “3Dfy”. U-CMR factors an image into 3D shape, texture, and camera, maintaining multiple camera hypotheses in a camera-multiplex instead of selecting one point estimate. U-CMR obtains state-of-the-art camera prediction results and learns a reasonable shape space approaching that of a method using keypoint supervision.

  • Problem

    In multiple-instance single-view collections, each object instance has only one view, and keypoints are tedious to label across the thousands of categories we may want to “3Dfy”.

    Source: §1 · p. 2

  • Method

    U-CMR factors an image into 3D shape, texture, and camera, maintaining multiple camera hypotheses in a camera-multiplex instead of selecting one point estimate.

    Source: §1 · pp. 2–3

  • Results

    U-CMR obtains state-of-the-art camera prediction results and learns a reasonable shape space approaching that of a method using keypoint supervision.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across four categories, U-CMR recovers reasonable 3D shape and texture without viewpoint and keypoint supervision.

    Source: §1 · p. 3, §5 · p. 14

    The approach does not model articulation and struggles with significantly articulated birds, including flying birds and birds with open wings.

    Source: §4.5 · pp. 13–14

Abstract

from arXiv · show

We present a learning framework that learns to recover the 3D shape, pose and texture from a single image, trained on an image collection without any ground truth 3D shape, multi-view, camera viewpoints or keypoint supervision. We approach this highly under-constrained problem in a "analysis by synthesis" framework where the goal is to predict the likely shape, texture and camera viewpoint that could produce the image with various learned category-specific priors. Our particular contribution in this paper is a representation of the distribution over cameras, which we call "camera-multiplex". Instead of picking a point estimate, we maintain a set of camera hypotheses that are optimized during training to best explain the image given the current shape and texture. We call our approach Unsupervised Category-Specific Mesh Reconstruction (U-CMR), and present qualitative and quantitative results on CUB, Pascal 3D and new web-scraped datasets. We obtain state-of-the-art camera prediction results and show that we can learn to predict diverse shapes and textures across objects using an image collection without any keypoint annotations or 3D ground truth. Project page: https://shubham-goel.github.io/ucmr

1 Introduction

The paper targets category-specific 3D reconstruction from collections where each object has only a single view, despite the annotation burden of existing approaches. U-CMR replaces keypoint annotations with a category template and maintains multiple camera hypotheses to handle the difficult camera optimization problem.

1 Introduction

  • The task is to create a 3D model and texture from one view of each object in a collection of related but differently shaped instances.The paper contrasts this “Multiple Instance Single View” setting with classical structure from motion, which uses multiple views of one instance.Source: §1 · p. 2
  • Keypoint labels are costly to mark on every instance, limiting their practicality across the thousands of categories one might want to reconstruct.The paper notes that learning a keypoint detector also requires keypoint labels.Source: §1 · p. 2
  • U-CMR factors an image into 3D shape, texture, and camera while replacing keypoint annotation with a single 3D template shape for the category.The approach is presented in a setting similar to CMR.Source: §1 · p. 2
  • Camera optimization is difficult because viewpoint space is discontinuous and multimodal, so iterative optimization can get stuck in bad local minima.Camera-multiplex maintains multiple camera hypotheses per training instance and refines them alongside shape and texture.Source: §1 · pp. 2–3
  • U-CMR obtains state-of-the-art camera prediction results without keypoint annotations and learns a shape space approaching shapes from a method that uses keypoint supervision.The evaluation covers birds, cars, motorcycles, and scraped shoe images.Source: §1 · p. 3

2 Related Work

Prior reconstruction methods commonly rely on 3D models or multiple views, while many available object images are single-view collections. U-CMR and related methods address weaker supervision settings, with different input supervision and reconstruction outputs.

2 Related Work

  • Methods using full 3D supervision are constrained by the cost or unavailability of ground-truth shapes, while multi-view methods require multiple images, silhouettes, or depth maps of the same object.The paper distinguishes multi-view methods with known viewpoints from those with unknown viewpoints.Source: §2 · p. 3
  • Unconstrained image collections commonly provide single views of different instances rather than simultaneous views of the same instance.The paper also notes this is a natural setting for non-rigid objects whose shape may change over time.Source: §2 · p. 3
  • Table 1 contrasts training inputs with inference outputs; U-CMR uses silhouette supervision and predicts full 3D shape, camera, and texture.MeshRCNN predicts shape in camera coordinates rather than a canonical frame.Source: Table 1 · p. 4
  • CMR predicts shape, texture, and camera from single images but requires mask and keypoint annotations; U-CMR reports considerably better camera predictions than CSM under the same required supervision.Kulkarni et al. learn dense canonical surface mappings with mask supervision and a template shape, but do not predict 3D shape.Source: §2 · p. 4

3 Approach

The approach represents shape as a deformable mesh, texture as a UV image, and viewpoint with a weak-perspective camera. U-CMR learns shape and texture alongside per-image camera hypotheses, then trains an image-to-camera predictor for inference.

3.1 Preliminaries

  • A fixed-topology mesh combines a learned category mean shape with image-specific vertex deformations.The mesh topology is chosen to be homeomorphic to a sphere.Source: §3.1 · pp. 4–5
  • Texture is represented as a UV image mapped onto the fixed mesh, while each camera uses weak-perspective scale, translation, and rotation.The UV map uses a spherical projection or a distortion-minimizing unwrap with manually defined seams.Source: §3.1 · p. 5

3.2 Our Method

  • For each training image, U-CMR maintains K camera hypotheses and renders the predicted shape and texture from each camera.The per-camera reconstruction losses define a distribution over camera poses, rather than a single camera estimate.Source: Figure 2 · p. 5, §3.2 · p. 6
  • Shape and texture are updated using expected loss across cameras, while each camera is separately refined against its silhouette and image reconstruction losses.The losses include perceptual image distance; shape regularization penalizes deformation magnitude and promotes smoothness.Source: §3.2 · p. 6, Model Update · pp. 6–7
  • After shape and texture training, a feed-forward predictor learns to select the best camera for each image, enabling single-image prediction of shape, texture, and viewpoint at test time.The selected training camera minimizes silhouette and image reconstruction losses.Source: Model Update · p. 7, Approach at test time · p. 7

4 Experiments

The experiments test single-image shape, texture, and camera reconstruction across birds, cars, motorcycles, and shoes, without keypoint supervision. U-CMR avoids camera collapse more successfully than weakly supervised alternatives, produces plausible shapes across categories, and has limitations for articulated objects and texture prediction.

4.1 Experimental Detail

  • The experiments cover birds, cars, motorcycles, and shoes, using CUB-200-2011, PASCAL-3D+, and shoes scraped from Zappos; the texture head directly predicts UV pixels.The camera-multiplex is initialized with 40 hypotheses for most categories and 8 for cars and motorcycles, then pruned to 4 after 20 epochs.Source: §4 · p. 7, §4.1 · pp. 7–8

4.2 Qualitative Evaluation

  • Without camera-multiplex, CMR-nokp can fit the image silhouette with a planar bird shape that becomes evident from an alternate viewpoint.The setup has an inherent ambiguity: a planar shape and texture can explain the silhouette for an arbitrary camera pose.Source: §4.2 · p. 9, Figure 3 · p. 9
  • Despite lacking viewpoint or keypoint supervision, U-CMR predicts plausible full bird shapes and textures on unseen CUB test images, including varied bird types.The reported shapes include water birds and songbirds, with sharp long tails and some leg and beak protrusions.Source: §4.2 · p. 9
  • PCA captures variation in body, head, and tail shapes, while the learned mean bird mesh becomes thinner and develops a more prominent tail than its template.The analysis uses shapes obtained on the training set.Source: §4.2 · p. 9, Figure 4 · p. 9

4.3 Quantitative Evaluation

  • At least 15 degrees lower average rotation error than CMR-nokp and CSM, U-CMR also achieves slightly higher camera-distribution entropy than supervised CMR and nearly matches its Wasserstein distance.The camera distribution is learned without keypoint annotations.Source: Table 2 · p. 10, §4.3 · p. 11

4.4 Evaluations on other categories

  • On cars and motorcycles, U-CMR yields accurate camera poses and diverse plausible shapes; shoe shapes are reasonable but less diverse, and Pascal3D+ mean IoU is comparable to more-supervised alternatives.Reported artifacts include concave car-side indentations and tapered, pointy shoe parts, associated with weak-perspective projection and non-spatially-adaptive regularization.Source: §4.4 · p. 11

4.5 Limitations

  • U-CMR does not model articulation and is expected to struggle with strongly articulated birds; texture can also fail on unseen parts, with background colors leaking into predictions.Flying or open-wing birds are examples of shape failures, and large deformation makes camera-pose identification challenging without keypoints.Source: §4.5 · pp. 13–14

5 Conclusion

U-CMR decomposes images of deformable objects into 3D shape, texture, and camera viewpoint, using camera-multiplex to represent possible viewpoints. Across four categories, it recovers reasonable shapes and textures without viewpoint or keypoint supervision.

5 Conclusion

  • U-CMR maintains a distribution of camera hypotheses to address the under-constrained recovery of shape, texture, and viewpoint.The camera-multiplex is intended to keep optimization from getting stuck in a bad local minimum.Source: §5 · p. 14
  • Across four categories, U-CMR recovers reasonable 3D shapes and textures without viewpoint or keypoint supervision.Source: §5 · p. 14

Overview

The supplementary material collects additional visualizations, comparisons, an ablation, and architecture details.

Overview

  • The supplement adds visualizations of shape, texture, and camera-multiplex, comparisons with CMR, random test-sample results, a texture-model ablation, and architecture details.Source: Overview · p. 17

6 More Results

Additional results visualize learned texture and shape variation, camera-multiplex training dynamics, comparisons with CMR, and the texture-prediction ablation.

PCA in Texture space

  • Learned texture variation spans overall color and part-specific patterns, with eyes and beaks visible in some texture-space axes.The visualization applies PCA to predicted UV textures and renders variation on the mean bird shape.Source: PCA in Texture space · p. 17, Figure 9 · p. 17

Learnt shape space for other categories

  • Learned mean shapes differ substantially from the input templates for cars, motorcycles, and shoes, with PCA axes showing category-specific deformations.Examples include a more prominent motorcycle front tire, a rounder car rear, and a slimmer, elongated shoe.Source: Learnt shape space for other categories · p. 17, Figure 10 · p. 18

Camera-multiplex visualization over time

Qualitative comparison to CMR

  • U-CMR shapes are qualitatively similar to CMR-ours, while CMR-official captures shape and texture less accurately but has pointier beaks and feet from its keypoint loss.The comparison includes CMR-official, trained with additional keypoint losses, and CMR-ours, which uses ground-truth camera pose during training.Source: Qualitative comparison to CMR · pp. 18–19
  • On a random subset of 15 test images, U-CMR predicts shapes similar to CMR-ours when birds are not articulating too much.Source: Qualitative comparison to CMR · p. 19

Ablation on texture prediction model

7 Training details

Training uses an encoder-decoder predictor for shape and texture, followed by a separate feed-forward camera-pose predictor. Supplementary figures compare U-CMR with CMR variants on test images.

7 Training details

  • The shape and texture predictor encodes each 256×256 image with ResNet-18, then predicts vertex deformations and a 128×256 UV texture map through separate heads.The texture head uses seven ResNet blocks with intermediate bilinear upsampling.Source: §7.1 · p. 20
  • After shape and texture training with camera-multiplex, a camera head predicts scale, translation, and rotation from the shared encoder representation.The encoder is frozen while the fully connected camera head is trained.Source: §7.1 · p. 20

7.1 Architecture details

  • Supplementary test-set figures compare U-CMR and CMR variants, showing predicted-view and novel-view textured meshes.The figures include selected examples and random test subsets.Source: Figure 13 · p. 21, Figure 15 · p. 23

Related papers

Loading 2007.10982v1…