Source-linked AI summary
Learning Category-Specific Mesh Reconstruction from Image Collections
Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, Jitendra Malik
TL;DR
The paper addresses single-image 3D reconstruction when category instances lack ground-truth 3D or multi-view supervision. It learns a deformable mesh, camera, and canonical texture prediction model from annotated image collections, producing diverse and meaningful shape, pose, and texture predictions while leaving articulation and novel texture synthesis as open challenges.
Problem
Single-image prediction of generic object shape and texture lacks methods that can learn from annotated image collections without ground-truth 3D or multi-view data.
Method
The framework predicts a category-specific deformable mesh, camera pose, and canonical texture flow from one image, training with masks, sparse keypoints, and smoothness priors.
Results
The learned model accurately predicts diverse bird shapes, estimates camera pose, and infers meaningful, realistic textures from category-level annotated image collections.
Takeaways & Limitations
Annotated single-view image collections can support category-level inference of shape, pose, and semantically consistent texture without requiring 3D training data.
Takeaways & Limitations
The formulation does not explicitly model articulation, texture synthesis beyond copying image pixels, or rare poses reliably.
Abstract
from arXiv · showhide
We present a learning framework for recovering the 3D shape, camera, and texture of an object from a single image. The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation. Our approach allows leveraging an annotated image collection for training, where the deformable model and the 3D prediction mechanism are learned without relying on ground-truth 3D or multi-view supervision. Our representation enables us to go beyond existing 3D prediction approaches by incorporating texture inference as prediction of an image in a canonical appearance space. Additionally, we show that semantic keypoints can be easily associated with the predicted shapes. We present qualitative and quantitative results of our approach on CUB and PASCAL3D datasets and show that we can learn to predict diverse shapes and textures across objects using only annotated image collections. The project website can be found at https://akanazawa.github.io/cmr/.
1 Introduction
The paper learns to infer a category-specific 3D mesh, camera pose, and texture from one image using only annotated image collections rather than ground-truth 3D or multi-view supervision. Its deformable mesh and canonical appearance representations support efficient prediction, texture inference, and semantic correspondence.
- Motivation: The model infers a novel object's rough 3D shape, camera pose, and appearance from a single image by learning a category-level mental model.Training uses annotated image collections with foreground masks and semantic keypoint labels, while each training instance provides only one view.
- Shape representation and inference method: A learned category-specific mean mesh is deformed with instance-specific predictions, while a predicted camera maps the canonical mesh into image coordinates.The shared 3D space implicitly supports correspondences across predicted instances.
- Contributions: The paper’s framework differs from prior approaches primarily in three aspects, although the supplied passage does not enumerate those aspects.The surrounding contributions identify learning from annotated image collections, texture inference, and the mesh-based prediction framework as central elements.
- Shape representation and inference method: The framework combines deformable meshes with learning-based prediction, providing surface-level reasoning, correspondence association, and efficient inference from an unannotated image.The mesh representation is memory-efficient, while the learned predictor enables single-image inference.
- Learning from an image collection: Unlike methods requiring ground-truth 3D or multi-view supervision, the model learns from annotated image collections with one available view per training instance.This setting is especially useful for categories where scans or multiple views of the same instance are difficult to obtain.
- Ability to infer texture: Texture inference is formulated in a canonical appearance space, addressing limitations of voxel-like representations and fitting-based deformable models for generic object textures.The approach targets a task with little prior work beyond face-specific methods that require large collections of textured 3D ground truth.
2 Approach
The framework predicts a category-specific deformable mesh, camera pose, texture, and semantic correspondences from one image, learning from annotated image collections without ground-truth 3D or multi-view supervision. It combines instance-specific evidence losses with category-level structure and canonical texture representations.
- Framework overview: A CNN predicts camera pose, shape deformation, and texture parameters from an input image through a shared latent representation.The deformation is added to a learned category-level mean shape, while texture is predicted as a UV image.
- Shape and camera representation: The mesh uses fixed spherical connectivity, with vertex locations formed by adding instance-specific deformations to learned mean vertex locations.This yields an instance-specific shape in a canonical coordinate frame.
- Shape and camera representation: Weak-perspective camera parameters include scale, 2D translation, and quaternion rotation predicted from the image.The predicted camera transforms the canonical mesh into image coordinates.
- Semantic correspondences: Semantic keypoints are associated with mesh vertices through a learned assignment matrix whose rows approximate peaked probability distributions.Keypoint locations are obtained by multiplying the assignment matrix by the predicted vertex positions.
- Learning from image collections: Training uses annotated images with foreground masks and keypoints, plus rough weak-perspective cameras from structure-from-motion, rather than ground-truth 3D or multi-view data.Instance losses enforce agreement with masks and keypoints, while priors such as smoothness encourage desired shape properties.
- Texture prediction: Texture prediction is posed in a canonical UV appearance space shared across deformed instances and generated by predicting texture flow from the input image.The flow bilinearly samples input pixels to produce the UV image, preserving image details better than direct pixel regression.
3 Experiments
The experiments evaluate single-view shape, camera, and texture prediction using annotated image collections, primarily on CUB birds and additionally on PASCAL 3D+ categories. Results show meaningful shape variation, texture transfer through canonical appearance, and competitive reconstruction accuracy.
- Experimental Setup: Experiments use annotated image collections to evaluate single-view prediction of shape, texture, and camera pose, with CUB birds as the primary example.CUB provides masks, visibility indicators, and 14 semantic keypoints; ground-truth 3D is unavailable for benchmarking.
- Qualitative Results: The model predicts plausible bird shapes, camera poses, and textures, capturing overall body variation, beaks, and large deformations such as flying poses.Reported errors include rare poses and asymmetric articulation.
- Qualitative Results: PCA of predicted deformations reveals common shape modes associated with fat or thin birds, wing opening, and tail or leg deformation.The modes are visualized by deforming the learned mean shape.
- Qualitative Results: Canonical appearance images enable transferring one instance’s predicted texture onto another instance’s predicted mesh with semantically consistent color placement.For example, colors corresponding to one bird’s body transfer onto the other bird’s body despite differing views.
- Quantitative Evaluation: On CUB, predicted deformations improve mask and keypoint reprojection over the learned mean shape, while comparable predicted- and SfM-camera results support accurate camera estimation.Full prediction reaches PCK 0.81 versus 0.80 for the mean shape at normalized distance threshold 0.1.
- Evaluation on Other Object Classes: On PASCAL 3D+, the method achieves mean IoU comparable to or better than alternate methods while additionally predicting texture.The comparison includes deformable fitting-based and volumetric prediction approaches using similar image-collection supervision.
4 Discussion
The framework is an encouraging but incomplete solution for single-view textured mesh prediction. The discussion identifies articulation, texture synthesis, multiple-view training, and reduced annotation as open directions.
- Discussion: The framework does not solve single-view textured mesh prediction in the general case.The authors describe the results as an encouraging step while identifying several remaining challenges.
- Discussion: The deformable shape model handles shape change and articulation similarly, but explicit articulation modeling may improve the representation.This is proposed as a direction for extending the model.
- Discussion: Texture synthesis currently copies image pixels, motivating mechanisms that can both copy image content and synthesize novel aspects.The authors identify this as a possible improvement to texture generation.
- Discussion: The approach learns from one view per training instance, while multiple views might yield better results and unannotated collections could further reduce supervision requirements.Both directions are presented as possibilities rather than established outcomes.
A1. Optimization Details
Optimization begins from an SfM-derived mean shape and uses mesh-based smoothness regularization. The mesh keeps fixed connectivity while learning vertex geometry.
- Mesh Geometry: Mesh geometry uses a subdivided icosphere with 642 vertices and 1280 fixed faces during learning.The fixed topology supports optimization of vertex locations without changing connectivity.
- Initialization: The initial mean shape is obtained by running SfM on annotated keypoints and is used to initialize the final learned mean shape.Figure 10 contrasts the initialization with the learned result.
- Smoothness: Laplacian smoothness minimizes mean mesh curvature through the norm of rows of LV, where L is a fixed sparse Laplace-Beltrami operator.The operator is constructed from mesh connectivity and applied to vertex locations.
A2. Additional Results and Comparisons
The paper presents qualitative predictions on test images, showing inferred shapes and textures from the predicted camera alongside novel views. Compared with directly sampled image textures, the inferred representation avoids holes and background contamination that degrade novel-view appearance.
- Randomly selected results: 40 randomly selected test images are shown with predicted 3D shape and texture from the predicted camera view.Each example also includes textured-shape renderings after rotations of 60, 180, and -60 degrees around the y-axis.
- Texture comparison: Directly sampled textures produce holes and include background pixels, causing unnatural texture in novel views.The comparison uses a symmetric texture map sampled from visible image regions after predicting the camera and mesh.
- Texture comparison: The method instead predicts texture in a canonical appearance space rather than copying visible image pixels directly.This distinction is the basis of the comparison with direct texture sampling.
- Randomly selected results: The visualization compares each input image with its predicted shape, predicted-camera rendering, and three rotated novel views.The same comparison layout is described for each column of the displayed results.