Source-linked AI summary
Self-supervised Single-view 3D Reconstruction via Semantic Consistency
Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, Jan Kautz
TL;DR
The paper tackles the ambiguity of predicting 3D shape, texture, and camera pose from a single image without conventional 3D, keypoint, multi-view, or template supervision. It enforces semantic consistency between self-supervised image parts and reconstructed meshes, and reports performance comparable to supervised category-specific methods. The approach remains sensitive to segmentation accuracy, rare camera poses, and omitted fine-grained shape details.
Problem
Single-view reconstruction of 3D shape, texture, and camera pose remains ambiguous without 3D shapes, templates, keypoints, or multi-view supervision.
Method
The model uses self-supervised co-part segmentation, semantic part invariance, canonical UV mapping, and consistency constraints to reconstruct meshes from images and silhouettes.
Results
The method performs comparably to state-of-the-art supervised category-specific reconstruction methods.
Takeaways & Limitations
The framework reconstructs objects without category-specific template meshes or semantic keypoints and supports categories lacking such labels.
Takeaways & Limitations
Performance is constrained by inaccurate part segmentation, rare training-set camera poses, and data-driven templates that omit some fine details.
Abstract
from arXiv · showhide
We learn a self-supervised, single-view 3D reconstruction model that predicts the 3D mesh shape, texture and camera pose of a target object with a collection of 2D images and silhouettes. The proposed method does not necessitate 3D supervision, manually annotated keypoints, multi-view images of an object or a prior 3D template. The key insight of our work is that objects can be represented as a collection of deformable parts, and each part is semantically coherent across different instances of the same category (e.g., wings on birds and wheels on cars). Therefore, by leveraging self-supervisedly learned part segmentation of a large collection of category-specific images, we can effectively enforce semantic consistency between the reconstructed meshes and the original images. This significantly reduces ambiguities during joint prediction of shape and camera pose of an object, along with texture. To the best of our knowledge, we are the first to try and solve the single-view reconstruction problem without a category-specific template mesh or semantic keypoints. Thus our model can easily generalize to various object categories without such labels, e.g., horses, penguins, etc. Through a variety of experiments on several categories of deformable and rigid objects, we demonstrate that our unsupervised method performs comparably if not better than existing category-specific reconstruction methods learned with supervision.
1 Introduction
The paper addresses ambiguous single-view reconstruction by representing objects through semantically consistent deformable parts. Its self-supervised model removes several conventional annotations and performs comparably to supervised category-specific methods.
- Motivation: Single-view prediction of shape, texture, and camera pose is ill-posed because different 3D explanations can match the same 2D evidence.Existing methods use 3D shapes, keypoints, shading, templates, or multiple views to resolve this ambiguity.
- Motivation: Semantic parts provide a category-level basis for associating divergent object instances and inferring their relative pose and 3D shape.The paper uses deformable-parts intuition, such as representing birds through recurring parts including wings, legs, and heads.
- Approach: The model learns semantic part segmentation from images and enforces consistency between corresponding 2D and 3D parts during reconstruction.This uses self-supervised co-part segmentation and semantic part invariance across deformed instances.
- Contributions: The reconstruction model requires no category-level template prior, annotated keypoints, camera pose, or multi-view images.It is intended to generalize to categories without well-defined keypoints, including penguins.
- Contributions: The method learns a category-level 3D shape template from scratch and uses object instances as a deformable parts model.The contribution list also reports improved self-supervised semantic co-part segmentation.
- Results: Across evaluated categories, the self-supervised method performs comparably to state-of-the-art supervised reconstruction methods.The comparison concerns methods trained with predefined templates or annotated keypoints.
2 Related Work
The paper situates its representation choice among point clouds, implicit surfaces, triangular meshes, and voxel grids. It adopts triangular meshes because alternatives have rendering or efficiency drawbacks.
- Representation choices: Point clouds and voxel grids are more compatible with deep learning architectures but suffer from differentiable-rendering or memory-efficiency limitations.The passage contrasts these representations with triangular meshes for 3D processing tasks.
- Representation choices: Triangular meshes are adopted as the paper’s 3D representation for reconstruction.The supplied passage begins the adoption rationale after comparing common 3D representations.
3D Shape Representation
Prior single-view reconstruction methods use varying supervision, while this paper relates its approach to learnable templates and self-supervised correspondence methods. A baseline comparison illustrates the effect of removing supervision.
- Single-view reconstruction: Single-view reconstruction methods have used ground-truth 3D meshes, differentiable rendering, multiple views, or other supervision levels.Ground-truth mesh supervision can require substantial annotation or restrict evaluation to synthetic data.
- Template-based methods: Earlier category-level template learning can avoid 3D supervision but still require annotated keypoints and keypoint-dependent template initialization.Related settings also restrict categories to rigid or structured objects such as cars or faces.
- Baseline comparison: In the baseline comparison, CMR without camera-pose and template-prior supervision fails to learn texture and shape, whereas the proposed model reconstructs correctly without both.The meshes are rendered in both the input view and a frontal bird view.
- Self-supervised correspondence: The paper uses self-supervised co-part segmentation to enforce semantic consistency for 3D reconstruction rather than only learning 2D correspondence.This distinguishes its use of correspondence from prior work that maps images to templates without using the mapping for reconstruction.
3 Approach
The approach reconstructs mesh shape, texture, and camera pose from single images and silhouettes by enforcing semantic consistency between self-supervised 2D parts and reconstructed 3D surfaces. An alternating procedure learns instance reconstructions, a canonical semantic UV map, and a category template without keypoints or a predefined mesh.
- Reconstruction network: The baseline encoder and three decoders jointly predict mesh deformation, texture flow, and camera pose from an input image.The reconstruction network follows the CMR architecture while operating in a self-supervised setting.
- Semantic consistency: Self-supervised SCOPS segmentation provides semantically consistent part probabilities across category instances, despite variation in part shape and size.SCOPS discovers parts using concentration and equivalence losses plus part-basis discovery.
- Semantic consistency: The method maps 2D part segmentations through predicted texture flow into UV space and then onto mesh surfaces using a predefined mapping function.Aggregated semantic UV maps address noisy segmentation and uncertain texture flow, while the mapping remains independent of 3D deformations.
- Semantic consistency: Probability-based consistency constrains rendered canonical part probabilities to match SCOPS predictions, helping resolve the camera-shape ambiguity.The constraint renders the canonical semantic UV map with the predicted camera pose and compares it with the input segmentation probability map.
- Alternating optimization: The E-step trains shape, texture, and semantic consistency objectives, while the M-step updates the canonical UV map and category template from reconstructed instances.The E-step uses silhouette overlap, perceptual image distance, probability and vertex constraints, and texture consistency; the first E-step starts from a sphere template.
- Alternating optimization: The template is updated from selected consistent predictions so it represents a category mean shape rather than an arbitrary form.The template begins as a sphere and is updated through a feed-forward procedure.
- Bidirectional improvement: Reconstructed meshes can improve self-supervised part segmentation because the learned canonical UV map reduces noise in instance-based semantic maps.Combining the canonical map with instance reconstruction and camera pose supplies supervision for SCOPS.
4 Experimental Results
The method is evaluated on rigid and non-rigid categories using qualitative reconstructions, quantitative comparisons, and module ablations. Results show reconstruction across varied instances, effective texture and pose prediction, and benefits from semantic consistency and improved part segmentation.
- Experimental settings: The method is evaluated on rigid cars and motorcycles, plus non-rigid birds, horses, zebras, cows, and penguins.Quantitative evaluation is limited mainly to birds and cars because other datasets lack ground-truth keypoints, meshes, or camera poses.
- Qualitative results: The learned templates capture category shape characteristics and assign semantic parts consistently across bird, horse, motorbike, and car categories.Bird templates include head, neck, belly, and back parts aligned with SCOPS predictions.
- Qualitative results: Single-view reconstructions capture highly divergent shapes, transfer fine input textures, and render views that match the original images under predicted camera poses.Examples include thin birds, ducks, and flying birds, with details such as eyes and back textures preserved.
- Quantitative evaluation: The model achieves comparable or better bird mask reprojection accuracy than CMR despite CMR using additional semantic-keypoint supervision.Mask reprojection accuracy is measured by IoU between rendered and ground-truth silhouettes.
- Ablation studies: Removing semantic consistency substantially worsens keypoint transfer, while removing texture cycle consistency worsens texture-flow-based transfer.The unconstrained baseline can obtain better mask IoU by overfitting silhouettes, but predicts incorrect texture flow and camera views.
- Ablation studies: Using original rather than improved SCOPS reduces keypoint transfer performance by 5.3% via texture flow and 2.5% via camera pose.The improvement is measured indirectly because ground-truth part segmentation is unavailable across the datasets.
5 Conclusion
The paper concludes that self-supervised semantic consistency enables single-view reconstruction of shape, texture, and camera pose from images and silhouettes. The framework creates category-level templates and semantic UV maps while comparing favorably with supervised reconstruction methods.
- Conclusion: The improved SCOPS method yields more consistent part-segment size and shape.
- Conclusion: The framework reconstructs 3D shape, texture, and camera pose using only category-specific images and silhouettes.It enforces semantic consistency between reconstructed meshes and images to reduce joint shape-and-pose ambiguity.
- Conclusion: The method produces category-level templates and canonical semantic UV maps that capture representative shapes and semantic parts.
- Conclusion: Experimental results demonstrate efficacy compared with state-of-the-art supervised category-specific reconstruction methods.
Appendix
The appendix supplies additional implementation details, module visualizations, quantitative and qualitative results, and a discussion of failure cases and limitations. It also includes reconstruction results for bird paintings.
- Appendix contents: The appendix presents implementation details, ablations, additional quantitative and qualitative results, and failure cases and limitations.
- Additional results: Bird paintings are included among the additional reconstruction results.
Computing Category-level Template
The category-level template is updated from selected reconstructed meshes whose viewpoints are mutually consistent. This avoids averaging incompatible reconstructions, such as horses facing opposite directions, into an incorrect template.
- Ambiguity handling: Side-view horses may be reconstructed with their heads on opposite sides, making naive aggregation produce an incorrect category-level template.
- Template update: The method selects a subset of reconstructed meshes with roughly matching viewpoints before updating the template.It first chooses an instance with the most reliable reconstruction, based on rendered silhouette quality.
Computing Canonical Semantic UV Map
The canonical semantic UV map is updated using an exemplar and a similarity-filtered set of training samples to reduce the influence of outliers.
- An exemplar with the smallest perceptual distance objective is selected to avoid training samples affected by inaccurate Iflow predictions.The exemplar anchors the subsequent UV-map update.
- The set U contains the top k training samples whose semantic UV maps are most similar to the exemplar under the L2 norm.This similarity criterion filters samples before updating the canonical map.
Network Architecture
The reconstruction network uses a ResNet18 encoder and separate decoders for shape, texture, and camera pose prediction.
- The encoder is a ResNet18 with four residual blocks pretrained on ImageNet.
- The shape decoder predicts vertex deformation ∆V through one fully connected layer.
- The texture decoder uses two fully connected layers followed by eleven upsample-and-convolution layers to predict texture flow Iflow.
Smoothness Term
The method combines smoothness and edge regularization with adversarial and semantic-consistency objectives, then evaluates their effects through ablations, transfer metrics, and cross-category reconstructions. Results show that semantic consistency reduces camera-shape ambiguity, while adversarial training improves details, but inaccurate segmentation and rare poses remain failure modes.
- Smoothness Term: A graph Laplacian constraint smooths reconstructed mesh surfaces, while edge regularization penalizes irregularly sized faces.These terms supplement the objectives described elsewhere in the method.
- Adversarial Training: Adversarial training renders meshes from random camera poses and uses real images to learn shape priors that constrain reconstructions across views.The reconstruction and discriminator networks are denoted R and D.
- Ablation Studies: The basic model matches observed images and silhouettes but produces implausible unobserved views; template learning improves plausibility, while semantic consistency reduces camera-shape ambiguity.Adding semantic consistency enables more accurate camera pose and shape predictions in the reported ablation.
- Texture Flow Consistency: The texture flow consistency constraint improves keypoint transfer, especially for uniformly colored birds, and the full method performs favorably against CSM.The comparison is visualized in Figure 17.
- Semantic Consistency Ablation: Both probability-based and vertex-based semantic consistency constraints contribute to reconstruction performance on mask IoU and keypoint transfer.The ablation is reported on the CUB-200-2011 dataset.
- Results and Limitations: The method reconstructs deformable and rigid categories including birds, horses, cows, motorbikes, cars, zebras, and penguins, but suffers from inaccurate segmentation, rare camera poses, and omitted fine-grained parts.Examples include inseparable wings or legs in some categories.