Source-linked AI summary
Visual Object Networks: Image Generation with Disentangled 3D Representation
Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman
TL;DR
Existing image generators produce realistic 2D images but do not model the underlying 3D world. VON jointly learns 3D shapes and 2D images by disentangling shape, viewpoint, and texture, producing more realistic images while enabling 3D manipulations. Its current scope is limited to individual objects because large-scale 3D geometric data for entire scenes is unavailable.
Problem
Existing generative models can synthesize photorealistic images but lack an understanding of the underlying 3D world, limiting 3D-oriented image operations.
Method
VON uses an end-to-end adversarial framework that independently models shape, viewpoint, and texture, generating 3D shapes, differentiable 2.5D projections, and textured images.
Results
VON produces more realistic images than recent 2D generative models and enables viewpoint changes, separate shape and texture editing, interpolation, and appearance transfer.
Takeaways & Limitations
The disentangled 3D representation supports image and shape synthesis from both 2D and 3D visual data collections and enables manipulations unavailable to prior 2D methods.
Takeaways & Limitations
The current method works only for individual objects because large-scale 3D geometric data for entire scenes is unavailable.
Abstract
from arXiv · showhide
Recent progress in deep generative models has led to tremendous breakthroughs in image generation. However, while existing models can synthesize photorealistic images, they lack an understanding of our underlying 3D world. We present a new generative model, Visual Object Networks (VON), synthesizing natural images of objects with a disentangled 3D representation. Inspired by classic graphics rendering pipelines, we unravel our image formation process into three conditionally independent factors---shape, viewpoint, and texture---and present an end-to-end adversarial learning framework that jointly models 3D shapes and 2D images. Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes. It then renders the object's 2.5D sketches (i.e., silhouette and depth map) from its shape under a sampled viewpoint. Finally, it learns to add realistic texture to these 2.5D sketches to generate natural images. The VON not only generates images that are more realistic than state-of-the-art 2D image synthesis methods, but also enables many 3D operations such as changing the viewpoint of a generated image, editing of shape and texture, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints.
1 Introduction
VON addresses the limits of 2D-only image generation by jointly modeling 3D shapes and 2D images through disentangled shape, viewpoint, and texture factors. Experiments show more realistic images and support multiple 3D manipulations.
- 2D-only generative models overlook the 3D nature of the world, limiting applications such as robotics, virtual reality, and gaming.
- VON jointly synthesizes 3D shapes and 2D images by decomposing image formation into conditionally independent shape, viewpoint, and texture factors.It projects generated shapes into 2.5D sketches before adding realistic texture to produce images.
- Conditional independence removes the need for paired 2D–3D data and dense 3D correspondence annotations, enabling use of separate image and shape collections.The model can synthesize objects with diverse shapes and textures.
- VON produces more realistic image samples than recent 2D deep generative models and enables viewpoint changes, separate shape or texture editing, interpolation, and appearance transfer.
2 Related Work
The related work spans adversarial 2D image synthesis, deep 3D shape generation, inverse graphics, and the proposed bridge from 3D shapes to textured 2D images.
- GAN-based research has addressed image generation, image-to-image translation, text-to-image synthesis, interactive editing, inpainting, and super-resolution.
- Deep generative models represent and synthesize 3D shapes using voxels, point clouds, octave trees, image-learned priors, and partial-scan completion.
- VON connects these directions by generating 3D shapes, projecting them into depth-and-silhouette 2.5D sketches, and adding texture through a differentiable image-formation pipeline.The model is trained end-to-end on both 2D and 3D data.
- Inverse-graphics research learns to invert graphics engines, including convolutional inverse graphics networks for images such as faces.
3 Formulation
VON models object images through conditionally independent shape, viewpoint, and texture factors, connecting synthesized 3D shapes to 2D images with differentiable projection and adversarial learning. The formulation supports unpaired 2D and 3D training data while preserving controllable object structure and appearance.
- 3 Formulation: VON samples images from independently encoded shape, viewpoint, and texture factors, using category-specific 3D shape and 2D image collections.The model assumes clean image backgrounds and requires no alignment between the 3D and 2D datasets.
- 3 Formulation: The synthesis pipeline generates voxels, projects them into viewpoint-dependent 2.5D sketches, and renders final images with sampled texture codes.At test time, the shape network, projection module, and texture network are composed into one differentiable sampling process.
- 3.1 Learning 3D Shape Priors: The 3D shape prior uses a category-specific volumetric GAN with WGAN-GP to improve shape quality and diversity over occasional 3D-GAN mode collapse.The generator maps Gaussian shape codes to voxel grids, while a volumetric discriminator distinguishes real from generated shapes; distance-function representations are also explored.
- 3.2 Generating 2.5D Sketches: Differentiable projection converts voxel occupancy into silhouette and depth by tracing camera rays and computing expected visibility and depth.The projection remains differentiable through expectation calculations and trilinear interpolation, enabling gradients to reach both shape and viewpoint inputs.
- 3.2 Generating 2.5D Sketches: Viewpoint codes represent camera elevation and azimuth and are sampled from an empirical distribution estimated by silhouette Intersection-over-Union matching.Candidate 3D models are rendered under sampled poses, and the pose with the largest overlap is selected for each input image.
- 3.3 Learning 2D Texture Priors: The texture network maps projected 2.5D sketches and texture codes to images while modeling object texture, illumination, and rendering effects.Adversarial losses operate on generated images and 2.5D sketches, while encoders and cycle-consistency losses connect the two domains.
- 3.3 Learning 2D Texture Priors: Latent-space cycle consistency encourages the texture generator to use its code for one-to-many mappings, while KL regularization makes test-time sampling possible.The losses address latent-code neglect in conditional generation and encourage encoded texture codes to remain close to a Gaussian distribution.
- 3.4 Our Full Model: The full adversarial objective jointly optimizes shape and texture generators, representation encoders, and discriminators, while silhouette masking prevents texture synthesis from changing object shape.The implementation uses the 2.5D silhouette to mask generated images and adds mask consistency for predicted depth.
4 Experiments
VON is evaluated against 2D GANs and 3D shape generation baselines using quantitative and human-perception measures, followed by demonstrations of 3D-aware manipulation applications.
- Evaluations: The evaluation compares VON with DCGAN, LSGAN, and WGAN-GP on image generation, and with 3D-GAN on shape generation.Experiments use ShapeNet shapes, Pix3D chair images, and additional crawled images; image quality is assessed with FID and human preferences.
- Evaluations: VON achieves the smallest Fréchet Inception Distance among the compared image-generation models.Table 1 compares DCGAN, LSGAN, WGAN-GP, VON on voxels, and VON using distance-function representations.
- Evaluations: 74%−85% of human responses preferred VON images over the compared 2D GAN baselines.The study sampled 200 generated-image pairs and showed each pair to five Amazon MTurk subjects.
- Evaluations: VON produces more natural 3D shapes than 3D-GAN on voxel and distance-function representations.Shape quality is evaluated using FID between generated and real shapes, with Table 3 reporting better results for VON.
- Applications: The disentangled representation enables independent viewpoint, shape, and texture changes, along with interpolation in shape and texture spaces.These applications are described as unavailable in previous 2D generative models.
- Applications: VON transfers texture inferred from a real image to newly generated shapes and viewpoints.Figure 6 demonstrates example-based texture transfer for cars and chairs.
5 Discussion
VON is a fully differentiable 3D-aware generative model that disentangles shape, viewpoint, and texture, producing images and shapes while supporting 3D manipulations. Its current scope is limited to individual objects, with higher-resolution modeling and natural-scene synthesis left for future work.
- VON disentangles image generation into shape, viewpoint, and texture under an adversarial learning framework using both 3D and 2D visual data.The model is fully differentiable and synthesizes both images and shapes.
- The disentangled representation enables 3D manipulations unavailable to prior 2D generative models.These include changing viewpoint and editing object shape and texture.
- The current method works only for individual objects because large-scale 3D geometric data for entire scenes is unavailable.Natural-scene synthesis is identified as a meaningful next step.
- Future work includes coarse-to-fine modeling for producing shapes and images at higher resolution.