Source-linked AI summary
Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
Yiyi Liao, Katja Schwarz, Lars Mescheder, Andreas Geiger
TL;DR
The paper addresses the difficulty of learning interpretable, controllable 3D scene factors from raw 2D images. It proposes an unsupervised model combining 3D representations with 2D image reasoning and demonstrates disentangled, view-consistent synthesis for simple multi-object scenes.
Problem
Existing image-based GANs make disentangling 3D properties such as camera viewpoint and object pose challenging and lack an interpretable, controllable representation.
Method
The model learns 3D primitives representing object scale, pose, and appearance through end-to-end adversarial training from images without labeled 3D primitives, segmentations, or pose annotations.
Results
The model disentangles multiple objects and provides controllability over camera viewpoint and object poses while generating locally coherent images across viewpoint changes.
Takeaways & Limitations
Jointly modeling scenes in 3D and 2D supports unsupervised 3D controllable image synthesis for simple multi-object scenes.
Takeaways & Limitations
The method occasionally generates multiple objects from a single primitive or changes object identity under very large viewpoint changes.
Abstract
from arXiv · showhide
In recent years, Generative Adversarial Networks have achieved impressive results in photorealistic image synthesis. This progress nurtures hopes that one day the classical rendering pipeline can be replaced by efficient models that are learned directly from images. However, current image synthesis models operate in the 2D domain where disentangling 3D properties such as camera viewpoint or object pose is challenging. Furthermore, they lack an interpretable and controllable representation. Our key hypothesis is that the image generation process should be modeled in 3D space as the physical world surrounding us is intrinsically three-dimensional. We define the new task of 3D controllable image synthesis and propose an approach for solving it by reasoning both in 3D space and in the 2D image domain. We demonstrate that our model is able to disentangle latent 3D factors of simple multi-object scenes in an unsupervised fashion from raw images. Compared to pure 2D baselines, it allows for synthesizing scenes that are consistent wrt. changes in viewpoint or object pose. We further evaluate various 3D representations in terms of their usefulness for this challenging task.
1. Introduction
The paper asks whether the 3D simulation pipeline, including content creation, can be learned from raw 2D images while retaining controllable and coherent 3D scene manipulation. It proposes jointly modeling generation in 3D and 2D space to disentangle meaningful scene properties.
- Motivation: 3D control is essential for coherent image synthesis when camera position or individual object poses change.The motivation includes virtual reality, simulation, gaming, and data augmentation, where changing one object should not alter another object's properties.
- Contribution: The proposed model combines a 3D generator, differentiable renderer, and 2D image synthesis model to learn content creation and rendering jointly.This formulation is intended to learn abstract 3D representations while synthesizing controllable new content from raw images.
- Motivation: Classical rendering provides full 3D control but requires expensive, time-consuming creation of photorealistic 3D assets.The paper identifies the concerted effort of many 3D artists as a practical burden.
- Problem: 2D GANs can synthesize photorealistic images from latent codes but typically entangle camera viewpoint, object pose, and object identities.Their latent dimensions do not automatically expose physically meaningful 3D properties.
- Contribution: The paper defines 3D Controllable Image Synthesis as unsupervised learning of manipulable 3D scene properties from images alone.The target properties include multiple objects' pose, shape, appearance, and the camera viewpoint.
2. Related Work
Prior work improves image decomposition, disentanglement, and view synthesis, but largely remains 2D or handles only single objects and static scenes. This paper targets generative, disentangled 3D control of multiple objects from 2D images alone.
- Image Decomposition: Image decomposition methods use sequential RNNs or iterative segmentation, but earlier approaches handle simple or binarized scenes and newer ones address occlusions.
- Image Synthesis: Many controllable image-generation methods learn interpretable features or object-level controls, but they operate on a 2D understanding of the scene.Full 3D control over object translations, rotations, and novel viewpoints remains challenging.
- 3D-Aware Image Synthesis: Differentiable-rendering work often learns discriminative 3D models, whereas this paper combines differentiable rendering with generative models in both 3D and 2D space.The paper focuses on learning from 2D images alone rather than using 3D supervision or 3D inputs.
- 3D-Aware Image Synthesis: Implicit methods rotate a global 3D latent feature vector, while explicit methods use feature volumes that can be projected differentiably into 2D image space.DeepVoxels requires hundreds of images of the same object and is not generative; HoloGAN and PLATONICGAN are generative single-object methods.
- Image Decomposition: Existing decomposition approaches are limited to single objects or static scenes, restricting control to object-centric rotations or camera viewpoint changes.The paper contrasts this with manipulating multiple objects individually.
3. Method
The model jointly reasons in 3D and 2D by generating abstract object and background primitives, projecting them into image-space features, and refining and compositing them into a final image. It uses disentangled primitive attributes and conditional adversarial training to support object-level control and foreground-background separation.
- Overview: The model generates abstract 3D object and background representations, projects them into 2D, and uses a 2D generator to synthesize and composite the final image.The 3D representations are learned from unlabeled images, without primitive-level supervision.
- 3D Representations: Each foreground primitive contains scale, rotation, translation, and appearance attributes, while a separate background primitive models the scene background.The primitive set is intended to disentangle foreground objects from the background and expose controllable geometric properties.
- 3D Representations: The evaluated representations include point clouds, cuboids, and spheres, whose geometry is transformed by scaling, rotation, and translation.Point clouds use sparse 3D points with latent features, whereas cuboids and spheres attach texture maps to their surfaces.
- 3D Representations: The background is represented as a spherical environment map so generated scenes can accommodate changes in camera viewpoint.The background feature map is attached to the inside of a large sphere centered at the origin.
- 3D Generator: An implicit 3D generator maps Gaussian noise to jointly generated primitive attributes through shared early layers and separate output heads.The generated primitives include N foreground primitives and one background primitive.
- Projection and 2D Generation: Differentiable projection produces per-primitive feature, alpha, and depth maps, which shared convolutional networks refine into object renderings before alpha composition.Mesh primitives use a differentiable mesh renderer, while point clouds use Gaussian smoothing and an auxiliary projected cuboid for alpha and depth.
- Training Objectives: The adversarial objective can compare either full composite images or background images, using two datasets to encourage foreground-background disentanglement.The condition c selects full-image or background-only discrimination.
4. Experiments
Experiments evaluate 3D controllable image generation on synthetic and real data, then analyze representations, disentanglement, viewpoint consistency, and failure cases. The model achieves competitive image quality while supporting object transformations and more consistent 3D manipulations than 2D baselines.
- Experimental setup: The evaluation uses synthetic car and indoor datasets, a real fruit dataset, several baselines, FID-based metrics, and ablations of representations and architecture components.The synthetic datasets vary in difficulty, including multi-object cars and indoor scenes with chairs, tables, and sofas.
- Controllable image generation: Competitive FID scores on car and indoor datasets are achieved relative to Layout2Im, despite the proposed method requiring no 2D bounding-box supervision.The comparison highlights the difference between competitive image quality and the supervision requirements of the methods.
- Controllable image generation: FID scores remain relatively stable under random rotations and translations, while unpaired background supervision enables foreground-background disentanglement in complex indoor backgrounds.Without background supervision, foreground primitives can vanish on complex backgrounds, causing the background to generate the entire image and increasing FID.
- Controllable image generation: Qualitative comparisons show that the model preserves latent factors and produces consistent object rotation and translation, whereas Layout2Im can change object identities, backgrounds, and other scene elements during manipulation.The 2D baseline disentangles objects more effectively than some alternatives but struggles with correct occlusion relationships.
- Controllable image generation: On real fruit images, the model changes object arrangements while handling occlusions and shadows.The qualitative results demonstrate plausible image synthesis from real data.
- Ablation study: Different 3D representations obtain similar FID scores, suggesting that a joint 3D-2D representation matters more than the precise representation form; spheres perform best overall.The authors hypothesize that spheres outperform cuboids because they suffer less from surface discontinuities in 3D space.
- Ablation study: A single primitive can achieve low FID but cannot fully control individual objects, and its object count changes with camera viewpoint.The multi-primitive design is therefore evaluated not only by image quality but also by object-wise controllability and consistency.
- Ablation study: The proposed architecture generates locally coherent images across camera viewpoints, while geometric consistency cues enable learning depth alongside object appearance without significant performance gains.The single-primitive baseline fails to preserve object identity under viewpoint changes.
5. Conclusion
The paper presents modeling in both 3D and 2D as crucial for accurate, view-consistent results, while disentangling multiple objects and controlling camera viewpoint and object poses. It frames these results as a first step toward unsupervised 3D controllable image synthesis.
- Modeling in both 3D and 2D is crucial for accurate and view-consistent results.
- The method successfully disentangles scenes with multiple objects while controlling camera viewpoint and object poses.
- The paper presents its results as a first step toward unsupervised learning of 3D controllable image synthesis.