Source-linked AI summary
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, Niloy Mitra
TL;DR
Existing image generative models largely disentangle individual properties without representing scenes as multiple controllable objects, while traditional graphics pipelines are costly. BlockGAN learns object-aware 3D scene representations from unlabelled 2D images, and the resulting representations disentangle objects and their properties while supporting novel scene creation through object manipulation.
Problem
Most image generative models focus on disentangling properties such as shape, pose, and appearance without considering scenes composed of multiple objects or providing control over individual objects.
Method
BlockGAN generates separate deep 3D features for background and foreground objects, combines them into scene features, and renders images while learning end-to-end from unlabelled 2D images.
Results
BlockGAN learns disentangled representations of objects and their properties, enabling object pose and identity manipulation, object addition and removal, and novel multi-object scene creation.
Takeaways & Limitations
Explicit 3D object features support object-aware scene manipulation, including changes that adapt shadows and highlights to object movement.
Takeaways & Limitations
BlockGAN assumes prior knowledge of object category and object count, and assumes object poses are uniformly distributed and independent.
Abstract
from arXiv · showhide
We present BlockGAN, an image generative model that learns object-aware 3D scene representations directly from unlabelled 2D images. Current work on scene representation learning either ignores scene background or treats the whole scene as one object. Meanwhile, work that considers scene compositionality treats scene objects only as image patches or 2D layers with alpha maps. Inspired by the computer graphics pipeline, we design BlockGAN to learn to first generate 3D features of background and foreground objects, then combine them into 3D features for the wholes cene, and finally render them into realistic images. This allows BlockGAN to reason over occlusion and interaction between objects' appearance, such as shadow and lighting, and provides control over each object's 3D pose and identity, while maintaining image realism. BlockGAN is trained end-to-end, using only unlabelled single images, without the need for 3D geometry, pose labels, object masks, or multiple views of the same scene. Our experiments show that using explicit 3D features to represent objects allows BlockGAN to learn disentangled representations both in terms of objects (foreground and background) and their properties (pose and identity).
1 Introduction
BlockGAN addresses the lack of object-aware 3D scene compositionality in image generation by learning controllable 3D object representations from unlabelled 2D images. It aims to preserve realistic object interactions while enabling object-level manipulation and scene composition.
- The computer graphics pipeline offers controllable, high-quality image generation but requires substantial labour, time, and cost.
- Most GAN-based disentanglement methods control properties such as shape, pose, and appearance without modelling scenes as multiple objects.
- 2D layer-based compositional methods limit manipulation of 3D properties and cannot adequately represent view-specific appearance.
- BlockGAN generates and combines 3D object features into 3D scene features before projection and rendering, supporting occlusion and appearance interactions.
- BlockGAN is trained end-to-end from unlabelled 2D images without multi-view images, paired images, pose labels, or 3D shapes.
- The model disentangles objects and their properties, separates objects from cluttered backgrounds, and supports adding, removing, and manipulating object features.
2 Related work
Prior work explores unsupervised disentanglement, 3D-aware synthesis, and object-aware image generation, but often relies on 2D object representations or restrictive assumptions. BlockGAN instead represents objects as composable 3D features with explicit poses.
- Unsupervised GANs provide image controllability without labels, but do not guarantee which latent factors are learned.
- 3D-aware neural synthesis improves image-generation quality and controllability through explicit 3D representations or learned renderable structures.
- Object-aware synthesis methods commonly generate layers or patches, while some natural-image methods assume object size or independent object appearance.
- BlockGAN represents objects as learned 3D features with corresponding poses and combines them into 3D scene features.
- This representation provides 3D pose control, supports realistic lighting and shadows, and permits adding objects not observed during training.
3 Method
BlockGAN constructs scenes by independently generating posed 3D object features, composing them, and learning perspective rendering into images. Its architecture supports variable object counts, camera-aware projection, and occlusion-aware image formation.
- 3.1. Learning 3D object representations.: The generator independently creates object features before transforming and combining them into unified scene features, injecting 3D compositionality.
- 3.1. Learning 3D object representations.: Each scene contains a background and one or more foreground objects represented as deep 3D feature grids generated from noise and pose parameters.Pose includes scale, rotation, and translation.
- 3.1. Learning 3D object representations.: Style-based generation maps each noise vector to AdaIN parameters, helping separate object pose from identity and improving training stability.
- 3.2. Composing 3D scenes.: The scene composer combines object features with an element-wise maximum, which is permutation-invariant and supports flexible object counts at test time.
- 3.3. Learning to render.: A projective camera transform resamples scene features from scene space into camera space using the viewing volume of a virtual perspective camera.
- 3.3. Learning to render.: The learned camera projection converts camera-space 3D features into view-specific 2D feature maps and preserves occlusion ordering.
- 3.4. Training.: Training uses an adversarial loss and, for natural images with cluttered backgrounds, an additional style discriminator loss.
4 Experiments
BlockGAN is evaluated on synthetic and natural datasets with increasing object and texture complexity. Experiments assess visual fidelity, object disentanglement, controllable manipulation, comparison with LR-GAN, and sensitivity to pose-distribution imbalance.
- Datasets: Experiments use 64×64 images from synthetic CLEVR, SYNTH-CAR, SYNTH-CHAIR, and natural REAL-CAR datasets with 1–4 foreground objects.The datasets range from simple shapes to natural images with complex texture and cluttered backgrounds.
- Qualitative results: BlockGAN disentangles foreground and background objects and supports explicit manipulation of each object’s pose and identity, including in cluttered natural images.Moving objects changes associated shadows and highlights, while background changes affect foreground appearance.
- Quantitative results: KID evaluation compares BlockGAN with WGAN-GP, LR-GAN, and HoloGAN, with BlockGAN achieving competitive or better visual fidelity.KID is computed on 10,000 generated images per model, while object disentanglement is not measured by KID.
- Scene manipulation beyond training data: At test time, BlockGAN can add, remove, stretch, split, combine, and manipulate objects while producing realistic shadows, highlights, and occlusion.These operations create scenes that were not observed during training.
- Comparison to 2D-based LR-GAN: Compared with LR-GAN, BlockGAN provides explicit object-location control, disentangles background from foreground, and supports adding foreground objects after training.LR-GAN uses 2D alpha-composited layers and changes foreground appearance when background noise changes.
- Non-uniform pose distribution: On REAL-CAR, BlockGAN struggles to learn the car’s full 360° rotation despite separating foreground and background effectively.The authors associate this difficulty with mismatch between the unknown real pose distribution and the uniform pose distribution assumed during training.
5 Discussion and Future Work
BlockGAN learns disentangled 3D scene representations from unlabelled images, supporting object- and property-level manipulation. Its scope is constrained by assumptions about object categories, counts, and pose distributions.
- BlockGAN disentangles scenes by object and by properties such as pose and identity.
- Additional 3D object features can be added at test time, enabling novel multi-object scenes even when training uses fewer or single objects.
- The method assumes prior knowledge of object categories and object counts during training.
- Training assumes object poses are uniformly distributed and independent, limiting application to scenes with varying objects and categories.
Broader Impact
BlockGAN offers controllable generation of realistic scenes and may support digital-content creation. The paper also notes that such image-editing capabilities could be abused.
- BlockGAN provides control over the identity and pose of multiple independent objects and the background when generating images.
- The approach could support tools for digital artists creating procedurally generated digital content.
- Because generated images can add, manipulate, or remove objects or people, the tool could be abused.
- At test time, BlockGAN samples new images from latent variables rather than directly accepting an image input.
A Additional results
Additional experiments compare BlockGAN with entangled 3D scene generation and test its behavior with more objects. The results show improved object-level disentanglement alongside failures under crowded-scene assumptions.
- A.1. Comparison to entangled 3D scene representation: Compared with HoloGAN, BlockGAN separates identity and pose for each object while learning scene-level lighting and shadow effects.
- A.1. Comparison to entangled 3D scene representation: HoloGAN associates pose with fixed object identity, so moving objects can incorrectly change foreground and background identities.
- A.2. Increasing the number of foreground objects: With six foreground objects, BlockGAN can generate and manipulate 3D object features, but the background generator may also produce foreground objects.
- A.2. Increasing the number of foreground objects: With six-object training data, BlockGAN sometimes generates scenes containing more or fewer than six objects.
- A.2. Increasing the number of foreground objects: The authors attribute this failure to independently sampled poses that do not capture physical interdependence between objects.
B Additional ablation studies
Ablations examine camera modeling, scene composition, style discrimination, object-count mismatch, and sample quality. Explicit perspective modeling and element-wise maximum composition support the intended disentanglement and generalization behavior.
- B.1. Learning without the perspective camera: Explicit perspective cameras provide cues that help resolve scale/depth ambiguity and disentangle position from identity.
- B.1. Learning without the perspective camera: Without perspective modeling, depth changes can alter object identity or appearance instead of cleanly translating the object.
- B.2. Scene composer: Element-wise maximum, summation, and MLP composers all combine objects into scenes, while element-wise maximum performs best by KID and generalizes to multiple objects.
- B.4. Incorrect number of objects: With mismatched object counts, changing a foreground generator behaves as expected for one object, whereas background changes can affect a foreground object when three are present.
- B.5. Comparison with other methods: Across synthetic and natural datasets, BlockGAN produces competitive or better sample quality than WGAN-GP, LR-GAN, and HoloGAN while offering explicit pose control.
- B.3. Style discriminator: Style discriminators separate foreground objects from cluttered backgrounds by classifying feature-level means and standard deviations across layers.
E Datasets
The paper uses modified and rendered synthetic datasets alongside natural car data, with controlled scene generation and documented object and camera variables.
- Synthetic datasets: CLEVR is modified with more colour and primitive-shape variety, while CLEVR scene setups render SYNTH-CARn and SYNTH-CHAIRn datasets.The synthetic scenes use a fixed grey background, jittered camera locations, random lighting, and randomly placed foreground objects.
- Dataset variables: Table 3 defines azimuth, elevation, scaling, and horizontal and depth translations, whose ranges represent uniform random distributions.These variables describe object rotation, camera elevation, object scale, and object translations relative to the global origin.
- Dataset sources: The dataset collection includes synthetic samples and a natural CAR dataset, with links provided for textured chair models and natural car data.The figure presents samples from the synthetic datasets; the supplementary material also links external model and dataset sources.
F.1. Training details.
Training uses fixed camera assumptions, dataset-specific latent dimensions and learning rates, Adam optimization, and a shared 50-epoch batch-training schedule.
- Camera model: The virtual camera uses a 35 mm focal length and 32 mm sensor size, corresponding to a 49.1-degree angle of view.The same camera setup is used for natural images.
- Sampling: Latent noise dimensions vary by dataset and role, with larger vectors assigned to foreground objects to reflect their relative visual complexity.For example, CLEVRn uses 20 background and 60 foreground dimensions, while natural REAL-CAR uses 100 and 200.
- Training: BlockGAN is trained with Adam, equal discriminator and generator learning rates, batch size 64, and 50 epochs.The generator is updated twice per discriminator update; learning rates are 0.0001 for synthetic datasets and 0.00005 for natural CARS.
- Infrastructure: All models are trained on a single GeForce RTX 2080 GPU.
- Architecture settings: The generator uses ReLU after AdaIN for synthetic datasets and LReLU after AdaIN for the natural CAR dataset.The supplementary tables specify separate foreground, background, generator, and discriminator architectures.
F.2. Network architecture.
The architecture represents objects, scenes, and cameras with explicit feature tensors and uses category-specific generators, while separate tables document the generator and discriminator components.
- Feature representations: Object and scene features each have dimensions 16 × 16 × 16 × 64, while camera features are 16 × 16 before up-convolutions to 64 × 64.Camera features use 64 channels for synthetic datasets and 256 channels for natural image datasets.
- Generator scope: The paper assumes category-specific generators because GANs empirically tend to perform better on category-specific datasets.A proposed future direction is a shared rendering layer for objects from different category-specific generators.
- Network components: Separate architectures are specified for the foreground object generator, background generator, overall generator, and discriminator.Tables 4–8 cover these components across synthetic and real datasets.