Source-linked AI summary
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields
Michael Niemeyer, Andreas Geiger
TL;DR
Existing approaches face costly, labor-intensive 3D content creation and often lack compositional control for complex scenes. GIRAFFE incorporates compositional 3D neural feature fields with neural rendering to synthesize controllable photorealistic scenes from raw, unstructured image collections without supervision, disentangling objects and enabling scene transformations.
Problem
Photorealistic, controllable image synthesis remains difficult because 3D content creation is hardware-expensive and labor-intensive, while existing 3D generative methods are mostly limited to single-object scenes and struggle with complex imagery.
Method
GIRAFFE represents individual objects as separate generative neural feature fields, composites them into scenes, and combines volume rendering with efficient 2D neural rendering while training from raw image collections.
Results
GIRAFFE disentangles objects from backgrounds and individual objects in multi-object scenes without supervision, while supporting controllable object and camera transformations and fast image synthesis.
Takeaways & Limitations
The compositional scene representation provides controllable image synthesis by separating object shape and appearance from the background and allowing objects to be translated or rotated within scenes.
Takeaways & Limitations
Disentanglement can fail under dataset biases or when assumed uniform camera-pose and object-transformation distributions mismatch their real distributions.
Abstract
from arXiv · showhide
Deep generative models allow for photorealistic image synthesis at high resolutions. But for many applications, this is not enough: content creation also needs to be controllable. While several recent works investigate how to disentangle underlying factors of variation in the data, most of them operate in 2D and hence ignore that our world is three-dimensional. Further, only few works consider the compositional nature of scenes. Our key hypothesis is that incorporating a compositional 3D scene representation into the generative model leads to more controllable image synthesis. Representing scenes as compositional generative neural feature fields allows us to disentangle one or multiple objects from the background as well as individual objects' shapes and appearances while learning from unstructured and unposed image collections without any additional supervision. Combining this scene representation with a neural rendering pipeline yields a fast and realistic image synthesis model. As evidenced by our experiments, our model is able to disentangle individual objects and allows for translating and rotating them in the scene as well as changing the camera pose.
1. Introduction
GIRAFFE addresses the need for controllable image synthesis by incorporating compositional 3D structure into a generative model trained on raw image collections. Its feature-field representation and neural rendering pipeline support object- and camera-level control while improving efficiency and realism.
- The model is trained on raw, unstructured image collections while enabling test-time control of camera pose, object poses, shapes, and appearances.The overview also describes synthesis of scenes containing more objects than appeared in the training images.
- Photorealistic image synthesis must also provide simple, consistent control over attributes such as object shape, size, and pose.Existing approaches often operate in 2D and do not model the compositional nature of scenes.
- Compositional 3D structure is intended to make image synthesis more controllable than 2D approaches that can entangle changes across objects.The comparison includes operations such as translating one object, circular translations, and adding objects at test time.
- GIRAFFE represents scenes with compositional generative neural feature fields and combines this representation with a neural rendering pipeline.The scene is volume-rendered into a feature image, which a 2D neural renderer converts into RGB output.
- The neural rendering pipeline produces fast and realistic image synthesis from a relatively low-resolution volume-rendered feature image.This design combines volume rendering with 2D neural rendering rather than directly producing high-resolution RGB images from the volume.
2. Related Work
Related work explores photorealistic synthesis, implicit 3D representations, and 3D-aware generative models. GIRAFFE differs by targeting compositional multi-object scenes from raw image collections without additional supervision.
- GAN-based methods achieve photorealistic image synthesis at resolutions of 1024^2 pixels and beyond, while many works seek unsupervised disentanglement.These approaches modify training objectives or architectures, or analyze latent spaces of pretrained models.
- Implicit neural functions and differentiable rendering support 3D reconstruction and novel-view synthesis, including NeRF-based representations for complex scenes.GIRAFFE uses a generative variant of NeRFs as part of its representation.
- 3D-aware generative models provide controllability but may use additional supervision, suffer voxel-resolution artifacts, or remain limited to single-object scenes.Prior methods also show reduced consistency for higher-resolution, complex, and realistic imagery.
- GIRAFFE incorporates compositional 3D scene structure to handle multi-object scenes while training on raw image collections without additional supervision.It also integrates a neural rendering pipeline for the target synthesis setting.
3. Method
GIRAFFE builds controllable scenes from compositional neural feature fields, combining object-level representations, volume rendering, and 2D neural rendering. It trains from raw image collections by sampling scene entities, latent codes, camera poses, and object transformations under an adversarial objective.
- 3.1. Objects as Neural Feature Fields: Neural feature fields map 3D points and viewing directions to density and feature vectors, using positional encoding to represent complex signals.The representation introduces an inductive bias toward canonical 3D shape orientations.
- 3.1. Objects as Neural Feature Fields: GIRAFFE represents each object, including the background, with a separate neural feature field and affine transformation to control individual pose, shape, and appearance.Object-to-scene transformations use scale, translation, and rotation parameters.
- 3.2. Scene Compositions: GIRAFFE composes N entities by summing their densities and combining features with a density-weighted mean, preserving gradient flow to every entity with positive density.The first N−1 entities are objects and the last is the background; the number of entities may be fixed or varied across the dataset.
- 3.3. Scene Rendering: The model volume-renders low-resolution feature images, then uses a 2D convolutional neural renderer with upsampling and skip connections to produce higher-resolution RGB images.Rendering features at 16^2 and upsampling them to 64^2 or 256^2 improves rendering speed and image quality according to the authors.
- 3.4. Training: Training samples the number of entities, latent shape and appearance codes, camera poses, and object transformations from dataset-dependent distributions, then applies a non-saturating GAN objective with R1 gradient penalty.The camera and object-transformation distributions are uniform over dataset-dependent valid ranges.
4. Experiments
Experiments evaluate GIRAFFE across datasets, resolutions, baselines, controllability tests, and ablations. The results show unsupervised scene disentanglement, object-level control, generalization beyond training distributions, and competitive image quality with fast neural rendering.
- Experimental Setup: The evaluation covers single-object and more challenging real-world datasets, including CompCars, LSUN Churches, FFHQ, and multi-object Clevr variants.FID is computed from 20,000 real and fake samples; comparisons include HoloGAN variants and a ResNet-based 2D GAN.
- Disentangled Scene Generation: GIRAFFE disentangles backgrounds, individual objects, and object alpha maps without supervision, including when trained on scenes with fixed or varying object counts.The model also generates plausible backgrounds despite training images containing objects, and disentanglement emerges near the beginning of training.
- Controllable Scene Generation: GIRAFFE controls individual object rotations, 3D translations, appearances, and camera elevation while independently varying shape and appearance.Separate latent codes for shape and appearance allow appearance changes without altering object shape.
- Generalization Beyond Training Data: The compositional representation generalizes beyond training data by increasing object translation ranges and adding more objects than appeared during training.These operations are demonstrated during test-time scene generation.
- Comparison to Baseline Methods: GIRAFFE achieves similar or better FID scores than baseline methods at 64^2 and 256^2 pixels, with more consistent synthesis for complex scenes and cluttered backgrounds.It controls an object independently of the background and matches or exceeds a ResNet-based 2D GAN despite using 0.41m rather than 1.69m parameters.
- Ablation Studies: Neural rendering reduces total rendering time versus [77] from 110.1ms to 4.8ms at 64^2 pixels and from 1595.0ms to 5.9ms at 256^2 pixels.Ablations also evaluate RGB skip connections, the final activation function, and upsampling choices; axis-aligned positional encoding encourages canonical object orientations.
5. Conclusion
GIRAFFE enables controllable image synthesis by combining compositional 3D scene representations with neural rendering. It disentangles objects, backgrounds, shapes, and appearances without explicit supervision.
- GIRAFFE incorporates a compositional 3D scene representation into a generative model for controllable image synthesis.
- The representation disentangles individual objects from the background and separates each object’s shape from its appearance without explicit supervision.
- Combining the scene representation with a neural renderer yields fast and controllable image synthesis.
- The authors identify learning object-level transformation and camera-pose distributions, plus scaling to complex multi-object scenes, as future directions.