BlobGAN-3D: A Spatially-Disentangled 3D-Aware Generative Model for Indoor Scenes

1. Introduction

• Existing 3D-aware methods largely target individual objects, leaving scene composition with many objects difficult to represent.Multi-object editing also requires disentangling individual objects without explicit supervision.Source: §1 · p. 1
• Prior approaches addressed object manipulation in 2D on relatively simple synthetic datasets, while GIRAFFE struggled with complex real-world multi-object scenes.BlobGAN showed object-level disentanglement on complex real-world scenes, but its 2D design does not represent objects’ 3D locations.Source: §1 · p. 2
• BlobGAN-3D extends 2D blobs to 3D and retains object disentanglement, enabling explicit camera control and realistic object-level editing.The method predicts 3D blob parameters, feature vectors, and style vectors; camera-conditioned volume rendering produces a feature grid for image synthesis.Source: §1 · p. 2
• On real-world indoor datasets, BlobGAN-3D achieves image quality comparable to 2D BlobGAN and other 3D-aware GANs while enabling camera control and object-level editing.The authors describe this as the first such combination in a challenging multi-object real-world scenario without supervised object disentanglement.Source: §1 · p. 2
• Explicitly controlling blob locations in 3D gives users more freedom to determine object occlusions.Moving blobs along the depth dimension also produces foreshortening, which the 2D BlobGAN cannot provide.Source: §1 · p. 2

2.1. Unconditional 2D GANs

• 2D GANs such as StyleGAN produce high-quality images and support control over disentangled latent spaces, but do not encode 3D content needed for novel views and some edits.StyleGAN is also used as the backbone in other image-synthesis work.Source: §2.1 · p. 2

2.2. 3D-aware GANs

• 3D-aware GANs use 3D representations to derive images; prior work emphasizes image quality and multiview consistency, while local editing methods are often limited to single objects.Some local-editing approaches support only single-object contexts.Source: §2.2 · p. 2
• BlobGAN-3D maps a latent vector to blob parameters, renders camera-ray densities into 2D density maps, and uses splatted feature and style grids to synthesize an RGB image.The feature grid is the synthesis input, and the style grid conditions the synthesis network.Source: Figure 2 · p. 3

2.3. Editability in GANs

• Editing methods span latent-code control, semantic-mask editing, and compositional object representations, but several struggle on complex real-world scenes.BlobGAN enables local edits on challenging indoor datasets but ignores 3D structure; BlobGAN-3D extends it to novel-view synthesis and more editing control.Source: §2.3 · p. 3

3. Method

• BlobGAN-3D extends BlobGAN’s per-blob representations to 3D ellipsoids in scene space, volume-renders them into a 2D feature map, and feeds that map to a StyleGAN2 synthesis network.Each blob has geometric parameters plus feature and style vectors.Source: §3 · p. 3, §3.1.1 · p. 3

3.1.2 Volume rendering

• Each blob’s density is evaluated from its distance to the blob center, with scale controlling blob size and an assumed clear depth ordering simplifying independent density-volume evaluation.Blob features remain spatially invariant to reduce the memory cost of computing spatially varying feature volumes.Source: §3.1.2 · p. 4

3.1.3 Occlusions and foreshortening effect

• 3D blob depths determine front-to-back compositing along camera rays, and perspective projection links projected object size to depth, unlike BlobGAN’s parallel projection.Depth ordering can change as the camera or blobs move in space.Source: §3.1.3 · p. 4

3.2. Image Generation

• Opacity maps are computed at a lower resolution for efficiency, then an upsampling block produces higher-resolution maps for image synthesis.The block adds a modulated 1 × 1 convolution and PixelShuffle output to bilinear upsampling; its convolution is initialized to zero.Source: §3.2 · p. 5

3.3.1 Viewpoint disentanglement in StyleGAN2

• Skip connections from the input to the first two synthesis-network blocks help multiview consistency to some extent, though the authors report room for improvement.The stated motivation is that upsampling from the low-resolution input feature grid can lose spatial detail and cause multiview inconsistency.Source: §3.3.1 · p. 5

3.3.2 Depth supervision

• An extra MSE loss compares each blob centroid’s learned depth with the depth estimator’s prediction at its projected image location.The authors say it is hard to optimize all blob depths unsupervised and that multiview rendering is unhelpful until those depths are correct.Source: §3.3.2 · p. 5
• Training first fixes the camera to a front view until texture quality is good, then corrects blob-centroid depths while rendering from randomly sampled camera poses.Source: §3.3.2 · p. 5

4.1. Settings

• Evaluation uses LSUN Bedroom and Conference Room datasets, FID and KID computed from 50K real and 50K generated images, and five GAN baselines.The baselines are BlobGAN, GIRAFFE, VolumeGAN, StyleNeRF, and EG3D; the datasets contain 3M and 200K images, respectively.Source: §4.1 · pp. 5–6
• The comparison tables characterize BlobGAN-3D as both 3D-aware and object-level disentangled, with additional 3D effects beyond BlobGAN.The table distinguishes object-level disentanglement from foreground/background-only disentanglement.Source: Table 2 · p. 6, Table 1 · p. 6

4.2. Quantitative results

• BlobGAN-3D achieves image quality comparable to BlobGAN and 3D-aware GAN baselines in the quantitative comparison.Table 3 reports image-quality results on two datasets.Source: §4.2 · p. 6, Table 3 · p. 6
• BlobGAN-3D is relatively more stable to object edits than BlobGAN, evaluated using FID after moving different numbers of blobs.The edit test moves randomly selected blobs by 0.2 in normalized horizontal coordinates.Source: §4.2 · p. 6, Table 4 · p. 6

4.3.1 Multiview rendering

• Multiview comparisons report more realistic texture and fewer artifacts for BlobGAN-3D than 3D-aware alternatives.Figure 7 presents qualitative multiview rendering comparisons with baselines.Source: §4.3.1 · pp. 6–7

4.3.2 Scene editing

• Moving, removing, and restyling blobs enable precise edits while most other objects remain similar and the edited scenes remain realistic.The experiments also show blob aspect-ratio changes and a foreshortening effect.Source: Figure 3 · p. 6, §4.3.2 · p. 7, Figure 4 · p. 7

• BlobGAN-3D extends 2D BlobGAN to 3D while retaining object disentanglement and realistic object-level editing on complex real-world indoor scenes.The conclusion presents multiview synthesis and restyling examples.Source: §5 · pp. 7–8, Figure 8 · p. 8
• The method achieves image quality comparable to BlobGAN and other 3D-aware GANs while supporting good multiview consistency.Figure 7 shows multiview synthesis results.Source: §5 · pp. 8–9, Figure 7 · p. 8
• Explicit control of blob locations in 3D enables additional editing capabilities, including a foreshortening effect.The paper also illustrates different decorating styles by swapping blob style vectors.Source: Figure 8 · p. 8, §5 · p. 9

• Figure 9 contrasts how BlobGAN and BlobGAN-3D compute hierarchical opacity maps.Source: §A.1 · p. 12
• Figure 9 also presents the architecture of the upsampler module.Source: §A.1 · p. 12
• Opacity maps are computed at 128 × 128 to balance computational complexity and performance.Source: §A.1 · p. 12

• During the third training stage, each iteration drops one randomly selected blob with probability 0.5.The stated aim is realistic synthesis when an object disappears, which may improve disentanglement.Source: §A.2 · p. 12
• For occlusion learning, forcing all blobs into descending depth order was found unhelpful because it constrained the MLP's learning ability.Source: §A.2 · p. 12
• The final design lets the network learn an arbitrary normalized depth for each blob, then sorts the depths.Source: §A.2 · p. 12

A.3. Training time and hyperparameters

• BlobGAN-3D was trained for around 200000K images per dataset on 4 NVIDIA A100 GPUs, versus 25000K images for other baselines.Blob-based rendering has a low convergence rate; the authors attribute BlobGAN and BlobGAN-3D’s long training time to rendering blobs before mapping them to a scene image.Source: §A.3 · p. 12
• The setup follows most BlobGAN settings, using AdamW, an exponential moving average, and an 8-layer 1280-channel mapping-network MLP.BlobGAN used an 8-layer 1024-channel MLP.Source: §A.3 · p. 12

A.4. Volume rendering

• Blob density is computed from squared Mahalanobis distance, with density increasing as a query point approaches the blob center.Each blob reuses its feature vector as the feature value; its density is modeled as a normal distribution and changes as the blob moves.Source: §A.4 · p. 12

B. Qualitative results

• The examples show object movement and resizing, blob visualizations, foreshortening, and camera movement horizontally, vertically, and for zoom.BlobGAN-3D is compared with original BlobGAN for object-editing examples.Source: B. Qualitative results · p. 12
• The method achieves good consistency in realistic multi-object editing on real-world indoor datasets despite some minor changes due to entanglement.Generated-image samples at truncation weight w = 0.3 are provided to show diversity; the paper uses w = 0.6 for its other figures and notes that higher weights increase image quality but lower diversity.Source: B. Qualitative results · p. 12

C. Ablation study

• Both depth-supervised and unsupervised models were trained for roughly the same number of iterations, but neither was trained to convergence because of limited resources.The figure compares multiview rendering and blob-center coordinates with and without depth supervision.Source: C. Ablation study · p. 12
• 0.116 average depth estimation loss with depth supervision, down around 36% from 0.197 without depth supervision.The comparison used 5,000 generated images per model under the same camera sampling strategy.Source: C. Ablation study · p. 12

D. Limitations

• Changing one blob can still affect other objects’ appearance or location because disentanglement is not perfect.A failure example shows a bedside disappearing or reappearing when the bed is moved, attributed to occlusion or failed disentanglement.Source: D. Limitations · p. 13, Figure 22 · p. 23
• Depth estimation can correct a blob centroid’s depth, but blob shape and size mismatches mean that estimate may not reveal the object’s 3D depth.Source: D. Limitations · p. 13
• BlobGAN and BlobGAN-3D take longer to converge than other GANs because they render blobs as an intermediate representation before mapping them to a scene image.Source: D. Limitations · p. 13