PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Discovery of Object Radiance Fields

Hong-Xing Yu, Leonidas J. Guibas, Jiajun Wu

arXiv:2107.07905v2cs.CVcs.AI

TL;DR

Object-centric scene representations should explain image formation, capture 3D structure, and be learned without supervision, but existing methods often lack one or more of these properties. uORF combines neural radiance fields with deep inference to infer object and background fields from one image; the paper reports 3D segmentation and scene-generation capabilities, including editing.

  • Problem

    Existing scene-decomposition methods often lack one or more of the desired properties: unsupervised learning, explaining image formation, and 3D awareness.

    Source: §1 · p. 1

  • Method

    uORF integrates conditional NeRF object representations with deep inference to infer object-centric 3D radiance fields from a single image.

    Source: §1 · p. 1, §5 · p. 9

  • Results

    The paper demonstrates uORF on 3D scene segmentation and generation, and reports generalization to novel scene arrangements and unseen combinations of object properties.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results suggest neural rendering can be integrated into deep probabilistic inference to learn factorized 3D object-centric representations from RGB images.

    Source: §5 · p. 9

    The representation models object appearance but entangles physical properties that may matter for downstream tasks.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

We study the problem of inferring an object-centric scene representation from a single image, aiming to derive a representation that explains the image formation process, captures the scene's 3D nature, and is learned without supervision. Most existing methods on scene decomposition lack one or more of these characteristics, due to the fundamental challenge in integrating the complex 3D-to-2D image formation process into powerful inference schemes like deep networks. In this paper, we propose unsupervised discovery of Object Radiance Fields (uORF), integrating recent progresses in neural 3D scene representations and rendering with deep inference networks for unsupervised 3D scene decomposition. Trained on multi-view RGB images without annotations, uORF learns to decompose complex scenes with diverse, textured background from a single image. We show that uORF enables novel tasks, such as scene segmentation and editing in 3D, and it performs well on these tasks and on novel view synthesis on three datasets.

1 INTRODUCTION

The paper targets unsupervised, 3D-aware object-centric scene representations that explain image formation, and proposes uORF to infer object and background radiance fields from one image. It presents neural rendering as a route to this goal and reports 3D segmentation, editing, and generalization capabilities.

1 INTRODUCTION

  • The paper argues that object-centric scene representations should be learned without supervision or category priors, explain image formation, and capture 3D geometric and physical properties for navigation, interaction, and manipulation.Prior methods each lack one or more of these aspects; the paper identifies complex, non-differentiable 3D-to-2D image formation, including occlusion, as the fundamental challenge to acquiring all three.Source: §1 · p. 1
  • uORF infers object and background radiance fields from a single image, using neural rendering and deep inference for unsupervised 3D scene decomposition.Training uses multi-view rendering and pixel-space reconstruction losses, with sparse multi-view RGB images rather than geometry or segmentation annotations.Source: §1 · pp. 1–2
  • Explicitly modeling background and foreground supports complex shapes and textured environments, including editing that moves objects or changes the background.The paper also introduces progressive coarse-to-fine training to improve representation quality while keeping computational cost affordable.Source: §1 · p. 2
  • Across three datasets of increasing scene complexity, uORF learns 3D object segmentation with fine shape details and recovers background appearance details.The reported examples include thin chair legs and irregular wooden-floor textures.Source: §1 · p. 2
  • The paper reports that uORF enables 3D scene segmentation and editing, and generalizes to novel scene arrangements and unseen combinations of object properties.These are presented as contributions enabled by inferring individual 3D object radiance fields from a single view.Source: §1 · p. 2

2 RELATED WORK

The related-work section surveys object discovery, probabilistic scene decomposition, 3D de-rendering, and neural scene representations. It distinguishes uORF's focus on unsupervised, single-image decomposition of multi-object 3D scenes from approaches with different supervision, scene scope, or inference goals.

Co-segmentation and object discovery

  • Traditional object-discovery methods locate objects across images but do not explain image formation or represent scenes in 3D (Russell et al., 2006; Sivic et al., 2005; 2008; Grauman & Darrell, 2006; Joulin et al., 2010; Rubio et al., 2012; Vicente et al., 2011; Rubinstein et al., 2013; Cho et al., 2015; Li et al., 2019; Vo et al., 2020).The section notes that these approaches model visual words or cluster image patches.Source: §2 · p. 2

Deep probabilistic inference for scene decomposition

Unsupervised 3D scene decomposition

  • Prior unsupervised 3D decomposition methods require shape pretraining, multi-view inference images, or dense depth supervision; the closest cited work demonstrates only a single textureless background (Elich et al., 2020; Chen et al., 2020; Stelzner et al., 2021).The paper contrasts this with its explicit modeling of objects and background for complex shapes and textured backgrounds.Source: §2 · p. 3

Scene de-rendering

  • Scene de-rendering methods reconstruct object-centric 3D representations but rely on manual annotations of specific objects and scene categories (Yao et al., 2018).The cited examples include semantic segments and geometric attributes for manipulation, and object-centric scene graphs for dynamic scenes.Source: §2 · p. 3

Neural scene representations and rendering

  • Unlike work on single objects or holistic scenes, uORF decomposes multi-object scenes into object NeRFs without segmentation annotations; unlike GIRAFFE, it addresses single-image inference rather than unconditional generation (Sitzmann et al., 2019; Tewari et al., 2020; Niemeyer et al., 2020; Mildenhall et al., 2020; Yu et al., 2020; Kosiorek et al., 2021; Jang & Agapito, 2021; Schwarz et al., 2020; Niemeyer & Geiger, 2021; Chan et al., 2020; Niemeyer & Geiger, 2020).The section places this distinction within work on neural scene representations, differentiable rendering, and NeRFs.Source: §2 · p. 3

3 APPROACH

The approach infers object-centric 3D scene representations from a single image, then composes and renders their radiance fields for multi-view training.

3 APPROACH

  • A single image yields object and background latents that condition radiance fields; multi-view rendered images provide training supervision without 3D geometry or segmentation annotations.The training loss combines reconstruction, perceptual, and adversarial terms.Source: §3 · p. 4, §3.3 · p. 5

3.1 OBJECT-CENTRIC LATENT INFERENCE

  • The model assumes a background and at most K foreground objects, using separate background-aware slots to bind image features and update object-centric latents.Foreground queries use different coordinate frames and are constrained to zero density outside a box early in training.Source: §3.1 · pp. 4–5

3.2 COMPOSITIONAL NEURAL RENDERING

  • Conditional NeRFs represent objects and background, whose radiance fields are combined by density-weighted means and rendered along camera rays.The object-field MLP is shared across objects, while the background uses a separate network because its geometry and appearance differ.Source: §3.2 · p. 5

3.3 MODEL LEARNING

  • Perceptual loss accommodates appearance uncertainty from a single view, while adversarial loss addresses multimodal uncertainty in object shape.The authors state that unimodal reconstruction can yield blurry mean-shape results.Source: §3.3 · p. 5
  • Coarse-to-fine training starts with downsampled images, then uses randomly cropped high-resolution patches to recover fine details.The coarse stage can recover rough object radiance fields, while details such as thin chair legs may be missing.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate uORF’s factorized 3D representations and generation on three increasingly complex synthetic datasets. Across segmentation, novel-view synthesis, and editing, uORF outperforms compared methods, with additional tests of generalization and loss design.

4 EXPERIMENTS

  • The three synthetic datasets progress from CLEVR-like primitive objects to textured-background rooms and then rooms with diverse chair shapes and backgrounds.The datasets are CLEVR-567, Room-Chair, and Room-Diverse.Source: §4 · p. 6

4.1 SCENE SEGMENTATION IN 3D

  • For 3D segmentation, NV-ARI evaluates the three non-input views, while Fg-ARI evaluates only foreground pixels and ignores background segmentation.Each scene has four views, with one input view and three novel views.Source: §4.1 · p. 6

4 EXPERIMENTS

  • uORF outperforms all compared methods on segmentation ARI and NV-ARI, novel-view synthesis metrics, and scene-editing metrics.The reported segmentation results cover ARI and novel-view ARI; the novel-view and editing evaluations report gains across all metrics.Source: §4.1 · p. 6, §4.2 · p. 7, Table 2 · p. 7

4.2 NOVEL VIEW SYNTHESIS

  • Factorized uORF preserves object features versus NeRF-AE; background-aware modeling recovers textured backgrounds, and patch-level fine training sharpens cube edges.In the Room-Diverse example, uORF roughly recovers object shapes; without progressive training, cubes can be confused with spheres.Source: §4.2 · p. 7

4.3 SCENE DESIGN AND EDITING IN 3D

  • On Room-Chair, uORF edits both object positions and background textures, evaluating each task on 500 test scenes with four rendered images.For background changes, foreground objects are repositioned so the background indicator image differs from the ground-truth image.Source: §4.3 · p. 8

4.4 GENERALIZATION AND ANALYSIS

  • On 11-object packed-CLEVR-11, uORF reasonably performs and beats baselines; on held-out color-shape combinations, it matches normally trained uORF (oracle).The packed arrangements and test object combinations are unseen during training; uORF also shows generalization to unseen object shapes in Appendix D.Source: §4.4 · p. 9, Table 5 · p. 9, Table 6 · p. 9
  • Perceptual and adversarial losses both significantly improve novel-view image quality.The paper presents this as an ablation on Room-Diverse.Source: §4.4 · p. 9, Table 7 · p. 9

5 CONCLUSION

uORF infers object-centric 3D radiance fields from a single image of complex multi-object scenes. The authors demonstrate scene segmentation and generation in 3D, and view the results as a promising direction for learning factorized representations from RGB images.

5 CONCLUSION

  • uORF learns to infer object-centric 3D radiance fields from a single image of complex multi-object scenes and demonstrates scene segmentation and generation in 3D.The authors describe the results as suggesting a promising direction for integrating neural rendering into deep probabilistic inference to learn factorized 3D object-centric representations from RGB images.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement reports release of code, datasets, and pretrained models, alongside instructions for use and training.

REPRODUCIBILITY STATEMENT

  • The authors provide training and test code, all three synthetic datasets, pretrained models for all three datasets, and instructions for using the code and training on new datasets.Appendix B describes implementation details for re-implementing the work.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement notes that the radiance-field representation models appearance while entangling physical properties that may matter for downstream tasks. The supplementary overview lists implementation, dataset, baseline, and additional-results materials.

ETHICS STATEMENT

  • The radiance fields model object appearances but entangle physical properties that may be crucial to downstream tasks in a non-interpretable way.The authors envision structured 3D object representations for specific downstream applications as a possible route toward greater transparency and interpretability.Source: ETHICS STATEMENT · p. 10

A SUPPLEMENTARY MATERIAL OVERVIEW

  • The supplement includes implementation and dataset details, additional results and failure cases, and a comparison clarifying that GIRAFFE addresses unconditional generation rather than conditional inference.It also includes a real-photo demonstration and analysis of sensitivity to slot initialization.Source: A SUPPLEMENTARY MATERIAL OVERVIEW · p. 13

B IMPLEMENTATION

uORF uses coordinate-aware object-centric inference and separates foreground objects from the background in its scene representation. A world-space background and a temporary foreground locality constraint further encourage disentanglement.

B.1 OBJECT-CENTRIC LATENT INFERENCE

  • With fixed camera focal length, the encoder adds four normalized pixel-coordinate channels to RGB features to represent foreground position and pose in viewer coordinates.Each coordinate axis is normalized to [−1, 1].Source: §B.1 · p. 13

B.2 COORDINATE SPACE AND LOCALITY CONSTRAINT FOR BETTER FORE-/BACK-GROUND DISENTANGLEMENT

  • The model represents foreground objects in viewer space and the background in world space, aggregating sparse-view information under a similar-background-geometry assumption.World-space background modeling empirically provides more detail than viewer-space modeling.Source: §B.2 · p. 13
  • Viewer-space positional information also discourages the background slot from decoding foreground objects.Source: §B.2 · p. 14
  • For the first 100K iterations, a foreground box covering roughly 90% of projected image pixels forces queried points outside it to have zero density.The constraint empirically helps prevent foreground slots from fitting background segments.Source: §B.2 · p. 14

B.3 NEURAL RADIANCE FIELD ARCHITECTURE.

The implementation uses conditional object radiance fields and background-aware slot inference, with staged losses and coarse-to-fine training. Training renders multiple views for supervision from each input image.

B.3 NEURAL RADIANCE FIELD ARCHITECTURE.

  • The section introduces the conditional object radiance field architecture and points to its architecture illustration.Source: §B.3 · p. 14

B.4 MODEL LEARNING

  • The latent-inference algorithm samples background and foreground slots from their respective priors, binds slots to object regions, then aggregates updates and recurrently updates the slots.Source: §B.4 · p. 15
  • Perceptual and adversarial losses begin after 100K iterations, and adversarial loss is used on Room-Diverse but not CLEVR-567 or Room-Chair.The perceptual feature extractor uses the fourth convolutional block of ImageNet-pretrained VGG16; the adversarial discriminator follows a modified StyleGAN2 architecture.Source: §B.4 · pp. 14–15
  • Training proceeds for 600K coarse iterations at 64×64 resolution, followed by 600K fine iterations using 64×64 crops from 128×128 images.Each batch uses one input image and four rendered supervision views; training takes about six days on one Nvidia RTX 3090 GPU.Source: §B.4 · p. 15

C.1 DATA

The experiments use three scene datasets of increasing complexity, with training and test sets sharing texture, shape, and color pools but differing in object arrangements and illumination effects. The appendix also describes Slot Attention and NeRF-AE baselines.

C.1 DATA

  • Across all three datasets, training and test sets share texture, shape, and color pools but differ in spatial arrangements and global-illumination effects.Source: §C.1 · p. 15
  • CLEVR-567 uses cylinders, cubes, and spheres in eight colors, with two possible object sizes and the original CLEVR camera intrinsics.Source: §C.1 · p. 15
  • Room-Chair uses a ShapeNet chair model with the same materials and colors as CLEVR-567.Room-Chair and Room-Diverse use latent dimension D = 64 and maximum object count K = 5.Source: §C.1 · p. 16
  • Room-Diverse draws object shapes from 1,200 ShapeNet chairs and uses eight diffuse colors; it alone receives adversarial loss because shape uncertainty occurs there.Source: §C.1 · p. 16

C.2 BASELINE ARCHITECTURES

  • The Slot Attention baseline uses six convolutional encoder layers and six convolution-transpose decoder layers, with 64 channels per layer.All models use 128×128 images.Source: §C.2 · p. 16
  • NeRF-AE follows NeRF without view direction input, uses positional-embedding frequency 5, and sets the number of slots to one.Its encoder resembles uORF’s but increases the base channel count from 64 to 256.Source: §C.2 · p. 16

D ADDITIONAL RESULTS

The additional results report generalization to unseen shapes and real photos, examine sensitivity to slot initialization, and describe an attention rank-collapse failure. The comparison with GIRAFFE distinguishes the methods’ primary goals.

D ADDITIONAL RESULTS

  • The decoder assumes Lambertian surfaces because specularity estimation from a single image is intrinsically ambiguous.The foreground decoder follows NeRF with fewer parameters, and its density output uses ReLU.Source: Figure 9 · p. 17
  • uORF yields the same performance level on Room-Diverse test scenes whose 500 chair shapes are disjoint from the 1,200 training shapes.The reported unseen/seen-shape comparisons cover novel-view synthesis and 3D segmentation.Source: Table 12 · p. 17, Table 13 · p. 17, D ADDITIONAL RESULTS · p. 17
  • A Room-Diverse-pretrained uORF discovers roughly positioned and oriented objects in cellphone photos, producing plausible segmentation and object-moving results despite unseen objects and imaging conditions.Source: D ADDITIONAL RESULTS · p. 17, Figure 10 · p. 18
  • Across 500 Room-Chair test scenes and five initialization seeds, mean ARI is 78.8% and its standard deviation is 1.7%.The reported variability indicates that the different seeds produce ARI results close to that mean.Source: D ADDITIONAL RESULTS · p. 17
  • Additional qualitative examples cover segmentation, novel-view synthesis, scene editing, loss evaluation, challenging arrangements, and unseen object appearances.Source: D ADDITIONAL RESULTS · p. 17
  • Attention rank-collapse occurs when foreground slots share nearly identical attention maps and decode nothing or all foreground objects.It can arise from degenerate slot initialization; changing the seed empirically addresses it, while a fundamental solution is left to future work.Source: D ADDITIONAL RESULTS · pp. 17–18

E COMPARISON TO GIRAFFE

  • Unlike GIRAFFE’s focus on unconditional generation, uORF aims to infer a 3D multi-object scene representation from one image as well as support rendering and editing.Source: E COMPARISON TO GIRAFFE · p. 18

E.1 COMPARISONS BETWEEN OUR UORF AND GIRAFFE

The comparison finds that GIRAFFE struggles to infer multi-object scenes from a single image, including reconstruction and novel-view synthesis. The discussion distinguishes its controllable image generation goal from uORF's inference of object distributions, which supports 3D segmentation and editing.

E.1 COMPARISONS BETWEEN OUR UORF AND GIRAFFE

  • GIRAFFE fails to reconstruct multi-object scenes from a single image and to synthesize novel views, let alone segment scenes in 3D.The comparison covers CLEVR-567 and Room-Chair; Figures 22 and 23 show qualitative comparisons, and Tables 14 and 15 present inference comparisons.Source: §E.1 · pp. 18–19, Table 14 · p. 19, Table 15 · p. 19

E.2 GIRAFFE INFERENCE ON CLEVR-2345

  • On CLEVR-2345, GIRAFFE cannot perform novel-view synthesis, and its 2D feature-map renderer fails when viewpoint changes become significant.The paper reports this for the author-provided pretrained model and attributes the wide-baseline failure to the renderer not being inherently 3D.Source: §E.2 · p. 19, Figure 25 · p. 32

E.3 DISCUSSION AND SUMMARY

  • uORF targets inference of the joint object distribution from a single image, unlike GIRAFFE's extrinsic-controllable image generation, enabling 3D segmentation and editing.The discussion identifies ambiguous object correspondences, unknown object counts, and optimization issues as challenges for GIRAFFE's multi-object inference.Source: §E.3 · p. 19
  • Using both perceptual and adversarial losses improves image quality; attention rank-collapse occurs when foreground slots share attention and decode to the same radiance field.In the illustrated failure, the shared slots decode to an empty radiance field rather than specializing to objects.Source: Figure 18 · p. 27, Figure 21 · p. 29
Loading 2107.07905v2…