PaperlayerView exact revision on arXiv

Source-linked AI summary

Object Scene Representation Transformer

Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas J. Guibas, Klaus Greff, Thomas Kipf

arXiv:2206.06922v2cs.CVcs.AIcs.LG

TL;DR

Supervised 3D scene decomposition requires costly annotations, while prior object-centric rendering methods struggle with complex scenes and computation. OSRT learns object-centric 3D representations through novel-view synthesis and uses a Slot Mixer for rendering. The authors report state-of-the-art decomposition and rendering more than 3000× faster than prior works.

  • Problem

    Prior object-centric 3D rendering methods struggle to generalize to visually complex scenes and incur high computational and memory costs.

    Source: §1 · p. 2

  • Method

    OSRT learns object-centric 3D scene representations from RGB via novel-view synthesis, using Slot Attention and a Slot Mixer decoder.

    Source: §1 · p. 2

  • Results

    OSRT achieves state-of-the-art object decomposition across easier and complex many-object scenes, and its Slot Mixer makes it more than 3000× faster than prior works.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The authors expect OSRT to support model design and scaling efforts for object-centric geometric scene understanding.

    Source: §5 · p. 10

    OSRT masks can bleed into the background, and some architectural choices can produce a 3D-spatial Voronoi tessellation instead of clear object segmentation.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

A compositional understanding of the world in terms of objects and their geometry in 3D space is considered a cornerstone of human cognition. Facilitating the learning of such a representation in neural networks holds promise for substantially improving labeled data efficiency. As a key step in this direction, we make progress on the problem of learning 3D-consistent decompositions of complex scenes into individual objects in an unsupervised fashion. We introduce Object Scene Representation Transformer (OSRT), a 3D-centric model in which individual object representations naturally emerge through novel view synthesis. OSRT scales to significantly more complex scenes with larger diversity of objects and backgrounds than existing methods. At the same time, it is multiple orders of magnitude faster at compositional rendering thanks to its light field parametrization and the novel Slot Mixer decoder. We believe this work will not only accelerate future architecture exploration and scaling efforts, but it will also serve as a useful tool for both object-centric as well as neural scene representation learning communities.

1 Introduction

OSRT targets unsupervised, 3D-consistent decomposition of complex scenes, where prior object-centric rendering methods struggled to scale. It combines novel-view synthesis with a Slot Mixer decoder, and the authors report state-of-the-art decomposition and efficiency across scene complexities.

1 Introduction

  • Predicting unseen viewpoints from RGB can encourage geometrically consistent scene representations and may help discover objects without direct supervision.Source: §1 · p. 1
  • Prior object-centric novel-view-synthesis methods struggled with visually complex scenes and computational cost, requiring thousands of decoding passes per rendered pixel.Independent object decoding compounds the hundreds of steps used by volumetric rendering.Source: §1 · p. 2
  • The Slot Mixer decodes all slots together, requiring one forward pass per rendered pixel regardless of slot count.This object-aware decoder is designed to scale to large numbers of objects with little computational overhead.Source: §1 · p. 2
  • OSRT learns object-centric 3D scene representations from RGB supervision, without additional regularizers, depth maps, or explicit background handling.Its Slot Attention module decomposes the scene representation into slots for objects or background.Source: §1 · p. 2
  • The authors study robustness and the role of novel-view synthesis in facilitating scene decomposition on complex datasets.Source: §1 · p. 2
  • OSRT achieves state-of-the-art object decomposition across easier and complex many-object scenes, outperforming prior methods quantitatively and in efficiency.Source: §1 · p. 2

2 Method

OSRT combines SRT’s light-field novel-view synthesis backbone with Slot Attention and the Slot Mixer decoder to form object-centric scene representations. Its decoder uses ray-conditioned slot allocation and mixing, while the alternative Spatial Broadcast design decodes slots independently.

2.1 Novel view synthesis

  • SRT encodes input images into a Set-Latent Scene Representation (SLSR), then uses camera rays as queries to predict RGB values for novel views.Training uses an L2 reconstruction loss for novel views.Source: §2.1 · p. 3

2.2 Scene decomposition

  • Because SLSR tokens have no clear one-to-one correspondence with scene objects, Slot Attention converts them into a user-sized set of object slots called SlotSR.Its attention normalizes over slots to encourage exclusive grouping of input tokens into object representations.Source: §2.2 · p. 3

2.3 Efficient object-centric decoding

  • The Slot Mixer combines ray-conditioned slot allocation, a Mixing Block that forms a weighted mean of slots, and a Render MLP that predicts RGB.The Allocation Transformer derives relevant slots for each ray, including object locations, boundaries, and occlusions; scalar slot weights can serve as novel-view object assignments.Source: §2.3 · pp. 3–4

2.4 Alternative decoder architecture

  • Spatial Broadcast decodes each slot independently and combines per-slot color estimates with softmax weights, but its full decoder cost grows linearly with slot count.The Slot Mixer instead blends slots with its Allocation Transformer and fully decodes only their weighted mean.Source: §2.4 · pp. 4–5

3 Related works

Related work spans neural rendering, global-latent scene models, and object-centric learning. OSRT draws on Slot Attention and light-field rendering while addressing the computational demands and scene-complexity limits of prior approaches.

Neural rendering

  • NeRF represents a scene with an optimized volumetric MLP, while SRT uses a light-field formulation to scale global-latent models to complex scenes.Other generalizing NeRF methods reproject 3D points into 2D feature maps but lack a readily usable global 3D-based scene representation.Source: §3 · p. 5

Object-centric learning

  • MONet, IODINE, SPACE, and Slot Attention demonstrate unsupervised image decomposition, while later work extends these inductive biases to more complex scenes and video.OSRT uses Slot Attention for its stated efficiency and effectiveness.Source: §3 · p. 5

3D object-centric methods

  • ROOTS, ObSuRF, and uORF extend object-centric methods to 3D, but ObSuRF relies on ground-truth depth during training and uORF uses explicit foreground/background assumptions.ObSuRF and uORF were shown on scenes somewhat more complex than CLEVR; ObSuRF retains high inference cost without ground-truth depth.Source: §3 · p. 5

Generative object-centric rendering

  • GIRAFFE and INFERNO study generative object-centric 3D rendering without rendering novel views of a specifically provided input scene.The paper does not explicitly compare with INFERNO, which is described as capable of modeling lower-complexity CLEVR-like scenes.Source: §3 · p. 6

4 Experiments

The experiments evaluate OSRT’s novel-view reconstruction and unsupervised scene decomposition across datasets of increasing complexity. They compare it with prior methods and examine the effects of decoder design, novel-view synthesis, input views, and other model properties.

4 Experiments

  • Evaluation measures novel-view reconstruction with PSNR and object decomposition with FG-ARI computed jointly across rendered views, making the latter sensitive to 3D inconsistencies.The experiments span datasets of increasing complexity, including MSN-Hard’s tightly packed ShapeNet scenes, realistic backgrounds, and varied camera poses.Source: §4 · p. 6

4.1 Comparison with prior work

  • Across datasets, OSRT outperforms ObSuRF on reported metrics except for nearly identical CLEVR-3D FG-ARI; on MSN-Hard, five input views further improve reconstruction and decomposition.On MSN-Hard, OSRT also produces higher-quality images and more accurate decompositions than ObSuRF in qualitative comparisons.Source: Table 1 · p. 7, §4.1 · p. 7
  • 32.5 fps versus ObSuRF’s 0.01 fps: OSRT renders novel views more than 3000× faster on an Nvidia V100 GPU.The reported speedup combines a roughly 100× advantage for light-field over volumetric rendering and a roughly 30× advantage for Slot Mixer over the SB decoder; OSRT also requires no ground-truth depth for training.Source: §4.1 · p. 7

4.2 Ablations and model analysis

  • Slot Mixer preserves useful object decomposition by using a transformer for slot mixing, not slot decoding; unlike the SB decoder, it avoids substantial memory and compute costs.The SRT decoder’s global aggregation slightly improves reconstruction but does not yield useful object decompositions; the SB decoder performs similarly to Slot Mixer on reconstruction and decomposition.Source: Table 2 · p. 7, §4.2 · p. 8
  • Without novel-view synthesis, OSRT reaches PSNR 28.14 versus 23.54 for the NVS setup, but FG-ARI falls to 0.198 versus 0.812.The comparison supports using NVS as an auxiliary task for unsupervised scene decomposition, even though reconstructing only input views gives higher PSNR.Source: §4.2 · p. 8
  • Additional test-time views improve OSRT’s PSNR and FG-ARI; it also retains FG-ARI 0.780 on grayscale MSN-Hard inputs versus 0.812 on RGB, and generalizes to scenes with more objects than training.For object-count generalization on CLEVR-3D, random slot initialization has a slight advantage out of distribution, while learned initialization appears advantageous in distribution. UpOSRT and VOSRT achieve PSNR 22.42 and 21.38, with FG-ARI 0.798 and 0.767, respectively.Source: §4.2 · pp. 9–10
  • Scene editing can reveal occluded objects by removing a slot and can render an object from another scene with correct occlusions; OSRT’s masks can also bleed into backgrounds or form spatial Voronoi tessellations.The tessellation failure occurs with some architectural choices and substantially lowers FG-ARI; mask bleeding and tessellation are also reported in other object-centric models.Source: §4.2 · p. 9, §4.3 · p. 10
  • OSRT achieves an FG-ARI ratio of 0.940 on MSN-Hard and 0.987 with five input views, compared with 0.707 for ObSuRF, indicating stronger 3D consistency.The ratio compares FG-ARI across views with per-view 2D-FG-ARI; 1.0 indicates perfect consistency.Source: §4.2 · p. 9

5 Conclusion

OSRT is presented as an efficient, scalable approach to unsupervised scene decomposition and rendering, with faster novel-view synthesis and scope for future work on moving objects.

5 Conclusion

  • More than 3000× faster than prior works, OSRT enables efficient novel-view synthesis while decomposing complex visual scenes beyond the reach of existing methods.The conclusion attributes the rendering efficiency to the Slot Mixer decoder.Source: §5 · p. 10
  • OSRT is an efficient and scalable architecture for unsupervised neural scene decomposition and rendering.Source: §5 · p. 10
  • Future work could extend this class of methods beyond static scenes to include moving objects.Source: §5 · p. 10

A.1 Author contributions

The author contributions span project leadership, modeling and implementation, infrastructure, experiments, evaluation, visualization, and paper writing.

A.1 Author contributions

  • Mehdi S. M. Sajjadi and Thomas Kipf co-initiated and co-led the project, with major roles in management, modeling decisions, implementation, experiments, and paper writing.Sajjadi proposed and implemented Slot Mixer; Kipf implemented the first model.Source: §A.1 · p. 13
  • Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, and Filip Pavetić contributed baseline or ablation experiments and infrastructure, with additional evaluation, metrics, visualization, and writing contributions.Source: §A.1 · p. 13
  • Mario Lučić, Leonidas J. Guibas, and Klaus Greff provided project advice; Guibas and Greff also contributed to paper writing, and Greff advised on metrics, visualization, baselines, datasets, experiments, and figures.Source: §A.1 · p. 13

A.2 Societal impact

OSRT performs 3D scene decomposition and novel-view synthesis without object or segmentation labels, but the investigation is limited to synthetic scenes with relatively simple scanned objects.

A.2 Societal impact

  • OSRT enables 3D scene decomposition and novel-view synthesis without collecting object or segmentation labels.Source: §A.2 · p. 13
  • The current investigation is limited to synthetically generated 3D scenes populated with relatively simple 3D-scanned everyday objects.The authors state that this work has no immediate impact on general society.Source: §A.2 · p. 13
  • The authors expect this class of methods may offer reliability and interpretability advantages over supervised scene or object understanding, while dataset-selection bias can still affect learned models.They note that individual learned object variables can be inspected by analyzing or rendering the corresponding slot.Source: §A.2 · p. 13

A.3.1 ObSuRF

The ObSuRF baseline was adapted for MSN-Hard's larger scenes and memory demands, with training choices informed by observed convergence behavior.

A.3.1 ObSuRF

  • ObSuRF was configured with 32 slots for scenes containing up to 31 objects, a wider positional-encoding frequency range, and smaller batches to accommodate peak memory use.The batch size was reduced from 64 to 32, and the frequency range changed from 2−7 to 211.Source: §A.3.1 · p. 13
  • Occlusion-loss annealing was delayed until step 180,000 because increasing its weight earlier led to a local minimum where one slot explained the entire scene.The loss weight was increased linearly from 0 to 1 over 80,000 steps.Source: §A.3.1 · p. 13
  • Training stopped after 800 k steps when loss and FG-ARI showed no further change, using four A100 GPUs for six days.Source: §A.3.1 · p. 13

A.3.2 uORF

uORF located objects on MSN-Easy but captured their shapes poorly, and its MSN-Hard runs did not learn object representations. The authors attribute the Hard-dataset difficulty to capacity constraints.

A.3.2 uORF

  • On MSN-Easy, uORF achieved an FG-ARI of 0.216 despite locating objects, but failed to capture their shapes precisely.The 128×128 input size makes these metrics informative about quality but not directly comparable with ObSuRF or OSRT.Source: §A.3.2 · pp. 13–14
  • On MSN-Hard, uORF captured only the background and horizon, not the objects, after repeated training attempts.The authors believe this resulted from uORF’s limited capacity, further constrained by reducing slot dimensionality to 10 so a 32-slot model would fit on a 40 GB Nvidia A100.Source: §A.3.2 · p. 14
  • Using 8 slots instead of the 32-slot configuration also failed to produce object representations on MSN-Hard.The authors tried this alternative without increasing slot dimensionality.Source: §A.3.2 · p. 14

A.4 Model and training details

The appendix reports OSRT’s model and training details, including SRT modifications that reduced model size while improving MSN-Hard reconstruction. It also documents a prior spatial-partitioning failure and provides complete results and qualitative visualizations across datasets.

A.4 Model and training details

  • The full OSRT model has 81 M parameters, and the experiments use identical hyperparameters across datasets unless noted otherwise.The encoder removes its last CNN block, increasing SLSR tokens fourfold while leaving Slot Mixer rendering speed unaffected because it operates on constant-size SlotSR.Source: §A.4 · p. 14
  • OSRT uses absolute rather than relative pose parametrization, drops camera-ID and 2D-position embeddings, and uses 5 pre-normalization self-attention encoder layers.UpOSRT is the exception, using relative poses by definition; the pose changes slightly improved reconstruction quality without further drawbacks.Source: §A.4 · p. 14
  • PSNR rose from 23.33 to 25.92 on MSN-Hard as the SRT modifications reduced model size from 74 M to 41 M parameters.Qualitative comparisons are shown in Fig. 7.Source: §A.4 · p. 14, Figure 7 · p. 15
  • Slot Mixer adds an MLP before its Allocation Transformer, and the authors report that this improves results and stabilizes training.The model also uses one Slot Attention iteration except on MSN-Easy, where three performed best.Source: §A.4 · p. 14

A.5 Failure cases

  • A prior OSRT variation produced a 3D-consistent decomposition that partitioned space rather than aligning slots with object positions.Figure 8 shows an example in which the decomposition cuts through objects.Source: §A.5 · p. 14, Figure 8 · p. 15

A.6 Results

  • Complete quantitative results are provided for CLEVR-3D, MSN-Easy, and MSN-Hard in Tables 5–7.The appendix also notes that MSN-Hard performance continued to improve beyond the reported training duration, without affecting the stated conclusions or comparisons.Source: §A.4 · p. 14, §A.6 · p. 15, Table 5 · p. 17

A.7 Videos

  • On simpler datasets, Slot Mixer matched the SRT decoder in reconstruction quality and achieved similar or better FG-ARI.Table 4 summarizes the decoder variants; MSN-Hard results are in Table 2 of the main paper.Source: Table 4 · p. 15
  • Figure 10 compares MSN-Hard qualitative outputs across ObSuRF, OSRT view counts and variants, and 2D-target reconstruction.The appendix also provides OSRT video renders and a foreground-only segmentation visualization aligned with FG-ARI’s foreground evaluation.Source: §A.7 · p. 15, Figure 9 · p. 16, Figure 10 · p. 18

Related papers

Loading 2206.06922v2…