PaperlayerView exact revision on arXiv

Source-linked AI summary

DORSal: Diffusion for Object-centric Representations of Scenes et al

Allan Jabri, Sjoerd van Steenkiste, Emiel Hoogeboom, Mehdi S. M. Sajjadi, Thomas Kipf

arXiv:2306.08068v3cs.CVcs.AIcs.LG

TL;DR

Few-view 3D scene models can render novel views and represent objects, but uncertainty about unobserved scene content can leave their renderings blurry. DORSal conditions video diffusion on frozen object slots to synthesize multiple views and support object-level editing. It reports improved FID over prior 3D scene-understanding methods and stronger performance on real-world Street View than prior 3D diffusion work.

  • Problem

    Few-view 3D scene models generalize across scenes, but uncertainty about unobserved content often makes their novel views blurry.

    Source: §1 · p. 1

  • Method

    DORSal encodes a few context views into frozen OSRT object slots and conditions a video-diffusion decoder on those slots to generate novel views.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    DORSal reports 5x-10x improvement in FID over prior 3D scene-understanding methods and significantly better performance on real-world Street View than prior 3D diffusion work.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Object-slot conditioning supports basic scene editing, including removing slots and transferring them between scenes.

    Source: §3.2 · p. 4, §5.2 · p. 8

    Removing a slot can have non-local effects or fail to remove an object, and Street View slots may have little effect when multiple slots represent one object.

    Source: A LIMITATIONS · p. 15

Abstract

from arXiv · show

Recent progress in 3D scene understanding enables scalable learning of representations across large datasets of diverse scenes. As a consequence, generalization to unseen scenes and objects, rendering novel views from just a single or a handful of input images, and controllable scene generation that supports editing, is now possible. However, training jointly on a large number of scenes typically compromises rendering quality when compared to single-scene optimized models such as NeRFs. In this paper, we leverage recent progress in diffusion models to equip 3D scene representation learning models with the ability to render high-fidelity novel views, while retaining benefits such as object-level scene editing to a large degree. In particular, we propose DORSal, which adapts a video diffusion architecture for 3D scene generation conditioned on frozen object-centric slot-based representations of scenes. On both complex synthetic multi-object scenes and on the real-world large-scale Street View dataset, we show that DORSal enables scalable neural rendering of 3D scenes with object-level editing and improves upon existing approaches.

1 INTRODUCTION

Few-view scene models generalize across scenes and use structured representations, but uncertainty about unobserved content often makes their novel views blurry. DORSal conditions video diffusion on frozen object slots to generate more precise views while retaining object-level editing.

1 INTRODUCTION

  • Few-view scene models can generalize to novel scenes and use structured object representations, yet uncertainty about unobserved content often leaves their novel views blurry.These models require only a few observations per scene, unlike single-scene-optimized NeRFs.Source: §1 · p. 1
  • DORSal encodes a few observations with OSRT, then conditions a video diffusion model on the resulting object slots to generate 3D-consistent novel views.The slots can also be removed or transferred between scenes for basic object-level editing.Source: §1 · p. 2
  • The method combines diffusion-based rendering with object-centric scene representations to address uncertainty while retaining scene editing capabilities.The introduction presents this combination as supporting scalable rendering across complex scenes and large sets of target views.Source: §1 · p. 2
  • DORSal reports 5x-10x improvement in FID over prior 3D scene-understanding methods, and significantly better performance on real-world Street View than prior 3D diffusion work.It also reports more precise novel views while staying true to scene content, with object-level editing capabilities.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries introduce object-centric scene representations and conditional diffusion as the basis for DORSal. OSRT extracts object slots from a few views, while diffusion models learn to reverse a noise process and can produce sharp, conditioned samples.

Object-centric Scene Representations

  • OSRT supplies DORSal with pretrained object representations as conditioning, while earlier volumetric object-representation methods were limited by memory and compute demands.OSRT uses light-field rendering for inference speedups of O(100) over SRT.Source: §2 · p. 2
  • OSRT turns a small set of input views into object slots, using an encoder and Slot Attention; the user-chosen slot count bounds the number of modeled objects.Its shared decoder renders each slot as RGB and alpha, then combines them with a softmax; L2 training can blur uncertain novel views.Source: §2 · p. 2

Generative Modeling with Conditional DDPMs

  • DORSal’s overview contrasts OSRT’s commonly blurry L2 predictions with a frozen OSRT encoder whose object slots and target poses condition a diffusion U-Net to denoise novel views.The resulting sharp renders remain decomposable into scene objects for edits.Source: Figure 1 · p. 3
  • Conditional diffusion learns to reverse a noise process; its denoiser predicts reparameterization noise as the signal is corrupted over diffusion time.The schedule decreases α_t and increases σ_t, and conditioning information can enter the denoiser through U-Net cross-attention.Source: §2 · p. 3

3 DORSAL

DORSal encodes context views into object slots and uses a video-diffusion decoder to render multiple views of the same scene. Its conditioning and sampling design supports view consistency, slot-based edits, and long camera paths.

3 DORSAL

  • DORSal encodes a few context views into OSRT Object Slots, then conditions video diffusion on those slots to synthesize multiple 3D-consistent views of the scene.The OSRT representation is pretrained and frozen.Source: §3 · p. 3

3.1 DECODER ARCHITECTURE & CONDITIONING

  • The Multiview U-Net combines per-frame convolutions with axial self-attention between frames, using object slots to avoid attending directly to large, often redundant conditioning features.Each view receives projected slot tokens shared across views and a projected token for its target camera pose; conditioning enters through FiLM and cross-attention.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 EDITING & SAMPLING

  • At inference, masking an individual slot can remove its object, and transferring slots between scenes provides another form of editing.The authors derive edit effects by comparing edited samples with unedited samples and evaluate affected pixels against ground-truth instance segments.Source: §3.2 · p. 4
  • Edit-based instance masks are formed from pixel-wise differences between edited and unedited views, followed by smoothing and per-pixel assignment to the edit with the largest difference.The procedure uses a per-pixel softmax across difference images, a median filter, and a final argmax across edits.Source: §3.2 · pp. 4–5, Figure 3 · p. 4
  • For camera paths longer than the training context, DORSal interleaves identity, half-context temporal shifts, and random frame shuffles while denoising in short stages.These stages aim to maintain within-block consistency, smooth transitions, and global consistency across frames.Source: §3.2 · p. 5

4 RELATED WORK

Related work spans neural scene representations, diffusion-based novel-view synthesis, and object-centric generative models. DORSal combines object-slot conditioning with video diffusion to target complex scenes, 3D-consistent views, and object-level editing.

Novel View Synthesis and 3D Scene Representations

  • NeRF supports high-quality novel-view synthesis through scene-specific volumetric rendering, while SRT and RUST scale set-latent representations to complex real-world data but may produce blurry views.Other latent 3D representations have struggled to scale beyond simple synthetic datasets.Source: §4 · p. 5

Diffusion Generative Models

  • Pose-conditional diffusion methods have advanced 3D novel-view synthesis, but have focused mainly on individual objects, and maintaining 3D consistency remains challenging.DORSal combines video diffusion with object-slot conditioning to target more consistent views, especially in real-world settings, and support object-level edits.Source: §4 · p. 5

Object-centric diffusion methods

  • LSD and SlotDiffusion pair Slot Attention with diffusion decoders for image or video segmentation, but do not address 3D scenes or novel-view synthesis.Source: §4 · p. 5

Novel View Synthesis and 3D Scene Representations

  • Figure 4 compares DORSal with 3DiM, SRT, and OSRT on MultiShapeNet and Street View.Source: Figure 4 · p. 6

Object-centric diffusion methods

  • OBJECT 3DIT uses supervised synthetic edits, whereas DORSal’s edits do not require edit supervision or specially prepared edit data; DisCoScene instead conditions on object-based layouts.DORSal uses learned object representations as conditioning, reducing the prior scene knowledge required.Source: §4 · p. 6

5 EXPERIMENTS

The experiments evaluate novel-view synthesis, object-level scene edits, and camera-path consistency on synthetic MultiShapeNet and real-world Street View scenes. DORSal improves visual precision over scene-representation baselines, transfers slot-based editing capabilities, and renders smoother camera paths with mixed-view training.

5 EXPERIMENTS

  • The evaluation compares novel-view synthesis, object removal and transfer, and camera-path rendering on challenging synthetic MultiShapeNet and real-world Street View scenes.MultiShapeNet includes 16–32 objects per scene and novel test objects; Street View has inconsistent camera poses, moving objects, and exposure and white-balance changes.Source: §5 · p. 6

5.1 NOVEL-VIEW SYNTHESIS

  • DORSal greatly outperforms scene-representation methods on FID while achieving slightly lower PSNR than SRT and (Sup-)OSRT, and strictly outperforms 3DiM across all metrics.The comparison to 3DiM uses 256 DDPM sampling steps for both methods.Source: Table 1 · p. 7, Table 2 · p. 7, §5.1 · p. 7

5.2 EVALUATION OF OBJECT-LEVEL EDITS

  • Scene-editing capabilities transfer from OSRT to DORSal to a large degree, despite DORSal lacking object-centric architectural priors and segmentation supervision.Sup-OSRT supplies DORSal’s slots and serves as an editing upper bound, with significantly reduced visual fidelity.Source: §5.2 · p. 8, Table 3 · p. 8
  • Combining half the slots from each of two scenes lets DORSal generate a composite scene with objects and background carried over from both sources.Transferred objects are rendered with scene lighting and shadows modeled for their new placement.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 CAMERA-PATH RENDERING

  • Mixed nearby and random training views improve camera-path consistency, yielding qualitatively smooth videos compared with random-view training.On 40 held-out scenes with 190 target views each, PSNR is 16.50db for DORSal, 17.47db for 3DiM, and 18.06db for DORSal trained on mixed views.Source: §5.3 · p. 9, Figure 7 · p. 9

6 CONCLUSION

DORSal is a generative model for precise novel-view rendering of diverse 3D scenes. It conditions a text-to-video architecture on camera poses and object-centric scene representations, supporting scene editing and suggesting a route for future model and data scaling.

6 CONCLUSION

  • DORSal renders precise novel views of diverse 3D scenes and supports editing by controlling objects through their scene-representation slots.The conclusion characterizes this as basic object-level editing.Source: §6 · p. 9
  • DORSal adapts an existing text-to-video architecture for controllable 3D scene generation by conditioning on camera poses and object-centric representations.The model is trained on large-scale 3D scene datasets.Source: §6 · p. 9
  • The text-to-video foundation may allow future improvements in that model class to transfer to compositional 3D generation and could support joint training on large-scale video and 3D scene data.Source: §6 · p. 9

ETHICS STATEMENT

DORSal demonstrates precise novel-view rendering and basic object-level editing, but its practical usefulness is currently limited. The authors also identify dataset selection bias as an unresolved concern for performance and interpretability.

ETHICS STATEMENT

  • DORSal enables precise novel-view rendering conditioned on Object Slots and basic object-level editing.Source: ETHICS STATEMENT · p. 10
  • The authors state that DORSal’s practical usefulness remains limited and foresee no immediate broader societal impact.Source: ETHICS STATEMENT · p. 10
  • Despite not using web-scraped image-text conditioning pairs, DORSal remains susceptible to dataset selection bias, whose effects on performance and interpretability require further understanding.Source: ETHICS STATEMENT · p. 10

A LIMITATIONS

DORSal has limitations in training, high-resolution rendering and editing, camera-path control, and the scope and reliability of object-level edits.

A LIMITATIONS

  • DORSal is not end-to-end trained and is limited by the quality of Object Slots from its separately trained OSRT model.End-to-end training would bring additional memory requirements, which the authors identify as a challenge for future work.Source: A LIMITATIONS · p. 15
  • At 128×128, DORSal has worse editing performance and cross-view consistency than at 64×64; scaling the model or cascading upsampling stages are proposed remedies.The authors note these approaches would require more compute and memory or additional conditional upsampling stages.Source: A LIMITATIONS · p. 15
  • On Street View, learning the dataset’s time-ordered camera prior makes rendering views along arbitrarily chosen test-time camera paths challenging.MultiShapeNet frames use random viewpoints, while Street View frames are ordered by time.Source: A LIMITATIONS · p. 15
  • Removing a slot can alter other objects or the background, and edits may be partial or ineffective; on Street View, multiple slots may represent one object.The authors say many Street View slots have little or no effect when removed, likely because OSRT often assigns multiple slots to a car.Source: A LIMITATIONS · p. 15
  • Current editing is limited to operations such as object removal and transfer; finer-grained edits remain future work.The paper suggests language-supervised co-training or more disentangled object representations as possible directions.Source: A LIMITATIONS · p. 15

B.1 EVALUATION

The appendix describes evaluation protocols and implementation details for novel-view synthesis, camera-path rendering, DORSal, 3DiM, and the SRT/OSRT representations.

B.1 EVALUATION

  • Novel-view evaluation uses 5 target views for MultiShapeNet and 3 for Street View; camera-path training uses 10 input and 10 target views at 64×64.The OSRT base model uses 5 input views, and Street View frames are cropped to 128×128 for novel-view evaluation.Source: §B.1 · pp. 15–16
  • Object-edit evaluation uses the first 1,000 MultiShapeNet scenes and a 7-pixel median kernel; the 64×64 model is less susceptible to edit side effects than the 128×128 model.The reported FG-mIoU scores are 60.6 for the 128×128 model and 70 for the 64×64 model.Source: §B.1 · p. 15

B.2.1 DORSAL

  • DORSal conditions a multiview U-Net on Object Slots, camera poses, and diffusion noise; its frame shuffling alternates identity, half-context shifts, and random permutations for consistency.The setup uses separately trained OSRT slots, with slot dropout of 0 for MultiShapeNet and 0.2 for Street View; attention connects frames and processes within-frame features.Source: §B.2.1 · pp. 16–17

B.2.2 3D DIFFUSION MODEL (3DIM)

  • 3DiM generates a target view by conditioning at each denoising step on a randomly selected view from the input set, then adds each generated view to that set.The experiments use the default approximately 471-million-parameter X-UNet and train for 320,000 steps.Source: §B.2.2 · pp. 17–18

B.2.3 SRT & OSRT

  • The object representations come from SRT/OSRT encoders that process input views with a CNN, a Transformer, and Slot Attention; the paper uses 32 slots.SRT and OSRT are separately trained for approximately 4 million steps per dataset.Source: §B.2.3 · p. 18

B.3 COMPUTE AND DATA LICENSES

DORSal training used eight TPU v4 chips, while the appendix also identifies the datasets’ sources and licenses and the ablation figure’s comparisons.

B.3 COMPUTE AND DATA LICENSES

  • DORSal was trained on 8 TPU v4 chips with batch size 8 for approximately one week to reach 1 million steps.MultiShapeNet was generated with Kubric and is under Apache 2.0; Street View imagery and publication permission were obtained from the authors.Source: §B.3 · p. 18
  • Figure 9 compares slot-dropout and OSRT variants, inference guidance weights, and median-filter kernel sizes for scene-edit evaluation..Source: Figure 9 · p. 19

C.1 ABLATIONS

The appendix reports ablation findings, run-to-run variation, cross-view edit consistency, and input-view reconstruction results.

C.1 ABLATIONS

  • Guidance weight 2 is optimal in the reported ablation, while slot dropout slightly harms MultiShapeNet edit metrics but improves Street View edit quality; removing instance supervision also reduces MultiShapeNet editing performance.The Street View model without slot dropout did not produce meaningful scene edits.Source: §C.1 · p. 19

C.2 VARIANCE

  • Across three training reruns, standard errors were approximately 0.004 for LPIPS and 0.2 for PSNR, which did not affect interpretation of the reported results.Source: §C.2 · p. 19

C.3 3D CONSISTENCY

  • 0.702 Edit FG-ARI versus 0.721 2D Edit FG-ARI indicates a small cross-view consistency gap under the reported evaluation.The 2D score averages per-view results and does not penalize inconsistencies between views.Source: §C.3 · p. 19

C.4 REPRODUCING INPUT VIEWS

  • Figure 10 shows additional Street View novel-view results, including input views and a 3DiM version conditioned on 5 ground-truth input views.Source: Figure 10 · p. 20
  • On 16 MultiShapeNet scenes with target views matching inputs, DORSal and sup-OSRT improved PSNR and LPIPS over their novel-view scores; DORSal’s lower LPIPS indicated faithful input reconstruction.The authors interpret this as evidence that DORSal hallucinates less in these instances.Source: §C.4 · p. 20, Table 5 · p. 20

C.5 QUALITATIVE RESULTS

Additional results show DORSal rendering realistic Street View views that adhere to scene content, while slot removal supports meaningful edits with recurring failure modes. With more input views, 3DiM performs differently across MultiShapeNet and Street View.

C.5 QUALITATIVE RESULTS

  • On Street View, DORSal renders realistic novel views that adhere to scene content, unlike 3DiM initialized with five ground-truth views.Figure 11 provides additional MultiShapeNet novel-view examples.Source: §C.5 · p. 20, Figure 11 · p. 21
  • Removing slots produces several meaningful scene edits, but some objects remain unaffected, edits can have side effects, and different slots can produce similar effects.Figures 12–14 show MultiShapeNet and Street View examples, including exhaustive one-slot removals for three Street View scenes.Source: §C.5 · p. 20, Figure 12 · p. 21, Figure 13 · p. 22
  • Objects absent from input views generally cannot be directly edited through slot removal; encoding such imagined objects in slots is proposed as future work.The authors also suggest object supervision as a possible way to address some editing issues.Source: §C.5 · p. 21

C.6 COMPARISON TO 3DIM USING ADDITIONAL INPUT VIEWS

  • With additional ground-truth views, 3DiM performs markedly better on MultiShapeNet but markedly worse on Street View; the authors hypothesize this reflects dataset-specific training performance.They suggest camera gaps and scene complexity challenge 3DiM on Street View, where more diverse input views can yield intermediate views that hurt performance.Source: §C.6 · pp. 21–22, Table 6 · p. 22
Loading 2306.08068v3…