MONet: Unsupervised Scene Decomposition and Representation

Christopher P. Burgess; Loic Matthey; Nicholas Watters; Rishabh Kabra; Irina Higgins; Matt Botvinick; Alexander Lerchner

Source-linked AI summary

MONet: Unsupervised Scene Decomposition and Representation

Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, Alexander Lerchner

arXiv:1901.11390v1Jan 22, 2019cs.CVcs.LGstat.ML

TL;DR

Learning useful object-based decompositions of realistic scenes without supervision remains an open challenge. MONet jointly trains a recurrent attention network and VAE to segment and represent scene components, achieving meaningful decomposition and disentangled representations on challenging 3D scenes. Its scope remains limited for more visually complex datasets and unexplored video decomposition.

Problem
Learning object-based representations for realistic visual scenes without supervision remains an open challenge, despite the potential value of common representations for reasoning and transfer.
Method
MONet jointly trains a segmentation network, recurrent attention process, and VAE to decompose scenes into masked components represented in shared latent slots.
Results
MONet learns semantically meaningful scene decomposition, disentangled representations, and completion of partially occluded elements on challenging 3D scenes.
Takeaways & Limitations
Unsupervised decomposition can distill dataset structure into reusable representations across scenes with varying numbers of objects.
Takeaways & Limitations
The study does not address increased visual complexity such as natural or large images with many objects, and does not explore video decomposition.

Abstract

from arXiv · show

The ability to decompose scenes in terms of abstract building blocks is crucial for general intelligence. Where those basic building blocks share meaningful properties, interactions and other regularities across scenes, such decompositions can simplify reasoning and facilitate imagination of novel scenarios. In particular, representing perceptual observations in terms of entities should improve data efficiency and transfer performance on a wide range of tasks. Thus we need models capable of discovering useful decompositions of scenes by identifying units with such regularities and representing them in a common format. To address this problem, we have developed the Multi-Object Network (MONet). In this model, a VAE is trained end-to-end together with a recurrent attention network -- in a purely unsupervised manner -- to provide attention masks around, and reconstructions of, regions of images. We show that this model is capable of learning to decompose and represent challenging 3D scenes into semantically meaningful components, such as objects and background elements.

1 Introduction

MONet addresses the open challenge of learning object-based representations for realistic visual scenes without supervision. It combines segmentation and representation learning to decompose scenes into meaningful components while targeting common, flexible representations and generalization.

Motivation: Object-based scene representation remains difficult for artificial systems despite advances in deep learning.Object perception supports understanding and interaction with the sensed world, but realistic visual scenes remain challenging to represent in terms of objects.
Limitations of prior work: Existing supervised methods require difficult-to-acquire segmentation labels and often learn segmentation without structured object representations.Prior unsupervised approaches were limited to simple visual data, while VAE-based representation methods largely ignored object structure.
Design desiderata: Useful multi-object representations should share a common object space, handle occlusion in 3D scenes, and support variable numbers of objects.The proposed desiderata also include generalizing to novel object counts, feature combinations, and object co-occurrences.
MONet: MONet jointly trains a segmentation network and variational autoencoder to learn image components and their representations.The model is unsupervised and uses decomposition to exploit the efficiency of operating on constituent objects.
Results: MONet achieves state-of-the-art decomposition on non-trivial 3D scenes, including generalization and occlusion-handling, while learning disentangled latent codes.The model is applied across datasets and represents scene elements in a common latent code.

2 Method

MONet decomposes scenes into spatially masked components, models each component with a VAE, and learns the decomposition through a recurrent attention process trained end-to-end. Its objective is motivated by evidence that semantically aligned decomposition improves reconstruction, supporting unsupervised discovery of structured scene elements.

Multi-Object Network: MONet models a scene compositionally by spatially decomposing it into parts and representing those parts independently across slots with a common latent code.An attention module supplies the spatial masks, while a component VAE models each masked part.
Multi-Object Network: The component VAE encodes each masked region and weights its decoder likelihood by the mask, leaving reconstruction outside that region unconstrained.This permits reconstructions to fill in occluded regions; the VAE also models the attention masks.
Multi-Object Network: The recurrent attention network conditions each step on the image and the remaining scope, producing masks while tracking unexplained pixels.The initial scope is s0 = 1, and each subsequent scope excludes the portion assigned to the current attention mask.
Multi-Object Network: The loss combines masked reconstruction likelihood, latent KL divergence weighted by β, and KL divergence between attention and decoded mask distributions weighted by γ.The first two terms derive from the standard VAE loss, while the final term encourages the VAE to model the attention masks.
Exploiting compositional structure: Element masks produce better reconstructions than all-in-one masks, whereas masks from the wrong scene yield higher reconstruction error, higher latent KL, and edge artifacts.These experiments use rendered 3D Objects Room scenes with provided masks to test whether semantically aligned decomposition improves reconstruction.
Exploiting compositional structure: The improved loss under semantically aligned decomposition supports discovering structurally aligned scene decompositions without supervision.The authors attribute the improvement to more efficient use of neural-network capacity when common structure across scene elements can be exploited.

3 Results

MONet decomposes scenes into object and background components using learned attention masks and slot-specific reconstructions. Across synthetic 2D and 3D datasets, it generalizes to varied object counts, occlusion, overlap, and visually similar components.

3.1 Results on Objects Room: MONet learns distinct attention masks for floors, walls, sky, individual objects, and some shadows in Objects Room scenes.Its masked reconstructions are accurate within the attended regions, while unmasked components coherently fill regions occluded by objects.
3.1 Results on Objects Room: The same trained model generalizes at test time to empty rooms, six-object scenes, and four identically coloured overlapping objects using extra decomposition steps.The model was extended to nine slots at test time, although only the first eight were shown.
3.2 Disentangled representations: Single-latent traversals reveal interpretable features that are specific to objects, specific to wall-plus-sky components, shared across element classes, or able to switch between them.These visual patterns were identified by inspection and are described as indicative of disentangling.
3.3 Results on Multi-dSprites: MONet robustly separates individual sprites and backgrounds in Multi-dSprites, including sprites behind multiple occlusions or difficult to distinguish visually.The Multi-dSprites model used five slots and the same basic architecture and hyperparameters as before, aside from the slot count.
3.4 Results on CLEVR: On CLEVR, MONet segments diverse scenes into single objects, reconstructs them with high quality, completes occluded regions, and distinguishes overlapping shapes with similar appearances.The CLEVR setup used eleven slots and handled images containing between 2-10 visible objects per image.

4 Related Work

Related approaches either rely on segmentation supervision, handle only simple scenes, impose restrictive generative assumptions, or abstract away general object representations. MONet instead targets unsupervised decomposition with shared object-wise representations.

4.1 Supervised Approaches: Supervised segmentation methods require pixel-wise labels, while many only segment images rather than learning structured object representations.The text also states that ground-truth segmentations are difficult to acquire for most datasets.
4.1 Supervised Approaches: MONet can use supervised-segmentation architectures such as U-Net as its attention module without requiring a supervised training signal.These architectures provide masks proposed by the attention module, but MONet trains without pixel-wise supervision.
4.2 Unsupervised Approaches: Vision-as-inverse-graphics methods use domain-specific assumptions about latent codes or generative processes, which the text describes as too strong for broad usefulness.Other self-supervised and adversarial approaches are described as lacking general principles for representing objects.
4.2 Unsupervised Approaches: AIR and SQAIR use recurrent object-wise inference but factor representations into what, where, and presence variables and cannot model occluded objects or background pixels.Their additive component interaction is described as restrictive, and they have not been shown to scale to larger numbers of objects.
4.2 Unsupervised Approaches: Earlier unsupervised decomposition methods based on iterative refinement have not been shown to work beyond small, binarized images or videos.These methods draw their appeal from classical clustering and expectation-maximization algorithms.

5 Conclusions and future work

MONet is a compositional generative model for unsupervised scene decomposition and representation learning. It learns semantically meaningful, disentangled components and generalizes across scene complexity, while leaving broader visual settings and video decomposition for future work.

5 Conclusions and future work: MONet performs unsupervised scene decomposition and represents individual components in a common disentangled code.The model is presented as a compositional generative model for scenes containing varying numbers of objects.
5 Conclusions and future work: The model learns masks and reconstructions corresponding to semantically meaningful scene parts, including walls, floors, and individual objects.It also completes partially occluded elements when doing so provides a more parsimonious representation.
5 Conclusions and future work: MONet trained on scenes with 1-3 objects remains accurate with fewer or more objects and novel scene configurations.This supports robustness to variation in scene complexity, at least in the number of objects.
5 Conclusions and future work: The paper identifies downstream use in models that leverage object representations, including reinforcement learning agents.The authors describe robustness to scene complexity as promising for such applications.
5 Conclusions and future work: The method has not been tested on natural images, large images with many objects, or partial scene decomposition.Such settings may require further model development because MONet must explain and decompose the entire scene.
5 Conclusions and future work: Video decomposition and fuller assessment of latent disentanglement remain unexplored.The authors specifically note the issue of reusing latent dimensions between semantically different components.

A Supplementary figures

The supplementary figures provide additional MONet decompositions across Multi-dSprites, Objects Room, and CLEVR scenes. They show slot-wise decompositions, including Objects Room generalization beyond the training distribution.

Supplementary figures: The supplementary figures collect additional decomposition examples from the datasets considered in the paper.They extend the examples shown in the main text.
Supplementary figures: Figure 7 shows all 5 MONet slots on randomly selected Multi-dSprites samples.It uses the same trained model and format as Figure 6.
Supplementary figures: Figure 8 shows all 9 slots for Objects Room, although the model was trained with 7 slots.The left three columns use the training distribution; later columns test extra objects and same-colour objects on a similarly coloured floor.
Supplementary figures: Figure 9 presents unsupervised MONet decompositions of CLEVR scenes using all 11 slots.The figure follows the same trained-model format as Figure 6.

B.1 Component VAE

The component VAE uses a convolutional encoder and spatial broadcast decoder to model each masked scene component. Its implementation fixes latent and loss-scale hyperparameters for the reported experiments.

B.1 Component VAE: The VAE encoder is a CNN followed by a two-layer MLP that parameterizes a 16-dimensional Gaussian latent posterior.The CNN uses 3x3 kernels, stride 2, ReLU activations, and channel sizes (32, 32, 64, 64).
B.1 Component VAE: The broadcast decoder spatially tiles each latent and concatenates coordinate channels before reconstructing image-component and mask distributions.The decoder uses a four-layer CNN with 3x3 kernels, stride 1, 32 output channels, and ReLU activations.
B.1 Component VAE: For MONet, the background and foreground component scales are fixed at σbg = 0.09 and σfg = 0.11, with β = 0.5 and γ = 0.5.These settings distinguish the first background component from the remaining foreground components.

B.2 Attention network

The attention network recurrently allocates scene regions to components by updating a scope mask. A U-Net-based architecture produces attention logits, while implementation details specify skip connections, recurrence, and optimization settings.

B.2 Attention network: At each attention step, the network receives the image and current scope mask in logarithmic units.The scope tracks the image proportion that remains unexplained by previous attention masks.
B.2 Attention network: The attention network uses a U-Net with five downsampling and five upsampling blocks, or six each for CLEVR.Each block combines convolution, instance normalization, ReLU, and factor-of-two nearest-neighbour resizing.
B.2 Attention network: Skip tensors from the downsampling path are concatenated with inputs in the corresponding upsampling blocks.The skip tensors are collected after ReLU activation.
B.2 Attention network: A three-layer MLP forms the non-skip connection between the U-Net paths and matches the final skip tensor dimension.Its hidden layers have size 128, and its reshaped output is concatenated with the final skip tensor.
B.2 Attention network: A final 1x1 convolution produces αk logits, whose log-softmax values update the next attention mask and scope.The recurrence adds log αk and log(1 − αk) to the current log scope.
B.2 Attention network: The experiments initialize weights with a truncated normal, use zero biases, and optimize with RMSProp at learning rate 0.0001 and batch size 64.The implementation is carried out in TensorFlow.

C Datasets

The experiments use synthetic and real-image datasets spanning controlled 2D sprites, rendered 3D rooms, and cropped CLEVR scenes, with varied image resolutions and scene contents.

Objects Room: Objects Room contains 64x64 RGB images of a cubic Mujoco room with coloured walls, floors, objects, and visible blue sky.The camera is randomly positioned on a ring and vertically oriented within a specified range.
Objects Room: Objects Room training scenes contain three objects, with 1-3 objects visible because of camera placement and scene composition.Objects vary in colour, size, shape, and arrangement while avoiding overlap.
Multi-dSprites: Multi-dSprites consists of 64x64 RGB images containing 1-4 randomly coloured dSprites composited with occlusion onto random RGB backgrounds.The sprites are sampled from the 64x64 dSprites dataset and colourised with uniform random RGB values.
CLEVR: CLEVR images are cropped from 320x240 images to 192x192 and resized with bilinear interpolation to 128x128.The crop uses y-coordinates 29 and 221 and x-coordinates 64 and 256.