PaperlayerView exact revision on arXiv

Source-linked AI summary

Object-Centric Learning with Slot Attention

Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, Thomas Kipf

arXiv:2006.15055v2cs.LGcs.CVstat.ML

TL;DR

Learning object-centric representations from raw perceptual input is difficult because common approaches do not capture scene compositionality and often require supervision or task-specific architectures. The paper introduces Slot Attention, which iteratively and competitively maps perceptual features to exchangeable slots that can bind to objects. Across unsupervised object discovery and supervised property prediction, it reports competitive or better performance than relevant approaches, with improved efficiency and generalization to more objects and slots.

  • Problem

    Learning object-centric representations from raw perceptual input is challenging, while typical distributed representations do not capture compositional scene structure and often require supervision or task-specific architectures.

  • Method

    Slot Attention is a differentiable, end-to-end architectural interface that uses iterative competitive attention to map perceptual representations into exchangeable slots.

  • Results

    Slot Attention matches or outperforms relevant approaches on unsupervised object discovery and supervised object property prediction while using less memory and computation.

  • Takeaways & Limitations

    Slots provide a common representational format in which each slot can bind to any input object, supporting systematic generalization to unseen compositions, more objects, and more slots.

  • Takeaways & Limitations

    The experiments use artificially generated datasets under well-controlled settings, and the object specialization is implicit and driven by the downstream task.

Abstract

from arXiv · show

Learning object-centric representations of complex scenes is a promising step towards enabling efficient abstract reasoning from low-level perceptual features. Yet, most deep learning approaches learn distributed representations that do not capture the compositional properties of natural scenes. In this paper, we present the Slot Attention module, an architectural component that interfaces with perceptual representations such as the output of a convolutional neural network and produces a set of task-dependent abstract representations which we call slots. These slots are exchangeable and can bind to any object in the input by specializing through a competitive procedure over multiple rounds of attention. We empirically demonstrate that Slot Attention can extract object-centric representations that enable generalization to unseen compositions when trained on unsupervised object discovery and supervised property prediction tasks.

1 Introduction

The paper addresses the difficulty of learning object-centric representations from raw perceptual input without supervision or task-specific architectures. It introduces Slot Attention as a task-dependent interface that supports unsupervised object discovery and supervised set prediction.

  • Object-centric representations may improve sample efficiency and generalization across visual reasoning, structured environments, multi-agent modeling, and physical simulation.
  • Learning object-centric representations from images or videos is challenging and often requires supervision or task-specific architectures.
  • Slot Attention interfaces perceptual representations such as CNN outputs with a set of task-dependent slot variables.
  • Slots are exchangeable, share a common representational format, and can bind to any object rather than specializing to fixed object classes.
  • The module is evaluated through image reconstruction for unsupervised object discovery and set prediction for supervised object property prediction.

2 Methods

Slot Attention maps input feature sets to exchangeable slots through iterative, competitive attention and recurrent updates. The resulting slots support unsupervised object discovery and supervised set prediction architectures while respecting permutation symmetries.

  • Slot Attention module: Slot Attention maps N input feature vectors to K output vectors called slots, which can describe objects or entities.The module interfaces with perceptual representations and produces task-dependent abstract representations.
  • Slot Attention module: Randomly initialized slots are iteratively refined to bind to parts or groupings of input features, and shared initialization supports testing with different slot counts.The procedure runs for iterations t = 1 . . . T and uses a common learnable Gaussian distribution for initial slot representations.
  • Slot Attention module: Softmax attention is normalized over slots, making slots compete to explain input features before recurrent updates refine their representations.The module uses dot-product attention, weighted-mean aggregation, and a recurrent update function.
  • Slot Attention module: The module is permutation-invariant to input ordering and permutation-equivariant to slot ordering, allowing slots to share a common format and bind to any object.These properties make the module suitable for sets and preserve interchangeable slot roles.
  • Unsupervised object discovery: For unsupervised object discovery, a CNN with positional embeddings and Slot Attention encodes images into slots that a decoder reconstructs into image space.The slots form a representational bottleneck and each slot is decoded independently with shared decoder parameters.
  • Set prediction: For set prediction, Slot Attention produces slot representations, shared per-slot classifiers predict properties, and Hungarian matching handles arbitrary prediction and label order.The encoder follows the same CNN-plus-positional-embeddings architecture used for object discovery.

3 Related Work

The paper situates Slot Attention among approaches for compositional scene modeling, set transformations, iterative routing, memory models, expert systems, clustering, and recurrent attention. Its distinguishing focus is mapping between sets of different cardinality while preserving permutation symmetry and shared slot roles.

  • Object discovery: Slot Attention is related to compositional generative scene models that represent scenes as collections of latent variables with a shared representational format.IODINE is identified as the closest approach, using iterative variational inference to infer object-level latent variables.
  • Neural networks for sets: Unlike many set methods that transform constant-cardinality sets, Slot Attention maps one set to another set with different cardinality while respecting permutation symmetry.The surrounding literature includes set encoding, generation, set-to-set mappings, graph neural networks, and Transformers.
  • Iterative routing: Compared with capsule-style iterative routing, Slot Attention retains permutation symmetry through shared treatment of input-output relationships.The cited capsule variant uses separately assigned input-output pairs and lacks permutation symmetry.
  • Interacting memory models: Slot Attention resembles interacting memory models through recurrent per-slot updates and communication among slots, but emphasizes exchangeable slot representations.The related models use sets of slots and pairwise interactions to reason about input elements.
  • Mixtures of experts: Unlike mixture-of-experts models with expert-specific specialization, Slot Attention uses a common representational format so any slot can bind to any input part.The comparison distinguishes shared slot roles from specialization to particular tasks or object types.
  • Soft clustering: Slot Attention differs from soft k-means by using learned linear projections for dot-product similarity and a parameterized, learnable update function.The related routing procedure treats slots as cluster centroids but changes both similarity and update mechanisms.

4 Experiments

Experiments evaluate Slot Attention on unsupervised object discovery and supervised object property prediction. Across these tasks, it produces competitive object-centric representations, generalizes to more objects or iterations at test time, and is more efficient than IODINE.

  • Experimental setup: The evaluation covers unsupervised object discovery and supervised set-structured object property prediction, using specialized state-of-the-art baselines for each task.The object discovery experiments use CLEVR, Multi-dSprites, and Tetrominoes; supervised prediction is evaluated on CLEVR10.
  • Object discovery: Slot Attention compares favorably with IODINE and MONet for unsupervised object discovery, while a simple ordered Slot MLP cannot model the task’s compositional structure.The unsupervised model is trained from mean squared image reconstruction error and evaluated using ARI against segmentation masks.
  • Object discovery: Segmentation scores improve when test-time iterations increase beyond the trained T = 3, with stronger improvement on CLEVR10 scenes containing more objects.The experiments also report strong segmentation performance when testing on more objects than seen during training.
  • Object discovery: 69.4 ± 0.9% ARI on binarized Multi-dSprites compares with 64.8 ± 17.2% for IODINE and 68.5 ± 1.7% for R-NEM, using shape cues only.On greyscale CLEVR6, Slot Attention achieves 98.5 ± 0.3% ARI.
  • Set prediction: For supervised property prediction, Slot Attention matches or outperforms DSPN, improves generally with more test-time attention iterations, and degrades gracefully when trained on CLEVR6 but tested with more objects.Its attention masks also naturally segment objects despite training only on property prediction.

5 Conclusion

The paper concludes that Slot Attention learns object-centric representations by iteratively grouping perceptual features into slots. Experiments show strong competitiveness across unsupervised scene decomposition and supervised property prediction, while suggesting extensions to other modalities and downstream tasks.

  • Conclusion: Iterative attention learns a grouping strategy that decomposes low-level perceptual input into a set of slot representations.The conclusion describes Slot Attention as a versatile architectural component for object-centric abstract representations.
  • Conclusion: Slot Attention is highly competitive with prior approaches on unsupervised scene decomposition and supervised object property prediction.The conclusion also highlights training speed and memory efficiency for unsupervised scene decomposition.
  • Future work: Future directions include video, graph-node clustering, point clouds, text, speech, reward prediction, visual reasoning, control, and planning.These applications are proposed as natural next steps rather than evaluated results in this paper.

Broader Impact

The broader-impact discussion emphasizes both Slot Attention’s generality and the limited scope of the current evidence. The experiments use artificial, controlled datasets, while attention masks may support more transparent predictions.

  • Scope: The paper evaluates Slot Attention only on artificially generated datasets under well-controlled settings where slots are expected to specialize to objects.Its specialization is implicit and driven by the downstream task.
  • Interpretability: Attention-mask visualizations provide a concrete way to inspect how input features are distributed across slots.The authors present this as a measure for assessing whether the module specialized in unwanted ways.
  • Interpretability: Attention coefficients may support more transparent and interpretable predictions, but their usefulness for explaining overall predictions requires further work.This qualification is especially relevant when input features are not human interpretable.

Supplementary Material for Object-Centric Learning with Slot Attention

The supplementary material documents limitations, ablations, additional results, theoretical proof, and implementation details. It is organized into sections covering these supporting materials.

  • Supplementary organization: Section A discusses limitations and potential future work, while Section B reports an ablation study on Slot Attention.These sections supplement the main paper’s discussion and evaluation.
  • Supplementary organization: Sections C–E provide further qualitative and quantitative results, the proof for Proposition 1, and implementation and experimental-setting details.The supplementary material therefore covers empirical, theoretical, and implementation support.

A Limitations

The paper identifies limitations involving background handling, translation symmetry, task-dependent clustering, and restricted communication between slots.

  • Slot Attention gives backgrounds no special treatment because all slots share the same representational format.The authors suggest assigning a separate background slot as future work.
  • Absolute positional encodings make Slot Attention non-equivariant to translations.Patch-based extraction or relative positional encodings are proposed as extensions.
  • Slot Attention does not inherently identify objects, colors, or spatial regions; the downstream task drives clustering and object specialization.
  • Slots communicate only through input-key attention normalized across slots, limiting explicit slot-to-slot interaction.The paper suggests message passing or self-attention for dynamically interacting object systems and set generation conditioned on a single vector.

B Model Ablations

The ablations identify which Slot Attention design choices improve performance, stability, convergence, and robustness across object discovery and property prediction. Positional information, recurrent updates, LayerNorm, suitable aggregation, learning-rate decay, and multiple attention iterations are beneficial, while several alternatives have task-dependent or limited effects.

  • Aggregation and normalization: Weighted averaging stabilizes training and yields significantly higher ARI and Average Precision than weighted summation, especially at stricter distance thresholds.A weighted sum followed by LayerNorm produces a similar effect.
  • Position embedding: Removing positional information significantly lowers object-discovery ARI and reduces object-position prediction, while categorical property prediction does not require it.The ablation separates spatial prediction from categorical property prediction.
  • Slot initialization: Shared slot initialization is preferable overall: per-slot Gaussian parameters slightly improve supervised prediction but reduce unsupervised performance and prevent adding slots at test time without retraining.The default uses shared learnable initialization parameters across slots.
  • Attention mechanism: Softmax over the slot axis creates competition among slots for explaining input parts, whereas input-axis normalization makes slot attention independent across slots.This competition is central to the module's iterative slot updates.
  • Attention iterations: More than one attention iteration improves performance, but training with more than three slows convergence; using more than three at test time generally improves performance.The authors suggest applying a loss after every iteration as a possible mitigation for slower training convergence.
  • Update and normalization components: LayerNorm and the learned GRU update improve performance, while the residual MLP is optional and has task-dependent effects.LayerNorm particularly improves position prediction in set prediction; the MLP may slow property-prediction convergence but slightly improve object discovery.
  • Optimization: Learning-rate decay clearly benefits both settings, while warmup mainly helps object discovery avoid suboptimal stripe-clustering solutions.The ablation results use reduced validation and training subsets, with nearly identical unsupervised training and held-out validation results.
  • Number of slots: The model is robust to using more training slots than necessary, although extra slots can help property prediction and slightly harm object discovery.A rough upper bound on the number of objects is described as a reasonable slot-selection strategy.

C.2 Set Prediction

Set prediction performance improves with iterative refinement, especially for crowded scenes, while loss design creates a position–attribute trade-off. Slot Attention generally matches or outperforms DSPN despite using a shallower encoder.

  • Qualitative results: Qualitative examples show that property prediction typically becomes more accurate with additional iterations, although position accuracy may decrease.The examples include identical, overlapping objects and crowded scenes with ten objects.
  • Results: Slot Attention generally achieves higher average AP and lower variance than DSPN, despite DSPN using a deeper image encoder.The comparison uses DSPN with 10 iterations and Slot Attention with the same coordinate scale.
  • Results: Prediction errors increase as the number of objects in a scene increases.The results are partitioned by fixed object counts and measure AP for each bin.
  • Results: Increasing test-time iterations from 3 to 5 significantly improves results for scenes with many objects but has little effect on scenes with few objects.The model was trained with 3 iterations.
  • Ablations: Computing the loss at every Slot Attention iteration improves AP at all distance thresholds, whereas DSPN benefits only at small distance thresholds.The authors conjecture that the DSPN pattern reflects an optimization issue.
  • Ablations: Increasing coordinate importance improves AP at smaller distance thresholds but can reduce accuracy on other object attributes.More accurate position modeling therefore introduces a measurable trade-off across prediction targets.

D.2 Proof

The proof establishes Slot Attention’s permutation properties by tracing equivariance and invariance through its constituent operations. Shared per-slot transformations preserve slot equivariance, while aggregation removes dependence on input order after the first iteration.

  • Proof: Shared linear projections and slot-update networks are permutation equivariant because they apply identical parameters independently across elements.The slot update applies the same network to every slot.
  • Proof: Attention scores and softmax remain equivariant because dot products sum over feature dimensions and the softmax preserves the corresponding transformations.The proof uses permutation invariance of summation for the second equality.
  • Proof: The update matrix is invariant to input permutations and equivariant to slot permutations because it sums over input elements.This operation preserves the slot ordering structure while discarding input ordering.
  • Conclusion: Overall, Slot Attention is permutation equivariant with respect to slots and becomes permutation invariant with respect to input order after the first iteration.This follows by combining the equivariant algorithmic steps with the input-invariant update.
  • Architecture: The implementation uses 64-dimensional slots and GRU states, with a 128-unit hidden layer in the residual feedforward block.The encoder and decoders are specified separately for CLEVR, Tetrominoes, and Multi-dSprites.

E.4 Set Prediction Architecture

The set prediction architecture applies shared slot-wise prediction heads and uses normalized training schedules. The appendix also describes MLP alternatives that replace Slot Attention for controlled comparisons.

  • Property prediction: Property prediction applies a shared-parameter MLP independently to each slot and trains the network with Huber loss.The MLP has one 64-unit hidden layer with ReLU activation.
  • Property prediction: The prediction head uses sigmoid outputs because discrete features are one-hot encoded and continuous features are normalized to [0, 1].This output design is documented with the property prediction architecture.
  • Baselines: The Slot MLP baseline replaces Slot Attention and uses the same downstream decoder or classifier.For set prediction, images are resized to 16 × 16 before flattening to reduce MLP parameters.
  • Baselines: The object-discovery Slot MLP baseline uses a deeper MLP and a separate shared-parameter slot-wise MLP because this significantly improves performance over a simpler MLP.The architecture is described in Table 10.
  • Optimization: Training uses linear learning-rate warm-up followed by exponential decay controlled by the decay rate and decay steps.The schedule parameters are listed among the shared and task-specific hyperparameters.
  • Optimization: Hyperparameters were tuned using small subsets of 320 object-discovery images and 500 property-prediction images.The selected learning rates and batch sizes were drawn from a limited candidate set.

E.8 Datasets

The experiments use CLEVR for set prediction and three multi-object datasets for object discovery, with metrics designed for instance-level matching and segmentation quality.

  • Set Prediction: CLEVR scenes contain 3 to 10 objects annotated with position, color, shape, material, and size.Targets are zero-padded and include an indicator distinguishing true objects from padding.
  • Object Discovery: Object discovery uses CLEVR with masks, Multi-dSprites, and Tetrominoes from the Multi-Object Datasets library.The datasets are used for unsupervised object-discovery experiments.
  • Metrics: ARI evaluates clustering similarity between predicted alpha masks and ground-truth instance masks, with 1 indicating a perfect match and 0 chance level.Background labels are excluded from the ARI computation.
  • Metrics: Average Precision counts a detection as correct when object properties exactly match and position falls within a specified distance threshold.An additional confidence score represents the predicted probability that an object is present.
Loading 2006.15055v2…