PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Gap to Real-World Object-Centric Learning

Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, Francesco Locatello

arXiv:2209.14860v2cs.CVcs.LG

TL;DR

Unsupervised object-centric methods have been limited on complex real-world images, often requiring motion or depth. DINOSAUR reconstructs self-supervised features through slot-based representations, achieving strong results on simulated data and scaling to real-world benchmarks. The approach is competitive with more involved computer-vision pipelines, while its grouping granularity remains unresolved.

  • Problem

    Existing unsupervised object-centric methods struggle with complex data, and methods using motion or depth cannot train on most real-world image datasets.

  • Method

    DINOSAUR groups encoded image features into slots and trains them to reconstruct features from self-supervised pretraining instead of reconstructing pixels.

  • Results

    DINOSAUR is the first image-based fully unsupervised approach reported to scale to real-world data, with significant improvements on simulated and real-world datasets.

  • Takeaways & Limitations

    DINOSAUR achieves competitive performance with more involved computer-vision methods on object localization, segmentation, and scene decomposition benchmarks.

  • Takeaways & Limitations

    The method mixes semantic and instance-level grouping, and a fixed slot count can be inappropriate for controlling decomposition detail.

Abstract

from arXiv · show

Humans naturally decompose their environment into entities at the appropriate level of abstraction to act in the world. Allowing machine learning algorithms to derive this decomposition in an unsupervised way has become an important line of research. However, current methods are restricted to simulated data or require additional information in the form of motion or depth in order to successfully discover objects. In this work, we overcome this limitation by showing that reconstructing features from models trained in a self-supervised manner is a sufficient training signal for object-centric representations to arise in a fully unsupervised way. Our approach, DINOSAUR, significantly out-performs existing image-based object-centric learning models on simulated data and is the first unsupervised object-centric model that scales to real-world datasets such as COCO and PASCAL VOC. DINOSAUR is conceptually simple and shows competitive performance compared to more involved pipelines from the computer vision literature.

1 INTRODUCTION

Object-centric learning could support robust, compositional perception and reasoning, but existing unsupervised methods struggle beyond simple synthetic scenes. Motion or depth can help grouping, yet these signals are unavailable in most real-world image datasets; DINOSAUR instead reconstructs self-supervised features.

  • Object-centric representations may improve generalization by aligning visual decomposition with physical-world causal mechanisms.They can also support robustness to distribution shifts, reasoning, and control.
  • Existing reconstruction-based object-centric methods succeed on simple synthetic datasets but fail to scale to complex synthetic or real-world data.Their slot bottlenecks group repeating visual patterns, while low-level cues such as color are insufficient for more complex scenes.
  • Motion and depth provide useful grouping signals, but most real-world image datasets lack these cues or annotations.This limits the applicability of approaches that depend on auxiliary information.
  • DINOSAUR reconstructs self-supervised features with high within-object homogeneity, adding an inductive bias without external motion or depth signals.The authors report that combining feature reconstruction with Slot Attention significantly outperforms other image-based methods and bridges the real-world gap.

2 RELATED WORK

Related work spans end-to-end object-centric models, increasingly challenging synthetic benchmarks, and annotation-free computer-vision tasks. These areas motivate evaluating unsupervised object discovery on more complex data and established real-world benchmarks.

  • Earlier object-centric models use architectural bottlenecks to discover repeating patterns across closed-world datasets and represent them as object entities.These bottlenecks may be semantic, discrete, or set-valued.
  • ClevrTex and MOVi increase synthetic-scene complexity, exposing poor performance from previous unsupervised object-centric methods.The paper characterizes MOVi as containing highly realistic objects with complex shape and appearance.
  • Unsupervised semantic segmentation and object localization provide established real-world benchmarks related to unsupervised object discovery.The paper evaluates its method on these computer-vision tasks.

3 METHOD

DINOSAUR uses an autoencoder-like pipeline that groups encoded features into slots and reconstructs self-supervised features rather than original images. It combines frozen pretrained representations, Slot Attention, and either independently decoded or autoregressively decoded feature reconstructions.

  • DINOSAUR extracts input features, groups them into slots, and decodes a target signal from those latent vectors.This follows the standard encoder-grouping-decoder structure of object-centric models.
  • A frozen DINO ViT produces patch features, which are encoded and grouped by Slot Attention before feature reconstruction.The encoder can use a ResNet or DINO ViT.
  • The decoder reconstructs self-supervised features instead of the original image inputs.The objective is intended to provide a higher-level signal than pixel reconstruction.
  • ImageNet pretraining transfers teacher knowledge from a larger image collection to the datasets being analyzed.The paper uses ImageNet for pretraining and frames the loss as a student-teacher distillation process.
  • DINO, MoCo-v3, MSN, and MAE all work reasonably well for real-world object grouping, while DINO is used primarily for performance and accessibility.The paper experiments with ResNet and Vision Transformer features.
  • Slot Attention iteratively lets slots compete for encoder features to produce K slot vectors.It begins from randomly sampled initial slots and largely retains the original Slot Attention formulation.
  • The MLP decoder processes each slot independently, whereas the Transformer decoder autoregressively reconstructs features jointly across slots.The MLP design is heavily parallelizable; the Transformer can maintain global reconstruction consistency but has training drawbacks.
  • Evaluation uses decoder or attention masks, with low-resolution feature-reconstruction masks bilinearly resized before comparison with ground truth.The Transformer decoder lacks an alpha mask, so alternative attention masks are used.

4 EXPERIMENTS

DINOSAUR is evaluated on synthetic and real-world object discovery, comparisons with computer-vision methods, and analyses of its reconstruction targets, encoders, and decoders. Across these experiments, it extends object-centric learning to unconstrained real-world images while revealing decoder- and feature-dependent grouping behavior.

  • Experimental setup: The experiments evaluate object discovery on two synthetic and two real-world image datasets, using FG-ARI and mBO-based metrics.The study also compares DINOSAUR with Slot Attention, SLATE, K-Means, and regular block masks.
  • Synthetic datasets: DINOSAUR achieves good object-discovery performance on MOVi-C and MOVi-E, while Slot Attention and SLATE perform similarly to naive block masks and worse than K-Means.The synthetic evaluation uses challenging scenes with realistic objects and complex appearance.
  • Real-world datasets: DINOSAUR is the first reported fully unsupervised object-centric model to succeed on unconstrained real-world images, capturing objects across varied sizes, appearances, and shapes.Slot Attention degrades to non-semantic grouping, while SLATE remains below the K-Means baseline and requires early stopping.
  • Computer-vision comparisons: On computer-vision benchmarks, DINOSAUR is competitive with more involved methods: comparable for object localization, behind state of the art for segmentation, and close to state of the art for scene decomposition.Its evaluated masks are 14 × 14, whereas stronger segmentation methods use additional training and original-resolution masks.
  • Analysis: Image reconstruction alone remains insufficient for semantic grouping, even with frozen or finetuned semantic ViT features; the resulting models produce striped masks or fail to yield meaningful objects.Training a scaled-up Slot Attention model from scratch diverges in the tested setting.
  • Analysis: Self-supervised targets from DINO, MoCo-v3, MSN, and MAE all perform well for object discovery, indicating that the feature-reconstruction signal generalizes across pretraining schemes.ResNet34 encoders trained from scratch perform similarly overall to pretrained ViT encoders, supporting feature reconstruction as the key component.
  • Analysis: Decoder choice creates a grouping trade-off: MLP decoders yield higher ARI and better instance grouping, whereas Transformer decoders yield higher mBO and tighter, more semantic masks.The Transformer decoder can group semantically related instances into one slot, while the MLP decoder separates instances more effectively.

5 CONCLUSION

DINOSAUR establishes image-based, fully unsupervised object-centric learning that scales to real-world data, while identifying unresolved challenges in grouping semantics and decomposition detail.

  • The experiments show significant improvements over previous approaches on simulated and real-world data, with competitive performance against more involved computer-vision pipelines.
  • The approach mixes semantic and instance-level grouping, while decoder type influences this behavior and a fixed slot count can produce inappropriate decomposition detail.
  • Future work includes testing learned slot representations on robustness, generalization, downstream usefulness, and compositional image generation.
  • DINOSAUR combines self-supervised dense representations with joint grouping and region-representation learning for unsupervised semantic segmentation.
  • Video-based STEVE and SAVi++ do not outperform image-based methods, and both show a large gap to DINOSAUR.

B.2 RESULTS ON THE KITTI DRIVING DATASET

DINOSAUR is evaluated on real-world driving images and related unsupervised vision benchmarks. It outperforms a motion-supervised method on KITTI foreground grouping, while revealing difficulties with small objects, scale variation, and high-resolution processing.

  • KITTI dataset and setup: 95 869 training images were sampled from 147 KITTI videos, with evaluation on 200 independently annotated frames.Images were evaluated at 1242 × 375 pixels, matching the comparison method.
  • KITTI evaluation: KITTI processing partitions each image into four nonoverlapping 375 × 375 windows, producing 4 · K masks that are spatially reassembled.Padding is added before processing and removed afterward.
  • KITTI results: DINOSAUR clearly outperforms Bao et al. (2022) on KITTI FG-ARI despite their method using motion supervision.The model used K = 9 slots during training and 36 slots for evaluation.
  • KITTI limitations: Visual inspection shows that foreground cars are captured, but small background objects are often grouped with non-objects such as houses.This yields relatively high FG-ARI but relatively low mBO because the metrics weight object scales differently.
  • COCO downstream evaluation: On COCO object-property prediction, DINOSAUR achieves 25% higher class accuracy than Slot Attention and exceeds it on y-coordinate prediction.Its x-coordinate performance is similar to Slot Attention, while Slot Attention drops on the y-coordinate.
  • Computer-vision benchmarks: DINOSAUR reaches competitive or better results on established object-localization and semantic-segmentation benchmarks, including PASCAL VOC and COCO.The broader comparison includes object localization, object segmentation, and scene decomposition.

C.2 UNSUPERVISED SEMANTIC SEGMENTATION

This section evaluates DINOSAUR for unsupervised semantic segmentation and scene decomposition. The method is competitive on coarse scene categories and improves over STEGO on the full 171-class setting, while STEGO does not satisfactorily solve object discovery on MOVi.

  • Task definition: Unsupervised semantic segmentation assigns every pixel a semantic class without direct correspondence between predicted labels and ground-truth classes.Successful models must consistently cover the same semantic content across images.
  • Datasets and tasks: DINOSAUR evaluates foreground-only object segmentation on PASCAL VOC and COCO, and foreground-plus-background scene decomposition on COCO-Stuff.COCO-Stuff is evaluated with 27 and 171 semantic categories.
  • Semantic-label construction: DINOSAUR converts slot representations into semantic labels by clustering slot-associated features with K-Means and matching clusters to classes using Hungarian matching.The clustered features combine slot masks with pretrained ViT features.
  • Scene decomposition: DINOSAUR is competitive with STEGO on the 27-class scene-decomposition setting.The comparison uses mIoU-based semantic segmentation evaluation.
  • Scene decomposition: 3.2 IoU points is DINOSAUR’s improvement over STEGO on the novel 171-class scene-decomposition setting.The result is presented as evidence that DINOSAUR handles the full complexity of real-world scenes somewhat better.
  • STEGO on MOVi: STEGO performs significantly worse than all object-centric methods on the MOVi object-discovery evaluation.Its clusters often merge several objects, producing cluttered and less geometrically consistent masks.

D ADDITIONAL ANALYSIS

The additional-analysis section surveys experiments on decoder choices, encoder pretraining, slot counts, and related design questions. It also includes supplementary localization, segmentation, and pretraining comparisons.

  • Analysis overview: Additional analyses examine decoder behavior, pretraining schemes, encoder choices, slot counts, decoder scale, and mask quality.The section includes comparisons involving object localization, semantic segmentation, and object discovery.
  • Supplementary questions: The supplementary experiments include a specific study of whether DINOSAUR performs semantic segmentation on MOVi.This question is listed as Sec. D.7.
  • Decoder analysis: Table 10 reports decoder-analysis results for object localization and unsupervised semantic segmentation using ViT-B/16 encoders and task-specific slot counts.The table reports mean ± standard deviation over 3 seeds.
  • Decoder analysis: The supplementary material separately analyzes poor Transformer-decoder performance with larger ViTs and compares masks from Slot Attention and the Transformer decoder.These analyses are listed as Secs. D.5 and D.6.

D.1 CHOICE OF PRE-TRAINING METHOD

The pretraining analysis finds that self-supervised features generally support real-world object grouping, whereas supervised ImageNet features are worse. It also shows that feature-reconstruction details and encoder choice affect performance.

  • Pretraining schemes: All tested self-supervised schemes perform well for object discovery, while supervised ImageNet classification provides a worse feature source.The specific self-supervised method matters less than using self-supervised training for real-world grouping to emerge.
  • Feature extractors: Larger ViT architectures generally yield better results, while ResNet features show high ARI but poor mBO.Visual inspection indicates that ResNet-trained models mix semantic and geometric grouping.
  • Decoder interaction: 42.3 ARI is achieved by MAE with the MLP decoder on COCO object discovery, despite MAE being worst among the Transformer-decoder self-supervised methods.This indicates that interactions between pretraining and feature reconstruction remain consequential.
  • Encoder training: Training an encoder from scratch is a viable alternative to a pretrained frozen ViT and sometimes improves performance.The authors attribute this partly to domain adaptation and, on COCO, possibly to higher ResNet spatial resolution.
  • Computational trade-offs: Frozen encoders have computational advantages because they require no backward pass and only one encoder forward pass.The comparison also finds pretrained ResNet features are a worse reconstruction target than ViT features.

D.3.1 TRAINING SENSITIVITY

Training performance is sensitive to the number of slots, especially on synthetic scenes, while real-world performance remains stable across a broader range. Slot count also trades off instance-level and class-level grouping.

  • MOVi-E: Changing the number of training slots greatly impacts MOVi-E performance because object splitting strongly affects ARI.Using the maximum number of slots covers crowded scenes but impairs performance on scenes containing fewer objects.
  • PASCAL VOC 2012 and COCO: On PASCAL VOC 2012 and COCO, DINOSAUR performs well across a range of slots when given a sufficient minimum.Slot Attention fails to produce meaningful results across all tested slot counts.
  • PASCAL VOC 2012 and COCO: COCO has different slot-count sweet spots for instance-level segmentation at 9–11 slots and class-level segmentation at 7 slots.Visual inspection indicates that the optimal slot count depends on scene complexity.
  • PASCAL VOC 2012 and COCO: Training with a fixed dataset-level slot count optimizes results for the average scene rather than each image’s visual complexity.The paper identifies handling variation in complexity and instance count as a future-work question.
  • PASCAL VOC 2012 and COCO: The number of slots creates a small trade-off between instance-level segmentation with more slots and class-level segmentation with fewer slots.This trend is reported for COCO object discovery using the ViT-B/16 encoder and Transformer decoder.

D.3.2 TEST-TIME ROBUSTNESS

DINOSAUR is fairly robust to changing the number of slots at test time across synthetic and real-world datasets. However, decoder capacity, slot dimensionality, and mask source affect performance, and grouping behavior differs between MOVi and real-world data.

  • Test-time slot count: DINOSAUR generalizes well when applied with a different number of slots at test time across MOVi-E, PASCAL VOC 2012, and COCO.Changing slots can even improve performance when the new configuration better matches object scales and dataset annotations.
  • Test-time slot count: Test-time slot changes can substantially improve or reduce performance depending on the training configuration and the object scales it emphasizes.Examples include improvement on MOVi-E from 11 to 24 slots and degradation from 24 to 11 slots.
  • Decoder and slot dimensionality: The MLP decoder has higher ARI, whereas the Transformer decoder has higher mBO on COCO object discovery.Scaling the MLP provides slight ARI benefits, while Transformer scaling has little impact overall.
  • Decoder and slot dimensionality: Transformer-decoder performance degrades below the block masks baseline at overly small or large slot dimensionalities.The analysis identifies instabilities associated with decoder capacity and slot dimensionality.
  • Mask source: Decoder attention masks consistently outperform Slot Attention masks across tasks, especially for segmentation.The comparison covers COCO object segmentation and Stuff-27 and Stuff-171 scene decomposition.
  • Grouping behavior: On MOVi-C variants containing same-category or duplicate objects, results remain similar, indicating that DINOSAUR does not apply semantic grouping on MOVi.The variants isolate whether repeated semantic categories or identical instances drive grouping.

E.1 ARCHITECTURE AND HYPERPARAMETERS

The appendix specifies DINOSAUR’s encoder, slot-attention, decoder, mask-evaluation, and experiment settings. The architecture reconstructs feature representations using either an MLP or autoregressive Transformer decoder.

  • Encoders: DINOSAUR uses Vision Transformer or ResNet34 encoders, with ViT configurations varying in patch size, token dimensionality, and attention heads.The ResNet34 variant uses stride-one initial convolution, group normalization, and a 512-dimensional 28 × 28 feature map for 224 × 224 images.
  • Slot Attention: Slot Attention transforms input features into K randomly initialized slots of dimensionality Dslots through normalization and multilayer perceptrons.The module follows the original Slot Attention formulation with stability-oriented scaling and epsilon additions.
  • Experimental settings: Experiments use dataset-specific slot counts and compare reconstruction targets, self-supervised pre-training methods, decoders, and downstream predictors.The appendix lists settings for MOVi-C, MOVi-E, PASCAL VOC 2012, COCO, and KITTI.
  • Decoders: The MLP decoder predicts feature dimensions plus an alpha value after broadcasting slots across image patches and adding positional encodings.The decoder uses four ReLU layers, with hidden sizes of 1024 for MOVi and 2048 for COCO.
  • Decoders: The Transformer decoder autoregressively reconstructs patch features in raster order while conditioning on the Slot Attention outputs.Evaluation uses the final decoder block’s cross-attention weights as slot masks, resized and converted to hard masks with arg max.

E.2 BASELINES

The baseline appendix describes fixed block masks, DINO K-Means, Slot Attention, SLATE, STEVE, SAVi++, and STEGO. It also records implementation choices and a training-time caveat for SLATE.

  • Non-learning baselines: Block-mask baselines divide images into equal rectangular regions, using dataset-specific mask counts for MOVi-C, MOVi-E, PASCAL VOC 2012, and COCO.The construction uses two, three, or four columns depending on the requested number of masks.
  • Non-learning baselines: DINO K-Means clusters patch tokens from the final DINO ViT block and treats each resized cluster as an object instance.The experiments use ViT-B/8 for MOVi and ViT-B/16 for PASCAL VOC 2012 and COCO.
  • Learned baselines: Slot Attention, SLATE, STEVE, SAVi++, and STEGO are evaluated using official or adapted implementations with dataset-specific configurations.SAVi++ uses both image and optical-flow reconstruction, while STEGO settings vary in cluster counts and training parameters.
  • Learned baselines: SLATE’s performance metrics degraded significantly after some training time, so evaluation used the training point with the best observed performance.This introduces an early-stopping choice into the reported baseline comparison.

F DATASET AND EVALUATION SETTINGS

The work evaluates scene decomposition, object discovery, segmentation, and localization across synthetic and real-world datasets using specified preprocessing, splits, and mask-based metrics. Additional qualitative masks vary slots, encoders, decoders, and pre-training schemes.

  • MOVi: MOVi is converted from video to images by sampling nine random frames per clip, with all frames from 250 clips used for evaluation.This yields 87,633 training images for MOVi-C and 87,741 for MOVi-E; evaluation uses the validation split rather than out-of-distribution test data.
  • Tasks and datasets: The evaluation covers multiple tasks and datasets, with training performed using only images and no labels.Table 21 provides the task–dataset overview, while the dataset descriptions specify the training and evaluation splits.
  • PASCAL VOC 2012: PASCAL VOC 2012 training uses 10,582 trainaug images, while object localization is evaluated on the 11,540-image detection trainval split.Unsupervised segmentation uses 20 object classes plus background, and unlabeled pixels are ignored for object discovery and segmentation.
  • COCO: COCO training uses 118,287 images and validation uses 5,000 images, with instance masks converted to segmentation masks by per-pixel class arg max for object discovery.Overlapping instances and crowd annotations are excluded from the relevant metric evaluation.
  • Preprocessing: Evaluation preprocessing resizes images so the minor axis is 320 pixels and uses 320×320 center crops for COCO-Stuff, PASCAL VOC, and COCO, while MOVi uses full 128×128 images.Object discovery on PASCAL VOC and COCO follows the same settings as scene decomposition; MOVi uses full-resolution inputs at 128×128.
  • Metrics: FG-ARI evaluates foreground pixels using object or instance masks, whereas mBO averages IoU after assigning each ground-truth object mask its highest-overlap prediction.Unlabeled pixels and overlapping instance-mask pixels are excluded from the relevant metric calculations.
  • Additional examples: Qualitative mask examples vary settings across MOVi-C, MOVi-E, PASCAL VOC 2012, and COCO, including slot counts, encoder types, decoder types, and pre-training schemes.Examples include 11 or 24 slots on MOVi-E, six slots on PASCAL VOC, and seven slots on COCO.
Loading 2209.14860v2…