PaperlayerView exact revision on arXiv

Source-linked AI summary

SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos

Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, Thomas Kipf

arXiv:2206.07764v2cs.CVcs.LG

TL;DR

End-to-end object-centric learning has struggled to discover and track objects in complex real-world videos without instance-level supervision. SAVi++ predicts depth from slot-based video representations and applies scaling practices, enabling emergent segmentation and tracking in complex synthetic scenes and Waymo Open driving videos. The paper presents this as evidence that object-centric neural networks can extend beyond simple synthetic environments, while noting important practical scope limitations.

  • Problem

    End-to-end computer vision methods struggle to discover compositional object structure in complex dynamic videos without explicit instance-level supervision.

  • Method

    SAVi++ is a slot-based video model that predicts log-transformed depth signals from its representation while incorporating architecture scaling and data augmentation.

  • Results

    SAVi++ learns emergent object segmentation and tracking in complex videos, including Waymo Open driving videos using sparse LiDAR depth, without direct segmentation supervision.

  • Takeaways & Limitations

    The results provide a proof of concept that end-to-end slot-based models can learn object-centric decomposition in complex real-world videos without detailed human supervision.

  • Takeaways & Limitations

    Waymo Open is relatively structured and vehicle-centric, and SAVi++ relies on first-frame object bounding-box cues in the explored conditional setup.

Abstract

from arXiv · show

The visual world can be parsimoniously characterized in terms of distinct entities with sparse interactions. Discovering this compositional structure in dynamic visual scenes has proven challenging for end-to-end computer vision approaches unless explicit instance-level supervision is provided. Slot-based models leveraging motion cues have recently shown great promise in learning to represent, segment, and track objects without direct supervision, but they still fail to scale to complex real-world multi-object videos. In an effort to bridge this gap, we take inspiration from human development and hypothesize that information about scene geometry in the form of depth signals can facilitate object-centric learning. We introduce SAVi++, an object-centric video model which is trained to predict depth signals from a slot-based video representation. By further leveraging best practices for model scaling, we are able to train SAVi++ to segment complex dynamic scenes recorded with moving cameras, containing both static and moving objects of diverse appearance on naturalistic backgrounds, without the need for segmentation supervision. Finally, we demonstrate that by using sparse depth signals obtained from LiDAR, SAVi++ is able to learn emergent object segmentation and tracking from videos in the real-world Waymo Open dataset.

1 Introduction

Object-centric learning aims to discover persistent entities and their interactions in dynamic scenes without explicit segmentation supervision. SAVi++ addresses limitations of motion-only slot learning by using depth signals and scaling practices to handle complex real-world videos.

  • Motivation: Object-centric representations model scenes as distinct entities with spatiotemporal coherence, interactions, and persistent characteristics.Such representations are motivated by their potential to improve sample efficiency, robustness, visual reasoning, and interpretability.
  • Challenge: Unsupervised discovery of compositional structure in real-world dynamic scenes remains a major computer-vision challenge.Earlier work focused largely on single-frame synthetic RGB images, while motion and depth provide additional perceptual cues.
  • Approach: SAVi++ enhances slot-based video learning by predicting depth signals from object-centric representations and adopting model-scaling improvements.The depth signals can come from RGB-D cameras or LiDAR sensors.
  • Results: SAVi++ is presented as the first end-to-end slot-based model to segment complex naturalistic real-world videos without direct segmentation or tracking supervision.The work also demonstrates emergent object decomposition and tracking using sparse LiDAR depth in Waymo Open driving videos.
  • Results: SAVi++ handles complex multi-object videos with complex shapes and backgrounds, many objects, static and dynamic objects, and static or moving cameras.These capabilities are reported on the multi-object video benchmark.

2 Related work

Related work has explored slot-based, contrastive, heuristic, motion-based, and depth-based approaches to object discovery. SAVi++ builds on these directions while targeting the scaling and supervision challenges of complex real-world multi-object video.

  • Object-centric learning: Slot-based methods use factorized latent spaces and per-object decoders to discover objects, while stronger priors impose constraints on size, presence, or propagation.Examples include IODINE, MONet, Slot Attention, SQAIR, and SCALOR.
  • Object-centric learning: Prior object-centric video methods have generally been unable to scale to complex real-world multi-object video data.Contrastive methods such as GroupViT and ODIN instead target semantic groupings in real-world imagery.
  • Object discovery in driving scenes: Driving-scene discovery methods often obtain pseudo-ground-truth labels with task-specific heuristics before supervised training.Their success therefore depends substantially on the quality of those heuristic labels.
  • Cross-modal learning: Optical-flow targets help object discovery beyond RGB autoencoding but primarily benefit moving objects, while depth targets from simulators or sparse LiDAR provide another training signal.These modalities are part of broader cross-modal approaches to self-supervised object-centric learning.
  • Scaling strategies for vision models: Self-supervised object-discovery methods have often used low-capacity backbones despite the common practice of scaling vision architectures with dataset complexity and applying strong augmentation.SAVi++ relates its scaling strategy to this gap in prior practice.
  • Depth estimation: Future extensions could combine SAVi++ with supervised monocular-depth advances such as ordinal regression, transformer architectures, or more complex instance-wise decoders.These possibilities are identified as directions for future work.

3 Methods

SAVi++ extends slot-based video learning with depth prediction and scaling strategies to represent complex scenes, including static objects and moving cameras. Its recurrent slots are decoded into depth maps and per-slot masks, while training uses dense or sparse depth targets and optional flow.

  • SAVi++: The encoder maps augmented frames to object slots with attention, updates slots recurrently, and decodes each slot into depth and an alpha mask.These per-slot outputs are combined to form the scene representation and yield temporally consistent object segmentation.
  • Motivation: SAVi’s optical-flow objective is limited when objects are static or cameras move, motivating SAVi++’s depth-based objective.SAVi also primarily applies where objects have independent motion and struggles to generalize to moving-camera scenes.
  • SAVi++: SAVi++ improves SAVi by adding depth prediction, encoder and transformer scaling, and stronger data augmentation.The model uses a ResNet34 followed by a transformer encoder and applies Inception-style consistent video cropping.
  • Datasets: The experiments span MOVi-C, MOVi-D, MOVi-E, and Waymo Open, increasing from moving objects to static objects and moving cameras.Waymo Open includes the full set of these scene characteristics.
  • Depth prediction: Depth is used as a prediction target because it avoids optical flow’s difficulty with static objects and moving cameras.SAVi++ can use depth with flow or in isolation; depth is available from RGB-D cameras, LiDAR, or multi-camera estimation.
  • Depth prediction: Sparse LiDAR depth is projected into the camera frame, and pixels without measurements are excluded from the loss.LiDAR point distances are transformed using camera–LiDAR calibration before training.

4 Experiments

Experiments evaluate SAVi++ on increasingly complex synthetic videos and real-world Waymo Open driving sequences, including ablations and comparisons with baselines. SAVi++ improves emergent segmentation and tracking, while depth, scaling choices, conditioning, and remaining scope limitations shape the results.

  • 4.1 Synthetic video: SAVi++ outperforms the BBox copy baseline on all MOVi datasets, demonstrating non-trivial segmentation and tracking capabilities.A learned propagation baseline without visual input fails to generalize to unseen evaluation videos.
  • 4.1 Synthetic video: SAVi++ yields markedly better mIoU than CRW on MOVi-C and MOVi-D, with similar performance on MOVi-E.Unlike SAVi++, CRW propagates pixel-level annotations rather than producing instance-level object segmentations and object representations.
  • 4.1 Synthetic video: SAVi++ improves over SAVi on MOVi-C and overcomes SAVi’s limitations on the harder MOVi-D and MOVi-E datasets.The improvement on MOVi-E raises mIoU from 30.7% to 47.1%.
  • 4.2 Ablation study: Depth targets, a stronger architecture, and data augmentation produce major improvements in emergent segmentation and tracking on complex video sequences.The evaluation specifically examines whether depth signals and scaling strategies improve performance across synthetic scene and camera complexities.
  • 4.2 Ablation study: Training only on depth targets still supports accurate segmentation and tracking, especially on MOVi-D and MOVi-E.On MOVi-E, jointly predicting optical flow is difficult under camera movement, while removing depth targets causes a large mIoU drop.
  • 4.3 Real-world driving data: On Waymo Open, SAVi++ markedly outperforms BBox copy, propagation, and clustering baselines for object tracking, with high bounding-box recall.Unconditional SAVi++ also produces markedly better object decompositions than plain SIMONe, while longer sequences show frequent consistent tracking until objects leave the scene.
  • 4.4 Discussion: The experiments provide a first proof of concept for emergent object-centric decomposition of complex real-world videos with an end-to-end slot-based approach.The authors position this result as evidence that object-centric deep neural networks are not inherently limited to simple synthetic environments.
  • 4.4 Limitations: Practical applicability remains constrained by first-frame bounding-box conditioning, reliance on ground-truth target signals, and Waymo Open’s structured domain.Performance also qualitatively lags behind supervised approaches, and improving temporal consistency—particularly unconditionally—remains future work.

5 Conclusion

SAVi++ uses depth signals in slot-based architectures to learn object tracking and segmentation in complex videos. The results support extending end-to-end object-centric perception beyond simple synthetic environments without detailed human supervision.

  • Depth signals enable emergent object tracking and segmentation in complex video data with slot-based neural architectures.
  • SAVi++ bridges synthetic multi-object video benchmarks and complex real-world driving videos through changes to the SAVi model.
  • Object-centric deep neural networks are not inherently limited to simple synthetic environments, although substantial open challenges remain.

B Additional results

Additional results examine qualitative masking and tracking behavior, synthetic MOVi comparisons, Waymo Open category performance, and robustness to sparse-depth noise.

  • Qualitative results: Unconditional tracking on long Waymo Open videos is less consistent than conditional tracking, with slots also explaining environmental objects or background.The conditional setting is shown for reference.
  • Qualitative results: Masks occupying more than 1300px per frame on average are dropped to improve visualization interpretability.The thresholding heuristic does not affect the conditional model.
  • Quantitative results: SAVi++ performs similarly or worse than SAVi on mIoU and FG-ARI for the simpler MOVi-A and MOVi-B datasets.The authors attribute this mainly to overfitting from the scaling strategy and limited depth benefit when all objects move.
  • Quantitative results: Waymo Open performance is very good for cars and also strong for rare cyclists, suggesting the model is not limited to car-like objects.The reported breakdown covers cars, people, and cyclists.
  • Quantitative results: Emergent tracking performance remains largely unaffected by sparse-depth noise with standard deviations up to σ = 40 cm.These results use Waymo Open sequences evaluated with noisy depth targets.

C Training setup

The training setup uses TPU-based optimization with scheduled learning rates and distinct slot initialization procedures for conditional and unconditional experiments.

  • Optimization: Models are trained for 500k steps, or 300k steps for ablations, on TPU accelerators with batch size 64 using Adam.The learning rate is warmed up to 0.0002 and then cosine-decayed to zero, with gradients clipped at 0.05.
  • Slot initialization: Conditional experiments initialize slots from first-frame object bounding boxes, whereas unconditional experiments use equal amounts of learnable parameter vectors.Bounding boxes provide rough initial cues for which objects slots should bind to.
  • Training signals: Optical-flow targets are converted into three RGB channels when used in experiments.

D.1 SAVi++

SAVi++ retains SAVi’s temporal slot architecture while scaling its encoder, slot-processing dimensions, decoder resolution handling, initialization options, and augmentation pipeline.

  • Architecture: SAVi++ shares parameters across time and uses SAVi-like building blocks, with the same slot initializer and decoder except for the high-resolution variant.
  • Encoder: The encoder uses a ResNet-34 backbone with group normalization and linear positional encodings projected to the feature-map size.The root convolution uses stride 1 except in SAVi++ HR.
  • Corrector/Predictor: Corrector and predictor query/key/value sizes increase to 256, while the predictor MLP hidden layer increases to 1024.The authors report that larger embeddings increase SAVi++ mIoU by a few percentage points.
  • Decoder: The Waymo Open decoder uses an 8 × 12 spatial broadcast grid, while SAVi++ HR adds a stride-2 transposed-convolution layer for 256 × 384 frames.
  • Initializer: Conditional initialization maps first-frame bounding boxes to slot states, while unconditional initialization learns K shared D-dimensional slot states.
  • Data augmentation: Inception-style random crops are resized to target resolutions, with minimum retained frame areas of 20% for MOVi and 75% for Waymo Open.
  • Data augmentation: Sparse Waymo Open LiDAR points are transformed with the crop and resize affine map, and points outside the cropped frame are discarded.

D.2 Baselines

The baselines include SAVi, SIMONe, CRW, bounding-box methods, modality clustering, and a supervised tracking variant, with adaptations and training details reported for comparison.

  • SAVi: The SAVi baseline predicts optical flow but not depth, uses no data augmentation or transformer encoder, and follows SAVi hyperparameters.
  • SIMONe: SIMONe encodes fixed-length video clips into time-shared object latents and per-frame global latents, but cannot run autoregressively.Training and testing must use the same sequence length.
  • CRW: CRW learns grid-structured representations through cycle consistency and typically requires first-frame segmentation labels for tracking.
  • CRW: The CRW implementation follows published training and evaluation practices, using stride-8 ResNet backbones and label propagation from the second-last ResNet stage.
  • CRW: ResNet34 could not train with the CRW cycle-consistency loss, so reported results use ResNet18 instead.The reported settings include six-frame clips, frame skip 1, batch size 16, and 125 training epochs.
  • Bounding-box baselines: Bounding-box copy renders repeated initial boxes as pixel segments, giving later boxes precedence where boxes overlap.
  • Bounding-box baselines: Learned bounding-box propagation uses SAVi++ initialization and prediction to map slots from time t to time t + 1 after receiving first-frame boxes.
  • Modality baseline: The k-Means baseline clusters per-pixel log-depth, optical flow, position, and time features to assess modality-level instance information.

E Datasets

Experiments use synthetic MOVi datasets and real-world Waymo Open videos, spanning increasing scene and camera complexity. The datasets provide different object compositions, motion settings, resolutions, and annotations.

  • Dataset sources: The study uses synthetic MOVi datasets from Kubric and the real-world Waymo Open dataset.Kubric’s dataset-generation pipeline is available under an Apache 2.0 license, while Waymo Open is governed by a non-commercial dataset license.
  • Synthetic MOVi datasets: MOVi-C contains three to ten dynamic household objects against high-resolution HDR-photo backgrounds with a static camera pose.Videos are sampled at 12 fps; models train on six-frame sequences and evaluate on 24-frame sequences.
  • Synthetic MOVi datasets: MOVi-D increases object count with one to three dynamic objects and 10 to 20 static objects while retaining MOVi-C’s camera and sequence settings.Training uses 9.75k videos, evaluation uses 250 videos, and tuning uses a separate 250-video set.
  • Real-world dataset: MOVi-E adds random linear camera movement to MOVi-D’s setting, while Waymo Open contains 20-second front-camera scenes with bounding-box annotations.Waymo videos are sampled at 10 fps and downsampled to 128 × 192, or 256 × 384 for SAVi++ HR; the dataset has 798 training and 202 validation scenes.

F Metrics

The paper next provides a dataset-specific overview of the metrics used to evaluate the experiments.

  • The following subsection gives a detailed overview of the metrics used for each dataset.
  • Metric definitions are organized around the evaluation datasets rather than presented as a single universal measure.
  • The metric overview establishes how quantitative performance will be measured in the subsequent dataset-specific sections.

F.1 MOVi

MOVi evaluation uses ground-truth pixel-level segmentations and focuses on segmentation quality, temporal consistency, and correct use of first-frame bounding-box conditioning.

  • Evaluation setup: MOVi provides ground-truth pixel-level segmentations for directly measuring learned segmentation quality.
  • Evaluation setup: Because SAVi and SAVi++ use first-frame ground-truth bounding boxes for conditional initialization, metrics are measured from the second frame onward.
  • Segmentation metrics: FG-ARI compares predicted and ground-truth masks while ignoring background pixels and is sensitive to temporal mask consistency but not mask ordering.
  • Segmentation metrics: mIoU follows the video Jaccard-Mean metric and measures both segment overlap and whether first-frame conditioning was correctly used.

F.2 Waymo Open

Waymo Open evaluation relies on ground-truth bounding boxes because pixel-level segmentations are unavailable. It measures tracking distance, segment recall, and bounding-box overlap from the second frame onward.

  • Evaluation setup: Waymo Open has ground-truth bounding boxes but no pixel-level segmentation ground truth, requiring alternative quantitative metrics.
  • Evaluation setup: Conditional Waymo evaluation measures metrics from the second frame onward because slots are initialized using conditioning information.
  • Tracking metrics: Center-of-Mass distance measures the average Euclidean distance between predicted-mask centroids and ground-truth box centers.Centroids use valid LiDAR-associated pixels when computing predicted segment locations.
  • Tracking metrics: Bounding Box Recall reports how often any segment is predicted when a valid ground-truth box exists, complementing CoM distance when no matching is used.
  • Bounding-box metrics: Bounding Box mIoU averages per-frame intersection-over-union across corresponding predicted and ground-truth box tracks.Predicted tracks come from a per-slot readout MLP jointly trained with SAVi++ using Huber loss.
  • Model selection: Final Waymo model selection primarily used B. mIoU because it is analogous to mIoU on MOVi.
Loading 2206.07764v2…