Source-linked AI summary
SCALOR: Generative World Models with Scalable Object Representations
Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, Sungjin Ahn
TL;DR
SCALOR addresses the difficulty of learning object-oriented representations in videos with many objects and dynamic backgrounds. It uses parallel discovery and propagation with proposal rejection, and models scenes containing up to a hundred objects, including natural scenes with moving objects.
Problem
Unsupervised sequential object-oriented representation learning has mainly been demonstrated on scenes with only a few objects, limiting evidence for crowded dynamic scenes.
Method
SCALOR is a probabilistic generative world model that parallelizes object discovery and propagation, rejects overlapping proposals, and models dynamic backgrounds.
Results
SCALOR models scenes with up to a hundred objects and is demonstrated on natural scenes containing tens of moving objects alongside dynamic backgrounds.
Takeaways & Limitations
SCALOR extends unsupervised object representation learning from sparse synthetic scenes to crowded videos with complex dynamic backgrounds and natural moving-object scenes.
Takeaways & Limitations
SCALOR's standard Gaussian zwhat prior makes multimodal object appearance difficult to model, especially for partially observed boundary objects.
Abstract
from arXiv · showhide
Scalability in terms of object density in a scene is a primary challenge in unsupervised sequential object-oriented representation learning. Most of the previous models have been shown to work only on scenes with a few objects. In this paper, we propose SCALOR, a probabilistic generative world model for learning SCALable Object-oriented Representation of a video. With the proposed spatially-parallel attention and proposal-rejection mechanisms, SCALOR can deal with orders of magnitude larger numbers of objects compared to the previous state-of-the-art models. Additionally, we introduce a background module that allows SCALOR to model complex dynamic backgrounds as well as many foreground objects in the scene. We demonstrate that SCALOR can deal with crowded scenes containing up to a hundred objects while jointly modeling complex dynamic backgrounds. Importantly, SCALOR is the first unsupervised object representation model shown to work for natural scenes containing several tens of moving objects.
1 INTRODUCTION
Unsupervised object-oriented representation learning must scale to crowded, temporally evolving scenes while modeling both foreground objects and dynamic backgrounds. SCALOR addresses these challenges through parallel processing and achieves substantially broader scene coverage.
- Unsupervised structured representations can provide interpretability, sample efficiency, reasoning, causal inference, compositionality, and transferability.Temporal modeling is also essential because objects, agents, and spaces evolve over time.
- Scalability is a primary limitation because prior models process objects sequentially and generally handle only scenes with a few objects.This limits their applicability to complex natural scenes and restricts the search capacity available for object attention.
- Dynamic-background modeling is another limitation: prior object-oriented models generally operate on scenes without backgrounds.Natural scenes commonly contain dynamic backgrounds alongside foreground objects.
- SCALOR parallelizes propagation and discovery, reducing per-image processing complexity from O(N) to O(1), where N is the number of objects.The model is designed as a probabilistic generative world model for sequential object-oriented representation learning.
- SCALOR is demonstrated on natural-scene videos containing tens of moving objects together with a dynamic background.
- SCALOR improves scalability by two orders of magnitude in object density and applies to nearly a hundred objects with more efficient computation than SQAIR.The contribution includes a parallel proposal–rejection process and dynamic-background modeling.
2 PRELIMINARIES: SEQUENTIAL ATTEND INFER REPEAT (SQAIR)
SQAIR is a probabilistic temporal generative model that represents video frames with object-wise latent variables and supports objects entering or leaving sequences. Its sequential propagation and discovery, together with absent background variables, limit scalability and natural-scene modeling.
- SQAIR represents each frame using latent variables for the objects present at that time.Each object latent variable includes existence, pose, and appearance factors.
- SQAIR models objects appearing or disappearing through propagation of existing objects and discovery of newly introduced objects.Objects not propagated can be deleted, while newly discovered objects are introduced after deciding how many to add.
- SQAIR has no latent variables for the background, so its representation is limited to foreground objects.
- SQAIR is trained with variational inference using an importance-weighted autoencoder objective.VIMCO handles discrete random variables, while the reparameterization trick is used for continuous variables.
- SQAIR processes objects sequentially with RNN-based propagation and discovery, yielding time complexity O(|O_t|) per step.Sequential discovery can fail beyond a few objects, and the model lacks background dynamics needed for natural scenes.
3 SCALOR
SCALOR combines parallel propagation and proposal-rejection discovery with foreground-background modeling to represent videos containing many objects and dynamic backgrounds.
- SCALOR represents each object with presence, appearance, position, and depth factors, using depth to help handle occlusion.
- SCALOR decomposes generation into propagation, discovery, background transition, and rendering modules.
- Propagation is fully parallel, unlike SQAIR’s sequential object-wise propagation.
- Proposal-rejection proposes one object latent variable per H × W grid cell and rejects proposals overlapping propagated-object masks beyond threshold τ.
- The posterior discovery process computes proposals over all spatial feature cells in parallel, improving scalability but creating propagation collapse when discovery dominates.
- The background posterior conditions on the input image and existing foreground objects so remaining image regions can be explained by the background module.
4 RELATED WORK
Related work includes object-oriented scene models with different approaches to temporal modeling, attention, detection, and background decomposition. SCALOR is positioned as a concurrent scalable sequential model distinguished by proposal-rejection and background modeling.
- AIR and SPAIR decompose scenes into latent object appearance, position, and size, while SPAIR uses spatially invariant attention for local features.
- Figure 2 compares SCALOR and SQAIR using tracking accuracy, object count, and reconstruction error.
- SPACE combines foreground object detection with background mixture decomposition and parallelizes latent inference.
- DDPAE models objects with appearance and position vectors while sharing an appearance representation across time-steps.
- A concurrently developed model similarly emphasizes scalability through parallelized SPAIR and sequential modeling, whereas SCALOR adds proposal-rejection and background modeling for complex natural scenes.
5 EXPERIMENTS
SCALOR is evaluated on synthetic scenes, dynamic backgrounds, future generation, crowded natural video, and ablations targeting tracking and scalability. Across these settings, it tracks many objects, separates foreground from background, generates future frames, and remains effective on natural pedestrian footage.
- Experimental setup: SCALOR is evaluated on synthetic MNIST/dSprites scenes and natural CCTV footage, covering detection, tracking, generation, and generalization.The experiments include quantitative comparisons with state-of-the-art baselines.
- Large-scale synthetic scenes: The five synthetic density settings range from Very Low Density with 2–4 objects to Very High Density with 90–110 objects.The average visible-object counts range from 2.9 to 90 across the settings.
- Large-scale synthetic scenes: SCALOR outperforms SQAIR in the tested low-density settings and maintains relatively high precision-recall in scenes containing about 100 objects.SQAIR could not be made to work in the higher-density settings, while SCALOR’s CountMAE was lower and its tracking ability was not significantly impeded as object count increased.
- Tracking and representations: SCALOR identifies newly introduced objects, propagates existing tracks, and infers consistent object representations despite substantial occlusion.The qualitative results show discovery, propagation, and object-wise rendering across dSprites and MNIST sequences.
- Dynamic background: With dynamic backgrounds, SCALOR decomposes videos into foreground objects and background while achieving tracking performance comparable to the no-background setting.The background module models background dynamics and can also model whole-environment dynamics conditioned on foreground latents.
- Future generation: SCALOR performs conditional generation from five observed frames and generation from scratch by sampling objects from a discovery prior and propagating them forward.The experiments separately examine future image/background generation and fully generated sequences.
- Natural-scene evaluation: On the Crowded Grand Central Station dataset, SCALOR maintains temporal trajectories, reconstructs background, and produces pedestrian detection, tracking, and instance segmentation.The qualitative evaluation includes extracted object trajectories and tracking-ID-colored segmentations.
6 CONCLUSION
SCALOR is a probabilistic generative world model for crowded dynamic scenes, using parallel discovery-propagation and proposal-rejection to scale object representations while modeling dynamic backgrounds.
- SCALOR models crowded environments containing dynamic objects and backgrounds.
- Parallel discovery-propagation and proposal-rejection mechanisms increase capacity from a few objects to up to a hundred.
- Propagation inference processes the image sequence and maintains object states across time.
- Discovery-proposal rejection accepts or rejects proposed objects after comparing them with propagated objects.
- The background module combines inferred foreground objects and masks with a separately decoded background to reconstruct the current frame.
B ADDITIONAL QUANTITATIVE RESULT
The additional quantitative results evaluate SCALOR against baselines using tracking, bounding-box precision and recall, reconstruction error, and negative log-likelihood.
- Table 1 reports Multi Object Tracking Accuracy and bounding-box precision-recall across SCALOR’s experimental settings.
- Table 2 compares SCALOR with SQAIR and VRNN using reconstruction error and negative log-likelihood.
C.1 FREQUENT DENSE DISCOVERY
The frequent dense discovery experiment tests whether SCALOR can discover many objects introduced at multiple time-steps, using discovery and propagation reconstructions and bounding boxes.
- Figure 9 compares inferred bounding boxes with discovery bounding boxes, discovery reconstruction, and propagation reconstruction.
- 10–15 objects are introduced at the first, fifth, and ninth time-steps, respectively.
- SCALOR discovers many newly introduced objects in each frame.
- Figure 10 presents inferred, overall, discovery, and propagation bounding boxes and reconstructions for a very high-density setting.
- The very high-density experiment places 90–110 objects in the environment, with around 90 visible at each time-step on average.
C.3 ABILITY TO HANDLE OVERLAP AND OCCLUSION
These experiments examine SCALOR under stronger overlap, occlusion, longer sequences, unseen shapes, and larger object counts.
- Overlap and occlusion: More aggressive object motion increases the frequency of overlap and occlusion.
- Overlap and occlusion: SCALOR preserves object identities when objects overlap in the highly occluded MNIST setting.
- Generalization: The longer-sequence experiment compares bounding boxes from the first and last 10 time-steps.
- Generalization: The generalization experiment tests unseen shapes and a larger number of objects.
- Generalization: Models trained on 10-step trajectories are tested on 20-step trajectories, and models trained on 15–25 objects are tested on 50–60 objects.
D EXPERIMENT DETAIL AND ADDITIONAL QUALITATIVE RESULTS ON GRAND CENTRAL STATION DATASET
The Grand Central Station experiments use long natural-scene sequences to provide qualitative evidence for reconstruction, background extraction, object segmentation, tracking, and conditional generation. Additional figures visualize these components and generated future frames.
- Experimental setup: 400k frames were created from spatially split natural-scene video, with 360k for training and 40k for testing.Sequences contain 10 resized 128 × 128 frames, using every other 7 frames as consecutive frames because pedestrian movement was slow at 25 fps.
- Conditional generation: Conditional generation results show five observed frames followed by five generated frames for overall scenes, backgrounds, segmented objects, and movement trajectories.The generated portion begins after the red line, corresponding to frames starting from time-step 6.
- Qualitative results: The qualitative results visualize input sequences, overall reconstructions, extracted backgrounds, foreground segmentation masks with IDs, object centers, and trajectories.The trajectories are extracted from transitions in the center positions represented by zpos latents.
E MODEL ARCHITECTURE DETAILS
The architecture uses shared convolutional components to process sequential pedestrian-detection inputs across discovery and propagation. Training details specify experiment-dependent batch sizes, learning rates, image-noise scales, and RMSprop optimization, while full architectural details are deferred to the code release.
- Architecture: A fully convolutional encoder produces an H×W feature map, which a convolutional LSTM processes to model sequence information.The convolutional LSTM is shared by the discovery and propagation modules.
- Architecture: The discovery and propagation modules share the zwhat encoder and decoder.This shared representation is part of the pedestrian-detection architecture.
- Training details: Natural-scene experiments use batch size 20 and learning rate 4e-5, while dSprites/MNIST experiments use batch size 16 and learning rate 5e-4.RMSprop is used for optimization, with image-distribution standard deviations of 0.1 for natural experiments and 0.2 for toy experiments.
- Implementation availability: Full architecture details are to be released with the authors’ code.The passage states this without providing those details in the section.