PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects

Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, Yee Whye Teh

arXiv:1806.01794v2cs.LGcs.CVstat.ML

TL;DR

AIR struggles with temporal consistency and with separating overlapping or partially observed objects in image sequences. SQAIR extends AIR with a spatio-temporal state-space model that explicitly represents object presence, appearance, and location through time. It achieves temporally consistent reconstructions and samples, improves disentangling of overlapping and partially observed objects, and supports unsupervised detection, tracking, and future-frame generation, while remaining limited to simple backgrounds and static cameras.

  • Problem

    AIR describes single images, leaving temporal consistency, persistent-object tracking, and dynamics underrepresented for image sequences.

  • Method

    SQAIR augments AIR with propagation and discovery processes that update persistent objects, introduce new objects, and encode presence, appearance, and location through time.

  • Results

    SQAIR achieves temporally consistent reconstructions and samples, improves handling of overlapping and partially observed objects, and outperforms AIR across moving multi-MNIST evaluation metrics.

  • Takeaways & Limitations

    The generative model learns unsupervised object detection and tracking and can extrapolate sequences into the future by modeling object variables through time.

  • Takeaways & Limitations

    SQAIR currently works only with simple backgrounds and static cameras, and its sequential inference has complexity linear in the number of objects.

Abstract

from arXiv · show

We present Sequential Attend, Infer, Repeat (SQAIR), an interpretable deep generative model for videos of moving objects. It can reliably discover and track objects throughout the sequence of frames, and can also generate future frames conditioning on the current frame, thereby simulating expected motion of objects. This is achieved by explicitly encoding object presence, locations and appearances in the latent variables of the model. SQAIR retains all strengths of its predecessor, Attend, Infer, Repeat (AIR, Eslami et. al., 2016), including learning in an unsupervised manner, and addresses its shortcomings. We use a moving multi-MNIST dataset to show limitations of AIR in detecting overlapping or partially occluded objects, and show how SQAIR overcomes them by leveraging temporal consistency of objects. Finally, we also apply SQAIR to real-world pedestrian CCTV data, where it learns to reliably detect, track and generate walking pedestrians with no supervision.

1 Introduction

The paper argues that spatio-temporal consistency is a useful inductive bias for unsupervised object learning. SQAIR extends AIR to sequences to discover, track, and generate moving objects while addressing AIR’s static-scene limitations.

  • Motivation: Spatio-temporal consistency captures expectations that objects persist, move, and change in appearance rather than appearing or disappearing arbitrarily.The authors propose this consistency as an inductive bias that can reduce supervision requirements.
  • Limitations of AIR: AIR decomposes static scenes into objects but often merges nearby objects because it lacks temporal context.AIR explicitly encodes object locations and appearances, yet its single-image focus limits separation of close objects.
  • Limitations of AIR: AIR also struggles to identify partially occluded objects, including objects extending beyond the scene frame.The paper points to Figure 7 as an example of this limitation.
  • Contribution: SQAIR extends AIR into a sequential model that discovers and tracks objects over time and generates extrapolated future frames from unlabelled image sequences.The model retains AIR’s strengths while outperforming it on moving MNIST digits.

2 Attend, Infer, Repeat (AIR)

AIR is a structured variational auto-encoder that decomposes a static image into object-specific latent variables. Its recurrent inference and decoder represent object count, presence, location, and appearance, but the model is designed for single scenes.

  • Model: AIR decomposes a static scene into constituent objects using separate triplets of latent variables.The triplets encode appearance, location, and presence for each object.
  • Latent representation: Each object representation explicitly encodes appearance, location, and whether the object is present.The number of objects is inferred through the presence variables.
  • Scope: AIR learns object decomposition without supervision, but its static-scene formulation cannot maintain temporal object identities or dynamics.These limitations motivate the sequential extension introduced later.
  • Generation: The decoder transforms object appearance variables into glimpses, spatially places them using location variables, and sums them to generate the image mean.The spatial transformer scales and shifts each decoded patch according to its location variable.
  • Inference: AIR’s recurrent inference network samples object latents sequentially and uses prior latent variables to avoid explaining the same object twice.The number of inference steps is determined by a sequence of presence variables ending in zero.

3 Sequential Attend-Infer-Repeat

SQAIR augments AIR with a spatio-temporal state-space model organized around propagation and discovery. This structure maintains persistent objects, introduces new ones, and supports temporally consistent inference and generation.

  • Sequential structure: SQAIR alternates propagation of existing objects with discovery of new objects at every time step.Propagation updates or forgets previous latents, while discovery detects objects entering the frame.
  • Latent representation: Each object latent contains appearance, location, and a binary presence variable indicating whether the object exists at that time step.The model retains latent variables for propagated and discovered objects.
  • Generative model: The propagation prior models persistent objects’ dynamics and appearance changes while removing latents for objects that disappear.The discovery prior samples latents for new objects entering the frame, and both priors are learned during training.
  • Inference: During inference, a temporal RNN updates propagated objects, while a relation RNN models interactions and occlusion between objects.A discovery network then infers additional latents for objects entering the frame.
  • Inference: SQAIR conditions discovery on propagated objects so objects explained by propagation are not rediscovered.This conditioning permits a different number of discovered objects at each time step.
  • Learning: SQAIR is trained as an importance-weighted auto-encoder by maximizing an importance-weighted evidence lower bound.The training objective is applied to the sequential generative model.

4 Experiments

SQAIR is evaluated on moving multi-MNIST and real-world pedestrian CCTV data for reconstruction, generation, object counting, and learned-representation utility. It outperforms baselines on several multi-MNIST metrics, preserves temporal consistency, and reveals preprocessing-related limitations on CCTV data.

  • Evaluation setup: SQAIR is evaluated on moving multi-MNIST and DukeMTMC pedestrian CCTV sequences using qualitative generation and quantitative comparisons.The quantitative analysis measures model evidence, reconstruction quality, KL divergence, object-counting accuracy, and digit-addition accuracy.
  • Moving multi-MNIST results: SQAIR achieves superior performance across model evidence, reconstruction, KL divergence, object counting, and digit-addition metrics compared with AIR and VRNN.Its convolutional variant outperforms AIR and the corresponding VRNN in model evidence and reconstruction performance, while its KL divergence is almost twice as low as VRNN's and lower by a larger factor than AIR's.
  • Moving multi-MNIST results: SQAIR's temporal information prevents AIR's tendency to infer overlapping objects as a single object.This improves inferred object-count accuracy and supports modelling partially visible and heavily overlapping digits.
  • Qualitative results: SQAIR reconstructions and samples preserve motion and appearance consistently through time, unlike the compared VRNN samples and reconstructions.Figure 5 attributes this consistency to the model's propagation component.
  • Pedestrian CCTV results: On DukeMTMC CCTV data, SQAIR produces reconstructions, conditional samples, and prior samples, but object-counting accuracy is 0.5712 on validation data.The experiment uses sequences of length five and conditions generation on the first four frames.
  • Pedestrian CCTV limitations: CCTV performance is constrained by noisy background subtraction, which causes spurious detections and re-detections of the same objects.The authors also used early stopping and conjecture that a larger dataset could improve accuracy and marginal likelihood.

5 Related Work

Related work spans supervised object detection and tracking, video prediction, decomposed image and video representations, and Bayesian nonparametric models. SQAIR differs by combining object decomposition, temporal modelling, explicit locations, and adaptive object counting in an unsupervised generative framework.

  • Object Tracking: Object detection and tracking are typically learned with supervision, while prior approaches include Siamese networks and attention-based recurrent models.These methods commonly use object bounding boxes and additional labels during training.
  • Video Prediction: Deterministic video-prediction models can produce blurry averages under multiple possible futures, whereas stochastic models generate multiple plausible videos.Unlike SQAIR, these stochastic approaches do not explicitly model objects or their positions, limiting interpretability.
  • Learning Decomposed Representations of Images and Videos: Decomposed-representation methods learn object appearance and position in images or videos, often through iterative inference procedures.The cited approaches achieve results similar to SQAIR but do not exhibit its object-counting behaviour.
  • Learning Decomposed Representations of Images and Videos: Because SQAIR adapts computation to the number of objects currently present, it can be more computationally efficient than methods whose complexity scales with a predefined object maximum.Other cited approaches also differ by handling only single objects or not explicitly modelling locations.
  • Bayesian Nonparametric Models: Bayesian nonparametric work models variable numbers of video objects and their locations generatively, but relies on mixtures of Dirichlet processes and complex inference algorithms.The paper identifies this as a related alternative direction rather than the same modelling approach.

6 Discussion

SQAIR extends AIR to image sequences with explicit temporal structure, achieving temporally consistent reconstructions and samples while improving object disentanglement and partial-object identification. Its sequential inference design also addresses identity swaps, though the current model remains constrained by simple backgrounds, static cameras, and computational complexity.

  • Contributions: SQAIR achieves temporally consistent reconstructions and samples while enhancing AIR’s ability to disentangle overlapping and partially observed objects.These improvements arise from incorporating temporal information into the generative model.
  • Limitations: SQAIR’s current limitations include operation only with simple backgrounds and static cameras, linear inference complexity in object count, additive image composition, and a lack of ablation studies.The authors also identify potential improvements through alternative disentangling objectives, autoregressive generation, and further component analysis.
  • Inference design: The proposal glimpse both bottlenecks current-observation information and provides a high-resolution view of the object at a predicted location.Using this encoded predicted-location glimpse resolved identity swaps observed with alternative bottlenecks.
  • Inference design: Propagation inference conditions object location on prior appearance, location, dynamics, and relations, then conditions appearance on the new location.Presence additionally depends on the new appearance and location and on the object’s prior presence.

D.2 SQAIR and AIR Model Architectures

The architectures differ in encoder and recurrent-network choices, while both SQAIR variants retain recurrent temporal and propagation components. Larger models can overfit, whereas smaller models preserve qualitative behavior but reduce quantitative performance; training also fixes the output-noise scale.

  • Recurrent components: Both SQAIR variants use recurrent networks for temporal and propagation processing, with GRUs specifically used in the propagation-prior and temporal RNNs.The MLP-SQAIR and AIR fully connected layers and RNNs have 256 units; CONV-SQAIR uses 128 units.
  • Architecture variants: MLP-SQAIR uses fully connected networks, while CONV-SQAIR uses CNN encoders and a subpixel-CNN glimpse decoder.CONV-SQAIR’s encoders share a CNN, followed by encoder-specific fully connected layers.
  • Capacity and training: Values greater than 256 units for MLP-SQAIR and 128 units for CONV-SQAIR resulted in overfitting.Models with as few as 32 units per layer showed the same qualitative behavior but lower quantitative performance.
  • Capacity and training: The output likelihood is Gaussian with fixed standard deviation 0.3 because learning this scale caused unstable early training.A rapidly decreasing learned standard deviation encouraged zero inference steps, preventing the model from learning.

D.3 VRNN Implementation and Training Details

The VRNN baselines use recurrent latent-variable architectures with matched latent dimensionality and are trained by optimizing the FIVO bound. Medium-sized models are selected for comparison because larger models overfit.

  • VRNN architecture: The VRNN backbone uses an LSTM whose features parameterize a Gaussian prior over each latent state.At time t, the LSTM receives data and latent features from the preceding step and maps its output to the prior parameters.
  • VRNN architecture: MLP-VRNN and CONV-VRNN use shared architectural conventions but differ in their data encoders and decoders.The convolutional variant uses CNN-based feature extraction and transposed-CNN decoding, whereas the MLP variant uses fully connected networks.
  • Training: Training optimizes the FIVO bound with RMSPROP at learning rate 10^-5 and momentum 0.9 until the test bound converges.The latent dimensionality is fixed to 165, matching the upper bound used for SQAIR or AIR per time-step.
  • Model sizes: The experiments compare small, medium, and large VRNN architectures by hidden size, layer count, feature maps, and parameter count.MLP-VRNN configurations contain 1.2M, 2.1M, and 9.8M parameters; convolutional configurations vary in feature-map depth and strides.
  • Model sizes: Medium-sized models are used for comparison with SQAIR because larger models exhibited overfitting.The selected model scale is therefore an experimental control motivated by validation behavior rather than maximum capacity.
  • Addition experiment: The addition experiment feeds model representations into a 19-way MLP classifier to predict sums of two digits.AIR and SQAIR use presence-weighted concatenated zwhat variables, while VRNN uses its full 165-dimensional latent vector.

E Details of the DukeMTMC Experiments

The DukeMTMC experiments evaluate pedestrian modeling on preprocessed CCTV clips containing up to three people per frame. The reported model adapts CONV-SQAIR with pedestrian-shaped glimpses and convolutional encoders.

  • Dataset: The dataset uses cameras one, two, five, six, and eight from DukeMTMC and contains 3,500 training and validation sequences of length five.Sequences are constrained so that at most three objects appear in any single frame.
  • Dataset: CCTV frames are color-inverted, background-subtracted, downsampled to 240 × 175, converted to grayscale, and randomly cropped to 64 × 64.These preprocessing steps produce the input fragments used for pedestrian experiments.
  • Training: Training uses the MNIST procedure except that the curriculum increases sequence length from three to five time-steps.Five time-steps is the maximum length of the DukeMTMC sequences.
  • Model: The pedestrian model follows CONV-SQAIR and uses 28 × 12 glimpses to reflect the expected aspect ratio of pedestrians.Glimpse and image encoders share a CNN with feature maps [16, 32, 64, 64] and strides [2, 2, 2, 1].

F Harder multi-MNIST Experiment

The harder multi-MNIST experiment allows digits to appear or disappear during sequences, testing whether SQAIR can discover and rediscover objects over time. Figures document both temporal consistency and several failure modes.

  • Dataset and detection: SQAIR detects new digits introduced mid-sequence and rediscovers digits that were previously absent or lost.The harder dataset permits objects to appear or disappear at arbitrary times while retaining the other multi-MNIST parameters.
  • Failure modes: Removing proposal glimpse extraction can produce identity swaps because PROP may ignore latent variables and change inference order.The figure compares bounding-box colors representing object indices or identities.
  • Failure modes: MLP-SQAIR can re-detect objects in later frames, begin tracking only after discovery, or repeatedly rediscover all objects.Some runs also re-detect an object after severe overlap with another object.
  • Failure modes: SQAIR sometimes produces incorrect digit reconstructions that remain temporally consistent or are corrected after a later re-detection.Examples include reconstructing 5 and 2 as 6 and 7, and correcting overlapping 6 and 8 after the third time-step.
  • Temporal consistency: SQAIR reconstructions are shown with marked glimpse locations and described as temporally consistent, unlike the CONV-VRNN reconstructions.CONV-VRNN often produces a lower-quality and different reconstruction at t = 1.

H.2 Samples

The samples compare SQAIR and CONV-VRNN for temporal coherence in generated videos. SQAIR preserves motion and appearance across time, whereas CONV-VRNN changes objects between consecutive frames.

  • SQAIR samples: SQAIR samples maintain temporal consistency in both object motion and appearance.The samples illustrate stable object evolution across generated frames.
  • SQAIR samples: SQAIR can introduce a third object even when training sequences contained at most two objects.This behavior appears in the final sample.
  • CONV-VRNN samples: CONV-VRNN samples lack temporal consistency because objects change between consecutive generated frames.The generated objects also do not resemble digits from the training set.
  • Conditional generation: Conditional SQAIR generation observes only the first three frames before producing subsequent frames.The figure displays the input and remaining ground truth above, with reconstruction followed by generation below.

I Reconstruction and Samples from the DukeMTMC Dataset

SQAIR produces temporally consistent reconstructions and samples on DukeMTMC, with appearance and motion resembling the pedestrian inputs despite noise and occasional detection errors. Conditional generation extends reconstructions beyond the first four observed frames.

  • Reconstructions are temporally consistent and similar in appearance to the input sequences, despite spurious detections and missed objects.
  • DukeMTMC samples show spatially consistent appearance and motion, with generated objects resembling pedestrians in the training data.The samples remain noisy, consistent with noise in the dataset.
  • Conditioning on only the first four frames, SQAIR reconstructs those frames and then generates subsequent frames.
Loading 1806.01794v2…