PaperlayerView exact revision on arXiv

Source-linked AI summary

STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos

Ali Athar, Sabarinath Mahadevan, Aljoša Ošep, Laura Leal-Taixé, Bastian Leibe

arXiv:2003.08429v4cs.CVcs.LGeess.IV

TL;DR

Video instance segmentation often relies on computational, non-end-to-end tracking-by-detection pipelines that detect objects per frame and associate them over time. STEm-Seg models each clip as a 3D spatio-temporal volume, learns embeddings that cluster each instance across the clip, and performs single-stage inference. It achieves state-of-the-art results across multiple datasets and tasks while generalizing across diverse settings.

  • Problem

    Existing video instance segmentation methods use multiple networks for frame-wise detection and temporal association, making them computationally demanding and often non-end-to-end trainable.

  • Method

    STEm-Seg uses a single-stage, proposal-free network that learns spatio-temporal embeddings, mixing functions, and clustering parameters over a 3D video volume.

  • Results

    STEm-Seg achieves state-of-the-art results across multiple datasets and tasks, including UVOS, VIS, and MOTS.

  • Takeaways & Limitations

    The unified approach supports category-aware and category-agnostic video instance segmentation and generalizes across diverse datasets.

  • Takeaways & Limitations

    TrackR-CNN’s ReID-based embeddings learn appearance models, which are limited when similar-looking objects occur together in YT-VIS.

Abstract

from arXiv · show

Existing methods for instance segmentation in videos typically involve multi-stage pipelines that follow the tracking-by-detection paradigm and model a video clip as a sequence of images. Multiple networks are used to detect objects in individual frames, and then associate these detections over time. Hence, these methods are often non-end-to-end trainable and highly tailored to specific tasks. In this paper, we propose a different approach that is well-suited to a variety of tasks involving instance segmentation in videos. In particular, we model a video clip as a single 3D spatio-temporal volume, and propose a novel approach that segments and tracks instances across space and time in a single stage. Our problem formulation is centered around the idea of spatio-temporal embeddings which are trained to cluster pixels belonging to a specific object instance over an entire video clip. To this end, we introduce (i) novel mixing functions that enhance the feature representation of spatio-temporal embeddings, and (ii) a single-stage, proposal-free network that can reason about temporal context. Our network is trained end-to-end to learn spatio-temporal embeddings as well as parameters required to cluster these embeddings, thus simplifying inference. Our method achieves state-of-the-art results across multiple datasets and tasks. Code and models are available at https://github.com/sabarim/STEm-Seg.

1 Introduction

STEm-Seg addresses video instance segmentation by replacing multi-network tracking-by-detection pipelines with a single-stage, bottom-up approach that clusters spatio-temporal pixel embeddings. The method is designed as a unified, category-agnostic framework and reports strong performance across UVOS, VIS, and MOTS.

  • 1 Introduction: Video instance segmentation assigns every pixel to one object instance or the background across multiple frames.The task is motivated by dense per-pixel instance annotations and applications including autonomous vehicles and robots.
  • 1 Introduction: Existing methods detect objects frame by frame, associate them over time, and resolve occlusions with multiple networks, making them computationally demanding and non-end-to-end trainable.These approaches follow the established tracking-by-detection paradigm.
  • 1 Introduction: STEm-Seg instead learns category-agnostic spatio-temporal embeddings that cluster pixels from the same instance across an entire 3D video volume.Instances are inferred by assigning pixels to their learned embedding clusters.
  • 1 Introduction: STEm-Seg outperforms proposal-based methods on UVOS, VIS, and MOTS tasks involving pixel-precise tracking.The approach is presented as a unified method for varied automotive, YouTube, and category-agnostic settings.
  • 1 Introduction: The method combines novel embedding mixing functions with a single-stage network that incorporates temporal context and avoids external temporal-association components.Inference is based on clustering within the 3D spatio-temporal volume.

2 Related Work

Prior video segmentation and tracking methods use proposal generation, frame-wise processing, optical flow, trajectories, or temporal association. STEm-Seg differs by clustering instances across space and time in a single end-to-end approach.

  • 2 Related Work: Image-level instance segmentation methods are divided between top-down proposal-based systems and bottom-up pixel-grouping systems.Some bottom-up methods use embedding mixing and displacement fields to address appearance ambiguity.
  • 2 Related Work: Video segmentation methods group moving regions using optical flow or point trajectories, whereas STEm-Seg handles both moving and static objects.Other approaches generate video proposals from action or image-level object cues before learning temporal feature representations.
  • 2 Related Work: Semi-supervised VOS supplies the objects’ ground-truth masks in the first frame, while unsupervised VOS provides no ground-truth information at test time.Unsupervised VOS aims to segment and track dominant foreground objects.
  • 2 Related Work: Some unsupervised VOS methods process frames separately, use K-means, or cannot separate different object instances.Other methods propagate information across time with attentive graph neural networks.
  • 2 Related Work: MOTS and VIS baselines commonly adapt Mask R-CNN or associate clustered proposals over time, unlike STEm-Seg’s single-stage end-to-end design.YouTube-VIS contains more diverse videos and object classes than MOTS, motivating methods that generalize across settings.

3 Our Method

STEm-Seg models a video clip as one 3D spatio-temporal volume and predicts embeddings, variances, and center heat maps for proposal-free instance clustering. Training optimizes instance-mask overlap, while inference extracts object tubes from predicted centers and distributions.

  • 3.1 Problem Formulation: The network assigns pixels in a T-frame video clip to background or one of K unknown object instances by clustering embeddings across space and time.It predicts instance tubes simultaneously rather than treating the clip as independent images.
  • 3.1 Problem Formulation: The network outputs an E-dimensional embedding E, positive per-pixel variances V, and an instance-center heat map H.The outputs parameterize Gaussian distributions for object instances.
  • 3.1 Problem Formulation: Each instance’s embeddings across the entire clip are modeled by one multivariate Gaussian distribution, whose probabilities define predicted mask tubes.Pixels are included when their instance probability exceeds 0.5.
  • 3.1 Problem Formulation: Training uses the Lovász hinge loss as a differentiable surrogate for the Jaccard index, allowing the network to learn Gaussian parameters through IoU optimization.The formulation optimizes the overlap between predicted and ground-truth mask tubes.
  • 3.2 Embedding Representation: Spatio-temporal mixing adds coordinate offsets, while free dimensions provide learnable embedding capacity beyond spatial and temporal coordinates.Free dimensions require fixed variances because otherwise the network may produce large variances instead of discriminative representations.
  • 3.2 Embedding Representation: At inference, the highest heat-map peak supplies an instance center, whose embedding and variances define a mask tube; assigned pixels are then masked out and the process repeats.Only one forward pass is required, and the clustering step has negligible runtime relative to that pass.

ENCODER

The network uses an encoder with two decoders: one predicts embeddings and variances, while the other predicts instance centers. Temporal squeeze-expand decoders incorporate temporal context while controlling memory and runtime.

  • ENCODER: The temporal squeeze-expand decoder reduces memory and runtime by applying less temporal pooling to larger feature maps.The paper separately validates the network’s ability to learn spatio-temporal context.
  • ENCODER: For datasets requiring category labels, an additional TSE decoder performs semantic segmentation and assigns each instance the highest-scoring averaged category logit.The task formulation itself remains category-agnostic.

4 Experimental Evaluation

Experiments evaluate STEm-Seg across multiple video segmentation and tracking tasks, with ablations showing how embedding design, temporal context, and mixed training data affect performance. The method achieves strong results across DAVIS, YouTube-VIS, KITTI-MOTS, and DAVIS’16 while using a single-stage architecture.

  • Embedding Formulation: Adding spatial and temporal coordinate offsets improves DAVIS J &F from 57.3% with the identity baseline to 62.6% with ϕxyt.A free dimension is slightly better than a temporal dimension in the compared function pairs, while temporal and free dimensions perform similarly on KITTI-MOTS.
  • Temporal Window Size: Increasing clip length from T = 4 to T = 16 raises DAVIS J &F from 62.2% to 64.7%, but performance decreases above T = 24.The ablation supports temporal context while indicating that longer windows are not uniformly beneficial.
  • Training Data: Combining image and video training data yields 64.4% DAVIS J &F, compared with 60.7% using only video data and 57.1% using only image data.The image data include COCO and Pascal VOC, while the video data include YouTube-VIS and DAVIS.
  • Semantic Head: Oracle category labels improve YT-VIS AP from 24.7 to 33.0, indicating that semantic classification limits the reported result on the custom validation split.The authors suggest that a better semantic classifier could further improve official validation performance.
  • Video Object Segmentation: STEm-Seg reaches 64.7% J &F on DAVIS’19, exceeding the second-best published score of 61.1% while using a single proposal-free network.UnOVOST scores 67.0% but uses several networks, heuristic post-processing, and is an order of magnitude slower.
  • Video Instance Segmentation: 34.6 AP surpasses MaskTrack-RCNN’s 30.3 AP on YouTube-VIS, and the advantage remains with the ResNet-50 backbone at 30.6 versus 30.3.STEm-Seg performs best across all reported evaluation metrics in this comparison.
  • Multi-Object Tracking and Segmentation: STEm-Seg achieves the highest pedestrian sMOTSA score of 50.4 on KITTI-MOTS, although Track-RCNN performs better for cars.The result demonstrates strong performance across classes but not uniform dominance on every object category.
  • DAVIS’16 Unsupervised Segmentation: On DAVIS’16, a simplified binary-segmentation setup achieves state-of-the-art results despite competing methods being specifically engineered for that task.The network uses one decoder with two output channels and trains on randomly selected YT-VIS and DAVIS clips.

5 Conclusion

The paper presents a unified bottom-up approach that models video clips as 3D space-time volumes and separates instances by clustering learned embeddings. It reports strong results across diverse datasets and task settings, including category-aware and category-agnostic evaluation.

  • The method models video clips as 3D space-time volumes and separates object instances by clustering learned embeddings.
  • Novel mixing functions improve the feature representation of spatio-temporal embeddings and yield considerable performance improvements over existing formulations.
  • The approach achieves state-of-the-art results across multiple diverse datasets under both category-aware and category-agnostic settings.
  • The method generalizes better across different datasets than existing dataset-specific state-of-the-art methods.
  • The paper also reports that the network’s temporal-context learning generalizes to video saliency detection.

Supplementary Material

The supplementary material describes a three-part training loss for embedding-based video instance segmentation. It explains how the components regulate embeddings, variance predictions, center probabilities, and IoU-oriented optimization.

  • The total loss is a linear combination of embedding, variance-smoothness, and center-regression losses.
  • The variance-smoothness loss encourages uniform variance values for each object instance and excludes background-pixel variances.
  • The center-regression loss aligns foreground pixels’ instance-center heat-map probabilities with probabilities computed from their embeddings.
  • The embedding loss uses the Lovasz hinge loss, a convex surrogate of the Jaccard index that directly optimizes IoU for binary segmentation.
  • The supplementary equations identify the Lovasz extension and hinge loss as the components used to compute the embedding loss.

II. Implementation Details

The implementation uses task-specific training schedules, image-derived sequence augmentation, and clip-consistent video augmentation. Training and inference are conducted on specified GPU workstations with modest batch and memory settings.

  • Training uses batch size 2 on two Nvidia RTX TITAN GPUs with 64GB RAM, while inference uses one Nvidia GTX 1080Ti GPU with 32GB RAM.
  • All tasks use SGD with an initial learning rate of 10^-3, followed by exponential decay to 10^-5.
  • The schedules run for 60k iterations on DAVIS’19, 150k on YouTube-VIS, and 100k on KITTI-MOTS.
  • Training combines real video clips with sequences synthesized from static images using random affine transformations and motion blur.
  • Actual video clips receive only random horizontal flipping, applied consistently to entire clips rather than individual frames.

III. Baselines for DAVIS’19 Unsupervised

The DAVIS’19 baselines use Mask R-CNN proposals generated per frame and then associate those proposals over time with optical flow or re-identification embeddings. The supplementary material also situates the comparison within mixing-function ablations across three datasets.

  • The proposal-based baselines generate per-frame Mask R-CNN mask proposals for all objects in each video.
  • The mixing-function ablation reports results on DAVIS’19, YouTube-VIS, and KITTI-MOTS, with DAVIS’19 results repeated for reference.
  • The Mask R-CNN proposal network is jointly trained on YouTube-VIS, DAVIS’19, and augmented COCO and Pascal-VOC images for 120k iterations.
  • OF-Tracker associates adjacent-frame proposals using optical-flow warping followed by linear assignment based on object overlap.
  • RI-Tracker associates proposals using Euclidean distances between re-identification vectors produced by a ResNet-50 triplet-loss network.

IV. Extended Ablations for Embedding Mixing Function

The benefit of embedding mixing functions depends on the dataset and representation dimensions. Free dimensions generally outperform temporal coordinates on DAVIS’19 and YouTube-VIS, while KITTI-MOTS shows a different pattern.

  • On DAVIS’19 and YouTube-VIS, a free dimension is more beneficial than a temporal coordinate dimension when total embedding dimensions are fixed.
  • KITTI-MOTS shows similar performance for ϕxyt and ϕxyff, with car and pedestrian sMOTSA values of 72.5 and 48.9 versus 73.2 and 47.3.
  • The authors report a mean KITTI-MOTS sMOTSA of 60.70 for ϕxyt, slightly above 60.25 for ϕxyff.
  • The temporal coordinate may help KITTI-MOTS because objects move faster and often enter or exit video clips midway.
  • Mixing-function trends differ between KITTI-MOTS object classes: ϕxyfff is highest for cars at 73.4 but substantially lower for pedestrians.

V. UnOVOST Training on KITTI-MOTS

The UnOVOST comparison was retrained and tuned on KITTI-MOTS using the same training data and backbone as STEm-Seg, but its final scores remained consistently low.

  • UnOVOST was evaluated on KITTI-MOTS after retraining and tuning its implementation for multi-object tracking and segmentation.
  • The comparison used Mask R-CNN with a ResNet-101 backbone initialized from COCO instance-segmentation weights and adapted to car and pedestrian classes.
  • The UnOVOST training data and backbone matched those used for STEm-Seg, supporting a controlled comparison of the frameworks.
  • A ReID network was trained on KITTI-MOTS instance crops using triplet loss and batch-hard sampling.
  • Despite grid-search tuning of detection-pruning and optical-flow association IoU thresholds, UnOVOST’s KITTI-MOTS scores remained consistently low.
  • TrackR-CNN adaptation details include shared initialization from COCO and Mapillary Vistas weights and a single ReID head for YouTube-VIS.
  • TrackR-CNN’s low-resolution 14x14 RoI-Align layer produced coarse masks on diverse, large YouTube-VIS objects, reducing AP under IoU thresholds from 0.5 to 0.95.

VII. Additional Qualitative Results

Additional validation examples show that STEm-Seg reliably segments and tracks diverse objects across datasets, including challenging scale changes, occlusions, semantic categories, and distant automotive scenes.

  • Across DAVIS’19, YouTube-VIS, and KITTI-MOTS, STEm-Seg reliably segments and tracks varied objects in diverse scenarios.
  • The qualitative results indicate robustness to brief occlusions and sudden pose changes.
  • On DAVIS’19, the method remains robust to large scale changes in the motocross-jump sequence.
  • On YouTube-VIS, STEm-Seg segments semantically challenging animal categories and preserves fine details such as skateboards and surfboards.
  • On KITTI-MOTS, the method tracks and segments cars and pedestrians from large distances and bridges occlusions.
Loading 2003.08429v4…