Source-linked AI summary
SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation
Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, Graham W. Taylor
TL;DR
Video object segmentation must handle changing appearance, occlusion, and temporal dependencies, while recurrent methods can compound errors and limit parallelism. SST uses sparse spatiotemporal Transformer attention over multiple frames to produce dense segmentation, achieving state-of-the-art benchmark results with improved scalability. The authors also report qualitative robustness to occlusions, while noting limits for long-sequence evaluation.
Problem
VOS requires tracking and segmentation through appearance changes and occlusions, while recurrent prior methods exhibit compounding error and are not parallelizable.
Method
SST is an end-to-end, feedforward VOS model whose sparse spatiotemporal attention propagates information across a history of frames for dense segmentation.
Results
SST achieves state-of-the-art VOS results, including an overall YouTube-VOS score G = 81.8 and a DAVIS 2017 mean J &F of 78.4 versus 74.9 for previous SOTA.
Takeaways & Limitations
Attention over multiple frames and positional encodings support spatiotemporally structured representations, while SST offers superior runtime scalability and qualitatively improved occlusion robustness.
Takeaways & Limitations
The current YouTube-VOS benchmark contains sequences with limited duration, constraining evaluation of SST on long sequences.
Abstract
from arXiv · showhide
In this paper we introduce a Transformer-based approach to video object segmentation (VOS). To address compounding error and scalability issues of prior work, we propose a scalable, end-to-end method for VOS called Sparse Spatiotemporal Transformers (SST). SST extracts per-pixel representations for each object in a video using sparse attention over spatiotemporal features. Our attention-based formulation for VOS allows a model to learn to attend over a history of multiple frames and provides suitable inductive bias for performing correspondence-like computations necessary for solving motion segmentation. We demonstrate the effectiveness of attention-based over recurrent networks in the spatiotemporal domain. Our method achieves competitive results on YouTube-VOS and DAVIS 2017 with improved scalability and robustness to occlusions compared with the state of the art. Code is available at https://github.com/dukebw/SSTVOS.
1. Introduction
VOS requires tracking and segmenting objects through appearance changes, occlusions, and motion, while prior approaches suffer from adaptation, compounding-error, and scalability limitations. SST addresses these issues with sparse spatiotemporal attention and reports strong benchmark performance with improved occlusion robustness.
- VOS simultaneously tracks and segments objects throughout video clips despite appearance changes, occlusion, disocclusion, and similar objects in motion.
- Prior VOS categories include online finetuning, mask refinement, and temporal feature propagation, each with inherent drawbacks.
- Recurrent mask-refinement and temporal-propagation methods accumulate error over time and cannot be parallelized across a single example.
- SST processes videos in one feedforward pass, avoids recurrent compounding error, requires no online finetuning, and is fully parallelizable.
- Naive spatiotemporal attention costs O((THW)2C), so SST uses sparse operators to make high-resolution video self-attention feasible.
- 81.8 overall score on YouTube-VOS 2019 validation accompanies an order-of-magnitude reduction in feature-matching FLOPs and qualitatively improved occlusion robustness.
- SST applies Transformer self-attention densely over space and time for VOS and links its inductive bias to correspondence calculations.
2. Related Work
Related VOS methods address temporal coherence through mask refinement or temporal feature propagation, while Transformers and correspondence matching motivate SST’s attention-based design. SST differs by propagating temporal features with sparse attention in a single feedforward operation.
- Online finetuning methods adapt segmentation networks using an initial frame, while some offline methods model frames independently without explicit temporal coherence.
- VOS research on temporal coherence divides into mask-refinement and temporal-feature-propagation approaches.
- Mask-refinement methods use feedforward models to refine previous masks, with variants incorporating first-frame features, spatial priors, optical flow, or probabilistic inference.
- Temporal-feature-propagation methods increase mask-feature expressiveness, but prior approaches used RNNs to encode and propagate spatiotemporal representations through time.
- FEELVOS performs end-to-end VOS with pixel-wise embeddings and global and local matching to reference and previous frames.
- SST shares FEELVOS’s end-to-end simplicity while extracting features from multiple frames with attention and positional encodings.
- Transformer self-attention is motivated by successful applications across NLP, vision-language, speech, and computer vision, alongside correspondence-matching work.
3. Method
SST processes a bounded history of video-frame embeddings with sparse spatiotemporal self-attention, producing context and object-affinity features for segmentation. Its sparse connectivity makes attention tractable while supporting correspondence-like computations across time and space.
- SST Architecture: SST propagates a history of τ frames and applies spatiotemporal self-attention to attend to previous and reference frames.The design targets optical-flow-like calculations and reference-mask propagation over time.
- SST Architecture: The architecture uses a CNN feature extractor, a Transformer encoder, and a convolutional decoder for video object segmentation.The encoder receives frame embeddings and passes spatiotemporal context and attention-derived features to the decoder; CFBI’s decoder is used for fair comparison.
- Object Affinity: Object-affinity tensors propagate segmentation information from past frames, while causal attention restricts each feature cell to previous timesteps and its connectivity pattern.Affinity values summarize attention to cells belonging to each generalized object, including background, and the current temporal slice supplies object-discriminative features for segmentation.
- Sparse Attention: Sparse attention replaces naïve O((THW)^2C) spatiotemporal attention, whose quadratic cost is prohibitive for high-resolution video.The sparse operators select connectivity patterns over the three spatial-temporal axes.
- Grid Attention: Grid attention reduces complexity from O(C(THW)^2) to O(C(T + H + W)THW), making video attention tractable.Grid interactions propagate through fixed numbers of feature cells in the X, Y, and T directions.
4. Experiments and Results
Experiments evaluate SST against state-of-the-art methods on YouTube-VOS and DAVIS 2017, then analyze positional encodings, temporal history, and sparse-attention variants. SST performs competitively across benchmarks, with qualitative evidence of improved occlusion handling and ablations favoring sinusoidal encodings, longer histories, and local or strided connectivity.
- YouTube-VOS: SST performs favourably against previous methods on YouTube-VOS 2018 and 2019 in overall score G, including methods using online fine-tuning or synthetic data.The comparison distinguishes online fine-tuning and synthetic-data methods from methods that use neither.
- Occlusion: SST’s temporal history robustly tracks all three fish through overlap, whereas CFBI confuses two fish after they overlap.This qualitative example concerns foreground occlusion followed by disocclusion in YouTube-VOS validation.
- DAVIS 2017: 78.4 mean J &F on DAVIS 2017 versus 74.9 for previous state of the art under a comparable setup without online finetuning or synthetic data.The evaluation uses region similarity J, contour accuracy F, and their mean J &F.
- Ablation studies: Sinusoidal temporal positional encodings produce lower variance across runs and form a robust prior for finding correspondences in VOS.The positional-encoding study varies Transformer layers and temporal history while measuring J &F on DAVIS 2017 validation.
- Ablation studies: Even a modest increase in temporal history improves the J &F score, while local and strided attention generally outperform grid attention.The temporal-history and sparse-attention studies use DAVIS 2017 validation ablations.
- Scope and future work: The paper identifies longer-sequence evaluation as future work because YouTube-VOS contains sequences with at most 36 labeled frames sampled at 6 fps.The discussion links this boundary to a dataset-creation challenge for evaluating VOS on long sequences.
5. Conclusions
SST is presented as an entirely attention-based VOS model with strong benchmark performance and runtime scalability, while memory requirements limit longer temporal histories.
- 81.8 overall score G on YouTube-VOS demonstrates state-of-the-art benchmark performance.
- SST has superior runtime scalability compared with previous state-of-the-art methods, including recurrent approaches.
- Memory requirements inherent in SST’s weak Markovian assumptions currently prevent extending the history to a desirable longer duration.
6. Additional Results
Qualitative comparisons show SST remains accurate through occlusions and disocclusions, where competing methods confuse similar instances or nonlocal regions.
- SST produces superior tracking in challenging sequences containing occlusions and disocclusions.Figures 4 and 5 compare SST qualitatively with CFBI and STM.
- Positional encodings may help SST distinguish similar instances under occlusion by providing positional information.
- SST remains robust to nonlocal failures in which CFBI confuses instances that are far apart.
7. Grid Attention Routing
Grid attention propagates information through structured spatial and temporal routes, enabling reference-mask values to travel along transitively corresponding feature paths.
- Grid Attention Routing: Under the simplified analysis, query, key, and value are all set to the video feature tensor T.
- Grid Attention Routing: GridAttn3 denotes composing three applications of self-attention on the video feature tensor T.
- Grid Attention Routing: Grid attention connects arbitrary source and target pixels through routes that propagate across the X, Y, and T directions in a fixed order.
- Grid Attention Routing: Replacing the inner attention value with a foreground mask causes outputs to reference mask values along paths of transitively corresponding video features.
- Additional Results: In qualitative examples, SST maintains more accurate object tracking during occlusion and disocclusion than CFBI or STM.