PaperlayerView exact revision on arXiv

Source-linked AI summary

Associating Objects with Transformers for Video Object Segmentation

Zongxin Yang, Yunchao Wei, Yi Yang

arXiv:2106.02638v3cs.CV

TL;DR

Multi-object VOS methods traditionally process each target separately, increasing computation and memory demands. AOT embeds targets into a shared identification space and uses a Long Short-Term Transformer for uniform hierarchical matching and propagation. It reports superior benchmark performance with improved efficiency, while its identity-bank capacity bounds the supported object count.

  • Problem

    Existing VOS methods decode single-positive-object features, so multi-object videos require separate matching and segmentation with multiple times the resources.

  • Method

    AOT assigns target identities in a shared embedding space and uses a Long Short-Term Transformer for hierarchical multi-object matching and propagation.

  • Results

    AOT achieves superior performance across multi-object and single-object VOS benchmarks, with AOT-T maintaining real-time multi-object speed.

  • Takeaways & Limitations

    The identification mechanism makes multi-object VOS processing as efficient as single-object processing while supporting a speed-performance trade-off through LSTT depth.

  • Takeaways & Limitations

    The identity bank must contain more identification vectors than the number of objects, with M set to 10 by default to match the maximum object count.

Abstract

from arXiv · show

This paper investigates how to realize better and more efficient embedding learning to tackle the semi-supervised video object segmentation under challenging multi-object scenarios. The state-of-the-art methods learn to decode features with a single positive object and thus have to match and segment each target separately under multi-object scenarios, consuming multiple times computing resources. To solve the problem, we propose an Associating Objects with Transformers (AOT) approach to match and decode multiple objects uniformly. In detail, AOT employs an identification mechanism to associate multiple targets into the same high-dimensional embedding space. Thus, we can simultaneously process multiple objects' matching and segmentation decoding as efficiently as processing a single object. For sufficiently modeling multi-object association, a Long Short-Term Transformer is designed for constructing hierarchical matching and propagation. We conduct extensive experiments on both multi-object and single-object benchmarks to examine AOT variant networks with different complexities. Particularly, our R50-AOT-L outperforms all the state-of-the-art competitors on three popular benchmarks, i.e., YouTube-VOS (84.1% J&F), DAVIS 2017 (84.9%), and DAVIS 2016 (91.1%), while keeping more than $3\times$ faster multi-object run-time. Meanwhile, our AOT-T can maintain real-time multi-object speed on the above benchmarks. Based on AOT, we ranked 1st in the 3rd Large-scale VOS Challenge.

1 Introduction

Semi-supervised VOS tracks and segments objects throughout a video from masks given in the first frame. AOT addresses the inefficiency of independently processing multiple objects by associating and decoding them uniformly with hierarchical attention.

  • Semi-supervised VOS tracks and segments object(s) across an entire video sequence from mask(s) given at the first frame.
  • Existing methods decode features containing a single positive object, requiring independent matching and post-ensemble segmentation for multi-object scenarios.
  • Independent multi-object processing limits contextual modeling and requires multiple times the GPU memory and computation of single-object processing.
  • AOT assigns each target a unique identity in a shared feature space, enabling uniform association and direct multi-object segmentation decoding.
  • AOT's Long Short-Term Transformer constructs hierarchical matching and propagation using long-term and short-term attention.
  • AOT variants achieve strong performance and efficiency across multi-object and single-object benchmarks, while AOT-T maintains real-time multi-object speed.

2 Related Work

VOS research progressed from test-time fine-tuning and optical-flow propagation toward deep matching methods that avoid online adaptation. However, these methods generally process only one positive target and rarely study hierarchical multi-object propagation.

  • Semi-supervised VOS propagates manual labeling from one or more annotated frames, generally the first frame, throughout the video.
  • Early DNN methods fine-tuned networks at test time on first-frame ground truth, but this restricted network efficiency.
  • Recent methods avoid online fine-tuning through object embeddings, nearest-neighbor classifiers, soft matching, and first- or previous-frame pixel matching.
  • Single-positive-object methods must match and segment each target separately, consuming multiple times the computation of single-object cases.
  • Attention-based VOS methods match object features and propagate masks, but typically consider only one positive target during attention.
  • Hierarchical attention-based propagation for multi-object matching remains rarely studied, motivating the proposed long short-term transformer block.

3 Revisit Previous Solutions for Video Object Segmentation

Previous VOS solutions extend single-object networks to multiple objects by independently matching each target and aggregating the predictions. This straightforward post-ensemble design avoids adapting networks to object counts but increases resource use and limits uniform multi-object modeling.

  • A common multi-object solution applies a single-object VOS network independently to each object and combines the outputs with an ensemble function.
  • The formulation uses the current frame, memory frames, and memory masks containing reference and past predicted masks for all objects.
  • Post-ensemble processing extends single-object networks without requiring adaptation to different object numbers.
  • Independent parallel processing requires multiple times the GPU memory and computation needed to match and decode a single object.
  • AOT instead associates and segments multiple objects in an end-to-end framework, making multi-object processing as efficient as single-object processing.

4 Associating Objects with Transformers

AOT associates multiple targets in a shared embedding space and decodes them uniformly, while LSTT hierarchically combines current-frame, long-term, and short-term attention for matching and propagation.

  • Identification Mechanism: Attention-based propagation computes correlations between current-frame queries and memory keys, then propagates memory values into current-frame locations.In common single-object propagation, mask information is embedded into the values before decoding the aggregated feature.
  • Identification Mechanism: The identification mechanism assigns each target a distinct identity embedding, enabling uniform multi-object association and direct decoding in one feature space.Masks are assigned vectors from a trainable identity bank and attached to attention values for propagation.
  • Identification Mechanism: Identification decoding predicts logits for all identity-bank entries, selects the assigned identities, and produces probabilities for all targets.The resulting multi-object probabilities can be optimized with common multi-class segmentation losses.
  • Long Short-Term Transformer: Hierarchical attention is introduced because a single attention layer cannot fully model the more complex association required by multi-object propagation.The framework constructs this hierarchy with a series of attention layers rather than only one single-object attention layer.
  • Long Short-Term Transformer: LSTT uses self-attention for within-frame target association, long-term attention for memory aggregation, short-term attention for nearby-frame smoothness, and a two-layer feed-forward MLP.Its attention modules use multi-head attention, with the presented formulas showing single-head operations for simplicity.
  • Long Short-Term Transformer: Short-term attention restricts matching and propagation to a local spatial-temporal neighborhood, improving efficiency over non-local processes.For the first frame, long-term and short-term attention become self-attention without changing network structures or parameters.

5 Implementation Details

The implementation evaluates AOT variants across backbone, LSTT depth, memory configuration, and training stages, using benchmark-specific settings and quantitative table annotations.

  • Network Details: AOT uses MobileNet-V2 as a lightweight encoder and FPN with Group Normalization as the decoder; the neighborhood size is 15 and the identity bank contains 10 vectors.The identity-bank size matches the maximum object number in the benchmarks.
  • Architecture Variants: AOT-Tiny, Small, Base, and Large vary LSTT depth and long-term memory size, while AOT-L also uses ResNet50 or Swin-B encoders.The variants use 1, 2, or 3 LSTT layers, with AOT-L employing an expanding memory schedule.
  • Architecture Variants: AOT-T, AOT-S, and AOT-B retain only the first frame in long-term memory, whereas AOT-L stores predicted frames every δ frames with δ set to 2/5 for training/testing.AOT-S has two LSTT layers, AOT-T one layer, and AOT-B/L three layers.
  • Training Details: Training has two phases: synthetic video pre-training generated from static images, followed by main training on VOS benchmarks with video augmentations.Both phases apply randomly sampled augmentations.
  • Evaluation Protocol: Table 1 evaluates multi-object performance on YouTube-VOS and DAVIS 2017, marking YouTube-VOS training, 600p inference, and extrapolated timing separately.The symbols Y, ∗, and ‡ identify these experimental conditions or timing conventions.

6 Experimental Results

AOT achieves strong accuracy and efficiency across multi-object and single-object VOS benchmarks, while ablations expose accuracy–speed and design trade-offs. Its qualitative strength is associating similar objects, but tiny-object handling remains limited.

  • Benchmark Results: 84.9% validation and 79.6% testing performance on DAVIS 2017 was achieved by R50-AOT-L (Y) at 18.0FPS.The reported multi-object speed matched AOT’s single-object speed on DAVIS 2016.
  • Benchmark Results: 91.1% performance on DAVIS 2016 was achieved by R50-AOT-L (Y), while maintaining about 2× the efficiency of KMN.AOT-B matched CFBI+ at 89.9% while running 5× faster.
  • Qualitative Results: AOT performs better than CFBI on multiple highly similar objects but fails on some tiny objects lacking specialized processing.The qualitative examples include carousels, zebras, ski poles, and a watch.
  • Ablation Study: 80.3% with local attention dropped to 74.3% without it, demonstrating the necessity of local attention.Larger local windows also produced better performance.
  • Ablation Study: 77.9% with one LSTT block reached 41.0FPS, enabling an accuracy–speed trade-off through the number of LSTT blocks.More LSTT blocks increased performance, while one block provided fast real-time speed.

7 Conclusion

The paper concludes that AOT uniformly associates, matches, and decodes objects for efficient VOS. Its hierarchical LSTT structure preserves flexibility between real-time speed and state-of-the-art performance.

  • Conclusion: AOT’s identification mechanism makes multi-object VOS processing as efficient as single-object processing.It associates, matches, and decodes all objects uniformly under multi-object scenarios.
  • Conclusion: LSTT constructs hierarchical object matching and propagation while allowing AOT’s speed–performance balance to be adjusted through its block count.The conclusion presents this flexibility as a practical property of the framework.
  • Conclusion: The authors propose AOT as a baseline for related tasks including video instance segmentation, interactive VOS, and multi-object tracking.The conclusion frames the identification mechanism as potentially useful for future study of these tasks.

A.1.1 Network Details

The network details describe encoder-specific resolution changes and sequence-based processing before LSTT, alongside a patch-wise identity strategy for low-resolution features.

  • Encoder and Feature Processing: MobileNet-V2 is modified to produce 1/16-resolution features, while ResNet-50 and SwinB remove their last encoder stage.Encoder features are flattened into sequences before entering LSTT.
  • LSTT Configuration: LSTT uses 256 input channels and eight attention heads across its attention modules.A depth-wise convolution is inserted to increase LSTT’s receptive field.
  • Identity Representation: Patch-wise identity bank separates input masks into non-overlapping 16×16 patches to construct low-resolution identification embeddings.This addresses the mismatch between high-resolution masks and LSTT features at 1/16 input resolution.

A.1.2 Training Details

AOT combines identification embeddings with hierarchical long- and short-term attention to associate multiple objects efficiently, while experiments show robustness, real-time variants, and remaining accuracy–efficiency limitations.

  • Hierarchical matching and propagation: Occlusion can defeat short-term matching, whereas long-term attention can recover an accurate attention map and produce a correct prediction.The visualization describes a red person occluded in frame t−1; long-term attention remains effective in the final layer.
  • Hierarchical matching and propagation: AOT’s LSTT constructs hierarchical multi-object matching and propagation, with deeper layers progressively aggregating object information for more accurate attention-based matching.Long-term attention supports non-local matching, while short-term attention focuses on nearby spatial-temporal neighborhoods; deeper layers sharpen the attention maps.
  • Quantitative evaluation: AOT-T reaches 79.9% on DAVIS 2017 at 51.4FPS and AOT-T/S reaches 80.2%/82.6% on YouTube-VOS, outperforming cited real-time methods.On DAVIS 2016, AOT-S is reported at 89.4% and 40.0FPS.
  • Qualitative evaluation: AOT-L is reported as robust to similar objects, occlusion, fast motion, and motion blur in qualitative multi-object evaluations.The cited qualitative results cover validation YouTube-VOS and DAVIS 2017 cases.
  • Implications and limitations: AOT may support real-time video-system applications, but the real-time AOT-T still has a large accuracy gap relative to SwinB-AOT-L.The paper identifies stronger yet efficient encoders and decoders as an open question.
Loading 2106.02638v3…