PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Features in Hierarchical Propagation for Video Object Segmentation

Zongxin Yang, Yi Yang

arXiv:2210.09782v3cs.CV

TL;DR

AOT-style hierarchical propagation can lose object-agnostic visual information as object-specific information accumulates in deep layers. DeAOT separates visual and ID propagation into two branches and adds GPM for efficient construction, achieving strong accuracy and speed across four benchmarks.

  • Problem

    AOT’s hierarchical propagation can lose object-agnostic visual information as object-specific information increases, although visual features are crucial for matching objects in attention-based VOS.

  • Method

    DeAOT decouples object-agnostic visual and object-specific ID embeddings into separate propagation branches and uses Gated Propagation Module with single-head attention for efficiency.

  • Results

    DeAOT variants outperform AOT counterparts in accuracy and speed and achieve top-ranked performance on four VOS/VOT benchmarks.

  • Takeaways & Limitations

    DeAOT provides a hierarchical VOS framework combining decoupled feature propagation with superior run-time speed compared to previous solutions.

  • Takeaways & Limitations

    R50-DeAOT-L may still fail to track multiple highly similar objects during serious occlusion, and evaluations use benchmark-specific default video settings.

Abstract

from arXiv · show

This paper focuses on developing a more effective method of hierarchical propagation for semi-supervised Video Object Segmentation (VOS). Based on vision transformers, the recently-developed Associating Objects with Transformers (AOT) approach introduces hierarchical propagation into VOS and has shown promising results. The hierarchical propagation can gradually propagate information from past frames to the current frame and transfer the current frame feature from object-agnostic to object-specific. However, the increase of object-specific information will inevitably lead to the loss of object-agnostic visual information in deep propagation layers. To solve such a problem and further facilitate the learning of visual embeddings, this paper proposes a Decoupling Features in Hierarchical Propagation (DeAOT) approach. Firstly, DeAOT decouples the hierarchical propagation of object-agnostic and object-specific embeddings by handling them in two independent branches. Secondly, to compensate for the additional computation from dual-branch propagation, we propose an efficient module for constructing hierarchical propagation, i.e., Gated Propagation Module, which is carefully designed with single-head attention. Extensive experiments show that DeAOT significantly outperforms AOT in both accuracy and efficiency. On YouTube-VOS, DeAOT can achieve 86.0% at 22.4fps and 82.0% at 53.4fps. Without test-time augmentations, we achieve new state-of-the-art performance on four benchmarks, i.e., YouTube-VOS (86.2%), DAVIS 2017 (86.2%), DAVIS 2016 (92.9%), and VOT 2020 (0.622). Project page: https://github.com/z-x-yang/AOT.

1 Introduction

Semi-supervised VOS propagates annotated object masks through video frames, but AOT’s hierarchical propagation can lose visual information as object-specific information accumulates. DeAOT addresses this with decoupled branches and an efficient propagation module, achieving strong accuracy and speed across benchmarks.

  • Semi-supervised VOS tracks and segments objects throughout a video using masks annotated on one or several frames.
  • AOT hierarchically propagates object-specific information into object-agnostic visual embeddings, while DeAOT separates visual and ID propagation into two branches.
  • DeAOT uses the Gated Propagation Module to offset dual-branch computation and replace AOT’s efficiency-bottlenecking multi-head attention with single-head attention.
  • 86.0% at 22.4fps and 82.0% at 53.4fps are achieved by R50-DeAOT-L and DeAOT-T on YouTube-VOS, respectively.DeAOT-T exceeds AOT-T’s 80.2% and 41.0fps.
  • Without test-time augmentations, SwinB-DeAOT-L achieves top-ranked results on YouTube-VOS, DAVIS 2017, DAVIS 2016, and VOT 2020.Reported scores are 86.2%/86.1%, 86.2%/82.8%, 92.9%, and 0.622 EAO, respectively.

2 Related Work

VOS research progressed from finetuning and template matching toward attention-based memory and transformer methods. AOT introduced hierarchical propagation and collaborative multi-object association, while DeAOT separates visual and object-specific embeddings.

  • Finetuning-based Methods: Early VOS methods finetuned segmentation networks on annotated frames or constructed pixel-wise matching maps.
  • Template-based Methods: Template-based methods avoid test-time finetuning by matching current-frame features with annotated frames treated as templates.
  • Attention-based Methods: Attention-based methods use memory networks and non-local attention to propagate mask information from past frames to the current frame.
  • Visual Transformers: AOT introduces hierarchical propagation through the Long Short-Term Transformer and uses an ID mechanism to associate multiple objects collaboratively.
  • Visual Transformers: DeAOT differs from AOT by placing object-agnostic and object-specific embeddings in separate branches with individual propagation processes.

3 Rethinking Hierarchical Propagation for VOS

Hierarchical propagation transforms object-agnostic current-frame features into object-specific embeddings by incorporating memorized mask information. The paper argues that this can discard visual information, motivating separate visual and ID branches.

  • AOT Hierarchical Propagation: Attention-based VOS propagation computes a correlation map between current-frame queries and memorized-frame keys, then aggregates memorized values.Q is the current-frame query, while K and V are memorized-frame keys and values.
  • AOT Hierarchical Propagation: AOT’s hierarchical layers progressively propagate ID(Y m) into current-frame features, converting object-agnostic embeddings into object-specific ID embeddings.The resulting feature can be decoded into ID or mask predictions.
  • Motivation: Because feature channels are limited, absorbing object-specific ID information can cause object-agnostic visual information to be forgotten.
  • Motivation: Increasing the amount of ID information by including more IDs causes AOT performance to drop substantially.Figure 2 presents this trend as the ID maximum number increases.
  • Dual-branch Propagation: DeAOT separates propagation into a Visual Branch for refining and matching visual embeddings and an ID Branch for object-specific information.

4 Decoupling Features in Hierarchical Propagation

DeAOT separates object-agnostic visual and object-specific ID propagation into two branches while sharing attention maps. It uses GPM, a single-head gated propagation design, to improve efficiency while preserving hierarchical propagation.

  • 4.1 Hierarchical Dual-branch Propagation: DeAOT propagates visual and ID embeddings in separate branches while sharing their attention maps.The visual branch matches objects and refines visual features; the ID branch propagates object-specific information for segmentation.
  • 4.1 Hierarchical Dual-branch Propagation: The visual branch computes attention maps from patch-wise visual embeddings and avoids direct dependence on object-specific ID embeddings.This allows visual embeddings to remain and become more contrastive without being biased toward object-specific information.
  • 4.1 Hierarchical Dual-branch Propagation: The ID branch propagates object-specific information from past frames and reuses the visual branch’s attention maps for object identification.The paper states that object identification mainly relies on visual features rather than ID indices.
  • 4.2 Gated Propagation Module: GPM replaces AOT’s multi-head LSTT propagation with more efficient single-head attention and removes the feed-forward module.LSTT’s long-term attention has complexity O(NTH^2W^2), making the head number an efficiency bottleneck.
  • 4.2 Gated Propagation Module: GPM uses gated self-, long-term, and short-term propagation in both DeAOT branches.Short-term propagation is restricted to a λ × λ neighborhood because motion across contiguous frames is treated as smooth.
  • 4.2 Gated Propagation Module: DeAOT performs better than AOT on tiny or scale-changing objects but still fails on highly similar objects under serious occlusion.These qualitative comparisons are reported in Figure 4.

5 Implementation Details

The implementation uses several encoders with a shared FPN decoder and specifies propagation, embedding, memory, and training settings for DeAOT variants.

  • Network Details: Experiments use MobileNet-V2 by default, alongside ResNet-50 and Swin-B encoders, with the same FPN decoder.The spatial neighborhood size λ is 15 and the maximum object number in the ID embedding is 10.
  • Network Details: DeAOT-T, DeAOT-S, and DeAOT-B use L = 1, 2, and 3 GPM layers respectively, while DeAOT-L uses L = 3 with an expanding long-term memory.The T/S/B variants consider only the reference frame as long-term memory, while DeAOT-L updates memory every δ frames.
  • Training Details: Training first pre-trains DeAOT on synthetic videos generated from static image datasets, then performs main training on VOS benchmarks.The optimization strategies and related hyperparameters are kept the same as AOT.

6 Experimental Results

Experiments across four VOS/VOT benchmarks show that DeAOT improves accuracy and efficiency over AOT and achieves strong results across multi-object and single-object settings. Ablations support dual-branch propagation, GPM, shared attention maps, and depth-wise convolution as important design choices, while severe occlusion remains challenging.

  • YouTube-VOS: DeAOT variants outperform AOT counterparts in both accuracy and run-time speed on YouTube-VOS 2018/2019.SwinB-DeAOT-L reaches 86.2%/86.1% J &F, while DeAOT-T reaches 82.0%/82.0% and 53.4fps.
  • DAVIS 2017: 85.2%/80.7% on DAVIS 2017 validation/test at 27fps surpasses R50-AOT-L in accuracy and efficiency.SwinB-DeAOT-L achieves 86.2%/82.8% on DAVIS 2017.
  • DAVIS 2016: 92.9% on DAVIS 2016 is reported for SwinB-DeAOT-L, outperforming all listed VOS methods.The benchmark contains 20 validation videos and evaluates single-object VOS.
  • VOT 2020: 0.622 EAO is achieved by SwinB-DeAOT-L on VOT 2020, while larger DeAOT variants outperform several state-of-the-art trackers in EAO and real-time EAO.VOT 2020 contains challenging long videos with fast motion and occlusion.
  • Qualitative Results: R50-DeAOT-L performs better than R50-AOT-L on tiny or scale-changing objects but may fail on multiple highly similar objects under severe occlusion.The qualitative comparison identifies a remaining limitation in challenging occlusion scenarios.
  • Ablation Study: Coupling visual and ID propagation reduces performance from 82.5% to 81.5%, while replacing GPM with AOT’s LSTT reduces it to 80.3%.The ablation concludes that dual-branch propagation and GPM are crucial for improving VOS performance.
  • Ablation Study: DeAOT remains robust to attention head number, unlike AOT, whose single-head version runs at 44.6fps versus 27.1fps for default eight-head AOT but loses 0.7% accuracy.The experiments also find that visual embeddings are essential for long-term and short-term attention maps, while ID embeddings facilitate self-propagation as positional embeddings.
  • Ablation Study: Removing GPM’s depth-wise convolution lowers performance from 82.5% to 81.1%, and kernel size 5 performs best among 3, 5, and 9.The depth-wise convolution enlarges receptive fields within GPM.

7 Conclusion

DeAOT is an effective and efficient framework that decouples visual and ID propagation and achieves state-of-the-art performance with superior runtime speed.

  • DeAOT is proposed as a highly effective and efficient framework for video object segmentation.
  • Its dual-branch design decouples visual and ID embedding propagation to avoid visual-information loss in deep propagation layers.
  • DeAOT variant networks achieve new state-of-the-art performance on four VOS/VOT benchmarks with superior runtime speed compared to previous solutions.
Loading 2210.09782v3…