PaperlayerView exact revision on arXiv

Source-linked AI summary

TubeFormer-DeepLab: Video Mask Transformer

Dahun Kim, Jun Xie, Huiyu Wang, Siyuan Qiao, Qihang Yu, Hong-Seok Kim, Hartwig Adam, In So Kweon, Liang-Chieh Chen

arXiv:2205.15361v2cs.CV

TL;DR

Video segmentation methods have diverged across semantic, instance, and panoptic tasks despite their shared tube-labeling structure. TubeFormer-DeepLab predicts class-labeled tubes in a unified framework, and reports gains over published baselines on several benchmarks.

  • Problem

    Video semantic, instance, and panoptic segmentation methods have diverged despite their shared underlying structure of assigning predicted labels to tubes.

    Source: §1 · p. 1

  • Method

    TubeFormer-DeepLab extends mask transformers to directly predict class-labeled tubes, with labels encoding task-specific values.

    Source: §1 · p. 2

  • Results

    Across several video segmentation benchmarks, TubeFormer-DeepLab advances state of the art, including +13.1 STQ over Motion-DeepLab on KITTI-STEP test set.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model provides a general solution for multiple video segmentation tasks without task-specific design.

    Source: §1 · p. 2, §9 · p. 11

    Clip values of T = 2 for VPS and VSS or T = 5 for VIS mean the model performs short-term tracking and may miss objects with longer track lengths.

    Source: §8 · p. 11

Abstract

from arXiv · show

We present TubeFormer-DeepLab, the first attempt to tackle multiple core video segmentation tasks in a unified manner. Different video segmentation tasks (e.g., video semantic/instance/panoptic segmentation) are usually considered as distinct problems. State-of-the-art models adopted in the separate communities have diverged, and radically different approaches dominate in each task. By contrast, we make a crucial observation that video segmentation tasks could be generally formulated as the problem of assigning different predicted labels to video tubes (where a tube is obtained by linking segmentation masks along the time axis) and the labels may encode different values depending on the target task. The observation motivates us to develop TubeFormer-DeepLab, a simple and effective video mask transformer model that is widely applicable to multiple video segmentation tasks. TubeFormer-DeepLab directly predicts video tubes with task-specific labels (either pure semantic categories, or both semantic categories and instance identities), which not only significantly simplifies video segmentation models, but also advances state-of-the-art results on multiple video segmentation benchmarks

1. Introduction

Video segmentation methods have diverged despite their shared structure: assigning labels to tubes of linked masks. TubeFormer-DeepLab predicts class-labeled tubes across tasks and reports gains on several benchmarks.

1. Introduction

  • The paper formulates video segmentation as assigning labels to tubes of temporally linked masks; labels encode semantic categories alone or categories plus instance identities, depending on the task.VIS labels foreground things, while VPS labels both foreground things and background stuff.Source: §1 · p. 1, Figure 1 · p. 1
  • Video semantic, instance, and panoptic segmentation methods have diverged despite their shared tube-labeling structure.Examples include feature warping for VSS, frame-level mask propagation for VIS, and separate semantic, instance, and tracking branches for VPS.Source: §1 · pp. 1–2
  • TubeFormer-DeepLab extends mask transformers to predict class-and-tube-embedding pairs, whose embeddings combine with convolutional video-pixel features to produce tube predictions.The paper describes this as a general framework that does not require task-specific design.Source: §1 · p. 2
  • The hierarchical dual-path transformer addresses video attention learning with latent memory between frame features and global memory between clip features.Latent memory learns task-specific attention; global memory learns spatiotemporally clustered attention for tube regions and is split into thing- and stuff-specific sets.Source: Figure 2 · p. 1, §1 · p. 2
  • Clip-level inference uses video stitching to merge clip predictions, while a Temporal Consistency loss encourages consistent predictions in overlapping frames.Source: §1 · p. 2
  • Clip-paste extends thing-specific copy-paste augmentation by randomly pasting thing regions, stuff regions, or both between video clips.Source: §1 · p. 2
  • TubeFormer-DeepLab beats published baselines by +13.1 STQ (KITTI-STEP), +21 mIoU (VSPW), +2.9 track-mAP (YouTube-VIS), +3.6 DSTQ (SemKITTI-DVPS), and +13.6 STQ and +3.9 VPQ (VIPSeg).The comparisons are on the test sets for KITTI-STEP, VSPW, SemKITTI-DVPS, and VIPSeg, and the YouTube-VIS-2019 validation set.Source: §1 · p. 2

2. Related Works

Prior work addresses video semantic, instance, and panoptic segmentation with distinct approaches, while the paper situates TubeFormer-DeepLab as a unified alternative. The related work also describes extensions to depth-aware video panoptic segmentation.

2. Related Works

  • VSS methods exploit temporal information through feature warping [29] [42] [52] [94] [95], while VSPW introduced a large-scale benchmark and a video-context baseline.Source: §2 · p. 2
  • Most state-of-the-art VIS methods use detection-based approaches that allow overlapping mask predictions [5] [9] [28] [40] [54] [57] [77] [84].The paper distinguishes its use of latent memory for per-frame segmentation from IFC [40], which uses memory features for inter-frame communication.Source: §2 · p. 2
  • VPS requires temporally consistent panoptic outputs, labels every pixel, and disallows overlapping instance masks [44].The related work contrasts this with complicated pipelines using task-specific heads and branches [44] [69].Source: §2 · p. 3
  • Depth-aware VPS adds per-pixel depth estimation to VPS, and the paper notes that Video K-Net [50] also develops a unified video panoptic framework.Source: §2 · p. 3

3. Method

TubeFormer-DeepLab formulates video segmentation as assigning task-specific labels to tubes of temporally linked masks, then predicts those tubes through a hierarchical mask-transformer architecture. Its training and inference strategies operate on clips and stitch clip predictions into video-level results.

3.1. Video Segmentation Formulation

  • Video segmentation tasks can be formulated as assigning labels to temporally linked mask tubes: semantic categories alone, or categories paired with instance identities.The latter supports instance segmentation for foreground things and panoptic segmentation for things and stuff.Source: §3.1 · p. 3

3.2. TubeFormer-DeepLab Architecture

  • The model predicts class scores and tube embeddings, whose dot products with video pixel features produce tube masks.The resulting tubes and their class predictions form the clip-level segmentation.Source: §3.2 · p. 4
  • A latent dual-path transformer passes messages between frame features and latent memory before global attention models video-clip features.The hierarchical structure addresses the difficulty of learning attention over large, multi-frame feature maps.Source: §3.2 · p. 4
  • Separate thing- and stuff-specific global memories reflect that a thing class may have multiple instances, while each stuff class has at most one mask.Stuff memory slots are assigned to ground-truth stuff classes rather than included in bipartite matching.Source: §3.2 · p. 5

3.3. Training Strategy

  • Training matches predicted and ground-truth tubes using a VPQ-style similarity, then optimizes that similarity with auxiliary tube-ID, semantic-segmentation, and instance-discrimination losses.The similarity multiplies predicted class probability by the Dice coefficient between the predicted and ground-truth masks.Source: §3.3 · p. 5
  • A temporal consistency loss minimizes the distance between tube logits on clips’ overlapping frames, while clip-paste augments training by pasting thing or stuff region tubes.Clip-paste is applied with probability 0.5; the consistency loss is L1 and affects pixel and global-memory paths.Source: §3.3 · p. 5
  • For depth-aware panoptic segmentation, a separate depth module predicts pixel depth from CNN backbone features; sharing decoded segmentation features slightly degrades performance.The depth loss combines scale-invariant logarithmic error and relative squared error.Source: §3.3 · p. 5

3.4. Inference Strategy

  • Video-level inference runs clip inference with T−1 overlapping frames, then stitches results by matching tubes on their overlap using IoU.At clip level, class labels and per-pixel tube IDs are selected with argmax; IDs below class confidence 0.7 are set to void.Source: §3.4 · pp. 5–6

4. Experimental Results

TubeFormer-DeepLab is evaluated across video panoptic, semantic, instance, and depth-aware panoptic segmentation datasets. It reports competitive or state-of-the-art results across these tasks, alongside ablations and visualizations of its hierarchical attention.

4.1. Datasets

  • The evaluation covers KITTI-STEP and VIPSeg for Video Panoptic Segmentation (VPS), VSPW for Video Semantic Segmentation (VSS), YouTube-VIS for Video Instance Segmentation (VIS), and SemKITTI-DVPS for depth-aware video panoptic segmentation (DVPS).Source: §4 · p. 6

4.3. Main Results

  • 65.25 STQ on KITTI-STEP, +13.6 STQ and +3.9 VPQ on VIPSeg, and 67.0 DSTQ (+3.4 over ViP-DeepLab) on SemKITTI-DVPS.On KITTI-STEP, the model ranks first among single unified approaches; on SemKITTI-DVPS, it reaches a new state of the art.Source: Table 1 · p. 6, §4.3 · pp. 6–7, Table 2 · p. 7
  • On VSPW, TubeFormer-DeepLab outperforms competing methods on the validation set, ranks fourth of 17 on the test set, and exceeds TCB by +21 mIoU there.The test-set comparison notes that other methods may use ensembles, multi-scale inference, or pseudo-labels.Source: §4.3 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
  • On YouTube-VIS, TubeFormer-DeepLab-B4 exceeds VisTR by +7.4 AP and IFC by +2.9 AP among end-to-end methods.The model predicts one unique mask per object, whereas other methods often generate overlapping masks favored by AP; its AR1 gains indicate benefit in non-overlapping segmentation.Source: Table 5 · p. 7, Table 6 · p. 7, §4.3 · p. 7

4.4. Ablation Studies

  • Adding latent memory and frame-latent attention to the baseline improves STQ by +1.7, while adding memory-to-latent and latent-to-memory attention shows no improvement.The authors conclude frame-latent attention is sufficient for effective hierarchical attention between the latent and global dual-path transformers.Source: §4.4 · p. 8
  • Temporal consistency loss improves longer-video inference by +0.5 STQ, and clip-paste augmentation adds a further +0.9 STQ.Reverting either shared semantic/panoptic predictions or separate thing/stuff global memories drops STQ by -1.1.Source: §4.4 · p. 8
  • Scaling the backbone from n = 1 to n = 3 raises STQ from 73.19 to 74.25, while n = 4 begins to saturate on KITTI-STEP.Each increase in n adds 13M parameters; the authors report better B4 performance on larger VSPW and YouTube-VIS datasets.Source: §4.4 · p. 8

4.5. Visualization

  • Global-memory attention is spatially and temporally separated for individual thing or stuff tubes, while latent-memory attention specializes to scene areas or semantically similar regions.The visualization uses three input frames and shows global-memory attention for selected predicted tube regions and latent attention for four of 16 latent memories.Source: Figure 4 · p. 8, §4.5 · p. 9

5. More Experimental Results

TubeFormer-DeepLab improves on task-specific baselines across video panoptic, semantic, and instance segmentation, with results reported for both per-pixel and per-mask inference.

5. More Experimental Results

  • +12 and +14 STQ, and +11.1 and +8.1 VPQ, over Motion-DeepLab and VPSNet, respectively, on KITTI-STEP validation.The comparison uses TubeFormer-DeepLab-B1 with ResNet-50 and axial attention.Source: §5 · p. 9
  • +20.2 mIoU over TCB on VSPW validation with TubeFormer-DeepLab-B1.Source: §5 · p. 9
  • +5.8 AP over STEm-Seg on YouTube-VIS-2019 validation among non-overlapping methods, with B1 also exceeding its ResNet-101 version by +1.8 AP and B4 by +10.8 AP.Source: §5 · p. 9
  • In the TubeFormer-DeepLab framework, per-mask inference improves over per-pixel inference by more than 2 AP; TubeFormer-DeepLab-B4 per-mask outperforms MaskProp, VisTR, and IFC, and performs comparably with Seq Mask R-CNN.Source: §5 · p. 9
  • +3.9 AR1 and +3.0 AR10 over the second-best Seq Mask-RCNN method, with TubeFormer-DeepLab using T=5 versus T=13 or 36 for other proposal-based approaches.Source: §5 · p. 9

6. Visualization

The visualizations show tube-specific global attention and task-dependent latent-memory attention across video panoptic, semantic, and instance segmentation, alongside depth-aware outputs.

6. Visualization

  • Global-memory attention clusters spatiotemporally around tube regions; one memory corresponds to a semantic category in VSS and an instance identity in VIS.Source: §6 · pp. 9–10
  • Latent memories specialize in scene regions, tube boundaries, distant regions, moving parts, or small objects, with instance-specific attention in VIS but not VSS.Source: §6 · p. 10
  • Figure 8 visualizes depth-aware video panoptic segmentation on SemKITTI-DVPS, where TubeFormer-DeepLab generates temporally consistent panoptic segmentation and monocular depth estimation results.Source: §6 · p. 10

7. Discussion

The discussion places video segmentation in the broader goal of unifying semantic and instance segmentation through panoptic segmentation, and presents Figure 5's VPS visualization.

7. Discussion

  • Panoptic segmentation already unifies semantic and instance segmentation, and Panoptic-DeepLab and Naive-Student achieved state-of-the-art results on all three tasks.Source: §7 · p. 10
  • Figure 5 presents KITTI-STEP input frames, global- and latent-memory attention, and video panoptic segmentation outputs.Source: Figure 5 · p. 11

8. Limitations

Clip-level inference limits the tracking horizon, so objects with longer tracks may be missed; the authors identify long-term tracking as future work.

8. Limitations

  • With clip values T=2 for VPS and VSS or T=5 for VIS, the model may miss objects whose tracks exceed the clip length.The authors leave efficient integration of long-term tracking for future work.Source: §8 · p. 11
  • The authors position TubeFormer-DeepLab as an initial unified approach and a potential baseline for future research.Source: §8 · p. 11

9. Conclusion

TubeFormer-DeepLab reframes video segmentation as assigning labels to video tubes, offering a general solution across multiple tasks. The conclusion presents it as a mask-transformer architecture addressing tasks previously handled by fundamentally divergent models.

9. Conclusion

  • TubeFormer-DeepLab is a novel architecture based on mask transformers, addressing tasks previously tackled by fundamentally divergent models.The conclusion highlights video semantic, instance, and panoptic segmentation as examples.Source: §9 · p. 11
  • TubeFormer-DeepLab offers a general solution to multiple video segmentation tasks by directly predicting class-labeled tubes.The labels encode different values depending on the target task.Source: §9 · p. 11
  • The proposed paradigm formulates video segmentation as partitioning video tubes with different predicted labels.The conclusion frames this as a new paradigm for video segmentation.Source: §9 · p. 11
Loading 2205.15361v2…