PaperlayerView exact revision on arXiv

Source-linked AI summary

VITA: Video Instance Segmentation via Object Token Association

Miran Heo, Sukjun Hwang, Seoung Wug Oh, Joon-Young Lee, Seon Joo Kim

arXiv:2206.04403v2cs.CV

TL;DR

VITA addresses the difficulty of long-video understanding in offline video instance segmentation by associating frame-level object tokens rather than referring to dense spatio-temporal backbone features. It achieves state-of-the-art results on YouTube-VIS 2019 and 2021 and OVIS, while supporting practical processing of long videos.

  • Problem

    Offline VIS methods rely on dense spatio-temporal features, which can hinder relevant-context retrieval and make long-video processing difficult.

    Source: §1 · p. 2, §3.2 · p. 4

  • Method

    VITA associates object-centric tokens from an image detector across frames to understand a video without using dense spatio-temporal backbone features.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    VITA achieves state-of-the-art performance on YouTube-VIS 2019 and 2021 and OVIS, reporting 49.8 AP and 45.7 AP on the YouTube-VIS benchmarks and 19.6 AP on OVIS with ResNet-50.

    Source: §1 · p. 2, §4.3 · p. 7

  • Takeaways & Limitations

    The object-token design supports complete-offline processing of long videos, including OVIS on a single 12GB GPU, and competitive performance with frozen frame-level detectors.

    Source: §1 · p. 2, §6 · p. 10

    VITA cannot process an infinite number of frames, and object tokens may struggle to identify complex behaviors spanning very long sequences because they do not explicitly use temporal information.

    Source: §5 · p. 10

Abstract

from arXiv · show

We introduce a novel paradigm for offline Video Instance Segmentation (VIS), based on the hypothesis that explicit object-oriented information can be a strong clue for understanding the context of the entire sequence. To this end, we propose VITA, a simple structure built on top of an off-the-shelf Transformer-based image instance segmentation model. Specifically, we use an image object detector as a means of distilling object-specific contexts into object tokens. VITA accomplishes video-level understanding by associating frame-level object tokens without using spatio-temporal backbone features. By effectively building relationships between objects using the condensed information, VITA achieves the state-of-the-art on VIS benchmarks with a ResNet-50 backbone: 49.8 AP, 45.7 AP on YouTube-VIS 2019 & 2021, and 19.6 AP on OVIS. Moreover, thanks to its object token-based structure that is disjoint from the backbone features, VITA shows several practical advantages that previous offline VIS methods have not explored - handling long and high-resolution videos with a common GPU, and freezing a frame-level detector trained on image domain. Code is available at https://github.com/sukjunhwang/VITA.

1 Introduction

Earlier offline VIS methods used dense spatio-temporal features to jointly track and segment objects, but could struggle with long sequences. VITA instead builds on object-centric tokens, achieves state-of-the-art results across VIS benchmarks, and offers practical benefits for long and high-resolution videos.

1 Introduction

  • Prior offline VIS approaches jointly predict trajectories and masks using dense spatio-temporal features, while follow-up work focused on designing interactions between frames with a small number of tokens.These approaches motivate the question of what information is important for understanding a video.Source: §1 · p. 1
  • VITA represents a video as relationships among object-centric tokens, avoiding dense spatio-temporal backbone features and enabling global analysis of the collected tokens.The paper motivates this design by noting that dense reference features can hinder Transformer layers from retrieving relevant information in long sequences.Source: §1 · p. 2
  • With a ResNet-50 backbone, VITA achieves 49.8 AP on YouTube-VIS 2019, 45.7 AP on YouTube-VIS 2021, and 19.6 AP on OVIS, including a 5.1 AP improvement over the previous best on YouTube-VIS 2021.The paper also reports OVIS results for a 292-frame video using a single 12GB GPU.Source: §1 · p. 2
  • VITA can process 1392 frames at once on a single 12GB GPU regardless of video resolution, and can use a parameter-frozen image detector with little performance loss.It adds 6% parameters to a Swin-L detector.Source: §1 · p. 2

2 Related Works

VIS research includes online approaches that associate tracklets across local windows and offline approaches that predict across a whole sequence. VITA offers a distinct offline approach based on object tokens rather than dense backbone features.

2 Related Works

  • Online VIS methods associate tracklets across local windows, using hand-crafted or learned matching, while propagation-based and cascaded approaches use memories or correspondence learning.MaskTrack R-CNN established a tracking branch on an image instance segmentation model; other tracking-by-detection approaches compare per-frame predictions before association.Source: §2 · pp. 2–3
  • Offline VIS methods jointly predict trajectories and masks across a whole video, using dense backbone features and varied designs for exchanging information between frames.Examples include memory tokens, mask-oriented representations, temporal backbone exchange, and instance queries.Source: §2 · p. 3
  • VITA proposes an offline VIS paradigm that interprets a video from a collection of object tokens instead of dense spatio-temporal backbone features.This is presented as an alternative to the dense-feature approach of earlier offline VIS methods.Source: §2 · p. 3
  • MOT global trackers associate frame-level predictions across a sequence using graph optimization, Transformer queries, or global set classification.These approaches seek identity matches or aggregate information across frames to classify tracklets.Source: §2 · p. 3

3 Method

VITA builds video-level instance understanding by associating object-centric information from frame-level detections, rather than relying on dense spatio-temporal backbone features. Its pipeline uses a frame-level detector, an Object Encoder and Decoder, and clip-wise losses.

3.1 Frame-level Detector

  • Mask2Former represents each image object with a frame query and uses those queries to classify and segment matched objects.Segmentation uses a dot product between dynamic 1×1 convolutional weights from the queries and per-pixel embeddings.Source: §3.1 · pp. 3–4

3.2 VITA

  • VITA runs the frame-level detector independently on each frame, then collects frame queries to build video-level information and predict categories and masks.The collected queries carry object-centric information; per-pixel embeddings are also used by VITA.Source: §3.2 · p. 4
  • Object Encoder converts frame queries into object tokens and uses temporally shifted local self-attention windows to exchange information across frames.The windows have size W and do not overlap within a partition.Source: §3.2 · p. 4, Figure 3 · p. 4
  • Earlier offline VIS decoders retrieve object context from dense spatio-temporal features, which can hinder information retrieval and make long-video memory demands intractable.VITA’s Object Decoder instead reads condensed object tokens into trainable video queries, supporting processing of long, high-resolution videos.Source: §3.2 · pp. 4–5
  • VITA’s class head predicts video-query categories, while its mask head generates per-query embeddings for mask logits across all frames.The mask logits result from multiplying the mask embeddings by the frame-level pixel embeddings.Source: §3.2 · p. 5

3.3 Clip-wise losses

  • Clip-wise training matches video predictions to ground truth with Hungarian assignment and trains video and frame queries to cluster by instance identity.The total loss combines frame-level, video-level, and similarity losses.Source: §3.3 · p. 5

4 Experiments

Across YouTube-VIS 2019 and 2021 and OVIS, VITA reports leading results. The experiments also examine its dataset settings, implementation, inference capacity, token pruning, similarity loss, and frozen-detector use.

4.1 Datasets

  • YouTube-VIS 2019 has short videos and few unique instances, while OVIS features heavy occlusion, similar-looking instances, and videos up to 292 frames.YouTube-VIS 2021 adds more objects with confusing trajectories; its additional validation videos average 39.7 frames.Source: §4.1 · pp. 5–6

4.3 Main Results

  • 49.8 AP on YouTube-VIS 2019, 45.7 AP on YouTube-VIS 2021, and 19.6 AP on OVIS are reported as state-of-the-art results.The comparisons cover YouTube-VIS 2019 and, with a ResNet-50 backbone, YouTube-VIS 2021 and OVIS.Source: Table 1 · p. 6, Table 2 · p. 7, §4.3 · p. 7

4.2 Implementation Details

  • The default setup uses a three-layer Object Encoder with W = 6, a six-layer Object Decoder, and 100 video queries.Training proceeds from COCO pretraining to VIS datasets with pseudo videos; reported scores are means of five runs.Source: §4.2 · pp. 6–7

4.4 Ablation Studies

  • W = 6 balances performance and inference scalability as larger Object Encoder windows provide a wider temporal receptive field.The study varies shifted attention-window size during inference.Source: §4.4 · p. 8, Table 3 · p. 8
  • At 360 × 640 with W = 6, VITA processes about 11× more frames than IFC; complete-offline inference also outperforms clip splitting with heuristic association on OVIS.The comparison uses a single 12GB Titan XP GPU; clip-wise outputs are matched by mask IoU using Greedy or Hungarian algorithms.Source: §4.4 · p. 8, Table 5 · p. 8, Table 4 · p. 8
  • Keeping 75% of frame queries causes only a 0.1 AP decrease while increasing VITA's maximum frame count from 1392 to 2635.Queries are retained according to their ascending “no object” probability.Source: §4.4 · pp. 8–9, Table 6 · p. 8
  • Similarity loss adds 1.8 AP, and visualizations show VITA recovering heavily occluded instances that the frame-level detector misclassifies.The authors attribute the gain to reduced embedding discrepancies between equal identities and the recovery to temporal information.Source: §4.4 · p. 9, Figure 5 · p. 9, Figure 6 · p. 10
  • VITA can be trained with a frozen COCO-pretrained detector, but domain mismatch produces extra foreground predictions that lower AP through false positives.The strategy is presented as useful when training resources or storage for separate image and video models are limited.Source: §4.4 · p. 9, Table 7 · p. 9

5 Limitations

VITA remains limited in the number of frames it can process and may struggle to identify complex behaviors spanning very long sequences because its object tokens do not explicitly use temporal information.

5 Limitations

  • VITA’s current architecture is limited in processing an infinite number of frames.The authors identify this as a limitation for ultimate long-video understanding.Source: §5 · p. 10
  • Object tokens do not explicitly use temporal information, which may hinder identification of complex behaviors spanning very long sequences.The authors identify explicit designs for these issues as a future direction.Source: §5 · p. 10
  • The authors identify both frame-count limits and long-span behavior recognition as constraints on ultimate long-video understanding.They propose devising explicit designs to address these issues as future work.Source: §5 · p. 10

6 Conclusion

VITA uses object queries decoded by independent frame-level detectors for offline video instance segmentation. The authors report state-of-the-art performance on popular VIS benchmarks and practical advantages for long and high-resolution video processing.

6 Conclusion

  • VITA achieves state-of-the-art performance on popular VIS benchmarks, including YouTube-VIS 2019 and 2021 and OVIS.The conclusion describes object-oriented information as effective for improving performance.Source: §6 · p. 10
  • VITA directly leverages object queries decoded by independent frame-level detectors rather than the approach used by existing offline methods.It is built on an off-the-shelf image instance segmentation model.Source: §6 · p. 10
  • VITA shows fast convergence and competitive performance when trained on frozen detectors.The conclusion links this design to absorbing spatial knowledge purely from an image object detector.Source: §6 · p. 10

Broader Impact

VITA targets end-to-end processing of long, high-resolution videos for VIS. The authors point to video editing as a possible application and caution that VIS research should consider privacy-violating misuse.

Broader Impact

  • VITA is designed for end-to-end VIS on long and high-resolution videos while targeting state-of-the-art performance.Source: Broader Impact · p. 10
  • The authors hope VITA can positively affect industrial areas such as video editing applications.Source: Broader Impact · p. 10
  • The authors caution that VIS research must be aware of potential misuse that violates personal privacy.Source: Broader Impact · p. 10

Appendix

The appendix provides additional training details and explains VITA’s inference procedure.

Appendix

  • The appendix provides more details about VITA’s training.Source: Appendix · p. 13
  • The appendix explains VITA’s inference procedure.Source: Appendix · p. 13
  • Appendix A covers training details, while Appendix B explains inference.Source: Appendix · p. 13

A.1 Implementation

VITA is trained in two stages, using pseudo-videos from still images for VIS fine-tuning. The appendix specifies its loss setup and a memory-driven choice of decoder outputs for training.

A.1 Implementation

  • VITA is pretrained for image instance segmentation on COCO, then fine-tuned on three VIS datasets with six-frame clips and dataset-specific iteration schedules.Training uses batch sizes of 16 for pretraining and 8 for fine-tuning.Source: §A.1 · p. 13

A.1.1 Pseudo-video generation

  • Single COCO images are resized, randomly cropped T times, and resized again to form pseudo-videos of length T.The initial short side is chosen from 400, 500, or 600 pixels; the final short edge is selected from 288 to 512 pixels in steps of 32.Source: §A.1.1 · p. 13

A.2 Loss function

  • The frame-level detector loss combines categorical cross-entropy with mask-related binary-cross-entropy and Dice losses; video-level losses use the same hyperparameters.The reported weights are λcls = 2, λce = 2, and λdice = 5; the video-level Dice loss extends the frame-level function across time, as in IFC [15].Source: §A.2 · p. 13

A.3 Building VITA on Mask2Former

  • VITA training uses outputs from Mask2Former’s last three decoder layers because using all nine outputs exceeds available GPU memory.The Object Encoder and Object Decoder follow standard DETR [5] Transformer architectures, with decoder attention order changed and dropout eliminated.Source: §A.3 · p. 13, §A.1 · p. 13

B Inference procedure

VITA’s inference procedure collects frame-level detector outputs sequentially and then processes the full-video frame queries with its Object Encoder. Previous Transformer-based offline VIS methods instead aggregate backbone features across the video before their VIS module.

B Inference procedure

  • VITA collects only frame queries and mask features from each frame, then sends the full-video frame queries to its Object Encoder.Both VITA and Mask2Former-VIS generate video-level predictions using video queries and mask features.Source: B Inference procedure · pp. 13–14
  • Previous Transformer-based offline VIS models aggregate the entire video’s backbone features before passing them to the VIS prediction module.The comparison procedure collects backbone features sequentially to avoid memory explosion before entering each model’s VIS prediction module.Source: B Inference procedure · pp. 13–14
Loading 2206.04403v2…