PaperlayerView exact revision on arXiv

Source-linked AI summary

In Defense of Online Models for Video Instance Segmentation

Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, Xiang Bai

arXiv:2207.10661v1cs.CV

TL;DR

Online VIS is attractive for long and ongoing videos, but its performance trails offline methods and frame association remains error-prone. The paper introduces IDOL, a contrastive-learning framework with stronger instance embeddings and history use, and reports state-of-the-art results across three benchmarks, surpassing prior online and offline methods.

  • Problem

    Online VIS supports long and ongoing videos, but its performance lags offline methods because frame association is error-prone, motivating methods that close this gap.

  • Method

    IDOL uses contrastive learning to produce discriminative, temporally consistent instance embeddings, with optimal-transport sample selection and temporally weighted softmax for association.

  • Results

    IDOL outperforms prior online and offline methods on YouTube-VIS 2019, YouTube-VIS 2021, and OVIS, achieving 64.3 AP, 56.1 AP, and 42.6 AP, respectively.

  • Takeaways & Limitations

    The results show that improving instance embeddings and association can bridge the online–offline performance gap across benchmarks.

Abstract

from arXiv · show

In recent years, video instance segmentation (VIS) has been largely advanced by offline models, while online models gradually attracted less attention possibly due to their inferior performance. However, online methods have their inherent advantage in handling long video sequences and ongoing videos while offline models fail due to the limit of computational resources. Therefore, it would be highly desirable if online models can achieve comparable or even better performance than offline models. By dissecting current online models and offline models, we demonstrate that the main cause of the performance gap is the error-prone association between frames caused by the similar appearance among different instances in the feature space. Observing this, we propose an online framework based on contrastive learning that is able to learn more discriminative instance embeddings for association and fully exploit history information for stability. Despite its simplicity, our method outperforms all online and offline methods on three benchmarks. Specifically, we achieve 49.5 AP on YouTube-VIS 2019, a significant improvement of 13.2 AP and 2.1 AP over the prior online and offline art, respectively. Moreover, we achieve 30.2 AP on OVIS, a more challenging dataset with significant crowding and occlusions, surpassing the prior art by 14.8 AP. The proposed method won first place in the video instance segmentation track of the 4th Large-scale Video Object Segmentation Challenge (CVPR2022). We hope the simplicity and effectiveness of our method, as well as our insight into current methods, could shed light on the exploration of VIS models.

1 Introduction

VIS methods are divided into online and offline paradigms, but offline models face resource limits while online models lag in performance. Analysis attributes this gap primarily to frame association, motivating IDOL’s contrastive-learning approach and strong benchmark results.

  • 1 Introduction: Offline models process whole videos in one step but are constrained by memory on long or ongoing videos, whereas online models process frames sequentially and track instances across time.Videos exceeding 50 frames can exceed a 32G-RAM GPU limit for offline processing.
  • 1 Introduction: Per-clip segmentation improves IFC mask quality by 3.7 AP on YouTube-VIS, but longer clips reduce IFC and SeqFormer performance on OVIS by 1.8 and 2.2 AP.The evidence indicates that multi-frame segmentation is not consistently robust across methods and dataset difficulty.
  • 1 Introduction: Frame association, rather than mask quality, is the main source of the online–offline performance gap, while offline association degrades sharply on complex videos.On OVIS, black-box association degrades IFC by 12.3 AP and SeqFormer by 20.9 AP.
  • 1 Introduction: IDOL learns discriminative, temporally consistent instance embeddings with contrastive learning, using optimal transport for sample selection and temporally weighted softmax for re-identification.These choices target false positives in crowded or occluded scenes and improve association consistency.
  • 1 Introduction: 64.3 AP, 56.1 AP, and 42.6 AP on YouTube-VIS 2019, YouTube-VIS 2021, and OVIS establish IDOL as state of the art, improving over prior online methods by 13.2–14.7 AP and offline methods by up to 2.1 AP.The method also achieved first place in the YouTube-VIS Challenge 2022.

2 Related Work

Related work covers online VIS systems that add association mechanisms to image-level segmentation and offline transformer models that model video-level instances. These approaches improve temporal modeling but face scalability or association challenges in longer and more complex videos.

  • 2 Related Work: Online VIS methods commonly build on image-level instance segmentation with tracking or association heads using appearance, semantic, spatial, and confidence cues.MaskTrack R-CNN is presented as a baseline in this paradigm.
  • 2 Related Work: Offline approaches such as VisTR, IFC, and SeqFormer model video-level instances or communicate across frames to improve performance while reducing computational demands.VisTR’s per-frame instance embeddings make longer videos and complex scenes difficult, whereas IFC reduces computation and memory usage.

3 Method

IDOL combines a Deformable DETR-based segmentation pipeline with cross-frame contrastive learning and memory-based association. It learns discriminative embeddings, dynamically selects contrastive samples, and uses temporally weighted history to stabilize instance matching under challenging video conditions.

  • 3.1 Instance Segmentation.: IDOL uses DeformableDETR with a dynamic mask head as its instance-segmentation pipeline and can be coupled with other segmentation methods with minor modifications.The pipeline predicts instance embeddings, boxes, class labels, and per-frame masks.
  • 3.2 Contrastive Learning between Frames.: Contrastive learning pulls embeddings of the same instance together across frames and pushes embeddings of different instances apart to improve cross-frame association.A lightweight contrastive head decodes embeddings from instance features.
  • 3.2 Contrastive Learning between Frames.: For each key-frame instance, IDOL dynamically selects low-cost reference predictions as positives and high-cost predictions as negatives using optimal transport.The method supports multiple positive samples and uses a contrastive loss over positive-negative embedding pairs.
  • 3.3 Instance Association.: IDOL addresses fast motion, occlusion, crowded objects, and unstable frame-level predictions with temporally weighted softmax matching and a memory-bank association strategy.Memory instances retain multiple temporal embeddings, which are combined by a temporally weighted sum before bidirectional similarity is computed.
  • 3.3 Instance Association.: Temporal embedding history and duration-based confidence help re-identify instances missing under occlusion and preserve association consistency and integrity.The best-scoring memory instance is assigned when similarity exceeds 0.5; otherwise, a high-confidence unmatched prediction starts a new identity.

4 Experiments

IDOL is evaluated through oracle analyses, benchmark comparisons, and ablations across YouTube-VIS 2019, YouTube-VIS 2021, and OVIS. The experiments examine association quality, clip-length effects, efficiency, and the contributions of contrastive embeddings and inference strategies.

  • Analysis of Current SOTA VIS Models: Oracle experiments show that association, rather than mask quality, is the principal source of the online–offline performance gap.With ground-truth instance IDs, CrossVIS outperforms IFC on both datasets, but predicted association causes a 9.4 AP drop for CrossVIS versus 3.3 AP for IFC on YouTube-VIS.
  • Analysis of Current SOTA VIS Models: Longer clips make offline association increasingly fragile, reducing IFC by 12.3 AP and SeqFormer by 20.9 AP on OVIS at clip length 30.At clip length 5, the corresponding drops are 3.4 AP for IFC and 6.8 AP for SeqFormer.
  • Analysis of Current SOTA VIS Models: IDOL’s frame-to-clip oracle gaps are much smaller than SeqFormer’s on OVIS, indicating more robust association in challenging videos.The comparison is based on oracle experiments with different clip lengths for IDOL and SeqFormer.
  • Main Results: 30.2 AP on OVIS surpasses the previous method by 14.8 AP, while the stronger Swin-L backbone reaches 42.6 AP.OVIS contains long videos, heavy occlusion, and complex motion, exceeding the computational capability of standard offline processing.
  • Ablation Study: Contrastive embeddings raise OVIS AP from 11.0 to 18.4, optimal transport adds 2.3 AP on both datasets, and the embedding association strategy raises AP from 18.4 to 26.7 on OVIS.On YouTube-VIS, the embedding association strategy increases AP from 31.8 to 42.5.

5 Conclusions

The paper argues that online VIS can overcome its performance disadvantage by addressing error-prone frame association with discriminative and robust instance features.

  • IDOL mainly targets the error-prone association between frames that creates the performance gap between online and offline VIS methods.
  • IDOL significantly outperforms online and offline methods and achieves new state-of-the-art results on three benchmarks.

A.1 Qualitative Results

Qualitative results highlight IDOL’s ability to distinguish similar instances, maintain temporal consistency during complex motion, and remain robust under occlusion and crowded interactions.

  • IDOL more accurately segments and tracks instances with similar appearances through contrastive learning.
  • Better temporal consistency in the learned embeddings benefits tracking under high-speed, large, and complex motions.
  • Stable, discriminative embeddings and temporally weighted softmax improve robustness in crowded scenes with heavy occlusions and position exchanges.

A.2 Optimal Transport

IDOL replaces fixed IoU thresholds for sample selection with an optimal transport formulation that produces more accurate training embeddings and reduces false positives.

  • Optimal transport selects more accurate positive embeddings than IoU thresholds in occluded and crowded scenes, reducing false positives.The IoU-based method can assign boxes from an occluding instance as positives for the key instance.
  • The method selects positives and negatives dynamically using prediction costs derived from top IoU values rather than fixed thresholds.

A.3 Temporally Weighted Softmax

The qualitative comparisons show that IDOL’s association strategy improves identity consistency when instances disappear, reappear, or exchange positions during difficult video dynamics.

  • Without temporally weighted softmax, occluded or disappearing instances can receive new or swapped identities when they reappear.
  • Temporally weighted softmax is designed to preserve identity consistency after occlusion and disappearance.
  • IDOL is more robust than prior methods for similar appearances, severe occlusions, crowded scenes, and frequent position exchanges.
  • IDOL performs better on high-speed, large, and complex motions than prior methods in the qualitative comparisons.
Loading 2207.10661v1…