PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Instance Segmentation

Linjie Yang, Yuchen Fan, Ning Xu

arXiv:1905.04804v4cs.CV

TL;DR

Video instance segmentation extends image instance segmentation to simultaneous detection, segmentation, and tracking of instances in videos, addressing the lack of a suitable large-scale benchmark. The paper introduces YouTube-VIS and MaskTrack R-CNN, whose tracking branch supports joint processing, and reports that the method achieves the best results across all evaluation metrics on validation and test sets.

  • Problem

    Existing video segmentation datasets lack the exhaustive instance labels and cross-frame identities required for video instance segmentation.

  • Method

    The paper introduces YouTube-VIS and MaskTrack R-CNN, which adds a tracking branch to Mask R-CNN for joint detection, segmentation, and tracking.

  • Results

    MaskTrack R-CNN achieves the best results under all evaluation metrics on both the validation and test sets.

  • Takeaways & Limitations

    Video instance segmentation is presented as a new task and benchmark direction combining object detection, segmentation, and tracking for video understanding.

  • Takeaways & Limitations

    The method can confuse identities when objects change appearance substantially or when multiple similar objects occlude one another.

Abstract

from arXiv · show

In this paper we present a new computer vision task, named video instance segmentation. The goal of this new task is simultaneous detection, segmentation and tracking of instances in videos. In words, it is the first time that the image instance segmentation problem is extended to the video domain. To facilitate research on this new task, we propose a large-scale benchmark called YouTube-VIS, which consists of 2883 high-resolution YouTube videos, a 40-category label set and 131k high-quality instance masks. In addition, we propose a novel algorithm called MaskTrack R-CNN for this task. Our new method introduces a new tracking branch to Mask R-CNN to jointly perform the detection, segmentation and tracking tasks simultaneously. Finally, we evaluate the proposed method and several strong baselines on our new dataset. Experimental results clearly demonstrate the advantages of the proposed algorithm and reveal insight for future improvement. We believe the video instance segmentation task will motivate the community along the line of research for video understanding.

1. Introduction

The paper introduces video instance segmentation, extending image instance segmentation to simultaneous detection, segmentation, and tracking across video frames. It addresses the lack of suitable benchmarks with YouTube-VIS and proposes MaskTrack R-CNN for the task.

  • Task motivation: Video instance segmentation simultaneously detects, segments, and tracks object instances across video frames.Unlike image instance segmentation, it must also maintain instance identities over time.
  • Novelty and scope: The paper presents video instance segmentation as a formally defined task and reports it as the first work to address it.The task is motivated by applications requiring video-level object masks, including video editing, autonomous driving, and augmented reality.
  • Dataset motivation: Existing video segmentation datasets lack either exhaustive instance labeling or object identities required by this task.This gap motivates constructing a dedicated benchmark.
  • Dataset contribution: YouTube-VIS contains 2,883 high-resolution videos, 40 object categories, 4,883 unique video instances, and 131k high-quality masks.The dataset is intended to support video instance segmentation and related video understanding tasks.
  • Method contribution: MaskTrack R-CNN adds a tracking branch to Mask R-CNN and is evaluated against several baselines on the new dataset.The paper presents the method as a joint approach to video instance segmentation.

2. Related Work

Video instance segmentation builds on image instance segmentation and video tracking while adding semantic instance correspondence across frames. Related video segmentation settings generally track objects without requiring semantic categories or the task’s exhaustive instance labels.

  • Image instance segmentation: Image instance segmentation groups pixels by semantic class and object instance, commonly using proposal generation followed by box and mask prediction.Video instance segmentation additionally determines object correspondence across frames.
  • Video object tracking: Video object tracking includes detection-based tracking and detection-free tracking, whereas video instance segmentation combines tracking with per-frame instance segmentation.The proposed task requires both segmentation and cross-frame identity assignment.
  • Distinction from related tasks: Existing video segmentation settings may omit exhaustive labeling or object identities, making them unsuitable for video instance segmentation.Some settings also treat targets as general objects without semantic categories.

3. Video Instance Segmentation

Video instance segmentation predicts categorized object instances as temporally bounded mask sequences with confidence scores. Evaluation compares hypotheses with ground truth using spatial-temporal IoU, AP, and AR, rewarding detection, accurate boundaries, and reliable tracking.

  • Problem definition: The task defines a category set C and represents each ground-truth object with a category label and binary masks spanning its visible video interval.Each object is associated with a starting and ending time within the T-frame sequence.
  • Problem definition: Each predicted instance hypothesis contains a category label, confidence score, and predicted binary-mask sequence with its own temporal interval.The confidence score is used by the evaluation metrics.
  • Task objective: A good method must detect all instances, track them reliably, and localize their instance boundaries accurately.The task minimizes the difference between ground-truth instances and predicted hypotheses.
  • Evaluation metrics: Evaluation uses AP and AR adapted from image instance segmentation, with AP averaged over ten IoU thresholds from 50% to 95% in 5% steps.AP is the area under the precision-recall curve, while AR is maximum recall for a fixed number of segmented instances per video.
  • Spatiotemporal IoU: The proposed IoU pads each ground-truth and predicted mask sequence to all T frames before comparing them.This sequence-level computation measures spatial-temporal consistency and penalizes failures to track objects across frames.

4. YouTube-VIS

YouTube-VIS is a new large-scale benchmark designed for video instance segmentation and related pixel-level video understanding tasks. It builds on YouTube-VOS while adding exhaustive instance-oriented annotations and challenging video cases.

  • Its videos include challenges such as occlusion, appearance change, and heavy camera motion, alongside high-quality annotations.
  • YouTube-VIS was created because existing video segmentation datasets lacked either exhaustive labeling or instance identities.
  • YouTube-VIS complements YouTube-VOS with more objects and supports video instance segmentation, video object detection, and video semantic segmentation.
  • Figure 2 reports the number of unique video objects across the benchmark’s 40 categories.

5. MaskTrack R-CNN

MaskTrack R-CNN extends Mask R-CNN with a tracking branch and external memory to identify instances across video frames. It combines appearance, semantic, spatial, and detection cues during post-processing while processing videos online.

  • 5. MaskTrack R-CNN: MaskTrack R-CNN adds a fourth tracking branch and external memory to Mask R-CNN’s classification, box regression, and mask branches.The method is built on Mask R-CNN’s two-stage architecture.
  • 5.2. New Tracking Branch: The tracking branch performs multiclass classification over previously identified instances plus a new unseen instance label.For N existing identities, each candidate box is assigned among N + 1 labels.
  • 5.2. New Tracking Branch: Previously identified instance features are stored in memory, updated with new features, or inserted when a candidate receives the new-instance label.
  • Training: The full network is trained end-to-end using combined classification, box, mask, and tracking losses.
  • 5.3. Combining Other Cues: At test time, candidate assignments combine appearance similarity with detection confidence, spatial overlap, and semantic consistency.This combined score is used only during testing, not network training.
  • Inference: The method processes each testing video sequentially online, matching hypotheses across frames while avoiding within-frame matching conflicts.

6. Experiments

Experiments compare MaskTrack R-CNN with baselines on YouTube-VIS, showing strong overall performance and examining tracking cues, failure cases, and oracle settings. The analyses indicate that end-to-end joint training, image-level prediction, and cross-frame association each shape performance, while similar-object identity remains challenging.

  • Main Results: MaskTrack R-CNN achieves the best results across all evaluation metrics on both the validation and test sets.Its differentiable instance-matching component enables the tracking loss to backpropagate during end-to-end joint training.
  • Main Results: Mask propagation baselines cannot handle objects appearing in intermediate frames, while FEELVOS reaches 26.9 AP on the validation set.Track-by-detection methods differ in their use of spatial and visual matching cues; SeqTracker performs better than other baselines but requires offline precomputation.
  • Main Results: The memory queue preserves identities through occlusion, and the method detects objects entering mid-video by adding them to external memory.Examples include recovering a surfboard after disappearance and detecting a deer entering in the second frame.
  • Main Results: Failures occur when an object changes appearance substantially or when multiple similar objects move and occlude one another.The method splits one deer into two identities in different poses and merges or confuses fish identities in an aquarium.
  • Ablation Study: Bounding box IoU and category consistency are the most important matching cues, with either removed causing AP to drop by around 5%.Detection confidence improves performance only slightly, so the method uses the cues as soft constraints because their estimates are imperfect.
  • Oracle Results: Image-level prediction is more critical than cross-frame association for improving video instance segmentation performance in the oracle analysis.Image Oracle performs much better than Identity Oracle, while Identity Oracle is only marginally better than MaskTrack R-CNN.

7. Conclusions

The paper introduces video instance segmentation as a combined detection, segmentation, and tracking task, alongside YouTubeVIS and an initial method. It identifies future work in spatiotemporal proposals, trainable matching, and motion-aware recognition and association.

  • Video instance segmentation combines object detection, segmentation, and tracking, creating challenges in rich and complex scenes.
  • The paper presents YouTubeVIS as an accompanying dataset and MaskTrack R-CNN as a method combining single-frame instance segmentation with object tracking.
  • Future directions include spatiotemporal object proposals and detection, end-to-end trainable matching, and motion information for recognition and identity association.
Loading 1905.04804v4…