Source-linked AI summary
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation
Gedas Bertasius, Lorenzo Torresani
TL;DR
Video instance segmentation requires classifying, segmenting, and tracking object instances across sequences despite challenges from high-resolution multi-frame processing and intertwined tasks. MaskProp adapts Mask R-CNN with mask propagation to generate and aggregate clip-level tracks, achieving state-of-the-art YouTube-VIS accuracy while using much less labeled data. Its main scope boundary is the need to balance clip length against GPU memory for high-resolution processing.
Problem
Video instance segmentation is challenging because accurate segmentation requires high resolution while tracking requires simultaneous multi-frame analysis, and separate task pipelines can be suboptimal.
Method
MaskProp adapts Mask R-CNN with a mask-propagation branch that propagates instance information within clips and aggregates dense clip-level tracks into video-level sequences.
Results
MaskProp achieves the best reported accuracy on YouTube-VIS, outperforming EnsembleVIS by 1.8 mAP and MaskTrack R-CNN by 16.3 mAP.
Takeaways & Limitations
MaskProp handles occlusions, disocclusions, and motion blur while remaining simpler and using substantially less labeled pretraining data than the ICCV 2019 challenge winner.
Takeaways & Limitations
Clip length must be long enough for occlusion and motion-blur handling but short enough to fit high-resolution inputs in GPU memory.
Abstract
from arXiv · showhide
We introduce a method for simultaneously classifying, segmenting and tracking object instances in a video sequence. Our method, named MaskProp, adapts the popular Mask R-CNN to video by adding a mask propagation branch that propagates frame-level object instance masks from each video frame to all the other frames in a video clip. This allows our system to predict clip-level instance tracks with respect to the object instances segmented in the middle frame of the clip. Clip-level instance tracks generated densely for each frame in the sequence are finally aggregated to produce video-level object instance segmentation and classification. Our experiments demonstrate that our clip-level instance segmentation makes our approach robust to motion blur and object occlusions in video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset, outperforming the ICCV 2019 video instance segmentation challenge winner despite being much simpler and using orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes).
1. Introduction
Video instance segmentation requires jointly segmenting, classifying, and tracking objects across frames, but high-resolution multi-frame processing and separate task pipelines create efficiency and accuracy challenges. MaskProp addresses these issues with a unified, simpler model that achieves higher accuracy than prior approaches with far less pretraining data.
- Video instance segmentation must segment, classify, and link every object instance across an entire video sequence.
- High-resolution video processing is difficult because strong backbones rarely fit more than one image per GPU, while tracking requires multiple frames simultaneously.
- Reducing resolution harms segmentation or detection, whereas separating segmentation and tracking often produces suboptimal results because the tasks are closely intertwined.
- The ICCV 2019 challenge winner used independently designed components for detection, classification, segmentation, and tracking, while MaskTrack R-CNN achieved 30.3 versus 44.8 video mAP.
- MaskProp adapts Mask R-CNN with a mask-propagation branch, aggregates dense clip-level tracks, and achieves higher accuracy with 1.3M versus 1B images and 860K versus 14M bounding boxes.
2. Related Work
Video instance segmentation extends image and video segmentation by requiring class-aware instance masks and identity links throughout a sequence. Prior methods either use a unified but weaker tracker or a complex ensemble, whereas MaskProp uses unified mask propagation for stronger results.
- Unlike image instance segmentation, video instance segmentation must determine instance correspondences across multiple frames.
- Video object segmentation is class-agnostic and often uses a first-frame ground-truth mask, whereas video instance segmentation finds, classifies, segments, and links all predefined object instances.
- MaskTrack R-CNN is a unified model that adds a tracking branch to Mask R-CNN, while EnsembleVIS separately models classification, detection, segmentation, and tracking.
- MaskProp remains unified like MaskTrack R-CNN but uses a more effective mask-propagation branch, while being simpler and using less labeled data than EnsembleVIS.
- MaskProp produces higher accuracy on YouTube-VIS than the compared prior methods.
3. Video Instance Segmentation
The task evaluates video-level instance mask tracks for predefined categories, requiring both accurate per-frame segmentation and reliable temporal linking. Its AP and AR metrics average category-wise performance across ten IoU thresholds.
- For each detected instance, the system outputs a video-level mask track, category label, and confidence score over L frames of size H×W.
- Video IoU compares predicted and ground-truth instance masks across frames, so high scores require accurate classification, segmentation, and tracking.
- AP and AR are computed separately for each category and averaged over ten IoU thresholds from 50% to 95% in 5% increments.
4. Mask Propagation
MaskProp adapts Mask R-CNN for video by propagating instance-specific features and masks across temporal clips, then aggregates overlapping clip tracks into video-level instances.
- Video-level instance segmentation: MaskProp builds clip-level instance tracks for each overlapping temporal clip and aggregates them into video-level segmentation tracks.Clips near video boundaries may contain fewer frames.
- Training objective: The propagation branch is trained within a multi-task Mask R-CNN objective that includes a soft IoU mask-propagation loss.The authors report that soft IoU performed slightly better than standard cross entropy for this task.
- Instance feature propagation: The model detects frame-level instances, computes instance-specific features by masking backbone features, and propagates them across frames using deformable convolution.Motion offsets come from feature differences between the source and target frames.
- Propagated instance segmentation: Propagated and target-frame features are combined to predict instance masks, while an instance-agnostic attention map removes pixels belonging to no object.Predicted masks are normalized across detected instances before attention-based masking.
- Track aggregation: Overlapping clip tracks are matched by comparing predicted masks in their shared frames with a soft IoU score, then assigning matching tracks a common video-level ID.Tracks overlap when their center-frame distance is less than the clip length.
5. Experimental Results
MaskProp is evaluated on YouTube-VIS against established baselines using quantitative, ablation, and qualitative analyses. It achieves stronger accuracy with effective mask propagation, benefits from stronger frame-level masks, and produces robust tracks under challenging visual conditions.
- Quantitative Results: MaskProp is evaluated on YouTube-VIS, which provides per-pixel segmentation, category, and instance labels across 40 object categories.The evaluation uses 2,238 training, 302 validation, and 343 test videos, with reported experiments conducted on the validation set.
- Quantitative Results: MaskProp outperforms all compared baselines on mean video average precision, video AP at IoU 75%, and average recall at one and ten scored instances.The comparison includes MaskTrack R-CNN, a matched reimplementation, and EnsembleVIS, the ICCV 2019 challenge winner.
- Ablation Experiments: 15.2 mAP and 9.7 mAP are the respective drops from MaskProp’s 46.6 mAP when replacing its propagation branch with FlowNet2 and MaskTrack R-CNN tracking.All compared variants use the same backbone and detection networks.
- Ablation Experiments: 13 frames is the best instance-track clip length, propagating masks to six previous and six subsequent frames; removing high-resolution refinement reduces video mAP by 1.9%.The clip-length result addresses occlusions, while the refinement result measures a separate architectural component.
- Qualitative Results: Stronger frame-level Mask R-CNN models improve video instance segmentation, while qualitative results show more coherent tracks during motion, occlusion, overlap, blur, and appearance variation.The feature visualizations indicate reliable instance-specific feature propagation despite motion blur, deformations, and large appearance changes.
6. Conclusion
The paper concludes that MaskProp is a conceptually simple architecture for video instance segmentation that achieves state-of-the-art results on YouTube-VIS. The authors identify extensions to weaker annotation settings and related tracking and pose-estimation problems.
- Conclusion: MaskProp is presented as a conceptually simple architecture that does not require large amounts of labeled pre-training data and achieves state-of-the-art YouTube-VIS results.The conclusion summarizes the method’s simplicity, data efficiency, and benchmark performance.
- Conclusion: Future work will extend MaskProp to settings with only bounding-box annotations and to tasks such as pose estimation and tracking.These are proposed application and annotation-scope extensions rather than evaluated results in this paper.
- Conclusion: MaskProp produces more robust and temporally coherent instance tracks than MaskTrack R-CNN, especially with large motion, occlusion, or overlapping objects.The qualitative comparison uses distinct colors for object instances across original frames and predictions.