Source-linked AI summary
Occluded Video Instance Segmentation: A Benchmark
Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip H. S. Torr, Song Bai
TL;DR
The paper asks whether video understanding systems can detect, segment, and track objects under heavy occlusion. It introduces the OVIS benchmark and a temporal feature calibration module, finding that current systems remain weak while the plug-in substantially improves two baselines. OVIS therefore exposes a challenging gap in occlusion-aware video understanding.
Problem
The paper investigates whether video understanding systems can simultaneously detect, segment, and track instances in heavily occluded scenes.
Method
The paper constructs OVIS and introduces temporal feature calibration, which uses a reference frame to complement missing object cues in a query frame.
Results
16.3 AP is the highest result among evaluated state-of-the-art algorithms on OVIS, while temporal feature calibration improves MaskTrack R-CNN and SipMask baselines.
Takeaways & Limitations
OVIS shows that occluded video instance segmentation remains far from practical deployment and motivates further work on occlusion handling or object de-occlusion.
Takeaways & Limitations
The paper identifies high annotation cost as constraining video instance segmentation dataset scale and points to synthetic or augmented occluded data as future directions.
Abstract
from arXiv · showhide
Can our video understanding systems perceive objects when a heavy occlusion exists in a scene? To answer this question, we collect a large-scale dataset called OVIS for occluded video instance segmentation, that is, to simultaneously detect, segment, and track instances in occluded scenes. OVIS consists of 296k high-quality instance masks from 25 semantic categories, where object occlusions usually occur. While our human vision systems can understand those occluded instances by contextual reasoning and association, our experiments suggest that current video understanding systems cannot. On the OVIS dataset, the highest AP achieved by state-of-the-art algorithms is only 16.3, which reveals that we are still at a nascent stage for understanding objects, instances, and videos in a real-world scenario. We also present a simple plug-and-play module that performs temporal feature calibration to complement missing object cues caused by occlusion. Built upon MaskTrack R-CNN and SipMask, we obtain a remarkable AP improvement on the OVIS dataset. The OVIS dataset and project code are available at http://songbai.site/ovis .
1 Introduction
The paper introduces OVIS, a benchmark for detecting, segmenting, and tracking heavily occluded video instances, and evaluates whether current systems can handle this setting. It also proposes temporal feature calibration to recover missing object cues and improve baseline performance.
- OVIS targets video instance segmentation in occluded scenes by requiring models to simultaneously detect, segment, and track object instances.
- 296k high-quality instance masks across 25 semantic categories make OVIS a large-scale testbed dominated by severe object occlusions.The dataset labels occlusion levels and provides an AP metric for evaluating performance across occlusion degrees.
- 16.3 AP is the highest result among nine evaluated state-of-the-art algorithms on OVIS, while heavily occluded performance reaches only 6.3 AP.The results indicate a substantial gap between current systems and human occlusion perception.
- Temporal feature calibration uses a reference frame to complement missing object cues in a query frame through calibrated feature embeddings.The module learns an offset guided by the query frame and adjusts reference features using deformable convolution.
- 15.4 AP with MaskTrack R-CNN and 14.3 AP with SipMask improve their corresponding baselines by 4.6 and 4.1 AP, respectively.The module is presented as a flexible plug-in applicable to both OVIS and YouTube-VIS.
2 Related Work
The related work situates OVIS within video instance segmentation and neighboring video recognition, segmentation, tracking, and occlusion-handling tasks. These tasks differ in whether they classify instances, track them, require masks, or model occlusion explicitly.
- Video Instance Segmentation: YouTube-VIS established video instance segmentation, whereas OVIS specifically targets severe occlusions and is reported to be much more challenging.YouTube-VIS is not designed to study occluded video understanding.
- Video Instance Segmentation: Video instance segmentation combines object classification, segmentation, and cross-frame instance association, distinguishing it from several neighboring tasks.The surveyed methods include tracking-by-detection, bottom-up clustering, and transformer-based sequence modeling.
- Other Related Tasks: Video object segmentation tracks and segments one or more salient objects, but does not distinguish semantic categories as video instance segmentation does.Semi-supervised variants receive a first-frame mask, while unsupervised variants do not.
- Other Related Tasks: Video semantic segmentation labels each frame semantically without requiring instance distinction or cross-frame object tracking.Cityscapes and CamVid are cited as representative datasets using sparse frame annotations.
- Other Related Tasks: Video panoptic segmentation combines consistent panoptic segmentation with instance association across frames, while open-world video object segmentation remains taxonomy-free.The related datasets include Cityscapes-VPS and UVO.
- Occlusion Understanding: Occlusion-focused approaches infer occluders, model occlusion relations, recover occlusion ordering, or complete invisible object regions.Compositional models are cited as another direction for handling partial occlusions.
3 OVIS Dataset
OVIS is a large-scale benchmark for video instance segmentation in severe occlusions, with exhaustive annotations, occlusion labels, and statistics designed to measure this challenge. Its dataset structure and comparisons with YouTube-VIS show longer instances, denser scenes, and substantially heavier occlusion.
- Dataset scope: OVIS targets simultaneous detection, segmentation, and tracking of object instances in occluded videos.
- Dataset scope: 25 semantic categories were selected, emphasizing commonly seen animals and overlap with established image instance segmentation datasets.
- Collection and annotation: 8,644 candidate videos were reduced to 901 challenging videos after re-checking, yielding a 10% pass rate under stringent collection standards.
- Benchmark utility: OVIS is also suitable for semi-supervised or unsupervised video object segmentation and object tracking because bounding-box annotations are provided.
- Dataset statistics: 296k masks and 5,223 instances make OVIS larger in masks than YouTube-VIS 2019 and 2021, while containing fewer videos because it favors long videos and instances.
- Dataset statistics: OVIS averages 12.77s per video and 10.05s per instance, requiring long-term tracking because its instances last longer than those in YouTube-VIS.
- Occlusion statistics: 26.3% of objects are severely occluded, 80.2% of instances are severely occluded in at least one frame, and only 2% are never occluded.
- Occlusion statistics: OVIS has mBOR 0.22 versus 0.07 for YouTube-VIS 2019 and 0.06 for YouTube-VIS 2021; about half its frames have BOR no less than 0.2.
4 Experiments
Experiments show that severe occlusion substantially degrades video instance segmentation, especially segmentation and heavily occluded instances. Temporal feature calibration improves performance across occlusion levels and is most beneficial in heavily occluded scenes.
- Main Results: At least 50% performance degradation occurs on OVIS relative to YouTube-VIS for most methods, exceeding 80% for heavily occluded instances.SipMask drops from 32.5 AP on YouTube-VIS to 2.2 AP for the heavily occluded OVIS validation group.
- Main Results: 4.6 AP and 4.1 AP improvements are achieved by calibrated MaskTrack R-CNN and SipMask over their corresponding baselines on OVIS.The calibrated models reach 15.4 AP and 14.3 AP, compared with 10.8 AP and 10.2 AP for the baselines.
- Discussions: 121% improvement over the MaskTrack R-CNN baseline is obtained by the identity oracle on OVIS, compared with only 4% on YouTube-VIS.The image oracle reaches 58.4 AP and the identity oracle reaches 25.5 AP on OVIS, indicating greater difficulty in image-level prediction and cross-frame association.
- Discussions: 34.1% segmentation error occurs under severe occlusion versus 12.1% without occlusion for MaskTrack R-CNN, making segmentation the most increasing error type.Classification error is higher than segmentation and tracking error, while larger input resolution yields only 0.5 AP and 0.3 AP gains for MaskTrack R-CNN and SipMask.
- Discussions: 6.6% classification-error reduction is achieved by temporal feature calibration under severe occlusion, compared with 2.7% without occlusion.Its gain over the local-sampling-plus-deformable-convolution baseline increases with occlusion level, unlike the comparison strategy's classification gains over MaskTrack R-CNN.
5 Future Directions
The paper identifies several unresolved directions for OVIS, including occlusion-aware modeling, synthetic occluded data, learning from occlusion annotations, pre-training, and broader evaluation settings.
- Occlusion-aware models: Occlusion-aware models remain a direct avenue for improving OVIS performance, including compositional models and de-occlusion.The paper suggests testing whether completing invisible object parts helps in this scenario.
- Occluded data generation: Synthetic occluded data could expand video instance segmentation training beyond relatively small annotated video datasets.The paper discusses partial-occlusion augmentation and simulator-generated structured amodal data.
- Learning from occlusion annotations: OVIS provides coarse per-object occlusion labels that future training paradigms could explicitly abstract as prior knowledge.The three levels are no occlusion, slight occlusion, and severe occlusion.
- Large scale model pre-training: Joint training with image datasets improves performance, while self-supervised pre-training could further strengthen frame embeddings when transferred to OVIS.The paper proposes exploiting large quantities of unlabeled data for pre-training.
- Dataset versatility: Future dataset development could formalize unsupervised, semi-supervised, and interactive tracks and extend OVIS to video panoptic segmentation.The authors also identify broader scene understanding as a motivation for extending OVIS.
6 Conclusions
The paper introduces OVIS as a benchmark for heavily occluded video instance segmentation and finds that existing systems perform substantially below their performance on YouTube-VIS.
- 6 Conclusions: OVIS contains 296k high-quality instance masks from 5,223 heavily occluded instances and evaluates video understanding under object occlusion.It is presented as the second benchmark dataset after YouTube-VIS.
- 6 Conclusions: OVIS is more challenging than YouTube-VIS because most OVIS objects undergo severe occlusion, whereas YouTube-VIS was not designed for occluded video understanding.The comparison concerns dataset design and observed experimental difficulty.