Source-linked AI summary
SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation
Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, Ling Shao
TL;DR
Single-stage instance segmentation is fast but typically less accurate than two-stage methods, motivating SipMask’s focus on preserving within-box spatial information. SipMask uses sub-region-specific spatial coefficients and alignment strategies, reporting gains over TensorMask and YOLACT plus real-time video results.
Problem
Single-stage instance segmentation offers speed and simplicity but generally lags two-stage methods in accuracy, with YOLACT losing spatial information by using one coefficient set per bounding-box.
Method
SipMask is a fast anchor-free single-stage method whose spatial preservation module generates separate coefficients for bounding-box sub-regions, complemented by feature alignment and mask alignment weighting.
Results
SipMask gains 1.0% mask AP and four-fold speed over TensorMask on COCO test-dev, gains 3.0% mask AP over YOLACT at comparable speed, and reaches 30 fps for video segmentation.
Takeaways & Limitations
Preserving spatial information enables SipMask to achieve state-of-the-art single-stage instance segmentation performance and extend to real-time video instance segmentation.
Abstract
from arXiv · showhide
Single-stage instance segmentation approaches have recently gained popularity due to their speed and simplicity, but are still lagging behind in accuracy, compared to two-stage methods. We propose a fast single-stage instance segmentation method, called SipMask, that preserves instance-specific spatial information by separating mask prediction of an instance to different sub-regions of a detected bounding-box. Our main contribution is a novel light-weight spatial preservation (SP) module that generates a separate set of spatial coefficients for each sub-region within a bounding-box, leading to improved mask predictions. It also enables accurate delineation of spatially adjacent instances. Further, we introduce a mask alignment weighting loss and a feature alignment scheme to better correlate mask prediction with object detection. On COCO test-dev, our SipMask outperforms the existing single-stage methods. Compared to the state-of-the-art single-stage TensorMask, SipMask obtains an absolute gain of 1.0% (mask AP), while providing a four-fold speedup. In terms of real-time capabilities, SipMask outperforms YOLACT with an absolute gain of 3.0% (mask AP) under similar settings, while operating at comparable speed on a Titan Xp. We also evaluate our SipMask for real-time video instance segmentation, achieving promising results on YouTube-VIS dataset. The source code is available at https://github.com/JialeCao001/SipMask.
1 Introduction
SipMask addresses the accuracy limitations of fast single-stage instance segmentation by preserving spatial information within detected bounding-boxes. Its spatial preservation design and alignment strategies improve delineation of adjacent instances while maintaining real-time performance.
- Single-stage methods are faster than two-stage methods but generally lag behind them in segmentation accuracy.
- YOLACT’s single coefficient set for an entire bounding-box loses within-box spatial information and hampers delineation of adjacent instances.
- SipMask’s lightweight spatial preservation module generates separate spatial coefficients for bounding-box sub-regions, splitting mask prediction into multiple sub-masks.
- Mask alignment weighting and feature alignment strategies better correlate mask prediction with object detection.
- 1.0% mask AP gain and four-fold speedup over TensorMask are reported on COCO test-dev.
- 3.0% mask AP gain over YOLACT is reported on COCO test-dev at comparable speed, while video segmentation reaches 30 fps on YouTube-VIS.
2 Related Work
Single-stage methods reduce computational and memory costs by avoiding proposal generation and feature pooling, but YOLACT’s single coefficient set discards spatial detail. SipMask preserves that detail through sub-region-specific coefficients and feature alignment without pooling.
- Mask R-CNN and FCIS use feature pooling, whereas YOLACT and SipMask avoid pooling and combine basis masks with coefficients.Mask R-CNN is computationally expensive after RoIAlign, while FCIS is memory demanding because of large position-sensitive maps.
- Single-stage methods generally target faster inference by avoiding proposal generation and feature pooling strategies.
- YOLACT predicts category-independent basis maps and one instance-specific coefficient set for each bounding-box.
- SipMask’s spatial preservation module generates multiple coefficient sets for bounding-box sub-regions, preserving spatial information within instances.
- SipMask also includes feature alignment to improve representation for box classification and spatial coefficient generation.
3 Method
SipMask is a fast anchor-free single-stage instance segmentation method that preserves spatial information by predicting separate mask components for bounding-box sub-regions. Its architecture combines aligned features, multi-layer basis masks, spatial mask prediction, and losses linking mask quality to detection quality.
- Overall Architecture: SipMask uses an FCOS-based anchor-free architecture with mask-specialized classification and regression branches.The classification branch predicts box scores and spatial coefficients, while the regression branch predicts box offsets and category-independent basis masks.
- Spatial Preservation Module: SipMask divides each bounding-box into k × k sub-regions and predicts a separate set of spatial coefficients for each region.These coefficients are combined with image-specific basis masks to generate separate regional map predictions.
- Spatial Preservation Module: The spatial preservation module aligns features using regressed box locations before generating coefficients for classification and mask prediction.Deformable convolution uses regression offsets to augment the regular sampling grid and produce aligned features.
- Mask-Specialized Regression Branch: SipMask generates basis masks from multiple FPN prediction layers to capture contextual information, support continuous masks, and remove background clutter.This differs from YOLACT, which uses a single FPN prediction layer for basis-mask generation.
- Spatial Mask Prediction Module: The spatial mask prediction module prunes each regional map, adds the pruned maps, and thresholds the result to produce an instance mask.For k = 2, the regions are quadrants; responses outside each corresponding quadrant are set to zero before integration.
- Spatial Mask Prediction Module: Regional mask prediction reduces the influence of adjacent overlapping instances in the final mask.This spatial separation is intended to improve delineation of spatially adjacent objects.
- Loss Function: The mask alignment weighting loss assigns higher weights to mask errors from accurately detected boxes and combines with regression and classification losses.The overall objective is L = Lreg + Lcls + Lmask, with Lmask weighting each instance loss by its detection quality.
- Video Instance Segmentation: SipMask extends to real-time video instance segmentation by adding a fully convolutional tracking branch parallel to the mask-specialized branches.The branch uses two convolutional layers whose outputs are fused into tracking feature maps.
4 Experiments
Experiments show that SipMask improves single-stage instance segmentation accuracy while retaining high speed, and extends effectively to real-time video instance segmentation.
- State-of-the-art Comparison: 1.0% mask AP gain and four-fold speedup over TensorMask are achieved under similar settings on COCO test-dev.SipMask also gains 2.7% on large objects relative to TensorMask.
- State-of-the-art Comparison: 3.0% mask AP gain over YOLACT is obtained with comparable speed, using 30 ms for YOLACT versus 32 ms for SipMask.SipMask++ also improves mask accuracy over YOLACT++ at the same speed under similar input and backbone settings.
- Ablation Study: Qualitative COCO results report high-quality instance masks in challenging scenarios, corresponding to 38.1 mask AP.Colors distinguish different object instances.
- Ablation Study: Adding SP, CBM, and WL progressively improves mask AP over the baseline, with the complete model gaining 3.1% mask AP.The baseline uses only the high-resolution P3 FPN layer and a single coefficient set for mask prediction.
- Ablation Study: Spatial coefficients provide the largest individual accuracy improvement among the evaluated components.The spatial coefficients and feature alignment together constitute the SP module.
- Ablation Study: Using separate spatial coefficients improves delineation of spatially adjacent instances compared with a single coefficient set.The qualitative comparison contrasts baseline masks with masks generated from separate coefficients for bounding-box sub-regions.
- Ablation Study: Increasing sub-regions from 1×1 to 2×2 yields a large performance gain, while further increases produce only marginal improvements.The study identifies 2×2 as an optimal tradeoff between speed and accuracy.
- Video Instance Segmentation Results: SipMask achieves 30 fps on YouTube-VIS while providing favorable real-time video instance segmentation performance.The video experiments use the same 640×360 input size for comparison with MaskTrack R-CNN.
5 Conclusion
SipMask is a fast single-stage instance segmentation method that preserves spatial information through sub-region mask prediction and improves alignment with detection. Experiments support state-of-the-art single-stage performance and real-time operation, including video extension.
- Conclusion: SipMask preserves spatial information by generating separate spatial coefficients for different bounding-box sub-regions.Its design also includes feature alignment and mask alignment weighting loss to correlate mask prediction with object detection.
- Conclusion: SipMask achieves state-of-the-art single-stage instance segmentation performance on COCO and extends to real-time video instance segmentation.At 544×544 input resolution, it operates in real time on a single Titan Xp with 32.8 mask accuracy on COCO test-dev.