Source-linked AI summary
SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation
Dongfang Liu, Yiming Cui, Wenbo Tan, Yingjie Chen
TL;DR
VIS methods commonly extend two-stage Mask R-CNN, leaving challenges in feature sharing, mask resolution, proposal redundancy, and tracking robustness. SG-Net proposes a one-stage, proposal-free network with interconnected task heads, sub-region mask prediction, and centerness-based tracking. On YouTube-VIS, it reports improved accuracy and inference speed, while qualitative errors expose sensitivity to underrepresented classes.
Problem
Dominant two-stage VIS methods face limited cross-task feature sharing, RoI-constrained mask resolution, redundant proposal features, and costly repeated prediction operations.
Method
SG-Net is a one-stage FCOS-based framework with interdependent detection, segmentation, and tracking heads, dynamically predicting masks over instance sub-regions and tracking centerness movement.
Results
SG-Net outperforms MaskTrack R-CNN by 8.2% and 7.9% in AP with ResNet-50 and ResNet-101, respectively, and reports competitive results across YouTube-VIS metrics.
Takeaways & Limitations
The study presents SG-Net as a compact one-stage VIS baseline that improves mask quality and inference speed compared with two-stage solutions.
Takeaways & Limitations
Qualitative failures are attributed to YouTube-VIS data bias, including underrepresentation of the earless seal class in training.
Abstract
from arXiv · showhide
Video instance segmentation (VIS) is a new and critical task in computer vision. To date, top-performing VIS methods extend the two-stage Mask R-CNN by adding a tracking branch, leaving plenty of room for improvement. In contrast, we approach the VIS task from a new perspective and propose a one-stage spatial granularity network (SG-Net). Compared to the conventional two-stage methods, SG-Net demonstrates four advantages: 1) Our method has a one-stage compact architecture and each task head (detection, segmentation, and tracking) is crafted interdependently so they can effectively share features and enjoy the joint optimization; 2) Our mask prediction is dynamically performed on the sub-regions of each detected instance, leading to high-quality masks of fine granularity; 3) Each of our task predictions avoids using expensive proposal-based RoI features, resulting in much reduced runtime complexity per instance; 4) Our tracking head models objects centerness movements for tracking, which effectively enhances the tracking robustness to different object appearances. In evaluation, we present state-of-the-art comparisons on the YouTube-VIS dataset. Extensive experiments demonstrate that our compact one-stage method can achieve improved performance in both accuracy and inference speed. We hope our SG-Net could serve as a strong and flexible baseline for the VIS task. Our code will be available.
1. Introduction
VIS jointly requires detection, segmentation, and tracking, but dominant two-stage approaches create feature-sharing, mask-resolution, redundancy, and efficiency challenges. SG-Net addresses these issues with a compact one-stage, proposal-free architecture whose interconnected heads use spatially granular masks and centerness-based tracking.
- VIS requires simultaneous detection, segmentation, and tracking of instances in video frames.
- Two-stage VIS methods struggle to share features across task heads and restrict mask resolution through resized RoI features.These approaches also repeatedly encode proposal-based features, including redundant candidate proposals.
- SG-Net is a compact one-stage method that treats detection, segmentation, and tracking as interconnected tasks with jointly optimized shared features.Its fully convolutional architecture is tied to FCOS.
- SG-Net dynamically divides each instance into sub-regions and segments them to enrich object details and produce more accurate mask edges.
- The proposal-free design avoids inference-time growth with the number of predictions, while centerness-based tracking is designed to handle appearance, shape, and size changes.
2. Related Work
Prior work largely relies on two-stage, RoI-based pipelines, while one-stage detection and segmentation methods offer alternatives for speed, resolution preservation, and flexible task integration. SG-Net builds on these directions with proposal-free spatially granular masks and center-based tracking.
- VIS methods predominantly follow Mask R-CNN-style two-stage pipelines that use proposals and RoI features for multiple tasks.
- FCOS and CenterNet show that one-stage detectors can simplify detection without predefined anchors, while generally targeting faster inference.
- Its tracking follows tracking-by-detection but models object movement from FCOS centerness rather than bounding boxes or masks.
- SG-Net uses interdependent, jointly optimized task heads without RoI operations or redundant RoI features to support fast, high-quality instance masks.
3. Our SG-Net
SG-Net is a fully convolutional VIS architecture built on FCOS, with interdependent detection, mask, and tracking heads. Its mask head uses dynamically sized sub-regions and target-oriented base masks, while its tracking head regresses centerness movements for cross-frame association.
- Overall Architecture: SG-Net uses an FCOS-based backbone with FPN features and convolutional detection, mask, and tracking heads.The detection head retains FCOS classification, box regression, and centerness branches.
- Mask Head: The mask head combines spatial attention, base mask generation, and granularity blending to produce instance masks.These modules operate collaboratively, with attention scores weighting corresponding base masks.
- Mask Head: Bounding boxes are dynamically divided into r1 × r2 sub-regions according to object shape and size, with 1D attention scores predicted for each region.The region counts are bounded by six along each dimension, and the division uses a 50-pixel unit.
- Mask Head: Target-oriented base masks are generated from P3–P5 FPN features after higher-level features are upsampled and concatenated.Features outside each detected bounding box are zeroed, and a 1 × 1 convolution produces a number of base masks matching the sub-region count.
- Mask Head: The mask prediction sums element-wise products of each instance’s attention scores and corresponding base masks before sigmoid binarization.Avoiding proposal generation and RoI operations reduces runtime complexity per instance.
- Tracking Head: The tracking head predicts 2D centerness movements and greedily matches detections across frames, spawning a new tracklet when no candidate lies within radius r.The radius is defined as the mean of the tracking object’s bounding-box width and height.
4. Experiments
On YouTube-VIS, SG-Net is evaluated against prior methods through main comparisons, qualitative examples, ablations, and architectural discussions. The experiments report improved accuracy and faster inference, while identifying gains from spatial granularity, feature sharing, centerness tracking, and mask-resolution choices.
- Main Results: SG-Net outperforms MaskTrack R-CNN by 8.2% and 7.9% in AP with ResNet-50 and ResNet-101, respectively.It also surpasses STEm-Seg and SipMask under both backbone settings.
- Main Results: SG-Net runs about ×2 and ×3 faster than MaskTrack R-CNN with ResNet-50 and ResNet-101, respectively.The best ResNet-50 model typically runs at 0.043s/image on a 1080Ti GPU.
- Main Results: Qualitative examples show improved segmentation and tracking on difficult cases involving missing regions, nearby same-class instances, and occlusions.Higher-resolution base masks, spatial attention, and centerness tracking are identified as contributing factors.
- Ablation Study: The full model gains 5.2% absolute accuracy over the baseline while maintaining fast inference after adding BMM, SAM, and GIoU loss.The baseline achieves 31.1% AP at 33.3 ms, while the full model combines the contributing components.
- Ablation Study: Increasing box divisions improves accuracy through 6 × ∗, but 8 × ∗ brings marginal gains and a significant speed penalty.The default configuration uses 6 × ∗ divisions; dynamic divisions are reported as faster than a fixed 2×2 grid at similar accuracy.
- Ablation Study: Upsampling base-mask features from stride-8 to half input resolution raises AP from 34.9% to 36.3% and AP@0.75 from 38.1% to 39.6%.The higher resolution improves object-detail preservation but introduces a speed penalty; deeper FPN features also improve accuracy with an affordable speed cost.
- Discussions: Instance-aware connections enable end-to-end joint optimization across detection, segmentation, and tracking heads.The design avoids proposal-based RoI operations, while sub-region mask prediction provides fine granularity and centerness tracking improves robustness to overlap and appearance variation.
5. Conclusion
SG-Net is presented as a simple one-stage VIS framework that improves mask quality and inference speed compared with two-stage solutions.
- SG-Net is a simple one-stage framework for video instance segmentation.
- Compared with two-stage solutions, SG-Net improves mask quality and inference speed.