Source-linked AI summary
A Survey on Instance Segmentation: State of the art
Abdul Mueed Hafiz, Ghulam Mohiuddin Bhat
TL;DR
Instance segmentation must combine object detection with pixel-level semantic segmentation while handling accuracy, efficiency, and difficult image conditions. This survey synthesizes the field’s evolution, techniques, datasets, issues, and future scope, reporting representative state-of-the-art results including 48.6 Mask AP and 29.8 Mask AP at 33 frames-per-second.
Problem
Instance segmentation must distinguish separate instances of the same class while addressing accuracy, computational efficiency, small objects, transformations, occlusions, and degraded imagery.
Method
The paper surveys instance-segmentation evolution, techniques, datasets, related work, taxonomies, strengths, weaknesses, and future research scope.
Results
The survey reports 48.6 Mask AP for Hybrid Task Cascade on COCO’s test-challenge subset and 29.8 Mask AP at 33 frames-per-second for YOLACT on COCO.
Takeaways & Limitations
Instance segmentation is presented as an advancing research area with broad relevance to robotics, autonomous driving, surveillance, and related machine-vision applications.
Takeaways & Limitations
The survey identifies unresolved scope boundaries including multi-scale resource costs, geometric transformations, occlusions, and limited work on degraded images.
Abstract
from arXiv · showhide
Object detection or localization is an incremental step in progression from coarse to fine digital image inference. It not only provides the classes of the image objects, but also provides the location of the image objects which have been classified. The location is given in the form of bounding boxes or centroids. Semantic segmentation gives fine inference by predicting labels for every pixel in the input image. Each pixel is labelled according to the object class within which it is enclosed. Furthering this evolution, instance segmentation gives different labels for separate instances of objects belonging to the same class. Hence, instance segmentation may be defined as the technique of simultaneously solving the problem of object detection as well as that of semantic segmentation. In this survey paper on instance segmentation -- its background, issues, techniques, evolution, popular datasets, related work up to the state of the art and future scope have been discussed. The paper provides valuable information for those who want to do research in the field of instance segmentation.
1. Introduction
Instance segmentation extends coarse-to-fine image understanding by assigning distinct pixel masks to separate instances while identifying their classes. This survey reviews its evolution, methods, datasets, challenges, and future research scope.
- Instance segmentation simultaneously addresses object detection and semantic segmentation by assigning different labels to separate same-class object instances.
- 1.2 Issues: The field seeks both accurate localization and recognition across varied objects and efficient computation with acceptable memory and real-time cost.
- 1.2 Issues: Multi-scale detection improves accuracy but increases inference time and memory requirements when detectors process image pyramids.
- 1.2 Issues: Small-object segmentation remains difficult because higher layers lose detail while lower layers provide weaker semantic information; improving resolution increases computational complexity.
- 1.2 Issues: Geometric transformations, occlusions, and image degradations remain unresolved challenges, with degradation research limited by high-quality benchmark imagery.
- The survey covers more than 100 papers, organizes the literature into four top-level research clusters, and discusses techniques, datasets, strengths, weaknesses, and future directions.
2. Instance Segmentation Techniques: A Taxonomy
The survey classifies instance-segmentation techniques into mask-proposal classification, detection followed by segmentation, pixel labelling followed by clustering, and dense sliding-window methods. These families trade accuracy, computational demands, and pipeline complexity differently.
- Classification-of-mask-proposals methods generate mask proposals and then classify them.
- Detection-followed-by-segmentation methods first detect object boxes and then segment objects within those boxes.
- Detection and segmentation techniques may depend on pipelining, which contributes to reported drawbacks.
- Pixel-labelling-followed-by-clustering methods label every pixel categorically before grouping pixels into object instances.
- Pixel-labelling-followed-by-clustering benefits from high-resolution masks but generally has lower benchmark accuracy and higher computational requirements.The intense computation arises from pixel labelling.
- Dense sliding-window methods use CNNs for mask-proposal generation, while TensorMask predicts masks and multiple-class classifications in parallel.The survey notes decent benchmark performance but identifies algorithmic complexity as an issue.
3. The Evolution of Instance Segmentation
The survey traces instance segmentation from early CNN-based RCNN systems through increasingly efficient detection frameworks. This evolution addresses training and inference bottlenecks while retaining region-based representations.
- 3.1 RCNN: RCNN integrated AlexNet with selective-search region proposals to explore CNN-based instance segmentation.
- 3.1 RCNN: RCNN achieved high detection quality but required slow, difficult multistage training and separate feature extraction for every proposal during testing.
- 3.2 Fast RCNN: Fast RCNN streamlined training through end-to-end learning, shared convolutional computation, ROI pooling, and joint classification and bounding-box regression.
- 3.2 Fast RCNN: 3 times faster training and 10 times faster testing were reported for Fast RCNN compared with RCNN.
- MultiPath Network modified Fast R-CNN with skip connections, a foveal element, and an integral loss function to improve localization.
- Faster RCNN replaced externally computed selective-search proposals with CNN-generated region proposals to address the Fast RCNN speed bottleneck.
3.5 Mask R-CNN
Mask R-CNN extends Faster R-CNN with a parallel mask branch for simultaneous detection and high-quality mask generation. The survey also situates non-local operations as a way to capture long-range dependencies and improve instance segmentation.
- Mask R-CNN: Mask R-CNN adds a parallel object-mask prediction branch to Faster R-CNN’s bounding-box recognition branch.Its head uses a feature pyramid network backbone.
- Mask R-CNN: Mask R-CNN is relatively easy to train and adds a small computational load to Faster R-CNN, which runs at 5 fps.
- Mask R-CNN: Mask R-CNN generalized to related tasks such as human-pose estimation and ranked first in all three COCO 2016 challenges.
- MaskLab extends Faster R-CNN with semantic-segmentation and instance-center-direction outputs for segmentation inside prediction boxes.
- Non-local Neural Networks: Non-local blocks capture long-range pixel dependencies by allowing each output position to aggregate information from all possible positions.
- Non-local Neural Networks: The non-local operation preserves the input signal’s size while combining pairwise affinities with unary representations under normalization.
- Non-local Neural Networks: The technique was demonstrated to improve instance segmentation on the COCO dataset.
3.8 Path Aggregation Network (PANet)
PANet enhances feature propagation for instance segmentation by adding localization signals, adaptive feature pooling, and mask-branch information flow. Hybrid Task Cascade extends cascading through interleaved box and mask refinement, spatial context, and semantic segmentation.
- PANet: PANet augments an FPN with bottom-up path augmentation, adaptive feature pooling, a box branch, and fully connected fusion.These components are shown in the PANet framework.
- PANet: PANet shortens information paths between lower-layer localization features and top-level features to improve proposal-based instance segmentation.Its adaptive feature pooling propagates relevant information from every feature level to proposal-generating subnetworks.
- PANet: PANet achieved first place in the 2017 COCO Instance Segmentation Challenge and second place in object detection without batch-training for large image sets.The techniques were described as relatively easy to implement with relatively small computational overhead.
- Hybrid Task Cascade: Hybrid Task Cascade interleaves box and mask branches across multiple stages instead of refining the two tasks independently.The mask branch can use updated bounding-box predictions, while masks from previous stages are passed to later mask branches.
- Hybrid Task Cascade: HTC adds a branch for predicting per-pixel semantic segmentation over the entire image to provide spatial context.This branch is illustrated in the semantic segmentation architecture.
- Hybrid Task Cascade: HTC obtained 38.4% mask AP without fine tuning and achieved 48.6 Mask AP on COCO’s test-challenge subset.The latter result secured first place in the COCO 2018 Challenge in object detection.
3.10 GCNet
GCNet analyzes non-local global-context modeling and argues that contexts are nearly query-independent across image positions. It therefore uses a lightweight global-context block that preserves accuracy with lower computational expenditure and outperforms Non-Local Networks and SENet on MS COCO.
- GCNet: GCNet reports that global contexts modeled by Non-Local Networks are almost the same for different query positions.This finding motivates a query-independent formulation.
- GCNet: GCNet proposes a three-step general model for global context and instantiates it as a lightweight Global Context Block.The block is described as efficiently modeling global context and structurally resembling SENet.
- GCNet: The Global Context Block uses a bottleneck representation with default reduction ratio r=16.For a feature map with C channels, the bottleneck hidden representation has dimension C/r.
- GCNet: GCNet outperforms both Non-Local Networks and SENet on MS COCO.
3.11 YOLACT
YOLACT is a fully convolutional, real-time instance segmentation model that separates mask generation into prototype masks and per-instance coefficients. It combines these components linearly to produce instance masks and achieved 29.8 Mask AP at 33 frames per second on COCO using one Titan XP GPU.
- YOLACT: YOLACT generates prototype masks and predicts mask coefficients for each instance as parallel subtasks.Instance masks are produced by linearly combining the prototypes with the instance-specific coefficients.
- YOLACT: 29.8 Mask AP at 33 frames-per-second was achieved on COCO using one Titan XP GPU.The experiment used a single GPU for training, and the result was reported as faster than other state-of-the-art approaches at that time.
3.12 Mask Scoring R-CNN
Mask Scoring R-CNN augments Mask R-CNN with a MaskIoU-based score that better aligns predicted mask quality with confidence. The survey also contrasts this approach with TensorMask’s structured dense representation.
- Mask Scoring R-CNN: Mask Scoring R-CNN adds a MaskIoU head that learns scores aligned with predicted mask quality rather than classification confidence alone.The head uses predicted masks and ROI features to predict MaskIoU for C classes.
- Mask Scoring R-CNN: The added network block combines instance features with the predicted mask to regress mask IoU and prioritize better predictions in COCO AP evaluation.
- Mask Scoring R-CNN: Extensive COCO experiments consistently improved various models and outperformed the efficient Mask R-CNN approach.
- TensorMask: TensorMask performs dense instance segmentation by predicting structured four-dimensional tensors over spatial and relative mask positions.Its tensor shape is (V, U, H, W), separating object position from relative mask position.
4. Datasets
The survey describes large datasets supporting instance and semantic segmentation, emphasizing COCO’s scale and Cityscapes’ urban-scene annotations. It also identifies Mapillary Vistas as a globally varied, densely annotated street-scene dataset.
- COCO Dataset: COCO supports recognition, segmentation, and captioning, with more than 80 object classes and over 82,783 training images, 40,504 validation images, and 80,000 testing images.Its testing images are divided into test-dev, test-standard, test-challenge, and test-reserve subsets.
- COCO Dataset: COCO’s test-standard subset contains 20,000 images and serves as the default testing set for competitions and state-of-the-art benchmarking.
- Cityscapes Dataset: Cityscapes provides semantic, instance-specific, and pixel-specific annotations for 30 object classes grouped into 8 urban-scene categories.It contains about 5,000 finely annotated images and 20,000 coarsely annotated images.
- Mapillary Vistas Dataset: Mapillary Vistas contains 25,000 annotated images with 66 classes and polygonal delineations for individually demarcating objects.Its images span locations, weather, seasons, and daytimes worldwide.
5. Summary and Discussion
The survey organizes detection frameworks into two-stage region-based and single-stage unified approaches, then summarizes their tradeoffs and recurring design choices. It highlights accuracy, efficiency, scale variation, and computational cost as central concerns.
- Detection frameworks: Two-stage frameworks generally achieve better accuracy than single-stage frameworks on platforms with rich computational resources.Most winning techniques in major challenges are usually two-stage, including Mask R-CNN.
- Detection frameworks: Single-stage detectors are usually faster because they omit preprocessing, use lighter backbones and fewer candidate regions, and employ fully convolutional detection subnetworks.
- Detection frameworks: Single-stage frameworks poorly detect small objects, whereas two-stage frameworks do not share this stated weakness.
- Design choices: Recurring design choices include fully convolutional frameworks, complementary information from correlated tasks, sliding windows, and fusing features across backbone layers.
- Challenges and techniques: Deep backbones such as ResNet and ResNeXt have succeeded but are computationally expensive, while rotation variation receives less attention because popular benchmarks contain limited rotation diversity.
- Challenges and techniques: Object scale, lighting, background, blur, resolution, and noise contribute to recognition challenges, with scale variation especially affecting small objects.
- Challenges and techniques: Common scale-handling strategies include image pyramids, features from different convolutional layers, and network up-scaling for small-object detection.Image pyramids improve scale handling but are computationally expensive.
6. Scope for Future Work
The survey identifies persistent accuracy, speed, and resource challenges as priorities for future instance-segmentation research. It specifically emphasizes computational expense and the difficulty of achieving real-time performance.
- Future challenges: COCO average precision remains around 50%, leaving substantial room for improvement in instance segmentation.
- Future challenges: Research continues to balance hardware requirements against algorithmic simplicity and speed against accuracy.
- Future challenges: Hardware constraints limit research scope because instance segmentation is computationally expensive.
- Future challenges: Real-time instance segmentation remains unresolved despite potential applications in autonomous vehicles, security, and biometrics.
7. Conclusion
The paper surveys the evolution of image segmentation toward instance segmentation and reviews important issues and techniques in the field.
- The survey places instance segmentation within the continuing evolution from coarse to fine image inference.
- It discusses important issues and various instance-segmentation techniques from holistic and individual perspectives.
- The paper covers the taxonomy and strengths of the techniques it reviews.