Source-linked AI summary
Sparse Instance Activation for Real-Time Instance Segmentation
Tianheng Cheng, Xinggang Wang, Shaoyu Chen, Wenqiang Zhang, Qian Zhang, Chang Huang, Zhaoxiang Zhang, Wenyu Liu
TL;DR
Real-time instance segmentation remains challenging because existing methods rely on dense localization and multi-level prediction, increasing redundant computation and latency. SparseInst instead uses sparse instance activation maps to aggregate instance-level features for recognition and segmentation, achieving fast inference with competitive accuracy.
Problem
Real-time instance segmentation remains challenging because dense localization produces redundant predictions and multi-level prediction increases latency.
Method
SparseInst represents objects with sparse instance activation maps, aggregates highlighted image features into instance features, and performs recognition and segmentation from them.
Results
SparseInst achieves a competitive trade-off between accuracy and inference speed for real-time instance segmentation.
Takeaways & Limitations
Instance activation maps provide a general framework for end-to-end real-time instance segmentation with potential application to practical scenes.
Abstract
from arXiv · showhide
In this paper, we propose a conceptually novel, efficient, and fully convolutional framework for real-time instance segmentation. Previously, most instance segmentation methods heavily rely on object detection and perform mask prediction based on bounding boxes or dense centers. In contrast, we propose a sparse set of instance activation maps, as a new object representation, to highlight informative regions for each foreground object. Then instance-level features are obtained by aggregating features according to the highlighted regions for recognition and segmentation. Moreover, based on bipartite matching, the instance activation maps can predict objects in a one-to-one style, thus avoiding non-maximum suppression (NMS) in post-processing. Owing to the simple yet effective designs with instance activation maps, SparseInst has extremely fast inference speed and achieves 40 FPS and 37.9 AP on the COCO benchmark, which significantly outperforms the counterparts in terms of speed and accuracy. Code and models are available at https://github.com/hustvl/SparseInst.
1. Introduction
SparseInst addresses the challenge of real-time instance segmentation with sparse instance activation maps instead of boxes or dense centers. Its compact, fully convolutional design targets fast inference while preserving segmentation accuracy.
- Real-time instance segmentation remains challenging and urgent for applications such as autonomous driving and robotics.
- Dense center-based localization can produce redundant predictions and computation while limiting receptive-field context.CondInst produces more than 5456 instances for a 512 × 512 input.
- Instance activation maps highlight informative, instance-aware regions and aggregate features from the whole image for recognition and segmentation.They suppress obstructive pixels, provide more context, and avoid extra operations such as RoI-Align.
- SparseInst implements this paradigm as a simple, efficient framework with a backbone, encoder, and decoder for activation-map generation, recognition, and segmentation.
- 40.0 FPS and 37.9 mask AP on MS-COCO test-dev demonstrate SparseInst’s speed-accuracy performance on one NVIDIA 2080Ti GPU.With 448× input, it reaches 58.5 FPS with competitive accuracy.
2. Related Work
Prior instance-segmentation methods are commonly organized as region-based or center-based approaches, while bipartite matching supports end-to-end predictions without NMS. SparseInst instead uses instance activation maps as an object representation for fast recognition and segmentation.
- Existing instance-segmentation methods are divided into region-based and center-based representations.
- Region-based Methods: Region-based methods detect bounding boxes and use RoI-Pooling or RoI-Align to extract region features for pixel-wise segmentation.Mask R-CNN is presented as a representative strong baseline.
- Center-based Methods: Center-based methods represent objects by center pixels, while related approaches also predict contours, mask coefficients, encoded vectors, or dynamic mask kernels.The passage notes limitations for objects with hollows or multiple parts in contour-based approaches.
- Bipartite Matching for Object Detection: Bipartite matching has been explored for end-to-end object detection to avoid NMS, and related methods have applied mask encodings or queries to instance and panoptic segmentation.
- SparseInst is motivated by instance activation maps as a concise object representation for instance-level recognition and segmentation, with fast speed as its aim.
3. Method
SparseInst represents objects with sparse instance activation maps that highlight informative regions and support instance-level recognition and segmentation. Its compact, single-level framework uses IAM-based decoding, bipartite matching, and direct mask generation for fast inference.
- Instance Activation Maps: Sparse instance activation maps highlight informative regions for each object, whose aggregated features support recognition and segmentation.The maps are instance-aware weighted maps, and the resulting features are used for consequent recognition and instance-level segmentation.
- SparseInst Framework: The backbone extracts multi-scale features, while the encoder enlarges contextual information and fuses them into single-level features for faster inference.The encoder uses an instance context module to enhance contextual information and outputs features at 1/8 input resolution.
- IAM-based Segmentation Decoder: The IAM-based decoder has instance and mask branches: one produces activation maps, instance features, and kernels, while the other provides mask features.The predicted kernels are multiplied with mask features to generate instance segmentation masks.
- IAM-based Segmentation Decoder: Group-IAM uses multiple activation maps per object and concatenates group features to obtain finer-grained instance representations.Its Fiam is implemented with a 4-group 3×3 convolution.
- IAM-based Segmentation Decoder: IoU-aware objectness uses estimated mask IoUs as objectness targets and rescoring via ˜pi = √pi · si to align classification confidence with segmentation quality.The method uses one-to-one assignment and does not add a separate IoU prediction head.
- Label Assignment and Bipartite Matching Loss: Bipartite matching assigns predictions to ground-truth objects using classification scores and mask Dice coefficients, with the Hungarian algorithm finding the optimal match.The training loss includes classification, objectness, and segmentation terms; mask loss combines Dice and pixel-wise binary cross entropy.
- Inference: At inference, SparseInst directly produces instances, scores, and masks, then thresholds masks without sorting or NMS.Avoiding sorting and NMS makes the inference procedure concise and fast.
4. Experiments
Experiments on COCO evaluate SparseInst’s accuracy, speed, component choices, and qualitative behavior. The results show strong speed–accuracy performance and identify design choices that improve accuracy while controlling latency.
- Instance Context Encoder: 1.5 AP and 2.2 APL improvements result from adding pyramid pooling, while multi-scale fusion adds 0.7 AP and 2.0 APL with negligible latency.These modifications enlarge receptive fields, add object context, and strengthen multi-scale representation for single-level prediction.
- Structure of the Decoder: 0.5 AP is gained by adding coordinate features with negligible latency, while deformable convolution improves larger objects at a cost of +1.7ms.Reducing decoder width or depth improves speed but lowers performance; increasing depth from 4 to 6 adds 0.4 AP.
- Hybrid Mask Loss: Removing dice loss causes an 8.1-point AP collapse, while adding extra BCE loss improves performance by 1.0 AP and larger-object performance by 1.8 APL.The ablation evaluates the hybrid mask loss under full-resolution instance segmentation, where foreground–background imbalance is severe.
- IoU-aware Objectness: 1.3 AP is gained from IoU-aware objectness, and objectness alone improves AP by 0.7 even without rescoring.The results attribute this improvement to more instance-aware features; cross-entropy performs better than L1 loss.
- Qualitative Results: Qualitative results show well-localized masks aligned with activation maps, including crowded and dense scenes where different instances are distinguished.Activation maps highlight prominent object regions across varying scales, positions, and categories, while masks have fine boundaries.
5. Conclusion
The paper introduces instance activation maps as a sparse, instance-aware representation that highlights informative object regions and supports feature aggregation for recognition and segmentation. SparseInst applies this highlight-to-segment paradigm in an efficient framework for real-time instance segmentation.
- Instance activation maps are instance-aware weighted maps designed to highlight informative regions of objects.
- SparseInst aggregates instance features according to activation maps for instance-level recognition and segmentation.
- SparseInst is presented as a conceptually novel, efficient, end-to-end framework for real-time instance segmentation.
- The authors report fast inference speed with highly competitive accuracy and a favorable speed–accuracy trade-off.
A.1. TIDE Error Analysis
TIDE analysis compares SparseInst with YOLACT++ and SOLOv2 across error types. SparseInst has lower miss error but higher classification and duplicate-detection errors than the compared methods.
- SparseInst without Group-IAM runs at 40.2FPS and reaches 36.9AP, exceeding YOLACT++ and SOLOv2 on the reported speed and accuracy pairs.YOLACT++ is reported at 38.6FPS and 34.1AP; SOLOv2 at 38.2FPS and 34.0AP.
- SparseInst has lower miss error, indicating that it can discover more objects.
- SparseInst has higher classification-error and duplicate-detection-error portions than YOLACT++ or SOLOv2.The passage attributes both error types to classification.
- TIDE categorizes errors as classification, localization, missing detections, background detections, duplicate detections, and combined classification-plus-localization errors.