Source-linked AI summary
SOLO: A Simple Framework for Instance Segmentation
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, Lei Li
TL;DR
Instance segmentation must separate an arbitrary number of objects while assigning each instance pixel-level masks, and common approaches rely on bounding-box detection or pixel grouping. SOLO instead classifies instances by location and predicts masks directly; the paper reports strong speed and accuracy alongside extensions to other vision tasks.
Problem
Instance segmentation must separate an arbitrary number of objects while assigning each instance a pixel-level mask, and common approaches use box-based detection or pixel embeddings followed by grouping.
Source: §1 · p. 1Method
SOLO assigns pixels to instance categories based on quantized center locations and object sizes, then directly predicts semantic categories and instance masks.
Source: §1 · p. 1, §3.1.1 · p. 4Results
SOLO reports state-of-the-art instance-segmentation speed and accuracy, including 38.8% mask AP at 18 FPS on MS COCO with a ResNet-50 backbone on one V100, and reports extensions to detection and panoptic segmentation.
Source: §1 · p. 2, §5 · p. 13Takeaways & Limitations
With minimal modifications, SOLO is extended to object detection, panoptic segmentation, and instance-level image matting.
Source: §1 · p. 2, §5 · p. 13The default SOLOv2 has similar performance to the dynamic version while being conceptually simpler.
Source: §4 · p. 7
Abstract
from arXiv · showhide
Compared to many other dense prediction tasks, e.g., semantic segmentation, it is the arbitrary number of instances that has made instance segmentation much more challenging. In order to predict a mask for each instance, mainstream approaches either follow the 'detect-then-segment' strategy (e.g., Mask R-CNN), or predict embedding vectors first then cluster pixels into individual instances. In this paper, we view the task of instance segmentation from a completely new perspective by introducing the notion of "instance categories", which assigns categories to each pixel within an instance according to the instance's location. With this notion, we propose segmenting objects by locations (SOLO), a simple, direct, and fast framework for instance segmentation with strong performance. We derive a few SOLO variants (e.g., Vanilla SOLO, Decoupled SOLO, Dynamic SOLO) following the basic principle. Our method directly maps a raw input image to the desired object categories and instance masks, eliminating the need for the grouping post-processing or the bounding box detection. Our approach achieves state-of-the-art results for instance segmentation in terms of both speed and accuracy, while being considerably simpler than the existing methods. Besides instance segmentation, our method yields state-of-the-art results in object detection (from our mask byproduct) and panoptic segmentation. We further demonstrate the flexibility and high-quality segmentation of SOLO by extending it to perform one-stage instance-level image matting. Code is available at: https://git.io/AdelaiDet
1 INTRODUCTION
SOLO reframes instance segmentation around location-based instance categories, directly predicting object classes and masks rather than relying on box detection or pixel grouping. The introduction reports strong speed and accuracy, plus extensions to other vision tasks.
1 INTRODUCTION
- Instance segmentation provides pixel-level localization with irregular object boundaries, whereas bounding boxes are coarse; existing approaches use detect-then-segment or embeddings followed by grouping.The introduction presents SOLO as a direct alternative to those step-wise paradigms.Source: §1 · p. 1
- SOLO assigns instance categories by quantized center location and object size, turning instance separation into classification and mask prediction.The introduction notes that most COCO object pairs have distinct centers, while many of the remaining close pairs differ in size; FPN assigns different sizes to feature levels.Source: §1 · pp. 1–2
- The framework is trained end-to-end from full instance-mask annotations and predicts masks directly, without box detection or grouping post-processing.The proposed variants are Vanilla SOLO, Decoupled SOLO, and Dynamic SOLO (SOLOv2).Source: §1 · pp. 1–2
- 500 masks take less than 1 ms for Matrix NMS in a simple Python implementation, and it yields 0.4% AP better than Fast NMS.Matrix NMS uses parallel matrix operations to avoid sequential, recursive suppression, which is especially costly when computing mask-pair IoU.Source: §1 · p. 2
- 38.8% mask AP at 18 FPS on MS COCO is reported for SOLO with a ResNet-50 backbone on one V100, outperforming state-of-the-art instance segmentation methods in speed and accuracy.A lightweight version reaches 37.1% mask AP at 31.3 FPS; the method also reports 44.9% AP for object detection from mask-derived boxes and state-of-the-art panoptic segmentation.Source: §1 · p. 2
2 RELATED WORK
The related-work section contrasts SOLO with top-down, bottom-up, and semi-direct segmentation methods, then situates its dynamic-convolution design and Matrix NMS among related techniques. It emphasizes SOLO's box-free, grouping-free prediction from an image directly to masks and class probabilities.
Top-down instance segmentation
- Top-down methods segment within detected boxes, while SOLO is box-free and avoids restrictions from anchor-box locations and scales.Mask R-CNN segments instances within detected bounding boxes; TensorMask uses dense sliding windows, and YOLACT crops assembled masks using predicted boxes.Source: §2 · p. 3
Bottom-up instance segmentation
- Bottom-up methods group pixels using embeddings, affinities, or subgrouping, whereas SOLO learns directly from instance-mask annotations and predicts masks without grouping post-processing.The section notes that bottom-up methods typically lag top-down methods in accuracy, especially on diverse scenes.Source: §2 · p. 3
Direct instance segmentation
- AdaptIS and PolarMask avoid boxes in training, but are described as step-wise or as using a coarse parametric mask representation; SOLO directly predicts masks and class probabilities.The section characterizes SOLO as fully convolutional, box-free, and grouping-free.Source: §2 · p. 3
Dynamic convolutions
- SOLOv2 uses dynamic convolutions to learn instance segmenters by absolute locations, applying position information globally rather than encoding it separately for each instance as CondInst does.The section says both methods use a somewhat similar dynamic scheme, but their methodologies differ.Source: §2 · p. 3
Non-maximum suppression
- Matrix NMS addresses both hard removal and sequential operations, processing 500 masks in less than 1 ms in simple Python and yielding 0.4% AP better than Fast NMS.Fast NMS is described as parallel but faster at the cost of performance deterioration.Source: §2 · p. 3
3 OUR APPROACH
SOLO assigns each instance to a grid location and jointly predicts its category and mask, framing instance segmentation as category-aware dense prediction. Its variants and training and inference strategies preserve this location-based formulation while reducing prediction cost and handling duplicate masks.
3.1 Problem Formulation
- Each grid cell predicts an instance’s semantic category and corresponding mask, establishing a direct association between category and class-agnostic mask.SOLO uses normalized pixel coordinates to make mask features position-sensitive, then gathers grid predictions and applies mask NMS.Source: §3.1 · p. 4, §3.1.1 · p. 4, §3.1.2 · p. 4
3.2.2 Decoupled SOLO
- Vanilla SOLO maps the input to S² mask channels, while Decoupled SOLO replaces them with S-sized outputs for the two grid axes.The decoupled mask combines the relevant row and column maps element-wise, reducing output space from H×W×S² to H×W×2S.Source: §3.2.2 · p. 4
3.2.3 Dynamic SOLO
- Dynamic SOLO predicts location-conditioned convolution kernels that operate on mask features to generate instance masks.Its unified high-resolution mask feature is formed by fusing FPN levels P2 to P5; normalized coordinates are fed to the deepest level.Source: §3.2.3 · pp. 5–6
3.2.4 Decoupled Dynamic SOLO
- Decoupled Dynamic SOLO divides mask-kernel weights into groups and reduces mask-feature channels, maintaining high performance with a compact representation.For A = 4, the example feature space shrinks from H×W×256 to H×W×64.Source: §3.2.4 · p. 6
3.3.1 Label Assignment
- Positive grid samples are selected within a scaled center region of each ground-truth mask, with each positive assigned a binary mask target.The center region uses scale ϵ = 0.2; row-major mask ordering is reported to work well.Source: §3.3.1 · p. 6
3.3.2 Loss Function
- Training combines category-classification and mask-prediction losses, using Dice Loss for masks because of its effectiveness and training stability.The mask loss is evaluated for positive samples, and λ is set to 3.Source: §3.3.2 · p. 6
3.3.3 Inference Pipeline
- Inference filters low-confidence predictions, generates binary masks, and uses mask NMS to select final instances.Vanilla and Decoupled SOLO send their top 500 scoring masks to NMS; Dynamic SOLO applies predicted kernels to mask features before thresholding masks at 0.5.Source: §3.3.3 · p. 6
3.3.4 Matrix NMS
- Matrix NMS updates mask scores in parallel and is reported as 9× faster than traditional NMS and more accurate.It computes pairwise IoUs for the top-scoring predictions and uses decay factors to update their scores.Source: §3.3.4 · p. 7
4 EXPERIMENTS
The experiments evaluate SOLO and SOLOv2 across instance segmentation, detection, panoptic segmentation, and instance-level matting, with ablations, qualitative analysis, and details of the matting method and training data.
4.1.1 MS COCO Instance Segmentation
- 39.7% mask AP: SOLOv2 with ResNet-101 surpasses other state-of-the-art methods on COCO test-dev and gains +5.0% APL over Mask R-CNN.The reported speed-accuracy comparison also shows SOLOv2 outperforming a range of methods in both speed and accuracy, with finer masks than Mask R-CNN.Source: Table 1 · p. 7, §4.1.1 · p. 8, Figure 5 · p. 8
4.1.2 Instance Segmentation on LVIS; 4.1.3 Instance Segmentation on Cityscapes
- About 1% AP: SOLOv2 outperforms the Mask R-CNN baseline on LVIS; on Cityscapes, it exceeds Mask R-CNN by 0.9% AP.On LVIS, SOLOv2 also achieves a 6.7% AP improvement for large objects (APL).Source: §4.1.2 · p. 8, §4.1.3 · p. 8
4.2.1 SOLO Ablation
- 35.8% AP: multi-level FPN prediction improves on single-scale SOLO’s 29.0% AP; increasing the grid from S=12 to S=24 raises AP from 27.2% to 29%.CoordConv yields 3.6% absolute AP gains, Dice Loss performs best without manually adjusted loss hyperparameters, and decoupled SOLO slightly exceeds vanilla SOLO (37.4% vs. 36.8% AP) by simplifying location-category prediction.Source: §4.2.1 · pp. 8–9
4.2.2 SOLOv2 Ablation
- SOLOv2 ablations find that explicit coordinates improve AP by 1.5%, unified mask features improve results, and dynamic heads outperform decoupled heads by 0.7% AP.Matrix NMS runs in less than 1 ms; the paper also reports real-time lightweight settings and says SOLOv2 consistently outperforms Mask R-CNN as training time increases, despite Mask R-CNN’s slight advantage in the first five epochs.Source: §4.2.2 · pp. 9–10
4.3 Other Improvements
- 42.8% AP: auxiliary semantic loss and a denser grid raise SOLOv2’s best single-model result from 41.7% AP on COCO test-dev.The auxiliary loss adds 0.3% AP and the denser grid adds 0.7% AP; decoupled dynamic SOLO maintains similar performance with a compact 64-channel mask feature instead of 256 channels.Source: §4.3 · p. 11, Table 6 · p. 12
4.4 Qualitative Analysis
- SOLO’s mask features are position-aware, scanning objects horizontally and vertically, while some channels activate across foreground instances.Qualitative outputs show promising segmentation across diverse scenes, with well-segmented boundaries, especially on large objects.Source: §4.4 · p. 11, Figure 7 · p. 12
4.5 Extension: Object Detection; 4.6 Extension: Panoptic Segmentation
- SOLOv2’s instance masks also produce bounding boxes, outperforming most recent detectors in speed and accuracy, while its panoptic extension achieves state-of-the-art results.The detection results require no box-based training or engineering; the panoptic extension adds a semantic segmentation branch and outperforms other recent box-free methods by a large margin.Source: §4.5 · p. 12, Figure 8 · p. 12, §4.6 · p. 12
4.7 Extension: Instance-level Image Matting; 4.7.1 Method
- SOLO extends to instance-level image matting, addressing trimap-free ambiguity by estimating a soft matte for each selected object instance.The method uses raw-image input to improve detail features, adds a mean average error term to the loss, and produces soft mattes without thresholding.Source: §4.7 · p. 13, §4.7.1 · p. 13
4.7.1 Method
- Predicted convolution kernels generate individual soft mattes for all objects.The kernels are convolved with the feature maps.Source: §4.7.1 · p. 13
4.7.2 Dataset
- The training set combines 880 human alpha mattes with 520 LVIS training images containing people [48].The authors note that no existing public dataset provides separate alpha-matte annotations for each instance.Source: §4.7.2 · p. 13
4.7.3 Results
- The model produces high-quality alpha mattes whose accurate object boundaries support natural-looking image editing and compositing.Detail refinement recovers boundary and detail features that are coarse in the input feature maps.Source: §4.7.3 · p. 13
- Human-matting training also yields reasonable results on other object categories because the matte prediction is class-agnostic.The model retains object recognition by freezing the category branch initialized from COCO instance-segmentation weights; enhanced matte features learned from humans benefit other categories.Source: §4.7.3 · p. 13
5 CONCLUSION
The conclusion presents SOLO as a simple, direct instance-segmentation framework with variants built on the same location-based principle. It also describes extensions to object detection, panoptic segmentation, and instance-level image matting.
5 CONCLUSION
- SOLO is presented as a simple, direct framework for instance segmentation, with variants following the same principle of segmenting objects by locations.The appendix introduces further results and instance-level image matting.Source: §5 · p. 13, Appendix A · p. 15
- The paper extends SOLO, with minimal modifications, to object detection, panoptic segmentation, and instance-level image matting.The conclusion presents these extensions as demonstrations of the framework’s flexibility and effectiveness.Source: §5 · p. 13
- The authors hope SOLO’s simplicity, efficiency, and strong performance will make it a cornerstone for instance-level recognition tasks.Source: §5 · p. 13
A.1 Introduction
The paper introduces soft instance segmentation (SOFI), which seeks high-quality soft mattes for individual objects without trimaps or manual scratch annotation. Its SOSO model extends SOLO, and the reported results include gains on the SOFI benchmark and image matting.
A.1 Introduction
- Instance-level soft mattes are presented as a way to handle arbitrary foreground objects and support more flexible, automated image editing.Source: §A.1 · p. 15
- The paper motivates SOFI by noting that instance segmentation’s binary masks and moderate-overlap evaluation do not provide the high-quality masks needed for image editing.Source: §A.1 · p. 15
- SOFI requires high-quality soft mattes for all object instances without trimaps or manual scratch annotation, targeting photo-realistic image manipulation.The task combines high-level instance and category discrimination with low-level shape-detail processing.Source: §A.1 · p. 15
- SOSO is a single-shot, fully convolutional model based on SOLO that directly predicts soft instance mattes and class probabilities by fusing high- and low-level features.Source: §A.1 · pp. 15–16
- SOFI achieves a relative 30.9% improvement over our baseline on the proposed SOFI benchmark.SOSO also outperforms the state-of-the-art trimap-free approach on image matting.Source: §A.1 · p. 16
A.2 SOFI Task
SOFI represents each object with a soft matte and class probabilities, and its benchmark combines instance recognition with pixel-transition evaluation. The appendix describes SOSO’s architecture, training, and inference pipeline.
A.2 SOFI Task
- SOFI maps an image to soft mattes and class probabilities for each instance, with matte values from 0 to 1 representing foreground-to-background transition.Source: §A.2 · p. 16
- SOFI evaluation combines per-instance recognition with per-pixel transition prediction, using SOFI-MSE and SOFI-SAD based on image-matting error functions.The errors are averaged over object classes and test images.Source: §A.2 · p. 16
- SOFI-200 contains 200 images and 537 high-quality alpha mattes; its training data also includes 880 human-matting mattes and 520 LVIS person-category images.Source: §A.2 · p. 16
A.3.1 Network Architecture
- SOSO builds on SOLOv2 with an FPN prediction head and a Details Refinement module that fuses low-level image details with upsampled features to produce matte features.Predicted convolution kernels apply to these features to generate individual soft mattes.Source: §A.3.1 · p. 17
A.3.2 SOSO Learning
- SOSO uses SOLO-style label assignment with soft mattes as mask targets, and trains category and matte prediction losses using mixed matting and instance-segmentation data.The category branch is frozen; matte supervision uses mean average error and Dice loss, normalized by the number of instances.Source: §A.3.2 · p. 17
A.3.3 Inference
- At inference, SOSO filters low-confidence grid predictions, applies their matte kernels to matte features, takes the sigmoid output, and uses Matrix NMS to remove redundant predictions.Source: §A.3.3 · p. 17
A.4 Experiments
The experiments evaluate SOFI-200 and also apply the SOFI-developed model to instance segmentation and image matting, with ablations examining component contributions.
A.4 Experiments
- Experiments report quantitative and qualitative SOFI-200 results, comparisons on instance segmentation and image matting, and ablations of component contributions.Source: §A.4 · p. 17
- Unless specified otherwise, models use ResNet-101 and COCO-pretrained weights, train for six epochs, and use synchronized SGD across eight GPUs with eight total images per minibatch.Source: §A.4 · p. 17
A.4.1 Main Results
On SOFI, SOSO substantially improves over SOLOv2 under the same training setting. The results also show that performance differs across existing instance-segmentation methods and motivate a dedicated benchmark for detailed, accurate instance segmentation.
A.4.1 Main Results
- SOLOv2 performs much better than Mask R-CNN and PointRend on SOFI when all are trained on COCO for instance segmentation.The passage attributes Mask R-CNN's coarse masks to its mask head's typical 28 × 28 resolution.Source: §A.4.1 · p. 17
- The SOFI results support introducing a dedicated task and benchmark for more detailed and accurate instance segmentation.Source: §A.4.1 · p. 17
- 30.9% relative improvement: SOSO significantly boosts performance over SOLOv2 under the same training setting.Training SOLOv2 on the matting dataset brings only slight improvements.Source: §A.4.1 · p. 17
A.4.2 Comparisons with Image Matting Methods
The proposed SOFI solution also applies to classical image matting, treated as SOFI with a single object. Comparisons report competitive trimap-free results on human matting and superiority over LFM on three of four whole-image metrics.
A.4.2 Comparisons with Image Matting Methods
- Classical image matting is treated as a special case of SOFI with a single object.Source: §A.4.2 · p. 17
- The SOSO overview maps an input image to object categories and corresponding soft mattes, while the details-refinement module recovers low-level feature-map details.The accompanying figures show matting outputs and image-editing applications, including a Bokeh effect and image composition.Source: Figure S3 · p. 18, Figure S4 · p. 18, Figure S5 · p. 19
- On the human matting dataset, the method is competitive on all four metrics over unknown regions using only raw RGB input, without ground-truth trimaps.The compared trimap-based methods receive ground-truth trimaps and are evaluated only over unknown regions.Source: §A.4.2 · p. 18, Table S2 · p. 19
- The method shows large superiority over LFM on three of four metrics calculated over the whole image.LFM takes an RGB image and predicts a single soft matte without instance labels.Source: §A.4.2 · p. 18, Table S3 · p. 19
A.4.3 Ablative Analysis
Ablations on the human matting validation set examine the Details Refinement module and loss components; the results favor raw RGB input and using both losses together.
A.4.3 Ablative Analysis
- Removing raw RGB input to Details Refinement sharply reduces results, indicating raw-input low-level cues matter for detail extraction.Source: §A.4.3 · p. 19
- Among single loss terms, MAE has advantages over Dice, while the model using both losses achieves the best performance.Source: §A.4.3 · p. 19
- Dice loss alone also achieves good matting metrics despite being proposed for per-pixel binary classification.Source: §A.4.3 · p. 19
APPENDIX B RESULTS ON OBJECT DETECTION
The appendix reports results for using SOLO in bounding-box-based object detection.
APPENDIX B RESULTS ON OBJECT DETECTION
- The appendix presents results on applying SOLO to bounding-box-based object detection.Source: Appendix B · p. 19
APPENDIX C VISUALIZATION
The appendix presents SOLO visualizations, reporting 39.7% mask AP for SOLOv2 with a ResNet-101 backbone and highlighting boundary detail quality, especially on large objects.
APPENDIX C VISUALIZATION
- Table S4 presents ablation experiments on the human matting validation set.Source: Table S4 · p. 19
- RGB input plays a key role in the Details Refinement module.Source: Appendix C · p. 19
- Instance boundaries are segmented well in diverse scenes, especially for large objects.Source: Appendix C · p. 19
- Table S5 reports that SOLO's mask-derived bounding boxes outperform most state-of-the-art methods in box AP on MS COCO test-dev.The passage does not provide specific box AP values.Source: Table S5 · p. 20
- 39.7% mask AP was achieved by SOLOv2 with a ResNet-101 backbone on COCO test-dev.The model was trained on COCO train2017; Figure S6 visualizes the instance segmentation results, with different objects shown in different colors.Source: Appendix C · p. 19, Figure S6 · p. 20
- Figure S7 plots loss curves for vanilla and decoupled SOLO trained for 36 epochs on MS COCO train2017 and tested on val2017.Source: Figure S7 · p. 20