PaperlayerView exact revision on arXiv

Source-linked AI summary

SOLO: Segmenting Objects by Locations

Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, Lei Li

arXiv:1912.04488v3cs.CV

TL;DR

Instance segmentation must separate a variable number of objects at pixel level, while existing approaches rely on detection or grouping steps. SOLO assigns pixels instance categories based on location and size, reducing the task to two classification problems. It achieves on-par accuracy with Mask R-CNN on COCO and outperforms previous one-stage methods, while remaining limited by mask prediction quality and grid assumptions.

  • Problem

    Instance segmentation is difficult because images contain a variable number of instances that must each receive a pixel-level mask.

  • Method

    SOLO assigns pixels instance categories defined by quantized object-center locations and sizes, using two pixel-level classification tasks with end-to-end mask supervision.

  • Results

    SOLO achieves on-par accuracy with Mask R-CNN on COCO and outperforms previous one-stage instance segmentation methods.

  • Takeaways & Limitations

    SOLO provides a simple, flexible, box-free framework that can also generate reasonable instance contours with almost no modification.

  • Takeaways & Limitations

    Replacing predicted masks with ground-truth masks raises AP to 68.1%, indicating substantial room to improve the mask branch.

Abstract

from arXiv · show

We present a new, embarrassingly simple approach to instance segmentation in images. Compared to many other dense prediction tasks, e.g., semantic segmentation, it is the arbitrary number of instances that have made instance segmentation much more challenging. In order to predict a mask for each instance, mainstream approaches either follow the 'detect-thensegment' strategy as used by Mask R-CNN, or predict category masks first then use clustering techniques to group pixels into individual instances. We view the task of instance segmentation from a completely new perspective by introducing the notion of "instance categories", which assigns categories to each pixel within an instance according to the instance's location and size, thus nicely converting instance mask segmentation into a classification-solvable problem. Now instance segmentation is decomposed into two classification tasks. We demonstrate a much simpler and flexible instance segmentation framework with strong performance, achieving on par accuracy with Mask R-CNN and outperforming recent singleshot instance segmenters in accuracy. We hope that this very simple and strong framework can serve as a baseline for many instance-level recognition tasks besides instance segmentation.

1 Introduction

SOLO reframes instance segmentation as dense classification by assigning pixels instance categories based on object center locations and sizes. This directly avoids the box-detection or pixel-grouping steps used by prevailing top-down and bottom-up approaches.

  • Instance segmentation must separate a variable number of objects while assigning each instance a pixel-level semantic mask.
  • Existing top-down methods detect bounding boxes before masks, whereas bottom-up methods learn pixel embeddings and require grouping post-processing.
  • SOLO introduces instance categories that encode quantized object-center locations and sizes, converting instance separation into classification.
  • An S × S grid provides S^2 center-location classes, with each output channel predicting the mask of the instance assigned to that location.
  • FPN assigns different object sizes to separate feature-map levels, enabling regular classification of instances across scales.
  • SOLO is end-to-end trainable using mask annotations alone, performs pixel-level segmentation without boxes or grouping, and achieves on-par COCO accuracy with Mask R-CNN.

2 Our Method: SOLO

SOLO reformulates instance segmentation into coordinated category and mask predictions over image grids. It uses position-sensitive mask prediction, feature pyramids, center-based labels, and a joint training and inference procedure.

  • Framework: Each grid cell predicts an object’s semantic category and corresponding instance mask, creating a one-to-one correspondence between category outputs and masks.For an S×S grid, the mask tensor contains S^2 channels, with channel k assigned to grid (i, j).
  • Framework: SOLO requires position-sensitive mask prediction because each mask is conditioned on its grid cell.Normalized x, y coordinates are concatenated with features to provide spatial information to the convolutional network.
  • Architecture: FPN assigns different object sizes to different feature-map levels, while prediction-head weights are shared across levels.The framework attaches category and mask prediction heads to feature-pyramid maps with potentially different grid numbers.
  • Label Assignment: Center sampling labels grid cells within a scaled center region of each ground-truth mask as positive category samples.With ϵ = 0.2, there are on average 3 positive samples per ground-truth mask.
  • Learning and Inference: The joint loss combines focal category loss with weighted mask loss, and Dice loss is selected for mask optimization.The mask-loss weight λ is set to 3; inference filters predictions, keeps the top 500 masks, and rescales confidence using maskness.

3 Experiments

Experiments evaluate SOLO on COCO, analyze its grid, FPN, coordinate, loss, and head-depth choices, and measure speed and remaining mask-prediction errors.

  • Main Results: 37.8% mask AP makes ResNet-101 SOLO competitive with Mask R-CNN and stronger than prior one-stage methods on COCO test-dev.The comparison uses single-model results under the reported evaluation setup.
  • Grid Number: 27.2% AP with S = 12 and 29% AP with S = 24 show that increasing grid number improves single-scale SOLO.The single-scale model is described as suitable when object scales vary little.
  • Multi-level Prediction: 35.8 AP with multi-level FPN prediction improves over single-scale SOLO by assigning different object scales to pyramid levels.The corresponding grid numbers from P2 to P6 are [40, 36, 24, 16, 12].
  • CoordConv: 3.6 absolute AP gains from CoordConv show that explicit input coordinates improve spatially variant, position-sensitive predictions.Adding more than one CoordConv layer brings no noticeable additional improvement.
  • Loss Function: Focal Loss outperforms ordinary BCE in the mask branch, while Dice Loss achieves the best and more stable AP according to the ablation caption.The loss comparison addresses the dominance of background pixels in instance masks.
  • Head Depth: Increasing head depth from 4 to 7 gives 1.2 AP, after which performance becomes stable; the experiments therefore use depth 7.The mask head needs sufficient representation capacity for position-conditioned masks.
  • Speed: 22.5 FPS at 34.2 mask AP is achieved by SOLO-512 on a single V100, while Res-101-FPN SOLO runs at 10.4 FPS.The smaller model uses a 512-pixel shorter image size instead of 800.
  • Error Analysis: 68.1% AP after replacing predicted masks with ground-truth masks indicates substantial remaining room to improve SOLO’s mask branch.The reported baseline mask AP rises from 37.1 to 68.1 in this error analysis.

4 Decoupled SOLO

Decoupled SOLO factorizes SOLO’s S² location-channel output into two axis-specific tensors, reducing redundancy while preserving mask prediction through element-wise multiplication. It achieves slightly better accuracy than vanilla SOLO and substantially reduces output-space and GPU-memory costs.

  • Motivation: S = 20 produces 400 channel maps in vanilla SOLO, motivating Decoupled SOLO’s more efficient factorization for sparsely located objects.The decoupled variant is described as equivalent while using a smaller output space.
  • Architecture: Decoupled SOLO replaces M ∈ R^(H×W×S²) with X and Y tensors of shape H×W×S, reducing output space from H×W×S² to H×W×2S.The two tensors correspond to the grid’s two axes.
  • Mask prediction: For grid location (i, j), the object mask is the element-wise product of the corresponding channel maps x_j and y_i after sigmoid operation.The formulation treats horizontal and vertical location categories as independent.
  • Results: 0.6 AP gains over vanilla SOLO are reported for Decoupled SOLO under the same hyper-parameters.The result supports comparable or slightly improved accuracy despite the reduced output space.
  • Results: 40.5 AP is achieved with a DCN-101 backbone, while training and testing require considerably less GPU memory because the output space is largely reduced.The paper characterizes Decoupled SOLO as an efficient, accuracy-equivalent SOLO variant.
  • Results: 37.8 mask AP is achieved on COCO test-dev with a Res-101-FPN backbone trained on COCO train2017.The figure visualizes the corresponding instance-segmentation results.

5 Conclusion

SOLO is presented as a direct, end-to-end instance-segmentation framework that maps raw images to instance masks. Its design removes grouping post-processing and box-detection and RoI operations while maintaining constant inference time.

  • 5 Conclusion: SOLO is a direct instance-segmentation framework that is trainable end to end.The framework maps raw input images toward the desired instance masks.
  • 5 Conclusion: Constant inference time is achieved while eliminating grouping post-processing, bounding-box detection, and RoI operations.These are respectively associated with bottom-up and top-down instance-segmentation approaches.

A.1 Multi-level Prediction

Multi-level prediction uses five FPN pyramids to handle objects at different scales. This gives a 6.8 AP improvement over single-scale SOLO, with larger grid numbers assigned to smaller instances.

  • A.1 Multi-level Prediction: Five FPN pyramids segment objects of different scales, with ground-truth mask scales explicitly assigning objects to pyramid levels.The multi-level design is evaluated against single-scale SOLO.
  • A.1 Multi-level Prediction: 6.8 AP gains are obtained over single-scale SOLO through multi-level prediction.The grid number increases for smaller instances because they have a larger existence space.

B.1 Single-scale 1× Training

The single-scale section reports 1× training results on val2017 and compares Mask R-CNN using Dice loss with its original BCE loss. Dice loss performs worse for Mask R-CNN in this comparison.

  • B.1 Single-scale 1× Training: Single-scale models are trained with a 1× schedule and evaluated on val2017.The results are listed in Table 9.
  • B.1 Single-scale 1× Training: -0.9 AP is reported for Mask R-CNN with Dice loss relative to its original BCE loss.The passage attributes this to the absence of foreground/background imbalance in local bounding-box segmentation.

B.3 SOLO for Instance Contour Detection

SOLO extends its framework from instance segmentation to instance contour detection by converting COCO masks into contours and optimizing contour and semantic-category branches in parallel.

  • SOLO converts MS COCO ground-truth masks into instance contours using OpenCV’s findContours function.
  • The contour branch is optimized with Focal Loss alongside the semantic category branch.Other settings remain the same as the instance segmentation baseline.
  • Figure 7 visualizes contour detections from a ResNet-50-FPN model trained on COCO train2017, using different colors for different instances.

B.4 Qualitative Results

The paper provides qualitative instance-segmentation visualizations, including Res-101-FPN results trained on COCO train2017 with a reported mask AP of 37.8 on COCO test-dev.

  • The paper presents these additional visualizations in Figure 8.
  • Figure 8 shows qualitative instance-segmentation results from a model using the Res-101-FPN backbone.
  • The model is trained on COCO train2017 and achieves a mask AP of 37.8 on COCO test-dev.
Loading 1912.04488v3…