PaperlayerView exact revision on arXiv

Source-linked AI summary

YOLACT: Real-time Instance Segmentation

Daniel Bolya, Chong Zhou, Fanyi Xiao, Yong Jae Lee

arXiv:1904.02689v2cs.CV

TL;DR

Existing instance-segmentation methods prioritize performance over speed and rely on sequential feature repooling, making real-time operation difficult. YOLACT removes explicit localization by generating shared prototype masks and per-instance coefficients in parallel, achieving competitive COCO performance at much higher speed while retaining high-quality masks; Fast NMS further reduces suppression cost.

  • Problem

    Existing instance-segmentation methods prioritize performance over speed, while repooling-based feature localization is sequential and difficult to accelerate.

  • Method

    YOLACT performs prototype-mask generation and per-instance coefficient prediction in parallel, then assembles masks through a basic linear combination without explicit localization.

  • Results

    YOLACT-550 achieves competitive COCO instance-segmentation performance at 3.8x the speed of the previous fastest method, while outperforming Mask R-CNN at 95% IoU with 1.6 versus 1.3 AP.

  • Takeaways & Limitations

    The shared prototype representation supports emergent instance localization and practical advantages including fast inference, high-quality large-object masks, and applicability to modern object detectors.

  • Takeaways & Limitations

    YOLACT trails state-of-the-art instance-segmentation methods in overall performance, with most errors caused by detector mistakes.

Abstract

from arXiv · show

We present a simple, fully-convolutional model for real-time instance segmentation that achieves 29.8 mAP on MS COCO at 33.5 fps evaluated on a single Titan Xp, which is significantly faster than any previous competitive approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. Finally, we also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty.

1. Introduction

YOLACT targets the gap between accurate but slow instance segmentation and real-time detection by using parallel prototype-mask and coefficient prediction without explicit localization. This design yields a fast, competitive framework with lightweight assembly and shared prototype representations.

  • Motivation: Real-time instance segmentation methods have lagged behind object detection because localization, repooling, and post-processing make mask generation difficult to accelerate.The paper positions YOLACT as a fast, one-stage counterpart to real-time detectors such as SSD and YOLO.
  • Method: YOLACT forgoes explicit localization by generating non-local prototype masks and predicting per-instance linear combination coefficients in parallel.Final masks are formed by combining prototypes and then cropping with predicted bounding boxes.
  • Prototype representation: Shared prototypes form a distributed representation across categories, with emergent behaviors including spatial partitioning, instance localization, contours, and directional maps.The prototypes learn to localize masks despite the model being fully convolutional.
  • Practical advantages: Parallel computation and lightweight assembly add only marginal overhead to a one-stage detector, allowing 30 fps even with ResNet-101.The entire mask branch takes approximately 5 ms to evaluate, and the approach can be added to almost any modern object detector.
  • Contribution: YOLACT is presented as the first real-time instance segmentation algorithm above 30 fps with competitive MS COCO results.The contribution is framed around closing the speed gap between instance segmentation and real-time object detection.

2. Related Work

Prior instance segmentation methods use repooling, multiple stages, or expensive post-processing and clustering, limiting their real-time viability. Prototype-based approaches generally represent features, whereas YOLACT uses prototypes to assemble instance masks.

  • Two-stage methods: Two-stage methods such as Mask R-CNN repool features for each region of interest, preventing real-time speeds even at reduced image sizes.Their subsequent per-region computations contribute to the speed limitation.
  • One-stage methods: One-stage methods use position-sensitive pooling, mask voting, or related computations, remaining far from real-time despite parallel processing.The paper contrasts these operations with YOLACT’s lighter linear assembly step.
  • Clustering and embedding methods: Semantic-segmentation and embedding-based methods require multiple stages or expensive clustering procedures, limiting their viability for real-time applications.These methods form instances through boundary detection, pixel clustering, or learned embeddings.
  • Real-time instance segmentation: Existing real-time methods such as Straight to Shapes and Box2Pix operate at real-time rates on selected datasets but have accuracies far from modern baselines.The cited speeds include 30 fps on Pascal SBD, 10.9 fps on Cityscapes, and 35 fps on KITTI.
  • Prototypes: Earlier prototype methods represent features, while YOLACT learns prototypes specifically to assemble masks for instance segmentation.The distinction concerns the role prototypes play in the representation and output pipeline.

3. YOLACT

YOLACT adds instance segmentation to a one-stage detector without explicit feature repooling by combining image-wide prototype masks with per-instance coefficients. This parallel design preserves spatial coherence, enables fast mask assembly, and yields emergent localization and partitioning behaviors.

  • Architecture: YOLACT splits instance segmentation into parallel prototype-mask generation and per-anchor mask-coefficient prediction, then linearly combines them into instance masks.The prototype branch uses an FCN, while the detection head predicts one coefficient per prototype for each anchor.
  • Rationale: 20.7 mAP is obtained by the fc-mask model, showing that directly reshaping fully connected outputs into masks is insufficient.YOLACT instead combines fully connected coefficient prediction with convolutional, spatially coherent prototypes.
  • Prototype Branch: Higher-resolution, deeper protonet features improve mask quality and small-object performance, while prototype outputs receive supervision only through the assembled mask loss.The implementation upsamples FPN features and trains with pixel-wise binary cross-entropy on assembled masks.
  • Mask Assembly: The mask assembly uses a single matrix multiplication followed by a sigmoid, with P containing h×w×k prototypes and C containing n×k coefficients.Only instances surviving NMS and score thresholding are assembled.
  • Emergent Behavior: Prototype activations learn image partitions and other spatial patterns, allowing combinations such as subtracting one prototype from another to separate overlapping same-class instances.The paper also reports directional, contour, background, and ground-related prototype behaviors.
  • Emergent Behavior: Increasing the prototype count can make coefficient prediction harder, because an error in one coefficient may erase a mask or introduce leakage from other objects.The authors report little degradation with k = 32 and mainly redundant edge-level variations at higher k.

4. Backbone Detector

The backbone detector follows RetinaNet while prioritizing speed and feature richness for prototype and coefficient prediction. The default uses ResNet-101 with FPN and a 550 × 550 input, outperforming a matched ResNet-101 SSD variant in both speed and performance.

  • Backbone Design: YOLACT prioritizes a backbone that combines speed with feature richness because prototype and coefficient prediction require strong features.Its detector design closely follows RetinaNet with an emphasis on speed.
  • Detector Configuration: The default detector uses ResNet-101 with FPN, 550 × 550 images, three anchors per feature location, and anchor scales from 24 to 384 pixels.Aspect ratios are [1, 1/2, 2], and image aspect ratios are not preserved to keep evaluation time consistent.
  • Backbone Results: The ResNet-101 backbone with these design choices performs better and faster than an SSD model modified to use ResNet-101 at the same image size.

5. Other Improvements

YOLACT adds speed-oriented and training-only improvements around its detector. Fast NMS parallelizes suppression with GPU matrix operations, while auxiliary semantic-segmentation supervision increases feature richness without test-time cost.

  • Fast NMS: Fast NMS lets detections suppress one another in parallel, enabling implementation entirely with standard GPU-accelerated matrix operations.It relaxes traditional NMS by allowing already-removed detections to suppress other detections.
  • Evaluation: 29.8 mAP at 33.0 fps is reported for the base model on COCO test-dev with the confidence threshold set to 0.3.
  • Fast NMS: Fast NMS computes a c × n × n pairwise IoU matrix for top detections, masks lower-triangle and diagonal entries, and thresholds each detection’s maximum higher-scoring IoU.
  • Fast NMS: 11.8 ms faster than Cython traditional NMS, Fast NMS reduces performance by only 0.1 mAP; against CUDA NMS, it is 15.0 ms faster with a 0.3 mAP loss.
  • Auxiliary Losses: Training-only auxiliary losses can increase feature richness without test-time speed penalties.YOLACT applies a semantic-segmentation loss using a 1×1 convolution on P3, with targets derived from instance annotations.

6. Results

YOLACT delivers competitive instance-segmentation performance across COCO and Pascal while emphasizing speed, mask quality, and temporal stability. Its quality advantages are linked to larger masks and avoiding feature repooling, while image size and backbone choices expose a speed–performance trade-off.

  • Evaluation: YOLACT is evaluated on COCO train2017, val2017, and test-dev, and on Pascal 2012 SBD using standard metrics.
  • 3.8x speed over the previous fastest COCO instance-segmentation method accompanies competitive performance for YOLACT-550.
  • Model Configurations: Larger image sizes increase performance but reduce speed, while ResNet-50 and DarkNet-53 offer faster alternatives that outperform YOLACT-400.
  • Mask Quality: 138 × 138 masks formed directly from original features produce noticeably higher-quality large-object masks than Mask R-CNN and FCIS.
  • Temporal Stability: YOLACT produces more temporally stable video masks than Mask R-CNN without temporal smoothing, even though training uses static images.

7. Discussion

The discussion highlights YOLACT’s speed-oriented design choices and Fast NMS, while acknowledging lower overall performance than state-of-the-art instance-segmentation methods. Results and ablations characterize trade-offs across NMS, prototypes, backbones, and image sizes.

  • YOLACT remains much faster but falls behind state-of-the-art methods in overall instance-segmentation performance.
  • Fast NMS: Fast NMS is around 12 ms faster than standard NMS with only a slight performance decrease.
  • COCO Results: YOLACT-550 reports 29.9 mask AP and 33.0 FPS in the presented results.
  • Prototype Ablation: The prototype count is set to 32 as a compromise between performance and speed.
  • Pascal Results: Pascal 2012 SBD results show YOLACT outperforming popular reported approaches while also being significantly faster.

A.1. Box Results

YOLACT also functions as a real-time object detector: its box results are comparable to YOLOv3 at similar speeds, while its mask branch adds minimal computation.

  • Detection Comparison: YOLACT achieves similar detection results to YOLOv3 at similar speeds without several YOLOv2 and YOLOv3 improvements.
  • Mask Branch Cost: 6 ms is the total evaluation time of YOLACT’s mask branch, demonstrating minimal mask computation.

A.2. More Qualitative Results

Additional qualitative examples show that YOLACT separates adjacent instances of the same class, including overlapping elephants, zebras, and birds. Displayed masks and boxes use different resolutions and padding conventions.

  • Adjacent Instances: Adjacent same-class instances are included to provide further qualitative evidence that YOLACT is not merely performing semantic segmentation.
  • Adjacent Instances: Overlapping elephants, zebras, and birds receive clearly separated instance masks in the qualitative examples.
  • Display Convention: Masks are cropped at prototype resolution with 1px padding, whereas displayed boxes use original image resolution without padding.
Loading 1904.02689v2…