PaperlayerView exact revision on arXiv

Source-linked AI summary

YOLACT++: Better Real-time Instance Segmentation

Daniel Bolya, Chong Zhou, Fanyi Xiao, Yong Jae Lee

arXiv:1912.06218v2cs.CVcs.LGeess.IV

TL;DR

Real-time instance segmentation lacked a fast one-stage counterpart to object detectors such as SSD and YOLO. YOLACT addresses this with parallel prototype-mask and coefficient prediction, while YOLACT++ improves the framework and remains real-time with competitive results.

  • Problem

    Existing instance segmentation methods prioritized performance over speed, leaving a gap for a fast, one-stage real-time model.

  • Method

    YOLACT generates image-wide prototype masks and per-instance linear coefficients in parallel, then combines them to form instance masks.

  • Results

    YOLACT++ delivered a significant boost over the original framework while remaining real-time, and YOLACT introduced Fast NMS as a 12ms-faster replacement with negligible performance penalty.

  • Takeaways & Limitations

    Prototype and coefficient prediction provides a competitive single-stage real-time instance segmentation approach, with prototype behavior supporting performance even at k = 32.

  • Takeaways & Limitations

    YOLACT can fail to localize individual objects when many objects occupy the same scene region, producing foreground-like masks instead.

Abstract

from arXiv · show

We present a simple, fully-convolutional model for real-time (>30 fps) instance segmentation that achieves competitive results on MS COCO evaluated on a single Titan Xp, which is significantly faster than any previous state-of-the-art approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. We also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty. Finally, by incorporating deformable convolutions into the backbone network, optimizing the prediction head with better anchor scales and aspect ratios, and adding a novel fast mask re-scoring branch, our YOLACT++ model can achieve 34.1 mAP on MS COCO at 33.5 fps, which is fairly close to the state-of-the-art approaches while still running at real-time.

1 INTRODUCTION

YOLACT targets the gap between accurate but slow instance segmentation and real-time detection by avoiding explicit localization and repooling. It generates image-wide prototypes and per-instance coefficients in parallel, achieving fast, high-quality masks with competitive COCO performance.

  • Motivation: Existing state-of-the-art instance segmentation methods prioritize performance over speed, leaving few real-time counterparts to SSD and YOLO.The paper frames its goal as filling this gap with a fast, one-stage instance segmentation model.
  • Motivation: Instance segmentation is difficult to accelerate because leading methods rely on feature localization, repooling, and substantial post-processing.These operations limit both two-stage and some one-stage approaches from reaching real-time speeds.
  • Approach: YOLACT avoids explicit localization by generating non-local prototype masks over the entire image and predicting linear-combination coefficients for each instance.Final masks are formed by linearly combining the prototypes and cropping with predicted bounding boxes.
  • Approach: Shared prototypes create a distributed representation across categories, with prototypes learning behaviors such as spatial partitioning, instance localization, contours, and directional maps.This emergent prototype behavior allows a fully convolutional network to localize instance masks in a translation-variant manner.
  • Advantages: The parallel design keeps mask computation lightweight, taking approximately 5 ms and adding only marginal overhead to a one-stage detector.The authors also report high-quality masks without repooling and describe the approach as broadly applicable to modern object detectors.
  • YOLACT++: YOLACT++ extends the framework with deformable convolutions, improved anchor choices, and fast mask rescoring to improve performance while retaining real-time operation.The paper also introduces Fast NMS, reported as 12 ms faster than traditional NMS with a negligible performance penalty.

2 RELATED WORK

Prior instance segmentation systems use repooling, multiple stages, or expensive post-processing that limits speed. YOLACT instead uses lightweight prototype-based mask assembly, while prototype learning differs from conventional feature-representation codebooks.

  • Two-stage methods: Mask R-CNN and related two-stage methods generate regions of interest before classifying and segmenting them, requiring per-ROI repooling and subsequent computation.These methods do not reach 30 fps even when image size is reduced.
  • One-stage methods: One-stage methods assemble position-sensitive maps or combine semantic and directional logits, but repooling, mask voting, or other computations still limit speed.Their conceptual speed advantage therefore does not generally translate into real-time performance.
  • Alternative approaches: Semantic-segmentation-based approaches add boundary detection, clustering, CRF inference, or embeddings, introducing multiple stages or expensive clustering.These design choices restrict their viability for real-time applications.
  • Real-time instance segmentation: Some real-time systems achieve target frame rates on specific datasets, but their accuracy remains far below modern instance-segmentation baselines.The paper contrasts these systems with its goal of combining real-time speed and competitive accuracy.
  • Prototypes: Classical prototype methods represent visual features, whereas YOLACT uses learned prototypes to assemble instance masks.This distinguishes YOLACT's prototype space from textons, visual words, and related feature-representation codebooks.

3 YOLACT

YOLACT adds instance segmentation to one-stage detection without explicit feature repooling by combining image-wide prototypes with per-anchor mask coefficients. The design preserves spatial coherence while keeping mask assembly lightweight and enables emergent instance localization through shared prototype combinations.

  • Architecture: YOLACT generates image-sized prototype masks with an FCN and predicts per-anchor mask coefficients in a parallel detection head.The prototypes are instance-independent, while each coefficient vector represents an instance in prototype space.
  • Architecture: 20.7 mAP is obtained by the fc-mask model, showing that simply producing anchor masks from fully connected outputs is insufficient.The authors attribute the problem to the mismatch between fully connected outputs and spatially coherent masks.
  • Mask Assembly: A single matrix multiplication linearly combines P, an h×w×k prototype tensor, with C, an n×k coefficient matrix, followed by a sigmoid to form instance masks.Only instances surviving NMS and score thresholding are included in C.
  • Mask Assembly: The final masks are cropped to predicted boxes at evaluation and to ground-truth boxes during training, with training loss normalized by ground-truth box area.This normalization is intended to preserve small objects in the prototypes.
  • Emergent Behavior: Prototype partitions can separate overlapping instances of the same class by combining maps, such as subtracting prototype 3 from prototype 2 for two umbrellas.The network learns spatially varying prototype behavior despite using fully convolutional components; final box cropping supplies explicit position awareness.
  • Emergent Behavior: Performance does not degrade with as few as k = 32 prototypes, whereas increasing k can make coefficient prediction harder and introduce redundant edge-level variations.The authors associate larger-k errors with vanished masks or leakage from other objects and report only slight AP95 gains at higher k.

4 BACKBONE DETECTOR

The backbone detector follows RetinaNet with an emphasis on speed while retaining feature richness for prototype and coefficient prediction. The default uses ResNet-101 with FPN and 550 × 550 inputs, and the authors report it is faster and better than a matched ResNet-101 SSD variant.

  • Backbone Design: The backbone prioritizes speed and feature richness because predicting prototypes and coefficients requires strong features.Its design closely follows RetinaNet with an emphasis on speed.
  • Backbone Design: The default detector uses ResNet-101 with FPN and a 550 × 550 base image size without preserving aspect ratio.The FPN modification omits P2, adds P6 and P7, and places three anchors per feature location with aspect ratios [1, 1/2, 2].
  • Backbone Design: The detector trains both its box and mask outputs.
  • Backbone Results: The backbone performs better and faster than SSD modified to use ResNet-101 at the same image size.The comparison uses the same 550 × 550 image size.

5 OTHER IMPROVEMENTS

The paper adds Fast NMS and training-only feature-enrichment losses while preserving real-time operation. Fast NMS parallelizes duplicate suppression, trading negligible accuracy for substantial speed gains.

  • Fast NMS: Fast NMS decides whether detections are kept or discarded in parallel using standard GPU-accelerated matrix operations.It relaxes traditional suppression by allowing already-removed detections to suppress others.
  • Fast NMS: Fast NMS computes class-wise pairwise IoUs for top detections, masks the lower triangle and diagonal, then thresholds each detection’s maximum higher-scoring IoU.The resulting matrix identifies detections to keep for each class.
  • Fast NMS: 11.8 ms faster than traditional NMS while reducing performance by only 0.1 mAP in the authors’ code base.Against Mask R-CNN’s CUDA implementation, it is 15.0 ms faster with a 0.3 mAP loss.
  • Auxiliary losses: Training-only auxiliary losses increase feature richness without adding test-time speed cost.The paper applies a semantic segmentation loss using modules evaluated only during training.
  • Mask scoring: The mask scoring branch uses six convolutional layers, ReLU nonlinearities, and global pooling, adding approximately 1 ms overhead.It avoids feature concatenation and fully connected layers.

6 YOLACT++

YOLACT++ improves the base framework with fast mask re-scoring, deformable convolutions, and revised anchor configurations. These changes improve mask quality and accuracy while retaining real-time speed.

  • YOLACT++: YOLACT++ adds a fast mask re-scoring branch, deformable-convolution backbone improvements, and anchor-scale and aspect-ratio optimization.The enhancements target mask ranking, feature sampling, and detector configuration.
  • Fast Mask Re-Scoring Network: The re-scoring network predicts mask IoU from each cropped mask and uses it to rescore masks alongside classification confidence.The predicted mask IoU is multiplied by the corresponding classification confidence.
  • Fast Mask Re-Scoring Network: Adding Fast Mask Re-Scoring increases speed overhead by 1.2 ms.The branch uses the full-image cropped mask and no fully connected layers, making it faster than the compared design.
  • Deformable Convolution: +1.8 mask mAP results from adding deformable convolutions, with an 8 ms speed overhead.The authors attribute the boost partly to sampling that aligns with instances across scales, rotations, and aspect ratios.
  • Results: YOLACT++-550 with ResNet-50 has the same speed as the base model while improving its performance by 4.3 mAP.Compared with Mask R-CNN, YOLACT++-R-50 is 3.9x faster and falls behind by only 1.6 mAP.
  • Anchor choices: Using multi-scale anchors per FPN level provides the best speed-versus-performance trade-off among the tested anchor configurations.The tested variations change anchor scales or aspect ratios relative to the original configuration.

7 RESULTS

The paper evaluates instance segmentation on MS COCO and Pascal 2012 SBD, with timings measured on a single Titan Xp. Results show competitive accuracy at high speed, including strong YOLACT++ performance.

  • Evaluation: The evaluation covers MS COCO instance segmentation and box detection plus Pascal 2012 SBD instance segmentation using standard metrics.COCO models are trained on train2017 and evaluated on val2017 and test-dev.
  • Implementation Details: All models are trained with batch size 8 on one GPU using ImageNet-pretrained weights.COCO training uses SGD for 800k iterations and takes 4–6 days on one Titan Xp.
  • Evaluation: All reported speeds are measured on a single Titan Xp without test-time augmentations.The authors note that some listed speeds may exceed the original papers’ reported speeds.
  • COCO Results: YOLACT-550 is 3.8x faster than the previous fastest COCO instance segmentation method while retaining competitive performance.At the 95% IoU threshold, it achieves 1.6 AP versus Mask R-CNN’s 1.3 AP.
  • COCO Results: 29.8 mAP at 33.0 fps is achieved by the base model on COCO test-dev.The figure uses a confidence threshold of 0.3 for all images.
  • Model Configurations: Increasing image size raises performance but significantly decreases speed, whereas ResNet-50 or DarkNet-53 offers faster alternatives to lowering image size.The authors report that these backbones perform much better than YOLACT-400 while being only slightly slower.
  • YOLACT++ Results: YOLACT++-ResNet-50 runs at 33.5 fps, is 3.9x faster than Mask R-CNN, and falls behind by only 1.6 mAP.The proposed enhancements improve YOLACT by 5.9 mAP for ResNet-50 and 4.8 mAP for ResNet-101.
  • Pascal Results: YOLACT clearly outperforms popular approaches on Pascal 2012 SBD while also being significantly faster.The Pascal and COCO FPS values are not directly comparable because Pascal has fewer classes.

7.3 Mask Quality

The paper attributes YOLACT’s mask quality to large, full-image masks without feature repooling. It also reports temporal stability on videos despite training only on static images.

  • Mask Quality: YOLACT produces higher-quality masks for large objects than Mask R-CNN and FCIS because its masks are larger and avoid repooling.The paper illustrates cleaner boundary following and less noise around an arm.
  • Mask Quality: The final mask size is 138 × 138, and masks are constructed directly from original features without repooling.These design choices preserve full-image information and avoid potentially misaligned feature transformations.
  • Temporal Stability: YOLACT produces more temporally stable video masks than Mask R-CNN despite training only on static images and using no temporal smoothing.The paper attributes this partly to higher mask quality and mostly to its one-stage design.

7.5 More Qualitative Results

Qualitative and component analyses show that YOLACT separates adjacent same-class instances while YOLACT++ improves ranking, detection quality, and speed through targeted additions.

  • Qualitative results: YOLACT clearly separates overlapping adjacent instances, including elephants, zebras, and birds of the same class.
  • Box results: YOLACT achieves similar detection results to YOLOv3 at similar speeds without several YOLOv2/YOLOv3 improvements.
  • Speed: 6 ms is the total evaluation time of YOLACT’s mask branch, demonstrating minimal mask-computation overhead.
  • YOLACT++ components: Optimized anchors improve box recall, while deformable convolutions improve feature sampling across scale, rotation, and aspect-ratio changes.
  • YOLACT++ components: Deformable-convolution overhead falls from 8 ms to 2.8 ms with only a 0.2 mAP drop under the reduced-layer configuration.
  • YOLACT++ components: Mask re-scoring ranks detections using IoU-based mask scores, elevating high-quality masks that classification confidence would otherwise under-rank.

7.8 Timing Breakdown

Timing is measured with GPU parallelization disabled, producing inflated component times while highlighting the model’s reliance on parallel execution.

  • Disabling GPU parallelization makes each component’s measured time much higher than in the original model.
  • The model runs three times faster with parallelism enabled, demonstrating effective exploitation of parallel execution.

8 DISCUSSION

YOLACT provides high-quality, temporally stable masks but remains behind state-of-the-art overall performance and has localization and leakage failure modes tied to crowded scenes and inaccurate boxes.

  • YOLACT remains somewhat below state-of-the-art instance-segmentation performance despite higher-quality, temporally stable masks, while running much faster.
  • Localization Failure: Crowded scenes can prevent the network from localizing each object in its own prototype, producing foreground-like masks.
  • Localization Failure: YOLACT++ partially mitigates localization failures with more scale-covering anchors and deformable convolutions for improved feature sampling.
  • Leakage: Because noise outside assembled masks is not suppressed, inaccurate or oversized boxes can cause leakage from nearby or distant instances.
  • Mitigation: Mask re-scoring ranks detections by mask quality, placing higher-quality masks above detections with misleading classification confidence.
  • Understanding the AP Gap: Replacing predicted masks with ground truth improves mask mAP from 33.7 to 35.1, indicating detector performance contributes substantially to the AP gap.

9 CONCLUSION

The paper concludes that parallel prototype and coefficient prediction enables competitive real-time instance segmentation, while YOLACT++ improves the framework without sacrificing real-time operation.

  • YOLACT predicts prototypes and per-instance coefficients in parallel, then linearly combines them to form final instance masks.
  • Experiments on MS COCO and Pascal VOC demonstrate the effectiveness of the approach and its individual components.
  • YOLACT++ significantly improves the original framework through backbone, anchor, and mask re-scoring enhancements while remaining real-time.
Loading 1912.06218v2…