Source-linked AI summary
BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation
Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, Youliang Yan
TL;DR
Fully convolutional instance segmentation methods have lagged behind Mask R-CNN in mask precision at similar computational cost. BlendMask combines high-level instance and low-level semantic information through a blender module, outperforming Mask R-CNN while being 20% faster and achieving 34.2% mAP at 25 FPS in its real-time version.
Problem
Dense instance segmentation in one-stage frameworks remains desirable, but independent heads struggle to share features with related tasks such as semantic segmentation.
Method
BlendMask uses a blender module that combines high-level instance information with low-level semantic features and predicts instance attention maps with a single convolution layer.
Results
34.2% mAP at 25 FPS is achieved by BlendMask-RT, while BlendMask outperforms Mask R-CNN and is 20% faster.
Takeaways & Limitations
BlendMask provides an efficient, easily integrated alternative to Mask R-CNN for many instance-level recognition tasks.
Takeaways & Limitations
Score-map representations become ambiguous when two instances share center positions or other relative positions, making it impossible to identify the described instance from the crop.
Abstract
from arXiv · showhide
Instance segmentation is one of the fundamental vision tasks. Recently, fully convolutional instance segmentation methods have drawn much attention as they are often simpler and more efficient than two-stage approaches like Mask R-CNN. To date, almost all such approaches fall behind the two-stage Mask R-CNN method in mask precision when models have similar computation complexity, leaving great room for improvement. In this work, we achieve improved mask prediction by effectively combining instance-level information with semantic information with lower-level fine-granularity. Our main contribution is a blender module which draws inspiration from both top-down and bottom-up instance segmentation approaches. The proposed BlendMask can effectively predict dense per-pixel position-sensitive instance features with very few channels, and learn attention maps for each instance with merely one convolution layer, thus being fast in inference. BlendMask can be easily incorporated with the state-of-the-art one-stage detection frameworks and outperforms Mask R-CNN under the same training schedule while being 20% faster. A light-weight version of BlendMask achieves $ 34.2% $ mAP at 25 FPS evaluated on a single 1080Ti GPU card. Because of its simplicity and efficacy, we hope that our BlendMask could serve as a simple yet strong baseline for a wide range of instance-wise prediction tasks. Code is available at https://git.io/AdelaiDet
1. Introduction
BlendMask addresses the limitations of fully convolutional instance segmentation by combining instance-level and fine-grained pixel information in a unified one-stage framework. Its blender improves mask accuracy and inference efficiency relative to established approaches while supporting broader dense prediction capabilities.
- Independent heads struggle to share features with related tasks such as semantic segmentation, complicating network architecture optimization.
- One-stage dense instance segmentation is desirable because conventional operations simplify cross-platform deployment and unified frameworks facilitate multi-task architecture optimization.
- Related approaches: Top-down methods lose local coherence and use redundant mask representations, whereas bottom-up methods retain pixel-level information but rely heavily on dense prediction.
- Contributions: BlendMask’s blender combines rich instance-level information with accurate dense pixel features, surpassing YOLACT by 1.9 points and FCIS by 1.3 points in mAP on COCO.
- Results: 37.0% mAP with ResNet-50 and 38.4% mAP with ResNet-101 on COCO outperform Mask R-CNN while being about 20% faster.
- Capabilities: BlendMask can perform panoptic segmentation without modification, while its bottom module outputs masks at much higher resolution than Mask R-CNN’s typical 28 × 28 mask head.
2. Related work
Related work contrasts two-stage detect-then-segment methods, position-sensitive bottom-up representations, and top-down coefficient-based approaches. BlendMask balances top- and bottom-level instance information to address their limitations while preserving efficiency.
- Anchor-free object detection: 2.7% higher box AP was reported for an anchor-free detector than anchor-based RetinaNet, simplifying the detection pipeline.Anchor-free detection removes predefined anchor-shape restrictions and freely matches targets to prediction features.
- Detect-then-segment instance segmentation: Two-stage instance segmentation detects objects before predicting proposal masks, with RoIAlign providing local coherence absent from one-stage top-down approaches.The passage identifies two issues in two-stage frameworks but truncates their full description.
- Detect-then-segment instance segmentation: FCIS and related methods predict shared position-sensitive score maps, but their representation becomes ambiguous when instances share center or relative positions.Higher-resolution location crops improve accuracy but increase computation quadratically.
- Detect-then-segment instance segmentation: YOLACT predicts mask coefficients with box predictions to linearly combine cropped mask bases, assigning instance-level information to the top level.The paper argues that scalar coefficients are suboptimal for encoding instance information.
- Refining coarse masks with lower-level features: BlendMask balances top- and bottom-level representation workloads through a blender module, improving bases-combination methods over YOLACT and FCIS without increasing computation complexity.Its design merges top-level coarse instance information with lower-level fine-granularity and avoids the subnet required by MaskLab and IMP.
3. Our BlendMask
BlendMask combines bottom-level position-sensitive bases with top-level instance attentions through a blender module. Its learned representations encode semantic and object-part information, helping separate overlapping instances efficiently.
- Architecture: BlendMask uses a detector network and mask branch containing bottom score-map bases, top instance attentions, and a blender that merges them.The blender is the key component that combines position-sensitive bases according to attentions for final prediction.
- Bottom module: The bottom module predicts K bases with shape N × K × H_s × W_s from backbone or feature-pyramid inputs.DeepLabV3+ is used as the decoder in the experiments, while other dense prediction modules are also compatible.
- Top layer: The top layer predicts attention tensors of shape N × (K · M · M) × H_l × W_l, whose 3D structure encodes coarse instance shape and pose.It uses one convolution layer on each detection tower, unlike YOLACT’s mask coefficients.
- Blender module: The blender crops bases using bounding-box proposals, resizes them to R × R, interpolates attentions to R × R, normalizes them across K, and sums weighted products into mask logits.Inputs are bottom-level bases, selected top-level attentions, and proposals; training uses ground-truth boxes and inference uses FCOS predictions.
- Learned representations: BlendMask bases encode both object occupancy and position-sensitive object-part information.Observed bases detect upper-right or bottom-left object parts, general object pixels, or object borders.
- Learned representations: Position-sensitive features help separate overlapping instances, enabling BlendMask to represent all instances more efficiently than YOLACT.This interpretation is based on the observed learned bases and attentions projected onto the original image.
4. Experiments
Experiments on MSCOCO show that BlendMask’s blender design improves mask quality through fine-grained, instance-aware features while preserving efficiency. Ablations identify effective resolutions, bases, FPN inputs, aligned sampling, and auxiliary improvements, while comparisons show advantages over competing methods and Mask R-CNN.
- Experimental Setup: Experiments use MSCOCO 2017, training on train2017 and evaluating ablations on val2017 and final results on test-dev with COCO mask AP metrics.The dataset contains 123K images with 80-class instance labels; metrics include AP, AP50, AP75, APS, APM, and APL.
- Ablation Studies: The blender surpasses alternative variations by a large margin, attributed to its fine-grained top-level attention map providing instance-aware guidance.The comparison is reported in Table 1 and illustrated in Figure 3.
- Ablation Studies: 1 point of mask AP is gained by increasing bottom pooling resolution from 28 to 56, while inference time rises only within 0.2ms.The baseline subsequently uses R = 56 and M = 7; higher attention resolutions add detail with roughly unchanged runtime, but gains slow at high resolutions.
- Ablation Studies: 4 bases achieve optimal accuracy because instance-level information is better represented by top-level attentions, and FPN sampling improves performance while reducing runtime.Subsequent experiments adopt K = 4 and use P3 and P5 as bottom-module inputs.
- Ablation Studies: Almost 2AP is gained from aligned bilinear sampling over nearest sampling, while semantic loss, wider bottom features, and stride reduction further improve detection and segmentation.The additional changes increase bottom-module width from 128 to 256 and reduce bases output stride from 8 to 4 using P2 and P5 inputs; YOLACT protonet variants are also compared.
5. Conclusion
BlendMask introduces an efficient, easily integrated blender module that combines high-level instance and low-level semantic information for dense prediction. It outperforms Mask R-CNN while being 20% faster, and BlendMask-RT reaches 34.2% mAP at 25 FPS on a single 1080Ti GPU.
- Conclusion: The blender module combines high-level instance and low-level semantic information for efficient dense prediction and integrates easily with mainstream detection networks.It is designed for instance-level dense prediction tasks.
- Conclusion: 20% faster: BlendMask outperforms Mask R-CNN without bells and whistles.The comparison is made using the paper’s stated framework and baseline.
- Conclusion: 34.2% mAP at 25 FPS: BlendMask-RT achieves this result on a single 1080Ti GPU card.This is the framework’s real-time version.