Source-linked AI summary
Conditional Convolutions for Instance Segmentation
Zhi Tian, Chunhua Shen, Hao Chen
TL;DR
Instance segmentation methods commonly rely on ROI operations and fixed mask-head weights, while fully convolutional approaches struggle to distinguish similar-looking instances. CondInst instead generates instance-conditioned mask-head filters dynamically, producing a fully convolutional and compact design that the authors report as more accurate and faster than Mask R-CNN without longer training schedules.
Problem
ROI-based instance segmentation uses cropping and fixed mask-head filters, while standard FCNs struggle to produce different predictions for visually similar instances.
Method
CondInst uses a controller to dynamically generate mask-head parameters conditioned on each instance, eliminating ROI operations with a compact fully convolutional mask head.
Results
CondInst outperforms Mask R-CNN in both accuracy and speed without requiring longer training schedules.
Takeaways & Limitations
Instance segmentation can be addressed with an instance-aware fully convolutional framework that is simpler and faster than ROI-based alternatives.
Abstract
from arXiv · showhide
We propose a simple yet effective instance segmentation framework, termed CondInst (conditional convolutions for instance segmentation). Top-performing instance segmentation methods such as Mask R-CNN rely on ROI operations (typically ROIPool or ROIAlign) to obtain the final instance masks. In contrast, we propose to solve instance segmentation from a new perspective. Instead of using instance-wise ROIs as inputs to a network of fixed weights, we employ dynamic instance-aware networks, conditioned on instances. CondInst enjoys two advantages: 1) Instance segmentation is solved by a fully convolutional network, eliminating the need for ROI cropping and feature alignment. 2) Due to the much improved capacity of dynamically-generated conditional convolutions, the mask head can be very compact (e.g., 3 conv. layers, each having only 8 channels), leading to significantly faster inference. We demonstrate a simpler instance segmentation method that can achieve improved performance in both accuracy and inference speed. On the COCO dataset, we outperform a few recent methods including well-tuned Mask RCNN baselines, without longer training schedules needed. Code is available: https://github.com/aim-uofa/adet
1 Introduction
CondInst addresses limitations of ROI-based instance segmentation by using instance-aware fully convolutional mask heads with dynamically generated filters. This design avoids ROI operations, reduces mask-head complexity, and is reported to improve both accuracy and inference speed.
- Motivation: ROI-based methods can include irrelevant content for irregular objects and require large-receptive-field mask heads, increasing computational complexity.Mask R-CNN, for example, uses four 3×3 convolutions with 256 channels in its mask head.
- Motivation: FCNs struggle when visually similar instances require different foreground and background predictions.For example, predicting one person while treating another similar-looking person as background is difficult without instance-specific information.
- Approach: CondInst dynamically generates mask-head parameters conditioned on each target instance instead of using fixed filters for all instances.A controller sub-network generates the parameters of an instance-specific mask FCN.
- Approach: CondInst is fully convolutional and avoids ROI resizing operations, enabling high-resolution masks with more accurate edges.The framework eliminates ROI operations used in many existing methods.
- Efficiency: The dynamically generated filters allow a compact mask head and substantially reduce per-instance inference cost.The method reports only ∼10% more computational time than the FCOS bounding-box detector while processing 100 instances.
- Results: CondInst achieves state-of-the-art performance while being faster in inference and can be applied to panoptic segmentation.The authors present it as a strong alternative to Mask R-CNN without requiring longer training schedules.
2 Instance Segmentation with CondInst
CondInst represents each instance with dynamically generated mask-head filters, enabling fully convolutional mask prediction without ROI operations. Its compact instance-aware heads use detector features and relative coordinates to produce high-resolution masks efficiently.
- Network Outputs and Training Targets: At each feature-map location, classification predicts the associated instance class and the controller predicts the parameters θx,y of its mask head.Center-ness and box heads operate in parallel with the controller, following FCOS.
- Overall Architecture: For an image with K instances, CondInst dynamically generates K mask heads whose filters encode the characteristics of their target instances.Each instance-specific head is applied to shared mask-branch features to predict only that instance’s mask.
- Overall Architecture: CondInst avoids ROI cropping and feature alignment by using instance-aware filters to represent instances instead of bounding-box-based ROIs.This representation can handle irregular shapes that are difficult to tightly enclose with bounding boxes.
- Overall Architecture: The detector uses FCOS with FPN feature maps P3 through P7, while a mask branch supplies the features consumed by generated mask heads.The mask branch is connected to P3 and produces features at 1/8 of the input resolution.
- Mask Branch: The mask branch reduces its output to 8 channels, and bilinear upsampling by 4 produces masks at 400 × 512 resolution for an 800 × 1024 input.This output is higher resolution than the 28 × 28 masks used by Mask R-CNN.
- Network Outputs and Training Targets: Locations are assigned to instances when their mapped image positions fall within an instance’s center region, while locations outside center regions are negative samples.The center region uses radius r = 1.5 times the feature-map down-sampling ratio, following FCOS.
- Loss Function: Training combines the FCOS loss and mask loss with λ = 1, using Dice loss for positive locations and dynamically parameterized mask heads.Relative coordinates from mask features to the filter-generation location are concatenated with Fmask before mask prediction.
- Inference: The mask head has three 1 × 1 convolutions with 8 channels and 169 parameters, adding less than 5 milliseconds for 100 detections and about 10% computation over FCOS.After box-based NMS, the generated filters are applied as K instance-specific mask heads to the combined mask features and relative coordinates.
3 Experiments
Experiments on MS-COCO examine implementation choices, mask-head design, mask upsampling, NMS, and comparisons with prior methods. CondInst combines compact mask prediction with competitive accuracy and inference speed.
- Implementation Details: Experiments use MS-COCO train2017, val2017, and test-dev splits, with ResNet-50 initialized from ImageNet-pretrained weights.Main results use test-dev; ablations use val2017.
- Mask Head Design: A three-layer mask head with 8 channels achieves 35.7% mask AP, while depths of 2–4 perform similarly and depth 1 is inferior.The depth-1 head is a linear mapping with overly weak capacity.
- Mask Branch: Varying the mask-branch output channels from 2 to 16 produces nearly unchanged performance, with Cmask = 8 selected by default.This indicates robustness to the mask-branch width within a reasonable range.
- Mask Head Inputs: Removing relative coordinates lowers mask AP from 35.7% to 31.4%, showing that instance shape and relative position contribute to mask prediction.The generated filters encode more than appearance cues.
- Mask Upsampling: Upsampling by factor 2 improves mask AP from 34.4% to 35.8%, especially for small objects from 15.1% to 17.0%; factor 4 gives 35.7%.Without upsampling, masks have 1/8 of the input resolution and lose boundary details.
- Comparisons: CondInst improves over original Mask R-CNN from 34.6% to 35.4% mask AP while reducing latency from 65ms to 49ms per image.Against Detectron2 Mask R-CNN, it reaches 35.9% versus 35.5% at 49ms per image for both.
- Comparisons: CondInst surpasses TensorMask and YOLACT-700, reaching 40.1% versus 31.2% mask AP against YOLACT-700 with the same ResNet-101 backbone.It is also reported as approximately 8× faster than TensorMask in one comparison.
4 Conclusions
CondInst is presented as a simpler instance segmentation framework that conditions its mask head on instances and dynamically generates its filters. It eliminates ROI operations, reduces mask-head complexity, and outperforms Mask R-CNN in accuracy and speed without longer training schedules.
- CondInst conditions the mask head on instances and dynamically generates its filters, unlike Mask R-CNN’s fixed-weight mask head.This design reduces the mask head’s parameters and computational complexity.
- CondInst eliminates ROI operations, resulting in a faster and simpler instance segmentation framework.
- CondInst outperforms Mask R-CNN in both accuracy and speed without requiring longer training schedules.