PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Convolutions for Instance Segmentation

Zhi Tian, Chunhua Shen, Hao Chen

arXiv:2003.05664v4cs.CV

TL;DR

Instance segmentation methods commonly rely on ROI operations and fixed mask-head weights, while fully convolutional approaches struggle to distinguish similar-looking instances. CondInst instead generates instance-conditioned mask-head filters dynamically, producing a fully convolutional and compact design that the authors report as more accurate and faster than Mask R-CNN without longer training schedules.

  • Problem

    ROI-based instance segmentation uses cropping and fixed mask-head filters, while standard FCNs struggle to produce different predictions for visually similar instances.

    Source: §1 · pp. 2–3

  • Method

    CondInst uses a controller to dynamically generate mask-head parameters conditioned on each instance, eliminating ROI operations with a compact fully convolutional mask head.

    Source: §1 · p. 3, §2.2 · p. 8

  • Results

    CondInst outperforms Mask R-CNN in both accuracy and speed without requiring longer training schedules.

    Source: §4 · p. 16

  • Takeaways & Limitations

    Instance segmentation can be addressed with an instance-aware fully convolutional framework that is simpler and faster than ROI-based alternatives.

    Source: §4 · pp. 15–16

Abstract

from arXiv · show

We propose a simple yet effective instance segmentation framework, termed CondInst (conditional convolutions for instance segmentation). Top-performing instance segmentation methods such as Mask R-CNN rely on ROI operations (typically ROIPool or ROIAlign) to obtain the final instance masks. In contrast, we propose to solve instance segmentation from a new perspective. Instead of using instance-wise ROIs as inputs to a network of fixed weights, we employ dynamic instance-aware networks, conditioned on instances. CondInst enjoys two advantages: 1) Instance segmentation is solved by a fully convolutional network, eliminating the need for ROI cropping and feature alignment. 2) Due to the much improved capacity of dynamically-generated conditional convolutions, the mask head can be very compact (e.g., 3 conv. layers, each having only 8 channels), leading to significantly faster inference. We demonstrate a simpler instance segmentation method that can achieve improved performance in both accuracy and inference speed. On the COCO dataset, we outperform a few recent methods including well-tuned Mask RCNN baselines, without longer training schedules needed. Code is available: https://github.com/aim-uofa/adet

1 Introduction

CondInst addresses limitations of ROI-based instance segmentation by using instance-aware fully convolutional mask heads with dynamically generated filters. This design avoids ROI operations, reduces mask-head complexity, and is reported to improve both accuracy and inference speed.

Motivation

  • ROI-based methods can include irrelevant content for irregular objects and require large-receptive-field mask heads, increasing computational complexity.Mask R-CNN, for example, uses four 3×3 convolutions with 256 channels in its mask head.Source: §1 · p. 2
  • FCNs struggle when visually similar instances require different foreground and background predictions.For example, predicting one person while treating another similar-looking person as background is difficult without instance-specific information.Source: §1 · p. 3

Approach

  • CondInst dynamically generates mask-head parameters conditioned on each target instance instead of using fixed filters for all instances.A controller sub-network generates the parameters of an instance-specific mask FCN.Source: §1 · p. 3
  • CondInst is fully convolutional and avoids ROI resizing operations, enabling high-resolution masks with more accurate edges.The framework eliminates ROI operations used in many existing methods.Source: §1 · pp. 3–4

Efficiency

  • The dynamically generated filters allow a compact mask head and substantially reduce per-instance inference cost.The method reports only ∼10% more computational time than the FCOS bounding-box detector while processing 100 instances.Source: §1 · p. 4

Results

  • CondInst achieves state-of-the-art performance while being faster in inference and can be applied to panoptic segmentation.The authors present it as a strong alternative to Mask R-CNN without requiring longer training schedules.Source: §1 · p. 4, §1.1 · p. 5

2 Instance Segmentation with CondInst

CondInst represents each instance with dynamically generated mask-head filters, enabling fully convolutional mask prediction without ROI operations. Its compact instance-aware heads use detector features and relative coordinates to produce high-resolution masks efficiently.

Network Outputs and Training Targets

  • At each feature-map location, classification predicts the associated instance class and the controller predicts the parameters θx,y of its mask head.Center-ness and box heads operate in parallel with the controller, following FCOS.Source: Figure 3 · p. 6, §2.2 · p. 8

Overall Architecture

  • For an image with K instances, CondInst dynamically generates K mask heads whose filters encode the characteristics of their target instances.Each instance-specific head is applied to shared mask-branch features to predict only that instance’s mask.Source: §2.1 · p. 6
  • CondInst avoids ROI cropping and feature alignment by using instance-aware filters to represent instances instead of bounding-box-based ROIs.This representation can handle irregular shapes that are difficult to tightly enclose with bounding boxes.Source: §2.1 · pp. 6–7, §2.2 · pp. 8–9
  • The detector uses FCOS with FPN feature maps P3 through P7, while a mask branch supplies the features consumed by generated mask heads.The mask branch is connected to P3 and produces features at 1/8 of the input resolution.Source: §2.1 · p. 7

Mask Branch

  • The mask branch reduces its output to 8 channels, and bilinear upsampling by 4 produces masks at 400 × 512 resolution for an 800 × 1024 input.This output is higher resolution than the 28 × 28 masks used by Mask R-CNN.Source: §2.1 · p. 7

Network Outputs and Training Targets

  • Locations are assigned to instances when their mapped image positions fall within an instance’s center region, while locations outside center regions are negative samples.The center region uses radius r = 1.5 times the feature-map down-sampling ratio, following FCOS.Source: §2.2 · p. 8

Loss Function

  • Training combines the FCOS loss and mask loss with λ = 1, using Dice loss for positive locations and dynamically parameterized mask heads.Relative coordinates from mask features to the filter-generation location are concatenated with Fmask before mask prediction.Source: §2.3 · p. 9

Inference

  • The mask head has three 1 × 1 convolutions with 8 channels and 169 parameters, adding less than 5 milliseconds for 100 detections and about 10% computation over FCOS.After box-based NMS, the generated filters are applied as K instance-specific mask heads to the combined mask features and relative coordinates.Source: §2.4 · p. 10

3 Experiments

Experiments on MS-COCO examine implementation choices, mask-head design, mask upsampling, NMS, and comparisons with prior methods. CondInst combines compact mask prediction with competitive accuracy and inference speed.

Implementation Details

  • Experiments use MS-COCO train2017, val2017, and test-dev splits, with ResNet-50 initialized from ImageNet-pretrained weights.Main results use test-dev; ablations use val2017.Source: §3 · p. 10, §3.1 · p. 10

Mask Head Design

  • A three-layer mask head with 8 channels achieves 35.7% mask AP, while depths of 2–4 perform similarly and depth 1 is inferior.The depth-1 head is a linear mapping with overly weak capacity.Source: §3.2 · pp. 11–12

Mask Branch

  • Varying the mask-branch output channels from 2 to 16 produces nearly unchanged performance, with Cmask = 8 selected by default.This indicates robustness to the mask-branch width within a reasonable range.Source: Table 2 · p. 11, §3.3 · p. 12

Mask Head Inputs

  • Removing relative coordinates lowers mask AP from 35.7% to 31.4%, showing that instance shape and relative position contribute to mask prediction.The generated filters encode more than appearance cues.Source: Table 3 · p. 11, §3.3 · p. 12

Mask Upsampling

  • Upsampling by factor 2 improves mask AP from 34.4% to 35.8%, especially for small objects from 15.1% to 17.0%; factor 4 gives 35.7%.Without upsampling, masks have 1/8 of the input resolution and lose boundary details.Source: §3.4 · p. 13, Table 4 · p. 13

Comparisons

  • CondInst improves over original Mask R-CNN from 34.6% to 35.4% mask AP while reducing latency from 65ms to 49ms per image.Against Detectron2 Mask R-CNN, it reaches 35.9% versus 35.5% at 49ms per image for both.Source: §3.6 · p. 13, Table 6 · p. 14
  • CondInst surpasses TensorMask and YOLACT-700, reaching 40.1% versus 31.2% mask AP against YOLACT-700 with the same ResNet-101 backbone.It is also reported as approximately 8× faster than TensorMask in one comparison.Source: §3.6 · pp. 14–15

4 Conclusions

CondInst is presented as a simpler instance segmentation framework that conditions its mask head on instances and dynamically generates its filters. It eliminates ROI operations, reduces mask-head complexity, and outperforms Mask R-CNN in accuracy and speed without longer training schedules.

  • CondInst conditions the mask head on instances and dynamically generates its filters, unlike Mask R-CNN’s fixed-weight mask head.This design reduces the mask head’s parameters and computational complexity.Source: §4 · p. 15
  • CondInst eliminates ROI operations, resulting in a faster and simpler instance segmentation framework.Source: §4 · p. 15
  • CondInst outperforms Mask R-CNN in both accuracy and speed without requiring longer training schedules.Source: §4 · p. 16

Related papers

Loading 2003.05664v4…