PaperlayerView exact revision on arXiv

Source-linked AI summary

RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Features

Gang Zhang, Xin Lu, Jingru Tan, Jianmin Li, Zhaoxiang Zhang, Quanquan Li, Xiaolin Hu

arXiv:2104.08569v1cs.CV

TL;DR

Two-stage instance segmentation produces coarse masks because downsampling loses detail, especially for large objects. RefineMask incorporates fine-grained features through multi-stage mask refinement and focuses later refinement on boundaries. It improves over Mask R-CNN across three validation benchmarks and surpasses the LVIS Challenge 2020 winner on test-dev.

  • Problem

    Two-stage instance segmentation loses detail through feature-pyramid downsampling and instance-wise pooling, leaving masks coarse, especially for large objects.

    Source: §1 · p. 1

  • Method

    RefineMask iteratively incorporates fine-grained semantic features during multi-stage instance-mask refinement and focuses later stages on boundaries.

    Source: §1 · p. 2

  • Results

    RefineMask outperformed Mask R-CNN by 2.6, 3.4, and 3.8 points on COCO, LVIS, and Cityscapes validation sets, respectively, and surpassed the LVIS Challenge 2020 winner by 1.3 points on test-dev.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RefineMask consistently refines higher-quality masks, including accurate boundaries in hard regions such as object boundaries.

    Source: §1 · p. 2

Abstract

from arXiv · show

The two-stage methods for instance segmentation, e.g. Mask R-CNN, have achieved excellent performance recently. However, the segmented masks are still very coarse due to the downsampling operations in both the feature pyramid and the instance-wise pooling process, especially for large objects. In this work, we propose a new method called RefineMask for high-quality instance segmentation of objects and scenes, which incorporates fine-grained features during the instance-wise segmenting process in a multi-stage manner. Through fusing more detailed information stage by stage, RefineMask is able to refine high-quality masks consistently. RefineMask succeeds in segmenting hard cases such as bent parts of objects that are over-smoothed by most previous methods and outputs accurate boundaries. Without bells and whistles, RefineMask yields significant gains of 2.6, 3.4, 3.8 AP over Mask R-CNN on COCO, LVIS, and Cityscapes benchmarks respectively at a small amount of additional computational cost. Furthermore, our single-model result outperforms the winner of the LVIS Challenge 2020 by 1.3 points on the LVIS test-dev set and establishes a new state-of-the-art. Code will be available at https://github.com/zhanggang001/RefineMask.

1. Introduction

Two-stage instance segmentation loses detail through feature-pyramid downsampling and instance-wise pooling, limiting mask quality, especially at boundaries. RefineMask adds fine-grained features through multi-stage refinement and improves results across benchmarks.

1. Introduction

  • Instance segmentation assigns pixels to semantic categories while distinguishing instances within each category.Recent top-performing methods generally use a two-stage paradigm, exemplified by Mask R-CNN.Source: §1 · p. 1
  • Two-stage methods lose image details through coarse feature-pyramid maps and instance-wise pooling, which reduces features to sizes such as 7×7 or 14×14.The lost details are important for high-quality masks, particularly when mapping predictions back to input space.Source: §1 · pp. 1–2
  • Semantic segmentation can use high-resolution features and avoids instance-wise pooling, achieving higher boundary-region accuracy than two-stage instance segmentation methods.Unlike instance segmentation, it does not need to distinguish object instances.Source: §1 · p. 2
  • RefineMask supplements instance features with fine-grained semantic features and progressively upsamples mask predictions, while focusing later refinement on boundaries.It retains two-stage instance distinction and produces higher-quality masks than Mask R-CNN, with details comparable to state-of-the-art semantic segmentation in hard regions such as boundaries.Source: §1 · p. 2
  • RefineMask outperformed Mask R-CNN by 2.6, 3.4, and 3.8 points on COCO, LVIS, and Cityscapes validation sets, respectively, and surpassed the LVIS Challenge 2020 winner by 1.3 points on test-dev.The single-model LVIS test-dev result established a new state-of-the-art.Source: §1 · p. 2

2. Related Work

Prior instance segmentation methods lose detail when pooling canonical-size features, while related refinement and semantic-feature approaches address different parts of the problem. RefineMask combines iterative fine-grained feature fusion with explicit boundary-region refinement.

2. Related Work

  • Instance segmentation methods pool canonical-size features from the feature pyramid, losing details needed for high-quality masks, especially on large objects; RefineMask adds fine-grained information stage by stage.This distinguishes its feature supplementation during refinement from adding semantic features before instance-wise pooling.Source: §2 · p. 2
  • RefineMask refines entire objects together, unlike PointRend's iterative point predictions at blurred areas; SharpMask instead focuses on object proposal generation.HTC uses an intertwined cascade mask head but is described as having too much computational cost.Source: §2 · p. 2
  • Each Semantic Fusion Module combines preceding-stage instance features with pooled semantic features, guided by the instance and semantic masks.A semantic head attached to P2 supplies fine-grained features to the multi-stage mask head.Source: Figure 2 · p. 3
  • RefineMask explicitly focuses on instance boundaries in later refinement stages, unlike prior boundary methods using polygon deformation, interior-pixel replacement, or contour supervision.The framework caption identifies this strategy as boundary-aware refinement (BAR).Source: §2 · pp. 2–3, Figure 2 · p. 3

3. RefineMask

RefineMask combines a high-resolution semantic head with a multi-stage instance mask head that fuses fine-grained information and focuses later refinement on object boundaries.

3. RefineMask

  • RefineMask uses a semantic head and mask head alongside FPN to perform high-quality instance segmentation.The semantic head supplies fine-grained features; the mask head incorporates them through multiple stages and explicitly focuses on boundary regions.Source: §3 · p. 3

3.1. Semantic Head

  • The semantic head processes FPN’s highest-resolution P2 map without downsampling and defines fine-grained features as semantic features plus a high-resolution semantic mask.A fully convolutional network with four convolutional layers and a binary classifier predicts the image-wide foreground mask under binary cross-entropy supervision.Source: §3.1 · p. 3

3.2. Mask Head

  • The 14×14 initial mask is iteratively refined with instance and pooled semantic features and masks to 112×112, while channel compression limits added computation.The Semantic Fusion Module concatenates four inputs, compresses them with a 1×1 convolution, applies parallel 3×3 convolutions with different dilations, and uses the masks to guide prediction.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Boundary-Aware Refinement

  • Later-stage supervision targets boundary regions defined by distance to the mask contour, using regions from both ground-truth and preceding predicted masks.The first stage predicts a complete 28×28 mask; the next stages output 56×56 and 112×112 masks, with supervision limited to selected boundary regions.Source: §3.3 · p. 4
  • At inference, the first stage produces a complete 28×28 mask and boundary mask; subsequent stages refine the mask until it reaches its finest resolution.The mask update combines boundary-focused predictions with the preceding mask, and the process repeats through later stages.Source: §3.3 · p. 5, Figure 5 · p. 5

3.4. Implementation Details

  • The implementation replaces Mask R-CNN’s default mask head with three refinement stages and keeps other hyperparameters and detection losses largely unchanged.Loss weights for the initial mask and three refinement stages are 0.25, 0.5, 0.75, and 1.0; the boundary threshold is 2 for training and 1 for inference.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate RefineMask on COCO, LVIS, and Cityscapes using mask AP; AP⋆ scores COCO predictions against LVIS annotations. Across benchmarks, RefineMask improves mask results over Mask R-CNN, with ablations examining stages, feature fusion, and boundary refinement.

4.1. Main Results

  • RefineMask gained 2.6 points AP over Mask R-CNN on COCO and 3.4 points AP on LVIS validation, then beat the LVIS Challenge 2020 winner by 1.3 points on test-dev.The LVIS validation model used a class-agnostic classifier in its last stage; the test-dev result used no extra human-labeled data.Source: §4.1 · p. 5, Table 1 · p. 6, Table 9 · p. 7

4.2. Ablations Experiments

  • More refinement stages improved performance, especially for large objects; three stages matched four with lower computational cost.Each added stage doubles the preceding stage's output size.Source: §4.2 · p. 6, Table 3 · p. 6
  • SFM outperformed the three alternative fusion modules, which the authors attribute to its larger receptive field and context capture.The comparison is reported in Table 4.Source: §4.2 · p. 6, Table 4 · p. 6
  • Adding fine-grained features improved results by 1.6 points AP and 2.5 points AP⋆, with larger-scale objects benefiting more.The ablation removed the semantic head to test the contribution of fine-grained features.Source: §4.2 · p. 6, Table 5 · p. 6

4.3. Comparison with Previous Methods

  • RefineMask improved accuracy over Mask R-CNN and compared favorably with PointRend and HTC, with comparable or faster inference in the reported comparisons.The efficiency comparison used a 1× schedule, R50-FPN, and a single Tesla V100 GPU.Source: §4.1 · p. 5, Table 2 · p. 6, Table 8 · p. 7

4.2. Ablations Experiments

  • Ablations support both multi-stage and boundary-aware refinement: multi-stage supervision improved mask results, while boundary awareness added 0.9 points AP⋆.Without boundary-aware refinement, each stage predicted a complete mask; later resolution increases offered little benefit to easy, non-boundary pixels.Source: Table 7 · p. 6, Table 6 · p. 6, §4.2 · p. 7

4.6. Qualitative results.

  • COCO visualizations show sharper RefineMask predictions than Mask R-CNN on difficult regions, including airplane tails and horse feet.Against the coarser COCO annotations, some predictions also appeared more accurate than the ground-truth masks.Source: Figure 6 · p. 8, §4.6 · p. 8

4.5. Experiments on Cityscapes

  • Cityscapes showed larger improvements than COCO and LVIS; the reported result is the median of three runs because performance had high variance.The Cityscapes models used fine annotations and 64 training epochs.Source: Table 11 · p. 8, §4.5 · p. 8

5. Conclusion

RefineMask is a multi-stage framework for high-quality instance segmentation. It incorporates fine-grained features during instance-wise segmentation to address detail loss from pooling and produce more accurate masks.

5. Conclusion

  • Two-stage methods rely on pooling instance features from a feature pyramid, losing details needed for crisp mask boundaries.The conclusion identifies RoIAlign as an example of the pooling operation involved.Source: §5 · p. 8
  • RefineMask iteratively incorporates fine-grained features during instance-wise segmentation to refine masks.The approach is presented as a multi-stage framework.Source: §5 · p. 8
  • The authors propose RefineMask as a strong baseline for high-quality instance segmentation.This is the paper's stated conclusion about the framework's potential role.Source: §5 · p. 8

A.1 Boundary Region Calculation

The appendix describes a convolutional operator that approximates the paper's boundary-region calculation for efficient implementation. Its comparison with the defined regions reports high overlap and visually similar outputs.

A.1 Boundary Region Calculation

  • A convolutional operator approximates the boundary-region calculation defined in Equation 1 for efficient implementation.For boundary regions of width 2, the operator is a 5×5 matrix; larger widths can be handled similarly with weights summing to zero.Source: §A.1 · p. 10
  • The width-2 operator produces zero outside boundary regions and can produce nonzero values on boundary regions.The appendix specifies a 5×5 operator for calculating boundary regions of width 2.Source: §A.1 · p. 10
  • The approximation and definition yield high boundary-region IoU and visually near-identical regions.Figure 7 presents examples from the two implementations; Table 12 reports their IoU comparison.Source: §A.1 · p. 10, Table 12 · p. 10

A.2 Direct Measurement of Boundary Quality

The paper adapts boundary F1 to instance segmentation as Fnpx, measuring boundary agreement within n pixels of an object's contour. Reported results show larger gains on COCO and indicate improved boundary quality.

A.2 Direct Measurement of Boundary Quality

  • Fnpx adapts the boundary F1 score to instance segmentation by measuring boundary agreement within n pixels of the object contour.For each instance, the score compares the ground-truth mask with its positive prediction having maximum IoU; instances without a positive prediction are ignored.Source: §A.2 · p. 10
  • Under Fnpx evaluation, RefineMask showed larger gains on COCO and outperformed Mask R-CNN with an X101-FPN backbone by a large margin.The authors interpret these results as further indication that RefineMask improves boundary quality.Source: §A.2 · p. 10
  • Figure 7 contrasts boundary regions approximated by convolutional kernels with regions defined by Equation 1.The figure shows the approximations in the top row and the defined regions in the bottom row.Source: Figure 7 · p. 11
Loading 2104.08569v1…