PaperlayerView exact revision on arXiv

Source-linked AI summary

Boundary-preserving Mask R-CNN

Tianheng Cheng, Xinggang Wang, Lichao Huang, Wenyu Liu

arXiv:2007.08921v1cs.CV

TL;DR

Instance segmentation methods based on pixel-wise classification can ignore object boundaries and shapes, producing coarse masks and imprecise localization. BMask R-CNN jointly learns masks and boundaries through a boundary-preserving mask head with feature fusion blocks. It outperforms Mask R-CNN on COCO and Cityscapes, with larger gains under stricter localization criteria and on more precisely annotated data.

  • Problem

    Pixel-level classification in instance segmentation overlooks object shape and boundary information, making precise mask localization difficult.

  • Method

    BMask R-CNN replaces the Mask R-CNN mask head with a boundary-preserving head that jointly learns masks and boundaries through feature fusion blocks.

  • Results

    BMask R-CNN outperforms Mask R-CNN by 1.7% AP on COCO val and 2.2% AP on Cityscapes test, with larger gains at higher IoU thresholds.

  • Takeaways & Limitations

    Boundary information improves mask alignment and localization, with especially notable benefits when evaluation requires more precise masks.

  • Takeaways & Limitations

    Coarse mask and boundary annotations in COCO limit BMask R-CNN’s performance, although it can still produce more precise and smoother boundaries.

Abstract

from arXiv · show

Tremendous efforts have been made to improve mask localization accuracy in instance segmentation. Modern instance segmentation methods relying on fully convolutional networks perform pixel-wise classification, which ignores object boundaries and shapes, leading coarse and indistinct mask prediction results and imprecise localization. To remedy these problems, we propose a conceptually simple yet effective Boundary-preserving Mask R-CNN (BMask R-CNN) to leverage object boundary information to improve mask localization accuracy. BMask R-CNN contains a boundary-preserving mask head in which object boundary and mask are mutually learned via feature fusion blocks. As a result, the predicted masks are better aligned with object boundaries. Without bells and whistles, BMask R-CNN outperforms Mask R-CNN by a considerable margin on the COCO dataset; in the Cityscapes dataset, there are more accurate boundary groundtruths available, so that BMask R-CNN obtains remarkable improvements over Mask R-CNN. Besides, it is not surprising to observe that BMask R-CNN obtains more obvious improvement when the evaluation criterion requires better localization (e.g., AP$_{75}$) as shown in Fig.1. Code and models are available at \url{https://github.com/hustvl/BMaskR-CNN}.

1 Introduction

Instance segmentation methods based on pixel-level classification can produce coarse boundaries and imprecise localization because they overlook object shape and boundary information. BMask R-CNN addresses this by jointly learning masks and boundaries, achieving stronger localization results across COCO and Cityscapes.

  • Problem: Pixel-level classification treats proposal pixels equally, overlooking object shape and boundary information needed for precise masks.Pixels near object boundaries are difficult to classify, contributing to coarse and indistinct segmentation results.
  • Motivation: BMask R-CNN unifies instance-level mask and boundary prediction in one network because boundaries provide a dual representation that guides masks toward groundtruth alignment.The resulting masks are described as more distinct and precise in object localization.
  • Method: The boundary-preserving mask head jointly learns object masks and boundaries, using feature fusion blocks to connect boundary and mask feature learning.Boundary features guide mask prediction and provide shape and localization information.
  • Evaluation: The method is evaluated extensively on COCO and Cityscapes, with larger improvements reported on the fine-annotated Cityscapes dataset.The paper also studies feature fusion blocks, boundary features, boundary losses, and the Sobel mask head.
  • Contribution: BMask R-CNN explicitly exploits object boundary information to improve mask-level localization within the Mask R-CNN framework.The method is presented as conceptually simple and novel.
  • Results: BMask R-CNN outperforms Mask R-CNN by 1.7% AP on COCO val and 2.2% AP on Cityscapes test.It also obtains higher AP gains as the mask IoU threshold increases.

2 Related Work

Prior instance segmentation work includes detection-based methods that generate proposals before mask prediction and segmentation-based methods that group pixels into instances. Related research also develops boundary and edge detection techniques relevant to boundary-aware segmentation.

  • Detection-based methods: Detection-based methods generate region proposals with object detectors and predict instance masks after RoI pooling or alignment.Mask prediction is performed for object proposals in this family.
  • Segmentation-based methods: Segmentation-based methods first perform pixel-level segmentation and then group pixels into individual objects.Examples use boundaries, line segments, instance centers, directions, or metric learning for grouping.
  • Boundary detection: Deep convolutional networks have advanced edge detection through image-to-image prediction, semantic boundary detection, label-misalignment studies, and geometric-aware losses.These approaches provide related techniques for modeling boundaries and object structure.

3 Boundary-preserving Mask R-CNN

BMask R-CNN addresses coarse Mask R-CNN masks by jointly learning object boundaries and masks, then fusing boundary features into mask prediction. Its boundary-preserving head uses multi-task learning and boundary-specific optimization to improve shape and localization information.

  • Motivation: Mask R-CNN relies on local pixel-level predictions that ignore object shape, producing coarse, indistinct, and sometimes false-positive masks.Visualized Mask R-CNN predictions are described as rough and imprecise.
  • Boundary-preserving mask head: BMask R-CNN replaces the Mask R-CNN mask head with a boundary-preserving head that jointly learns object boundaries and masks end-to-end.The two representations are mutually related: mask features support boundary learning, while boundary features guide mask prediction.
  • RoI features: Boundary RoI features are extracted from the finest-resolution FPN level, kept at higher resolution, then downsampled to match mask-feature resolution for fusion.Mask RoI features use a scale-specific pyramid level, whereas boundary features preserve spatial information from P2.
  • Feature fusion: The architecture fuses mask and boundary features through convolutional blocks so boundary shape and localization information enrich mask features.The mask branch uses four consecutive 3 × 3 convolutions before boundary-feature fusion, followed by two 3 × 3 convolutions; the predictor uses 2 × 2 deconvolution and 1 × 1 convolution.
  • Boundary optimization: Boundary supervision combines dice loss with binary cross-entropy to address the imbalance and coarse-boundary issues of boundary prediction.The boundary targets are generated with a Laplacian operator and thresholded at 0; the experiments set λ = 1 and ϵ = 1.
  • Multi-task learning: A joint loss combines inherited classification and box-regression losses with Mask R-CNN mask loss and the newly introduced boundary loss.The paper motivates joint training by the cross-links between boundary and mask features, which can enhance representations for both predictions.

4 Experiments

Experiments evaluate BMask R-CNN on COCO and Cityscapes using Mask R-CNN as the baseline, including backbone comparisons and detailed component studies. BMask R-CNN consistently improves mask performance, with larger gains at stricter IoU thresholds and when annotations are more precise.

  • Experimental setup: Experiments evaluate BMask R-CNN on COCO and Cityscapes, with detailed ablations on COCO and Mask R-CNN as the baseline.COCO results use train2017 for training and val2017 or test-dev2017 for evaluation; Cityscapes contains urban scenes and eight object categories.
  • COCO results: 1.4, 1.7 and 1.5 AP improvements over Mask R-CNN are obtained with ResNet-50-FPN, ResNet-101-FPN and HRNetV2-W32-FPN respectively on COCO.The improvements remain stable across the evaluated backbones and input sizes.
  • COCO results: ≈2.3 AP gains on AP75 are reported across the evaluated COCO backbone comparisons, indicating stronger improvement under more precise mask localization.Bounding-box AP improves only very slightly over Mask R-CNN.
  • Comparison with prior methods: BMask R-CNN with ResNet-101-FPN surpasses the compared state-of-the-art instance segmentation methods on COCO test-dev2017 without additional bells and whistles.The comparison uses models trained on COCO train2017 and evaluated on COCO test-dev2017.
  • Localization quality: BMask R-CNN obtains larger gains than Mask R-CNN as the IoU threshold increases, showing better localization performance.This trend is illustrated by the AP curves in Fig. 1.

4.2 Ablation Experiments

Ablation experiments progressively add BMask R-CNN components and isolate the contribution of boundary supervision. Boundary supervision improves mask performance over both mask supervision and no boundary supervision.

  • The ablation study gradually adds boundary supervision, feature fusions, BCE-Dice loss, and specialized boundary RoI extraction.
  • Boundary supervision improves mask results by 0.8 AP over mask supervision and 0.7 AP over no supervision.The architecture remains the same while boundary targets and supervision are varied.

Effects of Boundaries:

Boundary-focused design choices address the spatial sparsity of boundaries and strengthen communication between boundary and mask features. Feature fusion substantially increases gains over Mask R-CNN.

  • RoI Feature Extraction:: Boundary prediction requires more precise spatial information because boundary pixels are spatially sparse.The study therefore varies RoI feature sources and resolutions for boundary learning.
  • Table 2 compares instance segmentation methods on COCO test-dev2017.The comparison includes state-of-the-art methods and identifies models marked with an asterisk as author implementations.
  • Feature Fusion:: With both mask-to-boundary and boundary-to-mask fusion blocks, BMask R-CNN improves over Mask R-CNN by 1.5 AP.Without fusion, the gain is 0.5 AP; adding further fusion blocks yields negligible improvements.
  • Loss Functions:: The combination of BCE and Dice loss leads to better performance than either individual loss for boundary learning.Weighted BCE produces less gain than BCE in boundary prediction.

Loss Functions:

The loss-function analysis compares boundary-learning objectives and visualizes their effects. Combining BCE and Dice produces boundaries that are both better localized and less affected by class imbalance.

  • Boundary Targets:: Table 4 evaluates changing the boundary head’s groundtruth, including a no-supervision condition.
  • RoI Features:: Table 5 evaluates different RoI feature extraction strategies on COCO val2017.
  • RoI Features:: Figure 5 illustrates different RoI feature extraction strategies for the boundary-preserving mask head.
  • Loss Functions:: Weighted BCE brings less gain than BCE in boundary prediction.
  • Loss Functions:: Combining Dice loss with BCE provides better-localized boundaries while avoiding the class-imbalance problem.BCE alone is precise but unclear, Weighted BCE produces thick coarse boundaries, and Dice lacks precise localization.
  • Computation Cost:: At equal computation cost, BMask R-CNN still achieves a significant gain over Mask R-CNN.

Computation Cost:

Experiments examine BMask R-CNN under controlled training settings and report Cityscapes performance. On Cityscapes, the method achieves a 2.2 AP gain over Mask R-CNN.

  • Experimental Setup:: Cityscapes experiments use ResNet-50-FPN, fine annotations, and images with original 1024×2048 resolution for inference.
  • Fusion Ablation:: Table 6 evaluates mask-to-boundary and boundary-to-mask fusion blocks.
  • Loss Ablation:: Table 7 evaluates different loss functions on COCO val2017.
  • Loss Analysis:: Figure 6 visualizes the impacts of different loss functions, with GT denoting groundtruth and W-BCE and D-BCE denoting Weighted BCE and Dice-BCE.
  • Cityscapes Results:: BMask R-CNN achieves 29.4 AP on the Cityscapes test set and gains 2.2 AP over Mask R-CNN.It outperforms previous methods without extra data.

4.4 Discussions

The paper discusses annotation limits, computation costs, and simpler boundary-prediction alternatives. It also reports that the Sobel mask head improves over Mask R-CNN but remains below the main method.

  • COCO’s coarse polygon mask and boundary annotations limit BMask R-CNN’s performance despite its ability to produce smoother, more precise boundaries.Precise annotations are time-consuming to obtain, creating a practical data-quality constraint.
  • The Sobel mask head predicts boundaries from masks using a Sobel operator followed by two 3 × 3 convolutions.It uses the same Dice-BCE loss as the main method.
  • The discussion includes COCO computation-cost results and Cityscapes validation and test results, but the supplied passages provide no numerical table entries.
  • 34.0 AP improves Mask R-CNN by 0.8 AP but remains 0.7 AP below the main BMask R-CNN method.This comparison uses a ResNet-50-FPN backbone with the remaining settings held constant.

4.5 Qualitative Results

Qualitative COCO comparisons show that BMask R-CNN produces clearer masks with boundaries that more closely coincide with the predicted boundary features than Mask R-CNN.

  • Mask R-CNN often produces coarse boundaries containing background and false-positive regions, whereas BMask R-CNN alleviates these issues.
  • BMask R-CNN’s predicted masks are clearer and highly coincident with its predicted boundaries.The figure compares raw mask and boundary predictions against ground-truth segmentation using a ResNet-101-FPN backbone.
  • Refining masks with predicted boundaries yields only minor improvement and is vulnerable to noise.

5 Conclusion

The conclusion presents BMask R-CNN as a boundary-guided approach for improving mask boundaries and localization, with stable gains across COCO and Cityscapes.

  • BMask R-CNN incorporates boundary information to guide mask learning for better boundaries and localization.
  • Experiments show remarkable and stable improvements on COCO and Cityscapes, especially for localization performance.
  • The method can also be plugged into Cascade Mask R-CNN for higher performance.
Loading 2007.08921v1…