PaperlayerView exact revision on arXiv

Source-linked AI summary

PointRend: Image Segmentation as Rendering

Alexander Kirillov, Yuxin Wu, Kaiming He, Ross Girshick

arXiv:1912.08193v2cs.CV

TL;DR

Image segmentation’s regular grids oversample smooth regions and undersample object boundaries, limiting efficient high-quality label-map prediction. PointRend treats segmentation as rendering, adaptively predicts selected points through subdivision, and improves instance and semantic segmentation while enabling practical high-resolution outputs.

  • Problem

    Regular-grid segmentation oversamples smooth regions and undersamples object boundaries, producing excess computation and blurry contours.

  • Method

    PointRend is a general module that adaptively selects points and predicts labels from interpolated CNN features using a subdivision strategy.

  • Results

    PointRend significantly improves strong Mask R-CNN and DeepLabV3 models on COCO and Cityscapes for instance and semantic segmentation.

  • Takeaways & Limitations

    PointRend produces sharper boundaries and makes high-resolution segmentation feasible by avoiding dense computation where coarse predictions suffice.

Abstract

from arXiv · show

We present a new method for efficient high-quality image segmentation of objects and scenes. By analogizing classical computer graphics methods for efficient rendering with over- and undersampling challenges faced in pixel labeling tasks, we develop a unique perspective of image segmentation as a rendering problem. From this vantage, we present the PointRend (Point-based Rendering) neural network module: a module that performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm. PointRend can be flexibly applied to both instance and semantic segmentation tasks by building on top of existing state-of-the-art models. While many concrete implementations of the general idea are possible, we show that a simple design already achieves excellent results. Qualitatively, PointRend outputs crisp object boundaries in regions that are over-smoothed by previous methods. Quantitatively, PointRend yields significant gains on COCO and Cityscapes, for both instance and semantic segmentation. PointRend's efficiency enables output resolutions that are otherwise impractical in terms of memory or computation compared to existing approaches. Code has been made available at https://github.com/facebookresearch/detectron2/tree/master/projects/PointRend.

1. Introduction

Image segmentation’s regular-grid outputs oversample smooth interiors while undersampling object boundaries, motivating PointRend’s adaptive rendering strategy. PointRend predicts labels at selected points and improves boundary detail and benchmark performance.

  • 1. Introduction: Regular grids oversample smooth regions and undersample object boundaries, causing excess computation and blurry contours.Low-resolution grids are used as a compromise between these sampling problems.
  • 1. Introduction: PointRend views image segmentation as rendering and adaptively selects a non-uniform set of points for label prediction.Its subdivision strategy computes high-resolution maps using an order of magnitude fewer floating-point operations than dense computation.
  • 1. Introduction: PointRend can be integrated into existing semantic and instance segmentation systems.The paper describes applications to systems such as Mask R-CNN and FCN.
  • 1. Introduction: PointRend interpolates feature-map values at selected points and uses a small point-head subnetwork to predict output labels.The module accepts regular-grid CNN feature maps and produces predictions over a finer grid.
  • 1. Introduction: PointRend improves strong Mask R-CNN and DeepLabV3 models on COCO and Cityscapes for instance and semantic segmentation.The paper reports sharper boundaries and quantitative gains despite metrics being relatively insensitive to boundary quality.

2. Related Work

Related segmentation systems rely on coarse or regular-grid representations that can limit fine detail or increase computation. PointRend instead applies adaptive output sampling and subdivision to refine uncertain regions efficiently.

  • 2. Related Work: Graphics renderers compute regular-grid pixel values over adaptively selected, non-uniform points to efficiently produce high-resolution images.Subdivision creates a quadtree-like sampling pattern, while adaptive sampling refines regions with greater variance.
  • 2. Related Work: PointRend differs from prior non-uniform semantic segmentation by sampling non-uniformly at the output rather than before network processing.The two approaches may be combined, while the prior method is currently unproven for instance segmentation.
  • 2. Related Work: Mask R-CNN predicts masks on a fixed 28×28 grid, which can oversmooth fine details for large objects.Bottom-up methods can produce more detailed output but lag behind region-based approaches on most instance-segmentation benchmarks.
  • 2. Related Work: Semantic segmentation systems commonly recover lower-resolution outputs with bilinear upsampling, while dilated convolutions increase memory and computation.Encoder-decoder architectures and skip connections are alternative ways to recover filtered details.
  • 2. Related Work: For instance segmentation, PointRend refines coarse masks by independently predicting selected points with a small MLP.The MLP uses interpolated backbone and coarse-mask features, and subdivision iteratively refines uncertain regions.

3. Method

PointRend treats segmentation as rendering: it adaptively predicts labels at selected points, concentrating computation near uncertain, high-frequency regions while producing finer outputs efficiently.

  • 3. Method: PointRend reframes segmentation as rendering an occupancy map from CNN feature maps, adapting computer-graphics ideas to label prediction.The module accesses interpolated point-wise features and predicts labels for selected points rather than every output pixel.
  • 3.1. Point Selection for Inference and Training: PointRend selects points adaptively near high-frequency areas, especially object boundaries, and develops the strategy for both inference and training.This targets the oversampling of smooth regions and undersampling of boundaries caused by regular-grid prediction.
  • 3.1. Point Selection for Inference and Training: During inference, PointRend repeatedly upsamples coarse predictions, selects the N most uncertain points, and predicts their labels on progressively finer grids.The selected points can be those with probabilities closest to 0.5 for binary masks; other locations receive interpolated values.
  • 3.1. Point Selection for Inference and Training: With M×M output resolution and M0×M0 starting resolution, PointRend requires no more than N log2(M/M0) point predictions.For M0=7, M=224, and N=282, it predicts 282·4.25 points, 15 times fewer than 224^2.
  • 3.1. Point Selection for Inference and Training: During training, PointRend uses non-iterative random sampling that biases points toward uncertain regions while retaining uniform coverage.It combines over-generation, importance sampling of the most uncertain βN candidates, and uniform sampling of the remaining (1−β)N points.
  • 3.2. Point-wise Representation and Point Head: Selected points receive point-wise features from interpolated CNN maps, combined with coarse prediction features to support region-specific segmentation.Fine-grained features can be extracted from one or multiple feature maps and concatenated; coarse features provide region-specific information for instance masks.

4. Experiments: Instance Segmentation

PointRend improves Mask R-CNN instance segmentation on COCO and Cityscapes while producing sharper, higher-resolution masks efficiently. Its adaptive subdivision concentrates computation on uncertain regions, making 224×224 outputs practical.

  • Inference design: PointRend refines coarse masks by selecting the most uncertain points at each subdivision step and updating their point-wise predictions.Inference refines a coarse 7×7 prediction to 224×224 in five steps, selecting at most 282 uncertain points per step.
  • Main results: PointRend outperforms Mask R-CNN’s default 4× conv mask head on both COCO and Cityscapes, quantitatively and qualitatively.The comparison uses mask AP, with AP⋆ additionally evaluating COCO masks against higher-quality LVIS annotations.
  • Efficiency: More than 30 times less compute and memory enables PointRend to produce 224×224 masks than the default 4× conv head at the same resolution.PointRend avoids computation where the coarse prediction is already sufficient, such as regions far from object boundaries.
  • Efficiency: At approximately 13 fps, unoptimized PointRend produces 224×224 masks at roughly the frame rate of a 56×56 4× conv head.The 56×56 design has lower COCO AP than the 28×28 baseline, 34.5% versus 35.2%.
  • Resolution and quality: Increasing output resolution improves visual boundary detail even when AP saturates because intersection-over-union metrics emphasize object interiors.The difference between 28×28 and 224×224 outputs is visually obvious despite relatively small AP differences.
  • Ablations: PointRend’s improvements persist with longer training schedules and larger models, while its point-head depth and width are not critical.Ablations also show that a broad range of training-sampling parameters, 2<k<5 and 0.75<β<1.0, gives similar results.

5. Experiments: Semantic Segmentation

PointRend extends semantic segmentation models with adaptive point-based refinement, improving accuracy while producing high-resolution predictions efficiently. On Cityscapes, it outperforms baseline architectures and avoids the higher costs of dilated-convolution alternatives.

  • PointRend extends instance-segmentation refinement to semantic segmentation models including DeeplabV3 and SemanticFPN.
  • On Cityscapes, the evaluation uses 19 categories, 2,975 training images, and 500 validation images, reporting median mIoU across five trials.
  • PointRend uses coarse semantic predictions plus interpolated fine-grained features and applies the same adaptive point-selection strategy as for instance segmentation.
  • DeeplabV3: PointRend achieves higher mIoU than baseline DeeplabV3, while dilated res4 inference remains inferior despite higher computational and memory costs.
  • DeeplabV3: PointRend reaches 1024×2048 resolution, representing about 2M points, while predicting only 32k adaptively sampled points.
  • SemanticFPN: SemanticFPN with PointRend improves over both its 8× and 4× output-stride variants without PointRend.

Appendix A. Instance Segmentation Details

The instance-segmentation experiments use standard SGD-based training configurations with dataset-specific schedules, batch sizes, resizing, and a longer COCO schedule variant.

  • Training uses SGD with 0.9 momentum, warmup over 1,000 updates, weight decay of 0.0001, horizontal flipping, and scale augmentation.
  • COCO: COCO training uses 16-image minibatches, a 60k/20k/10k update schedule, and random resizing between 640 and 800 pixels.
  • Cityscapes: Cityscapes training uses 8-image minibatches, an 18k/6k update schedule, and random resizing between 800 and 1,024 pixels.
  • COCO: The longer COCO schedule uses 210k/40k/20k updates while keeping all other settings unchanged.

Appendix B. Semantic Segmentation Details

The semantic-segmentation implementations follow DeeplabV3 and SemanticFPN training setups with model-specific crop sizes, schedules, augmentation, and optimization details.

  • DeeplabV3: DeeplabV3 uses SGD with 0.9 momentum, 16-image minibatches cropped to 768×768, a 90k-update polynomial schedule, and weight decay of 0.0001.
  • Both configurations apply linear warmup over 1,000 updates, while SemanticFPN additionally uses color augmentation and crop bootstrapping.
  • SemanticFPN: SemanticFPN uses SGD with 0.9 momentum, 32-image minibatches cropped to 512×1024, and a 40k/15k/10k learning-rate schedule.

Appendix C. AP⋆Computation

The appendix reports an evaluation error in the first arXiv version's COCO mask AP⋆ computation against LVIS annotations.

  • The first arXiv version used an incorrect list of categories absent from each evaluation image, producing lower COCO AP⋆ values.
Loading 1912.08193v2…