PaperlayerView exact revision on arXiv

Source-linked AI summary

Pointly-Supervised Instance Segmentation

Bowen Cheng, Omkar Parkhi, Alexander Kirillov

arXiv:2104.06404v2cs.CV

TL;DR

Instance segmentation needs costly pixel-level masks, motivating a simpler supervision format that can approach fully supervised performance. The paper combines bounding boxes with 10 randomly sampled object/background points, trains existing models directly, and introduces Implicit PointRend; Mask R-CNN reaches 94%–98% of fully supervised performance while annotation is about five times faster.

  • Problem

    Full object-mask annotation is substantially more time-consuming than bounding-box annotation, while prior weakly supervised methods remain behind fully supervised performance on large-scale datasets.

  • Method

    The paper collects bounding boxes with randomly sampled points labeled as object or background, uses them to supervise existing models directly, and proposes Implicit PointRend for point-based mask prediction.

  • Results

    94%–98% of fully supervised performance is achieved by Mask R-CNN with 10 annotated points per object across multiple datasets, while annotation is approximately 5 times faster than full-mask annotation.

  • Takeaways & Limitations

    Point-based supervision provides a strong weakly supervised baseline that is applicable to existing instance-segmentation models across datasets and scene types.

  • Takeaways & Limitations

    The method requires filtering mask predictions outside predicted boxes, although more sophisticated point selection could address this issue.

Abstract

from arXiv · show

We propose an embarrassingly simple point annotation scheme to collect weak supervision for instance segmentation. In addition to bounding boxes, we collect binary labels for a set of points uniformly sampled inside each bounding box. We show that the existing instance segmentation models developed for full mask supervision can be seamlessly trained with point-based supervision collected via our scheme. Remarkably, Mask R-CNN trained on COCO, PASCAL VOC, Cityscapes, and LVIS with only 10 annotated random points per object achieves 94%--98% of its fully-supervised performance, setting a strong baseline for weakly-supervised instance segmentation. The new point annotation scheme is approximately 5 times faster than annotating full object masks, making high-quality instance segmentation more accessible in practice. Inspired by the point-based annotation form, we propose a modification to PointRend instance segmentation module. For each object, the new architecture, called Implicit PointRend, generates parameters for a function that makes the final point-level mask prediction. Implicit PointRend is more straightforward and uses a single point-level mask loss. Our experiments show that the new module is more suitable for the point-based supervision.

1. Introduction

The paper introduces a point-based annotation scheme that combines bounding boxes with randomly sampled object/background points, addressing the cost of full-mask annotation while retaining high segmentation performance. It also shows that existing models can use this supervision directly and proposes Implicit PointRend for point-based training.

  • Motivation: Full-mask annotation remains substantially more complex and time-consuming than bounding-box annotation, motivating weaker supervision.COCO polygon masks take 79.2 seconds per instance on average, while bounding boxes take about 7 seconds.
  • Method: The scheme combines each object bounding box with randomly sampled points labeled as object or background.The annotation format is designed to be simple and can be simulated from existing instance-segmentation ground truth.
  • Results: Mask R-CNN trained with point supervision reaches 94%–98% of fully supervised performance across multiple datasets.The setup uses 10 annotated points per object and applies to COCO, PASCAL VOC, Cityscapes, and LVIS.
  • Method: Point-based supervision trains Mask R-CNN, PointRend, and CondInst without changing their architectures or training pipelines.The standard mask loss is computed at the annotated points by interpolating mask predictions there.
  • Results: The annotation scheme takes approximately 5 times less time than polygon-based mask annotation while preserving near-full-supervision performance.Bounding-box annotation plus 10 point labels takes 16 seconds per object, compared with polygon-based mask annotation in COCO.
  • Implicit PointRend: Implicit PointRend generates object-specific function parameters for point-level mask prediction and uses a single point-level mask loss.It removes importance-point sampling during training and avoids intermediate prediction interpolation.

2. Related Works

Related work spans fully supervised mask prediction, weak supervision from image-level labels or boxes, and point-based supervision in interactive and other vision tasks. The paper distinguishes its scheme by combining simple point labels with boxes and directly supervising existing instance-segmentation models.

  • Fully supervised segmentation: Fully supervised instance-segmentation models directly predict object masks from ground-truth masks, using region-level or image-level representations.Examples include Mask R-CNN-based methods, YOLACT++, and CondInst.
  • Weak supervision: Weakly supervised methods use image-level labels or bounding boxes, often generating proposals or pseudo-ground truth for training.The cited approaches show promise on smaller datasets but lack competitive large-scale COCO results in the supplied passage.
  • Point-based supervision: Point supervision has been applied across several vision tasks, while interactive instance-segmentation systems commonly require repeated model inference during annotation.The proposed scheme uses a simpler annotation process than those interactive pipelines.

3. Pointly-Supervised Instance Segmentation

The paper introduces pointly supervision: bounding boxes plus randomly sampled, binary-labeled interior points, enabling standard instance segmentation models to train without architectural changes. Ten points per object approach full-mask performance across datasets while reducing annotation time, and targeted augmentation and self-training further improve results.

  • 3.1. Annotation format and collection: Point supervision uses N randomly sampled points inside each bounding box, with annotators assigning object/background labels and models applying the standard mask loss at interpolated point locations.The scheme can be simulated from existing instance masks and is compatible with standard mask-predicting architectures.
  • 3.1. Annotation format and collection: 16 seconds per object for a box plus 10 points is approximately five times faster than COCO polygon-mask annotation.The estimate combines 7 seconds for the box with 0.9 seconds per point, versus 79.2 seconds for a polygon mask.
  • 3.3.1. Ablation of the annotation design: P10 provides a strong annotation-time trade-off: P20 improves performance by only about 0.3 AP while doubling annotation time.Performance improves rapidly through tens of points and then shows diminishing returns.
  • 3.3. Main results: 94%–97% of fully supervised performance is achieved by Mask R-CNN with P10 across four datasets, without changing its architecture or default training procedure.The comparison spans PASCAL VOC, COCO, Cityscapes, and LVIS.
  • 3.2. Training with points: Point-based augmentation improves the higher-capacity ResNeXt-101 model by 0.4 AP, while changing little for the smaller Mask R-CNN model.The strategy subsamples half of the available points at each training iteration.
  • 3.3.2. Main results: Self-training raises COCO Mask R-CNN performance from 36.1 AP to 36.7 AP, reaching 98% of the reported fully supervised performance.Pseudo-ground-truth masks are generated from the point-supervised model using ground-truth boxes.

4. Implicit PointRend Model

Implicit PointRend replaces PointRend’s coarse mask context with object-specific point-head parameters and a single point-level loss. On COCO, it matches PointRend with full masks and performs better under 10-point supervision, reaching the performance of fully supervised Mask R-CNN.

  • Motivation: Point supervision exposes a larger performance gap for standard PointRend than for other methods, motivating a design tailored to point-based training.The authors report that PointRend’s full-mask advantage does not carry over under point supervision.
  • Architecture: Implicit PointRend generates object-specific point-head parameters instead of using PointRend’s coarse mask prediction for region-level context.The point head predicts masks at arbitrary locations from fine-grained features and relative point coordinates.
  • Training and inference: The module uses one point-level mask loss and avoids PointRend’s importance sampling, while retaining adaptive subdivision during inference.Training can use annotated points directly, and inference progressively refines predictions from a uniform grid.
  • Main results: Implicit PointRend significantly outperforms PointRend under 10-point supervision while performing on par with PointRend under full mask supervision.The comparison uses AP (P10) and AP (M) on COCO with a ResNet-50-FPN backbone.
  • Main results: 96% of full-mask performance is achieved with point supervision, and 10-point Implicit PointRend matches fully supervised Mask R-CNN across backbone capacities.The comparison includes R50, R101, and X101 backbones.

5. Conclusion

The paper introduces random point-and-box supervision and Implicit PointRend as a simpler module for point-supervised instance segmentation. It concludes that this scheme offers a strong accuracy–annotation-time trade-off.

  • Conclusion: The proposed point annotation scheme labels bounding boxes and random points, and existing mask-prediction models can use it without architectural or training changes.The scheme is designed to provide weak supervision while preserving standard model pipelines.
  • Conclusion: The annotation scheme provides the best reported trade-off between annotation time and accuracy among the compared instance-segmentation schemes.The conclusion also identifies Implicit PointRend as a simple module addressing challenges specific to point supervision.

Appendix

The appendix provides additional analyses of the annotation tool, point-supervision overfitting, data augmentation, annotation time, and Implicit PointRend.

  • Appendix: The appendix expands the paper’s analysis with tool details, overfitting and augmentation ablations, annotation-time comparisons, and a detailed Implicit PointRend study.These analyses are organized across appendix sections A, B, D, and E.

A. Annotation Pipeline

The annotation pipeline combines object spotting and categorization with either boxes, points, or polygons. Point annotation uses a two-view tool, achieves about 90% agreement with masks, and can be more efficient under common pipeline timings.

  • Annotation tool: A trained annotator labels each point in 0.8–0.9 seconds, while the tool presents whole-object and zoomed views to support point classification.The interface shows the object context and a centered zoomed patch for each randomly presented point.
  • Annotation quality: Approximately 90% of point labels agree with COCO instance masks, with most disagreements attributed to inaccurate polygon boundaries.The comparison is against ground-truth masks and includes boundary-related discrepancies.
  • Time–performance trade-off: Point-based annotation is more efficient for pipelines whose spotting and categorization stages take 2–237 seconds per instance when targeting 31.8 AP on COCO.The analysis compares total annotation time across supervision forms.
  • Annotation pipeline: The pipeline separately accounts for spotting and categorization time, which is shared across annotation forms, and the subsequent instance-labeling time.For COCO, the reported shared pipeline time is 43.2 seconds per instance.

B. Overfitting and Point-based Augmentation

Longer training increases the performance gap between full-mask and point-based supervision, while point-based augmentation is designed to counter this overfitting effect.

  • Overfitting with longer training schedules: 1.1 AP separates full-mask and point-based supervision after a 3× schedule, compared with 0.9 AP under a 1× schedule.For ResNet-50-FPN, the reported 3× results are 37.2 AP versus 36.1 AP, while the 1× results are 35.2 AP versus 34.3 AP.
  • Point-based data augmentation: Sampling only half of the available points at each iteration is proposed as a point-based augmentation to counter reduced training-data variability.The augmentation randomly samples half of the points for a box instead of using all points.
  • Point-based data augmentation: Using 1 or 3 sampled points is too aggressive, whereas sampling 5, 7, or 9 points produces similar results.

C. More Analysis on Point-based Annotation

Random point supervision is close to full supervision and remains robust for rare LVIS categories, though direct comparison with multiple clicks is unavailable.

  • Point sampling schemes: Random points are already close to full supervision while being more efficient to collect and simulate.The authors do not directly compare multiple clicks because reliable click simulation is unavailable and large-scale collection is prohibitively expensive.
  • Point-based annotation for rare objects: LVIS rare-category AP varies modestly across three regenerated 10-point datasets, with standard deviation 0.26 versus 0.78 for mask supervision.The three reported rare-class AP means are 10.8±0.85, 10.2±0.30, and 10.7±0.40.

D. COCO Annotation Time

COCO annotation-time accounting shows that ten point labels added to a bounding box are substantially faster than polygon-based mask annotation.

  • COCO annotation time: Category labeling and instance spotting contribute 28.8 and 14.4 seconds per instance, respectively, to the COCO annotation totals.Both stages are performed with eight workers per image, and the reported total sums their annotation time.
  • COCO annotation time: Polygon-based mask annotation itself takes approximately 79.2 seconds per instance, while bounding-box annotation is estimated at 7 seconds after spotting.
  • COCO annotation time: For datasets that already have bounding boxes, ten point labels take 9 seconds per instance, over eight times faster than 79.2-second polygon masks.

E.1. Ablation study

The ablations identify relative box coordinates and image-level point features as important components of Implicit PointRend, with augmentation benefiting the new module most.

  • Point-based data augmentation: 0.9 AP is gained by point-based augmentation for Implicit PointRend, while Mask R-CNN and PointRend show no significant improvement.The experiments use a ResNet-50-FPN backbone and compare point-supervised models.
  • Point-level feature representation: Adding relative coordinates improves performance by 1.4 AP with point supervision and 0.8 AP with mask supervision.Without coordinates, the mask head is translation invariant and cannot distinguish similarly appearing points within one box.
  • Point-level feature representation: 1.6 AP from p2 image-level features and 1.4 AP from relative coordinates improve point-supervised Implicit PointRend.Positional encoding adds a further 0.3 AP for point-based supervision.
  • Point-level feature representation: Image-level p2 features further improve performance by 1.6 AP with point supervision and 1.4 AP with mask supervision.The ablation concludes that both coordinate and image-level features are essential to overall Implicit PointRend performance.

E.2. Mask supervision results

Under full mask supervision on COCO, Implicit PointRend is competitive with established instance segmentation methods while outperforming CondInst. Its remaining performance gap to PointRend is small and mainly associated with large objects.

  • Implicit PointRend matches PointRend without coarse mask prediction or importance sampling during training.The comparison is reported on COCO with full mask supervision.
  • Implicit PointRend outperforms CondInst by a large margin under full mask supervision.
  • Table 10 reports that Implicit PointRend performs better than Mask R-CNN and comparably to PointRend on COCO val2017.
  • The gap between Implicit PointRend and PointRend is less than 0.3 AP and mostly comes from large objects.
Loading 2104.06404v2…