PaperlayerView exact revision on arXiv

Source-linked AI summary

Boundary IoU: Improving Object-Centric Image Segmentation Evaluation

Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C. Berg, Alexander Kirillov

arXiv:2103.16562v1cs.CV

TL;DR

Existing Mask IoU-based evaluation can be less sensitive to boundary errors, particularly for large objects, while prior boundary-focused measures have limitations. The paper proposes Boundary IoU and derives Boundary AP and Boundary PQ, finding that the new metrics better track boundary-quality improvements generally overlooked by current evaluations. Boundary IoU also has a documented scope limitation: it can give perfect scores to non-identical masks when mismatches lie beyond the evaluated boundary bands.

  • Problem

    Current evaluation metrics have limited sensitivity to mask prediction errors near object boundaries, and prior boundary-focused measures have documented limitations across error types and object sizes.

  • Method

    The paper analyzes Mask IoU, Trimap IoU, and F-measure across error types and object sizes, then proposes Boundary IoU and task-level Boundary AP and Boundary PQ.

  • Results

    Boundary AP and Boundary PQ reveal boundary-quality improvements that Mask IoU-based evaluations generally ignore, while Boundary IoU is more sensitive to large-object boundary quality without over-penalizing small-object errors.

  • Takeaways & Limitations

    Boundary-sensitive evaluation provides a clearer quantitative gradient for boundary segmentation quality and supports more granular analysis of segmentation-related errors.

  • Takeaways & Limitations

    Boundary IoU can award a perfect score to non-identical masks when their mismatching pixels lie farther than d from the corresponding boundaries.

Abstract

from arXiv · show

We present Boundary IoU (Intersection-over-Union), a new segmentation evaluation measure focused on boundary quality. We perform an extensive analysis across different error types and object sizes and show that Boundary IoU is significantly more sensitive than the standard Mask IoU measure to boundary errors for large objects and does not over-penalize errors on smaller objects. The new quality measure displays several desirable characteristics like symmetry w.r.t. prediction/ground truth pairs and balanced responsiveness across scales, which makes it more suitable for segmentation evaluation than other boundary-focused measures like Trimap IoU and F-measure. Based on Boundary IoU, we update the standard evaluation protocols for instance and panoptic segmentation tasks by proposing the Boundary AP (Average Precision) and Boundary PQ (Panoptic Quality) metrics, respectively. Our experiments show that the new evaluation metrics track boundary quality improvements that are generally overlooked by current Mask IoU-based evaluation metrics. We hope that the adoption of the new boundary-sensitive evaluation metrics will lead to rapid progress in segmentation methods that improve boundary quality.

1. Introduction

Standard segmentation metrics can underrepresent boundary-quality progress, especially for large objects. The paper analyzes this gap and proposes Boundary IoU with task-level Boundary AP and Boundary PQ metrics.

  • Motivation: 86% relative AP improvement on COCO from 2015 to 2019 included few papers addressing mask boundary quality.The paper links this imbalance to metric sensitivity and the difficulty of quantifying other error types.
  • Motivation: Mask IoU is less sensitive to boundary errors in larger objects because interior pixels grow quadratically while boundary pixels grow linearly.Mask IoU weights all pixels equally, allowing interior area to dominate boundary discrepancies.
  • Analysis: The analysis finds that Mask IoU is less sensitive to large-object errors, while Trimap IoU and F-measure exhibit asymmetry or instability.The study generates varied error types on high-quality LVIS masks across object sizes.
  • Method: Boundary IoU computes IoU over mask pixels within a fixed distance of prediction and ground-truth boundaries, improving boundary sensitivity across scales.The measure is designed to assess large-object boundaries without over-penalizing small-object errors.
  • Task-level metrics: Boundary AP and Boundary PQ extend boundary-sensitive evaluation to instance and panoptic segmentation, respectively.Experiments cover COCO, LVIS, and Cityscapes, using real and synthetic predictions to compare boundary tracking against mask-based metrics.
  • Results: The new metrics reveal boundary-quality improvements that Mask IoU-based evaluations generally ignore.The paper proposes these metrics as a way to support progress toward higher-fidelity mask predictions.

2. Related Work and Preliminaries

Segmentation evaluation measures differ in which pixels they consider and how they respond to mask errors. The paper contrasts mask-based and boundary-based measures and introduces Boundary IoU's unified boundary-region construction.

  • Mask-based measures: Mask IoU divides intersection pixels by union pixels and is symmetric, but its responsiveness to boundary quality is unbalanced across object sizes.It evaluates all mask pixels, so large-object interiors can outweigh boundary errors.
  • Boundary-based measures: Boundary-based measures evaluate contour pixels or nearby regions rather than every pixel in the object mask.This focuses evaluation on contour alignment between prediction and ground truth.
  • Boundary-based measures: Trimap IoU computes IoU within a narrow band around the ground-truth contour, responding similarly to comparable errors across object scales.Its boundary region depends only on the ground-truth mask.
  • Boundary IoU: Boundary IoU computes IoU on pixels within distance d of both prediction and ground-truth contours, using a symmetric boundary-focused construction.Table 1 characterizes measures by symmetry, prediction preference, and error insensitivity.
  • Notation: In the notation, d denotes the pixel width of the boundary region, formed by dilating contour pixels by d pixels.A contour is the one-dimensional set of mask pixels touching the background; the boundary is the resulting two-dimensional region.
  • Boundary-based measures: F-measure combines contour precision and recall, but approximate matching can ignore substantial errors when object size is comparable to d.The approximation allows duplicate matches to reduce the computational cost of high-resolution evaluation.

3. Sensitivity Analysis

The sensitivity analysis compares segmentation measures by applying controlled synthetic errors of varying types and severities to LVIS masks across object sizes. This design enables robust, model-independent analysis while acknowledging that simulated errors may not fully represent real model errors.

  • Method: Sensitivity analysis measures how consistency scores change as controlled errors increase in magnitude.The analysis compares response curves across error severities.
  • Method: Pseudo-predictions are generated from ground-truth annotations to control error type and severity across mask sizes.This avoids bias toward particular segmentation models.
  • Error types: Scale errors use dilation or erosion, with severity controlled by the morphological kernel radius.These perturbations alter mask size around the ground-truth object.
  • Error types: Boundary localization errors add Gaussian noise to polygon vertices, while object localization errors shift masks by controlled pixel offsets.The respective severity controls are Gaussian-noise standard deviation and shift length.
  • Error types: Boundary approximation errors simplify polygons, whereas inner mask errors add random holes controlled by tolerance and hole count.Inner mask errors assess interior defects despite being uncommon in modern segmentation methods.
  • Implementation: The study samples high-quality LVIS validation masks and reports means and standard deviations over pseudo-predictions at each fixed severity.This aggregates results across randomly sampled instances.

4. Analysis of Existing Segmentation Measures

The analysis finds that Mask IoU becomes less sensitive to boundary errors as objects grow, while Trimap IoU and F-measure have symmetry, continuity, or scale-related limitations. These findings motivate Boundary IoU as a measure intended to address the identified shortcomings.

  • Mask IoU: Mask IoU scores a fridge at 0.97 versus a wing at 0.81 despite visually similar annotation discrepancies on equal-resolution crops.Boundary IoU produces closer scores of 0.87 versus 0.81.
  • Mask IoU: Mask IoU tolerates more misclassified pixels per contour-length unit as objects become larger.Interior pixels grow quadratically with scale, while contour pixels grow linearly.
  • Mask IoU: Fixed-pixel dilation or erosion significantly lowers Mask IoU for small objects, while Mask IoU increases with object area.Raising the minimum Mask IoU threshold would not remove this bias and would relatively over-penalize smaller objects.
  • Trimap IoU: Trimap IoU is asymmetric, favors larger predictions, and can ignore errors outside the ground-truth boundary region.Its response differs for dilated and eroded predictions under increasing scale error.
  • F-measure: F-measure can change nearly discontinuously with small error changes and may award significant errors a perfect score when d is large relative to small objects.Its sharp response curves can produce high-variance task metrics.
  • Discussion: The authors conclude that Trimap IoU and F-measure cannot replace Mask IoU as the main measure for broad evaluation metrics, despite Mask IoU’s large-object bias.These limitations motivate the proposed Boundary IoU measure.

5. Boundary IoU

Boundary IoU evaluates agreement between mask regions near the ground-truth and prediction contours, aiming to reduce Mask IoU’s scale bias while retaining sensitivity to boundary errors. Analysis shows improved responsiveness for large objects without over-penalizing small ones, with symmetry advantages over Trimap IoU and softer behavior than F-measure.

  • Definition and design: Boundary IoU computes IoU over original mask pixels within distance d of the ground-truth and prediction contours.It preserves mask pixels near contours rather than computing IoU directly on boundary regions.
  • Scale sensitivity: Boundary IoU reduces Mask IoU’s bias toward large objects across the considered error types.With fixed error severities, its value grows more slowly with object area.
  • Scale sensitivity: For smaller objects, Boundary IoU behaves identically to Mask IoU and avoids over-penalization across the analyzed error types.When d is sufficiently small, interior pixels contribute less, while small objects remain close to Mask IoU because contour-adjacent pixels comprise much of the mask.
  • Comparison with prior measures: Boundary IoU is symmetric and penalizes errors away from the ground-truth boundary, addressing two limitations of Trimap IoU.It considers pixels close to both prediction and ground-truth contours together.
  • Comparison with prior measures: Boundary IoU decreases gradually as contours diverge, unlike F-measure’s hard matching threshold.F-measure treats matches within d as perfect and rejects matches beyond d.
  • Parameter selection: Selecting d requires accounting for annotation consistency and current-method performance, with d decreased as performance improves.The paper suggests annotation consistency as a lower bound for d.
  • Limitation: Boundary IoU can assign a perfect score to non-identical masks when mismatched interior pixels lie farther than d from the boundaries.The paper suggests combining Mask IoU and Boundary IoU by taking their minimum to penalize such cases.

6. Applications

The paper replaces Mask IoU within instance- and panoptic-segmentation evaluation with a boundary-sensitive formulation, yielding Boundary AP and Boundary PQ. Experiments show that Boundary AP tracks boundary-quality improvements in synthetic and real predictions while also retaining sensitivity to localization and categorization improvements.

  • Metric construction: Replacing Mask IoU with min(Mask IoU, Boundary IoU) produces the Boundary AP and Boundary PQ evaluation metrics.The change is intended to improve boundary sensitivity while retaining tracking of other prediction improvements.
  • Implications: The proposed metrics reveal boundary-quality improvements generally ignored by Mask IoU-based evaluations.The paper presents instance-segmentation results in the main text and refers to the appendix for Boundary PQ analysis.
  • Instance segmentation: Boundary AP evaluates categorization, localization, and segmentation quality simultaneously, unlike AF, which ignores false-positive rates.The metric is tested on COCO, LVIS, and Cityscapes.
  • Synthetic predictions: Synthetic 28×28 predictions expose large-object fidelity deficits that Mask APL fails to capture but Boundary APL captures.The synthetic predictions are generated by downscaling and upscaling ground-truth masks before binarization.
  • Real predictions: Boundary AP captures PointRend’s mask-quality improvements more strongly for its higher-resolution output and for large objects.PointRend is evaluated with 28 × 28 or 224 × 224 output resolution against Mask R-CNN’s 28 × 28 output.
  • Real predictions: Boundary AP shows BMask R-CNN outperforming PointRend for small objects, while PointRend’s 224 × 224 output is preferable for larger objects.This pattern matches the paper’s subjective visual-quality assessment.
  • Real predictions: Boundary AP tracks improvements from better localization and categorization similarly to Mask AP when models predict their own boxes.Different backbones do not significantly change either metric when ground-truth boxes are supplied.

7. Conclusion

Boundary IoU provides a boundary-sensitive quality gradient that can support more precise evaluation and analysis of instance and panoptic segmentation errors.

  • Boundary IoU provides a quantitative gradient that rewards improvements in boundary segmentation quality.
  • Boundary IoU enables Boundary AP for instance segmentation and Boundary PQ for panoptic segmentation.
  • Boundary IoU can support more granular analysis of segmentation-related errors in complex instance and panoptic segmentation tasks.

A. Additional Measure Analysis

The additional analysis identifies limitations of existing boundary-focused measures and a limitation of Boundary IoU itself, while illustrating how these measures respond to different mask errors.

  • Trimap IoU under-penalizes inner mask errors because it evaluates only a band around the ground-truth boundary.
  • F-measure can ignore scale-type errors for smaller objects, while mF-measure reduces this limitation by averaging thresholds.
  • Boundary IoU can assign a perfect score to two non-identical masks.The example uses a disk and a ring sharing the same center and outer radius, with the ring’s inner radius d pixels smaller.
  • The proposed mitigation for this Boundary IoU limitation is to use min(Mask IoU, Boundary IoU) in real-world evaluation metrics.

B.1. Instance Segmentation

The instance-segmentation experiments evaluate Boundary AP across several datasets and synthetic or real predictions, showing that it captures boundary-quality differences that Mask AP can miss.

  • The evaluation covers COCO, LVIS, and Cityscapes instance-segmentation datasets.LVIS provides higher-quality ground-truth masks, while Cityscapes contains high-resolution street-scene annotations.
  • Synthetic predictions are generated by downscaling masks to fixed resolutions, upscaling them with bilinear interpolation, and binarizing them.
  • Boundary APL captures reduced fidelity in lower-resolution predictions for large objects, unlike Mask AP.The comparison covers synthetic 28 × 28, 56 × 56, and 112 × 112 predictions.
  • On real predictions, Boundary AP better captures mask-quality improvements across LVIS and Cityscapes evaluations.The comparison includes Mask R-CNN, PointRend, and Boundary-preserving Mask R-CNN using ground-truth boxes.
  • BMask R-CNN outperforms PointRend for smaller objects but trails it for large objects, where PointRend’s 224 × 224 resolution improves boundary quality.

B.2. Panoptic Segmentation

The panoptic-segmentation evaluation replaces Mask IoU with a Boundary IoU-based combination and tests Boundary PQ on COCO panoptic and Cityscapes, including synthetic predictions and reference models.

  • Boundary PQ replaces Mask IoU with min(Mask IoU, Boundary IoU) both for true-positive matching and Segmentation Quality.
  • The panoptic evaluation uses COCO panoptic and Cityscapes, with dilation widths of 2% and 0.5% of the image diagonal, respectively.
  • Synthetic panoptic predictions are generated by downscaling ground-truth maps and upscaling them with nearest-neighbor interpolation.The image-level process treats things and stuff segments uniformly.
  • Boundary PQ better tracks boundary-quality improvements than Mask PQ for synthetic predictions across panoptic datasets.The difference between the metrics is mainly caused by their Segmentation Quality components, while Recognition Quality is comparable.
  • The reference evaluation reports Boundary PQ for various models on COCO panoptic validation and Cityscapes validation sets.
Loading 2103.16562v1…