PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation

Zhaohui Zheng, Ping Wang, Dongwei Ren, Wei Liu, Rongguang Ye, Qinghua Hu, Wangmeng Zuo

arXiv:2005.03572v4cs.CV

TL;DR

Object detection and instance segmentation need better handling of difficult box regressions and redundant detections. The paper introduces CIoU loss and Cluster-NMS using geometric factors, and reports consistent AP and AR gains while maintaining high inference efficiency. It also identifies slower convergence and limited performance in some regression settings as scope boundaries.

  • Problem

    Existing ℓn-norm and IoU-based losses do not fully distinguish difficult regressions, while NMS creates accuracy and inference-efficiency bottlenecks.

  • Method

    CIoU loss incorporates overlap area, normalized central point distance, and aspect ratio, while Cluster-NMS implicitly clusters boxes and incorporates geometric factors during suppression.

  • Results

    CIoU loss yields consistent AP and AR gains against ℓn-norm and IoU-based losses, while Cluster-NMS improves precision and recall with high inference efficiency across detection and segmentation models.

  • Takeaways & Limitations

    Geometric factors can improve both bounding-box regression and NMS across state-of-the-art object-detection and instance-segmentation models.

  • Takeaways & Limitations

    CIoU loss yields lower metrics for some small or median objects, while its improvements are consistent for large objects in the discussed models.

Abstract

from arXiv · show

Deep learning-based object detection and instance segmentation have achieved unprecedented progress. In this paper, we propose Complete-IoU (CIoU) loss and Cluster-NMS for enhancing geometric factors in both bounding box regression and Non-Maximum Suppression (NMS), leading to notable gains of average precision (AP) and average recall (AR), without the sacrifice of inference efficiency. In particular, we consider three geometric factors, i.e., overlap area, normalized central point distance and aspect ratio, which are crucial for measuring bounding box regression in object detection and instance segmentation. The three geometric factors are then incorporated into CIoU loss for better distinguishing difficult regression cases. The training of deep models using CIoU loss results in consistent AP and AR improvements in comparison to widely adopted $\ell_n$-norm loss and IoU-based loss. Furthermore, we propose Cluster-NMS, where NMS during inference is done by implicitly clustering detected boxes and usually requires less iterations. Cluster-NMS is very efficient due to its pure GPU implementation, and geometric factors can be incorporated to improve both AP and AR. In the experiments, CIoU loss and Cluster-NMS have been applied to state-of-the-art instance segmentation (e.g., YOLACT and BlendMask-RT), and object detection (e.g., YOLO v3, SSD and Faster R-CNN) models. Taking YOLACT on MS COCO as an example, our method achieves performance gains as +1.7 AP and +6.2 AR$_{100}$ for object detection, and +0.9 AP and +3.5 AR$_{100}$ for instance segmentation, with 27.1 FPS on one NVIDIA GTX 1080Ti GPU. All the source code and trained models are available at https://github.com/Zzh-tju/CIoU

I. INTRODUCTION

The paper addresses limitations in bounding-box regression and NMS by incorporating geometric factors into both training and inference for object detection and instance segmentation.

  • Motivation: Existing losses struggle to distinguish difficult regression cases, while NMS limits detection accuracy and inference efficiency.The paper identifies these problems in dense-box detection and instance-segmentation pipelines.
  • Contributions: Cluster-NMS implicitly clusters detected boxes, supports geometric factors, and usually requires fewer iterations with GPU-based suppression.Its design targets improved AP and AR while maintaining high inference efficiency.
  • Experiments: CIoU loss and Cluster-NMS were evaluated across YOLACT, BlendMask-RT, YOLO v3, SSD, and Faster R-CNN.The experiments cover both instance segmentation and object detection models.
  • Results: The methods produce consistent AP and AR gains for object detection and instance segmentation while preserving real-time inference.The paper reports these gains across the evaluated model families.
  • Contributions: CIoU loss combines overlap area, normalized central point distance, and aspect ratio to improve bounding-box regression.These factors are formulated to be invariant to regression scale.

II. RELATED WORK

Related work covers bounding-box regression losses and NMS methods, emphasizing tensions among localization quality, detection accuracy, and inference efficiency.

  • Bounding-box regression: Bounding-box regression is used across one-stage, two-stage, and multi-stage object detectors.Representative frameworks include R-CNN, YOLO, and SSD series models.
  • Bounding-box regression: ℓn-norm losses are widely used but are sensitive to varying scales, motivating IoU-based alternatives.Examples include MSE, Smooth-ℓ1, square-root parameterizations, and Huber loss.
  • Non-Maximum Suppression: NMS is a key bottleneck for detection accuracy and inference efficiency in object-detection and instance-segmentation pipelines.Soft-NMS, Weighted-NMS, and related methods improve suppression behavior but can be time-consuming.
  • Non-Maximum Suppression: Fast NMS accelerates inference but can reduce performance because many boxes are over-suppressed.This trade-off motivates the paper’s efficient Cluster-NMS approach.
  • IoU-based losses: IoU loss aligns more closely with the evaluation metric but fails to distinguish non-overlapping boxes.GIoU adds a penalty term and improves precision over MSE and IoU loss in YOLO v3 and Faster R-CNN.

A. Analysis to IoU and GIoU Losses

The simulation study evaluates regression losses across broad combinations of box distance, scale, and aspect ratio, using aggregate regression error over iterations.

  • Simulation Experiment: The experiment uses 7 unit target boxes with aspect ratios from 1:4 through 4:1 and 5,000 uniformly scattered anchor points.Target centers are fixed at (10, 10), while anchor locations vary within a circular region.
  • Simulation Experiment: Each loss drives stochastic-gradient-descent regression, and performance is evaluated using ℓ1-norm error through iteration T = 200.The procedure records regression error across cases and iterations.
  • Simulation Experiment: 1,715,000 regression cases vary bounding-box distance, scale, and aspect ratio.The simulation uses comprehensive geometric configurations rather than uncontrolled detection benchmarks.
  • Simulation Results: The resulting error curves summarize regression error across iterations for the different loss functions.The curves are reported in Fig. 3(b).

2) Limitations of IoU and GIoU Losses:

IoU-based losses incompletely model bounding-box geometry: IoU fails for non-overlapping boxes, while GIoU can converge slowly for extreme aspect ratios. CIoU addresses these cases by combining overlap, normalized center distance, and aspect ratio.

  • IoU loss leaves non-overlapping anchor boxes unmoved because its gradient is always zero.
  • GIoU enlarges the workable regression basin but still produces large errors for extreme aspect ratios and converges slowly.Its enclosure penalty can become small or zero for inclusion relationships, causing behavior close to IoU loss.
  • IoU and GIoU primarily optimize overlap area, whereas a complete regression loss should also model distance and aspect ratio.
  • CIoU combines overlap area, normalized central-point distance, and aspect ratio, with all three factors normalized and scale-invariant.Its aspect-ratio term is downweighted when IoU < 0.5 and becomes necessary when IoU ≥0.5.
  • CIoU can rapidly minimize box distance and converge faster than GIoU, including cases involving containment or extreme aspect ratios.

IV. CLUSTER-NMS

Cluster-NMS replaces sequential suppression with iterative matrix operations over detected boxes, targeting lower iteration counts and GPU-efficient inference. Its procedure starts from score-sorted boxes and updates binary reservation decisions using IoU thresholding.

  • Original NMS sequentially traverses score-sorted boxes to suppress redundant detections, creating an inference bottleneck.
  • Cluster-NMS iteratively computes an IoU matrix, masks suppressed boxes, and suppresses each box when its active overlap exceeds threshold ε.
  • Cluster-NMS performs suppression on implicit clusters and can execute its matrix operations purely on GPU.The design is intended to reduce iterations while preserving efficient inference.
  • The algorithm outputs a binary vector in which 1 reserves a detection and 0 suppresses it.

A. Cluster-NMS

Cluster-NMS iteratively excludes already suppressed boxes from later suppression decisions, avoiding Fast NMS’s over-suppression while retaining parallel matrix execution. Its iteration parameter spans Fast NMS to original NMS behavior.

  • Fast NMS directly suppresses boxes from the IoU matrix and can over-suppress because already suppressed boxes still influence decisions.
  • Cluster-NMS uses the current binary NMS result to exclude suppressed boxes before recomputing suppression.
  • When T = 1, Cluster-NMS becomes Fast NMS; when T = N, it produces the same result as original NMS.
  • Cluster-NMS implicitly groups non-overlapping detections into clusters, performs suppression in parallel between clusters, and generally stops in fewer iterations.
  • Compared with CUDA NMS in TorchVision, Cluster-NMS is slower in the reported implementation because of engineering accelerations, although it can adopt similar optimizations.

B. Incorporating Geometric Factors into Cluster-NMS

Geometric factors extend Cluster-NMS beyond hard IoU suppression to score penalties, center-distance handling, and weighted coordinate updates. These variants target improved precision and recall while retaining parallel efficiency.

  • Geometric factors can be introduced into Cluster-NMS to improve detection precision and recall.
  • Cluster-NMSS applies a Gaussian Soft-NMS-style penalty to scores using overlap with higher-scoring boxes.
  • Normalized central-point distance can be added to Cluster-NMS, producing DIoU-NMS-equivalent behavior or a combined score-penalty variant.
  • Weighted Cluster-NMS updates box coordinates using classification scores and IoU-derived weights rather than deleting redundant boxes.
  • In the illustrated example, 10 detections form 3 clusters, and Cluster-NMS reaches the original NMS result after 2 iterations although the largest cluster contains 4 boxes.
  • Cluster-NMSW matches Weighted-NMS output while being 6.1 times faster, and its distance-aware variant yields consistent AP improvements.

C. Theoretical Analysis

The analysis establishes that Cluster-NMS can reproduce original NMS exactly after sufficient iterations and often converges sooner by processing independent box clusters in parallel.

  • Cluster-NMS after T iterations produces the same final suppression result as original NMS.
  • Binarizing the upper-triangular IoU matrix at threshold ε simplifies the proof without changing Cluster-NMS output.
  • The proof tracks Cluster-NMS subvectors across iterations and matches each update to original NMS at the corresponding iteration.
  • For a largest cluster containing M boxes, processing stabilizes after M iterations because boxes in other clusters do not affect it.
  • Different clusters can be processed in parallel, explaining why Cluster-NMS usually requires fewer iterations.

V. EXPERIMENTAL RESULTS

The experiments evaluate CIoU loss and Cluster-NMS across modern instance-segmentation and object-detection models on PASCAL VOC and MS COCO benchmarks.

  • CIoU loss and Cluster-NMS are evaluated on YOLACT, BlendMask-RT, YOLO v3, SSD and Faster R-CNN.Experiments use PASCAL VOC and MS COCO, with implementations in C/C++ and PyTorch.

A. Instance Segmentation

On YOLACT, CIoU improves regression outcomes over Smooth-ℓ1, while Cluster-NMS incorporates geometric factors to improve AP and AR without sacrificing efficiency.

  • CIoU loss outperforms Smooth-ℓ1 loss on most AP and AR metrics when using the same Fast NMS strategy.
  • Cluster-NMS matches original NMS results while retaining efficiency comparable to Fast NMS.
  • Overlap-based score penalties improve both AP and AR, especially APL and ARL for large objects.
  • Adding normalized distance in Cluster-NMS+D raises AR while leaving AP metrics on par with Cluster-NMSS.
  • Across borrowed and retrained models, Cluster-NMS with geometric factors improves AP and AR, with Cluster-NMSS+D balancing precision and recall through β.

2) BlendMask-RT:

The broader evaluations compare CIoU and Cluster-NMS across detection and segmentation settings, reporting gains while preserving real-time or near-original-NMS efficiency.

  • The evaluation spans instance segmentation and object detection models, including YOLACT, BlendMask-RT and YOLO v3.
  • 1) YOLO v3 [9]: CIoU yields 5.67% AP and 8.95% AP75 gains over the evaluated YOLO v3 loss baselines on PASCAL VOC.The reported improvement is also consistent under GIoU-based evaluation.
  • 1) YOLO v3 [9]: Cluster-NMS is only slightly slower than CUDA-accelerated original NMS, while geometric-factor variants retain real-time inference.
  • 1) YOLO v3 [9]: Cluster-NMSW matches Weighted-NMS results with much higher efficiency, and adding distance further improves especially AR.
  • The SSD experiments compare IoU, GIoU, DIoU and CIoU losses and report AP plus NMS evaluations.

3) Faster R-CNN [15]:

For Faster R-CNN, CIoU provides moderate gains because RPN-filtered boxes already overlap ground truth, while combining CIoU with Cluster-NMS improves all evaluated metrics. The discussion attributes CIoU’s broader effectiveness to jointly modeling geometric factors, with some adverse effects for smaller objects.

  • Faster R-CNN: CIoU produces moderate Faster R-CNN gains because RPN-filtered boxes already overlap their ground-truth boxes.The paper therefore reports less significant AP gains than for YOLO v3 and SSD.
  • Faster R-CNN: Cluster-NMSW +D applied after CIoU training brings further improvements across all evaluation metrics.This combination supports incorporating geometric factors into NMS alongside the regression loss.
  • Discussion: CIoU’s three geometric factors improve regression broadly, but small or median objects can show lower metrics than with alternative losses.The paper attributes this possible adverse effect to the relative importance of central-point distance and aspect ratio at smaller scales.
  • Discussion: For large objects, central-point distance and aspect ratio are both important, producing consistent improvements across evaluated models.Average precision and recall over all objects validate CIoU’s effectiveness, while aspect-ratio treatment remains open for future work.
  • Discussion: Different detectors may require different Cluster-NMS variants, while the framework preserves high inference efficiency and supports geometric factors during inference.The paper identifies Cluster-NMSW +D for YOLO v3 and BlendMask-RT and Cluster-NMSS+D for YOLACT.
  • Conclusion: Across object detection and instance segmentation models, CIoU and Cluster-NMS consistently improve AP and AR while supporting real-time inference.The conclusion describes Cluster-NMS as GPU-based and applicable to state-of-the-art models.
Loading 2005.03572v4…