PaperlayerView exact revision on arXiv

Source-linked AI summary

Point-Set Anchors for Object Detection, Instance Segmentation and Pose Estimation

Fangyun Wei, Xiao Sun, Hongyang Li, Jingdong Wang, Stephen Lin

arXiv:2007.02846v4cs.CV

TL;DR

Center-point regression can lack informative features for distant targets under object deformation and scale or orientation variation. The paper introduces task-specific Point-Set Anchors and applies them across detection, instance segmentation, and pose estimation, achieving performance competitive with state-of-the-art methods.

  • Problem

    Center-point features may provide limited information for predicting distant keypoints or box boundaries under object deformation and scale or orientation variation.

  • Method

    Point-Set Anchors place task-specific point sets near regression targets, sample their transformations like anchor boxes, and support keypoint regression in PointSetNet.

  • Results

    The approach achieves performance competitive with state-of-the-art methods for object detection, instance segmentation, and human pose estimation.

  • Takeaways & Limitations

    Point-set anchors provide informative features and task-specific initializations for keypoint regression across three high-level recognition tasks.

  • Takeaways & Limitations

    Instance-segmentation evaluation calculates IoU from predicted bounding boxes and ground-truth rectangles because mask IoU is computationally expensive.

Abstract

from arXiv · show

A recent approach for object detection and human pose estimation is to regress bounding boxes or human keypoints from a central point on the object or person. While this center-point regression is simple and efficient, we argue that the image features extracted at a central point contain limited information for predicting distant keypoints or bounding box boundaries, due to object deformation and scale/orientation variation. To facilitate inference, we propose to instead perform regression from a set of points placed at more advantageous positions. This point set is arranged to reflect a good initialization for the given task, such as modes in the training data for pose estimation, which lie closer to the ground truth than the central point and provide more informative features for regression. As the utility of a point set depends on how well its scale, aspect ratio and rotation matches the target, we adopt the anchor box technique of sampling these transformations to generate additional point-set candidates. We apply this proposed framework, called Point-Set Anchors, to object detection, instance segmentation, and human pose estimation. Our results show that this general-purpose approach can achieve performance competitive with state-of-the-art methods for each of these tasks. Code is available at \url{https://github.com/FangyunWei/PointSetAnchor}

1 Introduction

The paper argues that center-point regression can lack informative features for distant targets under geometric variation, and proposes task-specific point sets for keypoint regression. Point-set anchors extend this idea through sampled transformations, supporting a general network applied across three recognition tasks.

  • Center-point features may provide little information for distant keypoints because objects vary in scale, orientation, and deformation.
  • Task-specific point sets place features closer to regression targets, using implicit box edges for segmentation and pose-distribution modes for pose estimation.
  • Point-set anchors generalize classical box anchors by sampling point sets across scales, aspect ratios, positions, and configurations.
  • PointSetNet modifies RetinaNet by replacing anchor boxes with point-set anchors and adding a parallel keypoint-regression branch.
  • Performance is competitive with state-of-the-art methods for object detection, instance segmentation, and pose estimation.

2 Related Work

Prior object representations range from box anchors and center points to specialized point configurations, but their task coverage and localization precision differ. The paper presents point-set anchors as a flexible representation illustrated across detection, segmentation, and pose estimation.

  • Rectangular anchors provide initial boxes for localization but are coarse for the finer localization required by instance segmentation and pose estimation.
  • Point-set anchors use implicit bounding-box samples for instance masks and frequent training-set poses for pose estimation.
  • Center-point representations flexibly predict offsets to box corners or human joints, but the paper argues their features are less informative than task-specific point-set features.
  • Polygon-based representations such as corner and extreme points are difficult to apply across instance segmentation and pose estimation because of restricted shape.
  • ExtremeNet detects extreme and center points before grouping, while Deep Snake uses a two-stage pipeline based on initial contours and contour deformation.

3 Our Method

The method defines task-specific point-set anchors and regresses object shapes from their offsets rather than from a single center point. PointSetNet extends RetinaNet with point-set anchors, shared prediction heads, and task-specific regression for detection, segmentation, and pose estimation.

  • 3.1 Point-Set Anchors: Point-set anchors contain ordered points defined for a specific task, such as pose keypoints or points sampled along an implicit mask bounding box.Pose anchors are initialized from frequent training poses, while mask anchors vary scale and aspect ratio.
  • 3.2 Shape Regression: The framework represents detection, segmentation, and pose estimation as shape regression from anchor points to ordered target points.Offsets are computed as target shape points minus the corresponding point-set anchor locations.
  • 3.2 Shape Regression: Instance-mask offsets use one-to-one anchor-to-contour correspondences, with Nearest Point, Nearest Line, and Corner Point with Projection as matching strategies.Corner Point with Projection subdivides the contour into top, right, bottom, and left parts and matches points using projection intersections.
  • 3.2 Shape Regression: Positive and negative point-set anchors are assigned using IoU for detection and segmentation, and OKS for pose estimation.The supplied method description specifies IoU thresholds of 0.6 for positive and 0.4 for negative assignments, while pose uses OKS thresholds of 0.5 and 0.4.
  • 3.3 PointSetNet: PointSetNet replaces RetinaNet’s rectangular anchors with point-set anchors and adds parallel mask or pose regression heads over a multi-scale feature pyramid.The architecture uses feature-map levels P3 through P7 and separate classification, mask or pose regression, and bounding-box regression subnetworks.
  • 3.3 PointSetNet: Training uses focal classification loss and L1 shape-regression loss, with point-set anchors sampled across scales and aspect ratios.The anchor design generates nine bounding-box configurations per feature-map location, while the loss balance weight differs between instance segmentation and pose estimation.

4 Experiments

Experiments evaluate Point-Set Anchors across instance segmentation, object detection, and pose estimation, including anchor design, feature extraction, refinement, and backbone choices. The results show that task-aligned anchors and shape-indexed features improve regression, while the full approach remains competitive with state-of-the-art methods.

  • Experimental setup: PointSetNet is evaluated on MS-COCO for instance segmentation, object detection, and multi-person human pose estimation.Instance segmentation and detection use COCO trainval35k for training, minival for ablations, and test-dev for state-of-the-art comparisons; pose experiments use COCO train2017.
  • State-of-the-art comparison: PointSetNet achieves performance competitive with state-of-the-art methods in object detection and instance segmentation, although its instance-segmentation AP is 0.2 lower than PolarMask.The reported configuration uses 60 contour anchor points, ResNeXt-101 with DCN, extended training, and multi-scale inference.
  • Pose anchor design: Mean pose anchors surpass center-point anchors by 24 AP and rectangle anchors by 28.9 AP.The comparison assigns ground-truth poses to anchors using an OKS threshold of 0.5 when possible.
  • Pose anchor design: Additional pose shapes, scale transformations, and rotation transformations improve performance by +2.9 AP, +1.7 AP, and +2.6 AP, respectively.More anchor shapes and transformations also introduce more negative anchors, making learning less efficient.
  • Pose feature extraction: Deep shape indexed features increase AP from 42.8 to 46.0, a +3.2 AP improvement, when using nine pose-based feature points.The improvement is greater when the feature is used in pose regression than when used only for person classification.
  • Pose model configuration: ResNeXt-101-DCN, multi-scale testing, and HRNet improve pose performance by 4.5 AP, 3.2 AP, and 4.1 AP, respectively.With HRNet and multi-scale testing, PointSetNet outperforms CenterNet by 5.7 AP on COCO test-dev2017.

5 Conclusion

The paper presents Point-Set Anchors as a generalization of classical anchors for high-level recognition tasks. PointSetNet applies them across object detection, instance segmentation, and human pose estimation, achieving competitive experimental results.

  • Contribution: Point-Set Anchors generalize and extend classical anchors for instance segmentation and pose estimation.They provide informative features and task-specific initializations for keypoint regression.
  • Architecture: PointSetNet replaces RetinaNet anchor boxes with point-set anchors and adds a parallel branch for keypoint regression.Variants are applied to object detection, instance segmentation, and human pose estimation.
  • Conclusion: Competitive experimental results across three tasks demonstrate the generality of point-set anchors.The evaluated tasks are object detection, instance segmentation, and human pose estimation.
Loading 2007.02846v4…