Source-linked AI summary
RepPoints: Point Set Representation for Object Detection
Ze Yang, Shaohui Liu, Han Hu, Liwei Wang, Stephen Lin
TL;DR
Object detectors rely on convenient rectangular boxes, but these provide coarse localization and can yield lower-quality features. RepPoints introduce adaptively learned point sets for localization and recognition, replacing box representations in an anchor-free detector. The resulting system achieves 46.5 AP and 67.4 AP50 on COCO with ResNet-101 and multi-scale training and testing.
Problem
Rectangular bounding boxes provide coarse localization that ignores object shape, pose, and semantically important areas, potentially degrading extracted feature quality.
Method
RepPoints learn a top-down set of points from localization and recognition targets to bound objects and identify semantically significant local areas without anchors.
Results
46.5 AP and 67.4 AP50 are achieved on the COCO benchmark with multi-scale training and testing using a ResNet-101 model.
Takeaways & Limitations
RepPoints provide a finer object representation that supports localization and recognition while enabling an anchor-free detector competitive with anchor-based methods.
Takeaways & Limitations
Bounding-box regression performs poorly when the initial representation is far from the target, and center-point initialization can face recognition-target ambiguity for overlapping centers.
Abstract
from arXiv · showhide
Modern object detectors rely heavily on rectangular bounding boxes, such as anchors, proposals and the final predictions, to represent objects at various recognition stages. The bounding box is convenient to use but provides only a coarse localization of objects and leads to a correspondingly coarse extraction of object features. In this paper, we present \textbf{RepPoints} (representative points), a new finer representation of objects as a set of sample points useful for both localization and recognition. Given ground truth localization and recognition targets for training, RepPoints learn to automatically arrange themselves in a manner that bounds the spatial extent of an object and indicates semantically significant local areas. They furthermore do not require the use of anchors to sample a space of bounding boxes. We show that an anchor-free object detector based on RepPoints can be as effective as the state-of-the-art anchor-based detection methods, with 46.5 AP and 67.4 $AP_{50}$ on the COCO test-dev detection benchmark, using ResNet-101 model. Code is available at https://github.com/microsoft/RepPoints.
1. Introduction
Bounding boxes are convenient throughout object-detection pipelines but provide coarse localization and can contaminate features with background or uninformative regions. RepPoints replace them with adaptively learned points that support finer localization, semantic feature extraction, and anchor-free detection.
- Motivation: Bounding boxes support anchors, proposals, final predictions, and regular-grid feature extraction, but their convenience comes with coarse object representation.They are widely used partly because detection metrics evaluate box overlap and rectangular regions are easy to subdivide.
- Motivation: Coarse boxes can ignore object shape, pose, and semantic local areas, causing background-influenced features and degraded classification quality.Regular cells inside a rectangle may cover uninformative foreground or background content.
- RepPoints: RepPoints represent objects as adaptively positioned points that circumscribe spatial extent and identify semantically significant local areas.The representation is learned from localization and recognition targets, using ground-truth boxes for localization supervision.
- RepPoints: RepPoints are learned top-down and end-to-end without additional supervision, unlike bottom-up representations requiring handcrafted clustering or object masks.This distinction provides fine-grained localization while avoiding the construction procedures used by several prior non-rectangular representations.
- Results: 46.5 AP and 67.4 AP50 are achieved on COCO with multi-scale training and testing using a ResNet-101 model.Without multi-scale training and testing, the detector achieves 42.8 AP and 65.0 AP50 on COCO.
2. Related Work
Prior object-detection representations largely use regular rectangular boxes or bottom-up point compositions, each imposing limitations on geometric flexibility or processing. RepPoints instead learns a flexible top-down representation that integrates with feature extraction and replaces boxes throughout detector stages.
- Bounding-box representations: Bounding boxes dominate detector stages because they are easy to annotate, align with benchmark evaluation, and support regular-grid feature extraction.They appear as anchors, proposals, and final localization targets.
- Implementation: Irregular RepPoints remain compatible with feature extraction through deformable convolution and with box-based benchmarks through a generated rectangular pseudo box.The pseudo box is obtained from RepPoints for training and evaluation compatibility.
- RepPoints: RepPoints replaces anchors, proposals, and final localization targets, using center points initially to produce an anchor-free detector.RepPoints proposals and final targets replace the corresponding bounding-box representations.
- Other representations: Bottom-up point methods compose objects from detected points but may rely on handcrafted clustering or post-processing steps.Corner-based approaches predict parts such as object corners before grouping them into whole-object boxes.
- RepPoints: RepPoints is constructed top-down without handcrafted clustering and learns extreme or semantic points using only ground-truth bounding-box supervision.This contrasts with methods such as ExtremeNet that require additional mask supervision.
- Deformation modeling: RepPoints targets geometric localization as well as semantic feature extraction, whereas deformable convolution and deformable RoI pooling are designed for feature extraction only.The paper specifically distinguishes RepPoints from these top-down deformation-modeling approaches.
3. The RepPoints Representation
RepPoints replaces coarse rectangular object representations with adaptive sample points, while retaining differentiable conversion to pseudo boxes for training and evaluation.
- Bounding Box Representation: Bounding-box regression performs poorly when the initial representation is far from the target and requires tuning loss weights because parameter scales differ.
- Bounding Box Representation: Bounding boxes encode object location compactly but represent only rectangular scope, omitting shape, pose, and semantically important local areas.
- RepPoints: RepPoints models an object as n adaptive sample points, with n set to 9 by default in this work.
- RepPoints: A differentiable transformation T converts RepPoints into a pseudo box, enabling use of ground-truth box annotations and standard detection evaluation.
- RepPoints: The paper considers min-max, partial min-max, and moment-based conversions for obtaining pseudo boxes from RepPoints.
- RepPoints: RepPoints learning combines localization and recognition losses, using pseudo-box distance to ground truth for localization.
4. RPDet: an Anchor Free Detector
RPDet uses RepPoints throughout an anchor-free multi-stage detector, beginning with center points and refining sample points for localization and recognition. Its design combines deformable convolution with point-based representation while reducing computational burden relative to RetinaNet.
- Detector Design: RPDet replaces conventional bounding-box representations with RepPoints throughout its multi-stage detection pipeline.
- Detector Design: Deformable convolution supports RPDet recognition by aggregating features at irregular sample points, while recognition feedback guides point positioning.
- Initial Representation: Center points provide the initial representation, making RPDet anchor-free and reducing the initial hypothesis space from four dimensions to two.
- Initial Representation: Only 1.1% of COCO objects have coincident center points when FPN is used, according to the paper’s observation.
- RepPoints Refinement: The first RepPoints set is refined from center points using localization and subsequent recognition objectives, while the second set targets finer localization.
- Relation to Deformable RoI Pooling: RepPoints is a geometric representation for semantic localization, whereas deformable RoI pooling targets stronger appearance features and can complement it.
- Efficiency: 210.9 vs. 234.5 GFLOPS: RPDet is reported as more efficient than one-stage RetinaNet with ResNet-50 despite using two localization stages.
5. Experiments
Experiments evaluate RepPoints against bounding-box representations, alternative supervision sources, anchor-based initialization, and state-of-the-art detectors on COCO. RepPoints improve detection accuracy, support anchor-free initialization, and complement deformable feature extraction.
- RepPoints vs. bounding box: 36.2 mAP is achieved by the bounding-box baseline, while RepPoints improve performance by +2.1 mAP with ResNet-50 and +2.0 mAP with ResNet-101.The compared detectors use the same network structures except for their object representations.
- Supervision source for RepPoints learning: Removing object-localization supervision causes a 4.5 mAP drop, showing that geometric localization is important for learning RepPoints.RPDet learns RepPoints using both localization and recognition losses.
- Supervision source for RepPoints learning: +0.7 mAP results from adding object-recognition loss to RepPoints learning, which places points at semantically meaningful object positions.The recognition feedback improves feature extraction for the subsequent recognition stage but does not benefit the bounding-box representation.
- Anchor-free vs. anchor-based: +1.1 mAP and +1.4 mAP are obtained by center-point initialization over single-anchor initialization for bounding-box and RepPoints detectors, respectively.Center points are a special RepPoints configuration used as the initial object representation.
- RepPoints and deformable RoI pooling: +0.8 mAP is obtained by applying deformable RoI pooling to RepPoints proposals, compared with +0.7 mAP for bounding-box proposals.The results indicate that deformable RoI pooling and RepPoints provide complementary benefits.
6. Conclusion
RepPoints represents objects with fine-grained localization information while identifying local areas significant for classification. RPDet achieves competitive detection performance without anchors.
- RepPoints models fine-grained localization information and identifies local areas significant for object classification.
- RPDet achieves competitive object detection performance without requiring anchors.
- Learning richer and more natural object representations like RepPoints is presented as a promising direction for object detection.
A1. Relationship between RepPoints and Deformable RoI pooling
The analysis distinguishes RepPoints from deformable RoI pooling by showing that proposal-based sampling remains tied to proposal geometry, whereas RepPoints can localize objects under localization supervision.
- Translation Sensitivity: Bounding-box regression learns to transform a proposal BP into the ground-truth box BGT from features pooled within BP.The pooling function may use direct cropping, RoIPooling, or RoIAlign.
- Translation Sensitivity: Pooled features should change when the proposal changes, making conventional proposal-based feature extraction translation-sensitive.The passage notes that RoIAlign’s improvement over RoIPooling is partly due to guaranteed translation sensitivity.
- Analysis of Deformable RoI Pooling: Deformable RoI pooling generates proposal-specific sample points and uses their pooled features for regression.A deformation function produces sample points SP, while a regression function uses them to predict TP.
- Analysis of Deformable RoI Pooling: Deformable RoI pooling cannot provide a geometric object representation because its sampled features must remain sensitive to differing proposals.Figure 5 illustrates the contradiction arising when similar object geometry is assumed across different proposals.
- Analysis of Deformable RoI Pooling: Figure 6 shows that deformable RoI pooling’s learned sample-point scale changes with proposal scale rather than adapting to object geometry.
- RepPoints: RepPoints features localize the object rather than representing the original proposal, enabled by localization supervision on the corresponding pseudo box.The paper states that localization supervision on sample points is crucial for RepPoints and contrasts this with deformable RoI pooling.
A2. More Benchmark Results for RPDet
The paper reports additional RPDet benchmark results on the MS-COCO validation set, using FPN models trained under the 2x setting with multiple backbone and testing configurations.
- RPDet benchmark results are reported on the MS-COCO validation set, also called minival.
- Multi-scale training randomly selects the image shorter side from [480, 960] for each mini-batch.
- Multi-scale testing is included among the reported evaluation settings.