Source-linked AI summary
Bottom-up Object Detection by Grouping Extreme and Center Points
Xingyi Zhou, Jiacheng Zhuo, Philipp Krähenbühl
TL;DR
Object detection has largely shifted toward top-down region enumeration and classification, despite limitations of rectangular boxes. ExtremeNet instead estimates four extreme points and a center, groups them geometrically, and achieves competitive detection and segmentation results. The approach retains bottom-up recognition as a viable alternative, though its brute-force grouping is computationally costly and can produce ghost boxes.
Problem
Top-down detection relies on rectangular region enumeration and classification, even though boxes include distracting background and are not natural representations for most objects.
Method
ExtremeNet predicts four category-specific extreme-point heatmaps and a center heatmap, then groups point combinations when their geometric center has a high center-map response.
Results
The method achieves 43.7% bounding box AP on COCO test-dev, outperforming reported one-stage detectors and matching sophisticated two-stage detectors.
Takeaways & Limitations
Bottom-up extreme-point estimation yields state-of-the-art detection and competitive instance segmentation without using COCO training instance segmentations.
Takeaways & Limitations
Brute-force center grouping costs O(n^4), and equally spaced colinear objects can produce infrequent but consistent high-confidence ghost-box errors.
Abstract
from arXiv · showhide
With the advent of deep learning, object detection drifted from a bottom-up to a top-down recognition problem. State of the art algorithms enumerate a near-exhaustive list of object locations and classify each into: object or not. In this paper, we show that bottom-up approaches still perform competitively. We detect four extreme points (top-most, left-most, bottom-most, right-most) and one center point of objects using a standard keypoint estimation network. We group the five keypoints into a bounding box if they are geometrically aligned. Object detection is then a purely appearance-based keypoint estimation problem, without region classification or implicit feature learning. The proposed method performs on-par with the state-of-the-art region based detection methods, with a bounding box AP of 43.2% on COCO test-dev. In addition, our estimated extreme points directly span a coarse octagonal mask, with a COCO Mask AP of 18.9%, much better than the Mask AP of vanilla bounding boxes. Extreme point guided segmentation further improves this to 34.6% Mask AP.
1. Introduction
ExtremeNet reframes detection as bottom-up estimation of four extreme points and one center point, grouped geometrically rather than through region classification. The approach remains competitive with established detectors and supports coarse or guided segmentation from extreme points.
- Motivation: Top-down detectors classify many rectangular regions, although boxes include distracting background and do not naturally represent most objects.The paper also critiques exhaustive location enumeration as lacking compositional visual understanding.
- Method: ExtremeNet predicts top-most, left-most, bottom-most, and right-most points plus an object center for each category.It uses multi-peak extreme-point heatmaps and a center heatmap.
- Method: Extreme points are grouped into detections if their geometric center receives a sufficiently high response in the center heatmap.The method enumerates O(n^4) combinations of extreme-point predictions.
- Design rationale: Extreme points lie on objects and have more consistent local appearance than corners, while geometric grouping avoids implicit feature learning.The paper reports that appearance-based grouping works significantly better than associative embedding in its experiments.
- Segmentation: Extreme points directly support a tighter octagonal object approximation and can guide DEXTR toward close-to-state-of-the-art instance segmentation.The octagonal approximation is described as more fine-grained than a bounding box.
- Results: 43.7% bounding box AP on COCO test-dev outperforms reported one-stage detectors and is on-par with sophisticated two-stage detectors.A Pascal VOC-pretrained DEXTR model reaches 34.6% Mask AP without COCO mask annotations.
2. Related Work
The paper places ExtremeNet among two-stage, one-stage, deformable-part, and bottom-up pose-based detectors. Its distinguishing choice is to detect parts in linear image-location space and group them using fixed geometry and center appearance.
- Two-stage detection: Two-stage detectors generate category-agnostic box proposals, crop regions or features, and classify them sequentially.
- One-stage detection: One-stage detectors assign labels directly to positive anchors, using region or anchor proposal mechanisms without a separate cropping module.
- ExtremeNet: ExtremeNet predicts four extreme points and one center in O(hw) space instead of anchors in O(h^2w^2) space.Its center map is scale- and aspect-ratio-agnostic and does not use bounding-box regression.
- Part-based detection: Unlike deformable part models, ExtremeNet uses fixed geometry for its center and four universal extreme-point parts with a keypoint network.
- Keypoint grouping: Bottom-up pose methods learn embeddings, affinities, or offsets to group keypoints from the same person.
- ExtremeNet: ExtremeNet instead performs pure appearance-based center grouping by exploiting the geometric structure of extreme points and their center.
3. Preliminaries
The preliminaries define extreme-point annotations, keypoint heatmap estimation, CornerNet’s corner-based alternative, and DEXTR’s extreme-point-guided segmentation.
- Extreme and center points: Bounding-box annotation conventionally uses top-left and bottom-right corners, which may lie outside the object and require adjustment.
- Extreme and center points: Extreme-point annotation clicks the top-most, left-most, bottom-most, and right-most object points, taking 7.2 seconds on average versus 34.5 seconds for box annotation.The resulting annotation is reported as on-par with the more time-consuming box annotation.
- Keypoint detection: Keypoint estimation predicts multi-channel heatmaps with fully convolutional encoder-decoder networks and trains them using supervised pixelwise losses.
- CornerNet: CornerNet predicts opposing box-corner heatmaps and uses a modified focal loss with α = 2 and β = 4.
- CornerNet: CornerNet regresses category-agnostic offsets with Smooth L1 loss to recover information lost during network down-sampling.The down-sampling factor is s = 4 for HourglassNet.
- Deep Extreme Cut: DEXTR takes four extreme points and a surrounding cropped image region to produce a category-agnostic foreground segmentation mask.
4. ExtremeNet for Object detection
ExtremeNet detects four extreme points and a center per class, then forms detections through geometric consistency rather than region classification. The method also addresses grouping failure modes and uses extreme points to approximate object masks.
- Keypoint detection: ExtremeNet predicts four extreme points and one center per class using 5 × C heatmaps and 4 × 2 offset maps.The center has no offset prediction, while extreme-point offsets are category-agnostic but extreme-point specific.
- Center grouping: Candidate extreme points are extracted as local heatmap peaks, and quadruples are retained when their geometry is valid and their computed center scores above τc.The method uses τp = 0.1 and τc = 0.1 in all experiments.
- Center grouping: The brute-force grouping procedure enumerates extreme-point quadruples for each class, with runtime O(n^4), although an O(n^2) alternative is slower in practice on MS COCO.The alternative is harder to accelerate on a GPU, while the brute-force method is practical when n ≤40.
- Ghost box suppression: Center grouping can create high-confidence ghost boxes for three equally spaced colinear objects of the same size.Soft non-maximum suppression halves a box score when contained detections have a combined score exceeding three times its own score.
- Edge aggregation: Edge aggregation handles non-unique extreme points by collecting responses along aligned object edges, strengthening the middle pixel's confidence.Without aggregation, weak edge responses may fall below τp or score below responses from slightly rotated objects.
- Mask estimation: Extreme points support a coarse octagonal object-mask approximation by extending each point along its corresponding edge segment.The construction uses segments of one-quarter of the entire edge length, truncated when they meet a corner.
5. Experiments
Experiments evaluate ExtremeNet on COCO through implementation details, ablations, object-detection comparisons, and instance-segmentation tests. Results show that geometric grouping and edge aggregation contribute to detection performance, while extreme points support coarse and DEXTR-refined masks.
- Experimental setup: ExtremeNet is evaluated on COCO train2017, val2017, and test-dev using average precision across IoU thresholds.The training split contains 118k images and 860k objects; ablations use val2017 and comparisons use test-dev.
- Experimental setup: The implementation follows CornerNet’s setup with 511 × 511 inputs, 128 × 128 outputs, Adam optimization, and extensive augmentation.Testing additionally uses flip and five-scale augmentation followed by Soft-NMS.
- Ablation studies: A 2.1% AP drop occurs when associative embedding replaces geometric center grouping.The drop is especially evident for medium and large objects, whereas geometric grouping avoids relying on object identity from extreme-point features.
- Ablation studies: 0.7% AP improvement comes from edge aggregation, while removing it reduces decoding time to 76ms and increases speed to 4.1 FPS.Edge aggregation is more effective for larger objects with long axis-aligned edges.
- Ablation studies: 0.3% AP improvement comes from simple ghost-box suppression, indicating ghost boxes are not a significant practical issue on MS COCO.The suppression penalizes boxes containing smaller detections whose summed scores exceed three times the candidate box’s score.
- State-of-the-art comparisons: 43.7% AP on COCO test-dev is on par with sophisticated two-stage detectors and higher than reported one-stage detectors in the multi-scale comparison.ExtremeNet is 1.6% AP higher than CornerNet, but single-scale performance is 0.3% AP lower; performance is stronger for small and medium objects than for large ones.
- Instance segmentation: 18.9% Mask AP from the predicted extreme-point octagon exceeds the 12.1% bounding-box baseline, while DEXTR raises Mask AP to 34.6%.The DEXTR result uses a model without COCO mask annotations; the result is 2.9% AP below ResNeXt-101-FPN and on par with Res50-FPN.
6. Conclusion
The paper presents a bottom-up detector based on estimating and geometrically grouping extreme points. On MSCOCO, it achieves state-of-the-art detection and competitive instance segmentation without using COCO training instance segmentations.
- The framework estimates four extreme points and groups them using purely geometric reasoning.
- The method achieves state-of-the-art detection results and competitive instance segmentation results on MSCOCO.
- The reported results use no COCO training instance segmentations.