PaperlayerView exact revision on arXiv

Source-linked AI summary

Objects as Points

Xingyi Zhou, Dequan Wang, Philipp Krähenbühl

arXiv:1904.07850v2cs.CV

TL;DR

Conventional detectors enumerate many candidate boxes and require non-differentiable post-processing, motivating a simpler alternative. CenterNet detects object centers as points and regresses other properties, achieving a strong speed-accuracy trade-off while performing competitively on 3D detection and human pose.

  • Problem

    Existing detectors enumerate many candidate object boxes and use non-differentiable post-processing, making detection wasteful and difficult to train end-to-end.

  • Method

    CenterNet represents each object by its bounding-box center, uses keypoint estimation to find centers, and directly regresses object properties without NMS post-processing.

  • Results

    CenterNet achieves a strong speed-accuracy trade-off on COCO and performs competitively for 3D bounding-box estimation and human pose, reaching 28.1% COCO AP at 142 FPS.

  • Takeaways & Limitations

    Point-based detection supports fast, end-to-end object recognition while extending to properties such as pose, 3D orientation, depth, and extent in one forward pass.

  • Takeaways & Limitations

    If two objects perfectly align and share the same center, CenterNet detects only one of them.

Abstract

from arXiv · show

Detection identifies objects as axis-aligned boxes in an image. Most successful object detectors enumerate a nearly exhaustive list of potential object locations and classify each. This is wasteful, inefficient, and requires additional post-processing. In this paper, we take a different approach. We model an object as a single point --- the center point of its bounding box. Our detector uses keypoint estimation to find center points and regresses to all other object properties, such as size, 3D location, orientation, and even pose. Our center point based approach, CenterNet, is end-to-end differentiable, simpler, faster, and more accurate than corresponding bounding box based detectors. CenterNet achieves the best speed-accuracy trade-off on the MS COCO dataset, with 28.1% AP at 142 FPS, 37.4% AP at 52 FPS, and 45.1% AP with multi-scale testing at 1.4 FPS. We use the same approach to estimate 3D bounding box in the KITTI benchmark and human pose on the COCO keypoint dataset. Our method performs competitively with sophisticated multi-stage methods and runs in real-time.

1. Introduction

CenterNet replaces exhaustive bounding-box detection with center-point keypoint estimation and direct regression of object properties. The approach is simple, efficient, extensible to 3D detection and pose estimation, and achieves strong COCO speed-accuracy results.

  • Motivation: Bounding-box IoU post-processing removes duplicate detections but is hard to differentiate and train, leaving most current detectors not end-to-end trainable.The paper motivates CenterNet as a simpler alternative to this pipeline.
  • Method: CenterNet represents each object as a single bounding-box center point instead of enumerating many candidate boxes.This reframes detection as keypoint estimation and avoids the wastefulness of sliding-window enumeration.
  • Method: Object size, dimensions, 3D extent, orientation, and pose are regressed directly from image features at each center location.A fully convolutional network produces a heatmap whose peaks identify center points.
  • Extensions: CenterNet extends to 3D object detection and multi-person pose estimation by predicting additional outputs at each center point.For 3D detection it predicts depth, dimensions, and orientation; for pose estimation it regresses 2D joint offsets.
  • Results: 142 FPS with 28.1% COCO bounding box AP is achieved using ResNet-18 and up-convolutional layers.A DLA-34 network reaches 37.4% COCO AP at 52 FPS, while Hourglass-104 with multi-scale testing reaches 45.1% COCO AP at 1.4 FPS.

2. Related work

Prior detectors rely on region proposals, anchors, or multiple keypoints, often requiring additional classification, grouping, or suppression stages. CenterNet instead uses a location-based, shape-agnostic center point with one positive anchor per object, avoiding manual overlap thresholds and NMS.

  • Object detection by region classification: RCNN and Fast-RCNN classify region candidates, but both depend on slow low-level region proposal methods.Fast-RCNN reduces computation by cropping image features rather than image regions.
  • Object detection with implicit anchors: Faster-RCNN samples fixed-shape anchors on a low-resolution grid and labels them using ground-truth overlap thresholds.Anchors are foreground above 0.7 overlap, background below 0.3, and ignored otherwise; generated proposals are classified again.
  • Object detection with implicit anchors: One-stage detectors replace proposal classification with multi-class classification and improve performance through shape priors, feature resolution, and loss re-weighting.These methods include anchor shape priors, different feature resolutions, and re-weighting among samples.
  • CenterNet’s anchor distinction: CenterNet treats each center point as a single shape-agnostic anchor assigned by location rather than box overlap, with one positive anchor per object and no NMS.This design also removes manual foreground-background thresholds.
  • Object detection by keypoint estimation: CornerNet and ExtremeNet use keypoints for detection but require combinatorial grouping after keypoint detection, significantly slowing both algorithms.CornerNet detects two box corners, while ExtremeNet detects four extreme points and centers; both use the same robust keypoint estimation network as CenterNet.

3. Preliminary

The preliminary model predicts keypoint heatmaps from images using fully convolutional encoder-decoder networks, with keypoint types covering human joints or object categories. Training uses Gaussian heatmap supervision and focal loss, while local offsets recover output-stride discretization error.

  • Keypoint Representation: The network maps an input image to a keypoint heatmap with output stride R = 4, where heatmap values indicate detected keypoints or background.The heatmap has C channels for keypoint types, including 17 human joints or 80 object categories.
  • Keypoint Representation: Keypoint types include C = 17 human joints for pose estimation and C = 80 object categories for object detection.The output stride downsamples predictions by a factor R.
  • Heatmap Supervision: Ground-truth keypoints are downsampled and rendered as object-size-adaptive Gaussians, with overlapping same-class Gaussians combined by element-wise maximum.The construction follows Law and Deng, and supervision is defined on the resulting heatmap.
  • Heatmap Supervision: The heatmap is trained with penalty-reduced pixelwise logistic regression using focal loss, normalized by the number of keypoints, with α = 2 and β = 4.The normalization makes all positive focal-loss instances sum to 1.
  • Offset Prediction: A shared local offset prediction for each center point is trained with L1 loss to correct discretization error caused by the output stride.Offset supervision applies at keypoint locations, while other locations are ignored.

4. Objects as Points

CenterNet represents each object by its bounding-box center, predicts object size from that center, and reconstructs boxes without IoU-based NMS or other post-processing. The same center-point framework extends to 3D detection and human pose by regressing task-specific properties.

  • Objects as Points: CenterNet detects each object as a single bounding-box center point and regresses its size from that point.A keypoint estimator predicts center points, while a size prediction provides each object’s width and height.
  • From points to bounding boxes: Peak extraction keeps the top 100 responses per category, then combines center offsets and size predictions to form bounding boxes.The method treats keypoint confidence as detection confidence and extracts local maxima independently for each category.
  • 3D detection: 3D detection adds separate heads for depth, three-dimensional dimensions, and orientation at each center point.Depth uses an inverse-sigmoid transformation, dimensions are regressed in meters, and orientation uses two bins with in-bin regression.
  • Human pose estimation: Human pose estimation treats k = 17 COCO joint locations as center-point properties, regressing joint offsets and refining them with joint heatmaps.Center offsets group detected joints with person instances, and invisible keypoints are excluded from the regression loss.

5. Implementation details

The implementation evaluates four network architectures, modifies ResNet and DLA-34 with deformable convolutions, and uses Hourglass-104 unchanged. Training uses 512 × 512 inputs with augmentation and Adam, while inference varies augmentation to trade speed for accuracy.

  • Architectures: The experiments use ResNet-18, ResNet-101, DLA-34, and Hourglass-104; ResNets and DLA-34 use deformable convolutions, while Hourglass is unchanged.Hourglass-104 generally provides the best keypoint estimation performance but is relatively large.
  • Architectures: ResNet uses three reduced-channel up-convolutional layers with preceding 3 × 3 deformable convolutions to produce output stride 4.The up-convolutional channels are 256, 128, and 64, and kernels are initialized as bilinear interpolation.
  • Architectures: DLA uses fully convolutional symmetric upsampling with iterative deep aggregation and replaces each upsampling-layer convolution with a 3×3 deformable convolution.The deformable convolutions augment skip connections from lower layers to the output.
  • Training: 512 × 512 training inputs produce 128×128 outputs, with random flipping, scaling between 0.6 to 1.3, cropping, color jittering, and Adam optimization.The 3D estimation branch uses no augmentation because cropping or scaling changes 3D measurements.
  • Inference: Inference uses no augmentation, flip augmentation, or flip plus multi-scale augmentation at scales 0.5, 0.75, 1, 1.25, and 1.5.Flip outputs are averaged before decoding, while multi-scale results are merged with NMS.

6. Experiments

Experiments show that CenterNet achieves strong speed–accuracy trade-offs on COCO, with low collision rates and limited need for post-processing. The same center-point formulation also performs competitively for 3D bounding-box and human-pose estimation.

  • COCO object detection: 28.1% COCO AP at 142 FPS is achieved by the fastest ResNet-18 model, while DLA-34 reaches 37.4% AP at 52 FPS.DLA-34 provides the best speed/accuracy trade-off and exceeds YOLOv3 in both speed and accuracy.
  • COCO object detection: 45.1% AP with multi-scale evaluation makes Hourglass-104 CenterNet outperform all existing one-stage detectors on COCO test-dev.Two-stage detectors remain more accurate but slower.
  • Robustness and ablations: Center-point collisions prevent CenterNet from predicting < 0.1% of objects, versus ∼2% for slow- or fast-RCNN and 20.0% for Faster-RCNN with 15 anchors.IoU-based NMS has only a minor effect, improving DLA-34 flip-test AP from 39.2% to 39.7% while leaving Hourglass-104 at 42.2%.
  • Robustness and ablations: Using original test resolution is slightly better than fixing it, L1 is considerably better than Smooth L1, and λsize = 0.1 gives a good result.Doubling the training schedule before the learning-rate drop increases performance by 1.1 AP, but costs a much longer schedule.
  • 3D bounding-box estimation: On KITTI, CenterNet performs on-par with Deep3DBox and Mono3D in AP and AOS, does slightly better in BEV, and is two orders of magnitude faster.The experiments use annotated vehicle 3D boxes and evaluate average precision at IoU threshold 0.5.
  • Human pose estimation: Direct keypoint regression is reasonably accurate but not state-of-the-art, whereas projecting outputs to the closest joint detection performs competitively with state-of-the-art multi-person pose estimators.The projection improves results throughout, especially in high-IoU regimes.

7. Conclusion

CenterNet represents objects as points, detects their centers, and regresses object size through a simple, fast, accurate, end-to-end differentiable detector without NMS post-processing. The approach is general and supports estimating additional object properties beyond 2D detection.

  • CenterNet represents objects as points, finds their centers, and regresses their size.The detector builds on successful keypoint estimation networks.
  • The algorithm is simple, fast, accurate, and end-to-end differentiable without NMS post-processing.
  • The general approach extends beyond simple 2D detection to estimate properties including pose, 3D orientation, and depth.

Appendix A: Model Architecture · Appendix B: 3D BBox Estimation Details

Appendix A refers to architecture diagrams, while Appendix B specifies training and decoding details for depth, 3D dimensions, and orientation in 3D bounding-box estimation.

  • Appendix A: Model Architecture: Architecture diagrams are provided in Figure 6.The appendix directs readers to Figure 6 for diagrams of the architectures.
  • Appendix B: 3D BBox Estimation Details: For each object instance k, outputs at the ground-truth center point are ˆd_k ∈ R, ˆγ_k ∈ R^3, and ˆα_k ∈ R^8.These values are extracted from three output maps at the ground-truth center location.
  • Appendix B: 3D BBox Estimation Details: Depth is trained with L1 loss after converting the output to the absolute depth domain.The ground-truth absolute depth d_k is measured in meters.
  • Appendix B: 3D BBox Estimation Details: 3D dimensions are trained with L1 loss in absolute metric.γ_k represents object height, width, and length in meters.
  • Appendix B: 3D BBox Estimation Details: Orientation uses an 8-scalar encoding divided into two angular bins, each containing four scalars.The bins are B1 = [−7π/6, π/6] and B2 = [−π/6, 7π/6].
  • Appendix B: 3D BBox Estimation Details: Within each bin, two scalars provide softmax classification and two encode the sine and cosine of the in-bin offset.The offset is measured relative to the bin center m_i.
  • Appendix B: 3D BBox Estimation Details: Classification uses softmax, angular values use L1 loss, and the predicted orientation is decoded using the bin with the larger classification score.The encoding is α̂ = [b̂_1, â_1, b̂_2, â_2], with bin targets c_i = 1(θ ∈ B_i) and offsets a_i = (sin(θ − m_i), cos(θ − m_i)).

Appendix C: Collision Experiment Details

The appendix measures object-center and IoU collision frequencies in COCO train 2017, then quantifies forced anchor assignments in an anchor-based detector. Center collisions are fewer than IoU collisions at threshold 0.7, while forced assignments disproportionately affect small objects.

  • COCO train 2017 contains 118287 images, 860001 objects, and 80 categories, including 356340 small, 295163 medium, and 208498 large objects.
  • 614 center-point collisions occur across the COCO training set.
  • 170220 forced assignments are required by RenitaNet, including 125831 small-object, 18505 medium-object, and 25884 large-object assignments.These represent 35.3% of all small objects, 6.3% of all medium objects, and 12.4% of all large objects.

Appendix D: Experiments on PascalVOC

On Pascal VOC, CenterNet is evaluated on VOC 2007 test after training on VOC 2007 and VOC 2012 trainval, using mAP@0.5. The best CenterNet-DLA model performs competitively with top-tier methods while maintaining real-time speed.

  • Dataset and evaluation: Pascal VOC contains 16,551 training images and 4,962 testing images across 20 categories, evaluated by mAP at IoU threshold 0.5.Training uses VOC 2007 and VOC 2012 trainval, with testing on VOC 2007 test.
  • Training setup: CenterNet experiments use modified ResNet-18, ResNet-101, and DLA-34 at 384 × 384 and 512 × 512 training resolutions.All networks train for 70 epochs with learning-rate drops at 45 and 60 epochs.
  • Results: The best CenterNet-DLA model performs competitively with top-tier methods while maintaining real-time speed.Results are reported in Table 6 for Pascal VOC 2007 test.

Appendix E: Error Analysis

Appendix E analyzes CenterNet errors by replacing output heads with ground truth, finding that center-map improvements matter more than size-map improvements and that the full pipeline misses about 0.5% of objects.

  • Error-isolation method: The analysis replaces each output head with ground truth, using rendered Gaussian heatmaps for centers and nearest ground-truth sizes for detections.This isolates errors from the center point heatmap and bounding box size predictions.
  • Error sources: Improving both size maps yields a modest performance gain, whereas improving the center map yields much larger gains.The comparison is reported in Table 7.
  • Error sources: 83.1 AP is reached when only the keypoint offset is not predicted.This is the maximum AP reported under that ablation.
  • Error sources: About 0.5% of objects are missed by the entire pipeline with ground-truth inputs, due to discretization and Gaussian heatmap rendering estimation errors.The remaining misses arise despite replacing output heads with ground truth.
Loading 1904.07850v2…