Probabilistic two-stage detection

Xingyi Zhou; Vladlen Koltun; Philipp Krähenbühl

Source-linked AI summary

Probabilistic two-stage detection

Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl

arXiv:2103.07461v1Mar 12, 2021cs.CV

TL;DR

The paper addresses the lack of a probabilistically coherent combination of proposal and classification scores in two-stage detection. It builds a detector with a strong one-stage object-likelihood first stage and conditional region classification, achieving higher accuracy and speed than precursor models. On COCO, it reports 56.4 mAP with single-scale testing and 49.2 mAP at 33 fps with a lightweight backbone.

Problem
Standard RPNs maximize proposal recall but do not provide sufficiently accurate object likelihoods, while one-stage detectors can be slower for large vocabularies and may use misaligned local features.
Method
The framework builds probabilistic two-stage detectors from state-of-the-art one-stage detectors, using their object likelihoods with conditional region classification and joint training.
Results
The detectors are faster and more accurate than one- and two-stage precursors, reaching 56.4 mAP on COCO test-dev with single-scale testing.
Takeaways & Limitations
The framework combines advances from one-stage and two-stage detection to improve accuracy and speed while using probabilistically principled final scores.
Takeaways & Limitations
One-stage detectors usually use heavier separate classification and regression branches, and can lose their speed advantage on large-vocabulary datasets.

Abstract

from arXiv · show

We develop a probabilistic interpretation of two-stage object detection. We show that this probabilistic interpretation motivates a number of common empirical training practices. It also suggests changes to two-stage detection pipelines. Specifically, the first stage should infer proper object-vs-background likelihoods, which should then inform the overall score of the detector. A standard region proposal network (RPN) cannot infer this likelihood sufficiently well, but many one-stage detectors can. We show how to build a probabilistic two-stage detector from any state-of-the-art one-stage detector. The resulting detectors are faster and more accurate than both their one- and two-stage precursors. Our detector achieves 56.4 mAP on COCO test-dev with single-scale testing, outperforming all published results. Using a lightweight backbone, our detector achieves 49.2 mAP on COCO at 33 fps on a Titan Xp, outperforming the popular YOLOv4 model.

1. Introduction

Two-stage detection lacks a probabilistically coherent combination of proposal and classification scores. The paper replaces weak proposal scoring with calibrated object likelihoods from a strong one-stage first stage, improving speed and accuracy.

Two-stage detectors first propose object locations and then classify them, but their stage combination lacks a probabilistic interpretation.
The framework uses a class-agnostic one-stage detector for object likelihood, followed by conditional region classification and score combination.
The first stage must estimate calibrated object likelihoods rather than only maximize proposal recall, which standard RPNs do not reliably provide.
The two stages are trained jointly to maximize ground-truth object log-likelihood, and the resulting final log-likelihood is used for detection scoring.
56.4 mAP on COCO test-dev with single-scale testing surpasses all published results reported in the paper.
49.2 mAP at 33 fps on a Titan Xp with a small backbone surpasses YOLOv4’s 43.5 mAP at the same speed and hardware.

2. Related Work

Prior detectors trade off architectural efficiency, proposal quality, and classification accuracy. The proposed framework combines a strong one-stage proposal mechanism with later region classification while using fewer proposals.

One-stage detectors densely predict object classes and locations, while conventional two-stage detectors generate proposals before per-region classification and refinement.
One-stage detectors commonly use heavier separate classification and regression branches, and their speed advantage can disappear for large vocabularies.
The framework factorizes probability across stages, uses a strong class-agnostic detector, and selects fewer higher-quality regions for later processing.
Existing two-stage detectors use weak RPNs that maximize top-1K proposal recall without using proposal scores at test time, slowing inference.
Point-based detectors and DETR-style models represent alternative architectures for producing detections without the conventional dense or proposal-based design.

3. Preliminaries

Object detectors estimate bounding-box locations and class likelihoods. One-stage models predict both jointly, whereas two-stage models factorize objectness and conditional classification across proposal and region-classification stages.

An object detector predicts each object’s bounding-box location and class-specific likelihood scores over a predefined class set.
One-stage detectors jointly predict locations and class likelihoods in a single network, commonly modeling each class with an independent sigmoid Bernoulli distribution.
One-stage detectors optimize log-likelihood or focal loss, use class probabilities to score boxes, and directly regress bounding-box coordinates.
Two-stage detectors first estimate objectness for proposals, then classify each candidate conditionally into object classes or background and refine its location.
The paper treats both stages as one class-likelihood estimate and studies how this interpretation changes first-stage design and training.

4. A probabilistic interpretation of two-stage detection

The paper factorizes two-stage class probabilities into object likelihood and conditional classification, then trains the coupled background objective through tractable lower bounds.

The model represents class distribution as a class-agnostic object likelihood from stage one and conditional categorical classification from stage two.
For annotated objects, maximum-likelihood training reduces to independent objectives for the first and second stages.
The background likelihood couples first- and second-stage probabilities, so its exact objective requires evaluating the second stage for all first-stage outputs.
The paper jointly optimizes two lower bounds instead of the exact background objective to avoid prohibitively slow dense second-stage evaluation.
The first lower bound trains second-stage background likelihood for high-scoring first-stage objects, but can be arbitrarily loose in a specified probability regime.
The second bound makes first-stage training equivalent to a binary one-stage detector or an RPN with strict negatives that encourages likelihood estimation rather than recall.
The final detection score multiplies conditional classification by class-agnostic detection likelihood, requiring a strong first-stage detector.

5. Building a probabilistic two-stage detector

The framework builds a probabilistic two-stage detector around a strong one-stage first stage that predicts calibrated object likelihoods, then combines it with a second-stage classifier. It adapts several one-stage designs and adjusts proposal-processing settings for the two-stage pipeline.

The first stage must predict accurate object likelihoods for the overall detection score, rather than maximize object coverage.The framework experiments with four one-stage detector designs as probabilistic first stages.
RetinaNet contributes a heavier head, stricter anchor assignment, and focal loss to improve calibrated one-stage likelihoods.The design uses these three components in the first-stage architecture.
CenterNet* upgrades CenterNet to multiple scales with an FPN and applies classification and regression branches across FPN levels P3-P7.The improved architecture uses center-based annotations, neighborhood positives, distance-to-boundaries regression, and gIoU loss.
ATSS and GFL provide alternative first-stage designs, using centerness or regression quality to improve proposal likelihood estimation.The probabilistic baselines merge classification and regression heads for a slight speedup.
Each one-stage first stage is paired with Faster R-CNN or Cascade R-CNN, which infers the conditional class likelihood P(Ck|Ok).The one-stage architectures infer P(Ok), while the second stage models classification conditioned on object presence.
Using FPN levels P3-P7 for both stages and raising NMS to 0.7 accommodates the probabilistic detector’s reduced proposal count.The cited hyperparameter changes are reported as necessary for probabilistic detectors and do not improve the RPN-based detector.

6. Results

Across COCO, LVIS, and Objects365, probabilistic two-stage detectors improve accuracy and often speed by combining strong one-stage proposals with two-stage heads and proposal-score modeling.

One- and two-stage comparisons: All probabilistic two-stage detectors outperform their one- and two-stage precursors, with fewer proposals often reducing runtime.Probabilistic FasterRCNN variants use 256 rather than 1K proposals, and some are faster than original FasterRCNN.
Real-time models: 45.6 mAP at 40 fps lets CenterNet2 outperform YOLOv4 and EfficientDet-B2 in the reported real-time comparison.Against realtime-FCOS under matched settings, CenterNet2 gains 1.6 mAP while adding only 4 ms.
Real-time models: 49.2 mAP at 33 fps demonstrates that a two-stage detector can match one-stage speed while delivering higher accuracy.This configuration uses a slightly different FPN structure and self-training.
State-of-the-art comparison: 56.4 mAP with single-scale testing is achieved on COCO test-dev, exceeding all published results in the paper’s comparison.The final model uses a stronger backbone, larger input resolution, heavy crop augmentation, a longer schedule, and self-training.
Ablation studies: 40.4 mAP follows from incorporating first-stage scores after improving proposal quality with a stricter IoU threshold and focal loss.Strengthening the RPN alone enables fewer proposals but does not improve accuracy; the improved proposals make score incorporation effective.

7. Conclusion

The paper interprets two-stage detection probabilistically, motivating a strong likelihood-estimating first stage and principled final detection scores. The resulting detectors combine accuracy and speed across one- and two-stage designs.

A strong first stage should estimate object likelihoods rather than maximize proposal recall.These likelihoods are combined with second-stage classification scores for final detections.
Probabilistic two-stage detectors are faster and more accurate than their one- or two-stage counterparts.
The framework integrates advances from one- and two-stage detector designs to combine accuracy with speed.

A. Tightness of lower bounds

The combined lower bound is close to the true maximum-likelihood objective. Its gap is at most log(2), with equality only in a limiting case where the objective tends to negative infinity.

The analysis optimizes two lower bounds and combines them through their maximum.
log P(bg) ≤max(B1, B2) + log(2).
Both parts of the max-bound come within log 2 of the actual objective.
The bounds are exactly log 2 away only at α = β →0, where the objective tends to negative infinity.
For all other values, the bounds are tighter than log 2.

B. Backbones and training details

The experiments use distinct backbone configurations and training schedules for default, large, and real-time models. The reported evaluation includes classification-loss ablations on LVIS validation.

Default backbone: The default model uses ResNet-50 with SGD and a 90K-iteration 1x training schedule.The learning rate is dropped by 10x at iterations 60K and 80K.
Large backbone: The large model uses ResNeXt-32x8d-101-DCN with deformable convolutions and a 180K-iteration 2x schedule.Its shorter-edge scale augmentation ranges from 480 to 960.
Real-time backbone: The real-time model uses DLA with BiFPN and trains with scale augmentation over a 256–608 shorter-edge range.Its long-edge limit is 900, and the model is first trained with a 4x schedule.
The LVIS v1 validation ablation compares classification losses for both the proposed and baseline detectors.All models use ResNet50-1x with FPN P3-P7 and multi-scale training, reporting mean and standard deviation over two runs.

C. Extra-large model details

The extra-large model combines a stronger backbone, larger resolutions, heavy crop augmentation, longer training, and self-training. These changes produce high COCO validation and test-dev mAP, while the real-time model also improves.

The road map switches from ResNeXt-101-DCN to Res2Net-101-DCN with BiFPN, yielding a 0.6 mAP improvement.The backbone change also speeds up training.
56.4 mAP on COCO test-dev is achieved with 1560 × 1560 testing after self-training on pseudo-labeled COCO images.The same model achieves 56.1 mAP on COCO validation at the larger test size.
49.2 mAP@ 30ms is achieved by the real-time model after advanced augmentation, FPN, and self-training modifications.The modifications improve the model from 45.6mAP@ 25ms.

D. Federated Loss for LVIS

Federated loss addresses sparse LVIS annotations by balancing positive supervision with sampled negative categories. The method outperforms competing baselines and improves proposal-network-based detectors in the reported comparisons.

Motivation: Sparse LVIS annotations create a trade-off: treating unannotated objects as negatives harms rare classes, while using only annotated images weakens background learning.The paper motivates federated loss as a middle ground for this annotation setting.
Method: Federated loss includes all positive annotations but samples only a random subset of negative classes for each training image.The sampled class subset is denoted S.
Method: Negative categories are sampled by square-root training frequency, with |S| = 50; binary cross-entropy applies to S while other classes are ignored.The subset is resampled per iteration, so one image can receive different class subsets across iterations.
Results: Federated loss more than offsets the slight drop from replacing softmax with sigmoid and significantly outperforms EQL and other baselines.The EQL comparison follows the authors’ LVIS v0.5 settings, including ignoring 900 tail categories.
Results: Under the same FasterRCNN, Res50-1x, no-augmentation setting, FasterRCNN-CenterNet performs better and runs faster than GA RPN and CascadeRPN.Table 11 compares these proposal networks using the stated common training configuration.

F. Dataset details

The paper evaluates on COCO, LVIS V1, and Objects365, which differ in dataset size and number of categories.

Dataset composition: COCO has 118k training, 5k validation, and 20k test images across 80 categories; LVIS V1 has 100k training and 20k validation images across 1203 categories.Objects365 adds 600k training and 30k validation images across 365 categories.
Dataset composition: Objects365 contains 600k training images and 30k validation images across 365 categories, while all three datasets use internet images with accurate annotations.The datasets are COCO, LVIS V1, and Objects365.