Source-linked AI summary
Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, Ping Luo
TL;DR
Dense object detectors rely on large sets of hand-designed candidates and associated assignment and post-processing procedures. Sparse R-CNN instead uses a fixed small set of learnable boxes and features with dynamic heads and direct set prediction. On COCO, it reports competitive accuracy, runtime, and convergence, including 45.0 AP and 22 fps with ResNet-50 FPN.
Problem
Dense-prior detectors require dense candidate design, heuristic many-to-one assignment, and often NMS, motivating a sparse detector in both boxes and features.
Method
Sparse R-CNN uses a fixed small set of learnable proposal boxes and features, one-to-one dynamic heads, and set-based loss for direct prediction.
Results
45.0 AP and 22 fps are reported for Sparse R-CNN with a ResNet-50 FPN model under the standard 3× training schedule, with accuracy, runtime, and convergence on par with established detectors.
Takeaways & Limitations
Sparse R-CNN provides a considerably sparse object-detection design that avoids dense candidates, global dense-feature interaction, and NMS post-processing.
Abstract
from arXiv · showhide
We present Sparse R-CNN, a purely sparse method for object detection in images. Existing works on object detection heavily rely on dense object candidates, such as $k$ anchor boxes pre-defined on all grids of image feature map of size $H\times W$. In our method, however, a fixed sparse set of learned object proposals, total length of $N$, are provided to object recognition head to perform classification and location. By eliminating $HWk$ (up to hundreds of thousands) hand-designed object candidates to $N$ (e.g. 100) learnable proposals, Sparse R-CNN completely avoids all efforts related to object candidates design and many-to-one label assignment. More importantly, final predictions are directly output without non-maximum suppression post-procedure. Sparse R-CNN demonstrates accuracy, run-time and training convergence performance on par with the well-established detector baselines on the challenging COCO dataset, e.g., achieving 45.0 AP in standard $3\times$ training schedule and running at 22 fps using ResNet-50 FPN model. We hope our work could inspire re-thinking the convention of dense prior in object detectors. The code is available at: https://github.com/PeizeSun/SparseR-CNN.
1. Introduction
Sparse R-CNN challenges dense object-detection pipelines by replacing their numerous hand-designed candidates with a fixed sparse set of learnable proposals and maintaining sparse interactions throughout the pipeline. It aims to avoid dense-candidate design, many-to-one assignment, and NMS while retaining competitive detection performance.
- Motivation: Dense-prior detectors use numerous anchors, reference points, or dense region candidates, and their performance depends on candidate design choices.These choices include anchor sizes, aspect ratios, counts, reference-point density, and proposal-generation algorithms.
- Motivation: Dense pipelines commonly require NMS because they produce redundant near-duplicate detections and use heuristic many-to-one label assignment during training.The many-to-one assignment makes training sensitive to heuristic assignment rules.
- Motivation: DETR outputs a sparse set of predictions from learned object queries, but each query still interacts with dense global image features, slowing training convergence.This motivates seeking a detector that is sparse in both boxes and features.
- Contribution: Sparse R-CNN uses a fixed small set of learnable proposal boxes, such as 100 boxes represented by 4-d coordinates, instead of hundreds of thousands of dense candidates.The proposals extract region features for subsequent recognition.
- Contribution: Learnable proposal features encode instance characteristics and generate customized parameters for each proposal’s exclusive dynamic object-recognition head.Both proposal boxes and proposal features are randomly initialized and optimized with the rest of the network.
- Contribution: Sparse R-CNN maintains a sparse-in sparse-out pipeline through one-to-one dynamic instance interaction, without dense candidates or global dense-feature interaction.The paper reports competitive accuracy, runtime, and training convergence on COCO, including 45.0 AP and 22 fps with ResNet-50 FPN.
2. Related Work
Prior object detectors largely follow dense or dense-to-sparse designs, whereas Sparse R-CNN is positioned as a sparse alternative that begins with learnable proposals rather than dense candidate generation.
- Dense method: One-stage detectors classify and regress dense anchors or reference points placed across feature-map grids.Anchor-free methods replace hand-crafted anchors with reference points but retain dense candidates and predefined training assignment principles.
- Dense-to-sparse method: Two-stage detectors first generate sparse foreground proposals from dense region candidates before refining locations and predicting categories.Their proposal-generation algorithms remain important because the first stage is built on dense candidates.
- Dense-to-sparse method: DETR directly outputs predictions from sparse object queries without hand-crafted post-processing, but its queries interact with global dense image features.It is therefore characterized here as another dense-to-sparse formulation.
- Sparse method: Earlier sparse detectors could avoid dense candidates but often trailed dense detectors in accuracy, while hand-designed regular-grid priors were sub-optimal.Sparse R-CNN instead applies learnable proposals to pursue a more effective sparse design.
3. Sparse R-CNN
Sparse R-CNN replaces dense proposal generation with a fixed set of learnable boxes and features, processes corresponding RoIs through dynamic heads, and trains fixed-size prediction sets with bipartite matching. Its design combines sparse inputs, one-to-one interactions, iterative refinement, and set-based loss.
- Architecture: Sparse R-CNN replaces hundreds of thousands of RPN candidates with a small set of learnable proposal boxes, such as 100 boxes.The network combines a backbone, dynamic instance interactive head, and task-specific prediction layers.
- Backbone: The ResNet-FPN backbone produces multi-scale feature maps at pyramid levels P2 through P5, each with 256 channels.The implementation aligns with Faster R-CNN to demonstrate simplicity and effectiveness.
- Learnable proposals: Learnable proposal boxes are N × 4 normalized center-coordinate, height, and width parameters updated by back-propagation.They replace predictions from the Region Proposal Network.
- Dynamic instance interaction: Each proposal box is paired with a proposal feature, and its RoI feature is processed by an exclusive dynamic head conditioned on that proposal feature.The proposal feature generates parameters for two consecutive 1 × 1 convolutions with ReLU activation.
- Iterative refinement: Iterative stages reuse newly generated object boxes and features as proposals for the next stage, while self-attention reasons over the set of object features.The sparse design and light dynamic head add only marginal computation overhead.
- Training: Set prediction loss performs optimal bipartite matching between fixed-size predictions and ground-truth objects, combining classification, L1, and generalized IoU losses.The loss is applied only to matched pairs and directly bypasses many-to-one matching.
4. Experiments
Experiments evaluate Sparse R-CNN on COCO through component ablations, detector comparisons, efficiency measurements, and proposal-behavior analyses. The results show strong accuracy, fast convergence and inference, robustness to initialization, and benefits from iterative feature reuse and additional stages.
- Main Results: 26.7 AP vs. 22.5 AP on small objects favors Sparse R-CNN over DETR, while Sparse R-CNN also performs better with simple FPN features.The paper notes that DETR-series comparisons use stronger feature-extraction methods, motivating a stronger Sparse R-CNN variant for fairness.
- Efficiency: 10× faster convergence over DETR accompanies 45.0 AP vs. 43.8 AP, 22 FPS vs. 19 FPS, and 36 epochs vs. 50 epochs against Deformable DETR.The reported comparison covers training convergence, accuracy, inference speed, and schedule length.
- Efficiency: 23 FPS with 100 proposals and 22 FPS with 300 proposals indicate limited inference-speed reduction from increasing proposal count.The paper attributes this to the light design of the dynamic instance interaction head.
- Ablation Studies: Adding iterative architecture improves performance by 13.7 AP, while reusing previous-stage features contributes an 11.7 AP gain over the original cascade.Performance reaches 36.2 AP at two stages and saturates at six stages; increasing proposals from 100 to 500 gives continuous improvement but requires more training time.
- Proposal Behavior: Learned proposal-box performance is relatively robust to initialization, and converged boxes cover the image while cascading heads refine positions and remove duplicates.The proposal behavior is reported as robust in both rare and crowded scenes, with crowded objects consuming more refinement stages.
5. Conclusion
The conclusion presents Sparse R-CNN as a purely sparse detector using learned proposals and dynamic heads, with direct final predictions without NMS. It reports accuracy, runtime, and training convergence on par with established detectors and encourages reconsidering dense priors.
- Conclusion: Sparse R-CNN uses a fixed sparse set of learned object proposals and dynamic heads for classification and localization.Its final predictions are directly output without non-maximum suppression post-processing.
- Conclusion: The reported accuracy, runtime, and training convergence are on par with well-established detectors.The conclusion frames this result as evidence for reconsidering dense priors in object detection.
A. Crowded Scene
CrowdHuman experiments assess Sparse R-CNN in crowded human-detection scenes. The method is reported as applicable to crowded scenes and outperforms several mainstream and end-to-end detector comparisons in Table 11.
- Evaluation: CrowdHuman evaluates crowded human detection using AP, mMR, and Recall under IoU 50.The benchmark uses approximately 15k training images and 4k validation images.
- Evaluation: Sparse R-CNN is trained for 50 epochs with 500 proposals and image resizing constrained to shortest sides between 480 and 800 pixels.The longest image side is capped at 1500 pixels, with other details matching COCO.
- Results: Sparse R-CNN achieves better performance than Faster R-CNN, RetinaNet, and FCOS on CrowdHuman.The comparison is reported from Table 11.
- Results: 23.1 AP and 2.5 AP improvements over DETR and Deformable DETR, respectively, are reported on CrowdHuman.The cited comparison concerns two recent end-to-end detectors.
- Conclusion: CrowdHuman experiments show that Sparse R-CNN is applicable to crowded scenes and may serve as a baseline across detection scenarios.This is the paper’s stated conclusion for the crowded-scene evaluation.
B. Self-supervised Pre-training
Sparse R-CNN benefits from self-supervised pre-training: replacing ImageNet-supervised weights with self-supervised ones yields consistent improvement.
- B. Self-supervised Pre-training: Sparse R-CNN obtains consistent improvement when ImageNet-supervised pre-training is replaced with self-supervised weights.The reported self-supervised methods include DetCo and SCRL.
C. Backbone Architecture
Sparse R-CNN performs better with the evaluated Transformer backbones than with its default CNN-based ResNet-50 backbone.
- C. Backbone Architecture: Both evaluated Transformer backbones achieve better performance than the CNN backbone on Sparse R-CNN.The evaluated Transformer backbones are PVT and Swin Transformer, while the default backbone is ResNet-50.