Source-linked AI summary
End-to-End Object Detection with Fully Convolutional Network
Jianfeng Wang, Lin Song, Zeming Li, Hongbin Sun, Jian Sun, Nanning Zheng
TL;DR
Fully convolutional detectors still rely on hand-designed NMS, motivating end-to-end detection without post-processing. The paper analyzes label assignment, introduces POTO with an auxiliary assignment and 3DMF, and reports competitive performance against NMS-based detectors on COCO and CrowdHuman.
Problem
Hand-designed NMS remains necessary in mainstream fully convolutional detectors, hindering fully end-to-end training; one-to-many assignment also produces duplicate predictions when NMS is removed.
Method
The framework combines prediction-aware one-to-one label assignment, a modified one-to-many auxiliary loss, and differentiable 3D Max Filtering across adjacent scales.
Results
The framework achieves competitive performance against state-of-the-art NMS-based detectors on COCO and CrowdHuman, including 1.1% mAP improvement on COCO and gains of 3.0% AP50 and 6.0% mMR on CrowdHuman.
Takeaways & Limitations
Fully convolutional detectors can discard NMS while retaining strong feature representation and competitive detection performance.
Abstract
from arXiv · showhide
Mainstream object detectors based on the fully convolutional network has achieved impressive performance. While most of them still need a hand-designed non-maximum suppression (NMS) post-processing, which impedes fully end-to-end training. In this paper, we give the analysis of discarding NMS, where the results reveal that a proper label assignment plays a crucial role. To this end, for fully convolutional detectors, we introduce a Prediction-aware One-To-One (POTO) label assignment for classification to enable end-to-end detection, which obtains comparable performance with NMS. Besides, a simple 3D Max Filtering (3DMF) is proposed to utilize the multi-scale features and improve the discriminability of convolutions in the local region. With these techniques, our end-to-end framework achieves competitive performance against many state-of-the-art detectors with NMS on COCO and CrowdHuman datasets. The code is available at https://github.com/Megvii-BaseDetection/DeFCN .
1. Introduction
The paper asks whether fully convolutional detectors can perform end-to-end detection without hand-designed NMS. It identifies label assignment and local multi-scale filtering as the central design dimensions and reports competitive results on COCO and CrowdHuman.
- Fully convolutional detectors commonly use multiple predictions per instance followed by NMS, which prevents fully end-to-end training.
- The framework combines prediction-aware one-to-one assignment with 3D Max Filtering so each instance can receive a single prediction without post-processing.
- One-to-many assignment supplies robust features but creates duplicate predictions, whereas one-to-one assignment is crucial for removing duplicate-removal post-processing.Fixed one-to-one assignment can introduce ambiguity and reduce feature discriminability.
- 3D Max Filtering applies differentiable max filtering across adjacent scales to improve convolution discriminability in local regions.
- 1.1% mAP improvement over the FCOS baseline with NMS was achieved on COCO using an FCOS detector with a ResNeXt-101 backbone.
- 3.0% AP50 and 6.0% mMR absolute gains over the FCOS baseline with NMS were achieved on CrowdHuman under a ResNet-50 backbone.
2. Related Work
Related work removes or reduces dependence on anchors and NMS through anchor-free, recurrent, attention-based, and matching-based approaches, but each has important limitations for dense prediction.
- Anchor-based detectors use predefined sliding windows as anchors, making training targets hand-designed and requiring careful hyper-parameter tuning.
- Anchor-free frameworks remove predefined anchors but still require NMS for duplicate removal, whose fixed threshold needs tuning and may be fragile in crowded scenes.
- Recurrent detectors support direct box prediction but were mainly evaluated on small datasets, and their iterative inference is inefficient.
- Attention-based relation networks and DETR use one-to-one assignment and direct set losses without post-processing, but massive predictions incur high computational cost in dense frameworks.
3. Methodology
The methodology combines prediction-aware one-to-one assignment with auxiliary one-to-many supervision and 3D Max Filtering to remove NMS while preserving feature representation. POTO selects foreground predictions dynamically using classification and regression quality, while 3DMF suppresses duplicates across local regions and adjacent FPN scales.
- Label-assignment analysis: One-to-many assignment supplies strong feature supervision but produces duplicate high-scoring boxes when NMS is removed.On the FCOS baseline, discarding NMS causes a 28.4% mAP absolute drop.
- Label-assignment analysis: One-to-one assignment reduces the gap between detectors with and without NMS, but its limited supervision lowers performance relative to the FCOS baseline.The Center rule yields 21.5% mAP absolute gains over the FCOS baseline without NMS, while unresolved performance gaps remain.
- Auxiliary supervision: The proposed mixture assignment combines POTO with a modified one-to-many auxiliary loss, retaining adequate supervision while discarding NMS.The auxiliary loss is added during training to improve feature representation, alongside the POTO classification assignment.
- Prediction-aware One-to-one Label Assignment: POTO dynamically assigns one foreground prediction per ground-truth instance according to prediction quality rather than a fixed hand-designed rule.Its matching quality combines spatial prior, classification confidence, and regression quality through a weighted geometric mean; α = 0.8 is the default.
- Prediction-aware One-to-one Label Assignment: POTO training generates a suitable permutation of dense predictions as foreground samples and optimizes separate foreground and background losses.The formulation uses the assigned foreground index set together with ground-truth labels and boxes and the corresponding predicted scores and coordinates.
- 3D Max Filtering: 3D Max Filtering extends max filtering across adjacent FPN scales and channels, improving local discriminability to suppress duplicate predictions.It is designed as a differentiable module embedded in the FPN head; the broader NMS configuration analysis considers adjacent scales and spatial duplicate-removal ranges.
4. Experiments
Experiments show that POTO, 3DMF, and auxiliary supervision enable fully convolutional detection without NMS, with competitive or superior performance across COCO and CrowdHuman.
- Visualization: POTO suppresses duplicate classification scores, while 3DMF further enhances suppression near confident predictions and across FPN stages.
- Label assignment: Multiplication fusion is more suitable than addition for end-to-end detection, yielding 0.7% mAP absolute gains over addition.
- Components: POTO with 3DMF narrows the no-NMS performance gap to 0.2% mAP on COCO.POTO alone gains 19.0% mAP over vanilla FCOS, while 3DMF adds 1.8% mAP.
- 3DMF: The strongest 3DMF setting uses spatial range φ = 3 and scale range τ = 2, reflecting local duplicate predictions across adjacent scales.
- Larger backbones: 1.1% mAP separates the end-to-end detector from the FCOS baseline with NMS on COCO using a ResNeXt-101 backbone.
- CrowdHuman: 3.0% mAP and 6.0% mMR absolute gains over FCOS are reported on the crowded CrowdHuman dataset.The method is described as more robust and flexible in crowded scenes.
5. Conclusion
The paper combines prediction-aware one-to-one assignment, 3D Max Filtering, and auxiliary loss to bridge fully convolutional networks and end-to-end object detection.
- The proposed framework achieves superior performance against many state-of-the-art detectors with NMS on COCO and CrowdHuman.
- The method shows potential in complex and crowded scenes and may benefit other instance-level tasks.
A. Auxiliary Loss
The auxiliary loss is evaluated with several one-to-many assignment rules, and the prediction-aware quality function performs best among the compared matching metrics.
- The auxiliary-loss study compares FCOS, ATSS, Quality-ATSS, and Quality-FCOS assignment rules.
- 1.3% mAP absolute gains over standard ATSS are obtained by the quality-based assignment rule.
B. Comparison to DETR
The paper compares fully convolutional detectors with DETR and reports advantages for the proposed framework in convergence speed and crowded scenes.
- The COCO and CrowdHuman comparisons use ResNet-50-based models, with NMS omitted except for FCOS.
- Convolutions have tested variants, including deformable convolutions, that can outperform DETR in the reported comparisons.
- The framework has advantages over DETR in convergence speed and crowded scenes.