PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientDet: Scalable and Efficient Object Detection

Mingxing Tan, Ruoming Pang, Quoc V. Le

arXiv:1911.09070v7cs.CVcs.LGeess.IV

TL;DR

Object detectors have become increasingly expensive, creating a need for efficient detection under constrained model-size and latency budgets. EfficientDet addresses this with weighted BiFPN feature fusion and compound scaling, achieving better accuracy and efficiency than prior detectors, including 55.1 AP with 7x fewer FLOPs than prior art.

  • Problem

    Increasingly expensive object detectors are difficult to deploy where model size and latency are constrained, making detection efficiency an important research gap.

  • Method

    EfficientDet combines learnable weighted bidirectional feature fusion with compound scaling across backbone, feature, prediction-network, and resolution dimensions.

  • Results

    55.1 AP: EfficientDet-D7x outperforms prior art by +4 AP with 7x fewer FLOPs, while achieving better accuracy and efficiency across resource constraints.

  • Takeaways & Limitations

    EfficientDet establishes a detector family that combines state-of-the-art accuracy with substantially lower parameter and computation requirements across resource constraints.

Abstract

from arXiv · show

Model efficiency has become increasingly important in computer vision. In this paper, we systematically study neural network architecture design choices for object detection and propose several key optimizations to improve efficiency. First, we propose a weighted bi-directional feature pyramid network (BiFPN), which allows easy and fast multiscale feature fusion; Second, we propose a compound scaling method that uniformly scales the resolution, depth, and width for all backbone, feature network, and box/class prediction networks at the same time. Based on these optimizations and better backbones, we have developed a new family of object detectors, called EfficientDet, which consistently achieve much better efficiency than prior art across a wide spectrum of resource constraints. In particular, with single model and single-scale, our EfficientDet-D7 achieves state-of-the-art 55.1 AP on COCO test-dev with 77M parameters and 410B FLOPs, being 4x - 9x smaller and using 13x - 42x fewer FLOPs than previous detectors. Code is available at https://github.com/google/automl/tree/master/efficientdet.

1. Introduction

The paper targets object detectors that improve both accuracy and efficiency across diverse resource constraints. It proposes weighted bidirectional feature fusion and compound scaling, combined with EfficientNet backbones, to develop the EfficientDet family.

  • Motivation: 167M parameters and 3045B FLOPs are required by an AmoebaNet-based NASFPN detector, creating deployment barriers for real-world applications.This computation is reported as 30x more than RetinaNet.
  • EfficientDet: 55.1% COCO AP is achieved by EfficientDet with much fewer parameters and FLOPs than previous detectors in the single-model, single-scale setting.EfficientDet combines EfficientNet backbones with BiFPN and compound scaling, and is reported to improve accuracy while reducing model size and computation.
  • Motivation: Prior efficient-detector methods often sacrifice accuracy and target only narrow resource ranges, despite applications spanning mobile devices to datacenters.The paper motivates coverage from approximately 3B to 300B FLOPs.
  • Challenge 1: efficient multi-scale feature fusion: Different-resolution inputs contribute unequally during feature fusion, motivating a weighted bi-directional feature pyramid network instead of undiscriminating summation.The proposed BiFPN is presented as a simple and highly effective response to the efficient multi-scale feature-fusion challenge.
  • Challenge 2: model scaling: Compound scaling jointly increases resolution, depth, and width across the backbone, feature network, and box/class prediction network.The approach addresses the observation that scaling feature and prediction networks matters alongside backbone and input-image scaling.

2. Related Work

Prior object detectors trade off flexibility and accuracy against simplicity and efficiency, while multiscale feature processing and model scaling remain central design challenges. EfficientDet follows the one-stage design and targets improved efficiency and accuracy through optimized architectures.

  • One-Stage Detectors: Two-stage detectors are generally more flexible and accurate, whereas one-stage detectors are simpler and more efficient through predefined anchors.One-stage detectors have recently attracted substantial attention because of their efficiency and simplicity.
  • One-Stage Detectors: EfficientDet mainly follows the one-stage detector design to achieve both better efficiency and higher accuracy with optimized network architectures.
  • Multi-Scale Feature Representations: Object detection requires effective representation and processing of multi-scale features, motivating pyramidal hierarchies, FPN’s top-down pathway, and PANet’s additional bottom-up path.Earlier detectors often predicted directly from pyramidal feature hierarchies extracted by backbone networks.
  • Model Scaling: Model scaling commonly improves accuracy by using larger backbones or increasing input image size.Examples range from mobile-size models and ResNet to ResNeXt and AmoebaNet, and from 512x512 to 1536x1536 inputs.
  • Model Scaling: Increasing channel size and repeating feature networks can also lead to higher accuracy, although existing scaling methods mostly focus on single or limited scaling dimensions.

3. BiFPN

BiFPN addresses multi-scale feature fusion with bidirectional cross-scale connections and weighted feature fusion. Its fast normalized fusion preserves accuracy while improving GPU efficiency, and the final design uses depthwise separable convolutions.

  • Feature fusion formulation: BiFPN formulates multi-scale fusion as transforming input features at different resolutions into aggregated output features.The proposed design targets efficient aggregation across feature levels.
  • Cross-scale connections: BiFPN simplifies bidirectional cross-scale connectivity by removing single-input nodes and adding same-level input-to-output edges for additional feature fusion.These optimizations produce a simplified bidirectional network while retaining useful cross-scale information.
  • Weighted feature fusion: Weighted feature fusion learns unequal contributions from different-resolution inputs instead of treating all features equally.Weights may be scalar, per-channel, or per-pixel; scalar weights provide comparable accuracy with minimal computational cost but require normalization for stability.
  • Fast normalized fusion: Up to 30% faster on GPUs, fast normalized fusion matches softmax-based fusion in learning behavior and accuracy without softmax’s extra latency.The method bounds weights using ReLU and a small epsilon for numerical stability.
  • Final BiFPN design: The final BiFPN combines bidirectional connections with fast normalized fusion and uses depthwise separable convolutions for feature fusion.Batch normalization and activation follow each convolution.

4. EfficientDet

EfficientDet combines an EfficientNet backbone, BiFPN feature network, and shared class/box prediction network into a scalable detector family. Its compound scaling jointly adjusts network dimensions to support varied resource constraints and improves efficiency over single-dimension scaling.

  • Architecture: EfficientDet uses ImageNet-pretrained EfficientNets as backbones, BiFPN for repeated top-down and bottom-up fusion, and class/box networks for predictions.BiFPN processes backbone features P3–P7 before feeding fused features to the prediction networks.
  • Compound scaling: Prior scaling methods typically enlarge only the backbone, input resolution, or FPN depth, whereas EfficientDet jointly scales network width, depth, and input resolution.The joint-scaling design is motivated by the limitations of focusing on a single or limited set of dimensions.
  • Compound scaling: 1.35 is the selected BiFPN width scaling factor after grid-searching values from 1.2 to 1.45, while BiFPN depth increases linearly.Prediction width is fixed equal to BiFPN width: Wpred = Wbifpn.
  • Model family: EfficientDet-D0 through D7 result from applying the scaling equations with φ = 0 through 7; D7x instead uses a larger backbone and feature level P3–P8.D7 and D7x share the same BiFPN and head, but D7 uses higher resolution.
  • Compound scaling: The proposed compound scaling is heuristic-based, yet it significantly improves efficiency over other single-dimension scaling methods.The paper evaluates this efficiency improvement in Figure 6.

5. Experiments

Experiments on COCO show that EfficientDet delivers strong accuracy with substantially lower model size, computation, and inference latency than prior detectors. A modified EfficientDet-D4 also outperforms DeepLabV3+ for semantic segmentation while using fewer FLOPs.

  • COCO detection setup: EfficientDet is evaluated on COCO 2017 using standardized single-model, single-scale comparisons without test-time augmentation.Models are trained on 118K images with SGD, synchronized batch normalization, SiLU activation, augmentation, and soft-NMS evaluation.
  • COCO detection results: 4x–9x smaller and 13x–42x fewer FLOPs than previous detectors across a wide range of accuracy or resource constraints.EfficientDet-D0 matches YOLOv3 with 28x fewer FLOPs, while matching RetinaNet and Mask-RCNN with up to 8x fewer parameters and 21x fewer FLOPs.
  • Inference efficiency: Up to 4.1x faster on GPU and 10.8x faster on CPU than previous detectors under same-machine latency comparisons.Latency is measured with batch size 1, including preprocessing and NMS postprocessing for V100 measurements.
  • Semantic segmentation: 1.7% better accuracy with 9.8x fewer FLOPs than DeepLabV3+ for Pascal VOC 2012 semantic segmentation.The comparison uses single-model, single-scale settings and excludes ensemble, test-time augmentation, and COCO pretraining.
  • Inference efficiency: EfficientDet models are 4x–9x smaller, 2x–4x faster on GPU, and 5x–11x faster on CPU than other detectors.Figure 4 compares model size and latency on a Titan V GPU and Xeon CPU with batch size 1.

6. Ablation Study

The ablations show that EfficientDet’s efficiency and accuracy arise from stronger backbones, weighted BiFPN fusion, fast normalized fusion, and joint compound scaling. These choices improve accuracy-efficiency tradeoffs while preserving fusion behavior and substantially accelerating computation.

  • Backbone and BiFPN: Replacing RetinaNet’s ResNet-50 backbone with EfficientNet-B3 improves accuracy by about 3 AP with slightly fewer parameters and FLOPs.The comparison uses RetinaNet training settings and starts from a ResNet-50 plus top-down FPN detector.
  • Feature fusion: 1.26x - 1.31x faster GPU execution is achieved by fast normalized fusion while maintaining similar accuracy to softmax-based fusion.The approach removes expensive softmax while retaining normalized weights, and the result holds across three detector sizes.
  • Feature networks: Weighted BiFPN achieves the best accuracy among compared feature networks with fewer parameters and FLOPs.The comparison uses the same backbone, class/box prediction network, and training settings across experiments.
  • Feature fusion: Normalized fusion weights always sum to 1, change rapidly during training, and show learning behavior very similar to softmax-based fusion.This behavior is observed for three randomly selected feature-fusion nodes in EfficientDet-D3.
  • Compound scaling: Compound scaling achieves better efficiency than scaling resolution, depth, or width alone by jointly balancing backbone, BiFPN, and prediction-network dimensions.The method jointly scales depth, width, and resolution across all three network components from the same baseline detector.

7. Conclusion

The paper studies architecture choices for efficient object detection and proposes weighted bidirectional feature fusion with customized compound scaling. These optimizations yield the EfficientDet detector family, which improves accuracy and efficiency across varied resource constraints.

  • The paper systematically studies network architecture design choices for efficient object detection.
  • It proposes a weighted bidirectional feature network and customized compound scaling to improve accuracy and efficiency.
  • EfficientDet consistently achieves better accuracy and efficiency than prior art across a wide spectrum of resource constraints.

Appendix

The appendix ablates training duration and scale jittering, finding that longer training benefits both RetinaNet-R50 and EfficientDet-D1, while EfficientDet is particularly sensitive. It also compares fixed and scaled resolutions, showing similar accuracy-latency trade-offs despite different resource profiles.

  • Experimental setup: The appendix evaluates training epochs and multi-scale jittering on RetinaNet-R50 and EfficientDet-D1 while keeping other hyperparameters unchanged.These ablations target two training hyperparameters for object detection.
  • Training Epochs: 300 epochs mark the point until which both models benefit from longer training, with EfficientDet benefiting particularly strongly.The paper uses 300 epochs as a trade-off between accuracy and training time.
  • Training Epochs: +2 AP is achieved by the reproduced RetinaNet over the default 37 AP using the paper’s training settings.The comparison is against the default RetinaNet result reported as 37 AP.
  • Scale Jittering: Large scale jitters can improve accuracy when training longer, whereas prior detectors commonly use the smaller [0.8, 1.2] jitter range.The appendix examines different jitter ranges and notes that the benefit of larger jitter depends on longer training.
  • Fixed vs. Scaled Resolution: Fixed- and scaled-resolution EfficientDet-D1–D6 models show very similar accuracy-latency trade-offs despite differing resolution preferences.Under similar accuracy constraints, fixed resolutions use more parameters but fewer activations and less peak memory.
Loading 1911.09070v7…