PaperlayerView exact revision on arXiv

Source-linked AI summary

Speed/accuracy trade-offs for modern convolutional object detectors

Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, Kevin Murphy

arXiv:1611.10012v3cs.CV

TL;DR

Practitioners lack fair, comprehensive evidence for choosing object detectors under differing speed, memory, and accuracy constraints. The paper unifies Faster R-CNN, R-FCN, and SSD experiments to trace these trade-offs, finding useful speed improvements and accuracy/speed operating points while documenting benchmarking limits.

  • Problem

    Practitioners need speed, memory, and accuracy information to choose detectors, but prior comparisons vary across architectures, feature extractors, resolutions, hardware, and software.

  • Method

    The authors implement Faster R-CNN, R-FCN, and SSD in a unified framework and vary architectural and training configurations to trace their speed/accuracy trade-offs.

  • Results

    Fewer Faster R-CNN proposals can substantially improve speed without much accuracy loss, SSD is less feature-extractor-sensitive, and some accuracy gains require sacrificing speed.

  • Takeaways & Limitations

    The reported comparisons and newly identified configurations are intended to help practitioners choose detectors for real-world deployment constraints.

  • Takeaways & Limitations

    Timing includes CPU postprocessing and is comparable within the study but may not be directly comparable with speeds reported elsewhere.

Abstract

from arXiv · show

The goal of this paper is to serve as a guide for selecting a detection architecture that achieves the right speed/memory/accuracy balance for a given application and platform. To this end, we investigate various ways to trade accuracy for speed and memory usage in modern convolutional object detection systems. A number of successful systems have been proposed in recent years, but apples-to-apples comparisons are difficult due to different base feature extractors (e.g., VGG, Residual Networks), different default image resolutions, as well as different hardware and software platforms. We present a unified implementation of the Faster R-CNN [Ren et al., 2015], R-FCN [Dai et al., 2016] and SSD [Liu et al., 2015] systems, which we view as "meta-architectures" and trace out the speed/accuracy trade-off curve created by using alternative feature extractors and varying other critical parameters such as image size within each of these meta-architectures. On one extreme end of this spectrum where speed and memory are critical, we present a detector that achieves real time speeds and can be deployed on a mobile device. On the opposite end in which accuracy is critical, we present a detector that achieves state-of-the-art performance measured on the COCO detection task.

1. Introduction

Modern convolutional detectors are accurate and increasingly deployable, but selecting among them requires balancing accuracy with speed and memory. This paper addresses that gap through unified experiments across detector designs and configurations.

  • Real deployments require balancing accuracy with running time and memory, because mobile, autonomous-driving, and server systems impose different constraints.
  • Existing reports rarely provide a complete speed/accuracy trade-off because results vary with feature extractor, image size, hardware, and software.
  • The authors implement Faster R-CNN, R-FCN, and SSD uniformly to trace trade-offs across meta-architectures, feature extractors, image resolutions, and related parameters.
  • Using fewer Faster R-CNN proposals can substantially increase speed without much accuracy loss, making it competitive with SSD and R-FCN.
  • SSD is less sensitive to feature-extractor quality than Faster R-CNN and R-FCN, while some accuracy gains require sacrificing speed.
  • Several evaluated meta-architecture and feature-extractor combinations are novel, and some supported the winning entry of the 2016 COCO detection challenge.

2. Meta-architectures

The paper treats SSD, Faster R-CNN, and R-FCN as related meta-architectures built around convolutional features, anchors, and classification-regression prediction. They differ mainly in whether and where proposal-specific computation occurs.

  • Modern anchor-based detectors assign each anchor a class prediction and a box-offset prediction using a combined classification and regression loss.
  • The study decouples meta-architecture from feature extractor so SSD, Faster R-CNN, and R-FCN can be evaluated with alternative backbones.
  • SSD: SSD uses one feed-forward convolutional network to directly predict classes and anchor offsets without a second-stage per-proposal classifier.
  • Table 1 organizes convolutional detection models by meta-architecture and specifies how boxes are encoded relative to matching anchors.
  • Faster R-CNN: Faster R-CNN first generates class-agnostic proposals with an RPN, then applies a second-stage classifier and box refiner to those proposals.
  • R-FCN: R-FCN reduces per-region computation by cropping features from the last pre-prediction layer rather than the proposal-prediction layer.

3. Experimental setup

The experimental setup is designed to make speed and accuracy comparisons more comparable across modern detectors. The authors recreate major pipelines in a common TensorFlow platform.

  • Prior comparisons are difficult because papers use different frameworks, hardware, optimization targets, and training-set variants.
  • The authors build a TensorFlow detection platform with recreated SSD, Faster R-CNN, and R-FCN training pipelines for apples-to-apples evaluation.

3.1. Architectural configuration

Architectural configuration varies both the feature extractor and detector-specific settings such as proposal count and output stride. These choices affect accuracy, speed, memory, and deployment suitability.

  • Feature extractors directly affect detector memory, speed, and performance through their parameter counts and layer types.
  • The study compares VGG-16, ResNet-101, Inception v2, Inception v3, Inception-ResNet, and MobileNet as feature extractors.
  • Depthwise separable convolutions reduce computational cost and parameter count by factorizing standard convolutions.
  • For Faster R-CNN and R-FCN, proposal-prediction layers are selected according to the original architectures, with analogous choices for other extractors.
  • SSD uses multiple feature maps at different scales, including high- and lower-resolution maps, for location and confidence predictions.
  • Most evaluated meta-architecture and feature-extractor combinations are novel, including Inception-based Faster R-CNN and detection uses of Inception-ResNet and MobileNet.
  • Reducing Faster R-CNN and R-FCN proposals from the typical 300 to as few as 10 exposes a speed-recall trade-off.
  • Using stride 8 instead of stride 16 improves mAP by 5% but increases running time by 63%.

3.2. Loss function configuration

The experiments standardize anchor matching, sampling ratios, and localization-loss choices while comparing how loss configurations affect training and performance.

  • Loss configuration: Loss-function configuration choices can affect training stability and final performance.The paper compares its choices with configurations used in other works.
  • Matching: Anchor targets are assigned by matching anchors to groundtruth instances.The paper considers greedy bipartite and many-to-one matching strategies.
  • Matching: The experiments use Argmax matching throughout, with thresholds recommended for each meta-architecture.Argmax matching allows many-to-one assignments, while low-overlap matches are discarded.
  • Sampling: Positive-to-negative anchor sampling ratios are fixed to the recommendations of the original paper for each meta-architecture.Sampling follows the matching step to achieve a desired balance of positive and negative anchors.
  • Location loss: The box encoding applies scalar multipliers 10 to center coordinates and 5 to logarithmic width and height terms.These multipliers are also used in several prior works.
  • Location loss: All experiments use the Smooth L1, or Huber, loss for localization.This follows the cited prior works.

3.3. Input size configuration.

The study varies input resolution to trade detection accuracy for speed, using high- and low-resolution settings across the meta-architectures.

  • Input resolution: Faster R-CNN and R-FCN scale images to M pixels on the shorter edge, whereas SSD resizes images to M × M.Thus, image-size configuration differs between the two-stage and SSD meta-architectures.
  • Input resolution: Downscaling the input is evaluated as a way to trade accuracy for speed.The paper explicitly compares high- and low-resolution versions of each model.
  • Input resolution: High-resolution models use M = 600, while low-resolution models use M = 300.These settings are trained for each model to explore the accuracy-speed trade-off.
  • Input resolution: At the same nominal setting, SSD processes fewer pixels on average than Faster R-CNN or R-FCN.This holds when the other variables are kept constant.

3.4. Training and hyperparameter tuning

The models are trained end-to-end with architecture-specific optimization choices and evaluated on COCO using standardized post-processing and official metrics.

  • Training: All models are jointly trained end-to-end with asynchronous gradient updates on a distributed cluster.Optimization differs by meta-architecture: SGD with momentum for Faster R-CNN and R-FCN, and RMSProp for SSD.
  • Training: Faster R-CNN and R-FCN use batch size 1, while SSD uses batch size 32 except when memory requires reduction.Learning-rate schedules are manually tuned for each feature extractor.
  • Implementation: The Faster R-CNN and R-FCN training procedure differs from their typical four-stage training procedure.The implementation also replaces ROI Pooling variants with TensorFlow crop-and-resize using bilinear interpolation.
  • Evaluation: Models are trained on COCO training images plus validation images, holding out 8,000 examples for validation.Test detections use non-maximum suppression at an IoU threshold of 0.6 and clip boxes to the image window.
  • Evaluation: The official COCO API evaluates final detections using mAP averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05.The API also reports additional metrics.

3.5. Benchmarking procedure

Benchmarking measures runtime and memory under a specified hardware and preprocessing setup, while noting limits on comparisons with external reports.

  • Timing setup: Timing uses GPU inference with batch size one on an Nvidia GeForce GTX Titan X system.The machine has 32GB RAM and an Intel Xeon E5-1650 v2 processor.
  • Benchmark caveat: The reproduced SSD-with-VGG result matches the ECCV report but not the higher number from the newer arXiv version.The newer version uses improved data augmentation that the authors did not test.
  • Benchmark caveat: The COCO dataset split is similar to but slightly smaller than the trainval35k set used in several other papers.This creates a dataset-setting difference when comparing reported results.
  • Timing setup: Images are resized so their smallest dimension is at least k, then cropped to k × k with k = 300 or 600.Timing averages results over 500 images.
  • Timing caveat: Post-processing is included in timing and can take about 40 ms for the fastest models, capping the measured rate at 25 frames per second.The comparison is internally consistent but may not be directly comparable with external speed reports because hardware, software, framework, and batch size differ.
  • Memory measurement: Inference memory is measured with tfprof and averaged over three images.The paper presents this as a more platform-independent measure of memory demand.

3.6. Model Details

The study uses six ImageNet-pretrained feature extractors across unified Faster R-CNN, R-FCN, and SSD implementations, with architecture-specific training and feature-map configurations. These details support systematic comparisons across detector meta-architectures and extractors.

  • Unified implementations: The unified implementations use TensorFlow and compare Faster R-CNN, R-FCN, and SSD meta-architectures.The implementations vary meta-architecture, feature extractor, and associated configuration choices.
  • Faster R-CNN: Faster R-CNN uses feature maps from architecture-specific intermediate layers, followed by crop-and-resize and max-pooling for region processing.Stride and learning-rate settings vary across VGG, ResNet, Inception, and Inception ResNet extractors.
  • Feature extractors: Six feature extractors are used, all pretrained on ImageNet-CLS.Table 2 identifies their properties and defines Top-1 accuracy as ImageNet classification accuracy.
  • R-FCN: R-FCN uses position-sensitive score maps with extractor-specific layers, spatial bins, resizing, proposal training, and learning-rate schedules.The configurations include ResNet 101, Inception V2, Inception ResNet, and MobileNet variants.
  • SSD: SSD adds convolutional prediction layers with decaying spatial resolution and uses extractor-specific feature maps and training procedures.Additional layers, normalization, activations, and learning-rate schedules differ across VGG, ResNet 101, Inception V2, Inception ResNet, and MobileNet.

4. Results

The experiments map accuracy, speed, memory, and configuration choices across 147 detector models. They identify distinct operating points, including fast SSD models, a mid-range R-FCN or reduced-proposal Faster R-CNN balance, and highly accurate but slow Faster R-CNN models.

  • Experimental setup: 147 model configurations are benchmarked across meta-architecture, feature extractor, stride, input resolution, and proposal count.Measurements include GPU timing, memory demand, parameters, and floating-point operations; some high-resolution SSD configurations were omitted before convergence.
  • Accuracy vs time: Running time ranges from tens of milliseconds to almost 1 second per image, with R-FCN and SSD generally faster and Faster R-CNN generally slower but more accurate.Reducing Faster R-CNN proposals can move it toward the speed of the faster meta-architectures.
  • Critical points on the optimality frontier: SSD with Inception V2 or MobileNet occupies the fastest end of the optimality frontier, while MobileNet is roughly twice as fast as Inception V2 with slightly lower accuracy.This comparison ignores postprocessing costs.
  • Critical points on the optimality frontier: R-FCN with ResNet and Faster R-CNN with ResNet using 50 proposals occupy the frontier’s middle elbow, balancing speed and accuracy.The paper identifies these configurations as a central speed/accuracy sweet spot among its models.
  • Critical points on the optimality frontier: Faster R-CNN with Inception ResNet at stride 8 achieves the best reported frontier accuracy and state-of-the-art single-model performance, but requires nearly 1 second per image.The corresponding critical-point results are summarized in Table 3.
  • Feature extractors and object size: SSD performance is less reliant on feature-extractor classification accuracy than Faster R-CNN and R-FCN, while SSD is competitive on large objects but typically poor on small objects.The detector-versus-classifier relationship is shown for low-resolution models, and object-size comparisons fix image resolution at 300.
  • Image resolution: Halving input resolution in both dimensions lowers accuracy by 15.88% on average while reducing inference time by 27.4% on average.Higher resolution particularly improves small-object mAP, by a factor of 2 in many cases.
  • Number of proposals: Using 50 Faster R-CNN proposals retains 96% of the accuracy achieved with 300 proposals while reducing running time by a factor of 3.For Inception ResNet, 300 proposals gives 35.4% mAP, whereas 10 proposals gives 29% mAP.

4.2. State-of-the-art detection on COCO

The paper’s ensemble reaches state-of-the-art COCO performance, but it is an extreme speed/accuracy trade-off point because inference requires many network evaluations per image.

  • 41.3% mAP@[.5, .95] on the COCO test set exceeds the previous best result of 37.1% mAP@[.5, .95].The model is an ensemble of five Faster R-CNN models using ResNet and Inception ResNet feature extractors.
  • The ensemble improves small-object recall by nearly 60% relative to the previous best COCO detector.
  • The state-of-the-art ensemble is not directly comparable to the paper’s single-model results because it requires approximately 50 end-to-end network evaluations per image.The authors present it separately as an extreme point on the speed/accuracy trade-off curves.
  • Five Faster R-CNN models were selected using held-out validation performance while encouraging diversity through category-wise AP-vector comparisons.Models differed in feature extractors, output strides, loss variations, and training-data orderings.
  • Multicrop inference was used with the five-model ensemble, without multiscale training, horizontal flipping, box refinement, box voting, or global context.
  • The reported individual model numbers were computed on a held-out validation set and are not strictly comparable to official COCO test-dev results.They are expected to be very close.

5. Conclusion

The paper experimentally compares major factors affecting modern object-detector speed and accuracy and aims to help practitioners choose deployment methods. It also identifies ways to improve speed with little accuracy loss.

  • The study compares major aspects influencing the speed and accuracy of modern object detectors.
  • Using many fewer proposals than usual can improve Faster R-CNN speed without sacrificing much accuracy.
Loading 1611.10012v3…