PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmark Analysis of Representative Deep Neural Network Architectures

Simone Bianco, Remi Cadene, Luigi Celona, Paolo Napoletano

arXiv:1810.00736v2cs.CV

TL;DR

Existing work provides limited evidence about how DNN computational cost relates to recognition accuracy across architectures and hardware settings. This paper benchmarks more than 40 image-recognition DNNs on a high-end workstation and an embedded system using multiple performance indices. The results reveal non-linear relationships and hardware-dependent throughput behavior relevant to architecture selection under resource constraints.

  • Problem

    Few studies analyze computational cost and how it relates to recognition accuracy across DNN architectures, despite practical deployment constraints.

  • Method

    The paper benchmarks more than 40 state-of-the-art DNN architectures on two hardware platforms using accuracy, complexity, memory, computational cost, and inference-time measurements.

  • Results

    Recognition accuracy does not increase with operation count; throughput limits achievable accuracy, and real-time performance differs between high-end GPUs and embedded systems.

  • Takeaways & Limitations

    The analysis supports selecting DNN architectures according to the resource constraints of practical deployments and applications.

Abstract

from arXiv · show

This work presents an in-depth analysis of the majority of the deep neural networks (DNNs) proposed in the state of the art for image recognition. For each DNN multiple performance indices are observed, such as recognition accuracy, model complexity, computational complexity, memory usage, and inference time. The behavior of such performance indices and some combinations of them are analyzed and discussed. To measure the indices we experiment the use of DNNs on two different computer architectures, a workstation equipped with a NVIDIA Titan X Pascal and an embedded system based on a NVIDIA Jetson TX1 board. This experimentation allows a direct comparison between DNNs running on machines with very different computational capacity. This study is useful for researchers to have a complete view of what solutions have been explored so far and in which research directions are worth exploring in the future; and for practitioners to select the DNN architecture(s) that better fit the resource constraints of practical deployments and applications. To complete this work, all the DNNs, as well as the software used for the analysis, are available online.

I. INTRODUCTION

The paper addresses limited analysis of how DNN computational cost relates to recognition accuracy by benchmarking more than 40 architectures across workstation and embedded hardware. It reports several non-linear accuracy, complexity, throughput, efficiency, and memory findings relevant to research and deployment.

  • Few prior studies analyze computational cost and, more importantly, how memory usage and inference time relate to recognition accuracy.
  • The benchmark compares more than 40 state-of-the-art DNN architectures on ImageNet-1k using accuracy, model complexity, memory usage, computational complexity, and inference time.
  • Recognition accuracy does not increase with the number of operations, and model complexity has no linear relationship with accuracy.
  • Desired throughput places an upper bound on achievable accuracy, while parameter-use efficiency differs across DNN models.
  • Almost all models achieve super-real-time performance on a high-end GPU, but only some guarantee it on an embedded system.
  • Even DNNs with very low model complexity require a minimum GPU memory footprint of about 0.6GB.

II. BENCHMARKING

The benchmark framework publicly implements DNN comparison experiments on a workstation and an embedded system with substantially different computational resources. Using both systems highlights how architecture choice affects memory usage and inference time.

  • The framework is implemented in Python with PyTorch, cuDNN-v5.1, and CUDA-v9.0, and its code and models are publicly available.
  • Experiments run on both a workstation and an embedded NVIDIA Jetson TX1 system.
  • The workstation uses an NVIDIA Titan X Pascal GPU with 3840 CUDA cores, whereas the Jetson TX1 uses a Maxwell GPU with 256 CUDA cores.
  • Using the two systems highlights the importance of computational resources for DNN memory usage and inference time.

III. ARCHITECTURES

The study benchmarks diverse DNN architectures, including effectiveness-oriented and efficiency-oriented models, under standardized model and input-handling procedures. It evaluates them using multiple complementary performance indices.

  • The analyzed architectures include models designed for higher effectiveness and models designed for greater efficiency in embedded vision applications.
  • Architecture names with a following number use that number to indicate the number of layers containing learnable parameters.
  • The benchmark covers AlexNet, VGG, Inception, ResNet, DenseNet, ResNeXt, Xception, DualPathNet, squeeze-and-excitation, MobileNet, and ShuffleNet families.
  • Models are directly collected or converted to PyTorch using the same sampling policies for a fair comparison.
  • Inputs are normalized RGB mini-batches with model-dependent spatial sizes of 331, 229, or 224 pixels.
  • The benchmark measures accuracy rate, model complexity, memory usage, computational complexity, and inference time.

A. ACCURACY RATE

Accuracy is evaluated for image classification on the ImageNet-1k validation set using Top-1 and Top-5 metrics. The reported procedure uses only the central crop.

  • Top-1 and Top-5 accuracy are estimated on the ImageNet-1k validation set for image classification.
  • Predictions use the central crop only, while multi-crop averaging can achieve slightly better performance.

B. MODEL COMPLEXITY

The benchmark measures model complexity through learnable parameters and parameter-file size, alongside total memory, computational cost, and accuracy-related visualizations. Figure 1 compares accuracy with FLOPs while ball size encodes model complexity.

  • Model complexity: Model complexity is analyzed using the total number of learnable parameters and the parameter-file size in megabytes.The parameter-file size helps estimate the minimum GPU memory required by each model.
  • Accuracy and complexity: Figure 1 plots Top-1 and Top-5 center-crop accuracy against single-pass FLOPs, with ball size representing model complexity.
  • Memory usage: Total memory consumption includes the network model and memory required to process batches of sizes 1, 2, 4, 8, 16, 32, and 64.
  • Computational complexity: Computational cost is measured in FLOPs, counting each multiply-add as two FLOPs.This convention treats multiplication and addition as separate operations.

E. INFERENCE TIME

Inference time is measured per image across batch sizes on both a high-end Titan X Pascal GPU and an embedded Jetson TX1 board. The reported accuracy-complexity plots compare architectures using single-forward-pass measurements on both platforms.

  • Measurement setup: Inference time is reported in milliseconds per image for the NVIDIA Titan X Pascal and NVIDIA Jetson TX1 across the evaluated batch sizes.Each reported time is averaged over 10 runs for statistical validation.
  • Inference-time comparison: The accuracy-complexity plots use Top-1 and Top-5 ImageNet-1k validation accuracy and single-forward-pass computational complexity measured on both platforms.
  • Observed trade-offs: NASNet-A-Large reaches the highest Top-1 and Top-5 accuracy while also having the highest computational complexity.
  • Observed trade-offs: Among models below 5 G-FLOPs, SE-ResNeXt-50 (32x4d) has the highest Top-1 and Top-5 accuracy with approximately 2.76 M-params.

B. ACCURACY-RATE VS LEARNING POWER

The paper evaluates how efficiently DNNs use their parameters through Top-1 accuracy density. This comparison identifies efficient architectures and highlights accuracy-efficiency trade-offs across models.

  • Accuracy density: SqueezeNets, ShuffleNet, MobileNets, and NASNet-A-Mobile use their parameters most efficiently.
  • Accuracy-efficiency trade-off: Among the most efficient models, NASNet-A-Mobile and MobileNet-v2 provide substantially higher Top-1 accuracy.
  • Accuracy-efficiency trade-off: Among models exceeding 80% Top-1 accuracy, Inception-v4 and SE-ResNeXt-101 (32x4d) use parameters most efficiently.
  • Measurement constraint: Inference measurements were averaged over 10 runs, with missing larger-batch measurements attributed to insufficient system memory.

D. ACCURACY-RATE VS INFERENCE TIME

The benchmark relates Top-1 accuracy to throughput on Titan Xp and Jetson TX1 systems, showing that higher throughput constrains the maximum achievable accuracy. It also reports memory requirements for different batch sizes.

  • Throughput trade-off: The throughput upper bounds have nearly identical intercepts but the Titan Xp slope is 8.3× smaller than the Jetson TX1 slope.The reported slopes are −0.0244 for Titan Xp and −0.2025 for Jetson TX1, indicating a lower accuracy decay on Titan Xp as throughput increases.
  • Throughput trade-off: At more than 250 FPS on Titan Xp, ResNet-34 provides the highest accuracy with 73.27% Top-1 accuracy.
  • Throughput trade-off: For accuracy above 75%, ResNet-50 achieves the maximum throughput at 18,83 FPS; above 80%, SE-ResNeXt-101 (32x4d) achieves 7,16 FPS.
  • Memory usage: On Titan Xp, most models require less than 1GB of memory at batch size 1, and none requires more than 1.5GB.

F. MEMORY USAGE VS MODEL COMPLEXITY

On the Titan Xp, model complexity reliably estimates total memory utilization for batch size 1. The relationship is linear but follows two lines with similar slopes and different intercepts.

  • Model complexity can reliably estimate total memory utilization on the Titan Xp for batch size 1.The analysis measures initial static allocation of model parameters against total memory utilization.
  • The relationship is linear, following two lines with approximately equal slopes of 1.10 and 1.15.The two lines have different intercepts of approximately 910 and 639.

G. BEST DNN AT GIVEN CONSTRAINTS

The best architecture depends on hardware memory and throughput constraints. Titan Xp supports higher reported accuracy than Jetson TX1 under the stated resource conditions, while Jetson requires at least 1GB memory for comparable accuracy.

  • 75.95% is the maximum Titan Xp accuracy under low memory usage, achieved by DPN-68 regardless of computational time.Low memory usage is defined as ≤0.7GB.
  • At least 1GB memory is needed on Jetson TX1 for accuracy comparable to the best Titan Xp models.Under this condition, ResNet-50 reaches 76.01% accuracy with half real-time throughput.

VI. CONCLUSION

This paper benchmarks more than 40 ImageNet-1k DNN architectures across accuracy, complexity, memory, computation, and inference time to guide architecture selection under deployment constraints. Its findings show that accuracy is not determined by operations or model complexity alone, throughput limits achievable accuracy, and hardware strongly affects real-time feasibility.

  • More than 40 ImageNet-1k DNN architectures are compared across accuracy, parameters, memory usage, computational complexity, and inference time.The study is intended to guide architecture selection for resource-constrained practical deployments.
  • Recognition accuracy does not increase with the number of operations, and this finding is independent of the computer architecture.SE-ResNeXt-50 (32x4d) is cited as achieving very high accuracy with relatively few operations.
  • Model complexity and accuracy have no linear relationship, and DNNs use their parameters with different levels of efficiency.Parameter efficiency is analyzed through accuracy density.
  • Desired throughput places an upper bound on achievable accuracy, while model complexity can reliably estimate total memory utilization.The memory-utilization relationship is reported as linear on the Titan Xp.
  • Almost all models achieve real-time or super-real-time performance on a high-end GPU, but only a few guarantee it on an embedded system.The experiments compare a workstation with a Titan X Pascal GPU and an embedded Jetson TX1 system.
  • Even DNNs with very low model complexity require about 0.6GB of GPU memory, and the models and analysis software are available online.The repository is intended to support further exploration and addition of new entries.
Loading 1810.00736v2…