PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepEdgeBench: Benchmarking Deep Neural Networks on Edge Devices

Stephan Patrick Baller, Anshul Jindal, Mohak Chadha, Michael Gerndt

arXiv:2108.09457v1cs.AIcs.DC

TL;DR

Strict latency requirements and incomplete comparisons of recent edge devices motivate evaluating DNN inference at the edge. The paper compares five devices and provides a measurement method; results favor Coral for quantized TensorFlow models, while Jetson Nano performs better at low MobileNetV2 inference-time fractions.

  • Problem

    Strict latency requirements can be difficult for distant cloud services to meet, and existing edge-device benchmarks may be outdated or omit major recent devices.

    Source: §I · p. 1

  • Method

    The paper compares inference time and power consumption across four SoCs and one microcontroller, evaluates accuracy, and provides an extensible measurement method.

    Source: §I · pp. 1–2, §VI · p. 10

  • Results

    For a quantized TensorFlow model, Google Coral Dev Board delivers the best inference-time and power-consumption performance; Jetson Nano is presumably more power efficient below 29.3% AI-computation time for MobileNetV2.

    Source: §VI · p. 10

  • Takeaways & Limitations

    Device choice depends on model and inference frequency: Coral leads for quantized TensorFlow models, while Arduino can suit small sporadic workloads despite low inference rates.

    Source: §VI · p. 10

    The paper identifies evaluation of additional SoCs and larger CNN models as future work.

    Source: §VI · p. 10

Abstract

from arXiv · show

EdgeAI (Edge computing based Artificial Intelligence) has been most actively researched for the last few years to handle variety of massively distributed AI applications to meet up the strict latency requirements. Meanwhile, many companies have released edge devices with smaller form factors (low power consumption and limited resources) like the popular Raspberry Pi and Nvidia's Jetson Nano for acting as compute nodes at the edge computing environments. Although the edge devices are limited in terms of computing power and hardware resources, they are powered by accelerators to enhance their performance behavior. Therefore, it is interesting to see how AI-based Deep Neural Networks perform on such devices with limited resources. In this work, we present and compare the performance in terms of inference time and power consumption of the four Systems on a Chip (SoCs): Asus Tinker Edge R, Raspberry Pi 4, Google Coral Dev Board, Nvidia Jetson Nano, and one microcontroller: Arduino Nano 33 BLE, on different deep learning models and frameworks. We also provide a method for measuring power consumption, inference time and accuracy for the devices, which can be easily extended to other devices. Our results showcase that, for Tensorflow based quantized model, the Google Coral Dev Board delivers the best performance, both for inference time and power consumption. For a low fraction of inference computation time, i.e. less than 29.3% of the time for MobileNetV2, the Jetson Nano performs faster than the other devices.

I. INTRODUCTION

The introduction motivates edge inference as a response to cloud latency constraints and gaps in existing device benchmarks. It presents a comparison of recent accelerator-based edge devices and a measurement method spanning inference speed, power, and accuracy.

I. INTRODUCTION

  • Cloud deployment can struggle with strict latency requirements when services are far from users, motivating computation at the network edge.The introduction describes edge computing as processing closer to where tasks are needed, reducing network strain and latency.Source: §I · p. 1
  • DNNs can require substantial storage and computing resources, while edge devices offer limited hardware resources despite their accelerators.Source: §I · p. 1
  • Existing edge-device benchmarks may be outdated or omit major recent devices, leaving comparisons for DNN inference incomplete.The paper focuses on recent accelerator-based devices for DNN inference.Source: §I · p. 1
  • The benchmark compares Asus Tinker Edge R, Raspberry Pi 4, Google Coral Dev Board, Nvidia Jetson Nano, and Arduino Nano 33 BLE.Source: §I · p. 1
  • The evaluation covers inference speed, power consumption during accelerated inference, and model accuracy with and without device-specific optimization.Source: §I · pp. 1–2

II. TARGET EDGE DEVICES, FRAMEWORKS AND MODEL FORMATS

  • Four inference frameworks are used: TensorFlow, TensorRT, TensorFlow Lite, and RKNN-Toolkit; the paper also provides an extensible measurement method and releases its data and method for further research.The method measures power consumption, inference time, and accuracy.Source: §I · p. 2

A. Target Edge Devices

The paper evaluates four SoCs and a microcontroller, with hardware differences that shape their edge-inference capabilities. The target-device overview includes specifications for the evaluated devices.

A. Target Edge Devices

1) Nvidia Jetson Nano:

  • The Jetson Nano uses a fully utilizable GPU, and this experiment tests its 4 GB RAM version in 10 W mode to maximize performance.Its GPU is based on Maxwell and has 128 CUDA cores.Source: 1) Nvidia Jetson Nano · p. 2

2) Google Coral Dev Board:

  • The Coral Dev Board combines Google’s Edge TPU with its SoC, and supported models can use the unit through PyCoral and TensorFlow Lite.The board is available with 1 GB or 4 GB RAM.Source: 2) Google Coral Dev Board · p. 2

3) Asus Tinker Edge R:

  • The Tinker Edge R uses Rockchip’s RK3399Pro NPU, rated at 3 TOPS, and supports model compilation through RKNN-Toolkit.The board also offers mPCIe, M.2, serial camera, and serial display interfaces.Source: 3) Asus Tinker Edge R · p. 2

4) Raspberry Pi 4:

  • The Raspberry Pi 4 lacks a dedicated neural-computation unit, unlike the other SoCs, but is distinguished by low price and availability.The evaluation uses its 4 GB variant and a 64-bit Raspbian system.Source: 4) Raspberry Pi 4 · p. 2

5) Arduino Nano 33 BLE:

The paper includes a microcontroller alongside SoCs and describes framework-specific routes for inference acceleration. Device support and model conversion constrain which accelerators can be used.

5) Arduino Nano 33 BLE:

  • The Arduino Nano 33 BLE represents microcontrollers and runs TensorFlow Lite Micro on a 64 MHz Cortex-M4 with 1 MB of flash.The paper states it can run networks up to 500 KB in size.Source: 5) Arduino Nano 33 BLE · p. 3

B. Inference frameworks and conversion between them

  • Device APIs convert and run models for accelerators, but framework support differs: Coral uses optimized TensorFlow Lite, Tinker Edge R supports Frozen Graph and TensorFlow Lite, and Jetson TensorRT does not support TensorFlow Lite.Source: B. Inference frameworks and conversion… · p. 3

1) Tensorflow:

  • TensorFlow exports pretrained models for edge inference, and its compatibility with Nvidia GPUs gives Jetson Nano access to its integrated GPU.The paper notes that other evaluated devices perform inference solely on the CPU in this TensorFlow context.Source: 1) Tensorflow · p. 3

2) TensorRT:

  • TensorRT turns a trained network into an optimized runtime engine using graph optimizations, layer fusion, and optimized kernels.Models from TensorFlow or other frameworks require external conversion tools to reach supported ONNX, Caffe, or UFF formats.Source: 2) TensorRT · p. 3

3) Tensorflow Lite:

The section describes model formats and conversion paths used to deploy pretrained TensorFlow models across edge devices. Support for quantization, operations, and device-specific toolkits determines which optimized formats are usable.

3) Tensorflow Lite:

  • TensorFlow Lite targets microcontrollers and IoT devices, and models can be converted from TensorFlow when their operations are supported.Unsupported operations can be implemented manually.Source: 3) Tensorflow Lite · pp. 3–4
  • Coral deployment requires a fully 8-bit-quantized TensorFlow Lite model with supported operations and compile-time constant tensor sizes meeting the stated dimensionality constraints.For the Arduino C++ API, the TensorFlow Lite model must also be converted to a C array.Source: 3) Tensorflow Lite · p. 4

4) RKNN-Toolkit:

  • Rockchip’s RKNN-Toolkit converts models from multiple frameworks into its .rknn format for inference on Rockchip AI-accelerated processors.Source: 4) RKNN-Toolkit · p. 4

C. Model formats

  • Pretrained TensorFlow models can use formats including SavedModel, TF1 Hub, Frozen Graph, and TensorFlow Lite, while TF-TRT and RKNN-Toolkit use other formats for accelerated inference.Source: C. Model formats · p. 4

1) SavedModel Format:

  • SavedModel contains the complete TensorFlow program, including trained parameters and operations, and by default cannot be trained further after saving.The paper describes it as the recommended export format.Source: 1) SavedModel Format · p. 4

2) TF1 Hub Format:

  • TF1 Hub is syntactically similar to TensorFlow 1 SavedModel but differs semantically to support module reuse, composition, and retraining.Loading it in the TensorFlow 2 Python API requires importing an additional module.Source: 2) TF1 Hub Format · p. 4

3) Frozen Graph Format:

The methodology describes model formats and a Python-based setup for measuring inference on edge devices. It separates testing from power monitoring and timestamps inference to define the measured workload.

3) Frozen Graph Format:

  • Frozen Graph stores model parameters as constants bound to operations; once loaded, it cannot be trained, though conversion to TF1 Hub is possible.The converted TF1 Hub model is also not trainable.Source: 3) Frozen Graph Format · p. 4

4) TFLite Format:

  • TFLite deploys converted TensorFlow models on resource-limited devices, but unsupported operations need custom runtime implementations.Conversion can include quantization, which the passage says can reduce model size and inference speed.Source: 4) TFLite Format · p. 4
  • For full device optimization, models need both an 8-bit-quantized TFLite version and another TensorFlow format; Raspberry Pi lacks a dedicated optimization unit.The authors select commonly used image-classification models from TensorFlow 1 Detection Zoo and Model Garden.Source: 4) TFLite Format · p. 4

III. METHODOLOGY

  • The measurement framework uses test.py for device inference and serial reader.py for power data, with Fig. 1 outlining component workflow.The methodology is divided into three task descriptions.Source: Figure 1 · p. 4, §III · p. 5
  • Inference and power monitoring run on separate devices to avoid consuming test-device resources, with NTP clock coordination and direct Ethernet connection.The stated local-network clock deviation is better than 1 ms.Source: §III · p. 5

A. Inference time calculation using test.py

  • test.py runs compatible models in TensorFlow, TFLite, RKNN-toolkit, or TensorRT with ONNX, records timestamps and outputs, and calculates accuracy after inference.The script supports configuring batch size, input datatype, image dimensions, and inference count; ImageNet or COCO data can be accessed over SFTP when device memory is insufficient.Source: A. Inference time calculation using… · p. 5
  • Inference timestamps bracket framework inference calls, so the measured time excludes image preprocessing and result evaluation.Source: A. Inference time calculation using… · p. 5
  • The generated data supports calculating overall and average inference time, alongside total testing-program time for N images.Fig. 2 plots power consumption over time with labels during repeated MobileNetV2 inference on the Jetson Nano.Source: A. Inference time calculation using… · p. 5, Figure 2 · p. 5

B. Power consumption calculation using serial reader.py

Power is sampled by a monitoring script and aligned to inference timestamps so consumption can be summarized for inference intervals.

B. Power consumption calculation using serial reader.py

  • A laboratory power supply with an integrated HM310P meter provides voltage and current readings through a serial Modbus USB interface.The authors implemented a Python library to control and read the device using user-contributed register documentation.Source: B. Power consumption calculation using… · p. 5
  • serial reader.py requests power, voltage, and current at intervals and can timestamp and save readings or plot them live.Its output can be plotted with test.py inference timestamps, as in the Jetson Nano MobileNetV2 example in Fig. 2.Source: B. Power consumption calculation using… · p. 5
  • The evaluation summarizes power readings within each batch's inference-timestamp interval to assess inference energy and average power.The stated outputs are overall energy consumption in wattminutes and average power in watts.Source: B. Power consumption calculation using… · pp. 5–6

C. Model accuracy calculation using test.py

The evaluation calculates classification accuracy from model outputs and tests devices under controlled configurations, including separate handling for the Arduino Nano 33 BLE.

C. Model accuracy calculation using test.py

  • Top-1 accuracy checks whether the highest-probability class matches ground truth, while Top-5 accepts a match among the five highest guesses.Accuracy is the relation of correct detections to all images.Source: C. Model accuracy calculation using… · p. 6
  • The script sorts output tensor values by weight and compares their class indices with ground truth; 5,000 ImageNet validation samples are used per run.This sample count is drawn from the 50,000 labeled validation images, providing an accuracy approximation.Source: C. Model accuracy calculation using… · p. 6
  • Arduino Nano 33 BLE testing uses a separate C++ TFLite script for MobileNetV1 inference, with inputs and outputs transferred over serial USB.The monitoring device runs test.py to handle dataset transmission and evaluation.Source: C. Model accuracy calculation using… · p. 6

IV. EXPERIMENTAL CONFIGURATIONS

  • Runs use 5,000 images for accuracy estimation and 1,000 repetitions of five images to reduce effects of power-reading delays on timing measurements.Source: §IV · p. 6
  • The evaluation disables unnecessary processes and tests native TensorFlow and TensorFlow Lite as well as optimized frameworks, collecting inference time, idle and inference power, and accuracy.The listed timing measures include average and total inference time for 5,000 images; power measures include average and total consumption over 1,000 repetitions of five images.Source: §IV · p. 6

A. Models for evaluation

Model selection balances device compatibility, especially the Arduino's memory limit, with the need to compare quantized and non-quantized configurations.

A. Models for evaluation

  • The Arduino Nano 33 BLE's 1 MB RAM limits the shared model choice to 8-bit-quantized MobileNet v1 0.25, available in TFLite and TF1 Hub formats.The model is referred to as MobileNetV1 Quant. Lite.Source: A. Models for evaluation · p. 6
  • The quantized TFLite model runs on the Asus Tinker Edge R, Google Coral Dev Board, Raspberry Pi, and Arduino, though its small size may reduce accuracy on other devices.On the Coral board, one-image inference takes less than 2 ms; timestamp offsets up to 1 ms may leave up to 50% of power data unassignable to the workload.Source: A. Models for evaluation · p. 6
  • A larger MobileNetV2 with 12 times as many parameters is also tested because longer evaluation runs improve inference-time measurement accuracy relative to incidental function-call overhead.Source: A. Models for evaluation · p. 6
  • Compatibility constraints split MobileNetV2 testing: the float Frozen Graph runs on Asus, Raspberry Pi, and Jetson, while its quantized TFLite version runs on Coral instead of Jetson.Source: A. Models for evaluation · p. 7
  • The non-quantized MobileNetV2 Lite configuration is included to examine the impact of quantization and TFLite use.All discussed classification models are trained on ImageNet ILSVRC2012.Source: A. Models for evaluation · p. 7

V. RESULTS

Inference time varied by device, model, and framework: Coral led on quantized MobileNetV2 with its dedicated AI unit, while Jetson Nano led among the reported non-quantized MobileNetV2 runs. CPU-only comparisons also show that framework and quantization effects differed across devices.

1) Inference performance without utilization of the dedicated AI units:

  • For MobileNetV2, inference times were 685.490 seconds on Nvidia Jetson Nano, 1037.7 on Raspberry Pi 4, 1389.4 on Coral Dev board, and 1652.1 on Asus Tinker Edge R.Figure 4 describes a 1 × 5000 images test run across devices using only CPU as well as dedicated AI units.Source: 1) Inference performance without… · p. 7, Figure 4 · p. 8
  • Compared with MobileNetV2, MobileNetV2 Lite reduced CPU inference time 43% on Tinker Edge R and 7% on Raspberry Pi 4, but increased it 32% on Coral and 28% on Jetson.Source: 1) Inference performance without… · p. 7
  • For CPU inference of quantized Lite models, Tinker Edge R led on MobileNetV2 Quant. Lite at 584.9 seconds, while its MobileNetV1 Quant. Lite NPU run took longer than CPU inference.The reported MobileNetV1 Quant. Lite times were 22.2 seconds on CPU and 24.0 seconds on the NPU.Source: 1) Inference performance without… · p. 7
  • Quantizing MobileNetV2 Lite reduced inference time 27.3% on Jetson Nano, 37.6% on Tinker Edge R, 39.3% on Coral, and 33.7% on Raspberry Pi 4.Source: 1) Inference performance without… · p. 7
  • Arduino Nano 33 BLE took 57534.297 seconds to infer 5,000 images, which the authors consider potentially acceptable when inference and idle power are also considered.Source: 1) Inference performance without… · p. 7

2) Inference performance with utilization of the dedicated AI units:

  • Devices generally ran models faster through their respective optimized APIs, except Tinker Edge R with MobileNetV1; its MobileNetV2 Frozen Graph and TFLite times were similar at 760.7 and 760.6 seconds.Source: 2) Inference performance with… · pp. 7–8
  • Coral Dev Board ran MobileNetV2 Quant. Lite in 20.788 seconds, while Jetson Nano led for non-quantized MobileNetV2 at 103.142 seconds with TensorFlow-TensorRT and 118.737 seconds with TensorRT.Jetson Nano did not have the quantized-model option in this comparison.Source: 2) Inference performance with… · pp. 7–8, Figure 4 · p. 8

B. Power consumption

CPU-only inference power varied across devices and models; Jetson Nano consumed the least for every model reported. The results also place Arduino's long inference runtime alongside its low idle draw.

B. Power consumption

1) Power consumption without utilization of the dedicated AI units:

  • Jetson Nano consumed the least CPU power for every model: 61.8 wattminutes for MobileNetV2, 52.3 for MobileNetV2 Lite, 33.5 for MobileNetV2 Quant. Lite, and 1.1 for MobileNetV1 Quant. Lite.Source: 1) Power consumption without… · pp. 8–9
  • Figure 6 predicts power consumption for different models and devices at different inference rates.Source: Figure 6 · p. 9

2) Power consumption with utilization of the dedicated AI units:

Dedicated AI units generally improve energy efficiency, but predicted device rankings depend on inference rate. Quantization reduces MobileNetV2 accuracy slightly, and memory pressure can erase Jetson Nano’s advantage on a larger model.

2) Power consumption with utilization of the dedicated AI units:

  • Coral uses 1.543 wattminutes across MobileNetV2 models; Tinker Edge R records 9.619 under MobileNetV2 Quant. Lite, versus Jetson Nano’s 13.630 under MobileNetV2.Devices are generally more energy efficient with their AI unit, except Tinker Edge R on MobileNetV1; Jetson Nano uses less power with TensorRT than TF-TRT for MobileNetV2 (13.640 versus 30.645 wattminutes).Source: 2) Power consumption with utilization… · p. 9

3) Power consumption prediction:

  • Power predictions plot maximum inferences per minute against wattminutes, using each model’s best power-efficiency score and LAN idle power.Initialization, preprocessing, and result evaluation are excluded, so actual real-world consumption is expected to be higher.Source: 3) Power consumption prediction · p. 9
  • Below 853 inferences per minute (<29.3% AI-computation time), Jetson Nano outperforms other devices; Raspberry Pi is more energy efficient than Coral below 238 inferences per minute.The latter comparison is for MobileNetV2 Quant. Lite and applies when 50.65% of the time is spent on AI computation.Source: 3) Power consumption prediction · p. 9
  • At 5 frames per minute, Arduino Nano 33 BLE consumes approximately 0.063 wattminutes, less than any other device, and this is its maximum MobileNetV1 inference rate.Source: 3) Power consumption prediction · p. 9

C. Accuracy

  • Using 5,000 of 50,000 validation images caused Raspberry Pi’s native TensorFlow accuracy to deviate; MobileNetV2 quantization lowered Top-1 by about 0.6% and Top-5 by about 0.4%.No significant accuracy changes were observed for most framework ports, though some differences appeared with RKNN-Toolkit, TensorFlow Lite Micro, and the Edge TPU in TensorFlow Lite.Source: C. Accuracy · p. 9

D. Performance on larger models

  • 433.331 seconds on Raspberry Pi without acceleration was similar to 416.745 seconds on Jetson Nano using its acceleration unit for SSD MobileNetV2 COCO inference.Jetson Nano’s shared GPU/CPU memory was insufficient during initialization, so slower SD-card swap space was used, making GPU inference slower than Raspberry Pi’s CPU inference.Source: D. Performance on larger models · p. 10, Figure 7 · p. 10

VI. CONCLUSION

DeepEdgeBench compares inference time and power consumption across four SoCs and one microcontroller, alongside a measurement method. Coral leads for continuous inference with quantized TensorFlow models in TFLite, while sporadic-workload rankings depend on inference rate and model size.

VI. CONCLUSION

  • Results vary with model size, quantization, framework, and anticipated inference rate, and the evaluation compares five devices across models and frameworks.The paper provides a measurement method for power consumption, inference time, and accuracy that can be extended to other devices.Source: §VI · p. 10
  • For quantizable TensorFlow models converted to TFLite, Google Coral Dev Board delivers the best inference-time and power-consumption performance.Jetson Nano can accelerate TensorFlow models on its GPU when they do not use only TFLite-compatible operations, and TensorFlow can also use its GPU for training.Source: §VI · p. 10
  • For sporadic computation, Arduino Nano 33 BLE is most power efficient for small models but has low inference rates; Jetson Nano presumably leads below 29.3% AI-computation time for MobileNetV2, otherwise Coral is most power efficient.Source: §VI · p. 10
Loading 2108.09457v1…