Source-linked AI summary
MLPerf Tiny Benchmark
Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Jeremy Holleman, Nat Jeffries, Csaba Kiraly, Pietro Montino, David Kanter, Sebastian Ahmed, Danilo Pau, Urmish Thakker, Antonio Torrini, Peter Warden, Jay Cordaro, Giuseppe Di Guglielmo, Javier Duarte, Stephen Gibellini, Videet Parekh, Honson Tran, Nhan Tran, Niu Wenxu, Xu Xuesong
TL;DR
TinyML lacks a fair and reliable way to compare systems across diverse hardware and software stacks. MLPerf Tiny addresses this gap with an open, modular suite of four benchmarks measuring accuracy, latency, and energy. Its submissions demonstrate flexibility across divisions, vendors, frameworks, and hardware platforms.
Problem
TinyML needs a fair, reliable, and reproducible benchmark because diverse stacks make direct comparison difficult.
Method
MLPerf Tiny provides four reference-backed benchmarks measuring inference accuracy, latency, and energy with modular submission divisions.
Results
Submissions covered open and closed divisions, hardware and software vendors, diverse frameworks, and platforms ranging from MCUs to accelerators and FPGAs.
Takeaways & Limitations
The benchmark accommodates contributions at different deployment-stack components while providing controlled comparison and community baselines.
Takeaways & Limitations
Streaming inputs and preprocessing choices can distort measurements because test-runner bandwidth and feature-extraction scope complicate faithful evaluation.
Abstract
from arXiv · showhide
Advancements in ultra-low-power tiny machine learning (TinyML) systems promise to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted and easily reproducible benchmark for these systems. To meet this need, we present MLPerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems. The benchmark suite is the collaborative effort of more than 50 organizations from industry and academia and reflects the needs of the community. MLPerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems. Additionally, MLPerf Tiny implements a modular design that enables benchmark submitters to show the benefits of their product, regardless of where it falls on the ML deployment stack, in a fair and reproducible manner. The suite features four benchmarks: keyword spotting, visual wake words, image classification, and anomaly detection.
1 Introduction
TinyML brings machine learning inference to ultra-low-power edge devices, but diverse deployment stacks make fair comparison difficult. MLPerf Tiny addresses this need with an open, modular benchmark suite measuring accuracy, latency, and energy.
- TinyML achieves inference under a milliWatt, enabling responsive, private, always-on applications near sensors.On-device inference avoids wireless communication costs, which are higher than compute at this scale.
- Diversity across the TinyML stack makes standardization and direct comparison challenging.
- MLPerf Tiny provides four standard benchmarks selected by more than 50 organizations in academia and industry.
- The suite measures accuracy, latency, and energy to capture tradeoffs among TinyML systems.
- Reference implementations provide open-source community baselines while modularity supports hardware and software users.
2 Challenges
TinyML benchmarking must handle severe resource constraints and heterogeneous hardware and software stacks. These conditions make fixed interfaces and conventional benchmarks unrepresentative, motivating flexible controlled evaluation.
- Low Power: Power measurement is difficult because devices consume drastically different power and may include different data paths and preprocessing steps.
- Limited Memory: TinyML devices typically have resources two orders of magnitude smaller than smartphones, making benchmark overhead and model size critical.
- Hardware Heterogeneity: TinyML systems span devices with different performance, power, and capabilities, including MCUs, event-based processors, and memory-compute architectures.
- Hardware Heterogeneity: Hardware heterogeneity complicates benchmarking because systems may lack standard features such as system clocks or debug interfaces.
- Software Heterogeneity: TinyML inference stacks are tightly coupled to hardware, so benchmark restrictions can produce unrepresentative performance results.
- Cross-product: A modular benchmark should let users demonstrate improvements at specific pipeline layers or through end-to-end solutions in a controlled setting.
3 Related Work
Existing ML benchmarks do not adequately represent TinyML workloads on resource-constrained hardware. They lack suitable model sizes, runtimes, power measurements, or compatible implementations, leaving a distinct need for a dedicated benchmark.
- Existing ML-related hardware benchmarks do not accurately represent TinyML workloads on tiny hardware.
- CoreMark is easy to implement but does not profile full programs or accurately represent machine learning inference workloads.
- MLMark uses actual ML inference workloads, but its models require too much memory and have runtimes unsuitable for MCU-class devices.
- MLMark also lacks power measurement, although power is critical for TinyML benchmarking.
- The existing MLPerf inference benchmark excludes MCUs and other resource-constrained platforms because it lacks small benchmarks and compatible implementations.
- These gaps establish a need for a TinyML benchmark that treats power as a first-class concern and uses a suitable methodology.
4 Benchmarks
MLPerf Tiny targets model inference with four reference-backed benchmarks spanning visual, speech, and anomaly-detection workloads. The suite uses compact datasets and models designed for embedded systems, with explicit quality targets.
- MLPerf Tiny measures model inference without pre- or post-processing and defines each benchmark with a dataset, model, and quality target.
- Visual Wake Words: Visual Wake Words detects whether an image contains at least one person, using MSCOCO 2014 images resized to 96x96.
- Visual Wake Words: The Visual Wake Words model is a 325KB MobilenetV1 with 96x96 inputs and two output classes.
- Image Classification: Image Classification uses CIFAR-10 and a customized ResNetv1 that outputs a probability vector of size 10; its TFLite model is 96KB.
- Image Classification: The image-classification reference reaches 86.5% accuracy on 200 CIFAR-10 test images, with an 85% top-1 accuracy target.
- Keyword Spotting: Keyword spotting uses Speech Commands v2 and a 38.6K-parameter depthwise-separable CNN designed for microcontroller-scale memory.
- Keyword Spotting: The quantized keyword-spotting reference reaches 91.6% accuracy on the full test set and has a 90% accuracy requirement.
- Anomaly Detection: Anomaly detection addresses unsupervised detection of abnormal machine behavior using a ToyADMOS/MIMII-derived DCASE2020 dataset and an autoencoder-based model.
5 Run Rules
MLPerf Tiny uses a modular, reproducible benchmark harness with separate configurations for latency/accuracy and energy measurement. Its run rules define standardized procedures for measuring performance, accuracy, and energy while supporting both closed and open submissions.
- Benchmark design: The benchmark harness is modular and includes closed and open divisions to balance reproducibility with flexibility.The closed division controls models, datasets, and quality targets, while the open division permits changes across the machine-learning pipeline.
- Benchmark framework: The framework coordinates benchmark execution because TinyML platforms often lack file I/O, standard input/output, and interactive resources.It is based on EEMBC’s software development platform and controls the device under test.
- Measurement procedure: Latency is measured across five runs using sustained inference, with median inferences per second reported as the score.Each run downloads input data, loads the tensor, and performs inference for at least 10 seconds and 10 iterations.
- Measurement procedure: Accuracy is computed from validation-set output probabilities using Top-1 percent and AUC, subject to a model-specific minimum threshold.A score is valid only when the model meets its required minimum accuracy.
- Measurement procedure: Energy measurement follows the latency procedure while additionally reporting total energy in the timing window and micro-Joules per inference.Energy results also use the median of five measurements.
- Reference results: Reference implementations cover a broad range of latency and energy while each meets the minimum accuracy requirement.Results can be stored, reloaded, and examined through an energy viewer.
6 Benchmark Assessment
The benchmark assessment used scheduled, peer-reviewed submissions to test whether MLPerf Tiny could standardize results while accommodating diverse hardware and software approaches. The v0.5 round demonstrated broad flexibility and exposed trends in formats, frameworks, hardware, and power consumption, while remaining a snapshot of the field.
- Assessment process: MLPerf Tiny accepts submissions twice yearly and transparently peer-reviews results and implementations for rule compliance.Organizations run the benchmarks on their own hardware/software stacks before submitting by the deadline.
- Submissions: The v0.5 submissions included open and closed divisions from hardware and software vendors, demonstrating the suite’s modular flexibility.The benchmark accommodated different submitters’ goals by allowing them to target specific elements of their systems.
- Insights: 8-bit integer was the most common inference format because it offered a performance boost with little impact on model accuracy.The submissions also ranged from open-source interpreters to hardware-specific inference compilers.
- Insights: The submissions spanned MCUs, accelerators, and FPGAs, with FPGA platforms using variable-precision models for increased performance.Reported platform power consumption ranged from µWatts to Watts.
- Insights: Organizations used MLPerf Tiny to showcase hardware-agnostic SDKs, accelerator microarchitectures, RISC-V AI microcontrollers, and the hls4ml workflow.These examples illustrate how the benchmark can evaluate contributions at multiple layers of the TinyML stack.
- Scope: The first submission round provides a snapshot of TinyML, while future rounds can track how the field evolves.No v0.5 submission modified the training dataset, and current efforts remained focused largely on models, frameworks, and hardware.
7 Impact
MLPerf Tiny has already served as a standard task set and public development-platform project for TinyML research. The paper frames benchmarking and community collaboration as ways to support responsible deployment while addressing potential misuse and electronic waste.
- Research impact: MLPerf Tiny has already acted as a standard set of tasks for TinyML research and has become public projects on a TinyML development platform.The benchmark is positioned as a basis for standardization through competition and comparability.
- Responsible deployment: TinyML may democratize AI and preserve privacy through inexpensive, on-device intelligence, but it could also enable unwanted monitoring and increase electronic waste.The paper proposes collaborative standards for responsible deployment to mitigate these potential negative impacts.
8 Limitations
MLPerf Tiny’s current scope leaves important aspects of TinyML evaluation for future work. The main boundaries concern benchmark evolution, preprocessing treatment, and limited architectural coverage.
- Benchmark scope and stability: The benchmark will evolve with new application domains while retaining a subset of long-term-stable benchmarks for historical comparison.Open-source development is intended to support reproducibility as the suite changes.
- Streaming inputs and preprocessing: Preprocessing is not yet fully included, creating tradeoffs between realistic streaming evaluation and flexibility for joint feature-model optimization.The choice can also distort results when submitters vary feature extraction or classify it as preprocessing.
- Layer types and architectures: The closed division currently relies mainly on fully connected and convolutional layers, while additional architectures such as recurrent neural networks may be added later.Open-division submissions can deviate from the current reference architectures.
9 Conclusion
TinyML systems are difficult to compare meaningfully because the field is complex and rapidly changing. MLPerf Tiny addresses this need with open-source benchmarks and reference implementations that assess energy, latency, and accuracy.
- Conclusion: TinyML’s hardware and software diversity makes measuring progress and comparing systems meaningfully difficult.The paper frames fair, replicable, and robust evaluation as necessary for systematic development.
- Conclusion: MLPerf Tiny assesses TinyML hardware, models, and runtimes using benchmarks for energy, latency, and accuracy.The suite and its reference implementations are open-source.
- Conclusion: The benchmark suite and reference implementations are available as open-source resources.The paper provides the MLCommons Tiny repository URL for access.
A Benchmark Framework
The benchmark framework coordinates execution of a benchmark behavioral model across the system’s components. It provides the coordination layer for running benchmarks.
- A Benchmark Framework: The framework coordinates execution of the benchmark’s behavioral model among the system’s components.This coordination is described as the framework’s central role.
- A Benchmark Framework: The benchmark’s behavioral model is executed through coordination among multiple system components.The passage defines the framework by its cross-component execution role.
- A Benchmark Framework: The framework organizes benchmark execution rather than defining a single hardware component.Its scope is the interaction of the components involved in running the behavioral model.
A.1 Framework Hardware
The framework supports latency, accuracy, and energy measurements through host-runner and DUT interactions, with separate performance and energy hardware configurations. It standardizes execution while accommodating platform-specific implementations.
- Framework components: The framework uses a host PC, runner application, DUT, and thin DUT firmware shim to execute benchmarks.The runner coordinates initialization, test-data loading, inference triggering, and result collection.
- Hardware configurations: Two hardware configurations support latency and accuracy measurement or energy measurement with electrical isolation and an energy monitor.Both configurations share initialization, input loading, inference triggering, and result collection.
- Energy measurement: Energy scores measure only power delivered to the DUT core, excluding level-shifter power associated with the framework.The runner supports three energy-monitor devices: ST LPM01A, Joulescope JS110, and Keysight N6705.
- DUT interface: The DUT API requires timestamping, UART communication, input loading, single-inference execution, and prediction-result output.Timestamp behavior differs between performance mode and energy mode.
- Platform integration: Platform-specific submitter code connects the benchmark API to optimized libraries or hardware accelerators, while boilerplate code remains unchanged.This partitions the implementation between reusable framework code and code ported to each platform.
- Measurement procedure: Latency and energy scores use the median of five runs, while accuracy is computed from validation-set Top-1 percent and AUC subject to minimum thresholds.Energy additionally reports micro-Joules per inference, and latency reports median inferences per second.