Source-linked AI summary
Benchmarking TinyML Systems: Challenges and Direction
Colby R. Banbury, Vijay Janapa Reddi, Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, Robert Hurtado, David Kanter, Anton Lokhmotov, David Patterson, Danilo Pau, Jae-sun Seo, Jeff Sieracki, Urmish Thakker, Marian Verhelst, Poonam Yadav
TL;DR
TinyML’s expanding hardware landscape lacks a widely accepted benchmark for fairly comparing systems and measuring progress. This position paper surveys the field, analyzes benchmark challenges, and proposes a path involving four benchmarks and an open/closed structure. It concludes that comparable hardware evaluation is needed to support continued progress in the rapidly evolving field.
Problem
TinyML lacks a reliable benchmark suited to its diverse, ultra-low-power workloads, including first-class power measurement and appropriate methodology.
Method
The paper reviews TinyML use cases, hardware, datasets, and models, analyzes benchmark-design challenges, and discusses four benchmarks with open and closed divisions.
Results
The paper identifies benchmark requirements and presents selected use cases, datasets, and models for four TinyML benchmarks.
Takeaways & Limitations
Comparable hardware evaluation is presented as necessary for continued progress and stability in the rapidly evolving TinyML field.
Takeaways & Limitations
Benchmark design is constrained by TinyML’s diverse systems, difficult energy measurement, limited memory, and software heterogeneity affecting reproducibility and comparability.
Abstract
from arXiv · showhide
Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workloads. Furthermore, we present our four benchmarks and discuss our selection methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf working group that is comprised of over 30 organizations.
1 INTRODUCTION
TinyML brings inference to ultra-low-power, microcontroller-class devices, enabling responsive, private, always-on applications. The paper argues that field complexity and software and hardware diversity require a fair, reliable benchmark and presents a collective call to establish one.
- Motivation: TinyML targets ML inference on devices typically consuming under 1 mW, avoiding wireless communication costs that exceed compute energy at this scale.On-device, near-sensor inference also supports responsiveness and privacy.
- Motivation: Its efficiency enables smart, battery-powered, always-on applications for real-time data collection and processing.
- System constraints: MCU constraints require hardware–software co-design because limited memory and MHz-scale compute restrict model inputs, layers, and sometimes model families.The paper cites devices with a few hundred kB of memory and limited onboard compute.
- Benchmarking gap: Field complexity and rapid hardware-driven progress obscure measurement and make design decisions difficult, motivating a reliable TinyML hardware benchmark.
- Paper scope: The paper discusses benchmark-development challenges and opportunities as a call to establish common benchmarking for TinyML workloads.Its viewpoints represent the TinyMLPerf working group, comprising over 30 organizations and 75 members.
2 TINY USE CASES, MODELS & DATASETS
TinyML spans established and emerging applications across audio, industrial sensing, activity recognition, and vision, but model and dataset selection remains difficult. The paper surveys representative use cases, datasets, and models while emphasizing that limited TinyML-specific data constrains realistic benchmarking.
- Use cases: TinyML includes wake-word speech classification, industrial anomaly detection, predictive maintenance, activity recognition, and emerging visual wake words.
- Use cases: Large-label image classification and object counting are suitable future always-on tasks but remain too compute- and memory-intensive for current TinyML hardware.
- Use cases: AR glasses could benefit from TinyML because always-on, battery-powered applications cannot tolerate cloud, edge-server, or mobile-device offloading latency.
- Datasets: Most deployed TinyML models use larger proprietary datasets, while available open datasets are often neither large nor TinyML-specific, limiting realistic workload representation.
- Models: Selecting representative models is challenging because MCU-class ML is recent and lacks widely accepted model families, although reproducible quality–accuracy rules could guide future models, runtimes, and hardware.
3 TINY HARDWARE CONSTRAINTS
TinyML hardware ranges from efficient 32-bit MCUs to specialized ultra-low-power inference engines, with power, memory, and compute constraints defining the platform class. On-chip memory, vector instructions, and specialized architectures extend feasible inference toward the ultra-low-power end.
- Hardware range: TinyML hardware typically operates at 1 mW or below, spanning efficient 32-bit Cortex-M7 and RISC-V PULP MCUs to novel inference engines.
- Power: TinyML devices can have power budgets up to four orders of magnitude smaller than state-of-the-art MLPerf systems.
- MCU capabilities: Cortex-M platforms use SIMD and DSP instructions to accelerate neural networks, SVMs, and 8-bit fixed-point feature computation.
- Memory: On-chip SRAM and embedded Flash avoid costly DRAM accesses when models fit within the available memory constraints.
- Specialized hardware: Specialized hardware can reach about one microjoule per inference and expands ML toward the ultra-low-power end of TinyML processors.
4 CHALLENGES
TinyML benchmarking is complicated by severe power and memory constraints, heterogeneous hardware, and diverse software deployment methods. A useful suite must balance optimality, portability, comparability, representativeness, and coverage across devices.
- Challenges: TinyML benchmark design faces four primary obstacles: low power, limited memory, hardware heterogeneity, and software heterogeneity.These challenges complicate systematic measurement and comparison of performance differences across systems.
- Low Power: Power measurements are difficult because devices consume drastically different amounts of power and vary in data paths, preprocessing, peripherals, and firmware.TinyML systems also lack spare cores that could absorb measurement overheads.
- Limited Memory: TinyML memory constraints are typically two orders of magnitude smaller than smartphone-scale resources, making benchmark overheads potentially significant.Benchmarks must represent multiple quantization and precision levels while remaining deployable across devices.
- Hardware Heterogeneity: Hardware heterogeneity spans general-purpose MCUs and novel architectures, while systems may lack standard features such as clocks or debug interfaces.Normalizing performance across such implementations is a key challenge, and existing benchmarks require re-engineering for this ecosystem.
- Software Heterogeneity: Hand coding can maximize numerical results but reduces reproducibility, comparability, and time efficiency, whereas proprietary code-generation tools limit portability.ML interpreters improve portability and generalizability but add modest binary-size and performance overhead.
- Benchmark Design: A benchmark suite should balance optimality with portability and comparability with representativeness while supporting multiple deployment options.The impact of deployment choice on results must be carefully evaluated.
5 RELATED WORK
Existing ML benchmarks do not adequately represent TinyML workloads on resource-constrained hardware. The paper therefore identifies a need for a benchmark that supports TinyML-specific workloads and treats power measurement as fundamental.
- Existing Benchmarks: Existing ML-related benchmarks do not accurately represent TinyML workloads on tiny hardware.The surveyed benchmarks are relevant to TinyML systems but fail to capture their distinctive requirements.
- CoreMark: CoreMark is easy to implement and uses real algorithms, but it neither profiles full programs nor accurately represents machine-learning inference workloads.
- MLMark: MLMark uses actual ML inference workloads, but its models require gigabytes of memory, have long runtimes, and omit power measurements.These characteristics make MLMark unrepresentative of MCU-class TinyML workloads.
- MLPerf: MLPerf’s current inference benchmark excludes MCUs and other resource-constrained platforms because it lacks small benchmarks and compatible implementations.
- Need for TinyML Benchmark: TinyML requires a benchmark tailored to its workloads, with power as a first-class metric and a methodology suited to tiny hardware.
6 BENCHMARKS
The paper proposes a TinyMLPerf benchmarking suite designed to balance comparability with the diversity of TinyML systems. It selects four representative use cases, datasets, and reference models, while prioritizing an evolving minimum viable benchmark.
- Benchmark Structure: The suite adopts MLPerf’s open and closed divisions to preserve comparability while accommodating heterogeneous TinyML systems.Closed submissions use an equivalent reference model; open submissions may deviate as necessary, including through software optimizations.
- Selection Methodology: Use-case selection prioritizes diversity, feasibility, and industry relevance.The process emphasizes broad field coverage, access to open-source datasets and models, and real-world applications.
- Use Cases: Four target use cases cover audio wake words, visual wake words, image classification, and anomaly detection.They span three input data types and range from relatively resource hungry visual wake words to lightweight anomaly detection.
- Datasets: Selected datasets define the use cases, train reference models, and generate on-device test sets.The datasets are also available for training new or modified models in the open division and were chosen for being open, well known, and industry relevant.
- Reference Models and Metrics: Four reference models serve as closed-division workloads and open-division baselines, including DS-CNN, MobileNetV1, and an eight-layer ResNet.The suite primarily measures inference latency, with optional energy measurement; open-division accuracy must remain within a threshold of the closed model.
- Evolution: The authors prioritize quickly establishing minimum viable benchmarks and iteratively addressing deficiencies as the suite evolves.Result submissions were planned for March 2021.
7 CONCLUSION
The paper concludes that TinyML needs hardware comparability to support continued progress and stability. It reviews the field, analyzes benchmark-development challenges, and presents four benchmark configurations with selected use cases, datasets, and models.
- TinyML requires comparability among hardware innovations to enable continued progress and stability.
- The paper reviews TinyML’s current landscape and the need for a hardware benchmark, then discusses associated challenges and a path forward.
- The authors select use cases, datasets, and models for four TinyML benchmarks.