Source-linked AI summary
An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks
Kiran Seshadri, Berkin Akin, James Laudon, Ravi Narayanaswami, Amir Yazdanbakhsh
TL;DR
Edge TPU performance must be understood across diverse neural-network structures and accelerator configurations. This paper evaluates three Edge TPU classes on nearly 423K convolutional networks and develops graph-based learned models that estimate accelerator metrics with about 97% accuracy and correlations above 99%.
Problem
The paper addresses limited understanding of how different Edge TPU architectures perform across convolutional neural networks with diverse structures.
Method
The authors evaluate three Edge TPU classes across nearly 423K networks and use graph neural networks to predict accelerator performance metrics from network graphs.
Results
The graph-based model estimates latency and energy consumption with around 97% accuracy and correlations above 99% with ground truth data.
Takeaways & Limitations
The learned model supports rapid accelerator architecture exploration and hardware/software co-design as an alternative to expensive cycle-accurate simulation.
Takeaways & Limitations
The evaluation lacks V3 results for one reported analysis because V3 was not available.
Abstract
from arXiv · showhide
Edge TPUs are a domain of accelerators for low-power, edge devices and are widely used in various Google products such as Coral and Pixel devices. In this paper, we first discuss the major microarchitectural details of Edge TPUs. Then, we extensively evaluate three classes of Edge TPUs, covering different computing ecosystems, that are either currently deployed in Google products or are the product pipeline, across 423K unique convolutional neural networks. Building upon this extensive study, we discuss critical and interpretable microarchitectural insights about the studied classes of Edge TPUs. Mainly, we discuss how Edge TPU accelerators perform across convolutional neural networks with different structures. Finally, we present our ongoing efforts in developing high-accuracy learned machine learning models to estimate the major performance metrics of accelerators such as latency and energy consumption. These learned models enable significantly faster (in the order of milliseconds) evaluations of accelerators as an alternative to time-consuming cycle-accurate simulators and establish an exciting opportunity for rapid hard-ware/software co-design.
1. Introduction
The paper evaluates three classes of Edge TPUs across nearly 423K diverse convolutional neural networks and derives architectural performance insights. It also develops learned models that estimate accelerator metrics with low error and high rank correlation.
- Evaluation scope: Nearly 423K convolutional neural networks are used to evaluate three Edge TPU classes deployed or planned for Google products.The networks span diverse structures and convolution operations.
- Architectural insights: The study analyzes how Edge TPU architectures perform across models with different sizes and structures, tile sizes, and substituted operations.It reports performance deltas after replacing one operation with another.
- Learned modeling: The learned models estimate critical Edge TPU performance metrics with around 3% error and approximately 0.99 rank-based correlation with ground truth.The authors position these models as alternatives to expensive cycle-accurate simulation for design exploration.
2. Edge TPU Microarchitecture
Edge TPUs use a parameterizable template organized as a two-dimensional array of SIMD processing elements. Their hierarchy combines shared PE memory, core-local parameter storage, and multi-way MAC lanes.
- Template architecture: The template-based accelerator organizes parameterizable microarchitectural components in a 2D array of processing elements.This design supports exploration of architecture configurations for different target applications.
- Processing elements: Each processing element performs arithmetic computations in SIMD fashion while an on-chip controller transfers data and instructions.The controller stages activations and parameters before execution on the processing elements.
- Memory hierarchy: PE memory stores activations, partial results, and outputs, while core memory stores parameters for cores containing multiple MAC lanes.The hierarchy provides shared memory at the PE level and parameter storage at the core level.
3. Edge TPU Software Ecosystem
The Edge TPU software stack compiles TensorFlow Lite models ahead of time and maps supported operations onto the accelerator. Parameter caching reduces repeated external-memory transfers during continuous inference.
- Compilation and runtime: TensorFlow Lite models are compiled ahead of time by the Edge TPU compiler, which maps supported neural-network operations onto the accelerator.The runtime library connects the accelerator with the TensorFlow Lite API.
- Parameter caching: Parameter caching reuses weights across consecutive inferences instead of reloading the entire model for each input.This optimization targets scarce on-chip scratch-pad memory.
- Parameter caching: Cached parameters reduce external-memory transfers, improving performance and energy efficiency for continuous inference.The benefit applies when weights are fully or partially retained on chip.
- Convolution mapping: Convolution mapping places input activations, partial sums, and outputs in PE memory and weights in core memory.The compiler may choose different tiling and mappings based on memory capacities and tensor sizes.
- Convolution mapping: A mapped convolution uses four SIMD lanes with a four-element activation and weight example, illustrating the accelerator’s computation steps.The figure distinguishes active and inactive data elements during execution.
4. Learned Performance Model
The paper represents convolutional networks as graphs and uses a graph neural network to learn accelerator performance from graph structure and features. The model encodes, updates, and decodes graph attributes, using a final global scalar as the predicted metric.
- Learned Performance Model: The learned model uses encoder, core, and decoder components, with the core performing message passing to capture structural dependencies.The final updated global feature is used as the predicted latency, energy, or other performance metric.
- Learned Performance Model: A graph neural network predicts accelerator performance by encoding operations as nodes, connectivity as edges, and graph-level information as global features.Message passing learns node representations that are aggregated into a graph-level representation.
- Input representation: The model maps NASBench-101 operations to feature vectors, assigns uniform edge and global features, and represents each sampled cell as a graph.Input, output, and valid operation types receive distinct scalar encodings.
- Learned Performance Model Structure: Graph Net updates edge, node, and global attributes through neural blocks, aggregating information across graph components.The core is assembled from edge, node, and global update blocks.
- Accelerator configurations: The studied Edge TPU configurations cover different peak-throughput and on-chip-memory domains, including high-TOPS and low-TOPS designs.Table 2 defines V1 as high peak TOPS, V2 as low peak TOPS with small on-chip memory, and V3 as low peak TOPS with large on-chip memory.
5. Methodology
The study evaluates three Edge TPU configurations on nearly 423K diverse convolutional neural network architectures using uniform cycle-accurate simulations and learned performance models.
- Workloads: The workload uses nearly 423K NASBench-101 convolutional neural network architectures with varied structures and convolutional operations.Architectures use 3×3 convolution, 1×1 convolution, and 3×3 max-pooling operations.
- Accelerator configurations: The three studied Edge TPU configurations differ in clock frequency, PE memory, cores per PE, and Y-PE count.V1, V2, and V3 operate at 800 MHz, 1066 MHz, and 1066 MHz; V3 has 2 MB PE memory and eight cores per PE, while V2 has 384 KB and one core per PE.
- Microarchitectural simulations: A fully parameterized cycle-accurate simulator measures workload latency and energy consistently across accelerator configurations.Parameter caching is enabled in all simulations.
- Learned performance model training: A graph network model is trained on the 423K data points using a 60% training, 20% validation, and 20% testing split.The model uses Adam with learning rate 1e-3 and two-layer feed-forward blocks with 16 neurons per layer.
6. Evaluation
Across 417,454 high-accuracy NASBench models, Edge TPU performance depends on model structure and the balance between caching, memory bandwidth, and accelerator organization. V2 leads overall high-accuracy performance, while V1 often benefits smaller or medium models and V3 favors models with many 1×1 convolutions.
- Overall results: 94.33% mean validation accuracy at 5.65 ms makes V2 the strongest configuration across 417,454 models meeting the 70% accuracy threshold.The result is attributed mainly to V2’s 32 KB core memory and 32 GB/s I/O bandwidth.
- Energy: Energy favors V2 for models below 3.0 ms but V1 for larger, slower models because V1’s larger PE memory reduces costly buffer refills.The latency–energy relationship is linear across the evaluated models.
- Accuracy–latency trade-offs: A 0.16% accuracy reduction from 95.055% to 94.895% yields up to 1.78× lower latency on V1 with 66% fewer parameters.The accelerator winner changes from V2 for the highest-accuracy model to V1 for the slightly less accurate model.
- Graph structure: Increasing graph width generally lowers latency through greater parallelism, whereas graph depth beyond three does not improve accuracy and usually increases latency.Depths four and five are an exception because their graphs have fewer average parameters.
- Operation structure: Latency rises with more 3×3 convolutions, especially when both convolution count and depth increase, because 3×3 operations carry more parameters.The first three latency buckets average 1.48, 2.0, and 3.0 3×3 convolutions, respectively.
- Parameter scale: V1 performs best for medium models with 5–30 million parameters because its larger on-chip SRAM caches more of the model.For larger models, V2 and V3 benefit from higher memory bandwidth as parameter streaming dominates latency.
7. Conclusion
The paper evaluates three Edge TPU classes across more than 423K convolutional neural networks, extracts interpretable architectural trade-offs, and develops learned models for performance estimation.
- The graph-based learned model estimates Edge TPU latency and energy consumption with around 97% accuracy and correlations above 99% with ground truth.The authors propose this accuracy as a basis for rapid architecture exploration and hardware/software co-design.