PaperlayerView exact revision on arXiv

Source-linked AI summary

Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottlenecks

Amirali Boroumand, Saugata Ghose, Berkin Akin, Ravi Narayanaswami, Geraldo F. Oliveira, Xiaoyu Ma, Eric Shiu, Onur Mutlu

arXiv:2109.14320v1cs.ARcs.LG

TL;DR

The Edge TPU’s monolithic design struggles with variation across edge neural-network models and their layers. Mensa uses multiple specialized accelerators and a runtime scheduler to place layers. Across 24 Google edge models, Mensa-G improves energy efficiency and throughput over both the Edge TPU and Eyeriss v2.

  • Problem

    The Edge TPU’s one-size-fits-all design struggles to achieve high throughput and energy efficiency across heterogeneous edge models and layers.

    Source: §1 · p. 1

  • Method

    Mensa schedules neural-network layers across small heterogeneous accelerators specialized for layer families, considering layer fit and inter-layer dependencies.

    Source: §1 · p. 2

  • Results

    3.0x higher energy efficiency and 3.1x higher throughput than the Edge TPU, averaged across 24 Google edge NN models, are achieved by Mensa-G.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluated edge models’ layer characteristics vary substantially, supporting a design that uses a small number of specialized accelerators rather than one monolithic accelerator.

    Source: §1 · pp. 1–2

    Mensa’s heuristic scheduler may not always produce the best mapping decisions attainable by a hypothetical oracle scheduler.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Emerging edge computing platforms often contain machine learning (ML) accelerators that can accelerate inference for a wide range of neural network (NN) models. These models are designed to fit within the limited area and energy constraints of the edge computing platforms, each targeting various applications (e.g., face detection, speech recognition, translation, image captioning, video analytics). To understand how edge ML accelerators perform, we characterize the performance of a commercial Google Edge TPU, using 24 Google edge NN models (which span a wide range of NN model types) and analyzing each NN layer within each model. We find that the Edge TPU suffers from three major shortcomings: (1) it operates significantly below peak computational throughput, (2) it operates significantly below its theoretical energy efficiency, and (3) its memory system is a large energy and performance bottleneck. Our characterization reveals that the one-size-fits-all, monolithic design of the Edge TPU ignores the high degree of heterogeneity both across different NN models and across different NN layers within the same NN model, leading to the shortcomings we observe. We propose a new acceleration framework called Mensa. Mensa incorporates multiple heterogeneous edge ML accelerators (including both on-chip and near-data accelerators), each of which caters to the characteristics of a particular subset of NN models and layers. During NN inference, for each NN layer, Mensa decides which accelerator to schedule the layer on, taking into account both the optimality of each accelerator for the layer and layer-to-layer communication costs. Averaged across all 24 Google edge NN models, Mensa improves energy efficiency and throughput by 3.0x and 3.1x over the Edge TPU, and by 2.4x and 4.3x over Eyeriss~v2, a state-of-the-art accelerator.

1. Introduction

Edge inference must serve diverse neural-network models on resource-constrained devices, but the Edge TPU’s monolithic design struggles with their varied layer characteristics. Mensa responds with specialized accelerators and layer scheduling, improving performance across the evaluated Google models.

1. Introduction

  • Edge TPU throughput and energy efficiency fall short across diverse models, while its large buffers can consume substantial energy yet fail to prevent off-chip traffic.The cited analysis reports 24% average peak throughput utilization, 37% of theoretical peak energy efficiency, and, for CNNs, 48.1% of static and 36.5% of dynamic energy spent on on-chip buffers.Source: §1 · p. 1
  • Layer characteristics vary across models and within each model, with up to two orders of magnitude of variation within a single model.The paper notes that edge models use techniques such as depthwise separable and pointwise group convolutions to reduce computation and layer footprint.Source: §1 · p. 1
  • Fixed-size PE arrays, large buffers, and fixed dataflows can miss reuse opportunities and leave PEs underutilized across diverse edge layers.The paper also notes that reconfigurable accelerators may lack customization of essential parameters and incur online reconfiguration overheads.Source: §1 · p. 2
  • Mensa assigns layers to small on-chip or near-data accelerators specialized for layer families, accounting for accelerator fit and inter-layer dependencies.Its runtime scheduler uses layer characteristics and communication costs to choose where each layer executes.Source: §1 · p. 2
  • Mensa-G uses three accelerators: Pascal for compute-centric layers, Pavlov for LSTM-like data-centric layers, and Jacquard for other data-centric layers.Pavlov and Jacquard are placed in the logic layer of 3D-stacked memories and use smaller PE arrays than Pascal.Source: §1 · p. 2
  • 3.0x higher energy efficiency and 3.1x higher throughput than the Edge TPU, averaged across 24 Google edge NN models, are achieved by Mensa-G.Mensa-G also reduces total inference energy by 66.0% versus the Edge TPU and improves efficiency and throughput by 2.4x and 4.3x over Eyeriss v2.Source: §1 · p. 2

2. Background

The paper evaluates four neural-network families and describes the Edge TPU and the evaluation’s scope. The supplied text introduces the TPU shortcomings but does not detail them in this section.

2. Background

  • CNNs capture spatial features, LSTMs and Transducers process sequences, and RCNNs combine convolutional spatial extraction with recurrent sequence prediction.The paper focuses on long-term recurrent convolutional networks (LRCNs) as a popular RCNN type.Source: §2 · pp. 2–3

3. TPU & Model Characterization

  • The evaluation uses a 64x64 PE array and two large SRAM buffers across 24 Google models, but model specifics are undisclosed.The authors expect similar performance and energy characteristics for public models such as MobileNet and ResNet because of similarities to some Google models.Source: §3 · p. 3

3.1. Google Edge TPU Shortcomings

  • The section introduces three Edge TPU shortcomings and says their causes are discussed in Section 3.2.4.The supplied passage does not enumerate the shortcomings.Source: §3.1 · p. 3

1. The accelerator often suffers from extreme underutilization of the PEs.

The Edge TPU operates well below peak throughput across the evaluated models, with especially severe underutilization in recurrent models.

1. The accelerator often suffers from extreme underutilization of the PEs.

  • 75.6% below peak throughput on average, the Edge TPU’s measured performance falls far short of its 2 TFLOP/s theoretical peak.Transducer and LSTM models achieve less than 1% of peak throughput; CNN and RCNN models average 40.7% of peak utilization.Source: §1 · p. 3, Figure 1 · p. 3
  • Less than 1% of peak throughput is achieved by Transducer and LSTM models, the most underutilized model types.CNN and RCNN models reach only 40.7% of peak utilization on average, with a minimum of 10.2%.Source: §1 · p. 3
  • 37.2% of maximum possible energy efficiency is achieved on average across models; LSTM and Transducer models reach 33.8%.Even the best CNN model achieves only 50.7% of maximum efficiency.Source: §1 · p. 3

3. The accelerator’s memory system design is neither effective nor efficient.

The Edge TPU spends substantial energy on both on-chip buffers and off-chip memory, while its buffers often fail to cache model parameters effectively. The accompanying analysis examines edge models at layer granularity, including LSTMs and Transducers.

3. The accelerator’s memory system design is neither effective nor efficient.

  • 50.3% of total energy on average goes to off-chip memory accesses; CNN on-chip buffers also consume 48.1% of static and 36.5% of dynamic energy.For LSTMs and Transducers, approximately three quarters of total energy is spent on DRAM accesses.Source: §3 · pp. 3–4, Figure 2 · p. 4
  • Even at eight times the Edge TPU buffer capacity, only 46.5% of LSTM and Transducer parameters are effectively cached.The larger buffer reduces latency by 37.6% and energy consumption by 40.3%; the existing buffers occupy 79.4% of total area.Source: §3 · p. 4

3.2. Layer-Level Study of Google Edge Models

  • The paper analyzes individual layers to understand the sources of Edge TPU shortcomings and introduces a focused analysis of LSTMs and Transducers.The supplied passage names the analysis but does not provide the three properties it identifies.Source: §3.2 · p. 4, §3.2.1 · p. 4

1. Large parameter footprint.

LSTM and Transducer layers have much larger parameter footprints than CNN layers. Figure 3 relates LSTM gate matrices and combined layer footprints to FLOP/B.

1. Large parameter footprint.

  • LSTM and Transducer layers average 33.4 MB in parameter footprint, significantly larger than CNN layers.Each LSTM gate averages 2.1 million parameters, and LSTM layers can reach 70 million parameters.Source: §1 · p. 4

3. Intra- and inter-cell dependencies.

Layer characteristics vary substantially across and within edge NN models, while the Edge TPU's monolithic design contributes to utilization and energy inefficiencies. The analysis links these shortcomings to layer dependencies, data reuse, and memory behavior.

3. Intra- and inter-cell dependencies.

  • The Edge TPU serializes LSTM gates as fully connected layers, missing opportunities for parallel computation across gates and reducing PE utilization.LSTM cell computation has inter-cell dependencies through the previous hidden vector and intra-cell dependencies because the current hidden vector requires outputs from all four gates.Source: §3 · p. 4

3.2.2. Analysis of CNNs.

  • Across four edge CNN models, layer MAC intensity and parameter footprint vary by 200x and 20x, respectively.FLOP/B also varies 244x across layers in five representative CNN models; pointwise layers reuse input activations, unlike depthwise layers.Source: §3.2.2 · pp. 4–5

3.2.3. Analysis of RCNNs.

  • RCNN layers combine CNN and LSTM characteristics, yielding higher footprints, lower FLOP/B ratios than CNN layers, and greater variation.RCNNs include both CNN and LSTM layers.Source: §3.2.3 · p. 5

3.2.4. Sources of Edge TPU Shortcomings.

  • Low-reuse layers with large parameter footprints can leave PEs idle while waiting on long-latency DRAM cache misses.The passage gives 1–64 FLOP/B and 0.5–5 MB footprints for such layers, and notes that 32 GB/s LPDDR4 bandwidth is two orders below 2 TB/s needed to sustain peak throughput at one FLOP/B.Source: §3.2.4 · p. 5
  • A single dataflow misses layer-specific reuse opportunities, while fixed-size PE arrays struggle with differing layer shapes and dependencies.The missed reuse can make PEs wait for data not retained on-chip; the authors also call for better MVM scheduling and appropriately sized PE arrays.Source: §3.2.4 · p. 5
  • On-chip buffers are ineffective for many parameter-heavy or low-reuse layers, while their large size also makes accesses costly for layers that benefit from caching.The Edge TPU spends 50.3% of total inference energy on off-chip parameter traffic; its 4 MB parameter buffer is effective for layers averaging only 0.21 MB in parameter footprint.Source: §3.2.4 · p. 5

3.3. Key Takeaways

  • The analysis identifies layer heterogeneity, the Edge TPU's monolithic design, and the need to customize accelerator components to layer characteristics as key takeaways.The components named are the PE array, dataflow, on-chip memory, and off-chip memory bandwidth.Source: §3.3 · p. 5

4. Mensa Framework

Mensa distributes NN layers across specialized accelerators and uses a runtime scheduler to map layers to hardware. Its two-phase mapping process considers both accelerator suitability and communication overhead.

4. Mensa Framework

  • Mensa is a framework for exploiting inter- and intra-layer variation across edge NN models to improve efficiency and performance.Its design distributes layers across smaller specialized accelerators rather than relying on one monolithic accelerator.Source: §4 · p. 5, §4.1 · p. 5

4.1. High-Level Overview

  • Mensa combines heterogeneous accelerators with a runtime scheduler that maps each layer using model and hardware characteristics.Layers tend to group into a small number of families, so Mensa implementations typically need only a small number of accelerators.Source: §4.1 · pp. 5–6

4.2. Mensa Runtime Scheduler

  • Phase I selects an ideal accelerator for each layer in isolation; Phase II accounts for communication costs when assigning its destination accelerator.Phase II may keep a layer on the previous destination when transferring operands would cost more than execution there; it can instead choose the ideal accelerator when compute or parameter-transfer conditions warrant it.Source: §4.2 · p. 6
  • The initial scheduler does not run multiple layers concurrently, and its heuristic may not match an oracle's best mapping.During execution, accelerators transfer activations through DRAM to avoid keeping data coherent across accelerators.Source: §4.2 · p. 6

5. Mensa-G: Mensa for Google Edge Models

Mensa-G groups the diverse layers in Google edge models into families and uses those characteristics to guide a three-accelerator design. Its accelerators specialize in compute-centric or data-centric layers, with the scheduler coordinating their use and inter-layer communication.

5.1. Identifying Layer Families

  • 97% of analyzed layers group into five families distinguished by parameter footprint and reuse, MAC intensity, and activation characteristics.The families span average Edge TPU PE utilization from 82% for Family 1 to 0.3% for Family 3.Source: §5.1 · pp. 6–7

5.2. Hardware Design Principles and Decisions

  • Families 1 and 2 share compute-centric characteristics, while Families 3 and 4 are data-centric; Family 5 joins the data-centric design despite its smaller parameter footprint.Different MAC intensity, parameter footprint, and reuse patterns affect PE array size, buffer size, and memory bandwidth needs.Source: §5.2 · p. 7

5.2.1. Determining the Number of Accelerators Needed.

  • Pascal serves Families 1 and 2, Pavlov serves Family 3, and Jacquard serves Families 4 and 5, based on their distinct dataflow and reuse needs.Pascal exploits parameter and activation reuse; Pavlov uses temporal reduction for activations; Jacquard reuses parameters but not activations.Source: §5.2.1 · p. 7

5.2.2. Template-Based Design Approach.

  • Mensa-G uses a common tiled architecture to ease integration with existing Edge TPU compiler toolchains, while accepting a limit on accelerator customization.The accelerators are independent and do not share resources.Source: §5.2.2 · p. 8

5.3. Pascal: Compute-Centric Accelerator Design

  • Pascal uses temporal reduction for output activations and spatial multicasting for parameters to reduce memory traffic in compute-centric layers.Its 32x32 PE array reaches 2 TFLOP/s peak throughput; its activation and parameter buffers are each 256 kB and 128 kB, respectively.Source: §5.3 · p. 8

5.4. Pavlov: LSTM-Centric Accelerator Design

  • Pavlov targets Family 3’s large parameter footprints by reusing weights across LSTM cells and placing the accelerator in 3D-stacked memory.Its 8x8 PE array reaches 128 GFLOP/s peak throughput; weights stream from DRAM, while activations use a 128 kB buffer.Source: §5.4 · pp. 8–9

5.5. Jacquard Accelerator Design

  • Jacquard targets Families 4 and 5 with temporal parameter reuse and high off-chip bandwidth from placement in 3D-stacked memory.It uses a 16x16 PE array and 128 kB buffers for both activations and parameters.Source: §5.5 · p. 9

5.6. Data Transformations & Communication

  • Mensa schedules layers across accelerators and coordinates inter-layer communication, including extra traffic from models with frequent skip connections.Most evaluated models communicate between accelerators 4–5 times; the evaluation accounts for the extra communication traffic.Source: §5.6 · pp. 9–10

6. Experimental Methodology

The evaluation models 24 quantized Google edge models and estimates accelerator, memory, interconnect, and buffer energy and performance through simulation and analytical modeling.

Models

  • The study evaluates 24 Google edge models used in mobile applications; the models are TensorFlow Lite, fully 8-bit quantized, and compiled with the Edge TPU compiler.The authors expect similar results for MobileNet and ResNet, which share similarities with some evaluated models.Source: §6 · p. 10

Energy Analysis

  • The energy model sums static and dynamic energy across the accelerator, DRAM, interconnects, and on-chip buffers, using CACTI-P and LPDDR4-based estimates.It assumes each 8-bit MAC consumes 0.2 pJ/bit.Source: §6 · p. 10

Performance Analysis & Simulation

  • An in-house simulator models the Edge TPU and Mensa accelerators, integrating analytical dataflow costs and CACTI-P buffer-latency estimates.The modeled 3D-stacked-memory accelerators access 256 GB/s internal HBM bandwidth; both systems are assumed to use 2 GB of HBM DRAM.Source: §6 · p. 10

7. Evaluation

Across 24 Google edge models, Mensa-G improves energy efficiency and throughput over the Edge TPU and Eyeriss v2. Its latency reductions are largest for LSTMs and Transducers.

7.1. Energy Analysis

  • Mensa-G significantly reduces energy across all models: on-chip and off-chip parameter-traffic energy falls 15.3x versus Baseline, while dynamic buffer and NoC energy falls 49.8x versus Base+HB and 6.2x versus Eyeriss v2.Mensa-G reduces static energy by 3.6x and 5.6x versus Base+HB and Eyeriss v2, respectively.Source: §7.1 · p. 10

7. Evaluation

  • Mensa-G increases throughput by 3.1x, 1.3x, and 4.3x over Baseline, Base+HB, and Eyeriss v2, respectively; reported average utilization gains are 2.5x, 2.0x, and 2.6x.Source: §7.2 · p. 11

7.1. Energy Analysis

  • Base+HB’s high memory bandwidth reduces energy by only 7.5% on average; Eyeriss v2 reduces energy versus Baseline by 6.4% across LSTM and Transducer models and 36.2% for CNN models.Source: §7.1 · p. 10

7.2. Utilization and Throughput Analyses

  • Mensa-G increases average PE utilization by 2.5x/2.0x/2.6x over Baseline/Base+HB/Eyeriss v2, with the largest gains for LSTMs and Transducers.Their average utilization and throughput improve 82.0x and 5.7x over Baseline.Source: §7.2 · p. 11

7.3. Latency Analysis

  • 1.96x/1.17x lower inference latency over Baseline/Base+HB on average accompanies 5.4x/1.26x reductions for LSTMs and Transducers.CNNs and RCNNs see 1.64x/1.16x latency reductions over Baseline/Base+HB.Source: Figure 12 · p. 11, §7.3 · p. 11

8. A Case for Mensa Beyond the Edge TPU

The authors argue that Mensa’s multi-accelerator approach can address model diversity beyond the Edge TPU. They also describe how its scheduler can incorporate accelerators for emerging model types.

Accounting for Diversity in Other State-of-the-Art Accelerators

  • The authors suggest that insights from Google edge models can inform systolic-array accelerators, including Eyeriss v2, MAERI, and HDA.The evaluated models include widely used models and emerging types such as LSTMs and Transducers.Source: §8 · p. 11

Adopting Multi-Accelerator Frameworks

  • Mensa can manage customized instances of other accelerator designs, with each instance provisioned for a subset of models.The authors argue that maximizing hardware efficiency can justify additional design effort under tight area and energy budgets.Source: §8 · p. 11

Providing Flexibility to Incorporate New NN Model Types

  • Mensa’s software scheduler can incorporate new accelerators and map layers to them without significant changes to runtime software.The authors present this as a way to reduce design and deployment costs for new model types that do not run optimally on the three proposed accelerators.Source: §8 · p. 12

9. Related Work

Prior accelerator studies often target one model type, while work on layer heterogeneity has largely focused on CNNs or dataflow reconfiguration. Mensa is distinguished by addressing diverse edge models and coordinating broader accelerator design choices.

Studies on a Single NN Model Type

  • Single-model-type studies, predominantly on CNNs, do not analyze diverse edge NN classes or optimize one accelerator for multiple model types.Many also use traditional models such as AlexNet and VGG, rather than state-of-the-art edge models.Source: §9 · p. 12

Studies on NN Heterogeneity

  • MAERI and Eyeriss v2 reconfigure on-chip networks, but cannot tailor key memory and bandwidth parameters to layers and require frequent online reconfiguration.Their fixed dataflow and reconfiguration make co-design with the memory system difficult.Source: §9 · p. 12

Other Heterogeneous Accelerators

  • ScaleDeep targets cloud CNN training, while Neurosurgeon uses older vision and speech models and offloads layers to the cloud.These targets differ from Mensa’s local edge inference setting and diverse model scope.Source: §9 · p. 12

Concurrent Heterogeneous Dataflow Accelerators

  • HDA varies dataflows, whereas Mensa jointly considers compute, dataflow, on-chip memory, and off-chip bandwidth across more model types and edge constraints.The passage notes that Transducers would likely retain bandwidth underutilization and memory-traffic inefficiency under HDA.Source: §9 · p. 12

10. Conclusion

The paper analyzes Edge TPU bottlenecks across 24 Google edge NN models and proposes Mensa, a heterogeneous accelerator framework with a layer scheduler. Its three-accelerator design improves energy efficiency and throughput over both Edge TPU and Eyeriss v2.

10. Conclusion

  • 3.0x energy efficiency and 3.1x throughput gains over Edge TPU, with 1.96x lower inference latency, across 24 Google edge NN models.Against Eyeriss v2, the design improves energy efficiency by 2.4x and throughput by 4.3x.Source: §10 · p. 12
  • Mensa integrates multiple small heterogeneous accelerators and schedules each layer according to its characteristics.The framework uses a runtime scheduler to choose the accelerator for each layer.Source: §10 · p. 12
  • The design uses three accelerators with customized compute, memory, and dataflow, based on the observation that layers group into a small number of clusters.The analysis covers CNNs, LSTMs, Transducers, and RCNNs.Source: §10 · p. 12

Related papers

Loading 2109.14320v1…