PaperlayerView exact revision on arXiv

Source-linked AI summary

MEDEM: Multi-Engine DL Accelerator Design Methodology

Fareed Qararyah, Mohammad Ali Maleki, Pedro Trancoso

arXiv:2609.37399v1cs.ARcs.DCcs.PF

TL;DR

Flexible multi-engine DL accelerators need combinations of engines suited to diverse workloads, but existing approaches lack a systematic design methodology. MEDEM co-designs candidate engines and selects a combination under a resource budget; its evaluation reports improvements over state-of-the-art designs across workloads and resource budgets.

  • Problem

    Existing multi-engine DL accelerator approaches lack a systematic methodology for co-designing engines and choosing combinations that minimize aggregate execution cost across diverse workloads.

    Source: §I · p. 2, D. Motivating a Two-Stage Design… · pp. 3–4

  • Method

    MEDEM co-designs a pool of candidate engines and selects a multiset of them to maximize workload coverage within a practical resource budget.

    Source: §I · p. 2

  • Results

    Up to 4.84× in energy-delay product (EDP) and 1.59× in throughput: MEDEM-derived CMEAs outperform homogeneous and heterogeneous designs modeled after Simba and Maelstrom across 51 workloads and varied resource budgets.

    Source: §I · p. 2

  • Takeaways & Limitations

    MEDEM identifies co-designed accelerators that outperform the evaluated state-of-the-art baselines across single- and multi-model workloads and resource budgets.

    Source: §I · p. 2

Abstract

from arXiv · show

Multi-engine deep learning (DL) accelerators are becoming increasingly prevalent as they address the heterogeneity and growing complexity of modern DL workloads. To efficiently process diverse DL workloads, these accelerators must incorporate combinations of engines with complementary capabilities to match the distinct computational characteristics of these workloads' heterogeneous kernels. However, existing multi-engine DL accelerator design approaches lack a systematic methodology, leaving fundamental questions unresolved. These include how to co-design engines for workloads with diverse computational characteristics and which engine combinations minimize aggregate execution costs (such as time or energy) across such workloads. Addressing these questions requires efficient exploration of exponentially large design spaces. To address these questions systematically, this work proposes Multi-Engine DL Accelerator Design Methodology (MEDEM). MEDEM defines generic engine abstractions, co-designs candidate instances (engines), and selects a combination of co-designed engines to minimize aggregate execution cost given diverse DL workloads and a resource budget. MEDEM encompasses a set of design strategies that efficiently navigate the exponentially large design spaces of engine co-design and combination selection, identifying highly optimized multi-engine accelerators. A comprehensive evaluation demonstrates that MEDEM identifies accelerators that outperform state-of-the-art designs, delivering geometric-mean improvements of up to 4.84x in energy-delay product (EDP) and 1.59x in throughput. The improvements are achieved using different resource budgets, demonstrating MEDEM's scalability, and using 51 single- and multi-model DL workloads, demonstrating its generalizability.

I. INTRODUCTION

MEDEM separates engine co-design from engine-combination selection to address the difficulty of designing flexible accelerators for diverse DL workloads. Its strategies explore these design spaces efficiently, and evaluation reports improvements over state-of-the-art baselines across workloads and resource budgets.

I. INTRODUCTION

  • Existing flexible multi-engine designs lack a reusable systematic methodology, may not transfer across resource settings, and can use overlapping engines that provide poor workload coverage.Workload coverage means processing a workload set with minimal aggregate execution cost, such as time or energy.Source: §I · p. 2
  • MEDEM first co-designs candidate engines for distinct kernel characteristics, then selects a multiset from that pool to maximize workload coverage within a resource budget.Co-designed Multi-Engine Accelerators (CMEAs) are accelerators designed using MEDEM.Source: §I · p. 2
  • HCo prioritizes hardware or mapping axes by their impact on execution cost, exploring high-impact axes exhaustively and lower-priority axes heuristically.Its pruning techniques remove redundant and provably suboptimal regions of the search space.Source: §I · p. 2
  • BASS searches combinations between replicating the best-average engine and maximizing specialization, incrementally refining candidates before selecting the lowest-cost combination.The candidates come from the pool generated by MEDEM’s engine co-design stage.Source: §I · p. 2
  • Up to 4.84× in energy-delay product (EDP) and 1.59× in throughput: MEDEM-derived CMEAs outperform homogeneous and heterogeneous baselines modeled after Simba and Maelstrom.The evaluation uses 51 single- and multi-model workloads and resource budgets ranging from edge-typical to cloud-typical.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

DL models comprise layers that perform tensor operations, including convolution and matrix multiplication. Although models may contain many layers, they often repeat only a few unique layer shapes.

II. BACKGROUND AND MOTIVATION

B. DL Computing Engines

DL engines differ in which operation dimensions they map spatially across processing elements, with implications for their microarchitecture and data reuse. Multi-engine accelerators use independently controllable engines, while joint exploration of engine designs and combinations creates search-space challenges that motivate decoupling the two tasks.

B. DL Computing Engines

  • Engines spatially map either output dimensions, called non-reductive (NR), or dimensions requiring partial-sum accumulation, called reductive (R).NR mappings need no spatial reduction logic; R mappings require spatial reduction primitives such as inter-PE adder trees or global accumulation registers.Source: B. DL Computing Engines · p. 3

C. Multi-Engine DL Accelerators

  • Multi-engine accelerators can specialize engines for heterogeneous kernels, execute different model layers concurrently, and scale through modular designs.They also expose mapping and scheduling optimizations for multi-model workloads.Source: C. Multi-Engine DL Accelerators · p. 3

D. Motivating a Two-Stage Design Methodology

  • Flexible multi-engine design remains largely expert-driven, while jointly exploring engine configurations and combinations can impose substantial search costs.The joint space is dominated by combinations containing inefficient engines, grows multiplicatively, and can constrain per-engine hardware-mapping co-design.Source: C. Multi-Engine DL Accelerators · p. 3, D. Motivating a Two-Stage Design… · pp. 3–4
  • Decoupling engine co-design from combination selection enables structured exploration of high-quality candidates and makes joint hardware-mapping co-design more practical.The isolated engine-design space is smaller, and its separation allows application of insights from prior co-design research.Source: D. Motivating a Two-Stage Design… · p. 4

III. MEDEM OVERVIEW

MEDEM takes DL models, hardware-resource budgets, and an optimization goal, then co-designs candidate engines and selects a multiset to minimize aggregate execution cost. Its engine abstraction supports varied compute and memory configurations, while HCo prioritizes high-impact design axes for exhaustive exploration.

A. MEDEM Flow and Core Modules

  • MEDEM’s two modules co-design candidate engines for unique layers, then select a multiset that minimizes aggregate execution cost across the workload set.Coverage is quantified using the geometric mean of execution cost.Source: A. MEDEM Flow and Core Modules · p. 4

B. MEDEM Supported Engine Architectures

  • The engine abstraction combines a 2-D PE array, cluster-level L1 buffers, and global L2 buffers, and supports R- and NR-engine instances with varied mappings and dataflows.Each PE contains a MAC unit and local registers; buffers are modeled as buffets.Source: B. MEDEM Supported Engine Architectures · p. 4

IV. CANDIDATE ENGINE CO-DESIGN

  • HCo co-designs a pool of candidate engines from unique model layers and resource budgets; the pool may be smaller than the layer set when one engine suits multiple layers.The module includes a model parser and HCo.Source: §IV · p. 4

Hierarchical Co-designer (HCo)

1) Importance-based Design Axes Prioritization:

  • For each multi-start simulated-annealing search, HCo explores PS exhaustively, OB heuristically, and tile sizes and shapes (T) exhaustively after pruning.Array-shape dimensions are limited to powers of two, making exhaustive PS exploration practical.Source: 1) Importance-based Design Axes… · p. 5

3) Pruning Redundant, Dominated, and Undesired Design Points:

HCo prunes valid but redundant, dominated, or undesired design points across interdependent axes. If pruning leaves no designs under the budget, it relaxes dominated and undesired pruning while keeping redundant points excluded.

3) Pruning Redundant, Dominated, and Undesired Design Points:

  • HCo classifies valid but suboptimal designs as redundant, dominated, or undesired, and applies pruning across interdependent axes to support exhaustive exploration of important ones.Redundant points are equivalent; dominated points are provably suboptimal; undesired points reflect preferences or implementation constraints.Source: 3) Pruning Redundant, Dominated, and… · p. 5

2) HCo Components and Flow:

  • For convolution with batching, interchangeable-dimension pruning reduces (7!)^3 loop-order possibilities to thousands by removing equivalent tile and loop-order permutations.The example assumes engines permit any loop order at any memory level.Source: 2) HCo Components and Flow · p. 5
  • Global buffers larger than the maximum effectual size are dominated, while tiles divisible by a larger fitting tile are pruned as guaranteed suboptimal.A smaller tile that is larger in at least one dimension is not necessarily dominated.Source: 2) HCo Components and Flow · p. 5
  • Undesired loop orders can be pruned by requiring dimensions associated with a layer output to be collocated as outermost at a memory level for output-stationary mapping.Collocated dimensions appear consecutively in a loop order.Source: 2) HCo Components and Flow · p. 5
  • When pruning leaves no valid designs within budget, HCo reintroduces dominated and undesired designs in order of increasing design-space expansion, while redundant designs stay excluded.Exploration is repeated after each reintroduction.Source: 2) HCo Components and Flow · p. 6

V. ENGINE SELECTION

MEDEM evaluates candidate engines against unique layers, weights their execution costs by layer importance, and uses BASS to search engine multisets spanning average performance and specialization. It selects the combination with the lowest weighted aggregate cost.

A. cost matrix construction

  • Cross mapping search evaluates every unique layer on every candidate engine, reusing HCo to find mappings when the engine was not co-designed for that layer.The best-mapping execution costs are weighted by layer importance, defined as the number of layer instances across input models, and recorded in a cost matrix.Source: A. cost matrix construction · p. 6

B. Balanced Average and Specialization Search (BASS)

  • BASS searches engine multisets across a spectrum from replicating strong average performers to heterogeneous specialization, then selects the combination with the lowest aggregate execution cost.The candidate combinations come from MEDEM’s co-designed engine pool.Source: B. Balanced Average and Specialization… · p. 6
  • BASS seeds multi-start local searches by uniformly replicating the top h engines, then greedily accepts swaps that reduce aggregate execution cost until reaching local optima.It compares the local optima from multiple heterogeneity settings and returns the one with minimum weighted aggregate cost.Source: B. Balanced Average and Specialization… · p. 6

C. Evaluator

The evaluator estimates execution cycles and memory accesses using a Timeloop-based model, then uses offline component-energy values to estimate energy. Evaluation compares MEDEM-derived accelerators with Simba-like and Maelstrom-like baselines across engine counts and resource configurations.

C. Evaluator

  • The evaluator’s average EDP error against Timeloop-Accelergy is 2%, with only a few sampled cases below 90% accuracy.It estimates execution cycles and memory accesses and uses offline energy-per-operation and energy-per-access values.Source: C. Evaluator · p. 7
  • The evaluator supports engines within MEDEM’s abstraction but is not differentiable, unlike DOSA’s evaluator for square systolic arrays with weight-stationary dataflow.It decodes Timeloop and converts part of its iterative program into mathematical models.Source: C. Evaluator · p. 7

1) Baselines:

  • Spotlight’s top 10 design points are reevaluated with MEDEM’s Timeloop-Accelergy-based evaluator, and the best result serves as the co-design baseline.This re-evaluation addresses differences between Spotlight’s original MAESTRO and Timeloop-based evaluation.Source: 1) Baselines · p. 7
  • Baseline comparisons use Simba-like homogeneous and Maelstrom-like heterogeneous accelerators, with engine counts from 4 to 64 representing resource budgets from edge to cloud.For each baseline, two resource configurations are considered, and the better-performing configuration is used throughout the evaluation.Source: 1) Baselines · p. 7

2) System Modeling:

The evaluation specifies system models, workloads, mapping heuristics, and per-engine design-space and optimizer settings. It tests single- and multi-model CNN and Transformer workloads under stated modeling assumptions.

2) System Modeling:

  • Engine energy and execution time use models validated against Timeloop and Accelergy, with 7nm values except the 28nm Spotlight comparison; the interconnect is a modeled mesh NoC.The NoC uses 32 GB/s per link and 0.7 pJ/bit per hop.Source: 2) System Modeling · pp. 7–8

3) Workloads:

  • The workload suite combines design-time CNNs and Transformers with unseen evolved or structurally shifted models, including randomly formed design-time and exhaustive unseen multi-model combinations.The evaluation uses batch size 64, int8 weights and activations, int32 accumulators, and assumes off-chip memory can hold large-model weights.Source: 3) Workloads · p. 8

4) Workload Mapping Across Engines:

  • Both layer-sequential and layer-pipeline mapping heuristics are applied, and the better result is used for CMEAs and baselines.Layer-sequential processes layers one at a time; layer-pipeline distributes layers across engines and pipelines execution.Source: 4) Workload Mapping Across Engines · p. 8

5) Design Space and MEDEM Parameters:

  • The per-engine design space explores shared convolution and matrix-multiplication parallelism options and all loop orders at each of three memory-hierarchy levels, while excluding PE arrays with dimensions below 8.HCo's simulated annealing uses 50 cooling steps, 100 iterations per step, α = 0.95, and 10 random-seed runs; aggregate variance was negligible.Source: 5) Design Space and MEDEM Parameters · p. 8

6) Evaluation Metrics and Granularity:

The evaluation reports geometric-mean EDP and representative throughput across workload sets. MEDEM-derived CMEAs outperform homogeneous and heterogeneous baselines, with gains across engine counts and workload compositions.

1) Evaluation of MEDEM-Derived CMEAs:

  • 4.84× lower EDP and 1.59× higher throughput versus Simba-L, and 2.75× lower EDP and 1.51× higher throughput versus Maelstrom-L, are achieved by CMEAs.Throughput gains are more modest than EDP gains; the evaluation attributes this to energy optimization requiring spatial and temporal reuse across the memory hierarchy.Source: 1) Evaluation of MEDEM-Derived CMEAs · p. 8, Figure 6 · p. 9
  • CMEAs consistently outperform the baselines at all evaluated engine counts, despite the exponentially growing combination space.The text notes that expected growth in gains with engine count does not always occur, given limited workload heterogeneity and imbalanced layer contributions.Source: 1) Evaluation of MEDEM-Derived CMEAs · p. 8
  • Improvements over Simba-L increase with model count, while the largest improvements over Maelstrom-L occur for two-model workloads.The paper describes the single- versus multi-model trends as mixed.Source: 1) Evaluation of MEDEM-Derived CMEAs · p. 9, Figure 6 · p. 9
  • CMEAs improve over baselines across the experiments, supporting the conclusion that co-designed rather than independently optimized engines improve workload coverage and minimize aggregate execution cost.Source: 1) Evaluation of MEDEM-Derived CMEAs · p. 9

2) Evaluation of MEDEM Generalizability:

MEDEM-derived CMEAs retain effectiveness on unseen workloads, and HCo's co-designed engines usually outperform Spotlight's. Homogeneous CMEAs also improve on both baselines, underscoring the role of engine co-design.

2) Evaluation of MEDEM Generalizability:

  • MEDEM-derived CMEAs outperform baselines on unseen workloads across engine counts, and the opposing changes versus Simba-L and Maelstrom-L largely balance when both are considered.The paper says generalization should hold when design-time workloads represent the workload landscape, because MEDEM relies on unique-layer distributions rather than workload topology.Source: 2) Evaluation of MEDEM Generalizability · p. 9, Figure 8 · p. 9

1) Evaluation of HCo:

  • HCo's engines outperform Spotlight's in 94.8% of experiments, while Spotlight wins in 5.2%; HCo exceeds Spotlight by more than 8× in roughly one-fifth of experiments.The paper attributes the results to importance-based prioritization of high-impact design-space axes, despite HCo being simpler than Spotlight.Source: 1) Evaluation of HCo · p. 9, Figure 8 · p. 9

2) The Impact of Engine Co-design:

D. Evaluation of MEDEM Engine Selection Stage

BASS outperforms common-case replication and maximum specialization across engine counts, while workload layer distributions explain why additional specialization can have diminishing returns. The selected combinations use a range of engine configurations.

D. Evaluation of MEDEM Engine Selection Stage

  • BASS outperforms both common-case replication (DCC) and maximum specialization (SPEC) across engine counts, with gains over SPEC increasing as engine count rises.The gap between SPEC and DCC narrows with engine count, suggesting diminishing returns from engine heterogeneity.Source: D. Evaluation of MEDEM Engine… · p. 10
  • Q1 models have four dominant unique layers accounting for 99% of MACs, while Q3 models have 20 unique layers but only a subset accounts for most MACs.The trade-off implies a heterogeneity sweet spot: further specialization can harm Q1-like workloads and yield diminishing gains for less dominant layers in Q3-like workloads; similar trade-offs appear in multi-model workloads.Source: D. Evaluation of MEDEM Engine… · p. 10, Figure 11 · p. 10
  • BASS selects fewer unique engines than the total engine count, consistent with avoiding overspecialization and diminishing returns from further specialization.The selected combinations include replicated engines as well as engines used once.Source: D. Evaluation of MEDEM Engine… · p. 10, Figure 12 · p. 11

E. Overview of CMEAs Co-designed Engines

  • Across experiments, MEDEM selects 17 unique engines; the most-used engine appears 17% of the time, and R and NR engines have similar usage at 49.2% and 50.8%.The five most-used configurations vary in PE-array aspect ratios and L1/L2 buffer sizes and input-to-weight ratios; these ratios reflect layer variation that must be captured to minimize off-chip accesses.Source: E. Overview of CMEAs Co-designed Engines · pp. 10–11, Figure 13 · p. 11, Table IV · p. 11

VII. RELATED WORK

Prior work explores engine co-design and flexible engine combinations, but MEDEM offers a systematic, scalable methodology that separates engine co-design from combination selection. The conclusion reports gains in EDP and throughput over state-of-the-art designs.

VII. RELATED WORK

  • Engine hardware-mapping co-design explores large, interacting spaces using genetic algorithms, Bayesian optimization, Q-learning, VAEs, and gradient descent.The literature identifies sample-efficient exploration as necessary for these spaces.Source: §VII · p. 11
  • Flexible accelerators use homogeneous or heterogeneous engines, from Simba and Gemini’s NVDLA-like engines, to Maelstrom’s NVDLA and ShiDiaNao,.MOHam combines NVDLA, Eyeriss, and ShiDiaNao.Source: §VII · p. 11
  • Model-specific multi-engine accelerators typically use reconfigurable platforms to dedicate compute engines to a target model,,,,.Their design space is smaller because engines target few layers and many mapping parameters are fixed in hardware.Source: §VII · p. 11
  • MOSAIC supports broad MAC and non-MAC layers, but its unstructured joint exploration of engine configuration and composition limits effectiveness and scalability [15].Source: §VII · p. 11
  • MEDEM co-designs candidate engines and selects a combination, decoupling these stages through an efficient, scalable systematic methodology.It uses generic engine abstractions and aims to improve coverage of diverse DL workloads.Source: §VII · p. 11, §VIII · p. 11

VIII. CONCLUSION

  • MEDEM achieves geometric-mean gains of up to 4.84× in EDP and 1.59× in throughput compared to the state-of-the-art.The gains are reported using representative DL workloads and varying resource budgets.Source: §VIII · p. 11

Related papers

Loading 2609.37399v1…