Source-linked AI summary
qGAMMA: A Framework for Optimal DNN Mapping on Multi-Precision Accelerators
Leandro Fiorin, Luigi Altamura, Cristina Silvano
TL;DR
Mixed-precision DNN mapping on accelerators with multiple precision pipelines presents system-level optimization complexity. qGAMMA searches mapping and accelerator configurations with a genetic algorithm and analytical performance estimation; experiments evaluate its effectiveness in estimating and optimizing performance.
Problem
Optimizing mixed-precision DNN execution on accelerators with multiple precision pipelines introduces complexities that have been largely unaddressed in system-level optimization.
Source: §1 · p. 1Method
qGAMMA uses a genetic-algorithm search and analytical performance estimation to explore mixed-precision DNN mappings and accelerator configurations.
Source: §4 · pp. 5–6Results
Experimental evaluation across DNN inference workloads demonstrates qGAMMA's effectiveness in estimating and optimizing performance.
Source: §1 · p. 2, §6 · p. 13Takeaways & Limitations
qGAMMA offers designers insights into trade-offs among mapping strategies, parallelization, and precision for mixed-precision applications.
Source: §1 · p. 1A direct comparison with existing mapping and exploration tools is not feasible because none support both mixed-precision DNN modeling and multi-precision accelerators.
Source: §5.2 · p. 9
Abstract
from arXiv · showhide
This work describes qGAMMA, a flexible framework for exploring and optimizing the mapping of mixed-precision Deep Neural Network (DNN) models onto general-purpose spatial accelerators that support a range of fixed- and floating-point precisions. qGAMMA navigates the design space using a domain-specific genetic algorithm and an analytical performance model based on hardware synthesis results. The framework identifies the optimal mapping and accelerator configuration that minimizes target metrics. Extensive evaluations across DNN workloads show that qGAMMA offers valuable insights for designing mixed-precision applications, enabling better trade-offs between performance, energy efficiency, and resource utilization.
1 Introduction
qGAMMA addresses the complexity of optimizing mixed-precision DNN execution on accelerators with multiple precision pipelines. It explores mappings and accelerator configurations to help designers trade off mapping strategies, parallelization, and precision.
1 Introduction
- Mixed-precision DNN execution on accelerators with multiple precision pipelines creates optimization complexity that system-level approaches have largely left unaddressed.The model may use a different precision for each layer, while the accelerator provides hardware modules for different precisions.Source: §1 · pp. 1–2
- qGAMMA was developed specifically to address mixed-precision mapping optimization, which prior mapping frameworks did not explicitly support.Existing frameworks generally assume a single, unspecified computation precision.Source: §1 · p. 2
- The work proposes a configurable accelerator architectural template for exploring multi-precision designs.The introduction describes it as modeling state-of-the-art multi-precision architectures.Source: §1 · p. 2
- qGAMMA includes an analytical performance-estimation tool for multi-precision accelerators based on the proposed architectural template.Source: §1 · p. 2
- Its heuristic search prunes inefficient mappings to reduce the optimization search space and improve exploration efficiency.Source: §1 · p. 2
- The experimental campaign evaluates qGAMMA's effectiveness in estimating and optimizing performance.Source: §1 · p. 2
2 Background and related work
CNN dataflow choices shape reuse, resource use, energy, and bandwidth demands. Existing mapping tools explore these choices, while qGAMMA explicitly includes mixed-precision mapping for multi-precision spatial architectures.
2 Background and related work
- Dataflow comprises computation order, parallelism and unrolling, and tiling, which shape reuse, data movement, resource use, energy efficiency, memory use, and bandwidth requirements.The 2D CNN3 algorithm has six nested MAC loops, or seven with batching, that can be ordered, tiled, and scheduled.Source: §2 · p. 3
- MAESTRO and Timeloop analyze dataflow and accelerator configurations to estimate activity, memory use, communication, reuse, latency, and energy.Latency is determined by the slowest component, and energy is estimated from activity counts and component-specific energy costs.Source: §2 · p. 3
- GAMMA uses a domain-specific genetic algorithm to search per-layer CNN dataflows for mappings that minimize latency and/or energy.It encodes dataflow characteristics as gene pairs for CNN dimensions and spatial parallelization, then evaluates selected dataflows with tools such as MAESTRO or Timeloop.Source: §2 · pp. 3–4
- LOMA, CoSA, and SALSA explore DNN mapping and scheduling through brute force, mixed-integer linear programming, or simulated annealing.These approaches transform loops and loop orders to search spatial-accelerator mappings.Source: §2 · p. 4
- qGAMMA explicitly incorporates mixed-precision into dataflow and mapping exploration, addressing a capability absent from prior frameworks that generally assume one unspecified precision.Among prior tools, only Timeloop supports analytical estimation for different precision values within single-precision computation pipelines.Source: §2 · p. 4
3 Accelerator architecture template
The target architecture uses hierarchically clustered processing elements, scratchpad buffers, main memory, and a network-on-chip. Each PE models configurable multi-precision computation with separate integer and floating-point pipelines.
3 Accelerator architecture template
- The template connects hierarchically clustered PEs, local L1 and shared L2 scratchpad buffers, and main memory through a NoC, with double buffering for data transfers.The architecture template is identified in Fig. 2.Source: §3 · p. 4
- Each PE has a configurable functional unit with separate integer fixed-point and floating-point pipelines, enabling independent optimization for energy efficiency at added area cost.The unit supports FP32, FP16, FP8, FP4, INT32, INT16, INT8, INT4, and INT2 operations.Source: §3 · pp. 4–5
- The functional unit performs one MAC at FP32 or INT32, while lower-precision operations partition its 32-bit lane into multiple operands.For precision Q, the lane is partitioned into P = 32/Q operands; integer and floating-point multipliers use different pipeline-stage counts.Source: §3 · p. 5
4 qGAMMA
qGAMMA extends GAMMA to explore mixed-precision DNN mappings and accelerator configurations using a genetic algorithm. It adds precision-aware dataflow and encoding, search heuristics, and analytical performance and energy estimation.
4 qGAMMA
- qGAMMA explores dataflows and accelerator configurations to minimize a chosen energy, latency, or energy-delay-product metric.Inputs include a mixed-precision DNN, an objective, and accelerator-resource constraints such as PE count, memory sizes, NoC bandwidth, and supported precisions.Source: §4 · p. 5
4.1 Enhanced dataflow model
- The enhanced dataflow model uses each layer’s weight and activation precision to generate the corresponding architectural and dataflow models.This extends the dataflow representation to account explicitly for operand precision in each DNN layer.Source: §4 · p. 5, §4.1 · p. 6
4.2 Extended genetic encoding
- The extended encoding adds a third genome level to model low-precision pipelines, with separate mappers for inter-cluster, within-cluster, and pipeline dataflows.The three-level genome is used for low-precision configurations; FP32 and INT32 retain the original two-level genome, and the inner cluster size follows the lowest precision used.Source: Figure 4 · p. 6, §4.2 · p. 7
4.3 Heuristic optimization
- Three heuristics prune the expanded search space by restricting cluster sizes and parallel dimensions and favoring power-of-two K and C values.The search space for a three-level genome is reported as O(1033) for ResNet18; cluster sizes are generally restricted to powers of two, with an exception for the inner cluster.Source: §4.3 · pp. 7–8
4.4 Performance estimation tool
- The MAESTRO-based estimator models layer precision and adds activity-based energy accounting for low-precision MACs, packed buffer accesses, and NoC transfers.The dataflow description is modified to include the desired computation precision for each layer.Source: §4.4 · p. 8
5 Experimental evaluation
The evaluation tests qGAMMA on three DNNs using a modeled 16-PE accelerator, comparing optimization settings and precision choices. Results show latency, energy, and search-time trade-offs, while direct comparisons with other mapping tools are unavailable.
Hardware accelerator model
- The modeled accelerator has 16 PEs, up to 512 bytes of L1 scratchpad per PE, and up to 512 Kbytes of global L2 scratchpad.Component values are scaled to 22-nm technology; main-memory energy is excluded because valid solutions avoid main-memory accesses for temporary results.Source: Hardware accelerator model · p. 8
DNN models
- Experiments cover MobileNetV1, ResNetV1, and mixed-precision ResNet18, whose layers combine INT8 and INT4 weights and activations.The ResNet18 variant is reported as fine-tuned to accuracy within 1.5% of FP32 and 1.6% of uniform INT8.Source: DNN models · pp. 8–9
Optimization setup
- Optimization targets latency, activity-based energy, or EDP, using per-layer clustering or a single clustering selected across layers.The shared-clustering phase uses the most frequent configuration from per-layer optimization; search stops if the best solution is unchanged for 150 generations.Source: Optimization setup · p. 9
5.2 Experimental Results
- Direct comparison with state-of-the-art mapping tools is infeasible because none support both mixed-precision DNNs and multi-precision accelerators.The evaluation instead compares with the original GAMMA search algorithm and modeling features where suitable.Source: §5.2 · p. 9
Heuristics validation - ResNet18
- Heuristics reduce search execution time by about 31% with early stopping and 51% without it, while latency falls only 4%.Early stopping is described as essential for ensuring search convergence, despite slightly increasing search time.Source: Heuristics validation - ResNet18 · p. 9, Figure 5 · p. 10
- FP16 improves ResNetV1 EDP by 6.96× and FP8 by 21.44× versus FP32 under per-layer optimization.Latency is approximately 3.0× lower with FP16 and 4.29× lower with FP8 versus FP32.Source: Heuristics validation - ResNet18 · pp. 10–11, Figure 6 · p. 11
- INT8 and INT4 reduce MobileNetV1 energy by 1.96× and 3.03×, respectively, versus INT16.Lower precision does not always reduce latency because the optimization targets energy and may select suboptimal PE parallelization.Source: Heuristics validation - ResNet18 · pp. 11–12, Figure 7 · p. 12
- 1.20× latency speedup and 17% lower energy are reported for mixed-precision versus uniform INT8 ResNet18.The comparison uses minimum-latency optimization and one clustering scheme across all layers.Source: Figure 8 · p. 13, Heuristics validation - ResNet18 · p. 13
6 Conclusions
The conclusion presents qGAMMA as a framework for optimizing mixed-precision DNN mapping on multi-precision accelerators. Evaluation across diverse inference workloads reports useful design insights and faster, more effective design-space exploration.
6 Conclusions
- qGAMMA combines a framework for mixed-precision DNN mapping with an accelerator architecture template for multi-precision designs.The conclusion describes evaluation across diverse DNN inference workloads and highlights insights for optimizing mixed-precision applications.Source: §6 · p. 13
- The framework integrates rapid DNN accuracy estimation across models and precision levels to support accelerator optimization.The conclusion characterizes this as enabling faster and more effective design-space exploration.Source: §6 · p. 13
- The reported evaluation presents qGAMMA as a source of insights for optimizing mixed-precision applications across diverse DNN inference workloads.Source: §6 · p. 13