PaperlayerView exact revision on arXiv

Source-linked AI summary

ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning

Sheng-Chun Kao, Geonhwa Jeong, Tushar Krishna

arXiv:2009.02010v1cs.ARcs.LGeess.SP

TL;DR

Assigning compute and memory resources to DNN accelerators under performance and platform constraints remains relatively unexplored, and the search space is too large for exhaustive tuning. ConfuciuX uses reinforcement learning with genetic-algorithm fine-tuning, and reports higher sample efficiency than alternate methods.

  • Problem

    Choosing compute and memory resources to optimize latency or energy under platform constraints is relatively unexplored, and the design space can make exhaustive search infeasible.

    Source: §I · p. 2

  • Method

    ConfuciuX uses reinforcement learning for global hardware-resource search and a genetic algorithm for fine-tuning.

    Source: §I · p. 2

  • Results

    ConfuciuX demonstrates the highest sample-efficiency compared with other optimization and RL methods; co-search improves results by 4% to 69%, and genetic-algorithm fine-tuning improves them by another 7% to 93%.

    Source: H. Summary · p. 12, §VI · p. 13

  • Takeaways & Limitations

    The paper concludes that machine learning shows promise within the DNN accelerator design workflow.

    Source: §VI · p. 13

Abstract

from arXiv · show

DNN accelerators provide efficiency by leveraging reuse of activations/weights/outputs during the DNN computations to reduce data movement from DRAM to the chip. The reuse is captured by the accelerator's dataflow. While there has been significant prior work in exploring and comparing various dataflows, the strategy for assigning on-chip hardware resources (i.e., compute and memory) given a dataflow that can optimize for performance/energy while meeting platform constraints of area/power for DNN(s) of interest is still relatively unexplored. The design-space of choices for balancing compute and memory explodes combinatorially, as we show in this work (e.g., as large as O(10^(72)) choices for running \mobilenet), making it infeasible to do manual-tuning via exhaustive searches. It is also difficult to come up with a specific heuristic given that different DNNs and layer types exhibit different amounts of reuse. In this paper, we propose an autonomous strategy called ConfuciuX to find optimized HW resource assignments for a given model and dataflow style. ConfuciuX leverages a reinforcement learning method, REINFORCE, to guide the search process, leveraging a detailed HW performance cost model within the training loop to estimate rewards. We also augment the RL approach with a genetic algorithm for further fine-tuning. ConfuciuX demonstrates the highest sample-efficiency for training compared to other techniques such as Bayesian optimization, genetic algorithm, simulated annealing, and other RL methods. It converges to the optimized hardware configuration 4.7 to 24 times faster than alternate techniques.

I. INTRODUCTION

DNN accelerators use dataflow to exploit reuse, but choosing compute and memory resources for a given dataflow and platform constraint remains an open design problem. ConfuciuX searches this space with reinforcement learning and genetic-algorithm tuning, outperforming alternate optimization methods in solution quality and search time.

I. INTRODUCTION

  • The paper situates resource assignment alongside dataflow, which determines computation order, parallelization, and tiling to capture reuse.Prior work explored dataflow strategies, while the choice of resources for a given dataflow is the focus here.Source: §I · pp. 1–2
  • Hardware resource assignment remains an open problem despite its substantial effect on latency and energy for a fixed dataflow.Prior hardware design-space exploration works cited in the passage rely almost exclusively on exhaustive search.Source: §I · p. 2
  • The resource-assignment objective is to minimize end-to-end latency or energy while satisfying platform constraints such as maximum power or chip area.For layer-pipelined deployment, PE and buffer resources must be divided across the DNN's layers.Source: §I · p. 2
  • ConfuciuX combines reinforcement learning for global search with a genetic algorithm for fine-grained tuning of hardware assignments.It takes the model, platform constraint, deployment scenario, and latency or energy objective as inputs, and selects PE and buffer assignments.Source: §I · p. 2
  • 4.7 to 24 × faster search than alternate schemes, while consistently outperforming them in latency or energy solution quality.Evaluations span popular DNN models, three dataflow styles, and cloud and IoT platform constraints.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

The paper models an accelerator through its processing elements, local buffers, dataflow, and design-point. These elements determine how computation and data movement are organized.

A. DNN Accelerator Architecture

  • Each processing element has a MAC and local L1 buffers for weights, activations, and partial sums.A shared L2 buffer prefetches data from DRAM, while the network-on-chip distributes operands and collects outputs.Source: A. DNN Accelerator Architecture · pp. 2–3
  • Dataflow orchestrates data movement from global SRAM to PE buffers through computation order, parallelization across PEs, and tiling.The paper follows the behavior of NVDLA-, Eyeriss-, and ShiDianNao-style dataflows while allowing flexibility in free dimensions.Source: A. DNN Accelerator Architecture · pp. 2–3
  • A design-point is one combination of PE count and buffer size, treated as independently tunable parameters.That combination also determines components such as the minimum L2 buffer size and the NoC bandwidth required for stall-free operation.Source: A. DNN Accelerator Architecture · p. 3

B. DNN Accelerator Performance and Cost Modeling

Accelerator performance and cost depend on the DNN layer, mapping, and hardware resources. Resource needs also differ by deployment scenario, making the constrained design space large and difficult to search exhaustively.

B. DNN Accelerator Performance and Cost Modeling

  • Different layer types expose different amounts of data reuse, while mapping and tile sizes are bounded by buffer capacity.A mapping consists of dataflow and tile sizes, and its total tile count depends on the model and dataflow.Source: B. DNN Accelerator Performance and… · p. 3
  • PE count determines peak throughput, while per-PE buffer size determines how much reuse each PE can exploit within its computation tile.MAESTRO and Timeloop model layer shape, mapping, and resources to estimate runtime, utilization, area, and power.Source: B. DNN Accelerator Performance and… · p. 3

C. DNN Model Deployment on Accelerators

  • Layer Sequential deployment runs layers one by one on the same accelerator, so one design-point may over-provision some layers and under-utilize resources.The design-point is chosen at design-time, often using a heuristic intended to perform well across most layers.Source: C. DNN Model Deployment on Accelerators · pp. 3–4
  • Layer Pipelined deployment runs the whole model across the accelerator, requiring PE and buffer resources to be partitioned among layers.The assumed accelerator can partition these resources at design-time or compile-time.Source: C. DNN Model Deployment on Accelerators · p. 4

D. Challenge: Design-Space for HW Resource Assignment

  • For MobileNet-V2 layers under NVDLA-style dataflow, design-points sweep 1–64 PEs and filter counts of 1–800, corresponding to L1 buffers of 6B–2800B.Each point has a distinct latency, energy, and area consequence.Source: D. Challenge: Design-Space for HW… · p. 4, Figure 4 · p. 4
  • With 128 PEs and 128 buffers, MobileNet-V2's design space is O(1072), making exhaustive exploration infeasible.The challenge applies to choosing a design-point across layers in LS or a PE-and-buffer combination for each layer in LP.Source: D. Challenge: Design-Space for HW… · p. 4

E. Optimization Methods for Design-Space Exploration

Design-space exploration methods trade search coverage against evaluation effort, while reinforcement learning can use state information for sequential resource-allocation decisions. The paper frames constrained PE and buffer assignment as a Markov decision process.

E. Optimization Methods for Design-Space Exploration

  • Exhaustive search finds a global optimum but is nearly impossible for vast spaces; grid search makes it more approachable through coarse sampling.Random search samples design-points and retains the best solution found.Source: E. Optimization Methods for… · p. 4
  • Simulated annealing sometimes accepts worse points, with a temperature parameter controlling how much randomness it adds to search.Genetic algorithms evolve encoded design-points through selection, mutation, and crossover across generations.Source: E. Optimization Methods for… · pp. 4–5
  • Bayesian optimization uses an objective surrogate and uncertainty to select the next values for evaluation, aiming to limit objective-function evaluations.The surrogate may use a Gaussian process, random forest, or tree Parzen estimator.Source: E. Optimization Methods for… · p. 5

F. Reinforcement Learning for Design-Space Exploration

  • PE and buffer assignment across DNN layers can be formulated as an MDP: sequential decisions minimize platform latency or energy within an area or power budget.The paper motivates RL as a way to increase search sample efficiency compared with baseline methods that do not use current-state information.Source: F. Reinforcement Learning for… · p. 5

III. CONFUCIUX

ConfuciuX uses REINFORCE for a coarse global search of per-layer PE and buffer assignments, then applies GA for fine-tuning. Its agent makes sequential layer-wise decisions under platform constraints using observations that encode layer information and previous actions.

III. CONFUCIUX

  • ConfuciuX uses REINFORCE for global search and a genetic algorithm for fine-tuning PE and buffer assignments evaluated by MAESTRO.The inputs include the target model, deployment scenario, objective, and platform constraints.Source: §III · p. 5

A. RL Agent

  • The agent treats each DNN layer as a time step, choosing PEs and buffers, and ends an epoch after 2N actions or a constraint failure.Rewards and the previous layer’s actions inform the policy, which is updated at the end of each epoch.Source: A. RL Agent · p. 5
  • REINFORCE uses an actor without a critic because the authors observed that critic-based algorithms inaccurately estimated values in the discrete, irregular design space and disturbed policy learning.The paper reports that REINFORCE worked best among the RL algorithms they tested.Source: A. RL Agent · p. 5
  • The RNN policy uses a 128-unit LSTM layer so recurrent connections can capture how earlier resource choices consume the area or power budget.The authors also evaluated MLP-based policy networks.Source: A. RL Agent · p. 5

B. Observation (State)

  • The 10-dimensional observation encodes layer shape and type, the layer index, and the previous layer’s PE and buffer actions.For MLP/GEMM layers, three dimensions describe the matrix dimensions (M, N, K); observations are normalized to [-1, 1].Source: B. Observation (State) · pp. 5–6

C. Action Space

ConfuciuX searches PE and buffer assignments while optimizing whole-model latency or energy under platform area or power constraints. Layer-wise rewards guide training, while constraint violations receive penalties.

C. Action Space

  • The agent chooses PE and buffer counts from 12 coarse-grained levels, then uses fine-grained GA search after RL converges.PE levels reflect marginal performance returns; buffer levels depend on dataflow, with NVDLA buffer sizing illustrated from a 3×3 weight example.Source: C. Action Space · p. 6

D. Platform Constraint and Objective

  • The design objective is minimum whole-model latency or energy while satisfying platform power or chip-area limits.The relationship between resource counts and cost is non-monotonic: more PEs can raise data-fetch latency, while added PEs and buffers can raise power but reduce energy through shorter execution.Source: D. Platform Constraint and Objective · p. 6

E. Reward Function

  • The RNN jointly allocates the remaining platform budget across layers, and the environment penalizes violations as the agent makes decisions.Because final performance arrives only at episode end, layer-wise feedback shapes rewards; violations receive the negative accumulated episode reward rather than a fixed threshold penalty.Source: D. Platform Constraint and Objective · p. 6, E. Reward Function · p. 6
  • The reward uses the lowest layer-wise performance observed across training to stabilize learning and keep feasible-assignment rewards positive.The authors state that this term emphasizes relative improvement when performance values, such as cycle counts, are large.Source: E. Reward Function · p. 6

F. Interactive Environment (Env)

The environment evaluates candidate accelerator assignments with MAESTRO and tracks platform-constraint consumption. After RL’s coarse search, a locally constrained genetic algorithm refines the assignment.

F. Interactive Environment (Env)

  • MAESTRO evaluates each design point from the DNN, dataflow, and hardware configuration, estimating latency, energy, runtime, power, and area.Its model includes PEs, L1 buffers, a shared L2 buffer, and an NoC, and supports layer-wise and model-wise evaluation.Source: F. Interactive Environment (Env) · p. 7

G. Local fine-tuning using GA

  • The two-stage search uses RL for coarse global exploration and GA for fine local search, avoiding the much larger action space of a finer-grained RL search.The authors describe GA as suitable for fine-tuning when initialized with the RL solution, while standalone GA can fail to learn constraints and optimize simultaneously.Source: G. Local fine-tuning using GA · p. 7
  • GA starts from the RL solution and makes conservative local mutations and crossovers to preserve useful layer-resource relationships and reduce invalid assignments.Mutation changes a gene by a local step; crossover swaps PE-buffer pairs between two layers within a parent genome.Source: G. Local fine-tuning using GA · p. 7

IV. EVALUATIONS

The evaluation methodology covers CNN and GEMM-based DNNs, multiple accelerator dataflows, and platform settings ranging from constrained devices to cloud and unconstrained platforms. It compares ConfuciuX with search and RL baselines.

A. Methodology

  • The evaluation includes MobileNet-V2, MnasNet, and ResNet-50 CNNs, plus GNMT, Transformer, and NCF GEMM-based models.Source: A. Methodology · p. 7
  • Experiments consider cloud, IoT, extremely small IoT, and unconstrained platforms, with NVDLA-, Eyeriss-, and ShiDianNao-style dataflows.The action space uses 12 levels each for PEs and buffers.Source: A. Methodology · pp. 7–8
  • The comparisons include simulated annealing, Bayesian optimization, GA, and several discrete and continuous RL algorithms; ConfuciuX comparisons use its global-search stage for fairness.The RL baselines include A2C, ACTKR, PPO2, DDPG, SAC, and TD3.Source: A. Methodology · p. 8

B. Per-layer study for LS deployment

Con’X finds layer-specific PE/buffer configurations that are as good as or better than the baselines and two heuristics. The best configuration varies by layer, so one assignment does not suit the whole network.

B. Per-layer study for LS deployment

  • Each layer has a distinct latency/energy landscape, and no single PE/buffer action pair suits all layers.The figure maps PE and buffer levels to latency/energy, with purple indicating lower values and red higher values.Source: B. Per-layer study for LS deployment · pp. 8–9, Figure 5 · p. 9
  • Overprovisioning can flatten latency gains, while layer type affects whether more PEs or larger buffers help.For Layer-12, latency is unchanged above the ninth PE level and third buffer level; for Layer-23 DWCONV, extra PEs can remain idle and fewer buffers increase fetching energy.Source: B. Per-layer study for LS deployment · p. 8
  • Con’X consistently finds each layer’s optimal action pair, with solutions as good as or better than baseline methods and two common heuristics.For layer-sequential deployment, designers can use the per-layer configurations to select a solution that performs well across most layers.Source: B. Per-layer study for LS deployment · p. 9, Figure 5 · p. 9

C. LP Deployment

For layer-pipelined deployment, ConfuciuX(global) searches for model-wide assignments under platform constraints, and the paper also extends the search to per-layer dataflow choices. It matches or outperforms the compared methods across evaluated settings, with reported gains in latency and energy and faster convergence than alternate RL algorithms.

C. LP Deployment

  • Across the evaluated DNNs, dataflows, and cloud/IoT constraints, ConfuciuX(global) reaches the same or better objective value than PPO2 and genetic algorithm.Genetic algorithm can find good values under loose cloud constraints but fails in some tight IoT and IoTx cases.Source: C. LP Deployment · p. 9
  • 86% lower latency and 70% lower energy on average across baselines: ConfuciuX(global) finds the most optimized MobileNet-V2 design points across four NVDLA-style platform constraints.Random, simulated annealing, and genetic algorithm fail to find feasible solutions under tight IoT constraints; Bayesian optimization also fails under IoTx.Source: C. LP Deployment · p. 10
  • ConfuciuX(global) converges to optimized values 4.7 to 24 times faster than alternate RL algorithms.The authors report that REINFORCE-based ConfuciuX finds better solutions than actor-critic algorithms; the tested critic network’s best RMSE is 5.3e+4 cycles.Source: C. LP Deployment · p. 10

D. Dataflow-HW co-automation

E. Benefit of Two-stage Optimization

ConfuciuX’s second-stage local genetic-algorithm tuning further improves assignments found by the first-stage global search. The paper also reports compile-time FPGA results, including gains from the two-stage MIX approach over a uniform NVDLA-style baseline.

E. Benefit of Two-stage Optimization

  • Local genetic-algorithm fine-tuning improves first-stage results by another 7% to 93%, reaching 66% to 99% improvement over initial valid values.The first-stage global search improves values by 56% to 99% over initial valid values while maintaining constraints.Source: E. Benefit of Two-stage Optimization · p. 11, Figure 9 · p. 11
  • ConfuciuX assigns resources heterogeneously across layers under area constraints, with fewer PEs and buffers allocated to MobileNet-V2 DWCONV layers.For ResNet-50, it assigns more buffers to layers with larger input/output channel sizes, including layers 37, 43, and 47.Source: E. Benefit of Two-stage Optimization · p. 11, Figure 10 · p. 12

F. LP deployment at compile time

G. Policy Network Exploration

ConfuciuX combines reinforcement learning for global hardware-resource search with genetic-algorithm fine-tuning. The paper reports policy-network design choices, search results, and how its fine-grained resource-assignment focus differs from prior work.

G. Policy Network Exploration

  • RNN-based policy networks converge to better results than MLP-based networks, and L=12 is the reported sweet spot for action-level granularity.The authors suggest RNNs may benefit from remembering constraint consumption in previous layers; decreasing L reduces complexity but worsens granularity.Source: G. Policy Network Exploration · p. 12

H. Summary

  • Joint REINFORCE search of hardware configuration and dataflow improves results by 4% to 69%, while specialized local GA fine-tuning adds 7% to 93%.The summary also reports that REINFORCE explores the design space more effectively and efficiently than baseline optimization methods and can converge faster with similar or better results than alternate RL methods.Source: H. Summary · p. 12

V. RELATED WORKS

  • Prior work studies DNN mapping and dataflow optimization, whereas ConfuciuX targets fine-grained compute-and-memory assignment within an accelerator.The related-work discussion notes that prior exploration often uses grid or exhaustive search with manually developed pruning, motivating automated search for LP and LS scenarios.Source: §V · pp. 12–13

VI. CONCLUSION

  • ConfuciuX addresses the relatively unexplored problem of assigning hardware resources for a DNN, dataflow style, and platform constraints, using RL search augmented by GA fine-tuning.The conclusion identifies future opportunities in learning dataflow and hardware behavior and in DNN-dataflow-hardware co-design.Source: §VI · p. 13

Related papers

Loading 2009.02010v1…