PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey

Hamza Kheddar

arXiv:2610.28699v1cs.LGcs.AIcs.ARcs.PFcs.ROcs.SYeess.SY

TL;DR

Fixed control and optimization struggle with dynamic workloads, while RL offers adaptive decision-making but faces hardware deployment constraints. This review synthesizes FPGA and IC RL implementations and identifies recurring architectural patterns alongside persistent limitations.

  • Problem

    Dynamic workloads challenge fixed control approaches, while translating RL to hardware must meet strict latency, power, determinism, and resource constraints.

    Source: §1 · p. 1

  • Method

    The review systematically synthesizes RL implementations on FPGA and IC platforms across algorithm families, architectures, applications, and hardware realization strategies.

    Source: §7 · p. 33

  • Results

    Effective designs commonly use fine-grained parallelism, pipelined datapaths, quantized arithmetic, and tightly coupled on-chip memory, with FPGA flexibility contrasting with IC determinism and low-power priorities.

    Source: §7 · p. 33

  • Takeaways & Limitations

    Reported application-level gains span latency, energy efficiency, throughput, and adaptability across networking, security, robotics, autonomous systems, sensing, and embedded optimization.

    Source: §7 · p. 33

    Broader adoption is constrained by scalability, portability, reproducibility, numerical-precision sensitivity, and limited robustness evidence beyond synthetic or small-scale environments.

    Source: §6.1 · p. 29, §6.1.1 · p. 29, §7 · p. 33

Abstract

from arXiv · show

Reinforcement learning (RL) enables adaptive and autonomous decision-making in embedded, edge, and heterogeneous computing systems. Field-programmable gate arrays (FPGAs) and integrated circuits (ICs) offer low latency, energy efficiency, and architectural flexibility for RL deployment, while introducing challenges in resource constraints, implementation complexity, and hardware-algorithm co-design. This review surveys more than 100 studies on RL implementations across FPGA and IC platforms, emphasizing architectures, implementation strategies, performance evaluation, and applications. The literature is organized by RL family, including model-based Markov decision processes, classical value-based methods, deep RL, actor-critic, policy-gradient, and multi-agent approaches. We further classify studies by abstraction level, hardware realization strategy, and application domain, examining how RL algorithms map onto reconfigurable and fixed hardware. Unlike surveys restricted to particular algorithms or platforms, this work provides a unified analysis of FPGA and IC implementations, covering acceleration, reconfigurable architectures, energy-aware control, and system-level optimization. Comparative assessments consider resource utilization, latency, throughput, power consumption, and reported performance gains. Finally, we identify recurring architectural patterns, practical limitations, open challenges, and research directions for efficient RL-driven hardware systems. The survey offers a consolidated reference for researchers developing and evaluating RL solutions on specialized hardware.

1. Introduction

Dynamic, heterogeneous workloads expose the limits of fixed control and optimization, while RL offers adaptive sequential decision-making under uncertainty. FPGA and IC hardware can reshape RL deployment but impose constraints in design flexibility and implementation complexity.

1. Introduction

  • Dynamic workloads and runtime disturbances challenge static controllers and heuristic optimizers built around fixed operating assumptions.The cited examples include networking, edge computing, cybersecurity, robotics, language processing, sensing, and embedded environments.Source: §1 · p. 1
  • RL adapts through environment interaction and reward-based policy updates, making it relevant to systems balancing performance, energy, latency, and reliability.The passages also note that hardware deployment must contend with iterative updates, stochastic exploration, memory-intensive computations, and strict latency, power, and determinism requirements.Source: §1 · p. 1
  • FPGAs offer fine-grained parallelism, customizable datapaths, and reconfigurability, while ICs support deterministic execution, compactness, and low-power operation.Both alternatives bring stricter design constraints and greater development complexity than software-based solutions.Source: §1 · p. 1

Acronyms and Abbreviations

The section defines abbreviations used in the review and outlines its scope, literature-selection approach, and organization. The review compares FPGA and IC RL implementations across algorithm families, hardware flows, metrics, applications, and challenges.

Acronyms and Abbreviations

  • Abbreviations include algorithm families and platforms such as A3C, PPO, PG, FPGA, ASIC, and SoC, alongside hardware and evaluation terms such as RTL, LUTRAM, and FIPS.The supplied abbreviation entries also define RL, MDP, MARL, and numerous other terms used throughout the review.Source: Acronyms and Abbreviations · p. 2

1.1. Related work and our contribution

  • The review addresses gaps in prior surveys by analyzing more than 120 FPGA/IC-related RL works within a unified, hardware-centric synthesis.Its stated coverage spans major RL families, abstraction levels and toolflows, evaluation metrics, application areas, and cross-cutting limitations.Source: §1.1 · pp. 2–3

1.2. Review methodology

  • The search uses Scopus and a structured Boolean query to identify English-language journal and conference studies from 2015–2026 on RL and FPGA/IC hardware.Screening excludes redundant or weakly relevant work and studies lacking meaningful evaluation or sufficient implementation detail.Source: §1.2 · pp. 3–4

1.3. Paper structure

  • The review proceeds from hardware background and RL taxonomy to research areas, application domains, limitations, future directions, and conclusions.Figure 1 is identified as the roadmap of the review.Source: §1.3 · p. 4, Figure 1 · p. 4

2. Background

The background describes FPGA and IC hardware resources and design flows, then surveys platforms and simulators for hardware-aware RL evaluation. It highlights the shared stages and distinct implementation paths of reconfigurable and fixed hardware.

2.1. FPGA and IC

  • FPGA fabrics combine CLBs, BRAM, DSP slices, routing, clock management, and configuration logic, while ICs integrate fixed logic, processors, accelerators, memories, and management infrastructure.RL can exploit FPGA fabric reconfiguration; IC schemes tune operating points, power states, scheduling, and resource allocation.Source: §2.1 · pp. 4–5, Figure 2 · p. 5
  • FPGA and IC flows share high-level modeling, domain-specific compilation, HLS, and RTL stages, but diverge in physical implementation and deployment flexibility.Domain-specific compilation can optimize operators, memory tiling, scheduling, quantization, and hardware mapping before generating hardware descriptions.Source: §2.1 · p. 5, Table 2 · p. 6

2.2. Platforms and Simulators

  • Research platforms support hardware-aware evaluation, including FPGA placement optimization, processing-in-memory inference simulation, FPGA-accelerated circuit simulation, and Q-learning IP generation.The Q-learning toolchain's support for Intel FPGAs and ASIC flows is identified as future work.Source: §2.2 · p. 6

3. Taxonomy of RL-based FPGA and IC platform

The taxonomy organizes hardware RL studies by learning paradigm and application, while the MDP section frames hardware-aware decisions through states, actions, transitions, rewards, and discounting. The review contrasts platform strengths and implementation trade-offs.

3. Taxonomy of RL-based FPGA and IC platform

  • The taxonomy groups FPGA RL platforms by model-based, value-based, policy-gradient, actor-critic, and multi-agent paradigms, linking algorithms to applications, hardware advantages, and limitations.The review uses RL broadly to include classical and deep methods under the shared MDP formalism.Source: §3 · p. 6, Table 3 · p. 7
  • FPGA and IC implementations are generally advantageous for latency-critical, energy-constrained deployment, whereas GPUs support flexible development and large-scale training.The trade-off includes greater design complexity, tighter resource and memory constraints, and lower post-deployment flexibility for FPGA/IC approaches.Source: §3 · pp. 6–7

3.1. MDP

  • An MDP represents hardware and workload configurations as states, reconfiguration choices as actions, and system dynamics and objectives through transitions and rewards.The passage distinguishes FPGA emphasis on reconfigurability and parallelism from IC emphasis on area efficiency, deterministic timing, and low energy.Source: §3.1 · p. 7
  • MDP-based FPGA work includes optimal interconnect-coverage planning and an FPGA coprocessor for generating sparse transition and reward models.The former reports near-theoretical-minimum configuration sets with 100% coverage but faces offline-computation and scalability limits; the latter uses HLS, pipelining, and fixed-point arithmetic.Source: §3.1 · pp. 7–8, Table 4 · p. 8

3.2. Q-learning

Q-learning estimates action values without an explicit transition model, and hardware implementations store and update those values in on-chip memory. FPGA designs map the update and action-selection work onto memories, arithmetic units, and control logic, with implementations spanning serial, parallel, heterogeneous, and precision-optimized designs.

3.2. Q-learning

  • Q-learning estimates the optimal action–value function without an explicit transition model, storing tabular or compactly parameterized values in on-chip memory for hardware updates.The survey contrasts this with MDP planning, which assumes access to the transition model.Source: §3.2 · p. 8
  • On FPGAs, fixed-point pipelines and parallel comparators implement Q-value updates, while multi-port memory and scheduled access accelerate table lookups.Exploration can use a lightweight control FSM for online adaptation of DVFS modes, reconfiguration, or scheduling.Source: §3.2 · p. 9
  • The generic agent separates a QL accelerator that updates the Q-matrix from a policy generator that reads updated values and selects actions.The policy generator may be implemented in software or hardware, while delayed state signals synchronize the update inputs.Source: §3.2 · p. 9, Figure 3 · p. 9
  • FPGA QL implementations span serial low-resource designs with four-cycle updates, parallel max-tree accelerators, and heterogeneous FARANE-Q PS–PL designs.A separate precision study found that reducing the datapath from 32 to 16 bits preserved optimal policy behavior and enabled a DSP-free Q-Updater.Source: §3.2 · p. 9

3.3. SARSA

SARSA updates action values using the next action actually selected by the behavior policy, rather than Q-learning’s greedy estimate. Its hardware implementations can simplify the update datapath, with latency independent of action-space width and tighter coupling between policy execution and learning.

3.3. SARSA

  • Unlike Q-learning’s off-policy greedy estimate, SARSA uses the action sampled from the behavior policy in its on-policy update [equation].The update bootstraps from the actual executed action rather than a hypothetical greedy one.Source: §3.3 · p. 10
  • Removing the max-action comparator tree reduces critical-path delays, LUT footprints, and DSP use, but requires the policy generator to feed its selected action back to the learner.The update then uses a single next-state action-value lookup rather than scanning all actions.Source: §3.3 · p. 10
  • QTAccel reuses QL pipeline stages and adds no LUT, DSP, or BRAM overhead, while keeping update latency stable across action-space widths.The FAQ accelerator integrates environment dynamics and Q-updates, forwarding executed actions directly to the learner.Source: §3.3 · p. 10
  • SARSA offers simpler datapaths, action-count-independent update latency, and tighter coupling between policy execution and learning in FPGA/IC Q-table systems.The survey presents these properties as making SARSA an attractive alternative to Q-learning in tightly integrated systems.Source: §3.3 · p. 10

3.4. Deep Q-learning

Deep Q-learning replaces a tabular action-value table with a neural network trained on sampled transitions, supporting high-dimensional states without an explicit transition model. FPGA implementations map neural-network training, replay, and target-network operations to hardware, trading Q-learning’s simpler datapath for greater generalization capability.

3.4. Deep Q-learning

  • DQN approximates action values with a neural network trained on replay-buffer minibatches, avoiding an explicit transition model while scaling to high-dimensional states.This replaces the single-step tabular update with minimization of a temporal-difference loss over sampled transitions.Source: §3.4 · p. 10
  • FPGA DQN maps network propagation to pipelined multipliers and quantized arithmetic, replay to BRAM or hybrid BRAM/DDR memory, and target updates to a secondary network.Compared with tabular Q-learning, DQN trades simpler hardware for generalization and expressiveness on complex sensory streams.Source: §3.4 · p. 10, Figure 4 · p. 11
  • Zhang et al.’s heterogeneous DQN pipeline outperformed CPU and GPU baselines in energy efficiency while supporting full training on FPGA.The design separates five training stages into independent hardware units and uses FPGA floating-point MACs and multi-mode matrix multipliers.Source: §3.4 · p. 10
  • STANN-RL combines reusable neural-network training blocks with RL-specific target synchronization and prioritized sampling, while an FPGA-based Adam optimizer improved convergence over fixed-point SGD [28].Other implementations include a 32-bit floating-point Cartpole training and inference pipeline and a dual-mode edge accelerator that reuses datapaths for inference and training.Source: §3.4 · p. 11

3.5. DDPG

DDPG extends off-policy, target-network learning to continuous actions using an actor and critic, avoiding discrete-action maximization but requiring coupled actor–critic training. The hardware literature presents DDPG as a balanced target for FPGA/IC acceleration and reports refinements for precision, parallelism, and resource use.

3.5. DDPG

  • DDPG uses an actor to produce continuous actions and a critic to evaluate them, replacing discrete-action maximization with state–action fusion in the critic.Joint training requires critic TD learning and actor gradients from ∇_aQ(s,a), while exploration noise is injected at the actor output.Source: §3.5 · p. 11
  • Replay memory can use BRAM or DDR, and actor–critic networks can share matrix–vector engines through time-multiplexing, supporting deterministic low-latency pipelines.The survey notes that smaller batch sizes than deep value-based algorithms ease memory-bandwidth pressure.Source: §3.5 · p. 11
  • Hardware refinements include fixed-point quantization-aware training without degraded policy quality, FPGA-parallel gradient computation, prioritized replay support, and shared-network, block-floating-point TD3-lite.DDPG was also applied to switched-reluctance motor torque-ripple minimization using continuous firing-angle offsets.Source: §3.5 · pp. 11–12
  • DDPG’s deterministic policy, moderate network size, lack of discrete maximization, and predictable updates yield balanced compute and memory characteristics suited to FPGA/IC acceleration.The survey describes suitability for both single-accelerator and heterogeneous-system designs.Source: §3.5 · p. 12

3.6. Actor-critic

Actor–critic (AC) couples policy selection with value evaluation, while hardware implementations commonly use asynchronous A3C pipelines. FPGA designs range from full training accelerators to distributed and CPU–FPGA systems, with reported throughput, training-time, and power-efficiency gains alongside synchronization and resource challenges.

AC learning pipeline

  • AC jointly executes action selection, value evaluation, and parameter updates, with a critic TD error driving both policy and value-function updates.The actor represents policy π(a|s; θ), and the critic estimates V(s; ϕ) or an advantage.Source: §3.6 · p. 12

Neuromorphic AC

  • The ESL-based spiking AC design preserves conventional spiking AC behavior while reducing logic-element and power costs, but evaluation covers only a small-scale navigation task.Source: §3.6 · p. 13

Asynchronous advantage actor–critic

  • A3C extends AC to asynchronous learners whose local pipelines accumulate gradients into shared global parameters.FPGA implementations can map this structure to replicated compute pipelines sharing global parameter memory.Source: §3.6 · p. 13

Soft actor–critic

  • SAC uses off-policy replay, a stochastic Gaussian actor, and dual critics; its entropy-regularized objective adds noise generation, log-probability, and temperature-adaptation hardware requirements.The SAC objective augments expected return with entropy maximization.Source: §3.6 · p. 13

Hardware mapping and challenges

  • AC FPGA architectures map inference, TD or advantage computation, and gradient updates to concurrent pipelines, but still face synchronization, memory-bandwidth, and actor–critic resource-balancing challenges.Modern FPGA fabrics are described as increasingly capable of continuous, low-latency AC learning in embedded and edge platforms.Source: §3.6 · pp. 13–14

Actor-critic accelerators

  • FA3C, distributed A3C, and A3C-HC report improved throughput, speedup, training time, or power efficiency over their stated GPU or CPU baselines.FA3C uses paired forward/backward engines; the distributed design avoids redundant actor–critic replication, while A3C-HC assigns control and simulation to CPUs and learning updates to the FPGA.Source: §3.6 · p. 13

3.7. Monte-Carlo

Monte-Carlo (MC) methods update from episode returns rather than critic values or TD errors, trading simpler control logic for trajectory storage and slower responsiveness. FPGA implementations include dynamic-tree MCTS acceleration using hybrid CPU–FPGA execution.

MC hardware mapping

  • MC removes critic and TD-error circuitry by accumulating empirical returns over an episode, but requires trajectory storage and return computation at episode boundaries.FPGA designs use fixed-point accumulators and FIFO or BRAM buffers, often offloading trajectory storage to external memory.Source: §3.7 · p. 14

MC trade-offs

  • MC trades frequent low-latency updates for simpler control logic and regular memory access, suiting short or deterministic episodes when unbiased returns are preferred to bootstrapped targets.Episodic updates reduce responsiveness.Source: §3.7 · p. 14

MC tree search

  • 6.8× throughput improvement over CPU baselines was reported for Meng et al.’s dynamic-tree MCTS accelerator, with algorithmic performance maintained across Atari benchmarks.Its butterfly interconnect and bank assignment support arbitrary tree shapes while minimizing runtime access conflicts.Source: §3.7 · p. 14

3.8. PPO

PPO uses clipped policy updates and truncated-rollout advantage estimates, which bring ratio, clipping, and minibatch-memory requirements to hardware. FPGA/SoC accelerators target these demands with GAE pipelines and shared systolic compute, reporting memory-footprint and throughput gains.

PPO method

  • PPO clips a surrogate policy objective using a probability ratio, changing actor updates without changing the critic and using advantages from truncated rollout segments.Source: §3.8 · p. 14

PPO hardware requirements

  • PPO hardware requires prior-policy log probabilities, ratio and clipping arithmetic, rollout buffers, and repeated minibatch passes, increasing memory pressure over online TD updates.Source: §3.8 · p. 14

PPO trade-offs

  • Truncated horizons and synchronized minibatch optimization give PPO moderate responsiveness and a regular update schedule, but couple actor, critic, and rollout memory; clip-and-ratio computation can become the bottleneck.PPO is described as suitable for real-time, non-stationary environments despite added control complexity.Source: §3.8 · p. 14

PPO accelerators

  • HEPPO reduces on-chip memory footprint by 4× through standardized, quantized GAE processing and reports up to 30% speed improvement in a single-SoC integration.Its design uses backward GAE recursion with dedicated arithmetic chains to minimize memory accesses.Source: §3.8 · pp. 14–15
  • Up to 27× throughput improvement over CPU/GPU baselines was reported for PPOAccel, which uses shared forward/backward systolic datapaths and memory-blocked layouts for multi-epoch updates.The survey attributes gains mainly to eliminating kernel-launch overheads and training/inference imbalance typical in PPO.Source: §3.8 · p. 15

3.9. Comprehensive Performance Metrics and Cross-Design Analysis

Cross-design comparison requires metrics that account for workload, hardware cost, energy, and latency because surveyed studies use different workloads and baselines. The review finds trade-offs across RL families and recommends fuller, consistently scoped reporting.

Metric framework

  • Energy, resource, and latency–energy metrics complement raw indicators because speed or throughput alone cannot show power use, hardware cost, or real-time suitability.The review identifies FIPS/W as informative for full-training accelerators and recommends interpreting latency alongside power and workload.Source: §3.9 · p. 15

Cross-design analysis

  • The reviewed FPGA/IC studies show substantial variation in workload granularity, hardware cost, energy efficiency, resource efficiency, and performance objectives.The review frames comparison around normalized criteria and application-level comparisons, alongside raw implementation characteristics.Source: §3.9 · p. 15, Table 5 · p. 16

Workload-dependent profiles

  • Tabular QL implementations use small resource fractions and tens of milliwatts, fitting compact state–action workloads rather than direct comparison with neural-network deep RL.Source: §3.9 · p. 15
  • DQN, PPO, DDPG, and AC accelerators support more complex environments but generally use more BRAM, DSP, and LUT resources, making speedup insufficient without power, latency, and utilization.Source: §3.9 · p. 15
  • AC/A3C accelerators face resource pressure from concurrent actor–critic networks, gradient engines, workers, and update logic, while replicated workers can constrain smaller-FPGA scalability.Source: §3.9 · p. 15

Reporting example

  • TD3lite reports full-iteration throughput, measured power, and resource use, enabling FIPS/W assessment; block floating-point quantization and structural simplification reduce resource footprint while maintaining high throughput.Source: §3.9 · p. 16

Cross-design analysis

  • Existing RL-based FPGA/IC studies report improvements in speed, throughput, power efficiency, and resource utilization, but heterogeneous workloads and baselines motivate normalized metrics.Source: §3.9 · p. 16

Reporting recommendations

  • The review recommends reporting decision and update latency, throughput, board or chip power, energy per operation, resource use, and FIPS/W for full-iteration acceleration, with power scope specified.Source: §3.9 · p. 16

4. Research Areas in RL-based FPGA and IC Platforms

This section organizes RL-based FPGA and IC research around accelerating RL execution and using RL to optimize hardware configuration, system behavior, and synthesis workflows. It covers parallel, quantized, specialized, and reconfigurable hardware, as well as HLS directive tuning and design-space exploration, while noting constraints including costly training, weak generalization, scalability, and deployment.

4.1.1. RL-based Acceleration

  • Replay, environment, and application-specific accelerators use parallelism and pipelining; a parallel K-ary sum-tree achieved 21–40× speedup in replay operations.Other designs offload neural inference or implement real-time control; common patterns include on-chip replay, fixed-point computation, and dataflow-oriented neural networks.Source: §4.1.1 · pp. 17–18

4.1.2. Multi-agent RL–based Approaches

  • Multi-agent accelerators target control latency, inter-agent communication, or neural-network computation, using low-latency updates, parallel AC pipelines, and sparse compressed training.Agent growth increases communication and memory demands, while algorithm-specific pipelines limit portability.Source: §4.1.2 · p. 18

4.1.3. Multi-FPGA/IC Platforms for Scalable RL Execution

  • A distributed system of 416 Zynq-7045 nodes evaluated 832 agents simultaneously, illustrating multi-device scaling for population-based RL.The 3D mesh supports synchronization, but communication, synchronization, workload partitioning, and portability remain challenges.Source: §4.1.3 · pp. 18–19

4.1.4. Energy-Efficient Acceleration of RL

  • Energy-efficient RL acceleration combines event-driven spiking designs, lightweight arithmetic, hierarchical learning, and precision-aware computation.Reported examples include up to 90% fewer adaptation iterations and 12% less computational workload; E2HRL achieved 1.1–11.4 GOP/s/W and 8.8×–11× improvements over GPU-based RL execution.Source: §4.1.4 · p. 19

4.2.1. RL-Based Reconfigurable Hardware Approaches

  • RL guides FPGA reconfiguration, physical cyber–physical control, and distributed-device workload management, adapting hardware to changing demands and conditions.Dynamic partial reconfiguration and hardware measurements support adaptation, but reconfiguration overhead, training instability, restricted state representations, and absent standardized benchmarks constrain deployment.Source: §4.2.1 · pp. 19–20

4.2.2. RL-based Energy and Power Management; 4.2.3. RL in Multi-FPGA/IC Infrastructures

  • RL-based power management adapts operating policies, while multi-FPGA infrastructure work optimizes routing and communication; both face reward-design and generalization challenges.Digital twins support training power controllers, whereas online feedback guides congestion-aware routing; large state/action spaces and runtime stability remain concerns.Source: §4.2.2 · p. 20, §4.2.3 · p. 20

4.2.4. RL-based System and Design Optimization

  • RL-based system optimization spans placement, scheduling, compiler and design-space exploration, and runtime control, but faces training overhead, reward sensitivity, and limited generalization.Examples include graph-neural-network placement, LLVM/HLS pass selection, PPO-guided logic synthesis exploration, and runtime coherence-mode selection.Source: §4.2.4 · p. 21

4.2.5. RL-based High-Level and Logic Synthesis

  • RL-assisted synthesis explores logic optimization and technology mapping, HLS directives, and hybrid design spaces, with learned policies reducing LUT use in reported work.The surveyed scope includes FPGA/ASIC mapping and circuit-specific optimization strategies.Source: §4.2.5 · p. 21
  • RL systems such as AutoAnnotate and AnnotationGym tune HLS configurations, while improved directive-selection policies increase stability and consistency across workloads.These approaches target Vitis and Intel toolchains and report performance improvements over baseline and manually tuned designs.Source: §4.2.5 · p. 21
  • RL combined with graph-neural-network predictors generates Pareto-optimal latency, resource-utilization, and timing trade-offs, while bit-width tuning balances accuracy, performance, and resource consumption.Source: §4.2.5 · pp. 21–22
  • Figure 6 shows RL assisting four synthesis tasks: HLS transformations, feedback from specifications and HLS reports, logic optimization, and FPGA/ASIC technology mapping.Source: Figure 6 · p. 22
  • RL shifts synthesis toward adaptive, data-driven optimization, but expensive training, weak cross-circuit and toolchain generalization, and opaque or unstable policies hinder reliability and industrial EDA integration.Agents learn from design feedback and tailor strategies to circuits, toolchains, and performance objectives; weak generalization often necessitates retraining.Source: §4.2.5 · p. 22

5. Applications of RL-based FPGA/IC platform

The applications survey organizes RL-based FPGA/IC systems by domain and shows how application constraints shape algorithm and hardware choices. Across these areas, RL serves both as a workload to accelerate and as an optimizer for system behavior.

5. Applications of RL-based FPGA/IC platform

  • Across domains, algorithm choice follows application needs, while implementations commonly use pipelining, quantized arithmetic, on-chip memories, hardware/software co-design, and parallel inference.Networking and autonomous systems often use DQN-family, TD3, PPO, or MARL methods; security, robotics, and sensing favor different RL families.Source: §5 · p. 22

5.1. Networking

  • Networking studies cover spectrum access, disrupted-network routing, waveform adaptation, satellite routing, and fault-tolerant network-on-chip routing.Chares improved wireless classification accuracy by up to 4.1× under unseen conditions, while continuous-control deployment can be sensitive to noise dynamics.Source: §5.1 · pp. 23–25

5.2. Hardware and embedded security

  • Security applications use RL for adaptive Trojan testing, biometric recognition, pre-silicon verification, and intrusion detection.AdaTest iteratively balances trigger coverage and test-generation time; an RL-TPG framework reports about 90% verification coverage.Source: §5.2 · p. 25

5.3. Robotics

  • Robotics implementations address FPGA real-number limitations through binarized networks and hardware acceleration, with reported speedups constrained by interfaces and deployment validation.DDPG training reached a 4.07× speedup for model one, while a rover QL accelerator achieved up to a 43-fold speedup over an Intel i5 CPU.Source: §5.3 · p. 26

5.4. Autonomous vehicles

  • Autonomous-vehicle studies optimize heterogeneous task placement, UAV waypoint tracking, and fuel-cell hybrid-vehicle energy management.The UAV FPGA controller delivered 204.96 actions per second at 33.57 μW and reduced tracking error by 58.14% versus PID-based controllers.Source: §5.4 · p. 26

5.5. Intelligent sensing and imaging systems

  • Sensing and imaging work applies RL to adaptive measurement selection, lightweight IoT learning, and scanning-probe microscopy control.The microscopy system reduced mean deflection error by nearly 90% and variance by about 88% against a fixed-gain controller, but evaluation used one platform and a narrow sample set [125].Source: §5.5 · p. 27

5.6. Other applications

  • Other applications include motor control, DNA sequence alignment, and converter control, with performance and validation boundaries varying by task.The motor-control study remained below state-of-the-art model predictive control, while the converter study lacked detailed FPGA resource and power analysis.Source: §5.6 · pp. 27–28

5.7. Case Study: FPGA-Based RL for Smart EV Charging

  • The EV-charging case study implements tabular QL as a modular hardware control loop, with dedicated Q-table, policy, state, action, and reward modules.It uses dual-port Action RAMs, fixed-point Q-updates, base-7 state encoding, base-3 action decoding, and explicit overcurrent penalties [121].Source: §5.7 · pp. 28–29, Figure 10 · p. 28

6. Challenges and Future directions

The survey identifies constraints on scalability, learning, portability, and deployment maturity across RL-based FPGA/IC systems. It recommends moving from specialized prototypes toward reusable and deployment-oriented platforms, with RL treated both as a workload and as an optimizer.

6.1.1. RL as the Workload

  • Tabular QL and SARSA scale poorly as state–action cardinality grows, because Q-table storage can exhaust BRAM or SRAM and implementations remain limited to small or moderate environments.Validation on synthetic or small-scale environments also leaves robustness under non-stationary workloads, aging, and uncertainty with limited evidence.Source: §6.1.1 · p. 29
  • On-device deep-RL training is constrained by computation and memory demands; external DDR reliance increases latency and power and weakens expected energy-efficiency gains.Backpropagation, gradient accumulation, parameter updates, replay buffers, and rollout memories contribute to the overhead [DQN, PPO, and AC variants].Source: §6.1.1 · pp. 29–30
  • Hardware exploration and reduced precision can threaten learning stability, while specialized designs and limited IC/SoC support constrain reuse and post-fabrication adaptability.Pseudo-random generators may introduce bias, quantization can distort estimates or gradients, and many accelerators require redesign for other tasks or algorithms.Source: §6.1.1 · p. 30

6.1.2. RL as the Optimizer

  • For RL as optimizer, reconfiguration overhead, labor-intensive toolchains, fragmented evaluation, and underexplored adversarial robustness limit deployment maturity.Frequent dynamic partial reconfiguration can offset gains, while inconsistent benchmarks impede fair comparison [DPR].Source: §6.1.2 · p. 30

6.2.1. RL as the Workload

  • Future workload-oriented designs should combine scalable hybrid architectures and transfer learning with lifelong learning, memory-aware dataflow, and adaptive precision.The survey also identifies supervised, unsupervised, and self-supervised learning as complementary approaches for efficient adaptation under hardware constraints.Source: §6.2.1 · pp. 30–31
  • Edge-oriented directions include FPGA acceleration for UAV and swarm systems, medical perception, high-energy-physics detectors, and speech recognition.The survey points to attention-heavy models and real-time perception pipelines as targets for hardware-efficient co-design [137].Source: §6.2.1 · pp. 31–32
  • The survey identifies FPGA/IC co-design for RL with LLMs and generative AI as an open direction, alongside quantum-RL control and hybrid quantum–classical architectures.Existing LLM–RL frameworks are described as predominantly CPU/GPU-evaluated, while FPGA work often omits RL or focuses on inference.Source: §6.2.1 · p. 32

6.2.2. RL as the Optimizer

  • Optimizer-oriented research directions include modular RL hardware and RL-aware toolchains, plus secure, robust, and trustworthy systems for adaptive defense.The survey also proposes exploring blockchain mechanisms with RL accelerators for secure policy updates and auditable decisions.Source: §6.2.2 · pp. 32–33

7. Conclusion

The survey synthesizes more than one hundred studies of RL on FPGA and IC platforms, identifying recurring design patterns and trade-offs across algorithms and hardware. It reports impacts across diverse application domains, persistent adoption limitations, and research directions for evaluation and integration.

7. Conclusion

  • The survey organizes more than one hundred representative studies across RL families, hardware abstraction levels, architectural strategies, and application domains.It describes a transition from software-centric experimentation toward practical, resource-aware hardware realizations.Source: §7 · p. 33
  • Effective FPGA and IC designs commonly use fine-grained parallelism, pipelined datapaths, quantized arithmetic, and tightly coupled on-chip memory hierarchies.FPGAs favor flexibility, reconfigurability, and rapid prototyping, while IC realizations prioritize determinism, compactness, and ultra-low power.Source: §7 · p. 33
  • No RL paradigm dominates across hardware and application contexts: lightweight tabular methods suit real-time embedded control, while deep RL scales to high-dimensional problems with greater architectural complexity.MARL extends to distributed and large-scale systems but brings coordination, non-stationarity, and interconnect-overhead challenges.Source: §7 · p. 33
  • RL-enabled FPGA/IC systems have demonstrated impact across networking, security, robotics, autonomous systems, sensing, imaging, and embedded optimization, with reported gains in latency, energy efficiency, throughput, and adaptability.The review identifies numerical-precision sensitivity, exploration–exploitation trade-offs under deterministic execution, limited metric standardization, and scalability, portability, and reproducibility challenges.Source: §7 · p. 33
  • Proposed research directions include standardized hardware-aware benchmarks, end-to-end system evaluation, integration with design automation and digital twins, and hybrid model-based and data-driven learning.Processing-in-memory, heterogeneous SoCs, and domain-specific accelerators are identified as opportunities for rethinking RL mapping onto silicon.Source: §7 · p. 33
Loading 2610.28699v1…