PaperlayerView exact revision on arXiv

Source-linked AI summary

All On-Board: Fully On-Chip Neuromorphic Q-Learning with Embedded CartPole Simulation

Steven C. Nesbit, Giovanni T. Michel, Gerd J. Kunde, Edward Kim, Andrew T. Sornborger

arXiv:2609.32317v1cs.NEcs.LGcs.SYeess.SY

TL;DR

Rising AI energy demands and the difficulty of implementing complete learning algorithms on neuromorphic hardware motivate this work. It runs a closed-loop Q-learning agent and CartPole simulation entirely on Loihi 2; for equivalent training success, Loihi 2 provides over 1,000× energy efficiency and needs 51% as much time as CPU training.

  • Problem

    Implementing complete learning algorithms on neuromorphic hardware remains challenging under constraints such as limited plasticity rules, local-only memory, asynchronous execution, and reduced precision, amid rising AI energy costs.

    Source: A. Energy Efficiency in Artificial… · p. 1

  • Method

    The authors implement a closed-loop Loihi 2 agent that co-locates Q-learning updates, action selection, reward/state handling, and CartPole simulation without host computation between transitions.

    Source: A. Energy Efficiency in Artificial… · p. 2

  • Results

    Over 1,000× energy efficiency for equivalent training success; Loihi 2 required 51% as much time to train the same number of successful agents as CPU training.

    Source: B. Performance Profiling · p. 8

  • Takeaways & Limitations

    The results demonstrate that a complete closed-loop reinforcement-learning loop can run on a neuromorphic processor without host computation between environment transitions.

    Source: A. Training Algorithm Performance · p. 8

    CartPole is only an initial benchmark; more complex control tasks are needed to assess real-world applicability.

    Source: B. Limitations · p. 8

Abstract

from arXiv · show

As AI models grow in size and usage, their energy demands increase dramatically, raising sustainability and economic concerns. Neuromorphic hardware, inspired by the energy efficiency of the brain, seeks to address this challenge by offering low-power, fast-processing alternatives to conventional computing. Such hardware is particularly well-suited to control systems deployed in resource-constrained environments, which are best trained via reinforcement learning (RL). This contribution presents the design and implementation of a fully on-chip, closed-loop Loihi 2 RL agent. Our neuromorphic circuit consists of a fully embedded Q-learning algorithm and an on-chip simulation of the CartPole-v0 environment on Loihi 2. Our Q-learning algorithm trained the same number of successful agents as the CPU implementation in only half the execution time and with two orders of magnitude less dynamic power. These findings demonstrate the viability of RL on neuromorphic hardware and highlight its promise for building energy-efficient, real-time, embedded AI systems.

I. INTRODUCTION

AI’s growing computational and energy costs motivate energy-efficient hardware, yet complete learning algorithms remain uncommon on neuromorphic systems. This work addresses that gap by co-locating Q-learning and CartPole simulation on Loihi 2.

A. Energy Efficiency in Artificial Intelligence

  • Neuromorphic systems use event-driven spiking neural networks, with computations occurring when spikes are generated.The passage describes this as a route to lower energy consumption and power density.Source: A. Energy Efficiency in Artificial… · p. 1
  • Complete learning algorithms remain uncommon on neuromorphic hardware because hardware constraints make on-chip learning challenging.Constraints include limited on-chip plasticity rules, local-only memory access, asynchronous execution, and reduced numerical precision.Source: A. Energy Efficiency in Artificial… · p. 1
  • This implementation co-locates Q-learning updates, action selection, reward/state handling, and CartPole simulation on Loihi 2, eliminating host computation between environment transitions.Earlier neuromorphic work included policies, value networks, and other learning approaches, but differed in computational boundary.Source: A. Energy Efficiency in Artificial… · p. 2

B. Q-Learning

Q-learning updates action values from interaction with an environment to learn a reward-maximizing policy. Its discrete state–action space and iterative updates align with neuromorphic hardware.

B. Q-Learning

  • Q-learning is a model-free method in which an agent learns an environment policy through interaction and cumulative-reward maximization.Its discrete state–action space and iterative weight updates can map to spike-based neural dynamics.Source: B. Q-Learning · p. 2
  • The action–value function Q(s_t, a_t) estimates expected cumulative reward for taking action a in state s at time t.The passage introduces its update through Equation 1.Source: B. Q-Learning · p. 2
  • The Q-value update uses the learning rate, immediate reward, discount factor, and maximum next-state Q-value to form a temporal-difference target and error.The update term U is the Q-value adjustment; γ sets the importance of future rewards.Source: B. Q-Learning · p. 2

C. CartPole Environment

CartPole is a benchmark control task that balances a pole on a moving cart. The paper embeds its physics simulation on Loihi 2 as part of a fully on-chip learning loop.

C. CartPole Environment

  • CartPole is a standard reinforcement-learning benchmark in which horizontal cart forces keep a pole upright as long as possible.Small angular deviations can quickly destabilize the pole, requiring rapid corrective actions.Source: C. CartPole Environment · pp. 2–3
  • The study retains standard CartPole-v0 to test whether the complete learning loop and nonlinear environment dynamics can run on one neuromorphic processor, not to establish state-of-the-art control.The passage notes that standard CartPole is a useful starting point but not a particularly challenging test of modern AI agents.Source: C. CartPole Environment · p. 3

A. Neuromorphic CartPole Simulation

  • The Loihi 2 simulation adapts OpenAI Gym’s CartPole-v0 and embeds the environment dynamics for on-chip Q-learning and inference.It was developed for deployment using Lava on Loihi microcode.Source: A. Neuromorphic CartPole Simulation · p. 3
  • CartPole combines standard-benchmark comparability, embeddable physics, and a discrete action space suited to neuromorphic architectures.Its dynamics relate applied force to cart acceleration and pole angular acceleration.Source: A. Neuromorphic CartPole Simulation · p. 3

1) Function Approximation:

The CartPole dynamics are implemented with microcode-compatible function approximations and a simpler random-number generator. The simulation updates four state variables at fixed time steps and checks for failure conditions.

1) Function Approximation:

  • Sine, cosine, and division approximations produce maximum errors of 0.03632 rad/s^2 in pole angular acceleration and 0.00166 m/s^2 in cart acceleration.Sine and cosine are approximated over [−0.418, 0.418] rad; division uses Newton’s method.Source: 1) Function Approximation · p. 3

2) Random Number Generation:

  • The on-chip LCG replaces MT19937 because replicating MT19937 was infeasible under Loihi 2 hardware constraints; it has a shorter period but is called at most 400 000 times per training run.The LCG uses a = 65 793, c = 4 282 663, and m = 223 to generate pseudorandom 23-bit unsigned integers.Source: 2) Random Number Generation · pp. 3–4

3) CartPole State Variables:

  • At each 0.02 s Euler step, the simulation updates cart position, cart velocity, pole angle, and pole angular velocity, then checks for failure at ±2.4 m cart position or 0.2095 rad pole angle.The process outputs pole angle, pole angular velocity, reward, cart position, and cart velocity as 24-bit signed integers.Source: 3) CartPole State Variables · p. 4

B. Q-Learning

The on-chip Q-learning circuit runs a nine-step loop across interconnected modules. It simulates CartPole, selects actions, computes temporal-difference updates, and applies Q-value changes.

B. Q-Learning

  • Five interconnected modules use a gating chain to synchronize the nine-step Q-learning loop.The overview describes the loop as the sequence coordinating the circuit’s algorithmic operations.Source: B. Q-Learning · p. 4
  • The CartPole module updates the state and supplies reward; the input layer encodes angle and angular velocity as a 72-neuron one-hot state, while the rate calculator tracks episode score and computes exploration and learning rates.Training halts when the episode score reaches 500.Source: B. Q-Learning · p. 4
  • The two-neuron output layer provides Q-values for left and right actions, while action selection can be overridden randomly and the reward is scaled to −8 or +8 for the target layer.The output layer also routes the current state–action Q-value for the next weight update.Source: B. Q-Learning · p. 4
  • The target layer computes TDtarget, and the update calculator uses it with the learning rate and current Q-value to compute α(TDerror) for the weight update trace.The trace determines the Q-value update sent to the output layer.Source: B. Q-Learning · p. 4
  • The stored action is sent to CartPole, and Hebbian learning applies the trace-based Q-value update before the gating chain restarts the next state–action cycle.The previous state is relayed to the input layer for the next iteration.Source: B. Q-Learning · p. 4

1) Gating Chain:

A nine-neuron gating chain sequences the algorithm’s operations. The policy network discretizes CartPole observations into a 72-state input and updates its Q-table through a plastic connection.

1) Gating Chain:

  • Nine gating neurons activate sequentially to control information propagation through the algorithm’s nine-step loop.The chain serves a role similar to gating in neuromorphic backpropagation.Source: 1) Gating Chain · p. 4

2) Policy Network:

  • Figure 3 maps neuron layers to rows and algorithm time steps to columns, with arrows indicating information flow and layers shown only when gated on.Its time-step columns correspond to Loihi 2 time steps; ϵ and α denote exploration and learning rates.Source: Figure 3 · p. 5
  • The policy input discretizes pole angle into six bins and angular velocity into twelve, forming 72 one-hot states to balance coverage with neuromorphic memory limits.The discretization was based on empirical testing of quantized off-chip Q-learning.Source: 2) Policy Network · p. 5
  • A two-neuron output layer connects to the input through a plastic Q-table matrix, and Hebbian updates use tag T and weight update dW to change only the active state–action pair.The relay St+1 stores state information for use in Hebbian weight updates.Source: 2) Policy Network · p. 5

3) Random Decision:

The random-decision module uses episode score to govern exploration and also supplies the learning rate for weight updates. It determines whether to override the Q-policy and which random action to take.

3) Random Decision:

  • The exploration probability is governed by Equation 11, and its output is also used as the learning rate in the weight update module.The rate calculator derives the exploration rate from the episode score.Source: 3) Random Decision · pp. 5–6
  • The random-action process compares an LCG-generated number with ϵ; if the number is no greater than ϵ, it selects a random action of −1 or 1.Otherwise, no random action is selected.Source: 3) Random Decision · p. 6

4) Decision Process:

The decision process selects an action from the output layer’s Q-values, with random action input able to override that decision. The weight-update module computes and sends Q-table changes, with fixed-point constraints shaping the update.

4) Decision Process:

  • The action-selection neuron chooses left or right from the signed difference between the two Q-values; random input overrides this decision.Equal Q-values trigger a random choice when no random-action input overrides the neuron.Source: 4) Decision Process · p. 6

5) Weight Update:

  • The target calculator adds γ max_a Q(s_t+1, a) to positive rewards but uses only the negative reward, assuming future reward after failure is zero.It sends a target onward only when a reward input arrives.Source: 5) Weight Update · p. 6
  • The update calculator scales TDtarget − Q(s_t, a_t) by the learning rate and 1/8, then computes separate updates for the two actions.Without a learning-rate input, the update is zero.Source: 5) Weight Update · p. 6
  • Updated Q-values are constrained to an 8-bit signed range, and their differences from previous values form traces that drive Hebbian learning.When both action values saturate at 126, one is set to 125 to preserve a difference.Source: 5) Weight Update · p. 6
  • Fig. 4 compares CPU- and Loihi 2-trained agents using solve-rate, top-agent, top-half, and all-agent panels.Panels a–b show the percentage scoring at least 195; c–h show score distributions for the listed agent groups.Source: Figure 4 · p. 7

III. RESULTS

The study trained 1,000 agents on each of CPU and Loihi 2 and evaluated their Q-tables in CartPole-v0. CPU training reached a higher peak solve rate, while Loihi 2 reached its peak earlier and achieved the same absolute number of successful agents in about half the wall-clock time.

A. Agent Performance

  • The evaluation trained 100 agents at each of 10 update counts on both platforms, then tested the resulting Q-tables in CartPole-v0.An agent solves the task with a mean score of at least 195 over 100 trials, each capped at 200.Source: A. Agent Performance · p. 7

1) CPU Training:

2) Loihi 2 Training:

  • 62% of CPU-trained agents solved CartPole-v0 at 90,000 updates, versus Loihi 2’s 36% peak at 50,000 updates; both produced the same absolute number of successful agents, with Loihi 2 taking half the wall-clock time.CPU success fell slightly to 61% at 100,000 updates; Loihi 2’s rate plateaued near 30% from 60,000 updates onward.Source: 2) Loihi 2 Training · p. 7, 1) CPU Training · p. 7

B. Performance Profiling

Performance profiling compared Q-learning training and inference on Loihi 2 and a CPU. Loihi 2 used much less power and time per update, while a library-based GPU run consumed more energy and took longer than Loihi 2.

B. Performance Profiling

  • 600× lower training power and 2.8× faster execution per weight update yielded over 1,000× energy efficiency for equivalent training success; Loihi 2 used 51% as much wall-clock time.The wall-clock comparison is despite the CPU taking 2.8 times longer per weight update.Source: B. Performance Profiling · p. 8
  • A direct GPU comparison was outside scope because it would require a fully on-device CartPole-v0 environment and integer-precision updates matching Loihi 2’s constraints.The stated constraints include 8-bit weights and 24-bit activations.Source: B. Performance Profiling · p. 8
  • The PyTorch/OpenAI Gym GPU run had a 67% success rate, but used three orders of magnitude more energy and took an order of magnitude longer than Loihi 2.The paper notes that this was not a direct matched comparison to its neuromorphic implementation.Source: B. Performance Profiling · p. 8

IV. DISCUSSION

The Loihi 2 agent matches CPU-trained agent success in half the training time, with energy efficiency over three orders of magnitude greater than CPUs. The discussion identifies CartPole as an initial benchmark and frames broader gains for real-time control as a path that still requires testing on more complex tasks.

A. Training Algorithm Performance

  • Energy efficiency is over three orders of magnitude greater than CPUs, while CPU-trained agent success is matched in half the training time.The implementation runs the Q-learning update and CartPole simulation on Loihi 2 without host computation between environment transitions.Source: A. Training Algorithm Performance · p. 8

B. Limitations

  • 36% versus 62% success rates leave the neuromorphic implementation at 58% of the CPU’s success rate.The authors suggest 24-bit fixed-point arithmetic and reduced RNG performance in random action selection as likely contributors.Source: B. Limitations · p. 8
  • CartPole is only an initial benchmark; more complex tasks such as UAS control are proposed for a more realistic applicability assessment.The implementation sustains over 17 kHz weight-update rates, above the cited 100–1 000 Hz sensing and actuation frequencies typical for quadcopter control.Source: B. Limitations · p. 8

C. Broad Implications

  • For discrete state–action control tasks with real-time requirements, the work presents neuromorphic RL as a path toward 1 000× energy-efficiency improvements.The authors state that these gains must extend to more complex benchmarks and deployed applications for real impact on AI sustainability.Source: C. Broad Implications · pp. 8–9
  • Together with neuromorphic backpropagation, these results support the synfire-gated synfire chain framework’s ability to support a broad class of AI algorithms previously realized only on von Neumann architectures.The authors describe mapping algorithms designed for globally clocked processors onto asynchronous neuromorphic chips as a plausible path toward larger, more capable neuromorphic AI models.Source: C. Broad Implications · p. 9
Loading 2609.32317v1…