PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control

Guangzhi Tang, Neelesh Kumar, Raymond Yoo, Konstantinos P. Michmizos

arXiv:2010.09635v1cs.NEcs.LGcs.RO

TL;DR

High-dimensional continuous control creates energy demands for mobile robots, while spiking networks face representation and learning challenges. The paper proposes a population-coded spiking actor trained with deep reinforcement learning and deployed on Loihi. It reports the same performance as a deep actor network with 140 times less energy per inference on Jetson TX2 comparisons.

  • Problem

    Deep reinforcement learning is energy-intensive for mobile robots with high-dimensional continuous control needs, while spiking networks face limited representation ability and challenges in learning such tasks.

    Source: §1 · pp. 1–2

  • Method

    PopSAN uses neuron populations with learnable receptive fields to encode observations and actions, pairing a spiking actor with a deep critic and integrating it with DRL algorithms.

    Source: §1 · p. 2, §2.1 · p. 2, §2.2 · p. 4

  • Results

    PopSAN achieved the same performance as deep actor networks across tasks and DRL algorithms, while Loihi-run PopSAN used 140 times less energy per inference than a deep actor network on Jetson TX2.

    Source: §1 · p. 2, §3.1 · p. 6, §4 · p. 7

  • Takeaways & Limitations

    The authors present PopSAN as a potential alternative to deep learning for real-world reinforcement learning applications where energy efficiency and robustness matter.

    Source: §4 · p. 8

Abstract

from arXiv · show

The energy-efficient control of mobile robots is crucial as the complexity of their real-world applications increasingly involves high-dimensional observation and action spaces, which cannot be offset by limited on-board resources. An emerging non-Von Neumann model of intelligence, where spiking neural networks (SNNs) are run on neuromorphic processors, is regarded as an energy-efficient and robust alternative to the state-of-the-art real-time robotic controllers for low dimensional control tasks. The challenge now for this new computing paradigm is to scale so that it can keep up with real-world tasks. To do so, SNNs need to overcome the inherent limitations of their training, namely the limited ability of their spiking neurons to represent information and the lack of effective learning algorithms. Here, we propose a population-coded spiking actor network (PopSAN) trained in conjunction with a deep critic network using deep reinforcement learning (DRL). The population coding scheme dramatically increased the representation capacity of the network and the hybrid learning combined the training advantages of deep networks with the energy-efficient inference of spiking networks. To show the general applicability of our approach, we integrated it with a spectrum of both on-policy and off-policy DRL algorithms. We deployed the trained PopSAN on Intel's Loihi neuromorphic chip and benchmarked our method against the mainstream DRL algorithms for continuous control. To allow for a fair comparison among all methods, we validated them on OpenAI gym tasks. Our Loihi-run PopSAN consumed 140 times less energy per inference when compared against the deep actor network on Jetson TX2, and had the same level of performance. Our results support the efficiency of neuromorphic controllers and suggest our hybrid RL as an alternative to deep learning, when both energy-efficiency and robustness are important.

1 Introduction

The paper addresses the energy cost of deep reinforcement learning for high-dimensional continuous control and the representation limits of spiking neural networks. It introduces PopSAN, combining population coding with deep reinforcement learning, and reports energy-efficient control performance on Loihi.

1 Introduction

  • Deep reinforcement learning can learn control policies for complex tasks, but its high energy consumption makes it ill-suited to some mobile robots with limited onboard resources.SNNs on neuromorphic processors are presented as an energy-efficient alternative, while the paper identifies scaling to complex tasks as a challenge.Source: §1 · p. 1
  • Prior spiking approaches struggled with high-dimensional continuous control because local learning can lack a global loss, one policy-gradient method used discrete actions, and rate-coded neurons have limited representation capacity.The need for precise encoding is especially challenging at small inference timesteps, which are expected to further reduce individual neurons’ representation ability.Source: §1 · p. 2
  • Population coding represents signals with neuron populations whose receptive fields capture parts of the signal, offering a route to better representation for high-dimensional control.The paper builds on prior demonstrations of population coding for stimuli and complex high-dimensional supervised learning tasks.Source: §1 · p. 2
  • PopSAN combines a population-coded spiking actor with deep reinforcement learning and, on Loihi, used 140 times less energy per inference than a deep actor network on Jetson TX2 while achieving the same performance.The method was integrated with DDPG, TD3, SAC, and PPO and evaluated on OpenAI Gym tasks.Source: §1 · p. 2

2 Methods

The methods pair a population-coded spiking actor with a deep critic, train its encoders and network parameters through DRL, and deploy the trained controller on Loihi for continuous control.

2.1 Population-coded Spiking Actor Network (PopSAN) embedded into DRL algorithms

  • PopSAN generates actions from observations while a deep critic estimates state or action values; it is integrated with DDPG, TD3, SAC, and PPO using each algorithm’s actor objective.For SAC and PPO, PopSAN predicts the action-distribution mean, while SAC uses a deep network for the standard deviation.Source: §2.1 · pp. 2–4

2.2 Population encoding and decoding in PopSAN

  • Each observation dimension is encoded by a neuron population with Gaussian receptive fields, and trainable means and standard deviations shape encoding; output populations are decoded into real-valued actions.The encoder can generate spikes probabilistically or through deterministic one-step soft-reset integrate-and-fire neurons.Source: §2.2 · p. 4

2.3 PopSAN training

  • DRL loss gradients train the PopSAN parameters, with SNN parameter gradients computed through extended spatiotemporal backpropagation and parameters updated every T timesteps.Input-population parameters are updated independently for each population.Source: §2.3 · pp. 4–5

2.4 Energy-efficient continuous control with Intel’s Loihi neuromorphic chip

  • For Loihi control, encoding and decoding are split between the host and embedded chip, and layer-wise rescaling maps the trained full-precision PopSAN onto Loihi’s low-precision hardware.The host computes encoding stimulation and final actions; the embedded chip generates spikes and computes firing rates.Source: §2.4 · p. 5

3 Experiments and Results

Across OpenAI Gym continuous-control tasks, experiments benchmarked PopSAN against deep actors and other SNN designs, tested population coding, and evaluated Loihi deployment. Results report parity with deep actors, stronger performance than the comparison SNN approaches, and a 140-fold energy advantage on Loihi over Jetson TX2.

3.1 Benchmarking PopSAN against mainstream DRL algorithms

  • PopSAN matched the corresponding deep actor networks across all tasks with both on-policy and off-policy DRL algorithms.The comparison used ten models per algorithm, trained for one million steps and evaluated on OpenAI Gym tasks.Source: §3 · p. 5, Figure 3 · p. 6, §3.1 · p. 6

3.2 Benchmarking PopSAN against other SNN design approaches

  • The compared SNN approaches did not match PopSAN even with five times more timesteps; PopSAN (T=5) consistently outperformed RateSAN (T=5,15,25).The authors attribute the gap to limited representation capacity and, for DNN-SNN, precision loss during conversion.Source: §3.2 · p. 6, Figure 4 · p. 7, Table 1 · p. 7

3.3 Learning in neuron populations

  • Learnable input encoders outperformed fixed encoders at every tested input population size, and their spike encodings increased distances between observations.Decreasing the input population size also hurt performance.Source: §3.3 · pp. 6–7
  • Increasing the output population size improved performance, while larger populations learned redundant action representations covering a wider range of action values.The tested output population sizes were 2, 3, 5, and 10 neurons per action dimension, with input population size fixed at 10.Source: §3.3 · p. 7

3.4 Evaluating continuous control on Loihi

  • 140 times less energy per inference than the Jetson TX2 deep actor network, PopSAN on Loihi also achieved high inference speed for real-time control.On HalfCheetah-v3, Loihi PopSAN rewards showed only a marginal decrease relative to PopSAN on a full-precision GPU.Source: §3.4 · p. 7

4 Discussion and Conclusion

PopSAN matched deep-network performance while using substantially less energy, and its learned population coding improved observation and action representations. Its integration with on-policy and off-policy algorithms broadens its applicability to continuous-control tasks.

4 Discussion and Conclusion

  • PopSAN achieved the same performance as deep networks while being two orders of magnitude more energy-efficient.The reported Loihi comparison found 140 times lower energy use than a deep actor network on Jetson TX2.Source: §4 · pp. 7–8
  • For Hopper-v3, larger output populations learned redundant action representations and yielded higher rewards, while learning in input populations improved representation and performance.Figure 5 shows receptive fields for output-population neurons corresponding to an action.Source: Figure 5 · p. 8
  • Table 2 compares power performance and inference speed across hardware.The supplied caption identifies the comparison’s scope but does not provide its values.Source: Table 2 · p. 8
  • Population coding increased representation capacity, improved observation and action encodings, and enabled complex high-dimensional control with fewer inference timesteps.The authors attribute the encoding gains to learnable population representations.Source: §4 · p. 8
  • PopSAN integrated successfully with on-policy and off-policy DRL algorithms, supporting its use across tasks requiring specialized solutions.The authors propose it as a potential alternative to deep learning when energy efficiency and robustness matter.Source: §4 · p. 8

1 PopSAN training using backpropagation

PopSAN training backpropagates loss gradients through the spiking network over time and updates its parameters periodically. The method also computes gradients for input-population parameters through neuron stimulation strength.

1 PopSAN training using backpropagation

  • The loss gradient with respect to the computed action trains PopSAN parameters, with each output population’s parameters updated independently.The paper says all PopSAN parameters are updated after every T timesteps.Source: §1 · p. 11
  • The SNN uses extended spatiotemporal backpropagation with a rectangular-function approximation to the gradient of a spike.The rectangular function is described as a surrogate for the spike gradient.Source: §1 · p. 11
  • Gradients flow from the output population through successive layers at each timestep, and parameter gradients collect contributions across timesteps.At t = T, the future-timestep additive term is absent.Source: §1 · p. 11
  • For input-population parameters, the loss gradient is backpropagated through neuron stimulation strength whether or not a spike occurred.Source: §1 · p. 11

2 Hyperparameters for training regular DRL and PopSAN

The experiments used open-source DRL codebases and generally matched PopSAN hyperparameters to those of the deep actor network unless specified otherwise. Configurations varied by algorithm and included population-network and auxiliary-network settings.

2 Hyperparameters for training regular DRL and PopSAN

  • Experiments built on OpenAI Spinning Up, OpenAI Baselines, and Stable Baselines, with PopSAN using deep-actor hyperparameters unless noted otherwise.Source: §2 · pp. 11–12
  • Deep actor DDPG, TD3, SAC, and PPO used 256-unit two-layer actor and critic networks, with algorithm-specific output, learning-rate, exploration, or optimization settings.DDPG and TD3 used tanh actor outputs; SAC used a Gaussian policy with separate mean and log-standard-deviation outputs.Source: §2 · p. 12
  • DDPG and TD3 PopSAN used input and output populations around two 256-unit LIF layers, population size 10, and learning rate 1e-4.Encoding was deterministic for HalfCheetah and Ant, and probabilistic for Hopper and Walker2d.Source: §2 · p. 12
  • SAC PopSAN used a spiking policy-mean network plus a dense log-standard-deviation network; PPO used a spiking policy with a learned log-standard-deviation parameter.Both used population size 10; SAC used deterministic encoding for all tasks, as did PPO.Source: §2 · p. 13

3 DNN to SNN conversion method

The DNN-to-SNN conversion procedure trains a deep actor, rescales its parameters for spiking activations, then selects an additional rescaling factor by evaluation reward.

3 DNN to SNN conversion method

  • The method first trains a deep actor with the chosen DRL algorithm and matches its architecture to PopSAN for a fair comparison.Source: §3 · p. 13
  • It transfers trained DNN parameters to the SNN and rescales layer biases and weights using maximum activations so firing rates can represent the DNN activation range.The firing-rate range is [0,1]; bias and weight rescaling use the current and previous layers’ maximum outputs.Source: §3 · p. 13
  • A grid search then selects an additional weight-and-bias rescaling factor between 0.1 and 1.0 by average reward over 10 evaluation episodes.The episodes were randomly sampled.Source: §3 · p. 13

4 Power measurement details

Power and inference speed were averaged across repeated test runs, while the supplementary population studies report benefits from learning input populations and using larger output populations.

  • Power consumption and inference speed were averaged over 10 runs, each covering 100k test observations.Source: §4 · p. 13
  • Learning in input populations improved input representation and performance across all environments.Source: Figure S1 · p. 13
  • Larger output populations resulted in higher rewards for all environments.Source: Figure S2 · p. 14
  • Power measurements used device-specific sensors; Loihi power was averaged across eight networks deployed simultaneously on the Nahuku chipset.Source: §4 · p. 14

5 Additional ablation studies for neuron populations

The additional ablations report PopSAN results for varying input and output population sizes in environments not covered in the main text.

  • Figures S1 and S2 report PopSAN results for input and output population sizes 2, 3, 5, and 10.Source: §5 · p. 14
  • The additional population-size results cover environments not covered in the main text.Source: §5 · p. 14
  • The ablations vary both input and output population sizes.Source: §5 · p. 14
Loading 2010.09635v1…