Spiking Neural Network Predicting Sequence of the External Worlds States in Model-Based Reinforcement Learning

1 Introduction

• The paper frames its contribution as a step toward entirely spiking model-based RL, a challenging research problem.The authors identify local learning as an alternative to backpropagation and note that most SNN-based RL work using it has focused on model-free RL.Source: §1 · pp. 1–2
• Applying backpropagation directly to SNNs is hindered by discrete spiking, while surrogate-gradient approaches have drawbacks in network size, timing, and neuroprocessor implementation.The paper notes that backpropagation requires a smooth, differentiable loss and lacks an explicit concept of time.Source: §1 · p. 2
• Fully spiking model-based RL remains largely unexplored, with prior examples relying on external arbitration or omitting temporal transitions.The authors describe their earlier work as demonstrating learning of external-world dynamics from scratch using spiking signals while accounting for time.Source: §1 · p. 2
• The work applies a previously learned external-world dynamics model to predict chains of subsequent states, supporting planning actions toward a target state.The paper presents this as the next step in the model's intended use.Source: §1 · pp. 1–2
• The study describes an inference SNN and compares its predictions with exact digital inference on the ping-pong RL task.The same task was used for learning the world-dynamics model.Source: §1 · p. 3

2 Discretized External World Dynamics

• The world-dynamics model is incorporated as a smaller SNN trained elsewhere, and the present network uses it to predict subsequent states from the current state.The network is intended to select the most probable next state.Source: §2 · p. 3
• Each world-state dimension is represented by its current discrete value and a time bin indicating how long that value has held.The paper uses an equal number of temporal intervals, T, for every dimension.Source: §2 · p. 3
• In the ping-pong example, five dimensions encode the ball's two coordinates, two velocity components, and the racket's Y coordinate.The figure caption identifies the ball coordinates and velocity components and racket Y coordinate as signals fed to the controller SNN.Source: §2 · p. 3, Figure 1 · p. 4

3.1 LIF Neuron

• The network uses leaky integrate-and-fire neurons, whose membrane potential changes with incoming synaptic spikes and fires when it exceeds a threshold.Forced firing can also be triggered by strong excitatory synapses, after which the membrane potential resets to zero.Source: §3 · p. 4, §3.1 · p. 4

3.2 Gating Synapses and Neuron Inactivation

• Gating synapses alter a neuron's activity time, enabling reflective connections to block firing or activate repeated input spikes within an interval.Negative reflective weights provide a refractory period; positive reflective connections repeat input while inter-spike intervals remain within the specified bound.Source: §3.2 · p. 5

3.3 Emitting Spike Trains

• A neuron can emit a spike train after firing, and a negative-weight gating spike can terminate that train early.The neuron parameter controlling the train determines its length when greater than one.Source: §3.3 · p. 5

3.4 Synaptic Plasticity

• Synaptic plasticity is used during learning, but inference operates with fixed synaptic weights.The described plasticity model includes 1-factor depression, 2-factor anti-Hebbian depression, and dopamine-based 3-factor updates with weight redistribution.Source: §3.4 · pp. 5–6

4.1 Representation of Time – the Network Indicating how Long Ago the Current Elementary State was Reached

• One small SNN per elementary state uses one input and T output neurons to indicate the current time value for that state.The corresponding output neuron emits a spike train for the current value.Source: §4.1 · pp. 6–7
• The time-representation module uses fast and slow forcing connections and blocking connections to determine how long input stimulation lasts.Figure 2 labels weights on blocking connections and delays on slow forcing connections.Source: Figure 2 · p. 6

4.2 The Learning Subnetwork

• The inference SNN incorporates the external-world dynamics model as a distinct imported part.Figure 4 marks the imported model with a green rectangle.Source: Figure 4 · p. 8
• The learning subnetwork is an ensemble of modified CoLaNET classifiers, one for each <sai, tai> pair, that find states able to transition immediately to a target state.The classifiers differ for zero and non-zero tai; the learned part is exported into the inference SNN.Source: §4.2 · p. 8

5 Testing the SNN Inferring External World Dynamics Rules on the ATARI Ping-Pong RL Problem

• The initial-state inputs activate Imagination neurons, whose persistent spikes feed the imported model and trigger the next predicted state while inhibiting competing states in the same dimension.The loop repeats predictions sequentially; the illustrated network is used for one-time prediction, though the authors note the idea could support permanent predictions.Source: §5 · pp. 8–9
• The experiment reproduces measurements from prior work using the same ping-pong environment, comparing inference mechanisms rather than evaluating the dynamics model itself.The earlier evaluation used a C++ program with comparable inference logic hard-coded.Source: §5 · p. 9

5.1 General Accuracy.

• The evaluation asks where the ball hits the left boundary and measures standard error plus the percentage of cases with no prediction.No-prediction cases are attributed to an endless loop in the predicted state sequence.Source: §5.1 · p. 9, Table 1 · p. 9
• The SNN and algorithmic inference produced very similar accuracy curves across starting horizontal positions, and their binned results were reported as very close.The comparisons concern hit-point accuracy by starting position and heatmaps of no-prediction fraction and intersection-Y error.Source: §5.1 · pp. 9–10, Figure 2 · p. 10, Figure 3 · p. 10

• The proposed structure incorporates an ensemble of CoLaNET networks expressing world dynamics to predict subsequent states.The dynamics model was created in another SNN context and inserted into the proposed structure.Source: §6 · p. 10
• The work advances the “only-spikes, only-SNN” approach by replacing C++ prediction-support procedures with SNN structures.The paper frames this as a significant step toward neurochip implementation.Source: §6 · p. 10
• The SNN inference results are close to those of the algorithmic implementation of the inference mechanism.Source: §6 · p. 11
• The next challenge is an SNN architecture that learns and performs inference simultaneously without physically transferring the learned model.Source: §6 · p. 11