PaperlayerView exact revision on arXiv

Source-linked AI summary

Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks

Zijie Xu, Bingrui Guo, Yiding Sun, Yiting Dong, Zhile Yang, Zhaofei Yu

arXiv:2610.07808v1cs.NEcs.LG

TL;DR

Low-timestep DSQNs can lose performance even as reducing simulation timesteps lowers computational costs. The paper analyzes their Q-value errors and introduces an auxiliary-ANN common-mode correction. CMC-DSQN improves low-timestep results on Atari and MiniAtar, surpassing the ANN baseline at T = 4.

  • Problem

    Competitive DSQN performance often relies on many simulation timesteps, while reducing timesteps can substantially degrade performance.

    Source: §1 · p. 2

  • Method

    CMC-DSQN replaces the SNN outputs’ common-mode component with an auxiliary ANN estimate, using the ANN during training and removing it at inference.

    Source: §1 · p. 2

  • Results

    Nearly 20% at T = 2: CMC-DSQN exceeds state-of-the-art DSQN baselines by nearly 20%, and at T = 4 surpasses the ANN baseline on Atari and MiniAtar.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Because the common-mode correction leaves greedy action selection unchanged, the auxiliary ANN can be removed at inference without additional deployment computation.

    Source: §1 · p. 2

Abstract

from arXiv · show

Spiking neural networks (SNNs) offer sparse and event-driven computation, making them attractive for energy-constrained reinforcement learning (RL) on edge devices. In value-based RL, deep spiking Q-networks (DSQNs) combine such efficiency with action-value estimation for decision making. However, existing DSQNs often require multiple simulation timesteps for competitive performance, increasing computational and energy costs, whereas reducing the timesteps can cause substantial performance degradation. We investigate this degradation from the perspective of Q-value estimation errors. By decomposing errors across actions into common-mode and differential-mode components, we find that low-timestep DSQNs suffer disproportionately from common-mode errors shared across action values, which are particularly detrimental to temporal-difference learning through bootstrapped targets. Based on this finding, we propose Common-Mode Compensation Deep Spiking Q-Network (CMC-DSQN), which uses an auxiliary ANN to compensate for common-mode errors in the SNN outputs. At inference, greedy action selection can be performed directly from the SNN outputs, allowing the auxiliary ANN to be completely removed and preserving the energy efficiency of SNNs. Extensive experiments on Atari and MiniAtar environments demonstrate substantial performance improvements under low-timestep settings. CMC-DSQN outperforms state-of-the-art DSQN baselines by nearly $20\%$ at $T=2$ and further surpasses the ANN baseline at $T=4$.

1 INTRODUCTION

Low-timestep DSQNs degrade despite the efficiency appeal of spiking computation. The paper attributes this degradation to common-mode Q-value errors and introduces an auxiliary-ANN correction that improves low-timestep results while allowing the ANN to be removed at inference.

1 INTRODUCTION

  • SNNs compute with sparse binary spikes, offering an event-driven alternative that can reduce computation and inference latency on neuromorphic hardware.Their sparse computation is attractive for RL agents that repeatedly make decisions on resource-constrained edge devices.Source: §1 · p. 1
  • Although DSQNs can achieve competitive performance, they often require many simulation timesteps, increasing inference latency and computational cost.Reducing timesteps is desirable, but low-timestep DSQNs can suffer substantial performance degradation.Source: §1 · p. 2
  • Low-timestep DSQNs are dominated by common-mode Q-value errors, which affect TD learning more directly than differential-mode errors.CliffWalking experiments found removing common-mode errors more effective than removing differential-mode errors.Source: §1 · p. 2
  • CMC-DSQN replaces the SNN outputs’ action-wise mean with an auxiliary ANN’s common-mode estimate, jointly optimized through temporal-difference learning.Because the correction does not change greedy action selection, the ANN can be removed at inference.Source: §1 · p. 2
  • At T = 2, CMC-DSQN exceeds state-of-the-art DSQN baselines by nearly 20%, and at T = 4 it surpasses the ANN baseline on Atari and MiniAtar.Figure 1 compares directly trained DSQNs on MiniAtar, averaging performance across environments and normalizing by the ANN.Source: Figure 1 · p. 1, §1 · p. 2

2 RELATED WORK

Prior work applies SNNs to policy learning and to value-based Q-learning, but low-timestep DSQN estimation errors remain insufficiently understood. This paper distinguishes its error decomposition and training-only auxiliary-ANN design from existing value-decomposition approaches.

2.1 SPIKING NEURAL NETWORKS FOR POLICY-BASED REINFORCEMENT LEARNING

  • Policy-based SNN studies focus on spiking policies for continuous control, using ANN-to-SNN conversion or actor–critic learning.This work instead investigates value-based RL through DSQNs.Source: §2.1 · p. 2

2.2 SPIKING NEURAL NETWORKS FOR VALUE-BASED REINFORCEMENT LEARNING

  • Value-based DSQNs use ANN-to-SNN conversion or direct surrogate-gradient training, with later work adding recurrent mechanisms, normalization, and neuron designs.Q-value estimation errors in low-timestep DSQNs remain insufficiently understood.Source: §2.2 · pp. 2–3

2.3 VALUE DECOMPOSITION IN DEEP Q-LEARNING

  • Unlike dueling-style methods that decompose the Q-function, this work decomposes estimation errors into common- and differential-mode components, finding common-mode errors dominate at low timesteps.CMC-DSQN uses a training-only auxiliary ANN without sharing the SNN encoder, unlike typical shared-encoder dueling architectures.Source: §2.3 · p. 3

3 PRELIMINARIES

The preliminaries introduce Q-value learning with bootstrapped temporal-difference targets and the spiking-network setup used in DSQNs. They describe how simulation timesteps trade value-estimation accuracy against computational cost.

3.1 DEEP Q-LEARNING

  • The RL setup treats the agent’s state, action, transition, reward, and discount factor as an MDP, with the objective of learning the optimal action-value function.The passage identifies the return and policy action-value function as quantities defined within this setup.Source: §3.1 · p. 3
  • DQNs approximate the optimal action-value function with a neural network and minimize TD loss against a bootstrapped target; estimated next-state values can propagate errors through successive updates.At inference, the agent selects the action with the largest estimated Q-value.Source: §3.1 · p. 3

3.2 SPIKING NEURAL NETWORKS

  • A DSQN runs an SNN for T simulation timesteps per observation; more timesteps generally improve value representation but raise computational cost.The SNN uses leaky integrate-and-fire dynamics with membrane decay, synaptic input, a firing threshold, and hard reset.Source: §3.2 · p. 3
  • Figure 2 presents deterministic CliffWalking trajectories for ANN and low-timestep SNN policies alongside state-wise common-mode error ratios relative to the optimal action values.The trajectories aggregate 20 random seeds; red crosses indicate states leading to infinite loops.Source: Figure 2 · p. 4

4 COMMON-MODE ERRORS IN DEEP SPIKING Q-NETWORKS

The CliffWalking analysis decomposes action-value estimation errors into common-mode and differential-mode components, then finds common-mode dominance in low-timestep SNNs and a more direct route into bootstrapped TD targets. Removing common-mode errors improves learning, while exact next-state Q-values also substantially improve it.

4 COMMON-MODE ERRORS IN DEEP SPIKING Q-NETWORKS

  • The deterministic CliffWalking case study permits exact computation of the optimal action-value function and each policy’s action-value function, enabling direct error analysis.Its small state space allows analysis of error effects on TD bootstrapping.Source: §4 · p. 4

4.1 COMMON- AND DIFFERENTIAL-MODE ERROR DECOMPOSITION

  • The analysis decomposes each action’s estimation error into a shared-across-actions common-mode component and a differential component capturing relative action-value errors.It also defines a state-wise ratio to quantify the common-mode error’s magnitude relative to the two components.Source: §4.1 · p. 4

4.2 COMMON-MODE ERROR DOMINANCE IN LOW-TIMESTEP SNNS

  • Low-timestep SNNs show less stable learning than an ANN and larger Q-value errors, with common-mode error dominating while the ANN’s two error modes are relatively balanced.The errors are measured against the exact Qπ of each learned greedy policy; state-wise analysis also finds common-mode dominance at nearly every state.Source: §4.2 · pp. 4–5, Figure 3 · p. 5

4.3 ERROR PROPAGATION THROUGH TEMPORAL-DIFFERENCE BOOTSTRAPPING

  • Common-mode error enters the bootstrap target directly, while differential-mode error affects it only through maximization, giving common-mode error a more direct path into successive TD updates.The differential contribution is bounded by the largest absolute action-wise differential error.Source: §4.3 · p. 5
  • Removing common-mode error improves learning, whereas removing differential-mode error does not; using exact Q* for bootstrapping also substantially improves learning.Together, these findings implicate bootstrap Q-value accuracy, especially its common-mode component, in low-timestep SNN learning.Source: §4.3 · pp. 5–6

5 COMMON-MODE COMPENSATION FOR DEEP SPIKING Q-NETWORKS

CMC-DSQN addresses low-timestep DSQNs’ common-mode errors by replacing the SNN’s shared output component with an auxiliary ANN estimate. The ANN is training-only, and CliffWalking results show improved learning stability and reduced Q-value errors.

5 COMMON-MODE COMPENSATION FOR DEEP SPIKING Q-NETWORKS

  • Exact Q references permit explicit error removal on CliffWalking but are unavailable in practical RL; estimating the entire error adds complexity and underperforms CMC-DSQN.The alternative auxiliary ANN must model both common- and differential-mode components.Source: §5 · p. 6

5.1 CMC-DSQN

  • Because Qref is unavailable during training and residual error moves with SNN learning, CMC-DSQN estimates the common-mode Q component with an auxiliary ANN learned jointly through TD learning.The method estimates the common-mode Q-value component rather than directly predicting the moving residual error.Source: §5.1 · p. 6

5.2 UNDERSTANDING COMMON-MODE COMPENSATION

  • CMC-DSQN substantially improves learning stability over the vanilla SNN, enables nearly all seeds to solve CliffWalking, and markedly reduces both common- and differential-mode Q-value errors.Figure 4 reports learning dynamics in panels (a) and (b), and common- and differential-mode RMSEs against the exact optimal Q-function in panel (c).Source: Figure 4 · p. 7, §5.2 · p. 7

5.1 CMC-DSQN

  • The auxiliary ANN does not change greedy action selection, so it is removed at inference and actions use the original SNN outputs without additional computational overhead.The paper states this preserves SNN energy efficiency.Source: §5.1 · p. 7

5.2 UNDERSTANDING COMMON-MODE COMPENSATION

  • CMC-DSQN achieves lower estimation errors than SNN-Q*, which bootstraps with exact optimal Q-values, by assigning common-mode error reduction to the ANN and differential-mode error reduction to the SNN.The paper says this division reduces the SNN’s representational burden and improves estimation of both error modes.Source: §5.2 · p. 7

6 EXPERIMENTS

The experiments evaluate CMC-DSQN against ANN-to-SNN conversion and directly trained DSQN baselines on Atari and MiniAtar, then test whether its gains depend on the compensation design. CMC-DSQN improves low-timestep performance, while ablations favor ANN-based common-mode compensation over the tested alternatives.

6.1 EXPERIMENTAL SETUP

  • The evaluation covers seven Atari games and all five MiniAtar environments, using T ∈ {2, 3, 4} for Atari and T ∈ {2, 4, 6} for MiniAtar.Models train for 30M Atari or 5M MiniAtar environment transitions; results average three or five seeds, respectively. Baselines include conversion and directly trained DSQN methods; APR of 100% corresponds to the ANN DQN average.Source: §6.1 · pp. 7–8

6.2 MAIN RESULTS

  • On Atari, CMC-DSQN improves aggregated performance across seven games, raising APR by more than 16 percentage points over vanilla DSQN at T = 2.The advantage persists as the simulation horizon increases.Source: Table 1 · p. 8, §6.2 · p. 9
  • On MiniAtar, CMC-DSQN reaches an APR of 90.30% at T = 2, nearly 20 percentage points above CaRe-BN, the strongest competing DSQN baseline.The comparison includes recent directly trained DSQNs and ANN-to-SNN conversion methods.Source: Table 2 · p. 8, §6.2 · p. 9
  • At T = 4, CMC-DSQN exceeds the ANN baseline in average normalized performance on both Atari and MiniAtar.The Atari and MiniAtar evaluations use Tables 1 and 2, respectively.Source: Table 1 · p. 8, Table 2 · p. 8, §6.2 · p. 9

6.3 ABLATION STUDIES

  • Replacing the auxiliary ANN with an SNN yields S-CMC, which consistently underperforms CMC-DSQN across tested timesteps.This comparison tests whether the gain comes simply from adding trainable parameters.Source: §6.3 · p. 9, Figure 5 · p. 9
  • QEC-DSQN, which compensates Q-value error rather than only its common-mode component, improves over vanilla DSQN but underperforms CMC-DSQN.This ablation supports the design choice to compensate only common-mode errors.Source: §6.3 · p. 9, Figure 5 · p. 9
  • Applying the compensation mechanism to ANN-based DQN produces little additional benefit, with CMC-DQN closely matching vanilla DQN.The reported comparison is consistent with common-mode error dominance in low-timestep SNNs but not ANNs.Source: §6.3 · p. 9, Figure 5 · p. 9

7 CONCLUSION

Low-timestep DSQNs are limited by dominant common-mode Q-value errors that affect TD bootstrapping more strongly than differential-mode errors. CMC-DSQN uses an auxiliary ANN to estimate the common-mode component during training, then removes it at inference; on Atari and MiniAtar it outperforms DSQN baselines at low timesteps and exceeds the ANN baseline at T = 4.

7 CONCLUSION

  • Low-timestep DSQNs are dominated by common-mode errors, which affect TD bootstrapping more strongly than differential-mode errors.CMC-DSQN replaces the SNN common-mode component with an auxiliary ANN estimate that can be removed at inference.Source: §7 · p. 9
  • On Atari and MiniAtar, CMC-DSQN substantially outperforms state-of-the-art DSQN baselines at low timesteps and surpasses the ANN baseline at T = 4.The auxiliary ANN is removed at inference, preserving the SNN energy-efficiency advantages.Source: §7 · p. 9
  • CMC-DSQN targets the Q-value estimation errors identified as a source of low-timestep DSQN degradation.The paper frames its method around common-mode errors and their stronger effect on TD bootstrapping.Source: §7 · p. 9

Related papers

Loading 2610.07808v1…