PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation

Youngeun Kim, Joshua Chough, Priyadarshini Panda

arXiv:2110.07742v1cs.CV

TL;DR

SNNs had largely been optimized for image recognition, leaving their use in semantic segmentation underexplored, despite segmentation’s relevance to scene understanding. This paper directly trains spiking FCN and DeepLab networks with surrogate gradients and evaluates them on static and event-based benchmarks. The authors report lower latency and better performance than ANN-SNN conversion, along with robustness and energy-efficiency advantages in parts of the evaluation.

  • Problem

    Most SNN optimization methods focused on image recognition, while semantic segmentation with spiking neurons had not been studied.

    Source: §I · pp. 1–2

  • Method

    The paper directly trains spiking versions of FCN and DeepLab using surrogate-gradient backpropagation with temporal batch normalization.

    Source: §I · p. 2

  • Results

    Surrogate-gradient learning achieves better performance with lower latency than ANN-SNN conversion on semantic segmentation.

    Source: §I · p. 2

  • Takeaways & Limitations

    The experiments show the feasibility of spiking semantic segmentation on static PASCAL VOC2012 and event-based DDD17, with reported robustness and energy-efficiency advantages over ANNs.

    Source: §I · p. 2, C. Analysis on Robustness · p. 9, D. Analysis on Energy-efficiency · p. 9

Abstract

from arXiv · show

Spiking Neural Networks (SNNs) have recently emerged as the low-power alternative to Artificial Neural Networks (ANNs) because of their sparse, asynchronous, and binary event-driven processing. Due to their energy efficiency, SNNs have a high possibility of being deployed for real-world, resource-constrained systems such as autonomous vehicles and drones. However, owing to their non-differentiable and complex neuronal dynamics, most previous SNN optimization methods have been limited to image recognition. In this paper, we explore the SNN applications beyond classification and present semantic segmentation networks configured with spiking neurons. Specifically, we first investigate two representative SNN optimization techniques for recognition tasks (i.e., ANN-SNN conversion and surrogate gradient learning) on semantic segmentation datasets. We observe that, when converted from ANNs, SNNs suffer from high latency and low performance due to the spatial variance of features. Therefore, we directly train networks with surrogate gradient learning, resulting in lower latency and higher performance than ANN-SNN conversion. Moreover, we redesign two fundamental ANN segmentation architectures (i.e., Fully Convolutional Networks and DeepLab) for the SNN domain. We conduct experiments on two public semantic segmentation benchmarks including the PASCAL VOC2012 dataset and the DDD17 event-based dataset. In addition to showing the feasibility of SNNs for semantic segmentation, we show that SNNs can be more robust and energy-efficient compared to their ANN counterparts in this domain.

I. INTRODUCTION

This paper extends spiking neural networks to semantic segmentation, where each pixel receives a class label. It compares conversion with direct surrogate-gradient training and introduces spiking versions of FCN and DeepLab.

I. INTRODUCTION

  • ANN-SNN conversion performs poorly on segmentation because spatial and channel activation variance is high, and it cannot directly use DVS spike-stream data.The paper notes that conversion can require thousands of time-steps to approach the pretrained ANN’s performance.Source: §I · pp. 1–2
  • Direct surrogate-gradient learning trains SNNs from input spikes by approximating the LIF neuron's backward gradient and optimizing spatial cross-entropy loss.Batch Normalization Through Time enables training the segmentation networks from scratch.Source: §I · p. 2
  • Surrogate-gradient training of spiking segmentation networks achieves better performance with lower latency than ANN-SNN conversion.The paper presents Spiking-FCN and Spiking-DeepLab and evaluates them on PASCAL VOC2012 and DDD17.Source: §I · p. 2
  • The paper introduces Spiking-FCN and Spiking-DeepLab as spiking adaptations of two representative ANN segmentation architectures.FCN uses an encoder-decoder with upsampling, while DeepLab uses dilated convolutions to cover wide receptive fields without computational overhead.Source: §I · p. 2
  • The reported experiments cover static PASCAL VOC2012 and event-based DDD17, and the spiking architectures are reported to provide more than 2× energy efficiency than standard ANNs.The paper motivates segmentation as pixel-wise scene understanding relevant to autonomous vehicles.Source: §I · pp. 1–2

II. RELATED WORK

Prior SNN work addresses non-differentiable neuron dynamics with several training strategies, but most methods target basic vision tasks. In ANN segmentation, FCN and DeepLab are representative architectures; semantic segmentation had not yet been studied with spiking neurons.

A. Spiking Neural Network

  • SNN training approaches include STDP, ANN-SNN conversion, and surrogate-gradient learning, while most optimization methods focus on basic vision tasks rather than semantic segmentation.STDP is limited to small-scale tasks by its locality; conversion can require many time-steps, while surrogate gradients support small latency and reasonable performance.Source: A. Spiking Neural Network · p. 3

B. Semantic Segmentation

  • ANN segmentation architectures preserve detail through FCN decoder skip information and use DeepLab dilated convolutions to enlarge receptive fields without added computational cost.U-Net, deconvolution networks, and RefineNet are other described approaches to feature detail or multi-scale input.Source: B. Semantic Segmentation · p. 3
  • Before this work, spiking-neuron semantic segmentation was unstudied; prior SNN region-segmentation work did not provide class information.The paper configures spiking versions of FCN and DeepLab to address that gap.Source: B. Semantic Segmentation · p. 3

III. PRELIMINARY STUDY

The preliminary study finds that ANN-SNN conversion performs poorly for semantic segmentation, while the LIF neuron model supplies the temporal dynamics used by spiking networks.

III. PRELIMINARY STUDY

  • ANN-SNN conversion substantially degrades performance from a pretrained ANN even with thousands of time-steps, as spatial and channel activation variation defeats proper scaling.The same balancing factor across layers or channels does not preserve delicate activations; conversion also cannot handle DVS video spike streams.Source: §III · p. 3

A. Leaky Integrate-and-Fire (LIF) Neuron

  • The LIF membrane potential combines decayed voltage from prior time-steps with weighted presynaptic spikes, and a neuron fires when its potential exceeds threshold.The leak factor λ and connection weight w_ij govern decay and input contribution, respectively.Source: A. Leaky Integrate-and-Fire (LIF) Neuron · p. 4
  • After firing, the experiments use a soft reset that subtracts the threshold, retaining residual information and performing better than a hard reset.The paper links retained residual information to preventing information loss.Source: A. Leaky Integrate-and-Fire (LIF) Neuron · p. 4

B. Input Representation

Static images are converted to rate-coded Poisson spikes, while DVS event streams are accumulated into frames for network input.

B. Input Representation

  • The figure contrasts static-image Poisson coding, whose accumulated spikes approximate the source image, with event-camera spikes grouped into frames.The two inputs use different representations before entering the network.Source: Figure 5 · p. 4, B. Input Representation · pp. 4–5
  • For static images, rate coding generates spikes proportional to pixel intensity by comparing each pixel with a random value at every time-step.Accumulated spikes over time represent an image similar to the original.Source: B. Input Representation · p. 4, Figure 5 · p. 4
  • For DVS input, spikes are accumulated within a time window to form frames that the network processes as video.The example window in the figure caption is 10 ms.Source: Figure 5 · p. 4, B. Input Representation · p. 5

C. Spiking Segmentation Network Architecture

The paper adapts DeepLab and FCN segmentation designs to spiking neurons and trains them with spatial loss and surrogate-gradient backpropagation through time.

C. Spiking Segmentation Network Architecture

  • Spiking-DeepLab uses dilated convolution in its final two feature-extractor layers to enlarge receptive fields while preserving feature-map resolution.A 3×3 kernel with dilation stride 2 covers a local area of 25 rather than 9; the last feature-extraction block omits average pooling.Source: 1) Spiking-DeepLab · p. 5
  • Spiking-FCN downsamples to learn semantic features, then uses learned transposed convolutions and intermediate encoder features to recover input resolution.Convolutional layers match channel counts between the downsampling and upsampling features; all layers use LIF neurons and the network ends with a spike accumulator.Source: 2) Spiking Fully-Convolutional… · pp. 5–6

D. Overall Optimization

  • Both architectures produce a two-dimensional probabilistic output map from accumulated spikes and compute pixel-wise cross-entropy against ground-truth labels.The last-layer membrane potential supplies the output map, and N and C denote normalization factor and number of classes.Source: D. Overall Optimization · p. 6
  • Training uses surrogate backpropagation through time, accumulating gradients across time-steps and layers to update weights.The nondifferentiable hidden-layer spike gradient is approximated with a piecewise-linear function; the final classifier uses conventional backpropagation after spike accumulation.Source: D. Overall Optimization · p. 6

A. Experimental Setup

Experiments evaluate spiking segmentation architectures on static-image VOC2012 and event-based DDD17, using matched training hyperparameters.

A. Experimental Setup

  • Experiments use PASCAL VOC2012 and DDD17, with VOC2012 evaluated through rate coding and DDD17 represented by accumulated positive and negative events.VOC2012 is resized to 64×64; DDD17 uses 50 ms event intervals and is resized from 346×200 to 64×64.Source: 1) Dataset · p. 6
  • The VOC2012 split contains 10,582 training and 1,449 validation images, while the DDD17 segmentation split contains 15,950 training and 3,890 testing frames across six classes.DDD17 labels cover sequence intervals from six original sequences.Source: 1) Dataset · p. 6
  • The spiking backbones have seven layers; Spiking-DeepLab adds dilated layers, while Spiking-FCN uses transposed-convolution upsampling, and both use BNTT.The paper configures the corresponding ANN architectures for comparison.Source: 2) Parameters · p. 7
  • Both networks use Adam at learning rate 3e-3, batch size 16, 60 epochs, 20 time-steps, leak factor 0.99, and membrane threshold 1.0.The learning rate decays by a factor of 10 halfway through training.Source: 2) Parameters · p. 7

B. Performance Comparison

ANN references outperform the spiking segmentation models overall, while the relative performance of Spiking-FCN and Spiking-DeepLab differs between VOC2012 and DDD17.

B. Performance Comparison

  • ANN references achieve higher performance than SNNs on both datasets, while the authors frame the experiments as testing feasibility rather than matching ANN performance.The comparison uses Spiking-FCN and Spiking-DeepLab against reference ANN methods.Source: B. Performance Comparison · p. 7
  • On VOC2012, Spiking-FCN performs worse than Spiking-DeepLab, whereas the two architectures perform similarly on DDD17.The authors attribute the VOC2012 gap to FCN's greater depth and the DDD17 similarity to that dataset's lower difficulty and more structured data.Source: B. Performance Comparison · p. 7
  • The authors connect the deeper FCN's weaker VOC2012 result to the difficulty of training deeper SNN architectures.They note prior work reporting greater training challenges for deeper SNNs.Source: B. Performance Comparison · p. 7

C. Analysis on Robustness

The authors assess segmentation robustness by adding Gaussian noise and comparing the relative MIoU drop from clean inputs. SNNs are more robust than ANNs on VOC2012, but similarly robust on DDD17.

C. Analysis on Robustness

  • Figures 8 and 9 show image, prediction, and ground-truth triplets for qualitative results on the VOC2012 validation set and DDD17 test set.Figure 10 compares changes in Mean IoU across Gaussian-noise standard deviations.Source: Figure 8 · p. 8, Figure 9 · p. 8, Figure 10 · p. 9
  • The relative accuracy drop compares clean-sample MIoU with noisy-sample MIoU across Gaussian-noise levels.Lower relative drop indicates greater robustness.Source: C. Analysis on Robustness · p. 9
  • SNNs are more robust than ANNs on VOC2012, but show similar robustness to ANNs on DDD17.The authors conclude that spiking segmentation offers improved robustness with traditional-camera images, despite the DDD17 result.Source: C. Analysis on Robustness · p. 9, Figure 10 · p. 9
  • The authors attribute VOC2012 robustness partly to Poisson coding, which converts static pixel values into randomly distributed temporal spikes.The DDD17 models do not use Poisson coding and predict from multiple sequential frames, while the ANN receives one input per prediction; the authors identify these as factors in reduced DDD17 robustness.Source: C. Analysis on Robustness · p. 9

D. Analysis on Energy-efficiency

The authors estimate inference energy from operation counts and measured layer-wise spiking rates, finding higher energy efficiency for SNNs than ANNs on both benchmarks.

D. Analysis on Energy-efficiency

  • The energy estimate counts MAC-operation energy and neglects memory and peripheral-circuit energy.ANN MACs are modeled as one floating-point addition plus one multiplication, while event-driven SNN MACs reduce to one floating-point addition and are weighted by layer spiking rate.Source: D. Analysis on Energy-efficiency · p. 9
  • Layer-wise spike rates are measured for Spiking-DeepLab and Spiking-FCN on VOC2012 and DDD17 to estimate SNN operation energy.Figure 11 reports the rates across layers; FCN upsampling layers have high rates, which the authors attribute to skip connections injecting spikes during forward propagation.Source: D. Analysis on Energy-efficiency · p. 9, Figure 11 · p. 10
  • Across two benchmarks, SNNs show higher energy-efficiency than ANNs; on static VOC2012, their greater efficiency is attributed to generating fewer spikes across all layers.Source: D. Analysis on Energy-efficiency · p. 9
  • The energy comparison uses energy values for a 45nm CMOS process.Table IV gives the process energy values, and Table V is the comparison of segmentation-network energy efficiency.Source: D. Analysis on Energy-efficiency · p. 9, Table IV · p. 10

VI. CONCLUSION

The paper studies directly trained spiking networks for semantic segmentation and reports more than 2× energy efficiency on static images. It also identifies a remaining performance gap with ANNs.

VI. CONCLUSION

  • ANN-SNN conversion requires more than one thousand time-steps and gives inferior performance, while BNTT with approximated-gradient training produces reasonable results with fewer time-steps.The study evaluates static PASCAL VOC2012 images and event-stream DDD17 data.Source: §VI · p. 10
  • The authors construct Spiking-DeepLab and Spiking-FCN and evaluate their performance, robustness, and estimated energy on two segmentation scenarios.They characterize the work as a feasibility study of training SNNs for semantic segmentation.Source: §VI · p. 10, Table V · p. 10
  • Spiking segmentation networks bring more than 2× energy efficiency on the static image dataset.The conclusion identifies closing the SNN–ANN segmentation performance gap as future work.Source: §VI · p. 10

Related papers

Loading 2110.07742v1…