PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Directly-Trained Spiking Neural Networks for Object Detection

Qiaoyi Su, Yuhong Chou, Yifan Hu, Jianing Li, Shijie Mei, Ziyang Zhang, Guoqi Li

arXiv:2307.11411v3cs.CVcs.AI

TL;DR

Object detection remains challenging for directly trained SNNs because it is a regression task requiring accurate bounding-box values. The paper introduces EMS-YOLO and its full-spike EMS-ResNet, reporting comparable same-architecture ANN performance with substantially lower energy consumption on frame- and event-based datasets.

  • Problem

    Directly trained SNNs lack an established solution for regression-based object detection, while converted SNNs depend on ANNs and perform poorly on sparse event data.

  • Method

    EMS-YOLO directly trains a deep SNN with surrogate gradients and uses a full-spike EMS-ResNet designed for deep, energy-efficient object detection.

  • Results

    5.83× lower energy consumption is reported with comparable performance to the same-architecture ANN on COCO and Gen1.

  • Takeaways & Limitations

    EMS-YOLO demonstrates directly trained SNN object detection in only 4 time steps across static-image and event-data settings.

  • Takeaways & Limitations

    The energy comparison assumes 32-bit floating-point operations implemented in 45nm technology, with fixed MAC and accumulation energy costs.

Abstract

from arXiv · show

Spiking neural networks (SNNs) are brain-inspired energy-efficient models that encode information in spatiotemporal dynamics. Recently, deep SNNs trained directly have shown great success in achieving high performance on classification tasks with very few time steps. However, how to design a directly-trained SNN for the regression task of object detection still remains a challenging problem. To address this problem, we propose EMS-YOLO, a novel directly-trained SNN framework for object detection, which is the first trial to train a deep SNN with surrogate gradients for object detection rather than ANN-SNN conversion strategies. Specifically, we design a full-spike residual block, EMS-ResNet, which can effectively extend the depth of the directly-trained SNN with low power consumption. Furthermore, we theoretically analyze and prove the EMS-ResNet could avoid gradient vanishing or exploding. The results demonstrate that our approach outperforms the state-of-the-art ANN-SNN conversion methods (at least 500 time steps) in extremely fewer time steps (only 4 time steps). It is shown that our model could achieve comparable performance to the ANN with the same architecture while consuming 5.83 times less energy on the frame-based COCO Dataset and the event-based Gen1 Dataset.

1. Introduction

Object detection is a challenging, energy-intensive vision task, while existing SNN approaches face long time horizons and depth-related training constraints. EMS-YOLO addresses these issues with directly trained, full-spike architecture and achieves comparable ANN performance with lower energy use.

  • Object detection requires recognizing multiple overlapping objects and precisely locating their bounding boxes.
  • ANN-based detectors provide high performance but incur high computational complexity and energy consumption.SNNs use binary spikes, asynchronous computation, and event-driven communication to reduce overhead.
  • Converted SNN detectors can require hundreds or thousands of time steps and may not capture sparse event-camera information effectively.Spiking-Yolo needs at least 3500 time steps, while Spike Calibration reduces this to hundreds but remains dependent on the original ANN.
  • Multi-scale object features require deep representation capacity, but existing detectors are limited to shallow or hybrid structures.Non-spiking components in hybrid designs may also prevent deployment on hardware allowing only spike operations.
  • EMS-YOLO directly trains a deep SNN with surrogate gradients and uses the full-spike EMS-ResNet to reduce redundant MAC operations.The design targets both static images and event data while avoiding gradient disappearance or explosion.
  • 5.83× lower energy consumption accompanies comparable performance to an ANN with the same architecture on COCO and Gen1.EMS-YOLO also requires only 4 time steps and outperforms advanced ANN-SNN conversion methods according to the reported contribution.

2. Related Work

Prior SNN object detectors primarily convert ANNs, creating latency, conversion-dependence, and event-data limitations. Directly trained and hybrid approaches improve flexibility, but deep full-spike detection remains constrained by residual-block design and energy costs.

  • ANN-SNN conversion approximates continuous ANN activations with average SNN firing rates, making converted performance dependent on the source ANN.
  • Converted SNNs can suffer performance loss at low time delay and work poorly on sparse event data.
  • Surrogate-gradient training enabled directly trained SNNs to operate at short time steps and lower power, while extending depth beyond earlier shallow models.TDBN extended directly trained SNNs from fewer than 10 layers to 50 layers.
  • Object-detection systems include frame-based and event-based sensors, with mainstream YOLO, SSD, Transformer, and RCNN detectors implemented using ANNs.
  • Early converted detectors required long inference times and could not support event-camera data, while hybrid models added ANN detection heads and parameters.
  • Existing directly trained residual designs address deep trainability unevenly, and non-spike shortcut convolutions can impose heavy energy consumption for varying feature dimensions and channels.

3. The Preliminaries of SNNs

The preliminaries describe spiking-neuron dynamics, surrogate-gradient and normalization tools, and the operation-based energy model used for SNN analysis. These foundations define how spikes are generated, trained, and compared with ANN computation.

  • 3.1. Spiking Neuron: LIF neurons model membrane-potential dynamics and emit biologically inspired spike signals, unlike ANN neurons that propagate only spatial activations.
  • 3.1. Spiking Neuron: The membrane potential integrates weighted presynaptic spikes with leakage controlled by the decay factor τ.
  • 3.1. Spiking Neuron: The Heaviside threshold controls firing, and the membrane potential resets to Vrest after a spike.
  • 3.2. Training Strategies: Surrogate gradients replace the nondifferentiable spike operation during back-propagation.The parameter a controls gradient-curve steepness while ensuring the gradient integral is 1.
  • 3.2. Training Strategies: TDBN normalizes sequential inputs across spatial and temporal domains using channel statistics, trainable parameters, and a threshold-dependent hyper-parameter.
  • 3.3. Energy Consumption: SNN energy is estimated from accumulation and multiply-accumulate operations, incorporating time steps, firing rate, and hardware energy costs.The comparison assumes 32-bit floating-point operations in 45nm technology, with EMAC = 4.6pJ and EAC = 0.9pJ.

4. Method

EMS-YOLO is a directly trained SNN for object detection that combines a YOLO-style detector with full-spike EMS-ResNet blocks. Its design targets efficient multi-scale feature extraction, bounding-box regression, and stable deep training.

  • Input Representation: Static images are reused at each time step, while event streams are split into temporal bins and processed as fixed time steps.Event inputs encode changes in light as asynchronous events with spatial coordinates, timestamps, and polarity.
  • EMS-ResNet: EMS-ResNet keeps shortcut information spiking by adding an LIF before convolution when channel numbers change and using separate blocks for channel transitions.EMS-Block1 handles constant or decreasing channels, whereas EMS-Block2 handles increasing channels through concatenation.
  • The EMS-YOLO Model: EMS-YOLO uses a backbone and detection heads composed mainly of EMS-Blocks to extract multi-scale object features.The first convolution converts inputs into spikes, and EMS-Modules can vary in block count and channel width for the task.
  • The EMS-YOLO Model: The detection head replaces multiple directly connected convolutions with EMS-Blocks to avoid performance loss from conventional multi-layer connections.The design uses a YOLOv3-tiny detection head as an example.
  • The EMS-YOLO Model: Last membrane potentials are fed into detectors to produce anchors, followed by non-maximum suppression for final classes and bounding boxes.Each prediction contains box coordinates, dimensions, class, and confidence values.
  • Analysis of Gradient Vanishing/Explosion Problems: Gradient Norm Equality analysis indicates that EMS-ResNet can avoid vanishing or exploding gradients during deep training.The analysis connects normalized Jacobian behavior with structural placement of EMS-Blocks and control of the input second moment.

5. Experiments

Experiments on COCO2017 and Gen1 evaluate EMS-YOLO using mAP, comparisons with ANN-based methods, and ablations of residual blocks, depth, and time steps. The results show strong detection performance with few steps and reduced energy consumption.

  • COCO2017 Dataset: EMS-YOLO achieved comparable performance to the same-structure ANN on COCO2017 while reducing energy consumption by 5.83×.The comparison used EMS-ResNet34 and the same experimental settings and data augmentation.
  • COCO2017 Dataset: At 4 time steps, directly trained EMS-YOLO outperformed ANN-SNN conversion methods that required at least several hundred time steps.
  • GEN1 Automotive Detection Dataset: On Gen1, EMS-YOLO reached mAP@0.5:0.95 of 0.267 versus 0.189 for the compared SNN method at the same number of time steps.The EMS-ResNet10 model used 6.20 M parameters and achieved a 21.15% spike firing rate.
  • Different Residual Blocks: EMS-Res18 reduced energy consumption by up to 4.91× relative to the ANN-ResNet18 baseline while maintaining comparable performance to other residual blocks.The baseline energy consumption was approximately 9.65mJ, defined as 1×.
  • Depth and Time-Step Ablations: Increasing the number of residual blocks strengthened feature extraction, while longer time steps produced higher object-detection accuracy in the reported ablations.The depth comparison used EMS-Res10, EMS-Res18, and EMS-Res34; time-step experiments included T = 1, 3, 5, and 7.

6. Conclusion

The paper introduces EMS-YOLO as the first deep directly trained SNN for object detection, using a full-spike EMS-ResNet to support efficient multi-scale feature extraction. The model achieves comparable performance to same-structure ANNs with short time steps on both static images and event data.

  • EMS-YOLO is presented as the first deep directly trained SNN for object detection.
  • EMS-ResNet is a full-spike residual block designed to eliminate redundant MAC operations from non-spike shortcut paths and residual connections.The design is intended to improve energy efficiency and neuromorphic-chip deployment.
  • EMS-YOLO achieves comparable performance to a same-structure ANN with very short time steps on both static images and event data.

A. Proof of Gradient Norm Equality

The analysis characterizes EMS-ResNet blocks through Jacobian moments and shows that suitable initialization can maintain gradient norms near equality. This provides a theoretical basis for training the architecture deeply without gradient disappearance or explosion.

  • Jacobian decomposition: The block Jacobian is represented using general linear transforms and matrix-moment principles.The analysis invokes general linear transforms, multiplication, and conditions on independent random matrices to characterize composed blocks.
  • Jacobian decomposition: EMS-Block1 and EMS-Block2 are analyzed as combinations of residual and shortcut paths with corresponding Jacobian matrices.The residual and shortcut paths are treated separately, including concatenation and maxpooling effects in EMS-Block2.
  • Gradient norm control: Proper initialization of batch-normalization layers controls the relevant moments in the shortcut path.The derivation relates the maxpooling and concatenation terms to channel counts and BN-layer initialization.
  • Gradient norm control: Controlling the input's second moment can satisfy ϕ(JJT) ≈1 for EMS-ResNet.The same condition is stated for the full EMS-ResNet after combining the block-level analysis.

B. Datasets Introduction

The paper evaluates object detection on COCO2017, a large frame-based dataset, and Gen1, a large event-camera automotive dataset. Together, the datasets cover conventional images and event data across different object-detection settings.

  • COCO2017 Dataset: COCO2017 provides 118K training images, 5K test images, and bounding-box labels for 80 object categories.It also supports segmentation, key-point detection, and captioning tasks.
  • GEN1 Automotive Detection Dataset: An event encodes a change in pixel light intensity over a temporal sampling interval.The event representation uses polarity and a contrast threshold in the logarithmic intensity relation.
  • GEN1 Automotive Detection Dataset: Gen1 records 39 hours of automotive scenes across pedestrian and car categories.The event-camera dataset is designed for automotive detection in diverse scenarios.
  • GEN1 Automotive Detection Dataset: Gen1 contains more than 255,000 bounding-box annotations at 304×204-pixel resolution.The annotations are produced from gray-level estimation by the ATIS sensor.

C. More Detailed Experiments

Additional experiments examine training settings, residual-block design, network depth, and detection-layer spiking on COCO2017. The ablations report energy savings for full-spike EMS-ResNet and performance changes associated with depth and detection-layer choice.

  • Experimental setup: The detailed COCO2017 experiments use channel-reduction models to test conclusion reliability rather than optimize peak performance.Models are trained with four Nvidia A100 GPUs, SGD, and a learning rate of 1E-2.
  • Different Residual Blocks: 5.98 times lower energy consumption is achieved by the full-spike EMS-ResNet in the residual-block ablation.The comparison is reported for the COCO2017 experiments with three time steps.
  • Numbers of Residual Blocks: Increasing the number of residual blocks makes the network converge faster and recognize objects more accurately at one time step.This depth experiment uses only 50 training epochs on COCO2017.
  • Spiking Detection layer: Spiking and non-spiking detection layers provide two ways to convert spike features into continuous bounding-box representations.The reported results indicate little performance difference between the two conversion methods.
Loading 2307.11411v3…