PaperlayerView exact revision on arXiv

Source-linked AI summary

Object Detection with Spiking Neural Networks on Automotive Event Data

Loïc Cordone, Benoît Miramond, Philippe Thierion

arXiv:2205.04339v1cs.CV

TL;DR

Automotive event data is sparse, asynchronous, and temporally rich, while prior SNN performance was insufficient for complex real-world detection. The paper introduces voxel-cube encoding and trains four spiking CNN families with modern surrogate-gradient methods. These models achieve new SNN classification results and enable SSD-based spiking detection on GEN1 Automotive Detection.

  • Problem

    Event-based object detection remained challenging because prior solutions did not fully exploit event-data properties or suit power-constrained deployment.

  • Method

    The paper combines voxel-cube encoding, surrogate-gradient training, PLIF neurons, and four spiking CNN architectures for classification and SSD-based detection.

  • Results

    The models set new state-of-the-art SNN classification results on two automotive event datasets and achieve 0.19mAP with less than 10M parameters for GEN1 detection.

  • Takeaways & Limitations

    Voxel-cube encoding enables performant automotive SNNs without requiring a high number of timesteps, while the resulting models support real-world event-based object detection.

Abstract

from arXiv · show

Automotive embedded algorithms have very high constraints in terms of latency, accuracy and power consumption. In this work, we propose to train spiking neural networks (SNNs) directly on data coming from event cameras to design fast and efficient automotive embedded applications. Indeed, SNNs are more biologically realistic neural networks where neurons communicate using discrete and asynchronous spikes, a naturally energy-efficient and hardware friendly operating mode. Event data, which are binary and sparse in space and time, are therefore the ideal input for spiking neural networks. But to date, their performance was insufficient for automotive real-world problems, such as detecting complex objects in an uncontrolled environment. To address this issue, we took advantage of the latest advancements in matter of spike backpropagation - surrogate gradient learning, parametric LIF, SpikingJelly framework - and of our new \textit{voxel cube} event encoding to train 4 different SNNs based on popular deep learning networks: SqueezeNet, VGG, MobileNet, and DenseNet. As a result, we managed to increase the size and the complexity of SNNs usually considered in the literature. In this paper, we conducted experiments on two automotive event datasets, establishing new state-of-the-art classification results for spiking neural networks. Based on these results, we combined our SNNs with SSD to propose the first spiking neural networks capable of performing object detection on the complex GEN1 Automotive Detection event dataset.

I. INTRODUCTION

Event cameras provide sparse, asynchronous data with precise timing, but this format is difficult for classical frame-based neural networks. The paper addresses this mismatch by combining spiking networks with voxel-cube encoding for automotive event processing.

  • Event cameras detect brightness changes and output binary events containing position, precise time, and polarity.
  • Spiking neural networks communicate through discrete binary spikes, making them naturally suited to sparse event data.
  • Specialized hardware has shown spiking neural networks consuming 50% less energy than traditional neural networks at the same accuracy.
  • The paper trains state-of-the-art spiking neural networks on real-world automotive event data for classification and object detection.
  • The work introduces voxel cube encoding, a new classification dataset, four spiking architectures, and SSD-based spiking object detectors.
  • Voxel cubes combine binary voxel grids with event cubes to preserve temporal information using few timesteps.

B. Spiking Neural Networks

Spiking neural networks model computation with temporal binary spikes, but their non-differentiability complicates training. Recent surrogate-gradient methods and frameworks enable deeper SNNs, while event-based object detection remains difficult for power-constrained deployment.

  • SNN neurons communicate through binary spikes, but their discrete outputs prevent direct use of standard backpropagation.
  • STDP modifies synaptic weights according to presynaptic and postsynaptic firing delays but has not proved effective on complex object-detection tasks.
  • Surrogate-gradient learning approximates the gradient of the non-differentiable spike function during backpropagation.
  • SpikingJelly and related frameworks support GPU-accelerated training of deeper SNNs and learnable neuron parameters such as PLIF time constants.
  • Event-based object detection remains challenging because existing approaches use dense or hybrid networks that do not fully exploit event sparsity and are difficult to embed under power constraints.

III. METHOD

The method converts microsecond-resolution events into binary voxel cubes. By moving fine temporal information into channels, it reduces the SNN timestep count while retaining temporal precision.

  • Events are accumulated over windows of ∆t seconds to form voxel grids with T timesteps in a 4D CTHW tensor.
  • Using many timesteps preserves temporal resolution but increases SNN computation, inference time, and energy consumption linearly.
  • Voxel cubes subdivide each ∆t window into n micro time bins and store those bins in the channel dimension.
  • Each voxel-cube channel records one polarity and micro-bin, with binary accumulation when multiple events fall in the same bin.
  • Transferring temporal information to channels reduces T without losing temporal precision relative to voxel grids.

B. Spiking Neural Networks models

The paper builds spiking versions of four CNN architectures using PLIF neurons and a spike-compatible classifier. Their voxel-cube input preserves temporal information while the models process event data over compact timestep sequences.

  • The models use strided convolutions, max pooling, batch normalization, and PLIF neurons as their core operations.
  • Voxel-cube encoding preserves temporal information through the channel dimension while using a limited number of timesteps.
  • The spiking classifier uses batch normalization, a 1 × 1 convolution, and PLIF neurons, then sums output spikes spatially and temporally.
  • Spiking VGG, SqueezeNet, MobileNet, and DenseNet replace ReLU activations with PLIF neurons and use the proposed spiking classifier.
  • Spiking VGG retains the VGG architecture while adding batch normalization before each spiking convolutional layer.
  • Spiking SqueezeNet replaces its convolutional layers with spiking equivalents while retaining its low-parameter design.

2) Spiking SqueezeNet:

The paper adapts popular CNN architectures for spiking processing and combines spiking backbones with SSD for object detection. Its SSD design aggregates multiscale features over timesteps and uses focal loss to address class imbalance.

  • 3) Spiking MobileNet:: MobileNet uses depthwise separable convolutions to reduce parameters and computations, but the spiking version removes the intermediate activation and reorders batch normalization.The resulting pointwise-convolution inputs are not spikes, while depthwise separable convolutions remain equivalent to normal convolutions with specific weights during training.
  • 4) Spiking DenseNet:: DenseNet preserves spike representations through channel-wise concatenations, so replacing ReLU activations with PLIF neurons supports deep spiking networks.The authors vary depth and growth rate to obtain different Spiking DenseNet versions.
  • Object detection architecture: SSD uses a backbone and multiscale predictor heads that output bounding boxes and classes, with the CNN backbone replaced by an SNN backbone and spiking extra layers.The extra layers use spiking convolutions, completing the spiking object-detection network.
  • Object detection architecture: Predicted bounding boxes and classes are summed over T timesteps, while post-processing remains outside the spiking-network scope on conventional hardware.This separates temporal accumulation inside the SNN from prediction filtering after inference.
  • Training: Focal loss replaces SSD hard-negative mining because the large number of background predictions creates class imbalance in one-shot detection.The authors report that the original hard-negative-mining approach did not achieve satisfying performance.
  • Object detection architecture: The architecture uses three extra convolutional blocks, smaller feature maps, and anchors with ratios from 0.5 to 0.8 to account for smaller objects.The three blocks progressively reduce feature-map size, and similar designs are used for spiking MobileNet + SSD and VGG + SSD.

IV. EXPERIMENTS

The experiments evaluate spiking networks on two automotive classification datasets and the GEN1 automotive detection dataset. The datasets use event recordings captured from cameras mounted on moving vehicles.

  • Datasets: The evaluation covers Prophesee NCARS, a new GEN1 Automotive Classification dataset, and the Prophesee GEN1 Automotive Detection dataset.Classification and detection experiments therefore use related automotive event-camera data at different task levels.
  • Prophesee NCARS: Prophesee NCARS contains 24k 100-ms samples captured by a GEN1 event camera mounted behind a moving car’s windshield.Samples represent either a car or background and are cropped from recordings with 304 × 240-pixel resolution.
  • GEN1 Automotive Detection: The Prophesee GEN1 Automotive Detection dataset contains 39 hours of recordings and over 255k manually annotated bounding boxes for cars and pedestrians.It was recorded with a GEN1 sensor mounted on a car dashboard.

3) Prophesee GEN1 Classification dataset:

The GEN1 classification dataset is generated by cropping annotated car and pedestrian boxes from the detection dataset. The study balances its classes and uses short binary voxel-cube inputs for training.

  • 3) Prophesee GEN1 Classification dataset:: Each GEN1 classification sample is a crop of a car or pedestrian bounding box containing the preceding 100 ms of events.Unlike NCARS, the dataset includes both car and pedestrian classes.
  • 3) Prophesee GEN1 Classification dataset:: The training set rebalances classes by undersampling cars and horizontally flipping oversampled pedestrian examples.This procedure is intended to avoid imbalance in the number of samples for each class.
  • Encoding: Both classification and detection samples are encoded as binary voxel cubes using 5 timesteps and 2 micro time bins.The authors selected this configuration as the best compromise between performance and number of operations.
  • Training: Classification models use AdamW with weight decay 1e−4, while object-detection models use an initial learning rate of 1e−3 and train for 200 epochs.Classification models train for 10 epochs; learning rates are scheduled with cosine annealing toward 0.
  • Training: Training uses SpikingJelly with 16-bit automatic mixed precision on an NVIDIA RTX A6000 and a 104-thread Intel Xeon Gold 6230R.The reported hardware provides a 48-GB GPU and a 104-thread CPU.

3) Performance metrics:

The evaluation combines task accuracy with measures of model size, operation count, and sparsity. Results show a trade-off between classification performance and resource requirements, while all tested SNNs remain sparse over five timesteps.

  • 3) Performance metrics:: Classification is measured with accuracy, while object detection is measured with COCO mAP over IoU thresholds [.50:.05:.95].COCO mAP is reported as the mean Average Precision over ten IoU thresholds.
  • 3) Performance metrics:: The reported efficiency metrics include parameter count, accumulation operations per timestep, and network sparsity.These metrics capture memory requirements, operation counts, and the fraction of activity relative to a dense equivalent.
  • 3) Performance metrics:: ACCs quantify SNN operations because spiking convolutions and PLIF updates use accumulation operations rather than multiplicative operations.Batch-normalization ACCs are excluded because they can be fused with convolutional layers.
  • Results: On NCARS, all reported SNN models outperform previous spiking results, while DenseNet, MobileNet, and VGG exceed 90% test accuracy.The smallest spiking SqueezeNet struggles to exceed 80% test accuracy.
  • Results: VGG provides the best accuracy on both classification datasets, whereas DenseNet offers competitive accuracy with few parameters and moderate ACCs per timestep.VGG has many parameters, while MobileNet is penalized by replacing depthwise separable convolutions with normal convolutions.
  • Efficiency: All SNNs have sparsity below 40% on both datasets, and with five timesteps require at most twice the operations of an equivalent dense ANN.The authors note that specialized hardware could consume less power because these operations are simple ACCs.

D. Automotive Object Detection

The paper combines spiking backbones with SSD for object detection on the Prophesee GEN1 event dataset, while studying timestep and encoding choices for accuracy and computation.

  • Automotive Object Detection: Spiking object-detection models use VGG-11, MobileNet-64, and DenseNet121-24 as pretrained backbones evaluated on filtered GEN1 boxes.Boxes with diagonals smaller than 30 pixels were filtered, and the backbones were pretrained on NCARS.
  • Automotive Object Detection: 0.19 COCO mAP is achieved by DenseNet121-24 + SSD with 8.2M parameters and 2.33G ACCs per timestep.The spiking models outperform a traditional neural network with over five times more parameters.
  • Timesteps and Encoding: SNN computation increases linearly with timesteps, making low-timestep voxel-cube encodings important for limiting computational cost.The voxel-cube representation preserves temporal information while minimizing the number of timesteps.
  • Timesteps and Encoding: The best NCARS results were almost always obtained with 10 timesteps, whereas one timestep performed significantly worse even with many micro time bins.Increasing micro time bins did not consistently improve results, although it helped when timestep counts were low.
  • Timesteps and Encoding: For 100ms samples, 5 timesteps and 2 micro time bins provided the best accuracy–encoding compromise used for all models except SqueezeNet.SqueezeNet was unable to learn with one or two timesteps.

B. Influence of Batch Normalization and PLIF neurons

Batch normalization and parametric LIF neurons materially improve the training of complex spiking networks, with normalization placement also affecting performance.

  • Batch Normalization: Removing batch normalization made networks significantly less performant or unable to learn, while placing it before convolutions produced better results.The placement effect is described as more consequential for these SNNs than for DNNs.
  • Batch Normalization: Batch normalization before convolutions can transform sparse spike feature maps into dense decimal representations during training.This allows convolution weights to receive meaningful updates even when they did not receive spikes.
  • Batch Normalization: Batch-normalization parameters can be fused with subsequent convolution parameters, allowing normalization during SNN training without preventing compatible inference.The authors connect this design to training larger and more complex convolutional SNNs.
  • PLIF Neurons: Replacing PLIF neurons with LIF neurons using τ = 2 reduced accuracy across all networks.PLIF neurons learn different time constants for each layer and reduce the number of hyperparameters requiring tuning.

C. Influence of depthwise separable convolutions

Depthwise separable convolutions make larger spiking MobileNets easier to train and more accurate, but their benefit depends on model size.

  • Depthwise Separable Convolutions: Spiking MobileNets with 32 or 64 input channels achieved higher accuracy with depthwise separable convolutions than with normal convolutions.The smaller parameter count also made training faster.
  • Depthwise Separable Convolutions: For normal-convolution MobileNets, adding parameters could reduce accuracy, whereas accuracy increased for MobileNets trained with depthwise separable convolutions.The authors presume that larger normal convolutions make surrogate-gradient training more difficult.
  • Depthwise Separable Convolutions: The smaller MobileNet variant was 5% more accurate with normal convolutions than with depthwise separable convolutions.The authors therefore recommend depthwise separable convolutions as a second resort when normal-convolution accuracy decreases as networks grow.
  • Overall Outcome: The paper reports new state-of-the-art spiking classification results and 0.19 mAP object detection with fewer than 10M parameters.These results use four architectures and SSD on automotive event data.
Loading 2205.04339v1…