PaperlayerView exact revision on arXiv

Source-linked AI summary

SDPAD: A Fully Spike-Driven Pipeline for End-to-End Autonomous Driving

Chengjun Zhang, Yuhao Zhang, Jie Yang, Mohamad Sawan

arXiv:2610.11583v1cs.ROcs.AI

TL;DR

Dense ANN planners are accurate but computationally heavy, while SNN planners still lag in planning accuracy and face architectural bottlenecks. SDPAD uses integer-spike perception and query-based planning in a single feed-forward pass. It narrows the SNN–ANN gap across nuScenes and NAVSIM, with NAVSIM results comparable to mainstream ANN planners and better than the prior SNN planner SAD.

  • Problem

    Dense ANN planners impose heavy computation, while existing spiking planners lack planning-grade architectures for latency-efficient BEV lifting and selective aggregation of scene context.

    Source: §1 · pp. 1–2

  • Method

    SDPAD converts a pretrained ANN perception stack to integer spikes, performs spike-driven BEV lifting, and decodes trajectories with Spike-QFormer in a single feed-forward pass.

    Source: §1 · p. 2

  • Results

    SDPAD narrows the SNN–ANN planning gap across nuScenes and NAVSIM; on NAVSIM it reaches 86.3% PDMS and exceeds SAD by 4.3 PDMS points.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    The results demonstrate that spiking architectures can rival dense ANN planners in complex driving tasks at a small fraction of their energy cost.

    Source: §5 · p. 8

    Direct surrogate-gradient training and on-chip neuromorphic deployment remain future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

End-to-end autonomous driving demands trajectory planners that are both highly accurate and cheap enough for edge deployment. State-of-the-art artificial neural network (ANN) planners meet the accuracy requirement at the cost of heavy dense computation, while spiking neural networks (SNNs)---though promising orders-of-magnitude energy savings through sparse, event-driven arithmetic---still lag far behind in planning accuracy. We present \textbf{SDPAD}, a fully spike-driven end-to-end planning pipeline that closes this gap. SDPAD converts a pre-trained ANN perception stack into integer-spike form via quantized ANN2SNN conversion, lifts multi-view images into the bird's-eye-view (BEV) space with a spike-driven-max (SDM) depth distribution (Spike-3D-Lift), and plans through the Spike-QFormer, a spiking query transformer in which ego, agent, and map queries distilled from the BEV scene are fused by learnable waypoint queries via cross-attention, followed by deformable spike-cross-attention refinement. Every operation is gated by integer spikes and inference is a single feed-forward pass without temporal simulation loops. On the nuScenes open-loop benchmark, SDPAD achieves an average $L_2$ error of 0.40\,m and a collision rate of 0.12\%, on par with strong ANN planners while consuming 69.9\,mJ---less than 2\% of recent ANN baselines. In closed-loop evaluation on the NAVSIM navtest split, SDPAD reaches 86.3 PDMS, surpassing the previous SNN planner SAD by 4.3 points and matching mainstream ANN planners at a fraction of their energy. To our knowledge, SDPAD is the first fully spike-driven planner evaluated in end-to-end autonomous driving, demonstrating that SNNs can rival dense ANNs in complex driving tasks.

1 Introduction

Dense ANN planners perform well but are difficult to deploy on embedded hardware, while existing spiking planners face latency, BEV-lifting, and scene-interaction bottlenecks. SDPAD combines integer-spike, single-pass computation with spike-driven BEV lifting and query-based trajectory planning, narrowing the gap with ANN planners.

1 Introduction

  • Dense ANN planners achieve strong performance but their heavy transformer or convolutional backbones challenge deployment under embedded vehicles’ latency and power limits.The introduction describes this as a performance–efficiency dilemma.Source: §1 · p. 1
  • Existing spiking planners rely on iterative simulation, dense floating-point softmax depth distributions, or decoders lacking query-based aggregation of heterogeneous scene context.The identified context includes the ego vehicle, surrounding agents, and map.Source: §1 · pp. 1–2
  • SDPAD converts a pretrained ANN perception stack to integer spikes and uses Spike-3D-Lift to replace softmax depth distributions with spike-driven-max BEV lifting.The resulting lifting uses spike-gated accumulation while preserving geometric fidelity.Source: §1 · p. 2
  • Spike-QFormer distills ego, agent, and map context into query groups that waypoint queries fuse through spiking cross-attention, followed by deformable spike-cross-attention trajectory refinement.Inference uses a single feed-forward pass without temporal unrolling.Source: §1 · p. 2
  • SDPAD substantially narrows the SNN–ANN planning gap on nuScenes and NAVSIM, with accuracy comparable to mainstream ANN baselines at a small fraction of their energy consumption.The paper identifies this as the result of its experiments across both benchmarks.Source: §1 · p. 2

2 Related Work

Prior work advances end-to-end planning and applies SNNs to perception and control, but fully spike-driven autonomous-driving planning and specialized query decoders remain gaps. SDPAD positions its spiking query decoder as a response to those limitations.

2.1 End-to-End Autonomous Driving

  • End-to-end planners combine perception, prediction, and planning, but heavy dense decoder stacks create a multiply–accumulate bottleneck for real-time edge deployment.SDPAD is described as matching state-of-the-art planning performance while replacing dense computation with spike-driven operations throughout its pipeline.Source: §2.1 · p. 2

2.2 Spiking Neural Networks

  • SNNs serve perception and robot navigation, but autonomous-driving planning has largely remained open-loop and single-modality, with few fully spiking frameworks.SAD is cited as a notable end-to-end spiking-planning benchmark on nuScenes.Source: §2.2 · pp. 2–3

2.3 Query-Based Scene Understanding for Planning

  • Query-based decoders distill sparse scene structure, but floating-point Softmax limits neuromorphic deployment and existing spiking transformers lack specialized multi-task query decoders.The paper proposes Spike-QFormer to extend the query-based bottleneck paradigm to spiking end-to-end driving.Source: §2.3 · p. 3

3 Method

SDPAD combines a spiking BEV perception pipeline with query-based waypoint planning and deformable refinement. Its method uses integer-spike operations across the pipeline, including depth lifting and attention.

3.1 Overall Architecture

  • SDPAD processes multi-view images into a spiking BEV representation for segmentation and 3D detection, then refines waypoints and outputs an ego-vehicle trajectory.The pipeline uses E-Spikeformer and Spike-3D-Lift for BEV encoding, followed by Spike-QFormer and Deformable SCA for planning.Source: §3.1 · p. 3

3.2 Preliminary

  • Integer ANN2SNN conversion uses quantized activations and parallelizes SNN dynamics over the temporal window to reduce mapping error and residual potential variance.The method employs an iterative Leaky Integrate-and-Fire neuron with a soft reset.Source: §3.2 · p. 3

3.3 Spike-Driven-Max for Spike-3D-Lift

  • Spike-Driven-Max replaces floating-point softmax depth distributions with discrete integer spike intensities for hardware-friendly feature lifting.It clips and rounds depth estimates, applies an exponential mapping and learnable normalization, then uses a straight-through estimator for the non-differentiable rounding step.Source: §3.3 · pp. 3–4

3.4 Spike-QFormer for Waypoint Decoding

  • Spike-QFormer builds ego, agent, and map queries from BEV context, then fuses them with learnable waypoint queries through spiking cross-attention.A Spike-MLP decodes waypoint tokens into candidate trajectories, with the active navigation command selecting the final output.Source: §3.4 · pp. 4–5

3.5 Deformable Spike-Cross-Attention

  • Deformable SCA refines waypoint tokens by sampling BEV features at predicted offsets and aggregating them with spiking attention weights.It replaces floating-point attention computations with spike-driven operations and preserves a residual connection.Source: §3.5 · p. 5

3.6 Loss

  • Training combines segmentation, agent detection, planning, and depth losses; planning applies command-conditioned L1 supervision to initial and refined trajectories over valid future horizons.Segmentation supervises the 200×200 BEV grid, while agent detection combines existence classification with state regression.Source: §3.6 · pp. 5–6

4 Experiments

Experiments compare SDPAD with ANN and SNN planners on nuScenes open-loop and NAVSIM closed-loop benchmarks, then assess decoder, quantization, and BEV depth-attention choices. Results report competitive planning alongside low-complexity design trade-offs.

4.1 Benchmarks

  • nuScenes evaluates 3-second trajectories sampled at 2 Hz using L2 displacement error and Collision Rate, while NAVSIM assesses 4-second predictions with PDMS across five driving criteria.NAVSIM uses an LQR controller to interpolate the predicted trajectories; PDMS integrates collision, drivable-area, time-to-collision, comfort, and ego-progress criteria.Source: §4.1 · p. 6

4.2 Implementation Details

  • For nuScenes, the model uses an E-SpikeFormer feature extractor and produces three command-conditioned trajectories; for NAVSIM, two E-SpikeFormer backbones process images and LiDAR BEV maps and predict eight waypoints.The nuScenes model is trained for 30 epochs and the NAVSIM model for 100 epochs, each using two NVIDIA A800 GPUs.Source: §4.2 · p. 6

4.3 Main Results

  • 86.3% PDMS on NAVSIM is comparable to mainstream ANN planners, while SDPAD exceeds SAD by 4.3 PDMS points; on nuScenes it is competitive with strong ANN planners and outperforms SAD.The comparisons are presented in Tables 2 and 1, respectively.Source: §4.3 · p. 6, Table 2 · p. 7, Table 1 · p. 7

4.4 Ablation Studies

  • Adding ego and agent queries to the map-query baseline lowers L2 from 0.98 m to 0.67 m, and adding Deformable-SCA reaches 0.64 m L2 and 0.26% collision rate.The full ego-agent-map combination reports 0.66 m L2 and 0.30% collision rate; Deformable-SCA increases parameters from 35M to 39M.Source: §4.4 · pp. 7–8, Table 3 · p. 7
  • Increasing quantization steps from 2 to 8 reduces L2 from 5.32 m to 0.64 m and collision rate from 6.12% to 0.26%, while energy falls from 84.73 mJ at 3 steps to 69.93 mJ at 8.The integer-spike representation encodes the firing count of conversion steps without inference-time unrolling.Source: Table 4 · p. 7, §4.4 · p. 8
  • Compared with Softmax, SDM cuts computational overhead by 23.6% while maintaining 36.60% mIoU and 13.94% mAP:0.5; it reports 0.64 m L2 and 0.26% collision rate at 3 seconds.SparseMax and I-LIF reduce computational complexity further but degrade perception fidelity, with I-LIF reaching 17.48% mIoU.Source: Table 5 · p. 7, §4.4 · p. 8

5 Conclusion

The conclusion presents SDPAD as a fully spike-driven, single-pass autonomous-driving pipeline. The authors identify direct surrogate-gradient training and on-chip neuromorphic deployment as future work.

5 Conclusion

  • SDPAD combines integer ANN2SNN conversion, spike-driven-max BEV lifting, and Spike-QFormer planning with deformable spike-cross-attention refinement.The pipeline operates on integer spike representations in a single feed-forward pass without temporal simulation loops.Source: §5 · p. 8
  • Experiments on nuScenes and NAVSIM support the conclusion that spiking architectures can rival dense ANN planners in complex driving tasks at a fraction of their energy cost.The conclusion also characterizes the results in terms of planning quality, energy efficiency, and closed-loop safety.Source: §5 · p. 8
  • Direct surrogate-gradient training and on-chip neuromorphic deployment are left for future work.Source: §5 · p. 8

A Implementation Details

The implementation first trains a dense ANN perception stack on nuScenes, then converts its activations to integer spikes while retaining the linear layers.

A.1 ANN pre-training and conversion

  • The dense ANN counterpart is trained on nuScenes before its weights are converted to the spiking model.Source: §A.1 · p. 11
  • Conversion replaces each activation with the integer-spike operator SN(·), using I-LIF with Vmax=8, while leaving all linear layers unchanged.Source: §A.1 · p. 11

A.2 Dataset and Input Settings

The appendix specifies distinct nuScenes and NAVSIM input and trajectory settings. nuScenes conditions predictions on navigation commands, while NAVSIM follows its official input protocol.

A.2 Dataset and Input Settings

  • For nuScenes, six surround-view images are downsampled to 256×704, with a 100×100 BEV grid and a 68-bin depth range.The planner predicts six waypoints over 3 seconds at 2 Hz, conditioned on left, straight, or right commands.Source: §A.2 · p. 11
  • For NAVSIM, the input combines front and front-left/right camera images, with a rasterized BEV map for the map-enhanced variant, and the planner predicts eight waypoints over 4 seconds.Source: §A.2 · p. 11

A.3 Energy Measurement Methodology

The energy estimate weights MAC and AC counts by their 45 nm per-operation costs, classifying operations from each layer’s input tensor. Spike firing rates are averaged per layer over the validation set, and ANN baselines use the same pipeline with all operations counted as MACs.

A.3 Energy Measurement Methodology

  • Energy is estimated with a 45 nm hardware model using 4.6 pJ per MAC and 0.9 pJ per AC operation.BatchNorm layers are folded into their preceding convolutions before counting.Source: §A.3 · p. 11
  • Integer-spike inputs scale operation counts by measured firing rate and are counted as ACs; other inputs are counted as MACs.Firing rates are measured per layer across the validation set and averaged.Source: §A.3 · p. 11
  • Energy is reported per nuScenes frame or NAVSIM simulation step, while ANN baselines use the identical pipeline with all operations counted as MACs.Source: §A.3 · p. 11

B Model Architecture Details

This section provides the per-stage architecture specifications and describes initialization choices for SDM normalization and deformable spike-cross-attention sampling. The choices set an adaptive normalization scale and a structured spatial prior anchored to initial trajectory predictions.

B Model Architecture Details

B.1 Spike-Driven-Max (SDM) Initialization

  • SDM’s learnable normalization factor starts near Dmax and can adapt during fine-tuning.N = Dmax + softplus(α), with α initialized to −5.Source: §B.1 · p. 11

B.2 Deformable SCA Initialization

  • Deformable SCA offsets start at zero, while concentric-ring biases cover spatial neighborhoods around each reference point.Each waypoint token’s reference point is its initial, unrefined trajectory coordinate, anchoring sampling to the model’s geometric prediction.Source: §B.2 · p. 11

C Additional Experimental Results

A spike-driven CenterNet detection probe evaluates the representation capacity of SDPAD’s Spiking BEV features. The quantized model trails state-of-the-art ANN-based detectors but outperforms PETRv2.

C Additional Experimental Results

  • The quantized model trails state-of-the-art ANN-based baselines but outperforms PETRv2 on nuScenes validation 3D object detection.The results are presented as evidence that SDM-derived Spiking BEV features encode rich spatial and semantic representations.Source: C Additional Experimental Results · pp. 11–12

D Qualitative Results

The extended qualitative evaluation shows representative nuScenes and NAVSIM scenarios, comparing predicted trajectories with ground truth. It also presents intermediate Spiking BEV features and nuScenes perception outputs.

D Qualitative Results

Loading 2610.11583v1…