Source-linked AI summary
Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipeline
Ole Richter, Yannan Xing, Michele De Marchi, Carsten Nielsen, Merkourios Katsimpris, Roberto Cattaneo, Yudi Ren, Yalun Hu, Qian Liu, Sadique Sheik, Tugba Demirci, Ning Qiao
TL;DR
Edge systems need low-latency, low-energy processing that communicates only relevant information from increasingly distributed sensors. This paper integrates an event-based camera with an asynchronous sCNN processor on one SoC, achieving 3.36μs latency through nine convolutional layers and 98.56% N-MNIST accuracy for Speck1 BPTT-CNN.
Problem
Edge computing needs local, real-time extraction of relevant information from diverse sensors while reducing data transmission, latency, and energy consumption.
Method
The paper presents a resource-efficient medium-scale sCNN processor combined with an event-driven vision sensor and fully asynchronous dataflow on a single ASIC.
Results
3.36μs latency was measured for processing one event through nine convolutional layers, while Speck1 BPTT-CNN achieved 98.56% N-MNIST accuracy with 0.6mW power.
Takeaways & Limitations
Integrating sensing and processing on one die supports low-latency visual processing with sparse computation, reduced raw-data communication, and a small energy budget.
Abstract
from arXiv · showhide
Edge computing solutions that enable the extraction of high-level information from a variety of sensors is in increasingly high demand. This is due to the increasing number of smart devices that require sensory processing for their application on the edge. To tackle this problem, we present a smart vision sensor System on Chip (SoC), featuring an event-based camera and a low-power asynchronous spiking Convolutional Neural Network (sCNN) computing architecture embedded on a single chip. By combining both sensor and processing on a single die, we can lower unit production costs significantly. Moreover, the simple end-to-end nature of the SoC facilitates small stand-alone applications as well as functioning as an edge node in larger systems. The event-driven nature of the vision sensor delivers high-speed signals in a sparse data stream. This is reflected in the processing pipeline, which focuses on optimising highly sparse computation and minimising latency for 9 sCNN layers to 3.36μs for an incoming event. Overall, this results in an extremely low-latency visual processing pipeline deployed on a small form factor with a low energy budget and sensor cost. We present the asynchronous architecture, the individual blocks, and the sCNN processing principle and benchmark against other sCNN capable processors.
I. INTRODUCTION
The paper targets low-latency, low-energy near-sensor processing by combining sparse event-based sensing with asynchronous spiking computation on one ASIC. It presents a resource-efficient medium-scale sCNN processor integrated with a machine-vision sensor.
- Motivation: Edge processing must extract relevant information locally with low latency and a small energy budget.The introduction frames on-demand sparse computation and event-based processing as responses to always-on sensing requirements.
- Motivation: Event-based sensors reduce data by communicating pixel intensity changes, lowering both sensor latency and output volume.Each pixel operates independently and transmits only changes in intensity rather than full frames.
- Approach: The fully asynchronous SoC processes sensory events with low latency and high throughput, then enters a low-power idle state when input is absent.The architecture avoids complex wake-up procedures when reducing power consumption.
- Contribution: The paper presents a resource-efficient medium-scale sCNN processor combined with an event-driven machine-vision sensor on a single ASIC.This integration is positioned against large-scale neuromorphic architectures with high synaptic resource costs.
- Paper scope: The paper organizes its presentation around asynchronous methodology, ASIC architecture, sCNN processing, component blocks, and processor comparisons.The stated structure culminates in comparisons of sCNN processors and vision-sensor processor combinations.
II. ASYNCHRONOUS LOGIC DESIGN METHODOLOGY
Speck1 uses asynchronous dataflow logic built from reusable state-holding primitives and handshake validity signals. Its buffers decouple neighboring channel handshakes while the architecture connects asynchronous processing to synchronous external components.
- Design methodology: Speck1’s design flow verifies feature tests against a golden model throughout IC design and through silicon validation.Later models additionally support automated production testing.
- Buffer design: The N-bit buffer holds input data, clears it after output acknowledgement, and decouples input and output handshakes using data validity.A C-element tree with OR2 gates signals whether all data channels contain data or are neutral.
- Dataflow primitives: The processing cores are pipeline systems composed from reusable dataflow primitives such as buffers, joins, splits, merges, and validity trees.These templates are built from pull-up/pull-down state-holding logic cells.
- Asynchronous logic: The QDI pipeline logic uses non-inverting pull-up/pull-down gates and hazard-free four-phase handshake behavior.Inversion is modeled by swapping true and false wires, while negative transitions occur during handshake reset.
- Integration: The architecture includes an asynchronous NoC-facing part and a synchronous part for integration with standard components, microcontrollers, and infrastructure.This split supports communication with synchronous off-the-shelf systems.
III. ARCHITECTURE
The Speck architecture combines convolution cores, an event-based sensor, sensor-event preprocessing, and decision readout through a unicast event-routing NoC.
- Architecture: The architecture comprises convolution cores, the sensor, sensor-event preprocessing, and a decision readout core.These components are connected by a unicast event-routing system called the NoC.
A. Sensor
Speck1 uses a 128x128 event-based sensor whose pixels encode logarithmic light-intensity changes and whose events pass through configurable preprocessing before reaching destination layers.
- Sensor: The sensor contains 128x128 individually operating Dynamic Vision Pixels that temporally encode incident light intensity on a logarithmic scale.Each pixel connects to a handshake buffer that decouples pixel reset and timing from arbitration-readout delays.
- Preprocessing: The sensor event preprocessor can accept built-in or off-chip events and can stream built-in events off-chip for monitoring or further processing.This supports both on-chip processing and external observation of sensor events.
- Preprocessing: Preprocessing stages pool, cut, rotate, mirror, filter channels, and shift the event stream before forwarding it to one or two destination layers.These operations adapt the raw event stream to downstream sCNN requirements.
- Readout: The arbitration system encodes sensor events into an address-event representation channel using column and row arbiter trees.The pixel-level arbitration endpoint is optimized to limit transistor count.
- Pixel implementation: Each pixel has a configurable kill switch that can remove hot-pixel defects by forcing the pixel and buffer into reset.The reported front-illumination fill factor is 45%.
B. Sensor event pre-processing core
The sensor event pre-processing core adapts raw AER events to sCNN input requirements through configurable spatial, polarity, and routing transformations.
- Pipeline role: The pre-processing pipeline conforms raw AER sensor events to sCNN requirements before forwarding them onward.It can flip, rotate, crop, reduce resolution, or ignore polarity.
- Interface: The chip accepts pixel events from its built-in sensor and external sources through an AER interface.Sensor events can also be sent off-chip for monitoring.
- Spatial transformation: Pooling scales each event’s 2D address space by 1:1, 1:2, or 1:4 independently along x and y.
- Spatial transformation: ROI cutting forwards a 1x1 to 128x128 patch from the input address space to the sCNN.
- Spatial transformation: Coordinate transformations can flip, invert, and swap x and y for sideways, overhead, or mirror-mounted sensors.
- Event filtering: Polarity filtering preserves both polarities as separate channels, filters one, or combines both on one channel.
- Routing: Each pre-processed event can be routed to up to 2 destination layers through the NoC with an attached routing header.
C. Network on Chip
The NoC routes sparse events between convolution cores, while the convolution pipeline maps each incoming event into kernel and neuron address sweeps.
- Network routing: The NoC uses a star topology and non-blocking AER routing for feed-forward network models.Separate parallel physical routing channels support concurrent event transport.
- Event representation: The convolution core receives events as {c, x, y}, where c identifies the incoming channel and x,y are spatial coordinates.Padding expands the event address before kernel and neuron mapping.
- Event computation: For each event, the kernel and neuron spaces are anchored and swept according to configurable stride and kernel size.The kernel and neuron addresses move in opposite directions, and the sweep repeats across output channels.
D. Convolution Cores
The convolution cores process events sparsely through address mapping, compressed memories, LIF neuron updates, configurable leak or bias, and downstream event pooling and routing.
- sCNN computation: Unlike frame-based CNNs, event-driven sCNNs compute convolution only at each arriving pixel position.They traverse all output neurons associated with that active pixel rather than sweeping a kernel across a complete image.
- Address processing: The convolution pipeline pads events, maps kernel and neuron anchors, compresses addresses, and sweeps corresponding memory locations.Kernel and neuron addresses are swept inversely, with a new kernel selected for each output channel.
- Kernel memories: Zero-weight kernel positions are skipped, and signed 8-bit weights are directed with compressed neuron addresses to parallel compute blocks.
- Neuron computation: Each signed 16-bit neuron state models a LIF neuron through read-add-check-spike-write operations.Threshold crossing emits an event and writes back either a threshold subtraction or reset state.
- Leak and bias: A configurable reference tick applies signed 16-bit leak or bias values to active neurons through an additional memory controller.
- Output processing: Output events are merged by sum pooling across 1, 2, or 4 neurons independently along x and y.
- Output processing: Channels are shifted and routing information is prefixed before one event is sent per destination, for up to 2 destinations.
- Additional capability: Convolution cores can also implement fully connected layers with up to 65K, 32K, and 16K synaptic connections, respectively.SRAM words with fabrication defects can be blacklisted and skipped during computation.
E. Readout core
The readout core converts event streams into configurable class counts or moving averages and exposes thresholded maximum results for external processing.
- Readout functions: The readout core simultaneously calculates up to 16 spike class counts or moving averages.Moving-average lengths are configurable and can be replaced by time-bin counting without averaging.
- Readout functions: It optionally compares values with fixed thresholds, computes the current maximum, and exposes results on chip pins or for further processing.
IV. RESULTS AND DISCUSSION
Speck1 combines an event-driven sensor with a dedicated sCNN pipeline, achieving low-latency processing and competitive recognition accuracy while reducing energy and memory demands.
- Pipeline performance: 3.36µs latency processes one event through nine convolution-and-pooling layers, while pipelining supports ≈30Mevents/s per neuron compute unit.The latency is measured between input and output request edges at the ASIC I/O pads.
- Architecture discussion: On-the-fly synaptic kernel mapping reduces memory requirements by sharing stored kernel weights across synapses, saving SRAM area and energy.The reported advantage is especially pronounced for 3x3, 4x4, and 7x7 kernels.
- Evaluation setup: The benchmark evaluates a deployed, quantised four-layer sCNN on N-MNIST using both offline PC testing and direct event-stream measurements on Speck1.Five training repetitions and five testing repetitions were conducted for each training method.
- Recognition results: 141µJ energy and 0.47mW power are reported for Speck1 ANN2SNN, while BPTT-CNN reports 180µJ and 0.6mW on N-MNIST.The benchmark uses 11K neurons and 9.3K synapses for both Speck1 configurations.
- Recognition results: 98.50% on-chip accuracy is achieved with BPTT-CNN training, compared with 86.17% for ANN2SNN training.The corresponding offline accuracies are 99.3075% and 94.2%, respectively.
V. CONCLUSION
The conclusion presents Speck as a fully integrated smart sensor that moves efficient sparse neural processing to the sensor edge. Its single-die design and asynchronous, event-driven pipeline target low-cost, low-energy, low-latency visual processing.
- Integrated smart sensor: Speck integrates the sensor and efficient sCNN processor into a single SoC, shifting visual inference directly to the sensor edge.The integration is presented as a fully integrated smart sensor architecture.
- System advantages: Raw sensory data remains on-chip, while on-the-fly synaptic mapping lowers memory requirements and supports larger, more cost-effective fabrication technologies.The architecture processes sparse event data through a deep sCNN pipeline optimised for low latency.