Source-linked AI summary
Single-chip photonic deep neural network for instantaneous image classification
Farshid Ashtiani, Alexander J. Geers, Firooz Aflatouni
TL;DR
Integrated photonic deep networks have lacked demonstrated scalable end-to-end implementations because of nonlinear-function and device-loss challenges. This paper presents an on-chip photonic deep neural network that classifies images through propagating optical waves, performing end-to-end classification within a single clock cycle.
Problem
Integrated photonic deep networks lack demonstrated scalable end-to-end implementations, with device bandwidth and propagation constraints remaining relevant scope boundaries.
Method
The chip routes overlapping image sub-images through on-chip neuron layers, combining optical linear computation with opto-electronic nonlinear activation.
Results
Within a single clock cycle, the chip performs end-to-end handwritten-letter image classification.
Takeaways & Limitations
The demonstrated photonic platform supports neural-network computation with low propagation loss, large bandwidth, low energy consumption, and ultra-low computation time.
Takeaways & Limitations
The architecture requires routing overlapping sub-images to input-layer neurons and depends on the bandwidth available in the integrated photonic platform.
Abstract
from arXiv · showhide
Deep neural networks with applications from computer vision and image processing to medical diagnosis are commonly implemented using clock-based processors, where computation speed is limited by the clock frequency and the memory access time. Advances in photonic integrated circuits have enabled research in photonic computation, where, despite excellent features such as fast linear computation, no integrated photonic deep network has been demonstrated to date due to the lack of scalable nonlinear functionality and the loss of photonic devices, making scalability to a large number of layers challenging. Here we report the first integrated end-to-end photonic deep neural network (PDNN) that performs instantaneous image classification through direct processing of optical waves. Images are formed on the input pixels and optical waves are coupled into nanophotonic waveguides and processed as the light propagates through layers of neurons on-chip. Each neuron generates an optical output from input optical signals, where linear computation is performed optically and the nonlinear activation function is realised opto-electronically. The output of a laser coupled into the chip is uniformly distributed among all neurons within the network providing the same per-neuron supply light. Thus, all neurons have the same optical output range enabling scalability to deep networks with large number of layers. The PDNN chip is used for 2- and 4-class classification of handwritten letters achieving accuracies of higher than 93.7% and 90.3%, respectively, with a computation time less than one clock cycle of state-of-the-art digital computation platforms. Direct clock-less processing of optical data eliminates photo-detection, A/D conversion, and the requirement for a large memory module, enabling significantly faster and more energy-efficient neural networks for the next generations of deep learning systems.
Photonic CNN image classifier chip
The photonic classifier chip forms 5x6 optical input arrays, routes overlapping sub-images through a three-layer nanophotonic network, and produces two outputs for up to 4-class classification. Balanced optical paths and opto-electronic activation support uniform signal distribution across the network.
- Input and distribution: 30 input-pixel signals are split into four overlapping 12-pixel sub-images, each routed to one neuron in the first layer.A nanophotonic distribution network uses waveguides, Y-junction splitters, and crossings.
- Input and distribution: Balanced Y-junctions and crossings equalize optical power across all 48 paths connecting the input array to the first layer.The balancing is designed to ensure uniform optical power distribution.
- Nonlinear activation: 7 micro-ring modulators realize the ReLU nonlinear activation function, driven by seven off-chip TIAs.Supply light is uniformly distributed among neurons in the second and third layers.
- Classification output: Two simultaneous outputs, Out1 and Out2, enable classification of up to 4 classes.The output layer contains two neurons.
- Network architecture: The network has three layers: four 12-input first-layer neurons, three 4-input hidden-layer neurons, and two 3-input output neurons.The first layer connects fully to the hidden layer, which connects fully to the output layer.
Image classification demonstration
The integrated PDNN chip classified printed handwritten-letter images almost instantaneously, achieving average accuracies of 93.7% for two classes and 90.3% for four classes. Classification used trained on-chip weights, optical processing, and output thresholding over repeated cross-validation iterations.
- Instantaneous processing: The PDNN classified each image almost instantaneously as it was formed on the input pixel array.The output layer represented the estimated class of the target object.
- Classification procedure: Classification thresholds were derived from measured output voltages and applied to all test images using a simple linear labeling process.Accuracy was calculated against reference labels.
- Classification procedure: 200 cross-validation iterations on randomly selected sample subsets were averaged to report classification accuracy.This procedure was used to demonstrate algorithm robustness.
- 2-class classification: 93.7% average accuracy was achieved for 2-class classification of 216 printed letters, comprising 108 “p” and 108 “d” samples.Printer-induced variations made the dataset more challenging.
- 4-class classification: 90.3% average accuracy was achieved for 4-class classification of 432 printed letters: “p”, “d”, “a”, and “t”, with 108 samples per class.The result remained accurate despite printer-induced variations and noise.
Discussion
The PDNN’s speed is primarily constrained by component bandwidths, yet its demonstrated architecture supports sub-100-picosecond neuron-layer processing and end-to-end classification within one 3-GHz GPU clock cycle. The chip can scale to higher-resolution and more complex image classification while eliminating image sensing, digitization, and large memory modules.
- Speed limitations and scaling: Sub-100-picosecond processing speed can be achieved for each neuron layer using commercial SOI blocks with tens-of-gigahertz bandwidth.Processing speed is mainly limited by the micro-ring modulators, SiGe photodiode, and TIA because computation occurs as waves propagate on-chip.
- Speed limitations and scaling: 3-dB bandwidths larger than 30 GHz enable the implemented PDNN chip to perform end-to-end classification within a single 3-GHz GPU clock cycle.The ring modulator and SiGe photodiodes used on the chip both have 3-dB bandwidths larger than 30 GHz.
- Speed limitations and scaling: The chip can scale to more pixels for instantaneous classification of higher-resolution images and more complex patterns.Routing overlapping sub-images for convolution is a scaling challenge that can be addressed with multiple photonic routing layers and/or tiling adjacent pixel arrays.
- Summary and applications: The first end-to-end photonic deep neural network classifier chip performs instantaneous image classification through optical-wave propagation, eliminating image sensing, digitization, and large memory modules.Integrated photonics provides large bandwidth and low propagation loss for this computation.
- Summary and applications: Low energy consumption and ultra-low computation time enable neural networks that are significantly faster and less power-consuming than conventional all-electrical solutions.The chip could support event-driven and salient-object detection as a standalone classifier or alongside electronic processors.
Methods · Image formation uniformity and path loss
The calibration array assesses image-formation uniformity and input path loss. Uniform illumination produces less than 5% pixel non-uniformity, while approximately 63 mW at the collimator yields about 4 µW per waveguide and about 42 dB total path loss.
- Methods: The calibration array measures individual pixel photocurrents while the chip is uniformly illuminated without obstruction.This procedure checks image uniformity before classification.
- Image formation uniformity and path loss: Less than 5% measured non-uniformity under uniform, unobstructed chip illumination is sufficient for feature extraction.Individual calibration-array pixel photocurrents are measured to verify image quality.
- Methods: 63 mW coupled to the collimator corresponds to about 3 µA measured photocurrent per photodiode.The same uniformity measurement estimates optical path loss from the collimator to chip-coupled power.
- Methods: 0.8 A/W photodiode responsivity converts the measured photocurrent to approximately 4 µW coupled into each grating-coupler waveguide.The estimate uses the measured photocurrent and photodiode responsivity.
- Image formation uniformity and path loss: About 42 dB total path loss is estimated between the optical collimator and each grating-coupler waveguide.The loss is derived from approximately 63 mW at the collimator and approximately 4 µW coupled into the waveguide.
- Image formation uniformity and path loss: 0.035 overlap between the input-pixel aperture area and beam spot contributes to the path loss.This overlap is listed as the first of three main loss factors.
- Image formation uniformity and path loss: 0.0048 aperture fill factor, defined by pixel area relative to aperture area, contributes to the path loss.The fill factor is listed as the second main loss factor.
- Image formation uniformity and path loss: About 5 dB measured grating-coupler loss is the third main path-loss factor, while the transparency-film transmission coefficient is almost one.The three factors explain the approximately 42 dB total loss.
Micro-ring modulator alignment algorithm
The chip uses seven micro-ring modulators to approximate ReLU activations, requiring resonance alignment because fabrication and temperature variations can shift ring wavelengths. A thermal control loop aligns the rings under thermally equilibrated operating conditions and is verified through neuron outputs before and after input illumination.
- Implementation: Seven micro-ring modulators approximate ReLU activations: four serve the first layer and three serve the second layer.The rings are placed at the outputs of the first and second layers.
- Alignment rationale: Shared supply-light wavelength requires aligned ring resonances for reliable, repeatable ReLU realization across neurons.Fabrication variations and temperature changes can shift resonance wavelengths, motivating compensation control loops.
- Alignment algorithm: The control loop sequentially adjusts N-doped-heater voltages to minimize |VSUM − VREF| while respecting the maximum heater voltage Vmax.Each heater has a measured resistance of about 1.9 kW, and the loop updates VREF when VSUM becomes smaller than VREF.
- Operating procedure: Alignment is performed after setting all weights and reaching thermal equilibrium; optimal heater biases are then recorded for classification.Weight-setting can heat the chip through the PIN attenuators, so alignment must account for the resulting thermal state.
Electronic control circuitry
The electronic control circuitry drives and configures the photonic classifier chip. It sets neuron weights, thermally aligns micro-ring modulators, and adjusts ReLU thresholds.
- Electronic control circuitry: The electronic system controls and drives the photonic components of the classifier chip.Its block diagram is shown in Extended Data Fig. 2.
- Electronic control circuitry: The microcontroller generates data and clock signals for the serial DAC array to set weights, while a serial interface writes the data.The DAC array provides weight-setting control for the photonic network.
- Electronic control circuitry: Seven heaters thermally tune the micro-rings, while seven bias voltages (Vb) adjust the threshold of the ReLU blocks.The circuitry adjusts the threshold voltage of each ReLU block.
- Electronic control circuitry: 66 PIN attenuators on chip set weights for neurons arranged as 4x12, 3x4, and 2x3 across the three layers.These dimensions correspond to the first, second, and third layers, respectively.
PDNN chip training process
The PDNN chip was trained by optimizing weight vectors in an identical digital ReLU network, then translating them to chip inputs and calculating class-separating thresholds. Thresholds were iteratively refined using measured training data before classification, with 25% and 50% of the datasets used for the 2- and 4-class cases, respectively.
- Weight optimization: Training optimized PDNN weight vectors in a Python/Keras digital neural network with identical architecture, ReLU activation, and stochastic gradient descent.Recorded training-image pixel values were fed into the digital network to find the optimal weight vectors.
- Threshold calculation: Threshold calculation was included in training to separate classes from the third-layer output combination Vout = Out1 – Out2.One threshold was used for the 2-class case and three thresholds for the 4-class case.
- Threshold calculation: As measured training data were fed to the algorithm, threshold values were updated toward optimal values, producing higher classification accuracies.Accuracy increased and converged toward its maximum as more data enabled more accurate threshold estimates.
- Chip programming: The optimized weight vectors were translated into PIN attenuator-array input voltages and written to the chip using a microcontroller and digital-to-analogue converters.A look-up table specified attenuation as a function of attenuator input voltage.
- Threshold calculation: Threshold values depend on input-data sequence, so the calculation was repeated multiple times to improve robustness before classification.The resulting threshold values were used during the classification phase.
- Data allocation: 25% of the 2-class dataset and 50% of the 4-class dataset were used to calculate thresholds, with the remaining data used for classification.The resulting classification accuracies are reported in the corresponding figures.
Chip fabrication
The photonic chip was fabricated in a 180 nm SOI process using low-loss nanophotonic waveguides, grating couplers, distribution components, photodiodes, and ring modulators with measured optical and electro-optic specifications.
- Fabrication process: 180 nm SOI fabrication used a 2 µm thick buried oxide.The chip was fabricated in the AMF 180 nm SOI process.
- Waveguides: <2 dB/cm waveguide loss was achieved with 220 nm-thick, 500 nm-wide single-mode nanophotonic waveguides.These waveguides were used for photonic routing.
- Optical coupling: 30% input and calibration grating-coupler efficiency contrasted with 40% supply-light coupling efficiency.The measured coupling efficiencies were about 30% and about 40%, respectively.
- Electro-optic components: 0.8 A/W photodiode responsivity and >30 GHz 3-dB bandwidth were measured, while ring modulators also exceeded 30 GHz bandwidth.The photodiodes had a 3-dB bandwidth greater than 30 GHz, and the ring modulators had a 3-dB bandwidth of more than 30 GHz.
Competing interests
The authors declare no conflicts of interest related to this article.
- The authors declare no conflicts of interest related to this article.