PaperlayerView exact revision on arXiv

Source-linked AI summary

An optical neural network using less than 1 photon per multiplication

Tianyu Wang, Shi-Yuan Ma, Logan G. Wright, Tatsuhiro Onodera, Brian Richard, Peter L. McMahon

arXiv:2104.13467v1physics.opticscs.ETcs.LGcs.NE

TL;DR

The paper addresses the high energy cost of deep-neural-network inference by experimentally testing whether optical matrix-vector multiplication can operate below one photon per multiplication. Using a massively parallel 3D free-space optical processor and standard neural-network training, it achieves high MNIST accuracy in the sub-photon regime. The results support low-optical-power ONNs while identifying scalable optical fan-out and fan-in as an engineering requirement.

  • Problem

    Deep neural networks’ growing computation and inference energy costs motivate evidence for optical processors predicted to offer greater energy efficiency than electronic processors.

  • Method

    The study uses a 3D free-space optical processor for massively parallel matrix-vector multiplication, with standard neural-network training and optical inference.

  • Results

    3.2 photons per multiplication yielded ∼99% MNIST accuracy, while 0.64 photons per multiplication yielded >90% accuracy.

  • Takeaways & Limitations

    The results provide a proof-of-principle that ONNs can achieve high accuracy with less than one photon per scalar multiplication and support a path toward 10^-16 J per scalar multiplication.

  • Takeaways & Limitations

    Practical overall energy efficiency still requires scalable optical fan-out and fan-in; attenuation used in the experiment is not an ultimate engineering solution.

Abstract

from arXiv · show

Deep learning has rapidly become a widespread tool in both scientific and commercial endeavors. Milestones of deep learning exceeding human performance have been achieved for a growing number of tasks over the past several years, across areas as diverse as game-playing, natural-language translation, and medical-image analysis. However, continued progress is increasingly hampered by the high energy costs associated with training and running deep neural networks on electronic processors. Optical neural networks have attracted attention as an alternative physical platform for deep learning, as it has been theoretically predicted that they can fundamentally achieve higher energy efficiency than neural networks deployed on conventional digital computers. Here, we experimentally demonstrate an optical neural network achieving 99% accuracy on handwritten-digit classification using ~3.2 detected photons per weight multiplication and ~90% accuracy using ~0.64 photons (~$2.4 \times 10^{-19}$ J of optical energy) per weight multiplication. This performance was achieved using a custom free-space optical processor that executes matrix-vector multiplications in a massively parallel fashion, with up to ~0.5 million scalar (weight) multiplications performed at the same time. Using commercially available optical components and standard neural-network training methods, we demonstrated that optical neural networks can operate near the standard quantum limit with extremely low optical powers and still achieve high accuracy. Our results provide a proof-of-principle for low-optical-power operation, and with careful system design including the surrounding electronics used for data storage and control, open up a path to realizing optical processors that require only $10^{-16}$ J total energy per scalar multiplication -- which is orders of magnitude more efficient than current digital processors.

I. INTRODUCTION

Deep neural networks’ growing energy demands motivate optical processors, whose predicted sub-photon efficiency depends on massively parallel matrix-vector operations. The paper presents a large-scale 3D free-space design that implements these operations through parallel optical multiplications and accumulations.

  • Motivation: 80-90% of the cost in large-scale deployments is for inference processing, intensifying concerns about the energy growth of state-of-the-art DNNs.Deep learning’s expanding models require increasingly large computation and energy consumption.
  • Motivation: Less than 1 photon per scalar multiplication is predicted for sufficiently large vector sizes at the standard quantum limit.A visible-wavelength photon carries energy on the order of 10^-19 J.
  • Motivation: Existing analog ONN demonstrations used vectors of at most 64, far below the >10^3 scale predicted for sub-photon efficiency.Reported optical energy consumption remained 10^-14-10^-13 J rather than the predicted 10^-18 J per scalar multiplication.
  • Approach: The reported 3D free-space processor supports vector sizes up to 0.5 million and enables matrix-vector and vector-vector operations below 1 photon per scalar multiplication.The resulting ONN performs image classification while matching theoretical predictions for quantum-limited optimal efficiency.
  • Approach: The processor computes matrix-vector products by encoding inputs as spatial-mode intensities and weights as modulator-pixel transmissivities.Its design performs scalar multiplications and additions in parallel during a single optical pass, with electronic nonlinear activation functions.
  • Implementation: 505,521 scalar multiplications can be performed in parallel using aligned 711×711 OLED and SLM pixel arrays.Large vector dot products are summed by focusing multiple spatial modes onto a single detector; sufficiently large mode counts provide precise readout even below one photon per mode.

III. PRECISION OF SUB-PHOTON DOT PRODUCTS

The optical processor computes very large dot products with sub-photon energy per multiplication, while maintaining useful numerical precision and MNIST classification accuracy. Its massively parallel design supports low-power ONN operation, though scalable optical fan-out and fan-in remain important engineering requirements.

  • Dot-product precision: 0.001 photons per scalar multiplication produced approximately 6% dot-product error for vectors of size approximately 0.5 million.Shot noise at the detector was the dominant error source in this regime.
  • Dot-product precision: Larger vectors produced lower numerical error for every tested photon budget between 0.001 and 0.1 photons per multiplication.At low photon budgets, shot noise dominated the error across vector sizes.
  • ONN classification: The ONN used fully connected layers with optical matrix-vector multiplications and electronic nonlinearities between layers.The benchmark was a four-layer fully connected network trained with quantization-aware training.
  • ONN classification: 3.2 photons per multiplication yielded approximately 99% MNIST accuracy, while 0.64 photons yielded more than 90% accuracy.The 3.2-photon result was nearly identical to the same trained network’s 99% digital-computer accuracy.
  • System implications: Up to approximately 0.5 million scalar multiplications were performed in parallel, enabling optical energy below 1 × 10^-18 J per scalar multiplication.The authors estimate that system-level design could reach 1 × 10^-16 J per scalar multiplication.
  • System implications: Practical high-efficiency ONNs still require scalable, miniaturized optical fan-out and fan-in with efficient modulators and detectors.The experiments achieved sub-photon operation partly through attenuation, which the authors identify as unsuitable for an ultimate engineering solution.

METHODS

The methods encode vectors and matrix elements optically, detect summed outputs, and calibrate measured dot products against digital ground truth. Accuracy experiments vary photon counts and use normalized non-negative vector pairs, with signed values handled by a procedural modification.

  • Optical encoding: An OLED display encoded input-vector elements as intensities, while a spatial light modulator encoded matrix elements through transmissivity.The setup used green OLED pixels centered around 525 nm and a phase-only reflective SLM for intensity modulation.
  • Optical encoding: Pixel-to-pixel alignment performed element-wise multiplication between the input and weight vectors before optical fan-in.This alignment maximized the matrix-vector multiplication size by using individual pixels rather than pixel patches.
  • Detection: A photon-counting detector measured integrated incident photon flux over a specified time window.Different detectors were used across experiments, including an MPPC for low-light measurements.
  • Calibration and evaluation: Accuracy characterization used normalized vector pairs with non-negative elements, while signed elements could be handled through a straightforward procedural modification.Normalization placed all dot-product answers between 0 and 1 for comparison with the full dynamic range.
  • Calibration and evaluation: The detector was calibrated by fitting a linear relationship between dot-product answers and detector output.Calibration used many different input-vector pairs before accuracy characterization.
  • Calibration and evaluation: Dot-product error was defined as measured output minus digital-computer ground truth, with photon counts adjusted using detector integration time and neutral-density filters.The characterization used randomly chosen grayscale natural-scene image pairs from STL-10.

Training of Noise-resilient Neural Networks

The study trained a four-layer fully connected network with quantization-aware training and then executed its matrix-vector multiplications optically, while applying nonlinear processing digitally.

  • Network architecture: The handwritten-digit classifier used four fully connected layers with 784 input neurons, two 100-neuron hidden layers, and 10 output neurons.ReLU activations determined predictions from the largest output value.
  • Noise resilience: Quantization-aware training quantized activations to 4 bits and weights to 5 bits to improve robustness against numerical error.Training was performed exclusively on a digital computer.
  • Optical inference: The optical processor executed three matrix-vector multiplications corresponding to forward propagation between successive network layers.Weights were loaded onto the SLM, while layer neuron values were encoded on the OLED display.
  • Optical inference: Dot-product photon counts were converted to numerical outputs through a calibration curve fitted using the first 10 MNIST test samples.The measured results were then communicated to a digital computer for subsequent network operations.

2. PROPERTIES OF THE OLED DISPLAY

The optical system used a commercially available OLED display and SLM-based intensity modulation, with an MPPC characterized for single-photon detection and optical-power measurement.

  • OLED display: The OLED was selected for its commercial availability, low cost, high pixel count, and ability to encode very large vectors.Unlike coherent sources, OLEDs cannot encode phase information and generally have slower modulation speeds.
  • OLED display: OLED pixels could be turned off completely, providing high intensity-modulation dynamic range and reducing background-light noise compared with LCD displays.LCD pixels retain residual light because their LED backlights remain active.
  • OLED display: The green OLED array contained approximately 2 × 10^6 pixels in a 1080 × 1920 square lattice with 57.5 µm pixel pitch.A band-pass filter narrowed the broad OLED emission spectrum for more precise and stable SLM phase modulation.
  • Intensity modulation: A phase-only SLM was converted into an intensity modulator using a half-wave plate and polarizing beam splitter.The SLM encoded intensity through the phase difference between extraordinary and ordinary polarized light.
  • Photodetection: The MPPC detected single-photon events through Geiger-mode photodiodes and provided high signal-to-noise ratio, broad measurement range, and moderate bandwidth.At low photon flux, photons could be enumerated by counting output spikes; spatial multiplexing increased the measurable flux range.
  • Photodetection: The MPPC power-meter noise floor corresponded to 1.25 pW, while photon counting verified measurable optical power near 10 fW.The experiments used direct voltage readout because their optical powers exceeded the analog noise floor.

5. ALIGNMENT OF THE OPTICAL IMAGING SYSTEM

The optical system required precise pixel-to-pixel alignment between the OLED and SLM to maximize parallel matrix-vector multiplication size. Field-of-view limitations from vignette, nonlinear distortion, and aberration constrained the aligned array to about 0.5 million pixels.

  • Alignment requirements: Precise de-magnification, resolution, and mechanical alignment were required to match OLED pixels to SLM pixels and prevent crosstalk.The OLED image had to match the SLM pixel pitch, remain no larger than one SLM pixel, and align in translation, rotation, pitch, and yaw.
  • Alignment requirements: 0.16× de-magnification matched the 57.5 µm OLED pitch to the 9.2 µm SLM pixel size.The zoom factor was mechanically tunable to precisely match the two pixel pitches.
  • Alignment scale: 711 × 711 ≈ 0.5 million pixels were aligned in practice, below the 3.6-million-pixel specification estimate.The practical limit was set by imaging errors toward the zoom lens field-of-view edges.
  • Field-of-view limitations: Up to 90% intensity loss from vignette across the aligned array required dimming central pixels for uniform dot-product contributions.The correction preserved equal brightness across pixels used in large dot products.
  • Field-of-view limitations: Nonlinear image distortion caused local zoom-factor variation and progressive pixel walk-off toward the field-of-view edges.Mechanical alignment could correct linear distortions but not nonlinear distortions completely.
  • Field-of-view limitations: Aberration expanded OLED pixel images and coupled optical energy into neighboring SLM pixels with incorrect weight transmissions.This coupling produced crosstalk and weight-modulation errors.

6. CORRECTION OF OPTICAL VIGNETTE

The system corrected spatial transmission nonuniformity and characterized imaging and temporal noise that limited low-light operation. Vignette correction equalized pixel contributions, while crosstalk and detector, OLED, and SLM fluctuations remained relevant error sources.

  • Vignette correction: Vignette correction used an attenuation map derived from a uniformly illuminated OLED image to equalize transmission across the region of interest.The corrected intensity was matched to the darkest corner of the selected region.
  • Imaging quality: Pixel walk-off and crosstalk coupled energy into neighboring SLM pixels with incorrect weights, and spatially varying crosstalk kernels quantified these errors.The kernels were measured near the center and corner of the field of view using sparse pixel grids.
  • Temporal noise: The optical-power SNR was about half the shot-noise-limited value under constant OLED brightness and SLM transmission.Excess noise arose from OLED intensity fluctuations, SLM phase instability, and detector intrinsic noise.
  • Temporal noise: OLED intensity fluctuations included raster scanning, pulse-width modulation, and thermal noise, motivating avoidance of very low pixel settings.Pulse-width modulation was observed only below approximately 35% brightness.
  • Temporal noise: The SLM oscillated at 53 kHz with 0.24% peak-to-peak power ripple, and its instability was relatively minor compared with OLED fluctuations.The ripple was measured from the first-order diffraction spot of a phase grating.
  • Temporal noise: Detector intrinsic noise became significant near 1 pW and necessitated photon counting at still lower photon flux.At high optical power, detector intrinsic noise was negligible compared with other conditions.

9. OPTICAL FAN-IN AND DETECTION ENERGY CONSUMPTION

Optical fan-in performs dot-product summation by accumulating photoelectrons on one detector, avoiding digital summation and intermediate storage. Large numbers of spatial modes improve low-photon SNR and reduce the energy cost of accumulation.

  • Optical fan-in: Optical fan-in sums spatial modes by physical photoelectron accumulation on one detector instead of digital summation across detector pixels.This skips digital summation circuits and reduces intermediate-result memory usage.
  • Detector design: The detector operated in high-gain Geiger mode for single-photon SNR, but lower gain could reduce energy per addition and prevent saturation.Detector area, capacitance, gain, and full-well capacity jointly constrain the energy-noise trade-off.
  • Detector design: At 1 photon per spatial mode, moderately low gain could produce more electrons per photon than the calculated 2.1-electron thermal-noise level.The thermal-noise estimate used a 27.5 aF detector capacitance at room temperature.
  • Noise scaling: 0.5 signal photons and 2 noise electrons per spatial mode yielded SNR 0.25 individually, while 10,000 modes enhanced SNR to 25.The enhancement follows the square-root scaling with the number of summed spatial modes.
  • Noise scaling: Less than 1 photon per accumulation remains possible even in the thermal-noise-limited regime when the vector size is sufficiently large.Summing many spatial modes increases total signal relative to aggregate noise.
  • Energy consumption: 100 aJ per addition was projected for optimized optical fan-in, versus approximately 10^2 fJ per electronic multiply-and-accumulate operation.The estimate was described as enabling at least 10^2-fold greater energy efficiency than state-of-the-art machine-learning accelerators.

10. COMPARISON TO THE STANFORD MATRIX-VECTOR MULTIPLIER

The 2D-block scheme generalizes the Stanford matrix-vector multiplier by arranging vectors and weights in 2D blocks, improving suitability for image data and reducing crosstalk. Its optical fan-out and fan-in are physically asymmetric, and the demonstrated fan-out remained digital.

  • Comparison to the Stanford Matrix-Vector Multiplier: The 2D-block scheme generalizes the Stanford multiplier’s fan-out, element-wise multiplication, and fan-in using 2D rather than 1D operand arrangements.This geometric change leads to different physical implementations.
  • Comparison to the Stanford Matrix-Vector Multiplier: 2D blocks preserve smooth local image features, reducing errors from minor shifting or blurring in image-classification inputs and weights.Natural-scene images and their trained weights usually vary gradually except near a few high-contrast boundaries.
  • Comparison to the Stanford Matrix-Vector Multiplier: Over an order of magnitude lower crosstalk is potentially achieved with large 2D blocks than with 1D arrays because boundary size scales differently.The text contrasts a square vector perimeter of 4√N with a 1D-array boundary of 2N.
  • Comparison to the Stanford Matrix-Vector Multiplier: 2D blocks reduce crosstalk while retaining high fill factor on both OLED and SLM panels.Introducing gaps would reduce fill factor and computational throughput.
  • Comparison to the Stanford Matrix-Vector Multiplier: The demonstration used digital fan-out, while fully optical fan-out is required for the total energy benefit of optical matrix-vector multiplication.Possible optical implementations include microlens arrays and beam-splitter arrays.

Part II Vector-Vector Dot Product Precision

The paper characterizes optical precision through vector dot products and converts signed vectors into non-negative representations. This preserves optical computation while adding constant digital overhead, with further reuse reducing matrix-vector overhead.

  • Vector-Vector Dot Product Precision: The dot-product result y is defined for input vector x and weight vector w, whose elements x_k and w_k are generally real numbers.These dot products serve as the precision characterization for general matrix-vector multiplication.
  • Vector-Vector Dot Product Precision: Signed-element dot products are converted into two non-negative-vector dot products, followed by 2 digital multiplications and 2 digital additions.The optical computation is doubled, while the digital overhead is constant and independent of vector size N.
  • Vector-Vector Dot Product Precision: The conversion represents signed dot products as a linear combination of dot products involving transformed vectors and an all-ones vector.All resulting dot products use non-negative elements and can be computed optically.
  • Vector-Vector Dot Product Precision: Neural-network inputs and post-ReLU activations are already non-negative, while hardware normalization and nonzero SLM transmission constrain the implemented representation.The vectors are normalized to [0, 1], and the SLM cannot achieve zero transmission.
  • Vector-Vector Dot Product Precision: For matrix-vector multiplication, the all-ones dot product is computed once and reused across matrix rows.A matrix of size N′ × N therefore needs N′ optical dot products plus one additional optical dot product.

A. Generation of Test Datasets

The experiments generated natural-image-based vector pairs, calibrated detector readouts against ground-truth dot products, and evaluated single-shot errors across photon budgets. Larger vectors and higher photon budgets improved precision until systematic error became limiting.

  • Generation of Test Datasets: Test vectors were randomly generated from STL10 natural-scene images, using color-channel patches sized to reach the target vector dimension.Natural images were selected to represent image-classification inputs with globally inhomogeneous and locally smooth structure.
  • Generation of Test Datasets: Element-wise multiplication and optical fan-in computed the dot products, with additional examples recorded for 64 × 64 vectors containing 4,096 elements.The camera captured the modulated light produced from OLED-displayed inputs and SLM-encoded weights.
  • Generation of Test Datasets: Photon budgets were controlled through MPPC detector integration time, and repeated measurements across randomly generated vector pairs provided noise statistics.The calibration dataset varied vector sparsity and used S × T total detector readouts.
  • Generation of Test Datasets: Optical energy or detected photons per dot product were divided by the number of multiplications to compare energy efficiency across vector sizes.At 525 nm, photon energy was approximately 0.4 aJ.
  • Generation of Test Datasets: A least-squares linear calibration converted mean detector readouts into dot-product results for vector pairs of length N = 505521.Calibration generally needed repeating only after setup changes or drift.
  • Generation of Test Datasets: Higher photon budgets reduced error until systematic error limited performance above 1 photon per multiplication, while larger vectors generally produced lower error.These trends were observed across the tested vector sizes and photon budgets.

Part III Optical Neural Network for Image Classification

The image-classification demonstration used a quantization-aware, hardware-aware training procedure for a three-layer MNIST multilayer perceptron. Training incorporated quantization and simulated optical imperfections, while inference used the hardware’s maximum allowable precision.

  • Part III Optical Neural Network for Image Classification: The MNIST classifier was a fully connected three-layer network with 784 inputs, two 100-neuron ReLU hidden layers, and 10 digit outputs.The network was implemented and trained in PyTorch.
  • Part III Optical Neural Network for Image Classification: Quantization-aware training used 4-bit neuron activations and 5-bit weights to adapt the model to the setup’s numerical precision.The selected weight precision matched the SLM’s extinction ratio of 50.
  • Part III Optical Neural Network for Image Classification: Data augmentation simulated hardware imperfections through random rotation, translation, zooming, and 3 × 3 blurring convolution.These transformations modeled misalignment, zoom mismatch, and intra-pixel crosstalk.
  • Part III Optical Neural Network for Image Classification: Training used stochastic gradient optimization with tuned learning rate, momentum, decay, and trial-specific parameters.Optuna was used for parameter fine-tuning.
  • Part III Optical Neural Network for Image Classification: Activation quantization was applied during digital training but not ONN inference, which used 7-bit OLED inputs and 8-bit SLM weights.The trained models performed better at full inference precision despite noise-aware training.

14. WORKFLOW FOR RUNNING OPTICAL NEURAL NETWORKS FOR INFERENCE

Inference alternates optical matrix-vector multiplication with digital bias and nonlinear activation steps, then selects the highest output score. The implementation uses shifted, normalized weights and an auxiliary input block to support signed dot products.

  • Inference workflow: Optical matrix-vector multiplication is performed for each layer during forward propagation, using the trained digital network’s exact weights.Photon usage is controlled by adjusting the number of detector samples summed for each multiplication.
  • Inference workflow: Digital processing applies biases and nonlinear activations between optical layers without modifying or retraining the network parameters.The resulting activations become the input vector for the next optical multiplication.
  • Inference workflow: The output-layer prediction is the class with the highest score.This is the final step after the sequence of optical multiplications and digital nonlinear processing.
  • Optical computation: An auxiliary all-transmissive input block computes the 1⃗·x offset needed to recover signed dot products from non-negative optical inputs.Only one additional block is needed for an entire layer, although an entire row is shown for illustration and redundancy.
  • Inference workflow: 89,400 multiplications and 89,400 additions are performed optically in each forward propagation across layers sized 784×100, 100×100, and 100×10.Digital assistance includes additions for signed-dot-product offsets, biases, and nonlinear activations.

15. ENERGY SCALING OF OPTICAL NEURAL NETWORKS

The energy analysis separates costs that can be amortized by increasing vector dimensions from costs that scale with matrix size. A near-future 4,096×4,096 design is projected to deliver petascale throughput at femtojoule-level energy per MAC, with substantial efficiency advantages over digital accelerators.

  • Energy-scaling categories: Energy costs scaling with input or output vector size decrease per operation as matrix-vector dimensions grow, whereas matrix-sized costs cannot be amortized.The two categories scale respectively as 1/N′ or 1/N, and remain constant with N or N′.
  • Target architecture: A near-future target uses 4,096×4,096 matrix-vector multiplication at a 1 GHz update rate with 4–5-bit weights and activations.The design assumes telecommunications-compatible input encoding and output readout rates.
  • Electronic contributions: ADC energy can reach 10 aJ/MAC for N′ = 4,096, based on 42 fJ per sample at 1 GHz.Higher sampling rates are generally less energy-efficient because of increased thermal noise.
  • Electronic contributions: DAC sampling can consume as little as 3.7 aJ/MAC in comparable 5-bit, 100 MHz electric-crossbar applications.The cited estimate is 15 fJ per DAC sample, comparable to the ADC sample energy.
  • Electronic contributions: On-chip wire transmission can be kept near 100 aJ per MAC through wire planning and optical transmission over distance.The underlying transmission cost is approximately 100 aJ/µm/bit.
  • Electronic contributions: Memory can cost about 4 pJ per 4-bit number without optimization, making storage relatively expensive compared with other energy categories.Much of this cost may arise from signal transmission during memory access.
  • Projected system performance: 17 POPS at 1 GHz and 1 fJ/MAC including memory are projected for the 4,096×4,096 design, corresponding to approximately 40 W total power.Excluding memory, the projected energy per MAC is several hundred aJ.
  • Projected system performance: Optical energy efficiency is projected to be 2–3 orders of magnitude better than state-of-the-art digital machine-learning accelerators and photonic chips.Under heat-limited average power, this corresponds to projected computation speeds 10×–100× higher than digital processors at the same power.
Loading 2104.13467v1…