Source-linked AI summary
Experimentally realized in situ backpropagation for deep learning in nanophotonic neural networks
Sunil Pai, Zhanghao Sun, Tyler W. Hughes, Taewon Park, Ben Bartlett, Ian A. D. Williamson, Momchil Minkov, Maziyar Milanizadeh, Nathnael Abebe, Francesco Morichetti, Andrea Melloni, Shanhui Fan, Olav Solgaard, David A. B. Miller
TL;DR
The paper addresses the lack of an analog backpropagation demonstration for nanophotonic neural networks and its relevance to rising AI energy demands. It implements in situ backpropagation on a foundry-manufactured hybrid photonic network, including an all-analog gradient-update protocol. The system solves classification tasks with accuracies similar to digital equivalents despite systematic error, while gradient accuracy remains important for convergence.
Problem
Nanophotonic neural networks lacked an experimental demonstration of analog backpropagation, despite growing interest in analog AI hardware and rising neural-network energy demands.
Method
The paper alternates programmable photonic mesh matrix operations with digital nonlinearities and implements in situ backpropagation using bidirectional propagation, grating-tap monitoring, and amplitude-phase readout.
Results
Model train accuracy reached 95% and model test accuracy 97% for the corrected phase-measurement experiment.
Takeaways & Limitations
In situ backpropagation can provide a feasible training approach for deep multilayer hybrid photonic neural networks while optically accelerating matrix operations and gradient computations.
Takeaways & Limitations
The paper's analysis relies on a stated convention that follows a simple example in Ref. 5.
Abstract
from arXiv · showhide
Neural networks are widely deployed models across many scientific disciplines and commercial endeavors ranging from edge computing and sensing to large-scale signal processing in data centers. The most efficient and well-entrenched method to train such networks is backpropagation, or reverse-mode automatic differentiation. To counter an exponentially increasing energy budget in the artificial intelligence sector, there has been recent interest in analog implementations of neural networks, specifically nanophotonic neural networks for which no analog backpropagation demonstration exists. We design mass-manufacturable silicon photonic neural networks that alternately cascade our custom designed "photonic mesh" accelerator with digitally implemented nonlinearities. These reconfigurable photonic meshes program computationally intensive arbitrary matrix multiplication by setting physical voltages that tune the interference of optically encoded input data propagating through integrated Mach-Zehnder interferometer networks. Here, using our packaged photonic chip, we demonstrate in situ backpropagation for the first time to solve classification tasks and evaluate a new protocol to keep the entire gradient measurement and update of physical device voltages in the analog domain, improving on past theoretical proposals. Our method is made possible by introducing three changes to typical photonic meshes: (1) measurements at optical "grating tap" monitors, (2) bidirectional optical signal propagation automated by fiber switch, and (3) universal generation and readout of optical amplitude and phase. After training, our classification achieves accuracies similar to digital equivalents even in presence of systematic error. Our findings suggest a new training paradigm for photonics-accelerated artificial intelligence based entirely on a physical analog of the popular backpropagation technique.
PHOTONIC NEURAL NETWORKS
The paper builds hybrid photonic neural networks by alternating programmable optical matrix operations with digital nonlinear transformations. The network propagates inputs forward, measures an output cost, and computes parameter gradients for training.
- PHOTONIC NEURAL NETWORKS: Trainable phase shifts η parameterize the photonic network and determine its optical linear transformations.The total number of phase-shifting elements across layers is denoted D.
- PHOTONIC NEURAL NETWORKS: Hybrid PNNs alternate analog photonic matrix-vector multiplications with digital nonlinear transformations across network layers.This combines optical linear processing with digitally implemented nonlinear functions.
- PHOTONIC NEURAL NETWORKS: During inference, input data are forward-propagated through L layers to produce the network output and evaluate the cost against ground-truth labels.The cost is represented as L(x, z) = c(bz(x), z).
- PHOTONIC NEURAL NETWORKS: Backpropagation updates η using gradients computed from the cost for individual training examples or averaged batches.The gradient is represented as ∂L/∂η ∈ R^D.
- PHOTONIC NEURAL NETWORKS: A ring-shaped 2D classification task achieved 90% accuracy on the actual photonic chip.The chip-labelled points agreed well with the ideal ring classification boundary.
BACKPROPAGATION DEMONSTRATION
The demonstrated architecture equips a programmable photonic mesh for bidirectional, globally monitored optical computation with complete amplitude and phase readout. These capabilities support forward, backward, and sum steps for experimentally implementing in situ backpropagation.
- BACKPROPAGATION DEMONSTRATION: The photonic mesh architecture experimentally implements backpropagation within a hybrid digital-analog model using universal linear optics for computationally intensive operations.The demonstration uses a foundry-manufactured photonic circuit and custom optical rig.
- BACKPROPAGATION DEMONSTRATION: Bidirectional light propagation sends and measures signals from either direction through the photonic circuit.This supports switching between forward and backward signal propagation.
- BACKPROPAGATION DEMONSTRATION: Global monitoring measures optical power at any waveguide segment using 3% grating taps and an IR camera.The proof-of-concept setup images taps throughout the chip with an automated stage.
- BACKPROPAGATION DEMONSTRATION: Self-configuring Generator and Analyzer circuits provide amplitude and phase detection for both forward and backward propagation.Their symmetry enables optical input generation and output measurement in both directions.
- BACKPROPAGATION DEMONSTRATION: The in situ protocol sends forward signals, backward adjoint signals, and a sum vector through each mesh to calculate gradient updates.The earlier protocol obtains gradients by subtracting forward and backward measurements from sum measurements digitally.
ANALOG UPDATE
The paper evaluates an all-analog gradient-update protocol that replaces digital subtraction with an adjoint-phase sweep and local optical processing. Experiments show that gradient accuracy worsens near convergence because of uncorrected photonic circuit errors.
- ANALOG UPDATE: The proposed analog update sweeps the adjoint phase ζ from 0 to 2π and extracts the gradient from the measured AC power component.The gradient is obtained as dη(0), the AC component when no adjoint phase is applied.
- ANALOG UPDATE: A summing interference circuit, gated integration, and high-pass filtering can compute dη(ζ) and update phase-shifter voltages entirely in the analog domain.The scheme is intended to avoid analog-digital conversion and additional memory complexity.
- ANALOG UPDATE: Digital subtraction and the proposed analog update are mathematically equivalent, and both can operate in parallel across photonic layers.The analog expression is identified as dη(0)/2.
- ANALOG UPDATE: The analog gradient measurement was experimentally simulated by programming summing vectors and recording dη(ζ) on the chip.The experiment optimized a four-point discrete Fourier transform target matrix under phase perturbations.
- ANALOG UPDATE: Analog gradient measurements become less accurate near convergence, likely because loss and thermal crosstalk remain uncorrected.These photonic circuit errors produce large gradient measurement errors.
PHOTONIC NEURAL NET TRAINING
The authors train multilayer hybrid photonic neural networks on 2D classification tasks using in situ backpropagation, with photonic matrix operations and digitally implemented nonlinearities. Performance varies with phase-measurement accuracy, but the system achieves successful model-level classification.
- Training setup: The experiments train three-layer hybrid photonic neural networks on circle and moons 2D classification datasets using in situ backpropagation.The networks use 4 × 4 matrix-vector multiplication layers, absolute-value nonlinearities, and a softmax decision layer.
- Training setup: The hybrid inference pipeline programs photonic matrix layers, measures output power, and digitally applies square-root nonlinearities before subsequent optical computations.The same packaged chip performs input, output, and matrix operations across all three layers.
- Evaluation protocol: Model metrics evaluate physical-device-learned parameters on the true model, distinguishing them from device cost and accuracy measured during hardware operation.The training updates occur on the device, while the reported model evaluation is performed computationally.
- Circle classification: 96% model test accuracy and 93% model train accuracy are achieved on the circle dataset despite noisy training curves and phase-shift gradient errors.The simulated and measured training curves show roughly correlated swings, with convergence visible in iteration-averaged curves.
- Moons classification: 87% model train accuracy and 94% model test accuracy are achieved on the moons dataset with measured phases, while corrected ground-truth phases yield 95% train and 97% test accuracy.Using ground-truth phase measurements reduces phase-shift gradient error by roughly an order of magnitude and improves training outcomes.
DISCUSSION AND OUTLOOK
The discussion presents analog in situ backpropagation as feasible for multilayer hybrid photonic neural networks despite gradient errors. It emphasizes gradient accuracy as central to training efficiency and identifies photonic acceleration of computationally intensive operations as the main opportunity.
- Discussion: Proof-of-principle experiments indicate that analog gradient measurement and photonic-network training remain efficient and feasible despite gradient error.The work frames these experiments as a foundation for analyzing practical photonic mesh accelerators and their tolerance to gradient errors.
- Discussion: In situ backpropagation is presented as the most practical and efficient approach for training deep multilayer hybrid photonic neural networks.The comparison concerns population-based, direct-feedback-alignment, and perturbative alternatives.
- Outlook: The hybrid approach optically accelerates O(N^2) matrix-vector products and matrix-gradient computations while implementing O(N) nonlinearities and derivatives digitally.The authors describe this division as reasonable because optical input and output modulation and measurement already require O(N) time.
- Gradient accuracy: Gradient accuracy plays an important role in reaching optimal training results.More accurate gradients produce convergence speeds and oscillations comparable to digital gradient calculations on the same example sequence.
- Outlook: The analog approach is positioned as an opportunity for energy-efficient artificial intelligence applications using photonic hardware.The authors also describe integration with existing machine-learning training protocols and autodifferentiation frameworks.
DATA AND SOFTWARE
The paper releases its software and data for reproducing the simulations, experiments, and circuit designs through Zenodo, GitHub, and associated photonics frameworks.
- Reproducibility: Software and data for the simulations and experiments are available through Zenodo and GitHub.The release includes experimental code via Phox.
- Reproducibility: Simulation and circuit-design code are provided through Simphox and Dphox, respectively.These tools accompany the experimental Phox framework.
CONTRIBUTIONS
The work involved photonic-chip fabrication and experiments, control software, custom electronics, manuscript preparation, and collaborative protocol and results discussions.
- Roles: SP taped out the photonic integrated circuit and ran the experiments with input from the listed coauthors.SP and ZS wrote device-control code, while TP designed the custom PCB with input from SP.
- Roles: SP wrote the manuscript with input from all coauthors, who also contributed to discussions of the protocol and results.
CONFLICTS OF INTEREST
The authors disclose a patent filing covering the analog backpropagation update protocol described in this work and report no other conflicts of interest.
- The authors filed a patent for the analog backpropagation update protocol discussed in this work.
- The patent application is identified as Prov. Appl. No.: 63/323743.
- The authors declare no other conflicts of interest.
METHODS
The methods combine a foundry-manufactured 6 × 6 photonic mesh with optical monitoring, bidirectional propagation, and analog or digital gradient measurement. Forward and backward optical operations implement matrix-vector products and vector-Jacobian products for training.
- Hardware platform: The experiment uses a foundry-fabricated 6 × 6 triangular photonic mesh containing 15 Mach-Zehnder interferometers and programmable titanium-nitride phase shifters.
- Optical rig design: Grating taps, an IR camera, and an XY translation stage measure optical inputs, outputs, and gradient-monitoring signals across the chip.The triangular circuit presents a 6 × 19 spot array, while the camera views 6 × 3 grating spots simultaneously.
- Forward inference operation: Forward inference programs phase shifters, injects optical amplitudes, measures mesh outputs, and reconstructs complex amplitudes from analyzer settings.The mesh evaluates y^(ℓ) = U^(ℓ)x^(ℓ), with output amplitudes and phases measured after propagation.
- Backpropagation protocol: During backpropagation, the system reverses generator and analyzer roles and sends error signals backward through each photonic layer to compute optical VJPs.The backward adjoint pass produces x^(ℓ)_aj = (U^(ℓ))^T y^(ℓ)_aj.
- Gradient measurement: Gradients are obtained either by digital power subtraction or by sweeping an auxiliary phase and reading the AC response of monitored power.For analog measurement, the gradient is d_θ(0)/2 after repeated sweeps from 0 to 2π.
Appendix A: Energy and latency analysis
The appendix compares energy and latency for analog and digital update strategies in hybrid photonic computation. Analog updates avoid storing prior gradient elements but require dedicated optoelectronic gradient-measurement units, while digital subtraction supports adaptive methods such as Adam.
- Energy model: Photonic matrix-vector products require roughly N · (E_inp + E_meas) energy for optical input and output measurement.The comparison treats input modulation and output measurement as the dominant setup costs for each matrix-vector product.
- Energy model: Digital electronic computation requires N^2 sequential multiply-and-accumulate operations for a matrix-vector product.
- Analog update: Analog backpropagation adds N^2 E_grad + N · (3E_inp + 2E_meas) to account for gradient measurement and additional optical input/output operations.
- Digital update: Digital subtraction can support adaptive updates such as Adam, but it requires storing N^2 elements and has higher gradient-processing energy because E_grad,digital ≫ E_grad,analog.The passage also notes that high-speed analog-digital conversion can consume up to 40 fJ per comparator.
- Phase-shifter modulation: MEMS phase shifters are presented as a promising alternative because they consume no static energy and can provide a linear voltage-to-phase relationship.A linear relationship makes the voltage gradient update proportional to the phase-shift gradient without a nontrivial scaling circuit.
Appendix B: Gradient accuracy
The appendix examines how gradient measurement accuracy affects optimization and how minibatch averaging interacts with gradient error. It emphasizes accurate phase measurements and notes that the reported phase-readout method increases gradient error by an order of magnitude.
- Gradient accuracy: Gradient accuracy can affect optimization and decrease as training approaches convergence.
- Minibatch effects: Minibatch averaging smooths noisy training curves because averaged gradients are smaller in magnitude.
- Minibatch effects: Minibatch gradients can have significantly higher normalized error than gradients from individual training examples.The passage attributes this to unchanged gradient-error variance combined with a larger relative contribution when the batch gradient is small.
- Measurement accuracy: The output phase measurement used in the study produces an order-of-magnitude increase in gradient error, underscoring the importance of accurate phase measurement.
- Calibration assumption: Operating phase shifters in a linear regime makes the phase-to-voltage derivative approximately constant and simplifies gradient updates.The voltage gradient follows ∂L/∂v_θ = (∂θ/∂v_θ)(∂L/∂θ), with the derivative obtained from calibration curves.
Appendix D: Comparison with other training algorithms
Backpropagation is presented as an efficient training method for multilayer neural networks, while the photonic implementation combines optical matrix operations with digital processing and calibration. The appendix describes the hardware protocol, its computational assumptions, and limitations affecting fully analog updates.
- Backpropagation is described as the most widely used and efficient known algorithm for training multilayer neural networks.
- Hybrid photonic neural networks alternate programmable optical matrix-vector multiplication with digital nonlinear transformations.The architecture uses photonic meshes for linear operations and digital processing between layers.
- The photonic mesh uses generator, matrix, and analyzer subunits to create, transform, and measure optical vectors in forward or backward propagation.The analyzer can be operated in reverse, and bidirectional operation supports backpropagation.
- Self-configuration programs vector-unit ports by minimizing power, with balanced binary-tree architectures requiring O(log N) measurement steps.The protocol sweeps phase settings to nullify open ports and maximize output-port power.
- The protocol performs γU-related computation off-chip because multiplying by γU is treated as an O(N) operation.The corresponding phase shifts are omitted from the demonstrated protocol under the assumption that they are inexpensive and digitally accountable.
- Purely analog updates may require nonvolatile memory for previous gradients; otherwise, digital subtraction is needed to aggregate update history.
Appendix H: Simulated error analysis
Simulations evaluated a larger N = 64 triangular photonic mesh on MNIST and tested in-situ backpropagation under amplitude, phase, and photon-shot-noise errors. With moderate noise levels, convergence was minimally influenced despite minor fluctuations, indicating robustness to these perturbations.
- Larger-scale simulation: The N = 4 prototype limits processing capability, so simulations extended evaluation to a two-layer triangular mesh with N = 64 inputs.The larger model transformed 28 × 28 MNIST images into 64-dimensional complex vectors for handwritten-digit recognition.
- Hardware-error model: The robustness analysis added amplitude-generation errors, phase-generation errors, and photon shot noise to the simulated training process.These corresponded to field amplitude error aerror, field error perror, and monitoring noise snoise.
- Robustness results: At moderate noise levels consistent with current photonic circuits, model convergence was minimally influenced despite minor fluctuations.The analysis evaluated normalized gradient errors and testset accuracies for each error type.
- Robustness results: The simulations therefore demonstrate robustness of in-situ backpropagation to noise and hardware errors that are difficult to eliminate completely.