Reprogrammable Electro-Optic Nonlinear Activation Functions for Optical Neural Networks

Ian A. D. Williamson; Tyler W. Hughes; Momchil Minkov; Ben Bartlett; Sunil Pai; Shanhui Fan

Source-linked AI summary

Reprogrammable Electro-Optic Nonlinear Activation Functions for Optical Neural Networks

Ian A. D. Williamson, Tyler W. Hughes, Momchil Minkov, Ben Bartlett, Sunil Pai, Shanhui Fan

arXiv:1903.04579v2Mar 12, 2019eess.SPcs.NEphysics.optics

TL;DR

Optical neural networks need practical nonlinear activation functions because conventional optical nonlinearities require large interaction lengths and high signal powers. This paper proposes a reconfigurable electro-optic architecture and shows improved performance on multi-input XOR and MNIST, increasing MNIST accuracy from 85% to 94%.

Problem
Optical nonlinearities are relatively weak, requiring large interaction lengths and high signal powers that constrain hardware footprint and energy consumption.
Method
The architecture taps a small portion of the optical input, converts it to an electrical signal, and uses electro-optic modulation to synthesize an optical-to-optical nonlinearity.
Results
The activation function enabled successful numerical demonstrations on multi-input XOR and MNIST, with MNIST accuracy improving from 85% to 94%.
Takeaways & Limitations
The reconfigurable activation architecture improves optical neural network expressiveness while supporting varied nonlinear responses and operation without reduced bandwidth or computational speed.
Takeaways & Limitations
The transimpedance amplifier may be challenging to integrate within the area available between neighboring interferometer output waveguides.

Abstract

from arXiv · show

We introduce an electro-optic hardware platform for nonlinear activation functions in optical neural networks. The optical-to-optical nonlinearity operates by converting a small portion of the input optical signal into an analog electric signal, which is used to intensity-modulate the original optical signal with no reduction in processing speed. Our scheme allows for complete nonlinear on-off contrast in transmission at relatively low optical power thresholds and eliminates the requirement of having additional optical sources between each layer of the network. Moreover, the activation function is reconfigurable via electrical bias, allowing it to be programmed or trained to synthesize a variety of nonlinear responses. Using numerical simulations, we demonstrate that this activation function significantly improves the expressiveness of optical neural networks, allowing them to perform well on two benchmark machine learning tasks: learning a multi-input exclusive-OR (XOR) logic function and classification of images of handwritten numbers from the MNIST dataset. The addition of the nonlinear activation function improves test accuracy on the MNIST task from 85% to 94%.

I. INTRODUCTION

Optical neural networks offer high bandwidth, low latency, and reconfigurability, but implementing nonlinear activations remains difficult because optical nonlinearities are weak, power-intensive, bandwidth-limiting, and typically fixed after fabrication. The paper proposes a reprogrammable electro-optic activation architecture that preserves speed while improving ONN flexibility.

Optical hardware platforms are attractive for machine learning because they provide ultralarge signal bandwidths, low latencies, and reconfigurability.
Weak optical nonlinearities require large interaction lengths and high signal powers, increasing physical footprint and energy consumption.
Resonant enhancement of optical nonlinearities trades operating bandwidth for stronger interactions, limiting information-processing speed.
Fabrication-fixed optical nonlinearities prevent ONNs from being reprogrammed for different activation responses and may constrain deep networks as signal power decreases.
Digital activation implementations require repeated optical-electronic conversion, limiting channel scalability and adding conversion latency.
The proposed electro-optic architecture measures a small portion of the optical signal and modulates the original signal, offering complete on-off contrast, varied responses, low thresholds, and preserved bandwidth.

III. NONLINEAR ACTIVATION FUNCTION ARCHITECTURE

The proposed activation converts a tapped fraction of optical power into an electrical voltage, then uses that voltage and an interferometer to produce an optical-to-optical nonlinear response. Electrical bias and optional signal conditioning make the response reconfigurable across ReLU-like and clipped behaviors.

A directional coupler taps fraction α of the optical power to a photodetector, while the remaining signal continues toward the interferometer.
The photodetector and transimpedance amplifier convert tapped optical power into voltage VG = G · R · α|z|2, which can pass through an additional nonlinear conditioner.
The conditioned voltage combines with bias Vb to phase-modulate the remaining optical signal, and the Mach-Zehnder interferometer converts this phase modulation into nonlinear amplitude transmission.
Increasing gain, responsivity, or tapped optical fraction increases phase shift per input power, but a larger tapped fraction also increases linear optical loss.
Electrical bias selects the nonlinear response: φb = 1.0π and 0.85π produce ReLU-like behavior, whereas φb = 0.0π and 0.50π produce saturating clipped responses.
A programmable electrical bias can be connected to the ONN control circuitry, enabling heuristic selection or training-based optimization of activation responses.

IV. PERFORMANCE AND SCALABILITY

The performance analysis evaluates how an ONN using integrated interferometer meshes and electro-optic activations scales with network depth and vector dimension. It focuses on power, latency, footprint, and computational speed, with activation thresholds characterized against gain and modulator Vπ.

The scalability analysis examines power consumption, computational latency, physical footprint, and computational speed as functions of layer count L and input dimension N.
The system analysis uses summarized parameter values to evaluate the ONN's performance and scaling characteristics.
Figure 3 maps constant activation-threshold contours against optical-to-electrical gain and modulator Vπ at photodetector responsivity R = 1.0 A/W.

A. Power consumption

The analysis separates ONN power consumption into optical-source and activation-circuit contributions, while treating mesh phase-shifter power as potentially negligible. Activation threshold and receiver-amplifier requirements determine the resulting power burden.

Power contributions: The power analysis focuses on optical-source supply and optical-to-electrical conversion circuits, assuming interferometer phase-shifter consumption can be negligible.This assumption is enabled by phase-change materials or ultra-low-power MEMS phase shifters.
Activation threshold: Activation threshold is defined as the minimum input optical power that triggers a nonlinear response.The threshold uses the phase shift required to produce a 50% transmission change relative to null input.
Activation threshold: 0.1 mW is the lowest activation threshold analyzed, requiring the ONN optical source to supply N · 0.1 mW.The threshold can be reduced using a small Vπ and large optical-to-electrical conversion gain.
Power contributions: 10–150 mW is the reported range for integrated optical receiver-amplifier power consumption, depending on implementation factors.This range motivates a conservative estimate for the optical-to-electrical conversion circuits across activations.

B. Latency

ONN latency combines propagation through the interferometer mesh with activation-function delay. The mesh term scales with network dimensions, whereas parallel activation circuits add a term independent of the vector dimension N.

Latency definition: ONN latency is the elapsed time from supplying input vector x0 to detecting prediction vector xL.For an integrated ONN, this is the travel time of an optical pulse through all L layers.
Latency scaling: The square interferometer mesh has propagation distance DW = N · DMZI, where DMZI is the length of each MZI.The activation layer additionally requires a delay line matching optical and electrical delays.
Latency scaling: The activation function adds latency independently of N because each activation circuit operates in parallel across all N-vector elements.The interferometer-mesh contribution retains the LN scaling predicted previously.
Operating assumptions: τoe = 100 ps is used as a conservative transimpedance-amplifier delay, with τrc ≈ 20 ps for a 50 GHz phase modulator.The analysis also assumes DMZI = 100 µm, neff = 3.5, and τnl = 0 ps.

C. Physical footprint

ONN footprint includes the quadratic interferometer mesh and linear activation-function contributions. The activation footprint can be dominated by delay-line length and electronic integration constraints.

Footprint model: The total ONN footprint combines the interferometer mesh area with the optical and electrical activation-function components.Electrical control lines are neglected in this analysis.
Footprint model: A = L · N^2 · AMZI + L · N · Af separates mesh area from activation-function area.AMZI is the area of one MZI, while Af is the area of one activation function.
Optical delay line: τopt = 120 ps corresponds to a waveguide delay-line length of approximately 1 cm for the activation function.A straight waveguide gives a large footprint but can maintain very low optical losses.
Electronic integration: The activation footprint transverse to propagation is dominated by optical-to-electrical conversion electronics, particularly the transimpedance amplifier.Amplifier-free receivers are proposed as one route toward tighter integration.
Footprint estimate: An N = 10 ONN layer is estimated at 11.0 mm × 0.6 mm when electronic transimpedance amplifiers are not integrated and Df ≤ DMZI = 60 µm.This estimate follows the stated footprint scaling under the assumed row-height constraint.

D. Speed

The electro-optic activation function is argued to preserve the speed of a linear ONN because high-speed detectors and modulators can support both input/output processing and interlayer transduction. Removing the matched delay line would trade footprint for speed.

Computational speed: The proposed activation function is argued to cause no speed degradation compared with a linear ONN using only interferometer meshes.The comparison is based on processing input vectors x0 per unit time.
Hardware operation: Integrated high-speed detectors and modulators can provide both ordinary ONN transduction and the optical-electrical/electrical-optical conversions required by the activation function.This supports using the same device classes between linear network layers.
Computational capacity: 10 GHz detector and modulator rates imply N^2·L·10^10 MAC/sec for an ONN, giving 10^12 MAC/sec at N = 10 for one layer.The estimate uses the equivalence between an N × N matrix multiplication and N^2 multiply-accumulate operations.
Speed–footprint trade-off: Removing the matched optical delay line can reduce activation footprint but limits speed to approximately (L · τele)^−1 when very long optical pulses are used.The speed then depends on the combined activation delay across all L nonlinear layers.

V. COMPARISON WITH THE KERR EFFECT

The paper compares its electro-optic activation with Kerr nonlinearities, showing that the electro-optic design offers tunable parameters that can strengthen the nonlinear response and lower activation thresholds.

The Kerr-effect alternative uses a nonlinear material response, whereas the electro-optic activation implements nonlinearity through a feedforward scheme.The Kerr effect is lossless and has no latency, while the electro-optic architecture provides tunability through electrical and optical design parameters.
The Kerr nonlinear parameter depends largely on waveguide design and material choice, while the electro-optic parameter has several adjustable degrees of freedom.The electro-optic design can vary tapped optical power, gain, responsivity, and modulator VπL.
The tapped-power fraction α should be minimized while keeping detected optical power above the photodetector’s noise-equivalent power.Increasing α raises the modulator voltage but also increases linear signal loss.
Tapping 10% of the optical power requires 20 dBΩ gain to match the nonlinear phase-shift threshold of a silicon Kerr waveguide with A = 0.05 µm2.Tapping only 1% requires an additional 10 dBΩ of gain for the same equivalence.
Reducing the phase-modulator VπL can further enhance electro-optic nonlinearity for a given applied voltage.The comparison in Fig. 4(b) treats VπL as an additional design parameter controlling phase-shift strength.

VI. MACHINE LEARNING TASKS

The paper evaluates its electro-optic activation on machine-learning tasks using numerical ONN simulations and physically measurable field quantities for training.

The study applies the electro-optic activation to an exclusive-OR logic task and a more complex machine-learning task.The XOR experiment is described in Sec. VI A, while the second task is considered in Sec. VI B.
The simulated ONNs are trained with neuroptica using on-chip backpropagation based only on physically measurable field quantities.Neuroptica is a custom Python simulator used to train the modeled networks.

A. Exclusive-OR Logic Function

A multilayer ONN with electro-optic activations learns a four-input XOR with very low error, while performance depends on the activation response and nonlinearity strength.

Exclusive-OR definition: A four-input XOR maps N binary inputs to one output, generalizing the two-input exclusive-OR relationship.The desired output is high when an odd-number pattern satisfies the multi-input XOR relation described in Fig. 5(b).
ONN architecture: The ONN uses L layers of N × N unitary interferometer meshes followed by N parallel electro-optic activations, retaining one final output.The lower N − 1 outputs are dropped after the final layer to produce y.
XOR learning result: The two-layer ONN learned the N = 4 XOR using gain g = 1.75π and biasing phase φb = π, with excellent agreement between learned and desired outputs.This configuration corresponds to the ReLU-like response shown in Fig. 2(a).
XOR learning result: The final MSE was below 10^-5 after training on all 16 binary input combinations.The 16 examples were processed in batches, and phase-shifter parameters were optimized by backpropagating the MSE gradient.
Activation-response dependence: Increasing nonlinearity improved XOR learning for the ReLU-like response, but very high nonlinearity broadened errors and could prevent convergence.For this response, the mean final MSE increased above gφ = 1.5π even as the best-case MSE continued to decrease.
Activation-response dependence: The activation response strongly affected performance: two other bias configurations produced substantially higher final MSE, demonstrating the importance of reconfigurability.The green response also improved with increasing nonlinearity, while its MSE range broadened from gφ = 1.0π.

B. Handwritten Digit Classification

The ONN classifies MNIST digits after reducing images to a small set of low-frequency Fourier coefficients. Adding electro-optic activation functions improves accuracy over a linear two-layer ONN, reaching 94% with a deeper trainable configuration.

Dataset and training: The ONN uses 70,000 grayscale 28×28 images of handwritten digits, with 60,000 training examples and 10,000 test examples.Training uses batches of 500 images.
Preprocessing and architecture: MNIST images are converted to Fourier space, and the N coefficients with smallest k are selected to reduce the ONN input size.The selected complex coefficients are fed into an L-layer ONN whose ten normalized output intensities form the digit probability distribution.
Activation-function benefit: 8% higher final validation accuracy is achieved with the activation function: 93% versus 85% for the linear ONN.This comparison uses a two-layer network with N = 16 Fourier components.
Compact representation: 93% prediction accuracy is obtained using only N = 16 complex Fourier components and 1024 free parameters.This accuracy is comparable with the 92.6% achieved by a fully connected linear classifier using 4010 parameters and all 784 real-space pixels.
Improved configuration: 94% testing accuracy is reached by adding a third ONN layer and making the activation-function gain trainable.The corresponding simulated system consumes 4.8 W, performs 7.7 × 10^12 MAC/sec, and has 1.5 ns prediction latency.

VII. CONCLUSION

The paper introduces an electro-optic architecture that synthesizes optical-to-optical nonlinearities for feed-forward ONNs. It taps a small portion of optical power for analog electrical processing and modulates the remaining original signal, supporting integrated implementation.

Contribution: The proposed architecture synthesizes optical-to-optical nonlinearities and serves as an activation function in a feed-forward ONN.Numerical simulations demonstrate its use on multi-input XOR and handwritten-digit classification tasks.
Operating principle: A tapped portion of optical input power undergoes analog electrical processing before modulating the remaining portion of the same optical signal.The architecture uses photodetectors and phase modulators rather than traditional all-optical nonlinearities.
Threshold and reconfigurability: The electro-optic approach uses electronic signal amplification to achieve a lower activation threshold than would be required from high optical signal powers.The conclusion contrasts this with the largely fixed responses of all-optical nonlinearities.
Integration: The activation architecture can use the same integrated photodetector and modulator technologies as fully integrated ONN input and output layers.This supports compatibility with integrated ONN implementations.