Source-linked AI summary
Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit
Tiankuang Zhou, Xing Lin, Jiamin Wu, Yitong Chen, Hao Xie, Yipeng Li, Jintao Fan, Huaqiang Wu, Lu Fang, Qionghai Dai
TL;DR
Existing optical neural processors have limited complexity and generally require dedicated designs for different neural-network models. This paper introduces a reconfigurable diffractive processing unit that supports large-scale optical networks, achieving benchmark digit and human-action recognition with accuracy reaching or exceeding advanced electronic methods.
Problem
Existing optical neural-network architectures have limited complexity and performance and cannot be reconfigured for different models after deployment.
Method
The authors construct a programmable diffractive processing unit with adaptive training to reconfigure feedforward and recurrent optical neural networks.
Results
The reconfigurable networks performed video-rate handwritten-digit and human-action recognition with accuracy reaching and sometimes exceeding advanced electronic computing methods.
Takeaways & Limitations
The DPU demonstrates a reconfigurable optical-computing approach that supports multiple neural-network architectures and benchmark recognition tasks.
Takeaways & Limitations
The demonstrated recurrent architecture does not yet address long-term-memory forgetting, motivating incorporation of an LSTM unit for longer sequences.
Abstract
from arXiv · showhide
Application-specific optical processors have been considered disruptive technologies for modern computing that can fundamentally accelerate the development of artificial intelligence (AI) by offering substantially improved computing performance. Recent advancements in optical neural network architectures for neural information processing have been applied to perform various machine learning tasks. However, the existing architectures have limited complexity and performance; and each of them requires its own dedicated design that cannot be reconfigured to switch between different neural network models for different applications after deployment. Here, we propose an optoelectronic reconfigurable computing paradigm by constructing a diffractive processing unit (DPU) that can efficiently support different neural networks and achieve a high model complexity with millions of neurons. It allocates almost all of its computational operations optically and achieves extremely high speed of data modulation and large-scale network parameter updating by dynamically programming optical modulators and photodetectors. We demonstrated the reconfiguration of the DPU to implement various diffractive feedforward and recurrent neural networks and developed a novel adaptive training approach to circumvent the system imperfections. We applied the trained networks for high-speed classifying of handwritten digit images and human action videos over benchmark datasets, and the experimental results revealed a comparable classification accuracy to the electronic computing approaches. Furthermore, our prototype system built with off-the-shelf optoelectronic components surpasses the performance of state-of-the-art graphics processing units (GPUs) by several times on computing speed and more than an order of magnitude on system energy efficiency.
Optoelectronic implementation of the reconfigurable diffractive processing unit
The DPU is a programmable optoelectronic neuromorphic processor that uses large-scale diffractive neurons and weighted optical interconnections to support diverse, high-complexity neural networks. High-throughput optoelectronic devices and optical computation enable video-rate inference, temporal multiplexing, and adaptation to system imperfections.
- Architecture: The DPU combines large-scale diffractive perceptron neurons with weighted optical interconnections as a programmable building block for diverse diffractive neural networks.Its design targets high model complexity and accuracy.
- High-throughput implementation: Gigabytes-per-second optoelectronic devices enable high-speed neural-network configuration and video-rate inference.The system uses a digital micromirror device and spatial light modulator for processing image and video signals.
- Reconfigurability: Massively parallel optoelectronic dataflow allows temporal multiplexing and programming of different optical neural-network architectures.The unit is controlled and buffered to customize network configurations.
- Optical computation: Almost all computational operations are performed optically, improving computing speed and system energy efficiency compared with existing electronic architectures.The optical allocation of computation is a central feature of the proposed optoelectronic AI architecture.
- System capabilities: The DPU supports programmable neural-network parameters, high-throughput devices, and massive neuron counts while adapting to optical-system imperfections.These capabilities arise from integrating diffractive optical elements into optical neural networks.
Adaptive training of optoelectronic diffractive deep neural network (D2NN)
Adaptive training enabled a three-layer optoelectronic D2NN to compensate for system imperfections and classify MNIST digits at 56 fps with 97.6% blind testing accuracy. The approach transferred a pre-trained model, experimentally fine-tuned its parameters, and updated SLM phase patterns to reduce accumulated errors.
- Model transfer: The pre-trained model was transferred to the optoelectronic D2NN by deploying its network structure and programming SLMs with layer-specific parameters.Model-transfer errors distorted wavefront connections, so sequential adaptive-optics-based compensation was derived to alleviate error accumulation.
- MNIST classification: A three-layer optoelectronic D2NN classified MNIST at 56 fps and achieved 97.6% blind testing accuracy on 10,000 digit images.The model was trained in silico on 55,000 MNIST training images and mapped digits from “0” to “9” into ten predefined regions.
- Adaptive fine-tuning: Adaptive training used two-stage fine-tuning of a pre-trained model to circumvent system error and improve recognition performance.Training could use either the full training set or a mini-training set containing 2% of the data, trading experimental accuracy against training efficiency.
- System adaptation: After each fine-tuning stage, refined parameters updated the SLM phase patterns to adapt the system to imperfections and reduce accumulated system errors.Adaptive training made DPU output intensity distributions between simulations and experiments more matched, especially at the last layer.
High-accuracy object classification with a diffractive network in networks (D-NIN-1)
D-NIN-1 enhances diffractive-network inference by using multiple feature maps, weighted external connections, and a DPU read-out layer for final decisions. On MNIST, the three-layer architecture improves accuracy and robustness over a three-layer D2NN, reaching 96.8% blind-testing accuracy after adaptive training.
- Feature-map fusion: Weighted summation of DPU outputs fuses the multi-channel diffractive feature maps for subsequent hidden-layer processing.The external connections implement weighted summation for the input feature maps of previous layers.
- Network connectivity: Complex internal and external neuron connectivity computes increasingly abstract hidden-layer features, followed by a DPU read-out layer for final decisions.The read-out layer produces the D-NIN-1 network outputs.
- Architecture and evaluation: Three-layer D-NIN-1 uses three diffractive feature maps in each hidden layer to improve MNIST classification accuracy and robustness over a three-layer D2NN.The feature maps correspond to three DPU layers in the demonstrated architecture.
- Experimental result: 96.8% blind testing accuracy was achieved across the whole test dataset after adaptive training with the programmed optoelectronic DPU system.Stacking multiple DPU layers at each hidden layer provides additional degrees of freedom and robustness for fine-tuning the pretrained model against system imperfections.
Configuring a diffractive RNNs (D-RNN) for human action recognition
The reconfigurable DPU was configured as a large-scale diffractive recurrent neural network for video-based human action recognition. Evaluated on Weizmann and KTH benchmarks, it achieved 88.9% blind testing frame accuracy on both databases, while D-RNN++ enhanced robustness with an electronic read-out layer.
- D-RNN configuration: The DPU’s recurrent connections enabled a standard D-RNN architecture for high-accuracy recognition of video sequences.The folded and unfolded D-RNN representations are shown in Figures 1e and 4.
- Benchmark evaluation: The D-RNN was evaluated on the Weizmann and KTH benchmark databases after preprocessing their video sequences for network input.Weizmann contains ten natural human-action categories and was split into 60 training and 30 test videos, with 30–100 frames per sequence.
- Benchmark evaluation: 88.9% blind testing frame accuracy was achieved for both Weizmann and KTH, corresponding to video accuracies of 100% and 94.4% for the two models.The selected sequence lengths were 3 for Weizmann and 5 for KTH, with an optimal fusing coefficient of 0.2 for both databases.
- D-RNN++ enhancement: D-RNN++ enhanced recognition accuracy and robustness by transferring the trained D-RNN hidden layer and replacing the DPU read-out layer with an electronic read-out layer.The electronic layer uses the last hidden state as input and learns its fully connected weights with fast ridge regression.
Discussion
The DPU used 8-bit phase modulation and 16-bit optical-field measurement, with binary inputs supplied by a DMD. Its speed, efficiency, input precision, and network capabilities could be further improved through advanced optoelectronic hardware and connectivity designs.
- Implementation: The system used 8-bit phase modulation accuracy, 16-bit optical-field measurement accuracy, and a DMD for binary unit inputs.The DMD was selected for high optical contrast, high speed, and easy calibration.
- Future improvements: Higher-throughput detectors, tuneable metasurface SLMs, and ASICs for dataflow control and unit programmability could further improve computing speed and energy efficiency.The passage gives SPAD arrays and tuneable metasurface SLMs as example optoelectronic devices.
- Future improvements: More advanced internal and external connectivities could expand D-NIN functionality by encoding multiple diffractive feature maps with multiple wavelengths.This is presented as an example of a possible connectivity improvement.
- Future improvements: Incorporating an LSTM unit into the D-RNN could reduce forgetting of long-term memory and enable longer input sequences.The passage specifically proposes LSTM integration for the D-RNN.
Conclusion
The study experimentally demonstrates a reconfigurable optoelectronic processor that adapts to different ANN architectures for large-scale optical neural information processing. High-bandwidth optoelectronic devices and adaptive training address insufficient model complexity and experimental model deviations.
- Conclusion: The DPU was experimentally demonstrated as a reconfigurable optoelectronic computing processor.It can be programmed to support different ANN architectures.
- Conclusion: The processor adapts to different ANN architectures for large-scale optical neural information processing.
- Conclusion: High-bandwidth optoelectronic devices and a novel adaptive training technique address insufficient model complexity and experimental model deviations.