Image sensing with multilayer, nonlinear optical neural networks

• Conventional imaging is inefficient because images are redundant, contain shared structure, and include information irrelevant to the intended sensing task.Machine-vision image sensing seeks only a specific subset of each captured image’s information.Source: §I · p. 2
• Existing optical encoders and most demonstrated ONNs perform only linear operations, making end-to-end systems mathematically equivalent to at most a single neural-network layer.Although ONNs can compute densely connected operations in parallel and potentially much faster than electronic DNNs, suitable nonlinearities for large-scale deep networks remain a major challenge.Source: §I · pp. 2–3
• Optical preprocessing can reduce sensor bottlenecks: using C-fold fewer pixels can improve frame rate, photons per pixel, power, size, weight, cost, and decision latency by roughly C-fold.Compression beyond 10 times is potentially feasible, but simple masks rarely provide sufficient computational capacity.Source: §I · p. 2
• Depth and nonlinearity are essential for efficient high-performance image processing because deep nonlinear networks can approximate practically relevant functions exponentially more efficiently than single-layer networks.Source: §I · p. 3
• 800:1 compression: A multilayer ONN sensor uses optical-to-optical nonlinear activation to conditionally encode images into low-dimensional latent features in a single shot.The system uses a commercial image intensifier as its nonlinear activation and supports incoherent optical inputs through microlens-array fan-out.Source: §I · p. 3
• High compression ratios: The nonlinear ONN consistently outperforms conventional image sensing and linear optical preprocessing across machine-vision datasets, flow-cytometry classification, and real-scene object detection and measurement.Source: §I · p. 3

Architecture

• A 2-layer fully connected ONN maps incoherent-light images into a lower-dimensional latent space while preserving information relevant to the sensing task rather than losslessly compressing images.A CMOS camera and digital post-processing read out the optical pre-processor.Source: §II · p. 3

Optical processing

• The device is designed for broadband, incoherent visible-light images, using microlens fanout to create copies of the input image for optical processing.Weighted copies are attenuated according to matrix elements, then focused by a lens to perform the optical summations.Source: §II · pp. 3, 5

Nonlinear activation

• The saturating microchannel plate in a commercial image intensifier provides local optical-to-optical nonlinear activation between the two optical matrix-vector multiplication layers.Its saturation produces a response qualitatively similar to the positive half of a sigmoid while preserving spatial parallelism and avoiding separate electronic read-out and re-injection.Source: Figure 1 · p. 4, §II · p. 5

Experimental comparison

• A beamsplitter enables reconfiguration of the sensor for direct imaging, single-layer linear ONN preprocessing, or the full multilayer nonlinear ONN encoder.The configurations are selected by changing the first-layer LCD from transparent to weighted transmission before the intensifier activation.Source: §II · p. 5

QuickDraw classification

• 79% QuickDraw test accuracy surpassed optical linear encoders (69.5%) and optimized digital linear encoders (74%), approaching ideal digital multilayer accuracy (82%).All encoders used a 4-dimensional latent space corresponding to 196:1 compression.Source: A. Nonlinear ONN encoders extract… · pp. 5, 7, Figure 2 · p. 6

Flow-cytometry classification

• 93% flow-cytometry test accuracy exceeded the linear ONN encoder’s 88.5% while compressing cell images at an effective 400:1 ratio.The nonlinear encoder also produced higher local density within cell-class clusters.Source: Figure 2 · p. 6, A. Nonlinear ONN encoders extract… · p. 7

Real-scene traffic-sign classification

• Across viewing angles from 0 to 80 degrees, the nonlinear multilayer ONN identified speed-limit signs better than the linear ONN in a real 3D-printed scene.The experiment processed photons from real 3D objects, although the microlens array’s limited field of view primarily captured each speed-limit sign.Source: Figure 2 · p. 6, A. Nonlinear ONN encoders extract… · p. 8

Image reconstruction

• The QuickDraw classification encoder supported reconstructions that captured coarse within-class attributes, including chair and hurricane orientation or shape, despite missing fine details.The encoder was trained only for classification, but its features preserved additional image structure.Source: B. The same optically compressed… · p. 8

Viewpoint estimation

• The traffic-sign encoder produced very accurate viewing-angle predictions, although performance decreased when predicting angles across all speed-limit classes simultaneously.A new digital backend was trained while reusing the previously trained multilayer optical encoder.Source: B. The same optically compressed… · p. 8

Anomaly detection

• Spectral clustering of the encoder’s normalized 4-dimensional features reliably identified anomalous doublet cell clusters as an additional class.The six prominent clusters represented five trained cell-organelle classes plus one anomaly class, showing preservation of structure beyond the initial labels without abandoning low-dimensional compression.Source: B. The same optically compressed… · p. 8

Image-sensing performance

• A 2-layer nonlinear ONN pre-processor consistently outperforms conventional imaging with direct down-sampling or linear-ONN pre-processing across a wide range of image-sensing tasks.This result is reported for the experiments summarized in Figures 2 and 3.Source: §III · pp. 8–9

Encoder reuse

• Classification-trained nonlinear ONN encoders can support new sensing tasks by retaining their weights while only new digital backends are trained.The same features enable image reconstruction, anomalous cell-organelle doublet detection, and viewing-angle inference for speed-limit signs.Source: Figure 3 · p. 9

Depth scaling

• At a compression ratio of 104, the 5-layer CNN3 pre-processor achieves nearly double the accuracy of shallower networks, with deeper models generally performing better across compression ratios.Figure 4 compares linear, MLP, CNN1, CNN3, and ResNet-based models for 10-class cell-organelle classification.Source: Figure 4 · p. 10, §III · pp. 10–11
• The depth-scaling simulations use a more challenging 10-class organelle dataset while modeling physical noise and strictly non-negative weights required for incoherent-light ONNs.Compression ratio is varied by changing the final optical layer’s output-neuron count, which sets the required pixel or photodetector count.Source: §III · p. 10

IV. DISCUSSION

• Nonlinear multilayer ONN pre-processors outperform direct-imaging and linear-optical sensors at high compression across diverse image-sensing tasks, with advantages that scale favorably with additional layers.They extend end-to-end image-system optimization to more powerful nonlinear optical pre-processing.Source: §IV · p. 11
• ONN sensors may reduce system complexity by preserving only task-relevant information and adapting to distortions through parameter adjustments, enabling smaller, cheaper, easier-to-manufacture optoelectronic systems.This contrasts with conventional sensors optimized to preserve potentially relevant information agnostically.Source: §IV · p. 11
• Recent advances in 2D-material optoelectronics, metasurfaces, VCSEL arrays, silicon nanophotonics, and micro-optoelectronics are making compact, mass-manufacturable ONN devices increasingly feasible.Demand for optical deep-learning accelerators is also driving platforms that process spatial, temporal, frequency, and mixed-domain information.Source: §IV · p. 11
• Emerging ONN platforms could extend optical sensing beyond incoherent images to spectroscopy, hyperspectral, LiDAR, and coherent inputs, while integrated ONNs could replace parts of the digital backend.Fully optical intelligent sensors are projected to reach multi-THz bandwidth, gigapixel effective spatial resolution, and picosecond-scale latency.Source: §IV · pp. 11–12

DATA AND CODE AVAILABILITY

• Demonstration, training, and data resources for the optical and digital neural networks are publicly available through the cited GitHub repository and Zenodo archive.The available materials cover data gathering, training, and all data generated in the work.Source: DATA AND CODE AVAILABILITY · p. 12

COMPETING INTERESTS

• T.W., Mandar M.S., L.G.W. and P.L.M. are listed as inventors on a U.S. provisional patent application for nonlinear optical neural network pre-processors for imaging.The application is identified as Serial No. 63/392,042.Source: COMPETING INTERESTS · p. 12

Multilayer optical-neural-network image pre-processor

• The pre-processor chains two optical matrix-vector multipliers around an image-intensifier optical-to-optical nonlinear activation before camera readout in a compressed latent space.The image intensifier provides gains around 800, resets the number of spatial optical modes, and uses saturation-based nonlinear responses calibrated across 36 illuminated regions.Source: Multilayer optical-neural-network… · pp. 17–18

QuickDraw image classification

• QuickDraw classification used a 1,600→36→4 multilayer optical encoder with nonlinear activations, compared against linear optical, direct-imaging, and digital baselines.The images were resized to 100×100 pixels, binarized, and displayed on a DMD; experimentally measured fanned-out inputs supplied training ground truth.Source: QuickDraw image classification · pp. 18–19

Optical-neural-network training

• Training used a calibrated digital twin with augmentation for optical imperfections, noise modeling, and layer-by-layer fine-tuning from experimentally collected activations.Augmentations included translations of ±5% and zoom mismatches of ±4%, with approximately 2% activation noise after both layers.Source: Optical-neural-network training · pp. 19–20

Flow-cytometry image classification

• Flow-cytometry experiments classified five cell-organelle classes using 1,000 hand-selected training images and held-out images displayed as binarized DMD inputs.The dataset included nucleolus, cytoplasm, centrosomes, cell mask, and mitochondria classes, while multi-cell images were later reused for anomaly detection.Source: Flow-cytometry image classification · pp. 20–21

Real-scene image classification

• Real-scene experiments classified eight speed-limit signs across viewing angles using a zoom-imaged physical scene and a two-dimensional compressed representation.The dataset contained 536 training and 176 validation images, with the compressed dimension reduced from four to two for this task.Source: Real-scene image classification · p. 21

Additional image sensing tasks based on different digital backends

• Additional digital backends reconstructed QuickDraw images, clustered cell-organelle feature vectors for anomaly detection, and estimated speed-sign viewing angles from two-dimensional features.Reconstruction used a learned multilayer decoder; anomaly detection used spectral clustering; angle estimation used a 2→50→100→1 perceptron with L1 loss for individual sign classes.Source: Additional image sensing tasks based… · pp. 21–22

Simulation of Deeper Optical Neural Networks for 10-class cell-organelle classification

• Deeper simulated ONN encoders generally produced higher-quality compression, especially at very high compression ratios, for 10-class cell-organelle classification.The comparison used linear, multilayer, and convolutional architectures with identical linear decoders; compression ratio was 10,000/N.Source: Simulation of Deeper Optical Neural… · pp. 22–23

Supplementary Note 1. OVERVIEW OF THE EXPERIMENTAL SETUP

• The encoder used two fully optical linear layers separated by an element-wise nonlinear image-intensifier activation.Each linear layer implemented optical matrix-vector multiplication, while the image intensifier provided the nonlinear activations.Source: Supplementary Note 1. OVERVIEW OF THE… · pp. 26–27

Supplementary Note 2. A FULLY OPTICAL MATRIX-VECTOR MULTIPLIER

• Matrix-vector multiplication decomposed into optical fan-out, LCD-based element-wise modulation, and optical fan-in, with input vectors represented by 40 × 40 = 1,600 pixels.Microlens arrays produced optical copies, LCD patches encoded matrix columns, and pooled detector energy completed the multiplication.Source: Supplementary Note 2. A FULLY OPTICAL… · p. 29, Figure 3 · p. 30

Supplementary Note 3. CALIBRATION OF OPTICAL MATRIX-VECTOR MULTIPLIERS

• The optical multipliers were calibrated by comparing measured neuron outputs with digitally computed outputs, while their best measured transmission was 2.9%.The reported optical loss did not support low-light operation, although improved imaging and phase modulation could increase transmission.Source: Supplementary Note 3. CALIBRATION OF… · pp. 30, 31, 33, 34

Supplementary Note 4. NONLINEAR ACTIVATION FUNCTIONS VIA AN IMAGE INTENSIFIER

• The image intensifier supplied spatially resolved nonlinear amplification, with a 21.7 µs decay time and nonlinear output response at a bandwidth of at least 16 kHz.Its nonlinear curves were relatively uniform across the 36 measured hidden-layer neurons, while neighboring-neuron cross-talk was reduced by blocking alternate optical copies.Source: Supplementary Note 4. NONLINEAR… · pp. 34, 36, 37

Supplementary Note 5. THE GENERAL PROCEDURE FOR TRAINING OPTICAL-NEURAL-NETWORK ENCODERS

• The general encoder used 1,600-pixel inputs, a 36-neuron hidden layer, and a 4-neuron bottleneck, with nonnegative weights, simulated double-exponential activations, and jointly trained digital backends.Training added optical-layer input noise of up to 0.02 and random ±5% translations and ±4% scaling to improve robustness and regularization.Source: Supplementary Note 5. THE GENERAL… · pp. 37–38
• Layer-by-layer retraining with experimentally collected nonlinear activations reduced the simulation-to-physical gap before final testing on held-out images.The protocol sequentially retrained the second optical layer and backend using CAM2 activations, then retrained the backend using CAM3 outputs.Source: Supplementary Note 5. THE GENERAL… · p. 38

Supplementary Note 6. RECOGNITION OF HAND-DRAWN FIGURES

• QuickDraw recognition used 10 hand-drawn object classes, with 3,000 training-validation images and 500 held-out test images after binarization and resizing.Ground-truth images captured the blurrier 40 × 40 patterns resolved by the LCD, which were used to train the uploaded optical weights.Source: Supplementary Note 6. RECOGNITION OF… · pp. 38–40

Supplementary Note 7. IMAGE-BASED CELL-ORGANELLE CLASSIFICATION

• Cell-organelle classification distinguished five stained HeLa-cell classes using 1,000 training-validation images and 200 additional test images, with the same binarization and encoder procedure.The classes were nucleolus, cytoplasm, centrosomes, cell mask, and mitochondria.Source: Supplementary Note 7. IMAGE-BASED… · p. 40

Supplementary Note 8. CLASSIFICATION OF 3D REAL-SCENE OBJECTS

• Real-scene speed-limit classification used eight sign classes across 712 viewing-angle images, with a 2-neuron bottleneck and a two-layer 2 →40 →8 digital decoder.The sign image was demagnified to approximately 1 mm × 1 mm so its optically fanned-out copies filled 40 × 40 LCD pixels.Source: Supplementary Note 8. CLASSIFICATION… · pp. 40–41

Supplementary Note 9. AUTOENCODERS FOR IMAGE RECONSTRUCTION

• A three-hidden-layer digital decoder reconstructed 28 × 28 QuickDraw images from the encoder’s 4-dimensional classification features, while batch normalization significantly improved performance.The latent space preserved features such as chair orientation, tent floors, and hurricane outlines, but reconstruction was largely limited by the 196:1 compression ratio.Source: Supplementary Note 9. AUTOENCODERS FOR… · pp. 42–45

Supplementary Note 10. ANOMALY DETECTION USING UNSUPERVISED LEARNING

• Using the unchanged cell-classification encoder, anomalous multi-cell images formed a separate latent cluster, and spectral clustering recovered six clusters corresponding to five normal classes plus anomalies.The latent vectors were 4-dimensional, and the first three principal components were visualized before clustering.Source: Supplementary Note 10. ANOMALY… · pp. 45–47, Figure 23 · p. 54

Supplementary Note 11. NONLINEAR PARAMETER REGRESSION

• A 2 →50 →100 →1 digital backend regressed viewing angle from the speed-limit classifier’s latent features using alternating even-angle training and odd-angle validation.The task minimized L1 loss, and angle-prediction performance decreased when predicting all sign classes jointly rather than one class at a time.Source: Supplementary Note 11. NONLINEAR… · pp. 48–49

CELL-ORGANELLE CLASSIFICATION TASK

• The simulations classified 93,050 cropped images across 10 cell-organelle classes, using 83,750 images for training and 9,300 for testing.Images included fields with no cell or multiple cells; two classes were excluded because of insufficient images or absent fluorescent signal.Source: CELL-ORGANELLE CLASSIFICATION TASK · p. 54
• Models varied bottleneck dimension and optical depth, with a linear non-negative encoder providing an approximately 70% accuracy baseline without nonlinearity.The digital backend was a fully connected N-to-10 layer, while approximately 2% relative input noise was added to each optical layer.Source: CELL-ORGANELLE CLASSIFICATION TASK · p. 55
• CNN1 achieved over 90% test accuracy versus 84% for MLP, demonstrating the benefit of adding an optical convolutional layer.CNN1 uses a 7 × 7 non-negative kernel, shifted ReLU activation, and primarily AvgPool operations.Source: CELL-ORGANELLE CLASSIFICATION TASK · pp. 56–57
• CNN3’s two additional non-negative convolutional layers enabled higher accuracy than CNN1, especially at high compression ratios.CNN3 is a five-layer convolutional network using one MaxPool operation after its first convolutional layer.Source: CELL-ORGANELLE CLASSIFICATION TASK · p. 57
• Practically realizing more powerful ONN encoders will require jointly designing compact hardware and optics-friendly neural-network architectures rather than directly adapting digital networks.The simulated ONN designs are described as ultimately speculative.Source: CELL-ORGANELLE CLASSIFICATION TASK · p. 57