PaperlayerView exact revision on arXiv

Source-linked AI summary

Image sensing with multilayer, nonlinear optical neural networks

Tianyu Wang, Mandar M. Sohoni, Logan G. Wright, Martin M. Stein, Shi-Yuan Ma, Tatsuhiro Onodera, Maxwell G. Anderson, Peter L. McMahon

arXiv:2207.14293v1physics.opticscs.ETcs.LG

TL;DR

Image sensors need to extract task-relevant information without digitizing large, redundant images, but ONN sensors had been limited by linear processing. The paper realizes a multilayer ONN sensor with an image-intensifier nonlinearity, achieving high-accuracy sensing at compression ratios up to 800:1 and outperforming linear preprocessing across diverse tasks.

  • Problem

    Conventional image sensing digitizes high-dimensional images, while ONN sensors lacked a suitable nonlinearity for deep, multilayer processing.

  • Method

    The authors build a multilayer ONN image sensor using a commercial image intensifier as an optical-to-optical nonlinear activation for incoherent-light images.

  • Results

    Across QuickDraw, flow-cytometry, and real-scene tasks, the nonlinear multilayer encoder outperformed linear optical preprocessing at high compression, reaching up to 800:1 compression.

  • Takeaways & Limitations

    Nonlinear multilayer ONN encoders can provide low-dimensional optical preprocessing for diverse image-sensing applications.

Abstract

from arXiv · show

Optical imaging is commonly used for both scientific and technological applications across industry and academia. In image sensing, a measurement, such as of an object's position, is performed by computational analysis of a digitized image. An emerging image-sensing paradigm breaks this delineation between data collection and analysis by designing optical components to perform not imaging, but encoding. By optically encoding images into a compressed, low-dimensional latent space suitable for efficient post-analysis, these image sensors can operate with fewer pixels and fewer photons, allowing higher-throughput, lower-latency operation. Optical neural networks (ONNs) offer a platform for processing data in the analog, optical domain. ONN-based sensors have however been limited to linear processing, but nonlinearity is a prerequisite for depth, and multilayer NNs significantly outperform shallow NNs on many tasks. Here, we realize a multilayer ONN pre-processor for image sensing, using a commercial image intensifier as a parallel optoelectronic, optical-to-optical nonlinear activation function. We demonstrate that the nonlinear ONN pre-processor can achieve compression ratios of up to 800:1 while still enabling high accuracy across several representative computer-vision tasks, including machine-vision benchmarks, flow-cytometry image classification, and identification of objects in real scenes. In all cases we find that the ONN's nonlinearity and depth allowed it to outperform a purely linear ONN encoder. Although our experiments are specialized to ONN sensors for incoherent-light images, alternative ONN platforms should facilitate a range of ONN sensors. These ONN sensors may surpass conventional sensors by pre-processing optical information in spatial, temporal, and/or spectral dimensions, potentially with coherent and quantum qualities, all natively in the optical domain.

I. INTRODUCTION

Conventional imaging digitizes large, information-redundant images before analysis, motivating optical encoders that compress images to task-relevant representations. This work applies multilayer, nonlinear ONN preprocessing to image sensing and reports compression up to 800:1 with strong performance across tasks.

  • Motivation: Conventional cameras digitize large images even though natural-image sparsity, shared structure, and task irrelevance make much of the information redundant.Image sensing seeks only a specific subset of each image for machine-vision tasks.
  • Motivation: Optical encoders preprocess images before digitization, targeting faster, smaller, and more energy-efficient image sensors.Reducing pixel count can improve frame rate, photons per pixel, power, size, weight, cost, and decision latency.
  • Motivation: ONNs perform densely connected neural-network operations in parallel in the optical domain, with potential speed advantages over electronic inference.They exploit many optical modes across space, time, or frequency.
  • Motivation: Developing a nonlinearity suitable for large-scale, deep ONNs remains a major challenge, while depth and nonlinearity are important for efficient neural-network image processing.Linear ONN preprocessing is mathematically equivalent to at most a single neural-network layer.
  • Contribution: The demonstrated sensor uses an optical-to-optical nonlinear activation to compress image data into a low-dimensional latent space, reaching compression ratios up to 800:1.It outperformed conventional image sensing and linear optical preprocessing across machine-vision, flow-cytometry, and real-scene tasks.

II. AN ONN-BASED IMAGE SENSOR WITH OPTICAL-TO-OPTICAL NONLINEARITY

The sensor implements a two-layer fully connected ONN that converts incoherent-light images into lower-dimensional optical features before camera readout and digital post-processing.

  • Architecture: A saturating microchannel plate inside a commercial image intensifier provides the optical-to-optical element-wise nonlinear activation after the first ONN layer.The resulting system is a two-layer fully connected optical neural network.
  • Architecture: The optical pre-processor maps incoherent-light input images into an abstract, lower-dimensional latent space.A CMOS camera performs sensor readout, followed by digital post-processing.
  • Optical input: A microlens array implements fanout by effectively creating copies of the input image for optical processing.The design targets broadband, incoherent visible-light images.
  • Optical computation: Optical matrix-vector multiplication attenuates copied image components according to weights and focuses their sums to form output-vector elements.These linear operations are followed by focusing light onto the image intensifier for nonlinear activation.
  • Experimental comparison: A beamsplitter enables reconfiguration for direct imaging, single-layer linear preprocessing, or multilayer nonlinear preprocessing.This provides matched configurations for comparing the three sensor modes.

A. Nonlinear ONN encoders extract information more effectively than linear encoders

Across benchmark, flow-cytometry, and real-scene tasks, the multilayer nonlinear ONN encoder extracted information more effectively than linear or direct-imaging frontends at matched low-dimensional bottlenecks.

  • QuickDraw classification: At a 4-dimensional latent bottleneck corresponding to 196:1 compression, the nonlinear multilayer ONN achieved better QuickDraw classification accuracy than the comparison sensors.Comparisons included direct imaging, linear ONN, and digital encoder frontends.
  • QuickDraw classification: 79% versus 82% test accuracy compared the experimental nonlinear ONN encoder with an ideal digital multilayer encoder.The experimental device was slightly less accurate than the ideal digital reference.
  • Flow-cytometry classification: 400:1 compression produced five-class cell-organelle classification accuracy better than the linear ONN encoder.The images came from flow-cytometry experiments.
  • Real-scene recognition: Across viewing angles from 0 to 80 degrees, the nonlinear multilayer encoder identified speed-limit signs better than the linear ONN encoder.The evaluation used a real-model three-dimensional scene.

B. The same optically compressed features enable a variety of image-sensing applications

The same optically compressed features supported multiple downstream image-sensing tasks through newly trained digital post-processors, including reconstruction, viewpoint prediction, and anomaly detection.

  • Image reconstruction: A new digital decoder reconstructed coarse intra-class QuickDraw attributes from features produced by an encoder trained only for classification.Reconstructions often lacked specific details but preserved orientations or shapes.
  • Viewpoint prediction: The traffic-sign encoder supported very accurate viewing-angle prediction after training a new digital backend.Performance decreased when predicting angle across all speed-limit classes rather than one class at a time.
  • Anomaly detection: Spectral clustering of four-dimensional cell-organelle features reliably identified anomalous doublet-cell images as an additional class.The result indicates that the encoder preserved structure beyond the originally trained classes.

III. MULTILAYER ONN SENSORS SCALE FAVORABLY TO COMPLEX IMAGE SENSING TASKS

The study evaluates multilayer nonlinear ONN pre-processors on complex image-sensing tasks and finds that increasing depth improves performance, especially under severe compression.

  • Task performance: A 2-layer nonlinear ONN pre-processor consistently outperformed direct imaging and linear ONN pre-processing across a wide range of image-sensing tasks.The encoder was evaluated on classification and other sensing tasks, including reconstructed images and anomaly detection.
  • Depth scaling: Deeper nonlinear ONN encoders produced higher classification accuracy across a wide range of compression ratios.The compared models included linear, MLP, CNN1, CNN3, and a digital ResNet reference.
  • Simulation setup: The simulations extended organelle classification to 10 classes and modeled physical noise with strictly non-negative weights for incoherent-light operation.The dataset was selected because it is more challenging than the earlier 5-class task.
  • Compression control: Compression was varied by changing the number of output neurons in the final optical layer, which sets the required photosensor pixel or photodetector count.A fully digital ResNet served as a reference for achievable performance.
  • Depth scaling: At a compression ratio of 10^4, the 5-layer CNN3 achieved nearly double the accuracy of shallower networks.The compression ratio corresponds to a bottleneck dimension of 1.

IV. DISCUSSION

The discussion presents nonlinear multilayer ONN sensors as high-compression optical frontends with favorable scaling and potential extensions beyond incoherent-light imaging.

  • Discussion: At high compression ratios, nonlinear multilayer ONN pre-processors outperformed both conventional direct imaging and linear optical pre-processing across diverse sensing tasks.The authors attribute the improvement to better extraction of image information in the high-compression regime.
  • System implications: ONN sensors may require preserving only task-relevant information, while parameter adjustment may accommodate distortions and manufacturing imperfections.The authors state that this could enable smaller, cheaper, and easier-to-manufacture optoelectronic systems in principle.
  • Technology outlook: Recent advances in optoelectronics and nanophotonics are making low-cost, small-footprint, mass-manufacturable ONN realizations increasingly feasible.The discussion highlights 2D-material optoelectronics, metasurfaces, VCSEL arrays, silicon nanophotonics, and micro-optoelectronics.
  • Technology outlook: Emerging ONN platforms could extend optical preprocessing to spectroscopic, hyperspectral, LiDAR, and coherent inputs.The proposed sensor paradigm is therefore not limited to incoherent-light images.
  • Technology outlook: Future all-optical intelligent sensors could bypass electronic bottlenecks, with projected operation at multi-THz bandwidth, gigapixel effective spatial resolution, and picosecond-scale latency.These figures are presented as a future possibility for integrated and free-space-facing ONN combinations.
  • Reproducibility: The work provides publicly available code and data for data gathering, training, and the optical/digital neural networks.The resources are linked through GitHub and Zenodo.

COMPETING INTERESTS

The authors disclose inventorship on a U.S. provisional patent application concerning nonlinear optical neural-network pre-processors for imaging.

  • Competing interests: T.W., Mandar M.S., L.G.W., and P.L.M. are listed as inventors on a U.S. provisional patent application for nonlinear optical neural-network pre-processors for imaging.The application is identified by Serial No. 63/392,042.

METHODS

The study implements a multilayer optical encoder using matrix-vector multiplication, an image-intensifier nonlinearity, and camera readout, then evaluates it across classification and reconstruction tasks. The experimental and simulated architectures vary depth, compression, and digital decoding to assess image-sensing performance.

  • Multilayer optical-neural-network image pre-processor: The optical pre-processor combines two optical matrix-vector multipliers with an optical-to-optical nonlinear activation and camera detection in a compressed latent space.The first multiplier and nonlinear activation precede the second multiplier; digital post-processing follows camera readout.
  • Multilayer optical-neural-network image pre-processor: The image intensifier supplies the nonlinear activation through local photoelectron amplification and saturation on its phosphor output.Its measured input-output gain was around 800, supporting multilayer operation and low-light use.
  • Multilayer optical-neural-network image pre-processor: Most experiments use a 1,600-dimensional input, a 1,600 by 36 first-layer matrix, and a 36 by 4 second-layer matrix, except for traffic signs using 36 by 2.The matrix convention identifies input length first and output dimension second.
  • QuickDraw image classification: QuickDraw evaluation uses selected 10-class images that are resized, binarized, displayed on a DMD, and divided into training, validation, and test sets.The dataset contains 3,000 training images and 500 test images after manual filtering.
  • Training and calibration: ONN weights are trained with a digital twin of the optical layers, including digital post-processing parameters, while measured optical inputs account for imaging imperfections.The measured inputs compensate for resolution limits and microlens-array aberrations affecting optically fanned-out images.
  • Image reconstruction: The 4-dimensional encoded features are decoded with a multilayer perceptron for image reconstruction, and larger decoders do not improve accuracy, indicating a bottleneck at the feature dimension.The decoder reconstructs 28 × 28 images using an architecture selected by random neural architecture search.
  • Simulation of deeper optical neural networks for 10-class cell-organelle classification: Deeper simulated networks generally produce higher-quality compression, with the advantage especially clear at very high compression ratios.The comparison includes a wide linear ONN, multilayer perceptrons, and deeper convolutional optical networks with nonlinear activations.

Experimental Setup

The experiment implements a two-layer optical neural-network encoder using optical matrix-vector multipliers, an image-intensifier nonlinearity, and camera read-out. Inputs include DMD-displayed images and reflected light from real scenes, while calibration characterizes optical computation and transmission.

  • Network architecture: The setup uses a two-layer optical neural network with fully optical linear layers and an image intensifier implementing the nonlinear activation between them.A CMOS camera performs read-out, followed by digital post-processing.
  • Input generation: DMD-displayed images and light reflected from physical objects serve as inputs to the ONN encoder.The experiments use DMD inputs for hand-drawn figures and cell organelles, and a 3D-printed traffic-sign scene for real-scene sensing.
  • Optical computation: Optical matrix-vector multiplication decomposes into fan-out, element-wise modulation, and fan-in using microlens arrays, LCDs, and lenses.The MLA creates identical optical copies, LCD modulation applies matrix weights, and optical fan-in sums contributions.
  • Optical computation: The first and second fully connected layers use different microlens-array configurations, with 676 lenslets in layer 1 and 63 lenslets in layer 2.Camera pixels corresponding to layer-2 neurons were summed digitally after read-out.
  • Calibration: Calibration compares experimentally measured optical matrix-vector outputs with digitally computed ground-truth vectors derived from optically fanned-out images.Separate monitoring-camera measurements characterize the two fully connected layers.
  • Calibration: The optical transmission of one matrix-vector multiplier is 2.9% at best, and the proof-of-concept setup cannot yet support low-light operation.The authors identify optimized imaging systems and phase modulation as routes to higher transmission.

Supplementary Note 5. THE GENERAL PROCEDURE FOR TRAINING OPTICAL-NEURAL-NETWORK ENCODERS

The ONN encoders were trained and calibrated layer by layer to account for physical nonlinear activations, noise, and hardware imperfections. Their latent features supported classification, reconstruction, anomaly clustering, and viewing-angle prediction, while reconstruction quality was constrained by the low-dimensional bottleneck.

  • Training procedure: Nonnegative optical weights, input noise injection, and random image transforms were used to improve robustness to incoherent-light constraints and hardware errors.Weights were clamped to [0, 1], while augmentation used ±5% translation and ±4% scaling.
  • Training procedure: The encoder weights were trained layer by layer, with physical nonlinear activations collected experimentally before retraining later optical and digital layers.After calibration, encoder and backend weights were fixed and evaluated on held-out test images.
  • Evaluation tasks: The QuickDraw dataset tested nonlinear ONN classification on 10 binarized hand-drawn classes selected for their greater variation than MNIST.The classes included clock, chair, computer, eyeglasses, tent, snowflake, pants, hurricane, flower, and crown.
  • Evaluation tasks: The latent-feature space preserved class information and salient visual attributes, including chair orientation, tent floors, and hurricane outlines.These attributes remained visible in reconstructions from the encoder’s latent features.
  • Evaluation tasks: Unsupervised spectral clustering separated five normal cell-image classes from an anomalous class, yielding six clusters corresponding to those categories.Cluster labels were assigned afterward to compute a confusion matrix against ground-truth labels.
  • Evaluation tasks: Viewing-angle prediction used an L1 loss, with even angles for training and odd angles for validation; performance decreased when predicting across all speed-limit classes.The experiment is described for predictions from the multilayer encoder’s latent-feature space.

Simulation of Deeper Optical-neural-network

Deeper ONN encoders were simulated for a 10-class cell-organelle task, comparing linear, multilayer perceptron, convolutional, and digital ResNet designs. Added convolutional depth improved accuracy, especially at high compression, but practical deployment requires co-design of compact hardware and optics-friendly architectures.

  • Task and architectures: The simulations evaluated ONN encoders on images from 10 cell-organelle classes using architectures with increasing optical depth and convolutional structure.The classes included cellmask, nucleolus, centrosome, mitochondria, cytoplasm, endoplasmic reticulum, nuclear envelope, nucleus, P bodies, and Cajal.
  • Task and architectures: The linear optical encoder provided a nonlinearity-free baseline of approximately 70% accuracy on the simulated dataset.Its input dimension was 10,000 and its output dimension was the bottleneck N.
  • Task and architectures: The simulated MLP used two fully connected layers with one saturating optical-to-optical nonlinear activation measured from the image intensifier.The network used 10,000-dimensional input and a 200-dimensional hidden layer.
  • Results: CNN1 achieved >90% test accuracy versus 84% for MLP, using an added convolutional layer and a shifted-ReLU-style activation.The convolutional layer used a 7 × 7 nonnegative kernel, one input channel, and 16 output channels.
  • Results: CNN3 added two convolutional layers beyond CNN1 and reached higher accuracy, especially at high compression ratios.CNN3 used one MaxPool operation after its first convolutional layer.
  • Results: A pretrained and retrained digital ResNet18 served as the state-of-the-art digital reference model for the cell-organelle dataset.Four additional layers adapted the input and classification output.
  • Limitations: Practical realization of more powerful ONN encoders is expected to require jointly designing compact hardware components and optics-friendly DNN architectures.The deeper ONN designs are described as ultimately speculative.
Loading 2207.14293v1…