Source-linked AI summary
Deep Learning Microscopy
Yair Rivenson, Zoltan Gorocs, Harun Gunaydin, Yibo Zhang, Hongda Wang, Aydogan Ozcan
TL;DR
Optical microscopy is limited by the resolution, field-of-view, and depth-of-field of its hardware. The paper trains a CNN to transform a single standard-microscope image into an enhanced output, achieving higher resolution with broader imaging coverage and rapid inference. The approach generalized across tissue samples and imaging conditions and extended spatial-frequency response and depth-of-field.
Problem
Optical microscopy must balance resolution against limited field-of-view and depth-of-field, motivating an approach that improves performance without changing microscope hardware.
Method
A CNN learns the statistical transformation from low-resolution to high-resolution microscopy images and then processes a single image from a standard microscope.
Results
The CNN improved resolution, field-of-view, and depth-of-field, resolved 0.345 µm, and produced images in approximately ~0.69 sec on a laptop.
Takeaways & Limitations
After training, the framework can blindly enhance images from different tissue types and imaging conditions and can extend computational imaging beyond optical microscopy.
Abstract
from arXiv · showhide
We demonstrate that a deep neural network can significantly improve optical microscopy, enhancing its spatial resolution over a large field-of-view and depth-of-field. After its training, the only input to this network is an image acquired using a regular optical microscope, without any changes to its design. We blindly tested this deep learning approach using various tissue samples that are imaged with low-resolution and wide-field systems, where the network rapidly outputs an image with remarkably better resolution, matching the performance of higher numerical aperture lenses, also significantly surpassing their limited field-of-view and depth-of-field. These results are transformative for various fields that use microscopy tools, including e.g., life sciences, where optical microscopy is considered as one of the most widely used and deployed techniques. Beyond such applications, our presented approach is broadly applicable to other imaging modalities, also spanning different parts of the electromagnetic spectrum, and can be used to design computational imagers that get better and better as they continue to image specimen and establish new transformations among different modes of imaging.
Introduction
Deep learning uses multilayer neural networks to analyze data, including images, and this paper applies a deep neural network to enhance optical microscopy without changing microscope hardware.
- Deep learning uses multilayer artificial neural networks for automated analysis of signals or data.
- Convolutional neural networks learn task-specific image transformations through trainable convolutional filters and nonlinear operations.
- The proposed network enhances optical microscopy using a single standard-microscope image without changing the microscope’s design or hardware.
Results and Discussion
The trained CNN transforms low-resolution microscope images into higher-resolution outputs while extending field-of-view and depth-of-field, generalizing across samples and imaging conditions.
- Training and testing: 179 registered low-resolution lung-tissue images and corresponding high-resolution labels were used to train the CNN.The low-resolution images used a 40×/0.95NA objective with a 150µm×150µm field-of-view, while high-resolution images used a 100×/1.4NA objective.
- Training and testing: After one-time training, the fixed CNN blindly outputs high-resolution images for samples beyond the training tissue type.It was tested on lung sections from another patient and on kidney tissue not included in training.
- Generalization: The same model enhanced spatial details in images acquired with a 100×/1.4NA objective, suggesting scale-invariance of the learned transformation.
- Field-of-view and depth-of-field: 6.25-fold larger FOV and significantly enhanced DOF were obtained relative to 100×/1.4NA objective images.The output FOV was approximately 379 × 379 µm, compared with the smaller high-NA field-of-view.
- Generalization: The framework was also tested across tissue images stained with Masson’s trichrome and H&E.The experiments examined whether a CNN trained on one stain could be applied to tissue samples stained with another dye.
- Resolution: 0.345 µm was resolved while modulation contrast increased across a substantial portion of the spatial-frequency spectrum, especially at high frequencies.
- Computation: ~0.69 sec per image was sufficient for average network inference on a laptop, with representative runtimes of ~0.695 sec and 0.037 sec.Self-feeding improved output images but increased runtime from ~0.037 sec to ~0.062 sec, with no noticeable changes after further cycles.
Methods
The study used de-identified human tissue sections with multiple stains and acquired images on an automated Olympus IX83 microscope using 40× and 100× objectives.
- Sample preparation: Samples included H&E-stained breast tissue from a breast cancer patient and Masson’s trichrome-stained lung and kidney tissue sections.The lung samples came from pneumonia patients, and the kidney sample came from a patient with moderately advanced diabetic nephropathy.
- Microscopic imaging: Images were acquired with an Olympus IX83 microscope, motorized stage, MetaMorph automation, 40×/0.95NA and 100×/1.4NA objectives, and a Qimaging Retiga 4000R camera.
Supplementary Tables
The supplementary tables report image-quality comparisons, training details, runtime measurements, and contrast values for a resolution test target.
- Supplementary Table 1 reports average SSIM for lung and breast-tissue datasets, comparing bicubic up-sampling with deep neural network outputs.
- Supplementary Table 2 lists deep neural network training details for the lung and breast-tissue datasets.
- Supplementary Table 3 reports average runtimes for regions of interest shown in Fig. 2.
- Supplementary Table 4 reports calculated contrast values for USAF resolution-test-target elements.