PaperlayerView exact revision on arXiv

Source-linked AI summary

Phase Imaging with Computational Specificity (PICS) for measuring dry mass changes in sub-cellular compartments

Mikhail E. Kandel, Yuchen R. He, Young Jae Lee, Taylor Hsuan-Yu Chen, Kathryn Michele Sullivan, Onur Aydin, M Taher A. Saif, Hyunjoon Kong, Nahil Sobh, Gabriel Popescu

arXiv:2002.08361v2eess.IVphysics.bio-phphysics.optics

TL;DR

Long-term live-cell imaging with cellular-compartment specificity remains challenging because fluorescence microscopy is limited by photobleaching, phototoxicity, and related artifacts. PICS combines quantitative phase imaging with AI to enable real-time, label-free monitoring, demonstrating week-long imaging with intact viability and specific measurements of nuclear and cytoplasmic growth.

  • Problem

    Long-term live-cell imaging with specificity for cellular compartments remains needed, particularly for studying cell growth, which is insufficiently understood.

  • Method

    PICS combines quantitative phase imaging with a U-Net-style deep convolutional neural network to learn mappings between label-free and stained images.

  • Results

    PICS enabled week-long live-cell time-lapse imaging with intact viability and high compartment specificity while simultaneously assaying nuclear, cytoplasmic, and total-cell growth.

  • Takeaways & Limitations

    PICS offers continuous, simultaneous monitoring of individual cellular components during long-term live-cell imaging.

  • Takeaways & Limitations

    A constant number of training epochs across resolutions and contrast somewhat limited PICS performance.

Abstract

from arXiv · show

Due to its specificity, fluorescence microscopy (FM) has become a quintessential imaging tool in cell biology. However, photobleaching, phototoxicity, and related artifacts continue to limit FM's utility. Recently, it has been shown that artificial intelligence (AI) can transform one form of contrast into another. We present PICS, a combination of quantitative phase imaging and AI, which provides information about unlabeled live cells with high specificity. Our imaging system allows for automatic training, while inference is built into the acquisition software and runs in real-time. Applying the computed fluorescence maps back to the QPI data, we measured the growth of both nuclei and cytoplasm independently, over many days, without loss of viability. Using a QPI method that suppresses multiple scattering, we measured the dry mass content of individual cell nuclei within spheroids. In its current implementation, PICS offers a versatile quantitative technique for continuous simultaneous monitoring of individual cellular components in biological applications where long-term label-free imaging is desirable.

Results · PICS Method

PICS combines quantitative phase imaging with a trained U-Net to infer fluorescence-like maps directly from label-free images in real time. The method supports long-term, viability-preserving imaging and performs across imaging conditions while retaining quantitative measurement capabilities.

  • PICS Method: PICS enables dynamic imaging from milliseconds to weeks without cell-viability concerns, supporting growth and proliferation assays for specific cellular compartments.The approach avoids chemical toxicity and photobleaching and can be applied in simultaneous multi-well experiments.
  • PICS Method: GLIM acquires four phase-shifted intensity images and combines them into a quantitative phase-gradient map, which is integrated using a Hilbert transform.The same camera and light path provide co-localized phase and fluorescence images, including nuclear and membrane channels.
  • PICS Method: PICS infers equivalent fluorescence signals directly from label-free phase images using a trained U-Net, with real-time rendering integrated into acquisition.Co-localized phase and fluorescence image pairs are used for training, after which inference is performed on live, never-labeled cells.
  • PICS Method: Residual learning produced much better performance under the same training conditions by predicting the difference between phase and fluorescence images.The residual connection adds the input to the output of the last convolutional block to generate the final prediction.
  • PICS Method: The method was evaluated across different image resolutions, fluorophores, cell lines, and both SLIM and GLIM imaging systems.Training used three in-focus images per unique field of view, spaced 2–3 depths of field apart as natural data augmentation.
  • PICS Method: At low resolution, including 10x imaging with 1.6 μm resolution, the PICS network achieved adequate performance.Figure 3 reports that performance improves with more data or training epochs while remaining adequate at the stated low resolution.

Effects of resolution on PICS performance

PICS maintained a Pearson correlation above 72% even with a 5x/0.08NA objective, while higher fluorescence and QPI contrast improved performance. Interferometric hardware further improved AI performance, and PICS produced uniform staining while correcting a staining defect.

  • Effects of resolution on PICS performance: Above 72% Pearson correlation was achieved even with a 5x/0.08NA objective when comparing predicted and actual fluorescence images.Pearson correlation quantified the match between computationally predicted and actual fluorescence images.
  • Effects of resolution on PICS performance: Higher fluorescence and QPI contrast, corresponding to greater spatial variance, yielded better PICS performance.The comparison held training time and training-pair quantities constant, with the number of epochs also kept constant across resolutions and contrast.
  • Effects of resolution on PICS performance: Interferometric hardware that decoupled phase and amplitude information improved the AI algorithm’s performance relative to QPI, DIC, and bright-field comparisons.These comparisons were presented in Supplementary Fig. 5.
  • Effects of resolution on PICS performance: PICS provided a uniform and consistent stain and corrected a staining defect.The staining correction was highlighted in Supplementary Fig. 5.

Training data set considerations for PICS · Time-lapse PICS of adherent cells

PICS achieved high-fidelity digital staining from as few as 20 image pairs and supported real-time multiplexed monitoring of unlabeled adherent cells over seven days. The approach maintained cell viability while increasing throughput relative to fluorescence acquisition and enabling multiple stain predictions.

  • Training data set considerations for PICS: 20 image pairs, representing roughly 500 SW cells and five minutes of training, were sufficient to generate high-fidelity digital stains.The experiments varied dataset size while holding other training parameters constant.
  • Training data set considerations for PICS: Networks that performed well on small training sets also performed well on larger, unseen validation sets.Certain images dominated training, and some cross-validation folds converged faster than others.
  • Time-lapse PICS of adherent cells: Fluorescence tags required an order of magnitude more exposure time than QPI frames, enabling higher-throughput acquisition while maintaining specificity.The throughput advantage becomes larger when separate exposures are used for individual fluorophores.
  • Time-lapse PICS of adherent cells: Many-day PICS monitoring caused no noticeable loss in cell viability, supporting nondestructive longitudinal imaging.This capability was demonstrated in the time-lapse experiment and emphasized by Figure 4 and Supplementary Video 4.
  • Time-lapse PICS of adherent cells: Seven-day imaging showed significantly increased cell-culture density, indicating continued multiplication during PICS monitoring.The time-lapse experiment used unlabeled SW480 and SW620 cells and predicted both nuclear and membrane fluorophores.
  • Time-lapse PICS of adherent cells: PICS can multiplex numerous stain predictions, with each additional inference channel adding approximately 65 ms to real-time inference.Multiple networks can be evaluated in parallel on separate GPUs, while one-stain computation is masked by acquisition.

Cell growth measurements of sub-cellular compartments · PICS of Spheroids · Discussion and Outlook

PICS combines quantitative phase imaging with AI to provide specific, real-time maps of unlabeled cellular compartments while preserving viability. It enables long-term quantitative growth measurements in cells and spheroids, including nuclear dry-mass estimation in optically scattering structures.

  • Cell growth measurements of sub-cellular compartments: PICS-DiI and PICS-DAPI independently quantified total-cell, cytoplasmic, and nuclear dry mass dynamically through individual-cell growth and mitosis.The compartment-specific maps were applied back to QPI images to measure dry mass.
  • Cell growth measurements of sub-cellular compartments: As confluence increased, growth saturated through contact inhibition, while nuclear dry mass remained stable and nuclear area distinguished SW480 from SW620 cells.In co-culture, SW620 cells had smaller nuclei but similar total dry mass to SW480 cells.
  • PICS of Spheroids: In HepG2 spheroids, GLIM suppressed multiple-scattering artifacts and enabled PICS nuclear mapping with a 4% average nuclear dry-mass error versus DAPI.The comparison used binary masks derived from PICS and DAPI images.
  • Discussion and Outlook: PICS uses automatic training and same-camera QPI–fluorescence acquisition, reducing registration and annotation burdens while supporting virtually unlimited predicted fluorescent channels.Training is performed once for each cell type and magnification, after which the stored network provides a reusable virtual stain.
  • Discussion and Outlook: Real-time inference required 65 ms per frame, faster than SLIM and GLIM acquisition, allowing specificity maps to be overlaid with QPI phase maps during imaging.The inference time was approximately one order of magnitude shorter than typical fluorescence exposure.
  • Discussion and Outlook: PICS enabled week-long time-lapse imaging with intact viability and high specificity for cellular compartments.The approach uses low-light quantitative phase input and nondestructive computation.
  • Discussion and Outlook: By multiplexing nuclear and lipid-bilayer specificity, PICS simultaneously assayed nuclear, cytoplasmic, and total-cell growth across many cell cycles.Training on fixed cells allowed PICS to mimic fluorescence stains otherwise incompatible with live-cell imaging.
  • Discussion and Outlook: PICS illustrates how deep learning can empower intrinsic quantitative phase contrast, extending specific, low-toxicity imaging toward thick tissues and spheroid viability assays.GLIM enabled subcellular specificity in optically turbid spheroids, while real-time feedback improved throughput and reduced toxicity.

Methods · Acquisition procedure · Real-time PICS

PICS combines quantitative phase acquisition with an optimized U-Net and TensorRT inference to generate synthetic fluorescence maps in real time. Its acquisition pipeline reconstructs phase from four label-free intensity images and overlays the synthetic stain, achieving approximately 15-fold faster imaging than the fluorescence training data.

  • Acquisition procedure: The acquisition software separates frontend event-list generation from backend processing, with TensorRT instrumented for real-time inference and a graphical interface for plate-reader-style imaging.The backend was adapted from the prior software architecture, while the frontend supports acquisition control.
  • Acquisition procedure: Each PICS image combines four label-free intensity images into a phase map before neural-network synthesis and phase–stain overlay.The sequence applies modulation, camera exposure and readout, phase retrieval, U-Net processing, and rendering.
  • Acquisition procedure: Approximately 15 times faster: PICS required less acquisition time than the roughly 1000 ms fluorescence images used for training.Under typical operation, image acquisition rather than computation limits the rate.
  • Methods: PICS translates quantitative phase maps into synthetic fluorescence using an optimized U-Net integrated with TensorRT for real-time inference.TensorRT tunes the network for specific network–GPU pairings and can operate directly on GPU memory.
  • Real-time PICS: The inference framework rescales inputs to the network’s required pixel size to accommodate differences in magnification and camera frame size.Scaling uses NVIDIA’s Performance Primitives library.
  • Real-time PICS: TensorRT network tuning requires a 30-second initialization, so one optimized network is built for the largest camera image and smaller inputs are mirror-padded.A 32-pixel mirror pad is applied to all inferences to reduce edge artifacts.

Multi-well plate reader operation · Training the neural networks · Cell culture

The study combines plate-reader tilt correction, U-Net-based neural-network training, and defined cell-culture protocols to support quantitative phase imaging experiments. Cell growth characteristics were consistent between experiments, indicating constant behavior of the particular subclone.

  • Multi-well plate reader operation: The plate-reader interface compensates for sample tilt by displaying each well as a 3D tomogram and interpolating the best-focus plane across mosaic tiles.Focus points define a Delaunay triangulation for interpolation; because well bottoms are flat, focusing on four corner points generally provides good results.
  • Multi-well plate reader operation: The interface also configures multichannel fluorescence acquisition alongside phase imaging and supports phase-specific stabilization and exposure settings.
  • Training the neural networks: The neural networks use a U-Net architecture selected to capture broad features typical of quantitative phase images.
  • Training the neural networks: Fifty-seven networks were trained without transfer learning using TensorFlow and Keras across workstations and isolated compute nodes.
  • Training the neural networks: Training used ADAM with mean squared error, while phase and fluorescence images were normalized using dataset extrema and a pixel-wise median filter.The filter was designed to bring values into the range [0,1].
  • Training the neural networks: Because broadband quantitative phase images exhibit sectioning and defocus effects, images were acquired as tomographic stacks and three in-focus images were selected per mosaic tile.
  • Cell culture: Growth characteristics were consistent between experiments, suggesting constant behavior of the particular subclone.

Time-lapse microscopy · Availability of computer code and algorithms: · Contributions:

PICS supported automated, nondestructive time-lapse imaging over a week, producing 202,860 GLIM images without appreciable focus drift. The supporting code was available from the corresponding author upon reasonable request, with responsibilities distributed across imaging, AI development, instrumentation, cell preparation, analysis, writing, and supervision.

  • Time-lapse microscopy: The week-long automated time-lapse procedure was repeated twice, including in Supplementary Fig. 11.The experiment was performed to illustrate PICS’s nondestructive specificity.
  • Time-lapse microscopy: Samples were imaged every sixty-eight minutes across multiwell plates, five depths, and 7 by 7 mosaic grids.Three cancer-cell conditions were plated in a 2 x 3 multiwell, with the procedure repeated for every well.
  • Time-lapse microscopy: 202,860 GLIM images were collected during a week-long automated time-lapse experiment without appreciable focus drift.Imaging used a temperature-controlled incubator, and the images were assembled into a mosaic by in-house software.
  • Time-lapse microscopy: After imaging, cells were fixed and stained with DiI and DAPI to produce an AI training corpus.This post-experiment staining supported training from the time-lapse samples.
  • Availability of computer code and algorithms:: The code and computer algorithms supporting the study’s findings were available from the corresponding author upon reasonable request.
  • Contributions:: M.E.K. designed and performed imaging experiments, while Y.R.H. and N.S. developed and trained the AI model.M.E.K. and Y.R.H. also instrumented real-time inference.
  • Contributions:: Cell culture, staining, spheroid provision, SW-cell provision, data analysis, manuscript writing, and project supervision were assigned across the listed contributors.N.S. supervised the AI work, and G.P. supervised the project.

Conflict of Interest: … Supplementary Note 2: Phase integration using the Hilbert transform

The supplementary material describes GLIM implementation and Hilbert-transform phase integration, including validation against the expected phase shift from a 3 μm polystyrene bead. It also reports a financial interest in Phi Optics, Inc. for one author, while the remaining authors declare none.

  • Conflict of Interest:: G.P. has a financial interest in Phi Optics, Inc., while the remaining authors declare no financial interest.Phi Optics develops quantitative phase imaging technology for materials and life science applications.
  • Supplementary Note 1: Gradient Light Interference Microscopy: PICS was tested with both SLIM and GLIM to show that it is not dependent on a particular quantitative phase imaging method.GLIM uses an infrared source and a DIC-based optical configuration with phase shifting by a liquid crystal variable retarder.
  • Supplementary information for Phase Imaging with Computational Specificity (PICS): The supplementary methods establish the optical and computational basis for applying PICS across quantitative phase imaging implementations.The described implementation spans SLIM/GLIM acquisition, phase-gradient recovery, and Hilbert-transform integration.
  • Supplementary Note 1: Gradient Light Interference Microscopy: GLIM reconstructs phase from four images acquired at π/2 phase shifts between the two beams, followed by one-dimensional integration.The phase-shifting components can be placed outside the microscope, but the integrated image can contain streak artifacts.
  • Supplementary Note 2: Phase integration using the Hilbert transform: GLIM recovers phase by measuring interference between laterally sheared beams and extracting the phase gradient along the contrast direction.A liquid crystal variable retarder cycles the phase shift to recover the gradient, scaled by the shear.
  • Supplementary Note 2: Phase integration using the Hilbert transform: The true phase map is obtained by applying a Hilbert-transform Fourier filter along the contrast direction, with regularization providing a Wiener-filter-like integral.The approach is described as a regularized version of Wiener filtering in the relevant frequency range.
  • Supplementary Note 2: Phase integration using the Hilbert transform: The Hilbert-transform reconstruction produced phase shifts in good agreement with the expected value for a 3 μm polystyrene bead embedded in immersion oil.The bead experiment is shown in Supplementary Fig. 2b.

Supplementary Note 3: Spatial Light Interference Microscopy · Supplementary Figure 1

SLIM uses spatial light modulation and four phase-shifted frames to reconstruct quantitative phase images and remove phase-contrast halos. The note contrasts SLIM’s higher sensitivity with GLIM’s stronger performance in strongly scattering samples and dense well plates, while indicating that PICS may generalize to other quantitative phase modalities.

  • Supplementary Note 3: Spatial Light Interference Microscopy: SLIM controls the phase between incident and scattered optical fields using a spatial light modulator matched to the objective’s back focal plane.The setup upgrades phase-contrast microscopy through spatially controlled retardance.
  • Supplementary Note 3: Spatial Light Interference Microscopy: Four phase-contrast-like frames, acquired with 90-degree SLM increments, enable recovery of the phase between the two optical fields.The phase is then obtained by estimating the transmitted-component shift and compensating for objective attenuation.
  • Supplementary Figure 1: SLIM reconstruction corrects the phase-contrast halo using a nonlinear Hilbert transform-based procedure.Supplementary Figure 1 shows both SLIM image reconstruction and the resulting halo-removed image.
  • Supplementary Note 3: Spatial Light Interference Microscopy: SLIM has higher sensitivity, whereas GLIM performs better for strongly scattering samples and dense well plates.In scattering samples, SLIM’s reference field vanishes exponentially; in dense plates, transmitted light is distorted by menisci or blocked by high walls.
  • Supplementary Note 3: Spatial Light Interference Microscopy: SLIM and GLIM images were acquired using three different Axio Observer Z1 microscopes.This identifies the microscope platforms used for the two imaging methods in the work.
  • Supplementary Note 3: Spatial Light Interference Microscopy: PICS is expected to apply to other imaging modalities, especially those where fluorescence can be overlaid with quantitative phase images.The work focuses on the authors’ SLIM and GLIM implementations while identifying broader modality applicability.
  • Supplementary Figure 1: Supplementary Figure 1 depicts ring illumination matched to the objective back focal plane and an SLM mask producing variable retardance.This arrangement effectively forms a phase-contrast microscope with controllable retardance.

Supplementary Figure 2 … Supplementary Figure 8

The supplementary analyses describe PICS image formation, neural-network design, applicability, training behavior, real-time implementation, and the effects of quantitative phase information. Together, they support integrated GLIM phase processing, improved fluorescence mapping, broad experimental applicability, and online acquisition constraints.

  • Supplementary Figure 2: Integrated GLIM images are generated by applying a Hilbert transform along the shear direction, producing an integrated phase profile consistent with the bead’s theoretical profile.The transform uses frequency-domain step-function multiplication, and the inverse transform’s imaginary component yields the integrated image.
  • Supplementary Figure 3: The PICS-DAPI network uses a modified U-Net with batch normalization before activation layers, fewer filters, and representations of intracellular textures and nucleolar edges.
  • Supplementary Figure 4: PICS was evaluated across stains, QPI modalities, and cell lines by comparing Pearson correlation between measured fluorescence and computationally inferred images.The tested applications included other QPI modalities and SW480 and SW620 cell cultures.
  • Supplementary Figure 5: Integrated GLIM phase produced the closest match to measured fluorescence, outperforming brightfield and DIC inputs as phase information was progressively isolated.The comparison used Pearson correlation across the full test set; DIC and integrated phase had similar standard deviations of 0.064 and 0.065, respectively, despite different qualitative performance.
  • Supplementary Figure 6: Supplementary Figure 6 tests how increasing numbers of QPI-fluorescence training pairs affect PICS network performance, including a condition with 80 training pairs.Each pair contains three focus levels; the 80-pair condition used 240 training images, 48 validation images, and 174 final test images.
  • Supplementary Figure 7: Training and validation losses were plotted for 57 neural networks to assess convergence and overfitting, with fold-to-fold differences in convergence observed during cross-validation.
  • Supplementary Figure 8: Real-time SLIM and GLIM acquisition overlaps reconstruction and inference with image capture, while GLIM’s variable-retarder modulation limits operation to one phase image every 80 ms.The pipeline includes modality-specific correction, GPU inference, and rendering; faster graphics cards or modulators could improve performance.

Supplementary Figure 9 … Video 4

The supplementary materials describe automated multiwell acquisition, digital-stain segmentation, comparative performance of fluorescence-equivalent imaging, nuclear dry-mass monitoring, and representative training, real-time, and long-term imaging videos.

  • Supplementary Figure 9: PICS supports automated multiwell acquisition through regions of interest, focus points, volume and mosaic settings, and phase-specific reconstruction parameters.The interface generates acquisition-event lists processed by the acquisition software and corrects sample defocus using focus points.
  • Supplementary Figure 10: Digital stains were binarized by thresholding the cumulative fluorescence-intensity histogram to distinguish sample from background.The threshold was identified from the histogram’s change in inflection, which marked the transition from background tracking to sample tracking.
  • Supplementary Figure 11: Over 72 hours, PICS semantic segmentation enabled watershed-based instance segmentation for monitoring nuclear dry mass and area in SW 480 and SW 620 subclones.The multiwell scan was acquired every two hours, and the comparison reports somewhat smaller SW 620 nuclei while total nuclear dry mass remains between the two subclones.
  • Supplementary Figure 11: 30% smaller nuclei were observed in metastatic SW 620 cells, while total nuclear dry mass remained comparable with SW 480 cells.The comparison comes from time-lapse monitoring of nuclear area and dry mass.
  • Supplementary Figure 12: Scattered-light GLIM images could produce fluorescence equivalents, whereas converting DAPI images into phase images performed substantially worse and missed structural details, underestimating cell area.The reported area underestimation is illustrated by red arrows.
  • Video 1: Video 1 shows co-localized GLIM and DAPI acquisition for PICS training in SW cells at 20x/0.8.The video documents paired acquisition of the two modalities used for training.
  • Video 2; Video 3: Video 2 demonstrates real-time GLIM with PICS-DAPI in SW cells at 20x/0.8, while Video 3 demonstrates real-time SLIM with PICS-DAPI in CHO cells at 10x/0.3.These videos provide real-time demonstrations across two imaging modalities and cell types.
  • Video 4: Video 4 shows GLIM and PICS imaging during a seven-day SW-cell time lapse at 20x/0.3.The video demonstrates extended longitudinal imaging of SW cells.
Loading 2002.08361v2…