Source-linked AI summary
Convolutional neural networks automate detection for tracking of submicron scale particles in 2D and 3D
Jay M. Newby, Alison M. Schaefer, Phoebe T. Lee, M. Gregory Forest, Samuel K. Lai
TL;DR
Submicron particle tracking is hindered by heterogeneous, low-SNR videos and extensive user intervention. The paper develops a machine-trained CNN that localizes particles from raw imaging data across diverse 2D and 3D conditions. The tracker achieves low false-positive and false-negative rates while improving automation and tracking throughput.
Problem
Submicron particle tracking is limited by heterogeneous imaging conditions, poor SNR, and extensive user involvement that introduces inefficiency and user variation.
Method
The paper develops an end-to-end recurrent CNN with over 6,000 tunable parameters, trained on diverse simulated video conditions for automated particle localization.
Results
The neural network tracker achieves substantially improved false-positive and false-negative performance over several existing methods, including a mean false-positive rate of ∼0.5% across tested SNR values.
Takeaways & Limitations
The tracker enables more automated and reproducible conversion of challenging 2D and 3D videos into particle position time series.
Takeaways & Limitations
The network does not recognize long filaments such as microtubules because they differ substantially from the particle shapes used in training.
Abstract
from arXiv · showhide
Particle tracking is a powerful biophysical tool that requires conversion of large video files into position time series, i.e. traces of the species of interest for data analysis. Current tracking methods, based on a limited set of input parameters to identify bright objects, are ill-equipped to handle the spectrum of spatiotemporal heterogeneity and poor signal-to-noise ratios typically presented by submicron species in complex biological environments. Extensive user involvement is frequently necessary to optimize and execute tracking methods, which is not only inefficient but introduces user bias. To develop a fully automated tracking method, we developed a convolutional neural network for particle localization from image data, comprised of over 6,000 parameters, and employed machine learning techniques to train the network on a diverse portfolio of video conditions. The neural network tracker provides unprecedented automation and accuracy, with exceptionally low false positive and false negative rates on both 2D and 3D simulated videos and 2D experimental videos of difficult-to-track species.
1 Significance Statement
High-resolution microscopy generates large 2D and 3D video datasets, creating demand for automated image analysis. Automated particle tracking is important for studying nanometer-scale particle movement in biological and therapeutic contexts.
- High-resolution light microscopes can collect terabytes of 2D and 3D video in a single day, creating demand for automated image analysis.
- Tracking nanometer-scale particles, including viruses, proteins, and synthetic drug particles, supports studies of mucosal barrier penetration and drug therapy design.
2 Introduction
Submicron particle tracking is difficult because heterogeneous, low-quality videos require extensive and variable user intervention. The paper addresses this by applying a recurrent, end-to-end CNN trained across diverse conditions, while emphasizing particle identification as central to tracking accuracy.
- Existing methods often require subjective, time-consuming parameter optimization for each heterogeneous video condition because ground truth is difficult to obtain.
- The method compiles simulations spanning varied microscopy conditions and uses thousands of learned parameters in a CNN for particle identification.
- Particle identification is presented as especially important for accuracy because the method improves false-positive and false-negative performance despite using simple adaptive linear assignment for linking.
- The recurrent network is trained end-to-end from raw pixels and outputs a probabilistic particle-versus-background classification at every pixel, using past and future observations.
- The three-layer CNN has over 6,000 tunable parameters optimized by machine learning, requiring no user-adjusted localization parameters and being tested across challenging video conditions.
3 Simulation of 4D greyscale image data
The simulation framework models four-dimensional greyscale microscopy data and the optical appearance of particles across varied conditions. It incorporates 3D-to-2D imaging, point-spread-function variation, noise, background changes, and photobleaching-related challenges.
- The simulation software represents a broad range of particle-tracking video conditions and includes how particles moving in 3D appear in a 2D camera image.
- A typical video is represented as a four-dimensional array containing three spatial dimensions and one time dimension, with 10-50 z-slices per volume.
- Figure 2 compares supervised, default, and manually adjusted tracking for 200 nm beads, including path lengths, ensemble-averaged MSD, and inter-user variation.
- Photobleaching can limit video length by gradually lowering SNR as imaging progresses.
- The simulated point spread function specifies pixel intensities from a particle at position (x, y, z), becoming dimmer and less focused away from z = 0 and developing diffraction-related disc patterns.
- PSF shape varies with microscope and camera parameters, making detailed physical modeling impractical for automated tracking in complex biological environments.
- Low SNR is addressed by adding Poisson shot noise with random intensity and slowly varying random background patterns to training videos.
4 An artificial neural network for particle localization
The method uses a layered convolutional neural network to localize particles from image data, producing detection probabilities for 2D and 3D tracking while incorporating temporal context. Its trainable parameters are learned from simulated videos spanning varied imaging and particle conditions.
- The network uses layered features that progress from detecting edges to recognizing curved lines and circular shapes.
- Each neuron processes a spatially local receptive field, combining nearby inputs through trainable kernel weights, biases, and a nonlinear activation.
- The three-layer network contains 12, 32, and 2 features, with outputs interpreted as particle detection probabilities at image coordinates.
- The network processes individual 2D slices to support both existing 2D data and reconstruction of 3D particle paths.
- A recurrent design uses observations from both past and future frames to improve particle-location predictions.
- Training optimizes network parameters on simulated particle videos varying point-spread shape, background, particle number, mobility, and signal-to-noise ratio.
Particle path linking
The tracker extracts candidate particles by thresholding neural-network detection probabilities, then links candidates across frames with adaptive linear assignment. The linking scheme uses detection confidence to accommodate particles entering, leaving, or becoming faint within videos.
- Candidate particles are connected components of pixels with detection probability q ≥ 0.5, and each receives the largest probability among its constituent pixels.
- Candidate positions are estimated by their centers of mass, yielding consistent average localization accuracy of 0.6 pixels.
- The linking model assumes that sampled particle motion approximates continuous motion while allowing particles to enter or leave the image.
- Link assignments are optimized as a linear-programming problem solved with the Hungarian-Munkres algorithm, using user-specified standard deviation σ.
- The adaptive linker makes the cost of leaving a particle unlinked depend on its detection probability rather than a fixed cutoff distance.
- σ was the only tracking parameter requiring user specification, and it rarely needed adjustment across approximately 600 test videos.
Performance evaluation and comparison to existing software
Across simulated 2D and 3D videos and experimentally derived 2D videos, the neural network tracker achieved accurate particle localization and tracking with substantially less manual effort. It maintained low error rates and strong particle recovery across varied imaging conditions.
- Evaluation design: 500 simulated 2D and 50 simulated 3D videos with randomized imaging and motion conditions provided ground-truth benchmarks for tracker evaluation.The 2D videos contained 100 paths across 50 frames; the 3D videos contained 300 particles across 20 z-axis slices.
- Comparison with existing software: Mosaic and Icy showed strong sensitivity to SNR and PSF radius, including false negative rates exceeding 50% or 40% and nearly fourfold changes in extracted particles.Mosaic exceeded 20% false positives below SNR 3, while Icy exceeded ∼40% false negatives across tested SNR values.
- Simulated 2D performance: ∼0.5% mean false positive rate across all tested SNR values distinguished the neural network tracker from Mosaic and Icy.It maintained fewer than 20% false negatives for SNR > 2.5 and less than one-pixel average localization error.
- Simulated 3D performance: ∼0.2% false positives and fewer than 5% false negatives in 3D made the neural network tracker the strongest performer among the compared 3D methods.It also achieved the lowest localization error and longest average path duration.
- Experimental 2D performance: 30–60 seconds per experimental video replaced 10–20 minutes of manual verification while producing comparable motion statistics and path lengths.The tracker also produced a greater number of traces, indicating fewer false negatives under experimental conditions.
Discussion
The neural network tracker targets difficult submicron-particle videos with fully automated analysis, improving throughput while supporting 2D and 3D tracking. Its robustness extends across varied particle and imaging conditions, but localization accuracy may remain insufficient for applications requiring exceptional precision.
- Submicron particles are difficult to track because they leave the focal plane, have low signal-to-noise ratios, vary spatially, and photobleach during imaging.
- 10–20x more time than video acquisition can be spent extracting particle paths, with user variation and tracker fatigue further limiting throughput.
- The CNN performs fully automated 2D and 3D tracking with low false-positive rates, lower false-negative rates and localization error, and longer average paths than leading software.
- The tracker remains robust across changing conditions, including rod-shaped salmonella and large polydisperse particles, while suppressing false positives from bright background objects.
- Localization accuracy was comparable to other methods, but applications such as microrheology may require additional accuracy from raw-image localization algorithms.
- Automation could broaden 3D particle-tracking adoption because 3D videos contain 10–50 times more data than comparable 2D videos and require substantial verification.
Supporting Information (SI)
The supporting information describes synthetic video generation, CNN architecture, training, and evaluation procedures. Simulations vary particle paths, backgrounds, noise, and image conditions, while the network converts normalized frames into per-pixel detection probabilities.
- Simulated videos: Synthetic videos approximate real-video appearance by randomizing generalized shape parameters rather than modeling optical-physics parameters.
- Simulated videos: Random Brownian particle paths are generated as ground truth, then combined with particle point-spread functions, background intensity, and noise to form image volumes.
- Network input and output: The CNN receives a single normalized image frame for a fixed z-axis slice and produces an output with the same size and dimension as the video.
- Architecture: The network uses three convolutional layers and one recurrent layer, with bilinear interpolation restoring the final two output likelihoods to the original image resolution.
- Architecture: The two output channels represent log likelihoods for background and nearby-particle presence, which are converted into detection probabilities.
- Training and deployment: Training minimizes cross entropy by gradient-based parameter updates on randomly generated synthetic frames, followed by TensorFlow deployment using optimized C++ operations.
- Evaluation: All 2D and 3D test videos used the same neural-network parameter values, while conventional methods were tuned across 10–15 parameter sets per method.
Methods
The methods describe preparation and imaging of viruses and nanoparticles for experimental particle-tracking videos. Videos were acquired at specified temporal and spatial resolutions, then analyzed using centroid-based trajectory processing.
- HIV, herpes simplex virus, and nanoparticles were prepared using previously described biological and chemical protocols before video acquisition.
- Sub-pixel tracking resolution was obtained by light-intensity-weighted averaging of neighboring pixels to estimate particle centroids.