Source-linked AI summary
Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy
Martin Weigert, Uwe Schmidt, Robert Haase, Ko Sugawara, Gene Myers
TL;DR
Existing nuclei-analysis methods often struggle with low-SNR images and densely packed objects. The paper extends shape-based STARDIST prediction to 3D star-convex polyhedra, addressing efficient representation, anisotropy, and overlap computation. STARDIST-3D outperformed watershed and U-Net baselines on two challenging datasets.
Problem
Low signal-to-noise ratios and densely packed nuclei limit existing automated detection and instance-segmentation methods.
Method
STARDIST-3D densely predicts star-convex polyhedra and object probabilities, adapting radial directions to anisotropy and using efficient intersection bounds for non-maximum suppression.
Results
STARDIST-3D outperformed strong watershed and U-Net baselines on two challenging datasets.
Takeaways & Limitations
The method is fast enough for typical large 3D volumes and is generally applicable to objects well represented by star-convex polyhedra.
Abstract
from arXiv · showhide
Accurate detection and segmentation of cell nuclei in volumetric (3D) fluorescence microscopy datasets is an important step in many biomedical research projects. Although many automated methods for these tasks exist, they often struggle for images with low signal-to-noise ratios and/or dense packing of nuclei. It was recently shown for 2D microscopy images that these issues can be alleviated by training a neural network to directly predict a suitable shape representation (star-convex polygon) for cell nuclei. In this paper, we adopt and extend this approach to 3D volumes by using star-convex polyhedra to represent cell nuclei and similar shapes. To that end, we overcome the challenges of 1) finding parameter-efficient star-convex polyhedra representations that can faithfully describe cell nuclei shapes, 2) adapting to anisotropic voxel sizes often found in fluorescence microscopy datasets, and 3) efficiently computing intersections between pairs of star-convex polyhedra (required for non-maximum suppression). Although our approach is quite general, since star-convex polyhedra include common shapes like bounding boxes and spheres as special cases, our focus is on accurate detection and segmentation of cell nuclei. Finally, we demonstrate on two challenging datasets that our approach (StarDist-3D) leads to superior results when compared to classical and deep learning based methods.
1. Introduction
3D fluorescence-microscopy nuclei analysis is difficult because low signal-to-noise ratios and dense packing undermine common instance-segmentation strategies. STARDIST-3D extends shape-based prediction to 3D using star-convex polyhedra, with parameter-efficient representations, anisotropy adaptation, and efficient overlap handling.
- Nuclei detection and segmentation commonly initiate studies of cellular expression patterns and developmental cell lineages.
- Semantic segmentation labels classes without separating nuclei, whereas instance segmentation assigns a distinct mask to each individual nucleus.
- Low signal-to-noise ratios and dense nuclear packing cause grouping-based methods to fuse touching nuclei and bounding-box methods to miss poorly approximated objects.
- STARDIST predicts object probabilities and radial distances to boundaries, defining star-convex shapes that are pruned with non-maximum suppression.
- STARDIST-3D densely predicts star-convex polyhedra for 3D volumes, retaining shape flexibility that includes bounding boxes and spheres as special cases.
- 64 radial values can faithfully represent 3D nuclei, avoiding the 1024 directions required by a naive 2D-to-3D extension; anisotropy adaptation is also critical.The method was reported to outperform watershed and U-Net baselines on two challenging datasets.
2. Method
StarDist-3D represents nuclei as star-convex polyhedra defined by radial distances along anisotropy-adjusted rays, predicts these parameters densely, and removes redundant candidates with NMS.
- 2.1. Star-convex polyhedra: Each object is represented by distances to its boundary along a fixed set of unit rays, whose triangulated endpoints form a star-convex polyhedron.The rays are selected using a spherical Fibonacci lattice and distributed over an ellipsoid representative of dataset objects.
- 2.1. Star-convex polyhedra: Anisotropy-adjusted rays account for squeezed nuclei shapes caused by unequal voxel sizes, with factors automatically computed from training-image bounding boxes.The final rays are obtained by scaling and normalizing the initial lattice vectors.
- 2.1. Star-convex polyhedra: As few as 64 rays can sufficiently reconstruct labeled 3D nuclei, while Fibonacci rays are more accurate than equidistant rays in the reported comparison.Reconstruction fidelity also depends critically on using an appropriate anisotropy factor for highly anisotropic data.
- 2.2. Model: A convolutional neural network densely predicts radial distances and an object probability for every voxel.The object probability is defined as the normalized Euclidean distance to the nearest background pixel.
- 2.2. Model: Training combines binary cross-entropy for object probabilities with a distance loss weighted on object pixels and regularized on background pixels.The distance-loss weighting promotes higher accuracy near object centers, which become polyhedron center candidates.
- 2.3. Efficient non-maximum suppression: NMS filters redundant polyhedron candidates using successively tighter overlap bounds before exact rasterized intersection when necessary.The filtering scheme uses bounding and inscribed spheres, convex hulls, and kernels, reducing runtime for large candidate sets.
3. Experiments
Experiments evaluate STARDIST-3D against watershed and U-Net-based methods on WORM and PARHYALE using IoU-thresholded accuracy. Results show consistent superiority, especially under PARHYALE’s low signal-to-noise ratio and anisotropy, and strong performance with limited annotations.
- Datasets: The study uses WORM and PARHYALE, which differ in resolution, signal quality, anisotropy, and available labeled training data.WORM has near-isotropic resolution and 18/3/7 training/validation/testing images, whereas PARHYALE has highly anisotropic images, lower signal-to-noise ratio, and 3/1/2 images.
- Methods and Evaluation: Methods are evaluated with accuracy across several IoU thresholds using Hungarian matching between predicted and ground-truth nuclei.Accuracy counts true positives, false positives, and false negatives, with a one-to-one assignment constraint.
- Results: STARDIST-3D consistently outperforms all compared methods, with the largest performance gap on PARHYALE.The authors associate this advantage with the shape model’s usefulness for disambiguating neighboring nuclei in challenging low-SNR images.
- Results: Without anisotropy-adapted radial directions, PARHYALE accuracy at τ = 0.5 drops from 0.593 to 0.291.This result demonstrates the importance of adapting the representation to anisotropic nucleus shapes.
- Results: With 4.15% of the WORM training and validation data, STARDIST-3D reaches accuracy 0.7 at τ = 0.5, matching U-NET+ trained on 100%.The reduced subset contains 472 instances in total.
4. Discussion
The paper presents STARDIST-3D for detecting and segmenting nuclei in volumetric fluorescence microscopy, including substantially anisotropic data. It outperforms watershed and U-Net baselines while remaining applicable to objects representable by star-convex polyhedra.
- Discussion: STARDIST-3D extends star-convex shape prediction to 3D fluorescence microscopy and handles substantially anisotropic nuclei.The method targets both detection and segmentation of cell nuclei in volumetric images.
- Discussion: STARDIST-3D outperforms the compared watershed and U-Net baselines while remaining easy to train and use.The paper attributes practical processing efficiency to its polyhedron parameterization and efficient intersection implementation.
- Discussion: The approach should generalize to objects whose shapes are well represented by star-convex polyhedra.This scope includes the cell-nucleus application studied in the paper.