Source-linked AI summary
Machine learning of hierarchical clustering to segment 2D and 3D images
Juan Nunez-Iglesias, Ryan Kennedy, Toufiq Parag, Jianbo Shi, Dmitri B. Chklovskii
TL;DR
The paper addresses how to improve segmentation during hierarchical region agglomeration. It proposes active learning across merge scales and reports improved segmentation against competing approaches, while noting remaining speed, scalability, and usability needs.
Problem
The paper addresses limited evidence for learning merge decisions across the changing region scales of hierarchical segmentation.
Method
The method actively selects region pairs during agglomeration, compares proposed merges with a gold standard, and learns from examples across all hierarchy levels.
Results
The method improves segmentation across EM and natural images, including a 28% reduction in VI error on the Berkeley Segmentation Dataset at ODS.
Takeaways & Limitations
Variation of information provides a region-based measure intended to make segmentation results more interpretable, particularly for neuronal EM images.
Takeaways & Limitations
The method still requires improvements in speed, scalability, and usability for neuronal reconstruction applications.
Abstract
from arXiv · showhide
We aim to improve segmentation through the use of machine learning tools during region agglomeration. We propose an active learning approach for performing hierarchical agglomerative segmentation from superpixels. Our method combines multiple features at all scales of the agglomerative process, works for data with an arbitrary number of dimensions, and scales to very large datasets. We advocate the use of variation of information to measure segmentation accuracy, particularly in 3D electron microscopy (EM) images of neural tissue, and using this metric demonstrate an improvement over competing algorithms in EM and natural images.
1 Introduction
The paper targets difficult image segmentation, especially large-scale neuronal EM data, by learning how to agglomerate superpixels across region scales using multiple cues and active examples.
- Motivation: Neuronal circuit reconstruction requires automated segmentation because nanometer-scale resolution and millimeter-scale neuron extent create huge image volumes.EM segmentation is difficult because adjacent neurons can share textures, internal structures resemble boundaries, and elongated shapes amplify boundary errors.
- Scope: The approach is demonstrated on both EM and natural images and is designed for arbitrary image dimensions.The paper also implements the ideas in the open-source Gala Python library.
- Motivation: Boundary-only segmentation and single-scale learning limit the use of information available during hierarchical region merging.The paper contrasts boundary maps and single-level learning with multiple cues combined through agglomerative learning.
- Contribution: The proposed active learning paradigm generates training examples across every level of the agglomeration hierarchy.The classifier proposes region pairs, compares them with ground truth, and uses the resulting examples to cover the feature space encountered at different region scales.
- Evaluation: Variation of information is advocated for evaluating segmentation because it better exposes region and topology differences relevant to neuronal EM images.The metric is intended to improve interpretability and analysis of segmentation results.
2 Methods
The method represents segmentation as hierarchical agglomeration on a region adjacency graph, learning merge priorities from features and gold-standard comparisons at successive scales.
- Graph representation: An initial superpixel oversegmentation is represented by a region adjacency graph whose nodes are regions and whose edges connect adjacent regions.Hierarchical agglomeration progressively merges graph nodes according to a merge priority function.
- Policy learning: The merge priority policy is decomposed into a feature map and classifier, with the classifier assigning priorities to candidate region pairs.Only local policy values need recomputation after a merge when an incident node changes.
- Training labels: The optimal agglomeration assigns each superpixel to the gold-standard segment with which it has greatest overlap, producing merge labels for region pairs.Labels indicate should merge, do not merge, or do not know depending on the regions’ relation to the gold standard.
- Active agglomeration: During each training epoch, proposed merges are checked against the gold standard; correct merges proceed, incorrect merges are retained as labeled examples, and the resulting classifier defines a new policy.The process repeats across epochs until the agglomeration matches the target agglomeration.
- Cues and features: The feature representation combines cue statistics for adjacent regions and their boundary, including histograms, quantiles, moments, size, and Jensen-Shannon divergence.Natural-image segmentation additionally uses orientation and convex-hull features, while EM data uses boundary, cytoplasm, mitochondria, and glia cues.
3 Results
The paper evaluates agglomerative segmentation with variation of information (VI), emphasizing its decomposition of false splits and false merges and its suitability for EM data. Active agglomerative learning modestly improves segmentation while substantially improving classifier probability calibration, and it also outperforms competing methods on natural images.
- Evaluation: The split-VI plot separates oversegmentation H(S|U) from undersegmentation H(U|S), making their tradeoff and agglomeration trajectories explicit.False merges move mainly rightward, whereas true merges move mainly downward.
- Evaluation: VI is used as the primary evaluation metric, especially for EM, because it addresses limitations of boundary precision-recall and Rand index for topology-sensitive segmentation.VI errors scale linearly with error size, unlike the Rand index’s quadratic scaling, and are more comparable across volumes.
- EM segmentation: Active agglomerative learning improved segmentation modestly but significantly compared with mean agglomeration and flat learning.Figure 4 compares learning and agglomeration methods using split VI, with shaded regions showing mean ± standard error.
- EM segmentation: Agglomerative learning shifted the minimum VI from p = 0.28 after flat learning to p = 0.51, indicating improved classifier probability estimates.The expected optimum near p = 0.5 is approached after agglomerative training.
- Natural images: On BSDS500, the algorithm significantly improved over state-of-the-art agglomerative methods and reduced the remaining error relative to human-level segmentation by at least 20% for all region metrics.The reduction for VI was 28% at the optimal dataset scale.
- Natural images: Agglomerative learning outperformed all other algorithms on both split VI and boundary precision-recall, and learning across scales helped combine cues for difficult images.The method remained effective even when boundary maps were far from ideal.
4 Discussion and conclusions
The method learns agglomerative merges across hierarchy levels by comparing candidate merges with gold-standard segmentations, improving segmentation on most natural-image test cases. The discussion highlights trade-offs involving competing methods, feature design, training-data requirements, and scalability.
- Method: The method learns merge decisions at all agglomeration scales by comparing candidate merges with a gold-standard segmentation.The learned examples are generated during agglomeration, so they match the segment sizes encountered at test time.
- Comparison with prior work: Compared with LASH, the method improves natural-image segmentation accuracy, while in EM the improvement appears mainly at higher undersegmentation values.LASH has an advantage earlier in EM agglomeration, whereas the proposed method improves at higher over-merging levels.
- Scalability and proofreading: Global optimization methods face theoretical scalability limits for volumes beyond the teravoxel range, whereas hierarchical methods could support massively parallel processing.The comparison also notes that hierarchical proofreading remains local, unlike global CRF re-optimization.
- Limitations and future work: The feature space remains incomplete: EM errors include pinch errors, motivating shape- and geometry-based features beyond voxel-sum features.The authors also identify standardized feature implementations and direct comparisons across method families as open needs.
- Limitations and future work: Training currently requires a full gold-standard segmentation, which may be difficult to obtain; a semi-supervised variant is being developed.The proposed variant is intended to require substantially less training data while achieving similar performance.