Source-linked AI summary
Superhuman Accuracy on the SNEMI3D Connectomics Challenge
Kisuk Lee, Jonathan Zung, Peter Li, Viren Jain, H. Sebastian Seung
TL;DR
The paper addresses whether automated EM neuron reconstruction can exceed the accuracy reached by leading convolutional methods on SNEMI3D. It combines a 3D U-Net variant with simulated defect augmentation, auxiliary long-range affinity prediction, and mean affinity agglomeration, and reports a submission that surpasses the benchmark’s human-accuracy estimate. The authors nevertheless caution that this does not show neuronal reconstruction is fully solved or that benchmark performance transfers directly to large-scale real-world reconstruction.
Problem
SNEMI3D had seen major accuracy gains from convolutional networks, but the benchmark had not yet established whether an automated method could surpass its human-accuracy estimate.
Method
A 3D U-Net variant predicts nearest-neighbor and auxiliary long-range affinities, trains with simulated image defects, and uses mean affinity agglomeration for postprocessing.
Results
The submission ranked first on SNEMI3D and surpassed the leaderboard’s estimate of human accuracy by a large margin.
Takeaways & Limitations
The result makes full automation of neural circuit reconstruction look more encouraging than before within the scope of the benchmark.
Takeaways & Limitations
The authors caution that human accuracy is ill-defined and that experts can still find errors, so surpassing the SNEMI3D estimate does not mean neuronal reconstruction is solved.
Abstract
from arXiv · showhide
For the past decade, convolutional networks have been used for 3D reconstruction of neurons from electron microscopic (EM) brain images. Recent years have seen great improvements in accuracy, as evidenced by submissions to the SNEMI3D benchmark challenge. Here we report the first submission to surpass the estimate of human accuracy provided by the SNEMI3D leaderboard. A variant of 3D U-Net is trained on a primary task of predicting affinities between nearest neighbor voxels, and an auxiliary task of predicting long-range affinities. The training data is augmented by simulated image defects. The nearest neighbor affinities are used to create an oversegmentation, and then supervoxels are greedily agglomerated based on mean affinity. The resulting SNEMI3D score exceeds the estimate of human accuracy by a large margin. While one should be cautious about extrapolating from the SNEMI3D benchmark to real-world accuracy of large-scale neural circuit reconstruction, our result inspires optimism that the goal of full automation may be realizable in the future.
1 Introduction
This work presents a SNEMI3D submission that surpasses the leaderboard’s human-accuracy estimate using a U-Net variant, simulated defect augmentation, long-range affinity prediction, and mean affinity agglomeration. The authors caution that this benchmark result does not establish that neuronal reconstruction is fully automated.
- Motivation: 3D neuron reconstruction from EM brain images is a foundational connectomics task, and convolutional networks have recently approached human accuracy on SNEMI3D.DeepEM3D was the first convolutional network reported to approach human accuracy on the benchmark.
- Approach: The submission combines extensive 3D convolution, simulated image defects, and long-range affinity prediction in a U-Net variant.The simulated defects include misalignments, missing sections, and out-of-focus sections.
- Postprocessing: Mean affinity agglomeration is used as postprocessing and provides a faster alternative to test-time augmentation while yielding a comparably large segmentation-accuracy gain.Test-time augmentation improves accuracy but substantially increases inference time.
- Caveat: Postprocessing benefits weaker networks more than stronger ones, and may become ineffective as convolutional networks improve.For the best net, the postprocessing improvement is relatively small.
- Caveat: The result surpasses the SNEMI3D human-accuracy estimate, but human experts can still find errors using interactive 3D segment viewing.The authors argue that human accuracy depends on expertise, procedures, and software tools.
- Implications: The authors conclude that full automation prospects are more encouraging, while the present SNEMI3D challenge is no longer adequate for evaluating highly accurate algorithms.They recommend modifying or replacing the challenge rather than treating neuronal reconstruction as solved.
2 Residual Symmetric U-Net
The model is a residual, symmetric U-Net adapted for anisotropic serial-section EM data. Its architecture combines coarse-to-fine refinement, residual modules, and selective three-dimensional processing with overlap-blended inference.
- Architecture: The network inherits U-Net’s contracting path, expanding path, and same-scale skip connections for coarse-to-fine refinement.Higher-level contextual information is progressively integrated with lower-level localization information.
- Architecture: Same convolutions replace valid convolutions, summation replaces concatenation in skip connections, and residual skip connections are added to every module.These choices simplify feature-map sizing and make every input-to-output path a residual subnetwork.
- Architecture: The modular design uses repeated convolutional blocks whose depth, scale, and width determine the network’s capacity.Each basic module contains three equal-width convolution layers interspersed with batch normalization and exponential linear units.
- Anisotropic 3D: To handle anisotropic EM data, the network avoids z-downsampling and uses exclusively 2D convolutions at the finest scale.The finest scale has maximal anisotropy and therefore receives specialized processing.
- Inference: Overlapping test-time patches are blended with a center-weighted bump function to reduce border-related accuracy loss.The implementation uses 50% overlap in all three dimensions and sets tx, ty, tz to 1.5.
- Inference: Test-time augmentation evaluates 16 rotated and flipped variants, but it is not used when measuring mean affinity agglomeration.The transformation set includes rotations and flips in the xy-plane plus z-direction flips.
3 Long-range affinity prediction as an auxiliary task
The network predicts long-range voxel affinities alongside nearest-neighbor affinities as an auxiliary training task. These additional predictions are not used at test time but are intended to improve the nearest-neighbor affinity task.
- Long-range affinities: Unlike earlier affinity-graph methods using only nearest-neighbor edges, this approach predicts selected long-range edges along cardinal directions.The long-range edges span 3, 9, and 27 voxels in x and y, and 2, 3, and 4 voxels in z.
- Affinity graph: Nearest-neighbor affinity maps produce the segmentation, while the additional maps represent long-range affinities introduced as auxiliary targets.Figure 2 depicts nearest-neighbor maps in the second column and long-range maps in the third through fifth columns.
- Affinity graph: The network produces twelve output images corresponding to the affinity maps shown in Figure 2.The long-range maps look qualitatively different from the nearest-neighbor maps.
- Training role: Long-range affinities are excluded at test time and are trained only to improve nearest-neighbor affinity prediction through auxiliary-task learning.The authors suggest that diversity among the auxiliary tasks may aid training.
4 Data augmentation
Training augmentation combines standard geometric and intensity transformations with simulated misalignments, missing sections, and out-of-focus sections. These defect simulations target common serial-section EM failure modes and may also encourage stronger use of 3D context.
- Standard augmentation: Standard augmentation uses 90° rotations, flips in all three dimensions, warping, and brightness and contrast perturbations.The warping and intensity transformations use code from the ELEKTRONN framework.
- Defect augmentation: Three novel augmentations simulate misalignments, missing sections, and out-of-focus sections encountered in serial-section EM imaging.The authors speculate that these transformations may improve accuracy even where defects are absent by forcing networks to exploit 3D context.
- Misalignment: Misalignment augmentation applies random x- and y-translations at a randomly selected z-location, using either slip-type or translation-type distortions.The same transformation is applied to the input image and target label stacks before dynamically generating the target affinity graph.
- Missing sections: Missing-section augmentation models partially or fully absent sections, motivated by imaging damage that can critically affect large-volume reconstruction.The paper describes both wholly missing sections and sections whose content is removed because of severe damage.
- Out-of-focus sections: Out-of-focus augmentation applies 2D Gaussian blur to up to five randomly selected slices, partially or fully, with standard deviation sampled between zero and five pixels.This models occasional microscope-focus failures during automated EM imaging.
5 Mean affinity agglomeration
The paper uses mean affinity as a simple, effective score for greedily agglomerating adjacent supervoxels after oversegmentation. Mean affinity smooths noise and is difficult to substantially outperform with learned GALA-type features.
- Adjacent supervoxels are greedily merged according to their agglomeration scores, with the highest-scoring pair merged at each step.The approach begins with an oversegmentation into supervoxels.
- Mean affinity provides the agglomeration score using a single hand-designed feature.
- Mean affinity smooths noise in affinity maps that could otherwise cause merge errors.
- Learned GALA-type features have been surprisingly difficult to use for substantially outperforming mean affinity agglomeration.The authors suggest this may reflect improvements in convolutional-network output since GALA was introduced.
6 Experiments
The experiments evaluate four training configurations on SNEMI3D, reserve validation data for model selection, and analyze performance further on the labeled AC3 stack. Evaluation uses the challenge’s adapted Rand F-score and variation of information for AC3 analyses.
- The SNEMI3D dataset contains one 1024×1024×100 labeled training stack and an equally sized test stack at 6×6×29 nm3 resolution.The training stack was split into the top 80 slices for training and bottom 20 slices for validation.
- Four nets combine proposed augmentation and long-range-affinity training: aug0, aug0-long, aug3, and aug3-long.aug0 omits the proposed augmentation, aug3 includes all of it, and the -long suffix denotes long-range-affinity training.
- Only aug3-long was submitted to the SNEMI3D leaderboard, with model selection and hyperparameter search performed strictly on the validation set.
- Post-challenge analyses used AC3, a publicly available labeled image stack of size 1024 × 1024 × 256.Although AC3 is a superset of the SNEMI3D test set, it was used only after the SNEMI3D submission.
- The SNEMI3D leaderboard measures segmentation quality with adapted Rand F-score, whereas AC3 analyses use variation of information.For automated segmentation S and ground truth T, VI separates oversegmentation and undersegmentation through H(S|T) and H(T|S).
7 Results
The submission ranked first on SNEMI3D and surpassed the leaderboard’s human-accuracy estimate. On AC3 and validation analyses, data augmentation, long-range affinities, and mean affinity agglomeration improved performance and robustness.
- The submission ranked first on SNEMI3D and surpassed the human-accuracy estimate provided by the challenge organizer.The authors describe this as the first demonstration of a fully automated algorithm surpassing human accuracy on a publicly available benchmark EM dataset.
- Mean affinity agglomeration achieved superhuman accuracy without costly test-time augmentation.The method uses agglomeration as postprocessing and avoids the inference-time cost associated with test-time augmentation.
- On AC3, proposed data augmentation, long-range affinity training, test-time augmentation, and mean affinity agglomeration each improved model performance.Test-time augmentation increased inference cost by 8-16×, while the other comparisons are reported across the model variants and postprocessing analyses.
- Misalignment augmentation made models substantially more robust to translation- and slip-type misalignment errors, with aug3 worsening beyond a certain slip extent.The validation analysis used mean affinity agglomeration as postprocessing.
- Missing-section augmentation improved robustness to multiple consecutive missing sections, and aug3-long produced the best qualitative result.Models without this augmentation failed immediately beyond two consecutive missing sections, whereas augmented models remained substantially more robust.
8 Discussion
Remaining reconstruction errors primarily arise from severe image defects, intrinsically difficult EM cases, agglomeration failures on self-touching objects, and underrepresented object classes. The discussion identifies automated error detection and correction as key missing components, with deep neural networks proposed for both tasks.
- 8 Discussion: Most remaining errors fall into four categories: severe image defects, intrinsically hard serial-section EM cases, self-touching objects, and underrepresented object classes.Examples include thin neurites parallel to the sectioning plane, dendritic spines contacting their shaft, glial cells near blood vessels, and soma-soma contacts.
- 8 Discussion: Severe image defects can cause misalignment errors that nearest-neighbor affinity graphs cannot properly handle.The authors suggest more robust alignment or removing defective regions and treating them as missing sections.
- 8 Discussion: Automated detection and correction of remaining errors are still missing from the neural circuit reconstruction pipeline.The discussion points to supervised deep learning for detection and coupled neural-network correction as possible approaches.
- 8 Discussion: Flood-filling networks are proposed as candidates for correcting individual erroneous objects through focused perceptual computation.The detector and corrector may be integrated into a single model rather than implemented separately.