Source-linked AI summary
Large-Scale Automatic Reconstruction of Neuronal Processes from Electron Microscopy Images
Verena Kaynig, Amelio Vazquez-Reina, Seymour Knowles-Barley, Mike Roberts, Thouis R. Jones, Narayanan Kasthuri, Eric Miller, Jeff Lichtman, Hanspeter Pfister
TL;DR
Large-scale electron-microscopy data make neuronal reconstruction a major analysis challenge. The paper presents a semiautomated pipeline combining sparse-annotation membrane classification and 3D segmentation fusion. It reports state-of-the-art reconstruction performance, including a 27,000 µm3 volume and spine necks reconstructed before proofreading.
Problem
Large EM data sets are difficult to analyze manually, while reconstructing neuronal anatomy and connectivity at nanometer resolution is important for studying brain structure.
Source: §1 · p. 2Method
The pipeline learns membrane appearance from sparse manual annotations, combines cell-region segmentation with 3D fusion, and provides semi-automated proofreading for merge errors.
Source: §1 · p. 3, §3.2 · p. 6, §4.1 · p. 8Results
The pipeline demonstrates state-of-the-art performance for dense tissue and long-range reconstructions, including a 27,000 µm3 volume and individual spine necks before proofreading.
Source: §1 · p. 3, §9 · p. 29Takeaways & Limitations
The workflow is designed to minimize manual effort and support parallelization on computer clusters and GPUs.
Source: §9 · p. 29The pipeline assumes previously aligned image data; registration and alignment are outside the paper’s scope.
Source: §3.1 · p. 5
Abstract
from arXiv · showhide
Automated sample preparation and electron microscopy enables acquisition of very large image data sets. These technical advances are of special importance to the field of neuroanatomy, as 3D reconstructions of neuronal processes at the nm scale can provide new insight into the fine grained structure of the brain. Segmentation of large-scale electron microscopy data is the main bottleneck in the analysis of these data sets. In this paper we present a pipeline that provides state-of-the art reconstruction performance while scaling to data sets in the GB-TB range. First, we train a random forest classifier on interactive sparse user annotations. The classifier output is combined with an anisotropic smoothing prior in a Conditional Random Field framework to generate multiple segmentation hypotheses per image. These segmentations are then combined into geometrically consistent 3D objects by segmentation fusion. We provide qualitative and quantitative evaluation of the automatic segmentation and demonstrate large-scale 3D reconstructions of neuronal processes from a $\mathbf{27,000}$ $\mathbf{\mu m^3}$ volume of brain tissue over a cube of $\mathbf{30 \; \mu m}$ in each dimension corresponding to 1000 consecutive image sections. We also introduce Mojo, a proofreading tool including semi-automated correction of merge errors based on sparse user scribbles.
1 Introduction
The paper addresses the challenge of analyzing large electron-microscopy data sets with a semiautomated 3D reconstruction pipeline. Its contributions span membrane segmentation, 3D fusion, large-scale reconstruction, and proofreading.
1 Introduction
- The paper presents the complete reconstruction pipeline and its application to large data, while noting that some component work was previously published.Source: §1 · p. 3
- Interactive random-forest training on sparse annotations reduced annotation effort and improved cell-region segmentation measured by variation of information.Source: §1 · p. 3
- The pipeline combines Kaynig et al.’s cell-region segmentation with Vazquez-Reina et al.’s segmentation fusion for 3D reconstruction from anisotropic EM data.Source: §1 · p. 3
- The authors extended segmentation fusion to branching structures and enabled parallel subvolume processing through pairwise matching of segmented blocks.Source: §1 · p. 3
- 27,000 µm3 of brain tissue was reconstructed automatically, including individual spine necks before manual proofreading.The authors describe this as the first such reconstruction in anisotropic serial-section EM data, to their knowledge.Source: §1 · p. 3
- Mojo uses sparse user scribbles, as described by Roberts et al., to support semi-automated correction of merge errors.Source: §1 · p. 3
2 Related Work
Prior work addressed parts of anisotropic EM reconstruction, while isotropic approaches faced resolution or field-of-view constraints. This paper compares its reconstruction scale and segmentation performance with earlier work.
2 Related Work
- As large EM volumes became available, automated analysis became a bottleneck; prior tools supported manual annotation and semi-automatic segmentation.Source: §2 · pp. 3–4
- Fully automatic isotropic methods using special staining can support long-range reconstruction, but sacrifice internal cell structures relevant to biological analysis.The paper describes conventional osmium-stained reconstructions that preserve structural information, including for mitochondria reconstruction.Source: §2 · p. 4
- Isotropic imaging is limited either to 30 nm voxels or a 20 µm2 field of view, whereas serial-section imaging records volumes of millions of cubic micrometers.Source: §2 · p. 4
- Nine times larger than Chklovskii et al.’s result, the demonstrated volume also had better neuronal-region segmentation with CRF than watersheds, reducing proofreading effort.Source: §2 · p. 4
- Earlier anisotropic-data methods often focused on segmenting 2D regions or grouping them across sections; some addressed both steps.Source: §2 · p. 4
3 Overview
The section outlines a reconstruction workflow for anisotropic serial-section images, its evaluation measure, and the data sets used to test it. The pipeline groups 2D segmentations into 3D objects and supports later proofreading.
3.1 Workflow
- The workflow combines interactive membrane detection, multiple 2D segmentation hypotheses, 3D segmentation fusion, and sparse-scribble proofreading.The data are assumed to have been aligned; registration is outside the paper’s focus.Source: §3.1 · pp. 5–6
3.2 Evaluation Measure
- Variation of information compares segmentations across both 2D region segmentation and 3D grouping, with scores reported in nats.The metric is based on entropy and mutual information.Source: §3.2 · pp. 6–7
3.3 Data Sets
- The large-scale test set contains 1,000 sections spanning a 30×30×30 µm³ volume, while the smaller 150-section set is split into training/validation and held-out test images.The first 75 images of the smaller set are used for training and parameter validation; the second half is reserved for testing.Source: §3.3 · p. 7
4 Region Segmentation
This section develops membrane-based 2D region segmentation from sparse interactive annotations, then evaluates the resulting methods. Its CRF approach improves on watershed segmentation, while interactive training outperforms fully annotated-image training after several rounds.
4 Region Segmentation
- The method segments membranes rather than cell interiors, aiming to adapt across animal types and staining protocols despite varying cell-region textures.Membrane boundaries are described as thin, smooth, and elongated.Source: §4 · p. 7
4.1 Membrane Classification
- A random forest trained on sparse membrane annotations uses image intensity, gradient, Hessian, Gaussian-difference, and steerable-filter features; interactive feedback focuses annotation on difficult classifications.The classifier uses 300 trees and reduces bootstrap samples to the minority-class size to address imbalanced training data.Source: §4.1 · p. 8
4.2 Interactive Training Evaluation
- After the third to fourth interactive pass, the sparse-annotation classifier outperforms training on fully annotated images.The second pass on one training set degraded performance after mitochondria were labeled as background, causing fuzzy membranes to be misclassified.Source: Figure 4 · p. 9, §4.2 · p. 10
4.3 2D Segmentation
- The 2D CRF combines random-forest membrane probabilities with isotropic boundary smoothing and an oriented term that favors membrane-aligned cuts.The isotropic term penalizes label changes between similar-intensity neighbors, while the directed term is asymmetric and penalizes cuts that violate smoothness along membrane direction.Source: §4.3 · pp. 10–11
- The smoothness terms are submodular, enabling global energy minimization by max-flow/min-cut; the resulting membrane labels are converted into neuronal-process regions by connected components and seeded region growing.The graph connects pixels to neighbors and to foreground/background terminals, with edge weights set by the smoothness and random-forest terms.Source: §4.3 · p. 12
4.4 Region Segmentation Evaluation
- 0.09 better variation of information than the best watershed segmentation is achieved by the CRF framework with gap completion.The optimal watershed segmentation performs 0.03 better than the best thresholded random-forest output.Source: §4.4 · p. 12
5 Segmentation Fusion
Segmentation fusion combines alternative 2D regions into geometrically consistent 3D neuronal processes, improving 2D region accuracy and matching strong 3D grouping results. Its overlap-based grouping can model branching, while thin, flexible processes remain challenging to match across sections.
5.1 Region Grouping Across Sections
- The 2D CRF’s isotropic smoothing closes small extracellular-space regions, while its anisotropic term limits shrinkage of elongated structures and enhances gap completion.Source: Figure 5 · p. 13
- Fusion selects among candidate 2D regions and links between adjacent sections; activated links require their connected regions to be selected, yielding a 3D segmentation.Region and link indicators form the labeling used to represent the volume.Source: §5.1 · p. 15
- Region similarity uses minimum relative overlap rather than texture correlation, a faster choice that gave equally good measurements on these EM data.Smaller regions from thin, flexible processes change more between sections, making overlap-based matching difficult; optical flow or warping would add significant computational overhead.Source: §5.1 · p. 15, §5.2 · p. 20
- Allowing multiple links to a segment models branching, while pruning links with little overlap helps prevent over-merging.The formulation also prevents overlapping activated segments within a section.Source: §5.1 · p. 16
5.2 Segmentation Fusion Evaluation
- Segmentation fusion and Sopnet achieved about the same region-grouping quality, with fusion 0.06 better in variation of information; both significantly outperformed agglomerative clustering.Lower variation-of-information scores indicate better groupings.Source: Figure 7 · p. 17, §5.2 · p. 17
- Correctly segmented regions rose from 68%(±4%) to 73%(±3%), while merge errors fell from 15.8% to 8.6% and split errors rose by 1%.The reported overall error-rate reduction was 5%, mainly from fewer merge errors.Source: §5.2 · p. 17, Figure 8 · p. 18
- Error rates were lower for larger regions; smaller regions tended toward merges, whereas larger regions tended toward splits.Source: §5.2 · p. 17, Figure 9 · p. 19
6 Semiautomatic Proofreading with Mojo
Automatic segmentation still requires proofreading to correct split and merge errors. Mojo presents the reconstruction as an overlay and lets users correct those errors through object relabeling or scribble-based subdivision.
6 Semiautomatic Proofreading with Mojo
- Manual proofreading is needed to guarantee correct neuron topology because some regions are split apart or span multiple objects.Source: §6 · p. 20
- Mojo overlays the automatic reconstruction on the EM image and provides separate modes for split- and merge-error correction.The interface allows users to select processes and choose whether corrections apply in 3D or only to the displayed section.Source: §6 · p. 20, Figure 11 · p. 21
- To correct a split error, users relabel target objects with a source object's label, choosing global, local-3D, or local-2D scope.Source: §6 · pp. 20–21
- To correct a merge error, users scribble distinct sub-objects within the object, and interactive segmentation separates them while keeping the correction inside that object.The method provides real-time feedback during scribbling.Source: §6 · p. 21
- Mojo does not enforce topological connectedness after each operation, giving users flexibility but leaving required topology constraints to them.Source: §6 · p. 22
- Precomputed multiscale tiles let Mojo load only visible image regions and operate with a fixed memory budget below 1 GB.The text identifies real-time relabeling of large volumetric objects as a non-trivial task.Source: §6 · p. 22
7 Parallel Implementation
The implementation divides reconstruction into image-level segmentation, sub-volume fusion, parallel matching of adjacent sub-volumes, and a global consistency step. Its reported job counts and runtimes describe the current MATLAB implementation, while proofreading remains limited mainly by human labor.
7 Parallel Implementation
- Each of 1,000 image jobs takes 9–13 hours for feature extraction, membrane classification, and CRF segmentation.Source: §7 · p. 23
- Fusion divides the data into 3,200 sub-volumes, with each job averaging 2–3 hours to compute adjacency weights and solve the grouping problem.Source: §7 · p. 23
- Pairwise matching processes 8,120 adjacent sub-volume pairs in parallel, taking about 5–6 minutes per pair to reconcile labels across overlaps.Merge and split operations align segment groups within the overlap region.Source: §7 · p. 23
- Global consistency and output generation run as a single job taking 1–2 hours; Mojo proofreading is limited mainly by the human proofreader, not computation.Parallel proofreading of the same volume would require locking or conflict resolution.Source: §7 · p. 23
8 Large-scale reconstruction results
The pipeline automatically reconstructed long-range neuronal processes in a 27,000 µm^3 volume, but examined spine necks and some branching structures still required proofreading. The results include both continuous multi-section spine traces and long-range merge errors.
Large-scale reconstruction
- 93 objects met the criteria for long-range reconstructions across the whole volume, without manual proofreading.The selected objects crossed from one face of the volume to another without obvious 3D geometry errors; the reconstructions included orthogonal and horizontal paths.Source: §8 · pp. 23, 26, Figure 13 · p. 25
Spine neck reconstruction
- None of the examined spines was correctly reconstructed automatically, although some required only one split-error correction and others retained multi-section traces.Spine necks running longitudinally across sections were more fragmented than orthogonal necks.Source: §8 · p. 26, Figure 15 · p. 27
Merge errors
- Region grouping supports branching and spine-neck reconstruction, but can also merge distinct processes; the illustrated long-range errors were easy for a human proofreader to detect.Automatic identification and correction of these geometry errors remained future work.Source: §8 · pp. 26, 29
9 Conclusions
The paper concludes that its pipeline provides state-of-the-art automatic reconstruction at nm resolution, including long-range processes, while targeting large data sets. Future work focuses on improving performance and making proofreading easier.
Conclusions
- The pipeline demonstrates state-of-the-art performance for dense neuronal-tissue reconstruction and long-range processes spanning many µm.The workflow targets large-scale data sets at nm resolution.Source: §9 · p. 29
- The workflow is designed to minimize manual effort and parallelize on clusters and GPUs, with most steps scaling linearly with processor count.Source: §9 · p. 29
Future work
- Future work targets pipeline performance, runtime optimization, improved segmentation algorithms, and easier proofreading, including a planned collaborative web version of Mojo.Source: §9 · p. 29