PaperlayerView exact revision on arXiv

Source-linked AI summary

Patch-based Querying Identifies Structures of Interest in Electron Microscopy

Niels Vyncke, Nicolas Nadisic, Yvan Saeys, Aleksandra Pižurica

arXiv:2610.06020v1cs.CV

TL;DR

Large vEM datasets strain downstream analysis that relies on expert preprocessing and annotation. The paper introduces interactive patch retrieval using self-supervised local descriptors and reports that it can reduce the search space while retaining most target occurrences.

  • Problem

    Growing vEM datasets outpace downstream analysis that relies on expert annotation, while supervised pipelines often require substantial labels and offer limited interaction.

    Source: §1 · pp. 2–3

  • Method

    The framework uses a few example patches and self-supervised local descriptors to retrieve ranked candidate regions, with interactive query refinement and no dedicated structure-specific detector.

    Source: §1 · pp. 3–4, §2.1 · p. 6

  • Results

    At reduction rates as high as 70%, recall is nearly one in most cases, indicating that most target structures are retrieved while most irrelevant patches are discarded.

    Source: §4.6.1 · p. 23

  • Takeaways & Limitations

    The framework offers a screening step that reduces manual inspection and supports locating varied cellular structures across vEM datasets, including strongly anisotropic SBF-SEM data.

    Source: §1 · p. 4, §5 · p. 35

    The method retrieves 2D candidate patches rather than 3D object boundaries, so it is intended for visual inspection or subsequent automatic processing.

    Source: §5 · p. 36

Abstract

from arXiv · show

Volume electron microscopy (vEM) has emerged as an essential sensing technique in biomedical research, allowing the three-dimensional imaging of biological cells and tissues at nanometer-scale resolution. The ability to generate extensive datasets has reached the limitations of downstream analysis processes, which depend significantly on the intervention of human experts for preprocessing and annotation. We propose an efficient and reliable patch-based retrieval framework based on self-supervised learning of local image descriptors to locate self-similar structures in vEM datasets. Given a few manual annotations of a given cellular structure, our method can retrieve similar structures across the EM volume. Our framework is interactive, allowing the human expert to refine the search queries and retrieve relevant image patches quickly and using little labeled data. Experiments on real-world vEM images of biological tissues demonstrate that our framework can reliably identify relevant cellular structures, generalize across different organelles and acquisition modalities, and substantially reduce the search space for downstream analysis.

1. Introduction

vEM produces detailed 3D views of cells, but growing datasets outpace analysis that depends on expert annotation. This work proposes interactive, patch-based querying with learned descriptors to locate structures of interest and reduce manual search.

1. Introduction

  • vEM enables 3D imaging of cells at nanometer-scale resolution, while growing datasets outpace downstream analysis constrained by expert annotation and limited interaction.Existing deep-learning approaches often require substantial expert-labeled data, and supervised pipelines may be fixed to a specific structure after training.Source: §1 · pp. 2–3
  • The study explores self-supervised local descriptors for vEM search, with lightweight unsupervised fine-tuning on each target dataset.The evaluation uses three FIB-SEM datasets and one SBF-SEM dataset, fine-tuned and evaluated separately.Source: §1 · pp. 3–4
  • The descriptors retrieve diverse structures, including mitochondria, endoplasmic reticulum, vesicles, and the canalicular system, and generalize to strongly anisotropic SBF-SEM data.The authors report no prior use of local image descriptors to search for self-similar structures in EM datasets.Source: §1 · p. 4
  • The framework uses local image descriptors and positive and negative query patches to retrieve candidate regions, reducing manual inspection without training a structure-specific detector.It is designed as an upstream search step, complementary to pixel-accurate segmentation, with queries that users can refine interactively.Source: §1 · pp. 3–4
  • Learned descriptors improve retrieval over SIFT, HOG, and LBP across multiple structures and datasets.The study also compares pretrained and fine-tuned descriptors through visual and quantitative evaluations.Source: §1 · pp. 4–5
  • The framework is evaluated as a screening step that can discard many patches while retaining most occurrences of the target structure, reducing the search space for downstream analysis.The evaluation covers three FIB-SEM datasets and one SBF-SEM dataset, including strongly anisotropic data.Source: §1 · p. 5

2. The proposed patch-based search framework

The framework learns compact patch descriptors with self-supervised autoencoders and uses positive and negative examples to retrieve candidate regions. A proof-of-concept visualization shows patches from distinct structures separating in the descriptor space.

2.1. Self-supervised learning of local image descriptors

  • The method operates on two-dimensional grayscale image patches; extending it to three-dimensional patches is outside the work’s scope.A patch is typically a small rectangular region of a 2D slice.Source: §2.1 · p. 5
  • Convolutional autoencoders learn compact, lower-dimensional patch encodings by preserving information needed to reconstruct each input.The encoder’s latent code serves as the descriptor; the decoder reconstructs the patch.Source: §2.1 · p. 6

2.2. Discriminative analysis of the latent feature space

  • Patches from four structures formed clearly separated clusters in the learned descriptor space, supporting its ability to distinguish anatomical structures.Each patch was represented by its average MSE distance to four structure-specific reference sets, then projected to three dimensions with PCA for visualization.Source: §2.2 · p. 7, Figure 3 · p. 8

2.3. Search with positive and negative query trees

  • Separate approximate-nearest-neighbor trees store positive and negative query encodings, enabling fast retrieval of nearby examples for each candidate patch.The positive examples contain the target structure, while the negative examples do not.Source: §2.3 · pp. 8–9
  • The similarity score subtracts negative-query similarity from positive-query similarity, and the framework returns the N patches with highest scores.The score ranges from −1 to 1; higher values indicate greater similarity to the query structures.Source: §2.3 · pp. 9–10

3. Implementation details

The implementation uses classical and variational autoencoders, pretrained on general-image datasets and fine-tuned separately on target vEM datasets. Fine-tuning adapts general features to dataset-specific properties, while the pretrained models provide the starting point.

3.1. Model architectures

  • The classical autoencoder uses three convolutional layers with max pooling and is trained with MS-SSIM, while the variational version adds mean and variance outputs and a weighted KL-divergence term.The variational encoder replaces the classical model’s final fully connected layer with parallel mean and variance layers.Source: §3.1 · p. 11

3.2. Pretraining and fine-tuning procedures

  • Fine-tuning is used because vEM datasets vary statistically, no equivalent to HPatches exists for task evaluation, and prior experiments found training from scratch inferior to pretrained models.The paper frames this as transfer learning to adapt general image features to biomedical image details.Source: §3.2 · p. 12

3.2.1. Pretraining

  • Pretraining uses 125,000 patches of size 65 × 65 from ImageNet, KonIQ, and Visual Genome for 400 epochs at learning rate 10−4.The train/validation/test split is 8:1:1, the latent space size is 32, and no data augmentation is performed.Source: §3.2.1 · p. 12
  • HPatches is used to evaluate and tune hyperparameters for image matching, retrieval, and verification tasks.The paper notes that vEM lacks an equivalent benchmark for evaluating descriptor models on such tasks.Source: §3.2.1 · p. 12, §3.2 · p. 12

3.2.2. Fine-tuning

  • Target-dataset fine-tuning uses continuous learning at learning rate 10−5 for 20 epochs, with 108,000 extracted patches.The remaining hyperparameters match those of the pretrained models.Source: §3.2.2 · p. 13

4. Results and discussion

The evaluation uses three FIB-SEM datasets and an additional SBF-SEM dataset, assessing retrieval with precision and recall, including average precision over thresholds.

4.1. Validation datasets

  • The study evaluates the framework on HeLa-EMBL, EPFL, and VIB FIB-SEM datasets, plus Platelet-EM SBF-SEM data.HeLa-EMBL includes expert labels for mitochondria and ER; EPFL is labeled for mitochondria, while Platelet-EM includes labels for the cell body and five organelles.Source: §4.1 · pp. 13–14

4.2. Evaluation metrics

  • Precision measures the fraction of returned patches overlapping the target, while recall measures pixel-weighted retrieval completeness, giving larger structures proportionally more weight.A one-pixel overlap is sufficient for precision because the goal is to locate a structure rather than delineate its boundary.Source: §4.2 · pp. 14–15
  • Average precision summarizes the area under precision-recall curves computed across thresholds, enabling comparison between retrieval techniques.The curves assess performance independently of a single chosen threshold.Source: §4.2 · p. 15

4.3. Query selection strategy

The query-selection strategy samples slices across the volume, then extracts tiled patches with clear positive or negative labels for querying and evaluation.

4.3. Query selection strategy

  • The volume is divided into groups of a chosen split size, and each group’s middle slice is selected as a query slice.The remaining slices are pooled as a separate group, as illustrated in Figure 5.Source: §4.3 · p. 15, Figure 5 · p. 16
  • Non-overlapping 80×80 patches tile each query slice; patches with at least 50% target overlap are positive, and patches with no overlap are negative.Intermediate-overlap patches are excluded from query construction to avoid ambiguous training signals.Source: §4.3 · p. 15
  • The same selection scheme is applied at query and evaluation time, with split sizes of 10, 20, 30, 50, or 100 slices, and nearest-neighbor k=1.At evaluation, all patches are scored; split size 100 is excluded for the 64-slice HeLa-EMBL dataset.Source: §4.3 · pp. 15–16

4.4. Visual evaluation of the local image descriptors

The visual evaluation uses t-SNE only to inspect latent-space organization, not to compute descriptors or perform retrieval. It finds improved separation after fine-tuning, strong local discrimination, and separation among four structures.

4.4.2. Local vs. global descriptor performance

  • Across groups of 10 slices, both autoencoders separate patches well locally but do not perfectly generalize across distant regions.The authors suggest providing more queries when accuracy decreases.Source: §4.4.2 · p. 17

4.4.3. Multiple structures

  • For VIB patches, both classical and variational autoencoders show nearly perfect separation among mitochondria, nuclear envelope, endoplasmic reticulum, and vesicle clusters.The evaluation uses patches from five slices and a 75% overlap threshold.Source: §4.4.3 · p. 17

4.4.1. Pretrained vs. fine-tuned descriptors

  • Fine-tuning improves descriptor separation: the classical AE shows perfect separation, while both pretrained and fine-tuned VAEs show near-perfect separation with a small negative region among positives.Pretrained AE patches separate but appear intermingled with little structure; fine-tuned AE subregions are closer together.Source: §4.4.1 · p. 17, Figure 6 · p. 18

4.5. Visual evaluation of retrieved patches

Retrieved-patch examples show that positive and negative queries return ranked candidate patches across datasets, with some false positives but useful results for human review.

4.5. Visual evaluation of retrieved patches

  • Although some returned patches are false positives, the reduced search space and informative results can help human experts enhance labeling.The authors describe false positives as acceptable after the search space has been substantially reduced.Source: §4.5 · p. 18
  • Similar retrieval examples are shown for HeLa-EMBL and EPFL; another example uses 100 positive and 100 negative queries to retrieve the top 25 patches across two slices.The queries are automatically extracted from a (pseudo-)labeled slice.Source: §4.5 · p. 18
  • For VIB, the top 12 patches retrieved by fine-tuned AE and VAE are ranked by decreasing similarity, with true positives and false positives marked in green and red.The examples use two positive and two negative queries.Source: §4.5 · p. 18, Figure 10 · p. 22

4.6. Performance evaluation of the proposed framework

Across datasets, patch retrieval can screen out much of the volume while retaining strong recovery. Fine-tuned descriptors also perform well across structures and acquisition settings.

4.6.1. Discussion on the PR-curves

  • At 70% reduction, recall is nearly one in most cases, meaning users can skip most patches while retrieving almost all structures.Similar results are reported for VIB and EPFL; performance improves with more query slices, likely because distant slices are less similar.Source: §4.6.1 · p. 23

4.6.2. Discussion on the average precisions

  • Fine-tuning raised median AP by +3.65 pp for the classical autoencoder and +2.80 pp for the variational autoencoder across 14 configurations.The classical autoencoder also significantly outperformed the fine-tuned variational autoencoder; gains were lower on VIB, a more challenging setting that the authors call a conservative estimate.Source: §4.6.2 · pp. 24, 26

4.6.3. Operational use in downstream analysis

  • Retaining the top 30% of patches preserved 93.8% of mitochondrial instances and 78.4% of mitochondrial pixels; retaining 50% preserved 95.4% and 88.9%, respectively.This allows downstream segmentation or quantitative analysis to focus on a smaller search space while retaining most target occurrences.Source: §4.6.3 · pp. 26, 28

4.6.4. Generalization to larger z-distances

  • At a z-gap of approximately 320 slices, retrieval recovered 8 of 9 structures; within ±5 slices, precision was approximately 90% and recall 83%.The authors report that retrieval quality declines smoothly with distance and that denser queries have diminishing returns beyond typical z-correlation lengths.Source: §4.6.4 · p. 28

4.6.5. Retrieval of a second, non-mitochondrial structure: endoplasmic reticulum

  • For ER retrieval, the fine-tuned classical autoencoder reached AP 0.969 at split size 10 and 0.882 at split size 30, comparable to mitochondria on the same dataset.The ER experiment adapted the positive-overlap threshold to 15% for its smaller, thin sheet-like structures, without changing model architecture or training procedure.Source: §4.6.5 · pp. 28, 32, Table 3 · p. 32

4.6.6. Comparison with handcrafted descriptors

  • The fine-tuned autoencoder achieved the highest AP for all seven structure/dataset combinations, including 0.79 versus 0.71 for SIFT on VIB vesicles.Handcrafted descriptors were reduced to the same 32 dimensions for comparison.Source: §4.6.6 · pp. 32–33, Figure 19 · p. 33

4.6.7. Generalization to a different acquisition modality: serial block-face SEM

  • On Platelet-EM SBF-SEM, AP reached 0.55 for mitochondria, 0.78 for alpha granules, and 0.81 for canalicular vessels; mitochondria had a 7% base rate.Retrieval transferred to strongly anisotropic SBF-SEM without architectural changes. Estimated workflow speed-up was approximately 4× (74–75% less analysis time), under the stated assumptions; retrieval took at most a few tens of seconds.Source: Table 4 · p. 34, §4.6.7 · p. 34, §4.6.8 · pp. 34–35

5. Conclusion

The conclusion presents patch retrieval as a target-flexible screening stage that narrows the search space before downstream analysis. It reports benefits from fine-tuning while noting boundaries in scope and output.

5. Conclusion

  • The framework reduces the search space for visual analysis or further processing, using flexibly chosen queries and allowing interactive query refinement.The conclusion states that query refinement improves performance.Source: §5 · p. 35
  • The framework returns candidate regions from a few example patches, complementing rather than replacing structure-specific segmentation.Retrieval determines where to look; segmentation delineates what is there.Source: §5 · p. 36
  • Fine-tuning improves pretrained models for vEM structures, with the classical autoencoder benefiting more than the variational autoencoder.The authors suggest the VAE’s additional β parameter requires careful tuning, which is impractical for this use case.Source: §5 · p. 36
  • Performance depends on target morphology and prevalence, declines with z-distance, and has not yet been broadly evaluated across structures, modalities, and unseen datasets.The method retrieves 2D candidate patches rather than 3D object boundaries and is intended for upstream use.Source: §5 · p. 36

CRediT authorship contribution statement

The contribution statement assigns formal analysis, investigation, methodology, software, visualization, and the original draft to N. Vyncke.

CRediT authorship contribution statement

Ethics statement

The study analyzes existing datasets and reports that it involved no new human or animal experiments.

Ethics statement

  • The study conducted no new experiments on human or animal subjects.The analyses relied exclusively on existing datasets.Source: Ethics statement · p. 37
  • The datasets were publicly available or obtained through data-sharing agreements subject to institutional policies.The statement names Ghent University and the VIB institute policies.Source: Ethics statement · p. 37
  • The authors state that ethical approval was not required for this study because it involved no new experimental procedures, animal handling, or interventions.Original data collection followed applicable laws, institutional guidelines, and ethics approvals as reported by the providers.Source: Ethics statement · p. 37
Loading 2610.06020v1…