PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecture

Rüdiger Schmitz, Frederic Madesta, Maximilian Nielsen, Jenny Krause, René Werner, Thomas Rösch

arXiv:1909.10726v3eess.IVcs.CVcs.LGq-bio.TO

TL;DR

Histopathologic diagnosis integrates information from widely different spatial scales, but prior end-to-end multiscale segmentation approaches had shown limited benefits over ensembles. This paper introduces multi-encoder FCNs with spatially preserving fusion and context-classification guidance; across three datasets, the models consistently improved on single-scale U-Nets.

  • Problem

    Histopathology segmentation must integrate features across widely different scales, while prior end-to-end multiscale approaches had shown only minor benefits over ensembles.

    Source: §1.1 · p. 2, §1.3 · p. 6

  • Method

    The authors develop multi-encoder FCNs that combine detail and context paths using spatially preserving fusion, with context-classification gating as an alternative.

    Source: §2.2.3 · p. 9, §6 · p. 31

  • Results

    Across three datasets, multi-scale architectures consistently improved segmentation over baseline single-scale U-Nets.

    Source: §6 · p. 31

  • Takeaways & Limitations

    The findings support integrating extensive multi-scale context into deep learning models for complex computational-histopathology tasks.

    Source: §6 · p. 33

    Evidence so far covers three public H&E whole-slide datasets using cross-validation; generalization to other data, tasks, and diseases and confirmation on independent test databases remain open.

    Source: §6 · p. 33

Abstract

from arXiv · show

Histopathologic diagnosis relies on simultaneous integration of information from a broad range of scales, ranging from nuclear aberrations ($\approx \mathcal{O}(0.1{\mu m})$) through cellular structures ($\approx \mathcal{O}(10{\mu m})$) to the global tissue architecture ($\gtrapprox \mathcal{O}(1{mm})$). To explicitly mimic how human pathologists combine multi-scale information, we introduce a family of multi-encoder FCNs with deep fusion. We present a simple block for merging model paths with differing spatial scales in a spatial relationship-preserving fashion, which can readily be included in standard encoder-decoder networks. Additionally, a context classification gate block is proposed as an alternative for the incorporation of global context. Our experiments were performed on three publicly available whole-slide images of recent challenges (PAIP 2019, BACH 2020, CAMELYON 2016). The multi-scale architectures consistently outperformed the baseline single-scale U-Nets by a large margin. They benefit from local as well as global context and particularly a combination of both. If feature maps from different scales are fused, doing so in a manner preserving spatial relationships was found to be beneficial. Deep guidance by a context classification loss appeared to improve model training at low computational costs. All multi-scale models had a reduced GPU memory footprint compared to ensembles of individual U-Nets trained on different image scales. Additional path fusions were shown to be possible at low computational cost, opening up possibilities for further, systematic and task-specific architecture optimization. The findings demonstrate the potential of the presented family of human-inspired, end-to-end trainable, multi-scale multi-encoder FCNs to improve deep histopathologic diagnosis by extensive integration of largely different spatial scales.

1. Introduction

Histopathologic diagnosis draws on features spanning nuclear, cellular, and tissue-architecture scales, much as pathologists inspect regions at different magnifications. Because prior end-to-end multiscale segmentation approaches had shown only minor benefits over ensembles, this work develops U-Net-based models to integrate widely different scales.

1.1. Clinical relevance and motivation

  • Histopathologic diagnosis combines features across scales, and pathologists repeatedly inspect regions at different magnifications.The scales span more than a thousand-fold, from sub-nuclear features to glandular and other higher-organizational features.Source: §1.1 · p. 3

1.2. Related works

  • Prior histopathology multiscale models showed only minor benefits over ensembles, while natural-image context methods used more similar scales than whole-slide histopathology requires.The authors therefore ask whether a human-inspired architecture can integrate otherwise unavailable information from widely different scales.Source: §1.2 · p. 5, §1.3 · p. 6

1.3. Contributions

  • The paper introduces U-Net-based FCNs with spatially preserving multi-encoder fusion or context-classification gating, and evaluates them against U-Net baselines across three public datasets.Its stepwise analysis also examines spatial correspondence, additional classification-loss guidance, and multiple path fusions.Source: §1.3 · p. 6

2. Model architectures

The model family builds on a standard U-Net and adds separate context paths that integrate information from different image scales. The paper describes spatially aligned feature fusion, context-classification gating, and variants for combining local and global context.

2.1.1. U-Net architecture; 2.1.2. Multi-scale ensembles as an upper bound for state-of-the-art performance

  • A ResNet18-based U-Net trained on full-resolution patches is the baseline; scale-specific U-Nets are also ensembled to establish an upper-bound comparison.The ensemble combines re-sampled predictions using hard votes, average voting, or logistic regression.Source: §2.1.1 · p. 7, §2.1.2 · pp. 7–8

2.2. The msY model family: Multi-scale multi-encoder architectures

  • The msY family extends the baseline Res-U-Net with multi-scale encoders to provide context and architectural information.Whole-slide image pyramids allow multiple scales to be loaded directly, with moderate overhead.Source: §2.2 · p. 8

2.2.1. Multi-scale merge block: spatial relationship preserving path fusion in multiscale multi-encoder models

  • The multi-scale merge block aligns context-encoder features spatially with the main path, concatenates them, and uses a 1 × 1 convolution to reduce feature maps.For multiple context encoders, their aligned features are concatenated before the 1 × 1 convolution.Source: §2.2.1 · pp. 8–9

2.2.2. Context classification gate

  • The context classification gate predicts which classes occur in a detail patch from global context, then gates segmentation probabilities while retaining an ungated feature path.The context classifier treats the task as multi-label classification; concatenating the original and excited features lets a 1 × 1 convolution learn how to combine them.Source: §2.2.2 · p. 9

2.2.3. msY-Net: Integrating context and tissue architecture

  • msY-Net processes a full-resolution patch and a scale-4 or scale-16 context patch in separate encoders, with their paths merged at the U-Net bottleneck.Only the full-resolution patch is segmented, so the context encoder has no decoder skip connections.Source: §2.2.3 · p. 9

2.2.4. msYI-Net and msY2-Net: Integration of global and local context

  • msY2-Net uses two context encoders, whereas msYI-Net adds a large-context classification path whose output gates the underlying msY-Net segmentation.Unlike the merged paths in msY-Net and msY2-Net, msYI-Net’s large-context path need not be spatially aligned and can use arbitrary patch scales.Source: §2.2.4 · p. 10

2.2.5. Context classification loss

  • A global-context encoder predicts detail-patch content with an additional binary cross-entropy classification loss added to the segmentation loss.The authors intended this guidance to improve gradient flow through the context encoder and focus it on detail-patch content.Source: §2.2.5 · p. 11

2.2.6. Early and multiple fusions by multi-scale merge blocks

  • Merge blocks can connect encoders directionally at different levels, and multiple fusions may enable further processing of combined features.Very early fusions provide little context because center crops have small receptive fields; multiple merges add learnable parameters only through their 1 × 1 convolutions.Source: §2.2.6 · p. 11

3. Materials & methods

The methods evaluate multi-scale histopathology segmentation across three challenge datasets, using shared preprocessing, cross-validation, and statistical comparisons with a single-scale baseline.

3.1. Datasets

  • Experiments used three publicly available challenge datasets—PAIP 2019, BACH 2018, and a 20-WSI CAMELYON 2016/MM subset—and five-fold cross-validation with splits conducted at the entire-WSI level.Source: §3.1 · p. 11, §3.1.3 · p. 17, §3.3 · p. 18

3.1.1. PAIP 2019

  • HCC’s pseudocapsule and architectural abnormalities make PAIP 2019 suitable for examining the value of large-scale context.Low-grade HCC may be identifiable from long-range tissue architecture despite minor nuclear abnormalities.Source: §3.1.1 · p. 15

3.1.2. BACH 2018

  • BACH 2018 provides 10 pixel-wise annotated breast-biopsy WSIs labeled as normal, benign, in situ, or invasive carcinoma.The closely interwoven diagnoses pose a challenging differential-diagnosis task.Source: §3.1.2 · p. 16

3.1.3. CAMELYON 2016/MM subset

  • CAMELYON 2016/MM selects 20 WSIs containing macrometastases, but diagnosis can still rely on individual tumor-cell differences, reducing the importance of multi-scale context.The subset retains isolated tumor cells and micrometastases alongside macrometastases.Source: §3.1.3 · p. 17

3.2. Preprocessing

  • Preprocessing adds an automatically thresholded overall-tissue class and applies Reinhard color normalization using tissue regions rather than white background.Images are then normalized to channel-wise zero average and unit variance in tissue regions.Source: §3.2 · pp. 17–18

3.3. Evaluation metrics

  • The primary outcome is a class-weighted Jaccard index; BACH additionally uses its challenge metric, based on pixel accuracy penalized by class distance.Validation evaluates fixed-size sub-images sampled to represent available classes.Source: §3.3 · p. 18

3.4. Statistical analysis

  • Performance is summarized by means and 95% confidence intervals across five splits, with corrected cross-validation t-tests limited to multi-scale versus baseline comparisons.The study states it is underpowered to test significance differences among multi-scale models.Source: §3.4 · p. 19

4. Experiments

The experiments explore multi-scale architectures on PAIP 2019, then assess selected models on BACH and CAMELYON and test context guidance, spatial alignment, and repeated fusion.

4. Experiments

  • Experiments first explore the msY family on PAIP 2019, then test selected architectures on BACH and CAMELYON and investigate design variants on PAIP.Training settings were optimized for the baseline Res18-U-Net and kept fixed across models and experiments.Source: §4 · p. 19

4.1.1. PAIP 2019

  • PAIP 2019 evaluates tumor extent and margin-status segmentation, weighting viable tumor more heavily than whole tumor in both loss and weighted Jaccard scoring.Comparisons with the baseline U-Net are the primary focus; comparisons among multi-scale variants are descriptive because the study is underpowered for significance testing.Source: §4.1.1 · pp. 20–21

4.1.2. BACH 2018

  • BACH models are fine-tuned from PAIP models and assessed on four breast-lesion classes using weighted Jaccard and the BACH challenge metric.The first 10 pseudo-epochs train only output layers before all weights are fine-tuned.Source: §4.1.2 · pp. 21–22

4.1.3. CAMELYON 2016/MM

  • CAMELYON 2016/MM models are fine-tuned from PAIP models, initially training only output layers and continuing until at least 60 pseudo-epochs or convergence.Fine-tuning pairs models and dataset splits by matching split index.Source: §4.1.3 · p. 22

4.2. Context classification loss

  • The context-classification-loss experiment compares PAIP models with and without that loss when using global-scale context.The tested models include msY(16), msY2, and msY(16),MM.Source: §4.2 · p. 22

4.3. Spatial alignment in multi-scale merge blocks

  • Spatial alignment is tested by replacing the msY2 merge block with concatenation and a 1 × 1 convolution, removing scale alignment and spatial-relationship preservation.Additional variants test random initialization and an initialization that approximately preserves the pretrained main-encoder weights.Source: §4.3 · pp. 22–23

4.4. Multiple merging

  • The multiple-merging experiment compares standard msY variants with models that fuse at every encoder level and measures the resulting GPU memory footprint.The msY2 global-context encoder uses ResNet34 in the maximum-variant experiment; the other models and encoders use ResNet18.Source: §4.4 · p. 23

5. Results

Multi-scale inputs substantially improved on the baseline U-Net, and the proposed multi-encoder models matched the best multi-scale ensemble with lower resource requirements. Results across datasets support these findings, while the ablations favor classification-loss guidance and spatially aligned fusion; additional fusion levels showed no relevant performance differences.

5.1. Multi-scale multi-encoder models improve histopathology image segmentation

  • Multi-scale input considerably improved on the baseline U-Net, while proposed multi-encoder models matched the best multi-scale ensemble with lower resource requirements.For msY2-Net, the parameter count was 76.3% of the corresponding three-U-Net ensemble and GPU footprint was reduced by 54.5% when trained in parallel.Source: §5.1 · p. 24
  • On CAMELYON 2016/MM, the performance increase from multiple scales was 6.8%, compared with 16.3% on both PAIP 2019 and BACH 2018.The authors describe the CAMELYON increase as smaller, consistent with differences in task nature and difficulty.Source: Table 3 · p. 26, §5.1 · p. 27

5.2. Context classification loss-based deep guidance for context encoder training

  • Additional classification loss consistently improved context-encoder training, with only moderate overhead in parameters and GPU memory.The comparison tested models trained with and without the classification loss on PAIP 2019.Source: §5.2 · pp. 27–28, Table 4 · p. 28

5.3. Necessity for spatial alignment in multi-scale merge blocks

  • Spatially aligned fusion outperformed both unaligned msY2-Net variants, regardless of how their merging 1×1 convolution was initialized.The authors conclude spatial matching is necessary when merging paths at the bottleneck in this manner.Source: §5.3 · pp. 28–29, Table 5 · p. 29

5.4. Additional path fusions can be added by introduction of multiple multi-scale merge blocks

  • The study found no relevant performance differences between single-merge and multiple-merge setups, while additional fusions caused only moderate increases in parameters and GPU memory.The additional connections did not lead to training instability and can be used for further architecture optimization.Source: §5.4 · p. 29, Table 6 · p. 30

6. Discussion and Conclusions

The multi-scale FCNs improved carcinoma segmentation across three datasets, while spatially aligned fusion and context integration were central to the discussion. The authors also identify limits to generalization and several avenues for further architecture and loss-function work.

6. Discussion and Conclusions

  • Multi-scale integration significantly improved segmentation over baseline single-scale U-Nets across three datasets, clinical scenarios, and tumor entities.The authors conclude that mimicking how human experts approach the task can support specialized machine-learning architectures.Source: §6 · pp. 31, 33
  • The proposed architectures were at least on par with U-Net ensembles while using fewer parameters and less GPU memory.Only common ensembling techniques were studied, so more sophisticated techniques could potentially improve ensemble performance.Source: §6 · p. 31
  • The encoder, decoder, and skip connections remain unchanged, leaving room for receptive-field, multiresolution, and wavelet-based extensions.The authors suggest that larger receptive fields in context encoders and multilevel wavelet transforms may benefit the models.Source: §6 · pp. 31–32
  • Multi-scale merge blocks supported additional path fusions at minor GPU-memory overhead, suggesting a search space for task-specific architectures.The blocks can be used at many levels, including all possible encoder levels, and extended to arbitrarily sized spatial scales.Source: §6 · p. 32
  • Adding a context content loss improved performance, while application- or architecture-specific loss design may offer further improvement.The study used binary cross entropy as its loss function.Source: §6 · p. 32
  • Evidence is limited to three public H&E WSI datasets and cross-validation; independent test databases and generalization to other data, tasks, and diseases remain untested.Performance remained dataset-dependent, with larger gains on tasks requiring multi-scale integration than on simpler cell-focused tasks.Source: §6 · p. 33

Supplementary Materials to: Multi-scale fully convolutional neural

The supplied passages give part of the paper title and its author list.

Supplementary Materials

  • The title concerns multi-scale fully convolutional neural networks for histopathology image segmentation, from nuclear aberrations to global tissue architecture.Source: Supplementary Materials to:… · p. 44
  • The listed authors are Rüdiger Schmitz, Frederic Madesta, Maximilian Nielsen, Jenny Krause, Stefan Steurer, René Werner, and Thomas Rösch.Source: Supplementary Materials to:… · p. 44

S1. Base model architecture

The baseline model architecture is detailed in a supplementary figure.

S1. Base model architecture

S2. Establishment of the CAMELYON 2016 macrometastases subset: CAMELYON 2016/MM

The CAMELYON 2016/MM subset was assembled by screening candidate slides for macrometastases and suitability for segmentation analysis.

S2. Establishment of the CAMELYON 2016 macrometastases subset: CAMELYON 2016/MM

  • Candidate whole-slide images were sorted by total tumor-region pixel size, then visually checked for macrometastases.Source: CAMELYON 2016/MM · p. 44
  • Slides with almost no non-cancerous tissue or almost no healthy–cancerous border were excluded as unsuitable for segmentation-performance analysis.Source: CAMELYON 2016/MM · p. 44
  • The resulting subset contains 20 slides, identified by their original CAMELYON 2016 filenames.Source: CAMELYON 2016/MM · p. 44

S3. Baseline model results

The baseline was a ResNet18-based U-Net, repeated in three independent training runs. Validation results report class-average Jaccard indices for whole and viable tumor at convergence.

S3. Baseline model results

  • Three independent training runs were conducted for the baseline model.Detailed results for the individual repetitions are given in Table S1.Source: S3. Baseline model results · pp. 44, 46
  • The baseline architecture is a ResNet18-based U-Net with an encoder that downsamples spatial dimensions and a bilinearly upsampling decoder.Encoder blocks downscale by a factor of 2, using max pooling or strided convolutions.Source: Figure S1 · p. 45, S3. Baseline model results · p. 45
  • Table S1 reports class-average validation Jaccard indices for whole and viable tumor at convergence.Source: Table S1 · p. 45

S4. Statistical significance tests and estimation of the statistical power of the study

The study used corrected resampled paired-sample t-tests with a one-sided test and Benjamini–Hochberg correction. Its estimated power supports comparisons against the baseline U-Net, but not reliable comparisons among multi-scale models.

S4. Statistical significance tests and estimation of the statistical power of the study

  • A corrected resampled paired-sample t-test addressed dependence between cross-validation splits, with a one-sided test and Benjamini–Hochberg correction for multiple testing.The one-sided test was used because multi-scale architectures can reproduce baseline U-Net behavior.Source: S4. Statistical significance tests and… · p. 46
  • The power estimate assumed a single-model Jaccard-index standard deviation of 0.045 and a difference standard deviation of 2 · 0.045, with a resampling correction factor of 1.5 for CAMELYON.Source: S4. Statistical significance tests and… · p. 46
  • 0.613–0.888 was the estimated statistical power for detecting the targeted 0.1–0.15 improvement over the baseline U-Net.The study was estimated to have power 0.253 for expected 0.05 differences between multi-scale models, so those within-family comparisons were considered underpowered.Source: S4. Statistical significance tests and… · p. 46

S5. GPU memory footprint analysis

GPU memory use during training was measured for forward and backward passes, then modeled across batch sizes. The regression slope represents per-patch GPU footprint, while the intercept represents model overhead.

S5. GPU memory footprint analysis

S6. BACH 2018

The BACH 2018 comparison reports that the models’ custom-metric performance lies in the range of the original challenge’s top-performing teams. That custom metric was excluded from the study because of its class-scale ambiguity and dominance by background contributions.

S6. BACH 2018

  • The models’ BACH 2018 custom-metric performance lies in the range of the original challenge’s top-performing teams: 1st place 0.69, 2nd 0.55, and 3rd 0.52.The metric was used in the original challenge, and Table S3 provides selected multi-scale architectures versus the baseline U-Net.Source: S6. BACH 2018 · p. 47, Table S3 · p. 49
  • The BACH 2018 custom metric was not used in the study because class-to-rational-scale mapping was unclear and the trivial background class dominated its score.Source: S6. BACH 2018 · p. 47
  • GPU memory footprint is reported in gigabytes per image patch in a batch.Source: S6. BACH 2018 · p. 49

Related papers

Loading 1909.10726v3…