Source-linked AI summary
HookNet: multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images
Mart van Rijthoven, Maschenka Balkenhol, Karina Siliņa, Jeroen van der Laak, Francesco Ciompi
TL;DR
Histopathology segmentation needs both high-resolution detail and contextual information, while larger receptive fields impose memory and optimization constraints. HookNet combines concentric multi-resolution patches through multiple encoder-decoder branches and aligned feature hooking, and improves overall performance over single-resolution models across the evaluated breast and lung tasks.
Problem
High-resolution segmentation must incorporate contextual information despite CNN memory and optimization constraints associated with larger receptive fields and inputs.
Method
HookNet uses multiple encoder-decoder branches fed by concentric multi-resolution patches, concatenating aligned contextual features into the target branch through hooking.
Results
HookNet increased overall performance compared with single-resolution models while handling subtle high-resolution differences and contextual information across two histopathology datasets.
Takeaways & Limitations
The framework supports high-resolution tissue segmentation when prediction depends simultaneously on fine details and contextual information.
Takeaways & Limitations
The study applied HookNet to histopathology whole-slide images and identifies broader applicability and constraint design as future research directions.
Abstract
from arXiv · showhide
We propose HookNet, a semantic segmentation model for histopathology whole-slide images, which combines context and details via multiple branches of encoder-decoder convolutional neural networks. Concentricpatches at multiple resolutions with different fields of view are used to feed different branches of HookNet, and intermediate representations are combined via a hooking mechanism. We describe a framework to design and train HookNet for achieving high-resolution semantic segmentation and introduce constraints to guarantee pixel-wise alignment in feature maps during hooking. We show the advantages of using HookNet in two histopathology image segmentation tasks where tissue type prediction accuracy strongly depends on contextual information, namely (1) multi-class tissue segmentation in breast cancer and, (2) segmentation of tertiary lymphoid structures and germinal centers in lung cancer. Weshow the superiority of HookNet when compared with single-resolution U-Net models working at different resolutions as well as with a recently published multi-resolution model for histopathology image segmentation
1. Introduction
Histopathology segmentation must capture both fine cellular details and broader tissue context, but CNN receptive-field expansion increases memory demands. HookNet addresses this challenge by combining multi-resolution, multi-field-of-view inputs in a multi-branch segmentation framework.
- Semantic segmentation groups pixels by concept and supports tumor detection and tissue characterization in histopathology.
- 1.1. Context and details in histopathology: Pathologists combine global tissue architecture, local cellular composition, and immune-cell distribution to distinguish histological patterns.
- 1.2. The receptive field and the field of view: CNNs can enlarge receptive fields for contextual information, but larger inputs and feature maps increase memory consumption and constrain optimization.
- 1.2. The receptive field and the field of view: Lower-resolution patches provide wider fields of view with the same model and input size, enabling greater contextual aggregation.
- Combining high-resolution details with contextual features remains an unsolved challenge for fine-grained, high-resolution segmentation.
- 1.4. Our contribution: HookNet uses multi-branch encoder-decoder CNNs that concatenate features between branches through hooking to produce fine-grained segmentation maps.
2. Materials
The study evaluates HookNet on breast and lung histopathology segmentation tasks using datasets with expert or manually guided annotations. The materials include multi-resolution tissue imagery and concentric patches for combining contextual and detailed views.
- The study evaluates multi-class breast tissue segmentation and TLS and GC segmentation in lung cancer sections.
- 2.1. Breast dataset: The breast dataset contains 86 cancer tissue sections comprising IDC (n=34), DCIS (n=35), and ILC (n=17).
- 2.1. Breast dataset: Breast slides were scanned at 0.24 µm/px, with ILC sections additionally restained using P120 to guide identification of lobular carcinoma cells.
- HookNet processes concentric multi-resolution patches through multiple encoder-decoder branches and combines them using a hooking mechanism.
- The lung dataset comprises 27 diagnostic H&E-stained slides from TCGA-LUSC with sparse annotations of TLS, GC, tumor, and other lung parenchyma.
3. HookNet: multi-branch encoder-decoder network
HookNet combines contextual and fine-grained information for histopathology segmentation through two encoder-decoder branches processing concentric patches at different resolutions. Its hooking mechanism and design constraints align and concatenate decoder features while supporting joint target and contextual training losses.
- Branches and inputs: HookNet uses context and target encoder-decoder branches to combine low-resolution, large-field-of-view context with high-resolution, fine-grained details.The branches process concentric multi-field-of-view, multi-resolution patches for target segmentation.
- Branches and inputs: Each branch is based on U-Net, with separate weights and encoder-decoder paths using valid convolutions, pooling, and nearest-neighbor up-scaling.The described design uses valid 3x3 convolutions, 2x2 max-pooling, and nearest-neighbor 2x2 up-scaling followed by convolution.
- Branches and inputs: Concentric patches provide matched target and context regions, with the target patch corresponding to the central region of the lower-resolution context patch.The input is a pair of RGB patches at target and context resolutions, with field-of-view correspondence determined by their resolutions.
- Hooking mechanism: Pixel-wise alignment requires even feature-map sizes before pooling and resolutions satisfying 2^D r_T ≥ r_C, where D is encoder depth.The constraints prevent misalignment through pooling, skip connections, and cross-branch hooking.
- Hooking mechanism: Hooking concatenates decoder feature maps from the context branch into the target branch, typically at the target decoder’s beginning after the bottleneck.The mechanism uses cropping when corresponding feature maps have equal spatial resolution, preserving pixel-wise alignment.
- Training: Training can use a target loss alone or combine high- and low-resolution pixel-wise categorical cross-entropies as L = λL_high + (1 − λ)L_low.The target branch produces the main high-resolution segmentation, while the context branch can also generate lower-resolution predictions.
- Training: The implementation addresses class imbalance through pixel-based sampling and uses depth-4 branches with 284x284x3 patches and mini-batches of 12.Convolutional layers use valid convolutions, L2 regularization, ReLU activations, and batch normalization.
4. Experiments
The experiments compare HookNet with single-resolution U-Nets across multiple resolutions and with the multi-resolution MRN model. The evaluation varies context resolution while maintaining a high-resolution target branch and documents architectural and training differences affecting comparison conditions.
- Baselines and resolutions: HookNet is compared with five U-Nets trained at 0.5, 1.0, 2.0, 4.0, and 8.0 µm/px.The U-Net baselines are denoted U-Net(r_t), while HookNet is denoted HookNet(r_t, r_c).
- Baselines and resolutions: All HookNet experiments use a 0.5 µm/px target branch, with context branches tested at 2.0 or 8.0 µm/px for breast tissue.These settings represent intermediate and extreme context resolutions.
- Hooking configurations: Hooking connects context features at depth 2 to the target decoder bottleneck for 2.0 µm/px context, and from the context decoder end for 8.0 µm/px context.The target branch has resolution 0.5 µm/px in these configurations.
- Comparison with MRN: HookNet is compared with MRN, differing in padding, branch structure, and up-sampling: HookNet uses valid convolutions, an encoder-decoder context branch, and target-decoder up-sampling.MRN uses same padding, an encoder-only additional branch, and multiple independent up-samplings.
- Training: Single-loss U-Net and HookNet models train in approximately 2 days, whereas context-loss HookNet and MRN require approximately 2.5 days.Training times were measured using a GeForce GTX 1080 Ti and 10 CPUs; the authors relate the increase to the extra loss and larger MRN feature maps.
5. Results
Across breast and lung segmentation, single-resolution models exposed a trade-off between contextual coverage and fine detail, while HookNet combined resolutions to improve overall performance and reduce context-related confusion.
- Single-resolution models: DCIS F1 rose from 0.47 at 0.5 µm/px to 0.86 at 8.0 µm/px, whereas ILC F1 fell from 0.85 to 0.20 across those resolutions.Limited context confused DCIS with IDC, while limited detail confused ILC with IDC.
- Single-resolution models: Lung performance generally increased with context, chiefly through higher GC F1 at 2.0 µm/px, while reduced detail increased Tumor–Other confusion.This mirrored the breast-tissue trade-off between contextual information and fine-grained detail.
- Multi-resolution models: HookNet achieved an overall F1 of 0.91 on breast tissue with HookNet(0.5, 8.0) and λ=0.75, improving nearly all classes over either single-resolution U-Net.ILC F1 increased by 0.66 versus U-Net(8.0), while DCIS F1 increased by 0.37 versus U-Net(0.5).
- Multi-resolution models: On lung tissue, HookNet increased TLS and GC F1 by 0.03 and 0.1 versus U-Net(0.5), respectively, but decreased Tumor F1 by 0.01.The Other-class F1 was unchanged; individual classes were best handled by a mixture of competing models, but HookNet had the highest overall F1.
- Multi-resolution models: HookNet outperformed MRN in overall F1 on both breast and lung segmentation when using the same fields of view.The best breast model emphasized the target branch with λ=0.75, whereas the best lung model used λ=1, ignoring context loss.
- Statistical comparison: Breast differences between HookNet and U-Net or MRN were significant, with p-values of 0.004 and 0.001, whereas lung differences were not significant.The authors associate the stronger breast benefit with wide contextual information and describe a smaller but still beneficial context contribution in lung tissue.
6. Discussion
HookNet combines contextual and high-resolution information for tissue segmentation while addressing feature-map alignment and practical design constraints. The paper demonstrates applications in breast and lung cancer histopathology, while identifying loss weighting, annotation density, and model capacity as important boundaries.
- Framework and design: HookNet combines context and details through multi-resolution feature hooking, using same-resolution concatenation, pixel-wise alignment, and valid convolutions to mitigate output artifacts.The framework uses MFMR patches and experimentally selects branch fields of view for task-specific segmentation.
- Applications: The framework produces multiclass segmentation models for breast and lung cancer H&E histopathology, including tumor and other tissue classes.The breast model distinguishes DCIS, IDC, and ILC; the methods are described as extendable to arbitrary class counts and other organs.
- Applications: TLS and GC segmentation could automate TLS detection at scale and support investigation of TLS count as a prognostic and predictive factor.Segmenting tumor and other tissue types also enables characterization of tumor morphology and tissue architecture, including tumor–stroma interfaces.
- Limitations and future work: HookNet and comparison models were limited to 50M parameters, enabling training on a single modern GPU with 11GB of RAM.Adding more branches will likely require multiple GPUs and could enable deeper or wider networks and faster inference.
- Loss weighting: λ=0.75 performs best for breast tissue segmentation, whereas λ=1.0 scores best for lung tissue segmentation.The authors identify extensive optimization of λ as future work.
- Limitations and future work: Sparse manual annotations limited performance assessment in transition zones between tissue types.The authors propose evaluating an additional densely annotated dataset and generating such annotations in future work.
7. Conclusion
The paper proposes HookNet for high-resolution tissue segmentation in histopathology, combining subtle details with contextual information. Across two datasets, the model improves overall performance over single-resolution models while handling both types of information.
- HookNet is a framework for high-resolution tissue segmentation applied to two datasets containing high-resolution and context-dependent tissue.
- HookNet increases overall performance compared with single-resolution models while simultaneously handling high-resolution subtle differences and contextual information.