Source-linked AI summary
Hierarchical Graph Representations in Digital Pathology
Pushpak Pati, Guillaume Jaume, Antonio Foncubierta, Florinda Feroce, Anna Maria Anniciello, Giosuè Scognamiglio, Nadia Brancati, Maryse Fiche, Estelle Dubruc, Daniel Riccio, Maurizio Di Bonito, Giuseppe De Pietro, Gerardo Botti, Jean-Philippe Thiran, Maria Frucci, Orcun Goksel, Maria Gabrani
TL;DR
The paper addresses the limited ability of cell-only or pixel-based representations to capture hierarchical tissue structure. It builds HACT graphs from cells and tissue regions, applies HACT-Net, and introduces BRACS for breast-cancer subtyping evaluation. Across the reported comparisons, HACT-based methods achieve superior classification performance to the evaluated baselines and are comparable to pathologists in supported tasks.
Problem
Cellular information alone incompletely characterizes complex tissue structure, while pixel-based processing misses biological entities and tissue composition relevant to pathology.
Method
The paper constructs hierarchical HACT entity-graphs from cells and tissue regions and uses HACT-Net to map their morphological and topological information to tissue classes, while introducing the BRACS dataset.
Results
HACT-Net achieves the best overall performance in the reported three-category evaluation, including approximately 66% F1-score for diagnostically challenging categories, and outperforms or compares favorably with evaluated baselines and pathologists.
Takeaways & Limitations
Hierarchical entity-graph representations provide a tissue-analysis framework that incorporates multiple biological entity levels for breast-tumor subtype classification.
Abstract
from arXiv · showhide
Cancer diagnosis, prognosis, and therapy response predictions from tissue specimens highly depend on the phenotype and topological distribution of constituting histological entities. Thus, adequate tissue representations for encoding histological entities is imperative for computer aided cancer patient care. To this end, several approaches have leveraged cell-graphs that encode cell morphology and organization to denote the tissue information. These allow for utilizing machine learning to map tissue representations to tissue functionality to help quantify their relationship. Though cellular information is crucial, it is incomplete alone to comprehensively characterize complex tissue structure. We herein treat the tissue as a hierarchical composition of multiple types of histological entities from fine to coarse level, capturing multivariate tissue information at multiple levels. We propose a novel multi-level hierarchical entity-graph representation of tissue specimens to model hierarchical compositions that encode histological entities as well as their intra- and inter-entity level interactions. Subsequently, a graph neural network is proposed to operate on the hierarchical entity-graph representation to map the tissue structure to tissue functionality. Specifically, for input histology images we utilize well-defined cells and tissue regions to build HierArchical Cell-to-Tissue (HACT) graph representations, and devise HACT-Net, a graph neural network, to classify such HACT representations. As part of this work, we introduce the BReAst Carcinoma Subtyping (BRACS) dataset, a large cohort of H&E stained breast tumor images, to evaluate our proposed methodology against pathologists and state-of-the-art approaches. Through comparative assessment and ablation studies, our method is demonstrated to yield superior classification results compared to alternative methods as well as pathologists.
I. INTRODUCTION
The paper motivates entity-based, hierarchical tissue representations because pixel-based and single-entity approaches incompletely capture pathological entities and tissue composition. It proposes HACT and HACT-Net, introduces BRACS, and evaluates performance against computational baselines and pathologists.
- Motivation: Manual breast-cancer diagnosis is tedious and time-consuming, with substantial intra- and inter-observer variability.
- Motivation: CNN-based pathology methods lack explicit pathological entities and disregard hierarchical tissue composition, limiting interpretability and use of entity-level knowledge.The paper gives epithelial cells, epithelium, and glands as an example of hierarchical composition.
- Motivation: Cell-graphs capture cell microenvironments but not tissue-region distributions, whereas tissue-graphs cannot depict cell microenvironments.
- Contributions: The proposed methodology combines HACT graph construction with HACT-Net-based histology-image analysis.
- Contributions: HACT is a hierarchical entity-graph representation, and BRACS is a public dataset of breast tumor regions-of-interest annotated with seven subtypes and challenging atypical cases.BRACS is intended to represent a more realistic breast-cancer analysis setting.
- Evaluation: Evaluation on BRACS compares the methodology with three independent pathologists and recent CNN and GNN approaches.The reported assessment finds performance comparable to pathologists on per-class and aggregated classification tasks while outperforming several recent CNN and GNN approaches.
III. PRELIMINARIES
The preliminaries define attributed entity-graphs and message-passing GNN operations, then describe GIN and PNA as alternative neighborhood-aggregation designs used in the hierarchical tissue-analysis pipeline.
- A. Notation: An attributed, undirected entity-graph G := (V, E, H) contains nodes, edges, and node features.The graph topology is represented by a symmetric adjacency matrix, and node neighborhoods contain adjacent nodes.
- B. Graph Neural Networks: Message-passing GNNs iteratively aggregate neighboring node features and update each node using its current features and the aggregated representation.
- B. Graph Neural Networks: The readout operation combines final node representations into a graph-level embedding.
- B. Graph Neural Networks: GIN uses sum aggregation followed by sum and an MLP, while PNA combines mean, standard deviation, maximum, and minimum aggregators with degree scalers and an MLP update.
- B. Graph Neural Networks: The proposed pipeline preprocesses an H&E-stained tumor region, constructs a HACT graph from morphological and topological entity information, and applies HACT-Net to map the graph toward tissue classification.
A. Pre-processing
Pre-processing reduces stain variability, identifies nuclei and tissue regions, and encodes their morphology and spatial organization into graph components for hierarchical tissue analysis.
- A. Pre-processing: Stain normalization addresses appearance variability caused by specimen preparation, staining, fixation, and imaging conditions.The method uses an unsupervised, reference-free algorithm estimating hematoxylin and eosin stain vectors with singular value decomposition.
- A. Pre-processing: Cell-graphs encode nuclei as nodes with morphological and normalized spatial features, while edges model nearby cellular interactions.The topology is initialized with k-nearest neighbors and pruned using a minimum-distance threshold.
- A. Pre-processing: Nuclei are detected with HoVer-Net, whose instance-aware distance representations help separate clustered and overlapping nuclei.
- A. Pre-processing: The overall pipeline constructs a hierarchical entity graph after preprocessing and applies a hierarchical graph neural network to classify tissue composition.
- A. Pre-processing: Tissue graphs represent higher-level tissue regions and their interactions as complementary entities for hierarchical tissue analysis.
2) Tissue-graph representation:
The tissue graph represents regions and their interactions, then links cell-level nuclei to tissue-level regions through a spatial assignment hierarchy.
- 2) Tissue-graph representation:: Tissue regions are first oversegmented into homogeneous superpixels and merged with neighboring regions sharing similar color statistics.SLIC performs the initial low-magnification oversegmentation, improving efficiency and reducing noisy-pixel effects.
- 2) Tissue-graph representation:: Tissue-region features combine CNN-derived superpixel features, averaged morphological representations, and normalized region centroids.
- 2) Tissue-graph representation:: A region-adjacency graph connects adjacent tissue regions, defining tissue topology through binary adjacency relationships.
- 3) Hierarchical Cell-to-Tissue graph representation:: The resulting HACT representation captures phenotypical and topological distributions across cell and tissue microenvironments.
- 3) Hierarchical Cell-to-Tissue graph representation:: HACT combines cell and tissue graphs with a binary cell-to-tissue assignment matrix encoding each nucleus’s spatial membership.Each nucleus is assigned to one tissue region; border cases use the region with maximum overlap.
C. Graph learning
HACT-Net processes cell-level and tissue-level graph information hierarchically, transfers cell embeddings into tissue nodes, and classifies the resulting graph representation.
- C. Graph learning: HACT-Net takes a HACT graph as input and uses separate cell- and tissue-level GNNs to process it from fine to coarse scale.
- C. Graph learning: The cell GNN applies PNA layers to iteratively build contextualized cell-node embeddings from neighboring cell features.The update uses degree-scalers and multiple neighborhood aggregators, including mean, standard deviation, maximum, and minimum.
- C. Graph learning: Cell embeddings and the cell-to-tissue assignment matrix initialize tissue-node features for subsequent tissue-level processing.Each tissue node receives information from the cell nodes mapped to it.
- C. Graph learning: The tissue GNN produces tissue-node embeddings whose receptive fields encode cell and tissue information across multiple hops.
- C. Graph learning: HACT-Net sums tissue-node representations into a graph embedding, which an MLP and softmax map to the tissue label during end-to-end training.Graph normalization and batch normalization help handle graph-size variation across samples.
V. DATASETS
BRACS is a large, diverse breast-tumor dataset of annotated regions of interest spanning seven diagnostic categories. Its natural variability in appearance, dimensions, staining, patterns, and artifacts is intended to support realistic evaluation.
- BRACS dataset: BRACS contains 4,391 tumor regions of interest from 325 H&E-stained breast carcinoma whole-slide images.The regions were annotated as Normal, Benign, UDH, ADH, FEA, DCIS, or Invasive.
- Dataset diversity: BRACS deliberately retains realistic variability in tumor-region appearance and dimensions rather than filtering atypical cases or artifacts.Reported variability includes staining, architectural patterns, glandular regions, tissue folds, tears, ink stains, and blur.
- Dataset organization: The BRACS train, validation, and test regions are partitioned at the whole-slide-image level to prevent regions from the same slide crossing splits.The dataset also exhibits substantial variation in tumor-region and graph-representation sizes.
- Dataset comparison: Compared with BACH, BRACS provides substantially larger train and test sets, supporting more robust evaluation.The BRACS train and test sets are nearly 10 and 6 times larger, respectively, than those of BACH.
- Dataset diversity: The dataset covers seven categories, including diagnostically challenging atypical precancerous classes ADH and FEA.These categories represent a broad histopathological spectrum and a practical diagnostic dilemma because of their risk of progression to cancer.
VI. RESULTS
The results section compares HACT-Net with CNN, cell-graph, tissue-graph, patch-graph, and combined graph baselines across BRACS and BACH classification settings. These baselines isolate the effects of scale, graph type, and hierarchical representation.
- Evaluation design: The evaluation compares proposed methods with CNN and GNN baselines on BRACS and BACH under multiple classification settings.The study includes ablations and a comparison with three independent pathologists.
- CNN baselines: Single-scale CNN baselines classify tumor regions at 10×, 20×, or 40× magnification before aggregating patch predictions.They use 128×128-pixel patches with a 64-pixel stride and a ResNet-50 backbone.
- CNN baselines: Multi-scale CNN baselines combine concentric patches from either 10×+20× or 10×+20×+40× magnifications.Their late-fusion strategy is intended to incorporate context across magnifications.
- GNN baselines: Graph baselines include CGC-Net for cell graphs and Patch-GNN for patch graphs, with Patch-GNN evaluated at 10×, 20×, and 40×.Additional baselines separately model cell graphs, tissue graphs, or concatenated cell- and tissue-graph embeddings.
B. Implementation
The implementation uses CNN-initialized cell and tissue graph representations, shared graph-network settings across methods, and weighted F1-score for evaluation. Experiments run in PyTorch and DGL on NVIDIA Tesla P100 GPUs and POWER8 processors.
- Graph representations: Cell graphs use 72×72 patches, while tissue graphs use 144×144 patches, with ResNet-34 or ResNet-50 initializing node features.These graph representations provide the inputs for the graph-learning comparisons.
- Graph architecture: CG-GNN, TG-GNN, CONCAT-GNN, and HACT-Net share the same graph-architecture options and hyperparameters.The compared models use the same implementation settings to support controlled evaluation.
- Graph architecture: The graph networks use 3, 4, or 5 PNA layers, two-layer PNA MLPs with 64 channels, and 128-dimensional graph embeddings.The output classifier is a two-layer MLP with 128 channels.
- Training and evaluation: Training uses Adam with a 10^-3 learning rate, batch size 16, and categorical cross-entropy.The final model is selected using the best validation weighted F1-score.
- Training and evaluation: Performance is evaluated with weighted F1-score to account for imbalanced numbers of regions across classes.The metric weights each class by its number of true instances.
- Computational resources: Experiments were conducted with PyTorch and Deep Graph Library on NVIDIA Tesla P100 GPUs and POWER8 processors.
C. Ablation studies
Ablation studies examine node-feature initialization, GNN layer type, and jumping knowledge on seven-class BRACS classification. The reported results favor hierarchical topology, CNN-based morphological features, and PNA layers.
- Ablation design: The ablations independently test node-feature initialization, GNN layer type, and jumping knowledge on seven-class BRACS classification.Each component is analyzed while the others are held fixed.
- Node features and topology: Cell-graph topology is more discriminative than tissue-graph topology, while combining both improves discriminative ability.The best performance is obtained with the HACT topology, supporting hierarchical representations in the ablation comparison.
- Node features and topology: Including morphological features significantly improves discriminative ability, with CNN-based features outperforming hand-crafted measures.The comparison indicates richer morphological information from CNN features than from hand-crafted features.
- GNN layer type: PNA layers outperform GIN layers for all four GNN constructions tested.These experiments use CNN-based node features and LSTM-based jumping knowledge.
3) Impact of jumping knowledge technique:
Across the evaluated classification settings, HACT-Net consistently provides the strongest overall performance, while ablations identify CNN initialization, PNA layers, and LSTM-based jumping knowledge as preferred components.
- Impact of jumping knowledge technique: LSTM-based jumping knowledge learns better dependencies between GNN layers than CONCAT, producing better graph embeddings.The ablation compares no jumping knowledge, CONCAT-based, and LSTM-based settings.
- Impact of jumping knowledge technique: The selected methodology uses CNN-based node initialization, PNA layers, and LSTM-based jumping knowledge.These choices are the conclusion of the component ablation experiments.
- 7-class classification: HACT-Net significantly outperforms CONCAT-GNN, indicating that hierarchical modeling better exploits complementary information from cell and tissue representations.The broader comparison also reports superior performance over CNN and GNN baselines.
- Classification settings: HACT-Net achieves the best overall performance in 4-class classification, including approximately 66% F1-score for the challenging Precancerous category.It also performs best for Normal, Precancerous, and Cancerous categories.
- Classification settings: HACT-Net consistently outperforms all baselines across six binary decision-tree tasks and achieves the best aggregated score.These tasks replicate the sequential diagnostic decisions used by pathologists.
1) Domain expert comparison on BRACS dataset:
On BRACS, HACT-Net is benchmarked against independent pathologists and shows stronger performance on diagnostically difficult atypia and hyperplasia classes, with comparable results for normal and cancerous categories.
- Domain expert comparison: HACT-Net outperforms domain experts on diagnostically challenging atypia and hyperplasia classes while remaining comparable for normal and cancerous categories.The comparison uses per-class F1-scores, weighted F1-score, and weighted accuracy.
- Dataset benchmarking: BRACS contains 626 TRoI images, compared with 240 breast biopsy slides in the reference dataset used for benchmarking.The comparison aggregates pathologist statistics over corresponding diagnostic categories.
- Domain expert comparison: Pathologist statistics show substantial inter-observer variability across the BRACS test set.The paper attributes differences in concordance to differing histopathology practices across regions.
2) Classification results on BACH dataset:
On BACH, HACT-Net attains accuracy comparable to state-of-the-art methods while using a single unified network, and its entity-based representation supports more interpretable attribution of diagnostic focus.
- Classification results on BACH dataset: The method applies rotation, mirroring, and color augmentation before extracting HACT graph representations for BACH training.Other graph augmentations were not used because they could disturb meaningful biological topology.
- Classification results on BACH dataset: HACT-Net achieves classification accuracy comparable to state-of-the-art methods on the BACH dataset.The reported accuracy differences are not significant given the 100-TRoI test set.
- Classification results on BACH dataset: HACT-Net uses a single unified network, whereas the other listed competitors use ensembles of multiple networks at inference.The evaluation was independently conducted by the BACH challenge organizers.
- Interpretability: Feature attribution highlights diagnostically relevant nuclei and tissue-region nodes, making entity-paradigm explanations more interpretable than pixel-based attribution.The analysis uses GRAPH-GRADCAM on CG and TG representations for sample TRoIs.
- Method rationale: The proposed representation addresses pixel-based processing’s context-resolution trade-off by modeling hierarchical biological entities and tissue composition.The method combines a hierarchical entity graph with a hierarchical graph neural network.