PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Graph-Based Deep Learning for Computational Histopathology

David Ahmedt-Aristizabal, Mohammad Ali Armin, Simon Denman, Clinton Fookes, Lars Petersson

arXiv:2107.00272v2cs.LGcs.CVq-bio.TO

TL;DR

Patch-wise CNN analysis in histopathology has limited access to global tissue context and inter-entity relationships. This survey reviews entity-graph construction, GNN architectures, applications, and explainability, finding broad success across cancer-analysis tasks while identifying unresolved challenges in construction, training efficiency, and interpretability.

  • Problem

    Patch-wise convolutional representations do not comprehensively capture global contextual information or the phenotypical and topological distribution of histological entities.

  • Method

    The survey systematically reviews entity-graph representations, graph deep-learning models, explainability tools, and applications organized by graph type, scale, task, and organ.

  • Results

    Graph-based deep learning has achieved superior performance across diverse cancer-analysis tasks and supports representations ranging from cell graphs to hierarchical cell-tissue graphs.

  • Takeaways & Limitations

    Graph representations enable pathology models to encode tissue structure and relationships while offering prediction outputs that can be examined through graph interpretability methods.

Abstract

from arXiv · show

With the remarkable success of representation learning for prediction problems, we have witnessed a rapid expansion of the use of machine learning and deep learning for the analysis of digital pathology and biopsy image patches. However, learning over patch-wise features using convolutional neural networks limits the ability of the model to capture global contextual information and comprehensively model tissue composition. The phenotypical and topological distribution of constituent histological entities play a critical role in tissue diagnosis. As such, graph data representations and deep learning have attracted significant attention for encoding tissue representations, and capturing intra- and inter- entity level interactions. In this review, we provide a conceptual grounding for graph analytics in digital pathology, including entity-graph construction and graph architectures, and present their current success for tumor localization and classification, tumor invasion and staging, image retrieval, and survival prediction. We provide an overview of these methods in a systematic manner organized by the graph representation of the input image, scale, and organ on which they operate. We also outline the limitations of existing techniques, and suggest potential future research directions in this domain.

I. INTRODUCTION

Computational pathology increasingly uses graph-based deep learning to represent relationships among cells, patches, and tissue regions that patch-wise CNNs do not explicitly capture. This survey organizes graph representations, architectures, applications, and open challenges for digital pathology.

  • I. INTRODUCTION: Graph neural networks are emphasized because graphs encode relationships between entities and can preserve irregular neighboring relations beyond fixed local pixel neighborhoods.The survey contrasts graph-based relational modeling with CNN-based image representations.
  • I. INTRODUCTION: Graph-based methods represent histological images as cell-, patch-, tissue-, or hierarchical cell-tissue graphs to model cellular morphology, topology, and tissue micro-architecture.Nodes can correspond to cells, patches, or tissue regions, while graph structure captures relationships among them.
  • I. INTRODUCTION: The survey reviews entity-graph construction, graph architectures, explainability tools, and applications spanning pathology analysis and disease characterization.Its application organization considers graph construction, feature-level fusion, task, and organ.
  • I. INTRODUCTION: The review addresses a gap by systematically covering graph-based representations and deep learning models for computational pathology, beyond earlier narrower reviews.Earlier surveys focused on handcrafted-feature cell graphs or briefly discussed GCNs in biology and medicine.
  • I. INTRODUCTION: Patch-wise CNNs capture local image relations but do not explicitly model complex neighborhood information, spatial context, or inter-nuclear relationships.Graph-based representations address these limitations by encoding relationships among tissue entities and their collective contribution to diagnosis.

II. GRAPH REPRESENTATION LEARNING IN DIGITAL PATHOLOGY: BACKGROUND

Graph-based pathology workflows transform whole-slide images into entity graphs whose nodes, features, and edges encode tissue content and relationships. GNNs then process these graphs for node- or graph-level analysis, with explainability as an additional workflow stage.

  • II. GRAPH REPRESENTATION LEARNING IN DIGITAL PATHOLOGY: BACKGROUND: Graph representations use node features X and an adjacency matrix W within G = (V, E, W) to describe graph entities and their connections.The graph contains n nodes, while d denotes input feature dimensions.
  • II. GRAPH REPRESENTATION LEARNING IN DIGITAL PATHOLOGY: BACKGROUND: A standard workflow transforms a WSI into graphs, defines entities and features, encodes relationships, applies GNNs and pooling, and supports model interpretation.The workflow includes architectures such as ChebNet, GCN, GraphSAGE, GAT, and GIN, plus graph explainers.
  • II. GRAPH REPRESENTATION LEARNING IN DIGITAL PATHOLOGY: BACKGROUND: Entity graphs represent histological images with biological or patch-defined nodes and edges encoding inter-entity interactions.Entities may be nuclei, tissue regions, patches, or superpixels.
  • II. GRAPH REPRESENTATION LEARNING IN DIGITAL PATHOLOGY: BACKGROUND: Node embeddings combine handcrafted morphological and topological features with learned representations extracted from image or entity-specific models.Examples include shape, size, orientation, intensity, chromaticity, and deep features from patches around nuclei or tissue regions.
  • II. GRAPH REPRESENTATION LEARNING IN DIGITAL PATHOLOGY: BACKGROUND: Edge topology encodes likely cellular or tissue interactions using proximity thresholds, nearest-neighbor rules, probabilistic models, Waxman models, or region adjacency graphs.Spatial centroids of superpixels can be used to construct region adjacency graphs.

3) Training paradigms:

Graph learning in computational pathology spans supervised, weakly or partially supervised, semi-supervised, and self-supervised settings. GNNs use message passing to support node- and graph-level predictions from graph structure and node content.

  • 3) Training paradigms:: Supervised, weakly or partially supervised, semi-supervised, and self-supervised paradigms provide different amounts or sources of annotation for graph learning.Self-supervised methods learn representations through proxy tasks and can later use annotations for fine-tuning.
  • 3) Training paradigms:: GCNs are categorized as spectral-based or spatial-based according to whether convolution is defined through graph spectral representations or spatial node relationships.Both categories process graph-structured data through neighborhood information.
  • 3) Training paradigms:: GNNs update node representations by aggregating neighboring features and combining them with the center node representation through message passing.The basic architecture consists of AGGREGATE and COMBINE operations applied across layers.
  • 3) Training paradigms:: Node-level prediction produces labels or regression values for individual nodes, whereas graph-level prediction produces one value for an entire graph after pooling.Graph-level outputs commonly support whole-graph classification or graph similarity computation.

1) ChebNet:

ChebNet approximates spectral graph filters with truncated Chebyshev polynomials, avoiding explicit Laplacian eigenvector computation. The survey then describes GCN as a localized first-order approximation of this spectral operation.

  • 1) ChebNet:: The spectral convolution operates on a graph signal using the normalized Laplacian, its eigenvectors and eigenvalues, and a parameterized filter.The normalized Laplacian is decomposed as L = UΛU^T.
  • 1) ChebNet:: ChebNet uses truncated Chebyshev polynomials to approximate spectral filters and avoid calculating graph-Laplacian eigenvectors.This approximation reduces the computational cost of spectral convolution.
  • 1) ChebNet:: ChebNet rescales the Laplacian through ˜L = 2L/λmax−I_N before evaluating Chebyshev polynomials.λmax denotes the largest eigenvalue of L, and the polynomials use recursively defined T0 and T1 initializations.
  • 1) ChebNet:: GCN is presented as a localized first-order ChebNet approximation that limits the convolution filter to K = 1 and approximates λ ≈ 2.These assumptions reduce parameters and help limit overfitting.
  • 1) ChebNet:: The graph convolution generalizes from a scalar node signal to feature matrices with C input channels and F output filters.The filter bank parameters are represented by Θ ∈ R^C×F, producing Z ∈ R^N×F.

3) GraphSAGE:

GraphSAGE learns node representations by aggregating neighborhood information with trainable functions, including mean, LSTM, and max-pooling aggregation.

  • 3) GraphSAGE:: GraphSAGE updates node embeddings through trainable neighborhood aggregation rather than simple neighborhood convolutions.The method also supports batch training to reduce memory use, at the cost of time efficiency.
  • 3) GraphSAGE:: The architecture uses max-pooling aggregation to construct spatial graph convolutions.
  • 3) GraphSAGE:: Graph attention networks modify graph convolution with masked self-attention over neighboring nodes.Attention coefficients determine how neighboring features contribute to updated node representations.
  • 3) GraphSAGE:: Multi-head attention applies K independent attention heads in parallel and concatenates their features to stabilize learning.

5) GIN:

GIN aggregates neighboring node representations by summation and updates each node with an MLP, while related architectures extend aggregation, pooling, and multi-scale representation learning.

  • 5) GIN:: GIN sums neighboring node representations, then updates each node through an MLP with injective aggregation and readout functions.These design choices are intended to provide maximum discriminative power.
  • 5) GIN:: Other surveyed architectures incorporate edge features, heterogeneous-graph flexibility, adaptive aggregation, or neighborhood-range selection.
  • 5) GIN:: FENet adaptively selects representations across convolution layers and uses sum-pooling to capture full graph structure.
  • 5) GIN:: MS-GWNN performs efficient, interpretable multi-scale graph convolution using localized graph wavelets and parallel scaling parameters.
  • 5) GIN:: Graph pooling reduces graph size and computational complexity while enabling hierarchical graph-level features.
  • 5) GIN:: DiffPool learns node-to-cluster assignment matrices, whereas SAGPool uses topology- and feature-aware self-attention for local pooling.

D. Graph interpretations

Graph interpretation methods address the need to identify clinically meaningful entities and relationships in graph-based pathology models through intrinsic attention and post-hoc explanations.

  • D. Graph interpretations: Interpretability is important because graph models encode cellular interactions, while relevant visual features remain difficult to identify for diagnosis and disease understanding.
  • D. Graph interpretations: Model-based interpretation uses internal weights or structure, whereas post-hoc methods extract information from learned relationships after training.
  • D. Graph interpretations: Attention mechanisms focus computation on relevant graph components and can also highlight important nodes as explanations.
  • D. Graph interpretations: Attention weighting computes a weighted combination of hidden states after scoring their similarity to a learned context representation.
  • D. Graph interpretations: The survey identifies post-hoc explainers based on excitation, node pruning, gradients, relevance propagation, and graph mapping.
  • D. Graph interpretations: The reviewed applications organize graph methods by construction, training paradigm, datasets, and pathology task rather than presenting an exhaustive literature review.

A. Cell-graph representation

Cell-graph methods represent nuclei as nodes and their spatial relationships as edges, enabling graph models to combine cellular features with tissue architecture for pathology tasks.

  • A. Cell-graph representation: Cell-graph pipelines detect cells, use their locations as vertices, construct spatial edges, and classify the resulting graph with a GCN.
  • A. Cell-graph representation: Cell-graphs address the limitation that patch-wise CNNs do not explicitly capture inter-nuclear relationships or global tissue information.
  • 1) Breast cancer:: Anand et al. combine nuclear morphology as vertex features with gland formation encoded as edge attributes derived from nearby-nuclei distances.
  • 1) Breast cancer:: Graph explainers can occlude nuclei clusters or use attention to visualize relative cell contributions to classification.
  • 1) Breast cancer:: Graph-based explanations preserve tumor macro-environment information that pixel-level heatmaps may fail to encode, motivating quantitative pathology-aware evaluation.
  • 1) Breast cancer:: GraphGrad-CAM++ achieved the best overall agreement with pathologists on explanation quality in the BRACS dataset.
  • 2) Colorectal cancer:: For colorectal grading, Adaptive GraphSAGE fuses multi-level features, while graph clustering groups cells for hierarchical representation.
  • 2) Colorectal cancer:: Graph-based deep learning methods outperformed classical graph-based and CNN-based methods, although restricting each node to its two closest neighbors limits message passing.

3) Prostate cancer:

Prostate cancer studies use graph representations to model gland and tissue organization, enabling weakly supervised classification and broader whole-slide analysis beyond small-patch CNN methods.

  • Prostate cancer: Weakly supervised prostate grading constructs cell-graphs from segmented nuclei, using nuclei as nodes and distances between neighboring nuclei as edges.The approach targets grade classification and stratification of low- and high-risk cases from image-level annotations.
  • Prostate cancer: Graph-based ROI methods represent important patches and their relationships, whereas tissue-graphs encode biologically defined morphological regions.Patch-graphs may contain multiple biological entities within each patch, while tissue-graphs are organized around relevant tissue morphology.
  • Prostate cancer: Outperformed CNN-based majority-voting, learned-fusion, and base-penultimate models for weakly supervised multi-class classification of arbitrarily sized breast-cancer ROIs.The method uses fixed-sized patch features as graph nodes, spatial proximity as edges, and two GCNs to propagate context and classify the ROI without patch-level labels.
  • Prostate cancer: Graph construction addresses the limitation of prior methods that build graphs from small WSI patches by modeling the entire slide at nuclei level.The pipeline segments and classifies nuclei, clusters them, constructs the graph, and performs graph classification.
  • Prostate cancer: Whole-WSI graph construction uses nuclei-level geometry and cellular organization, clustering nuclei into tissue-level nodes before graph classification.The resulting graph supports HER2 and progesterone-receptor prediction using a GIN model.

2) Colorectal cancer:

Colorectal studies apply graph learning to tumor staging, lymph-node metastasis, mutation prediction, survival analysis, and retrieval by modeling spatial or feature relationships among tissue regions.

  • Colorectal cancer: Graph attention MIL models colorectal tumor and tissue partitions as instances within WSI-level bags to predict tumor-node-metastasis stage.This explicitly represents spatial relationships between tumor and other tissue partitions, which CNN patch-selection approaches do not capture.
  • Colorectal cancer: Outperformed CNN-based and attention-based MIL models for colorectal lymph-node metastasis prediction using instance feature extraction, selection, and bag-level graph classification.The framework uses a self-supervised VAE-GAN feature extractor, maximum mean discrepancy for feature selection, and ChebNet with SAGPool.
  • Colorectal cancer: The proposed FENet significantly improved mutation prediction across APC, KRAS, and TP53 tasks, outperforming ChebNet, GraphSAGE, and GAT baselines.Its ensemble combines predictions from multiple non-isomorphic subgraphs on the TCGA colon adenocarcinoma dataset.
  • Lung cancer: Survival prediction integrates local patch features with global patch-graph topology and uses survival-specific supervision with adaptive attention-based patch selection.Topological features fine-tuned with survival labels outperformed CNN-based models on TCGA-LUSC and NLST datasets.
  • Lung cancer: A fully connected graph over important lung-cancer patches outperformed attention-MIL CNN models when classifying lung adenocarcinoma versus lung squamous cell carcinoma.DenseNet extracts patch features, which are pooled and passed to fully connected layers for final classification.
  • Colorectal cancer: Self-supervised GNN retrieval learns ROI representations whose Euclidean distances measure similarity, with contrastive learning improving representations and outperforming supervised classification methods without class labels.A separate hashing framework builds patch-graphs from spatial adjacency and CNN feature similarity, then encodes them for large-scale WSI retrieval.

4) Skin cancer:

Graph-based histopathology methods represent cells, tissue regions, and their hierarchy to capture complementary morphological and spatial information. Across reviewed applications, hierarchical representations improved breast cancer classification while remaining dependent on accurate graph construction and cell detection.

  • Hierarchical graph representation: Cell-graphs and tissue-graphs provide complementary information, but cell-only representations omit tissue macro-architecture and tissue-only representations omit cellular microenvironments.Hierarchical modeling is therefore presented as a way to combine multilevel structural information.
  • Multi-scale graph representation: MS-GWNN uses parallel graph-wavelet scales to encode multilevel tissue structure for breast cancer classification on BACH and BreakHis.The approach performs node prediction at multiple scales and combines the resulting representations for graph-level classification.
  • Hierarchical graph representation: HACT combines a cell-graph, tissue-graph, and cell-to-tissue hierarchy to represent cellular morphology, tissue properties, and their relative distribution.The cell-graph captures low-level cellular structure, while the tissue-graph captures higher-level tissue sections and spatial distribution.
  • Hierarchical graph representation: The enriched HACT representation outperformed CNN-based, standalone cell-graph, and tissue-graph models for five-class BRACS breast cancer subtype classification.Evaluation used WSI-level splits to assess generalization to unseen slides.
  • Interpretability: Graph-based methods support interpretable pathology analysis by highlighting nuclei and tissue-region nodes used for tumor-region classification.GraphGrad-CAM is used to identify the regions receiving emphasis from HACT-Net.

2) Colorectal cancer:

The reviewed approaches combine graph representations with CNN or multimodal features to model relationships among pathology entities and integrate histology with other biomedical information. These methods span tumor invasion, cervical-cell classification, and survival prediction, while the field remains relatively nascent and open to further development.

  • Tumor invasion and staging: GCN-based frameworks combine local and global tissue patterns to estimate tumor invasion and produce interpretable staging scores.The framework uses graph representations to explain how tissue regions contribute to invasion assessment.
  • Cervical-cell classification: A cervical-cell classifier builds a graph of CNN-feature cluster centroids, encodes intrinsic similarities with a GCN, and incorporates the result into CNN features.K-means clustering provides graph nodes before relation-aware representations are fused for classification.
  • Multimodal fusion: A multimodal renal-cancer framework fuses histology patch features, cell-graph features, and genomic features into a tensor modeling cross-modal interactions for survival prediction.The reviewed work reports that this multimodal tensor outperformed deep-learning-based feature fusion.
  • Scope: Graph-based deep learning has been used to capture phenotypical and topological distributions in biomedical data, including histological cancer tissue.The survey positions these applications within broader biomedical graph representation learning.
  • Future directions: Graph applications in digital pathology remain nascent, with unresolved challenges and unexplored approaches that could support more comprehensive biological analysis.The survey identifies entity construction, expert knowledge, model complexity, training, and explainability as future directions.

A. Entity-graph construction

Entity-graph construction determines which biological structures and relationships a pathology model can represent. The survey describes complementary cell, patch, tissue, and hierarchical graphs, while highlighting preprocessing dependence, fixed topology choices, and limited automation as major constraints.

  • Entity definition: Cell-graphs encode cellular morphology and interactions but require reliable cell detection, scale poorly with many cells, and omit tissue macro-architecture.Representative-node sampling and clustering can reduce redundancy, but most methods still assume cell-cell interactions are the dominant information source.
  • Entity definition: Patch-graphs provide contextual ROI information, but patch resolution and size trade off against context and depend on the data.Patch entities are not necessarily biologically defined, limiting how directly the graph corresponds to tissue structures.
  • Entity definition: Tissue-graphs represent neighboring regions and address some patch limitations, yet they cannot capture local cellular information on their own.Combining cell- and patch-level features still fails to exploit the hierarchy between representation levels.
  • Hierarchical representation: Hierarchical graphs better represent histological structure by combining cellular and tissue interactions, but remain constrained by cell-detection accuracy and graph complexity.The number of nodes drives computational constraints, while cell-centered construction remains vulnerable to detection errors.
  • Automated graph generation: Most surveyed pipelines use sequential preprocessing for tissue masks, nuclei, superpixels, features, and graph construction, making performance dependent on each stage.The survey identifies automated graph generation as underused and notes that its effectiveness in histopathology remains insufficiently investigated.
  • Automated graph generation: Fixed thresholds, patch settings, sampling ratios, KNN distances, superpixel choices, and feature selections limit graph generalization across tissues, organs, and tasks.Learning graph structure from data is presented as a potential alternative to predefined topology parameters.

C. Complexity of graph models

The survey identifies computational complexity, memory demands, and annotation requirements as key challenges for graph models in histopathology. It also highlights simpler architectures, alternative graph-processing strategies, and weakly or self-supervised training as responses.

  • C. Complexity of graph models: Graph models can represent large tissue regions and topological structure, but their complexity makes architecture selection an important hyper-parameter.The survey contrasts this representational power with the computational complexity of classical graph-based models.
  • C. Complexity of graph models: GNN training requires substantial memory and inference time because whole graphs and intermediate node states are typically stored.Graph sampling is proposed to reduce training costs, but efficient training remains uncommon in surveyed applications.
  • C. Complexity of graph models: Simpler graph architectures such as SGC reduce GCN complexity by removing intermediate nonlinearities and collapsing weight matrices.Other approaches bypass GCN processing entirely through cell-graph community detection, which outperformed several deep and graph-based baselines in tissue classification.
  • C. Complexity of graph models: Self-supervised and weakly supervised methods address the scarcity of manually labeled histopathology data for node embeddings and graph classification.Weakly supervised methods learn from few labeled samples, while self-supervised methods learn embeddings directly from histopathology images.
  • C. Complexity of graph models: Graph-based methods can scale to arbitrary tissue dimensions and incorporate structural information among instances, supporting weakly supervised classification.SegGini is reported to outperform CLAM and CACNN for weakly supervised prostate cancer classification.

E. Explainability of graph models

The survey presents explainability as essential for clinician trust but insufficiently developed in computational pathology. Existing methods mainly provide instance-level explanations, while global validation and clinician-oriented evaluation remain limited.

  • E. Explainability of graph models: Existing explainers mainly support cell-graph analysis, although hierarchical explanations could expose nuclei, tissue parts, and their interactions.Such multi-level explanations may better align with pathologists’ assessments and support trust in AI frameworks.
  • E. Explainability of graph models: Only one surveyed study evaluated explanation quality for clinicians; at the concept level, GraphGrad-CAM++ showed the highest agreement with pathologists.GraphGrad-CAM and GNNExplainer followed in the reported agreement ranking.
  • E. Explainability of graph models: Gradient- and perturbation-based explainers usually operate on single instances, making population-level understanding time-consuming and underexplored.The survey distinguishes instance-level explanations from the need to understand predictions collectively across input graphs.
  • E. Explainability of graph models: Model-level explainers such as XGNN and PGExplainer aim to characterize graph patterns associated with model behavior, but their suitability for histopathology node classification remains unknown.XGNN generates graph patterns through reinforcement learning, while PGExplainer provides instance explanations with a global model view.
  • E. Explainability of graph models: Interpretability is critical for clinician trust and informed cancer diagnosis, yet GNN explainability in digital pathology remains insufficiently studied.The survey frames explainability and quantitative evaluation around clinician usability as open priorities.
Loading 2107.00272v2…