Source-linked AI summary
A graph-transformer for whole slide image classification
Yi Zheng, Rushin H. Gindra, Emily J. Green, Eric J. Burks, Margrit Betke, Jennifer E. Beane, Vijaya B. Kolachalama
TL;DR
Patch-based WSI classification can impose slide-level labels on individual patches while neglecting spatial and overall WSI-level information. The paper addresses this with GTP, which combines contrastive patch representations, graph-based WSI modeling, and a vision transformer. GTP achieved high classification performance and produced class-associated regions that closely matched pathologist assessments, while its feature-extraction stage was computationally intensive and its demonstrated application focused on lung tumors.
Problem
Patch-based methods can introduce label noise by assigning each patch the WSI label and may neglect spatial connectivity and overall WSI-level information important for disease grading.
Method
GTP uses contrastive learning to generate patch embeddings as graph-node features, then fuses graph-based representation with a vision transformer for WSI-level classification.
Results
GTP achieved high WSI-level classification performance, exceeded other deep-learning architectures, and generated GraphCAM regions that closely matched pathologist-derived assessments.
Takeaways & Limitations
GTP provides an interpretable framework for class-specific WSI classification by combining regional and WSI-level information.
Takeaways & Limitations
Contrastive patch-feature extraction was computationally intensive, the graph-node construction could be redesigned, and applicability beyond lung tumors remains to be demonstrated.
Abstract
from arXiv · showhide
Deep learning is a powerful tool for whole slide image (WSI) analysis. Typically, when performing supervised deep learning, a WSI is divided into small patches, trained and the outcomes are aggregated to estimate disease grade. However, patch-based methods introduce label noise during training by assuming that each patch is independent with the same label as the WSI and neglect overall WSI-level information that is significant in disease grading. Here we present a Graph-Transformer (GT) that fuses a graph-based representation of an WSI and a vision transformer for processing pathology images, called GTP, to predict disease grade. We selected $4,818$ WSIs from the Clinical Proteomic Tumor Analysis Consortium (CPTAC), the National Lung Screening Trial (NLST), and The Cancer Genome Atlas (TCGA), and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from adjacent non-cancerous tissue (normal). First, using NLST data, we developed a contrastive learning framework to generate a feature extractor. This allowed us to compute feature vectors of individual WSI patches, which were used to represent the nodes of the graph followed by construction of the GTP framework. Our model trained on the CPTAC data achieved consistently high performance on three-label classification (normal versus LUAD versus LSCC: mean accuracy$= 91.2$ $\pm$ $2.5\%$) based on five-fold cross-validation, and mean accuracy $= 82.3$ $\pm$ $1.0\%$ on external test data (TCGA). We also introduced a graph-based saliency mapping technique, called GraphCAM, that can identify regions that are highly associated with the class label. Our findings demonstrate GTP as an interpretable and effective deep learning framework for WSI-level classification.
1 Introduction
Whole slide image analysis must handle large, spatially structured pathology data rather than treating every patch as an independent copy of the slide label. The paper introduces GTP, which combines graph representations, contrastive patch features, and vision transformers for WSI-level classification and interpretation.
- Motivation: WSIs can exceed a gigabyte, making efficient analysis of complete high-resolution slides difficult for traditional image-analysis routines.Deep learning has enabled applications including disease classification, tissue segmentation, mutation prediction, and immune-infiltrate profiling.
- Motivation: Pathologists examine slides at multiple scales, combining zoomed regional evaluation with zoomed-out assessment of the entire WSI.The paper uses this workflow as motivation for integrating regional and slide-level information.
- Approach: GTP embeds image patches as graph nodes using contrastive learning and applies a vision transformer to predict a WSI-level label.The framework is described as a graph-based vision transformer for digital pathology.
- Study design: The study used 4,818 WSIs from three national cohorts to distinguish normal, LUAD, and LSCC slides.The cohorts were CPTAC, NLST, and TCGA.
- Interpretability and evaluation: GraphCAM generates WSI-level saliency maps that identify regions associated with the output class label and can be compared with pathologist annotations.The paper also reports ablation and sensitivity analyses comparing GTP with current WSI-classification methods.
2 Materials and methods
The study uses WSIs and clinical data from three cohorts to build GTP, which represents patches as graph nodes, applies graph convolution and transformer processing, and generates GraphCAM saliency maps.
- Study population: WSIs and corresponding clinical data came from CPTAC, NLST, and TCGA, covering LUAD, LSCC, and normal tissue.These cohorts supplied the study population for WSI-level classification.
- Graph-Transformer: GTP represents each selected image patch as a graph node and connects spatially adjacent patches within the WSI.The adjacency matrix allows each patch to connect with up to eight neighboring patches.
- Patch representation: Contrastive learning extracts D-dimensional patch features, replacing raw patch pixels as graph node inputs.The learned feature extractor is trained without manual labels and later supplies node features for GTP.
- Graph-Transformer: Graph convolution propagates and aggregates information across connected patches before transformer processing.The graph convolution uses self-loops and a symmetric normalized adjacency matrix.
- Graph-Transformer: The transformer treats graph feature nodes as tokens and uses adjacency information to model interactions before mapping transformer states to labels.Its architecture includes multihead self-attention, multilayer perceptrons, layer normalization, and a learnable class embedding.
- Class activation mapping: GraphCAM propagates class-specific gradients and relevance through attention maps, then maps transformer relevance back to graph nodes through learned pooling assignments.The resulting graph activation map reconstructs WSI regions associated with the target class.
3 Experiments
The experiments trained GTP with patch features learned from NLST, evaluated it on CPTAC with five-fold cross-validation, and tested it independently on TCGA. They compared graph and transformer components, alternative methods, and GraphCAM visualizations against expert annotations and failure cases.
- Experimental settings: 2,071 CPTAC WSIs were used for five-fold cross-validation, while 2,082 TCGA WSIs formed the independent test set.Approximately 1.8 million NLST patches were used exclusively for contrastive learning and patch-specific feature generation.
- Experimental settings: 512×512 non-overlapping patches at 20× magnification were extracted from each WSI, excluding patches with more than 50% non-tissue area.
- Comparative and ablation studies: GTP was compared with AttPool and TransMIL using the same contrastive-learning feature extractor, and the transformer component was removed to assess the graph contribution.Hyperparameters for AttPool and TransMIL were fine-tuned on the CPTAC and TCGA cohorts.
- Interpretability analysis: GraphCAMs were compared with pathologist annotations using binarized maps, while additional figures displayed cross-validation saliency maps and LUAD-to-LSCC or LSCC-to-LUAD failure cases.The figure set covers original WSIs, GraphCAMs, expert annotations, prediction probabilities, and failure-case predicted labels.
- Comparative and ablation studies: Contrastive learning was evaluated against supervised ResNet variants, a convolutional autoencoder, and contrastive learning trained on STL-10 rather than NLST.The supervised ResNet comparison assigned each patch the label of its WSI.
- Evaluation: ROC and PR curves, AUC, precision, recall, specificity, and accuracy were computed for the three-label task, with mean AUCs and variance aggregated across five folds.DeLong’s statistical test assessed whether AUCs differed significantly between models.
4 Results
GTP achieved strong WSI-level classification performance, with external-test degradation concentrated in tumor classes, and outperformed comparator methods. Its graph-transformer design, contrastive-learning features, and GraphCAM interpretations were supported by ablations and region-overlap analyses.
- Mean area under the ROC and PR curves exceeded 0.9 for all classification tasks on CPTAC test data.
- External TCGA performance dropped slightly, especially for LUAD and LSCC, while most non-tumor WSIs were correctly classified.
- GTP achieved the best performance compared with TransMIL and AttPool.
- GraphCAMs overlapped pathologist annotations with mean maximum IoU = 0.817 ± 0.165 across 20 TCGA cases.
- Replacing contrastive learning with pretrained ResNet18 or a convolutional autoencoder reduced three-label classification performance on CPTAC and TCGA.
- Reducing hidden-state dimension or GCN layers degraded performance, whereas three transformer blocks sufficiently integrated pooled graph information.
- Increasing patch size, using 4-node connectivity, or adding 10% patch overlap reduced accuracy, while smaller patches increased graph computational cost.
5 Discussion
GTP integrates graph-based WSI representations with vision transformers to capture both regional and slide-level information for classifying normal, LUAD, and LSCC WSIs. GraphCAM further provides class-specific saliency maps whose identified regions closely match pathologist assessments, although feature extraction and graph construction remain computational and scope-related limitations.
- Model significance: GTP classification performance exceeded other deep learning architectures incorporating state-of-the-art configurations across reported model performance metrics.
- Interpretability: GraphCAM regions closely matched pathologist-derived assessments, supporting its use for identifying disease-related regions of interest on WSIs.
- Model significance: GTP integrates graphs with vision transformers to capture spatial connectivity and local region-specific information for WSI classification.The framework differentiates normal WSIs from those containing LUAD or LSCC.
- Interpretability: GraphCAM generates class-specific heatmaps that identify WSI regions associated with a selected output class.Class-specific maps are useful when pathology images contain features related to multiple classes, such as mixed LUAD and LSCC histology.
- Limitations: Contrastive learning for patch-level feature vectors was computationally intensive, and alternative feature-extraction methods may improve model performance.
- Limitations: The graph representation depends on patch-derived nodes, while broader applicability beyond lung tumors requires extension to other cancers and computational pathology tasks.