Source-linked AI summary
Attention-Based Deep Neural Networks for Detection of Cancerous and Precancerous Esophagus Tissue on Histopathological Slides
Naofumi Tomita, Behnaz Abdollahi, Jason Wei, Bing Ren, Arief Suriawinata, Saeed Hassanpour
TL;DR
Existing microscopy-image analysis methods are limited by bounding-box annotation requirements. This paper presents an attention-based model that analyzes high-resolution esophagus tissue images, with performance reported as higher than or marginally better than the sliding-window method.
Problem
Previous methodology for analyzing microscopy images is limited by bounding box annotation.
Method
The paper presents an attention-based model that extracts grid-based CNN features and applies attention modules to estimate each tile’s diagnostic importance.
Results
The model’s classification performance on the dataset is higher than that of the baseline and marginally outperforms the current sliding window method on esophagus tissue.
Takeaways & Limitations
The attention-based model supports detection of Barrett esophagus and esophageal adenocarcinoma on histopathology slides.
Takeaways & Limitations
The model’s performance has room for improvement, and its outperformance of the baseline was not consistent across every class.
Abstract
from arXiv · showhide
Deep learning-based methods, such as the sliding window approach for cropped-image classification and heuristic aggregation for whole-slide inference, for analyzing histological patterns in high-resolution microscopy images have shown promising results. These approaches, however, require a laborious annotation process and are fragmented. This diagnostic study collected deidentified high-resolution histological images (N = 379) for training a new model composed of a convolutional neural network and a grid-based attention network, trainable without region-of-interest annotations. Histological images of patients who underwent endoscopic esophagus and gastroesophageal junction mucosal biopsy between January 1, 2016, and December 31, 2018, at Dartmouth-Hitchcock Medical Center (Lebanon, New Hampshire) were collected. The method achieved a mean accuracy of 0.83 in classifying 123 test images. These results were comparable with or better than the performance from the current state-of-the-art sliding window approach, which was trained with regions of interest. Results of this study suggest that the proposed attention-based deep neural network framework for Barrett esophagus and esophageal adenocarcinoma detection is important because it is based solely on tissue-level annotations, unlike existing methods that are based on regions of interest. This new model is expected to open avenues for applying deep learning to digital pathology.
BACKGROUND
Barrett esophagus increases esophageal adenocarcinoma risk, while histologic diagnosis and existing whole-slide analysis remain difficult and annotation-intensive. The proposed attention-based approach addresses these limitations by using contextual tile weighting and tissue-level supervision.
- Clinical motivation: Barrett esophagus predisposes patients to esophageal and gastroesophageal-junction adenocarcinoma, with a 30 to 125 times higher cancer risk than the general population.
- Clinical motivation: Esophageal adenocarcinoma has an average 5-year survival rate below 15% in the United States.
- Diagnostic challenge: Histologic diagnosis of Barrett esophagus requires identifying metaplastic columnar epithelium with goblet cells, and pathologists show moderate interobserver variability.
- Existing methods: Sliding-window analysis requires pathologist-drawn ROI bounding boxes and heuristic aggregation, creating laborious, task-dependent, and poorly scalable workflows.
- Existing methods: Independent window classification ignores correlations between neighboring windows, although adjacent tissue can contain informative diagnostic characteristics.
- Proposed direction: The proposed soft-attention framework dynamically mines ROIs from high-resolution slides without explicit supervision and is trainable with tissue-level labels.
Dataset
The study assembled digitized H&E whole-slide biopsy images from a tertiary academic medical center and converted them into labeled single-tissue images. Expert lesion annotations established reference labels and supported baseline training, but were not required for the proposed model.
- Data source: Images came from endoscopic esophagus and gastroesophageal-junction mucosal biopsies collected at Dartmouth-Hitchcock Medical Center.
- Dataset split: 180 whole-slide images were digitized from H&E-stained tissue at 20× magnification, with 116 images assigned to development and 64 to testing.
- Preprocessing: Whole-slide images containing multiple tissue pieces were separated into 256 high-resolution images, each covering a single tissue piece.
- Reference standard: Two expert pathologists annotated lesion bounding boxes for reference standards and sliding-window baseline training, with disagreements resolved through discussion and senior consultation.
- Annotation strategy: The proposed attention-based model did not need the lesion bounding boxes used to train the baseline.
- Labels: The processed dataset comprised four classes: normal, BE-no-dysplasia, BE-with-dysplasia, and adenocarcinoma.
Methodology
The model combines grid-based CNN feature extraction with an attention module that weights tiles according to diagnostic relevance. Both components are optimized end-to-end using only tissue-level classification labels.
- Grid-based feature extraction: The CNN divides each high-resolution image into nonoverlapping tiles and produces a structured k×r×c feature map that preserves local geometric relationships.
- Slide classification: The attention module operates on extracted grid features to classify the whole slide.
- Training: Feature extraction and attention are jointly optimized end-to-end through classification loss backpropagation.
- Training: The model requires neither ROI bounding-box annotations nor manually developed heuristic aggregation rules.
EXPERIMENTS
The evaluation applied the proposed model to tissue images from patients at risk of esophageal cancer and compared it with a state-of-the-art sliding-window method. Images were tissue-filtered, labeled into four classes, and split by whole-slide origin to reduce leakage risk.
- Evaluation design: The study compared the proposed attention-based classifier with the state-of-the-art sliding-window method on high-resolution biopsy images.
- Preprocessing: Preprocessing removed white background and retained only image regions containing tissue.
- Preprocessing: Whole-slide images were separated into large single-tissue images averaging 5,131×5,875 pixels.
- Labeling: Each image received an overall label based on its lesions, assigning the highest-risk class when multiple lesion classes were present.
- Evaluation design: 379 processed images were created, with one third reserved for testing and all tissue images from a whole-slide image kept in the same split.
Sliding Window Baseline
The sliding-window baseline classifies cropped regions and aggregates their predictions for whole-slide inference, using annotated bounding boxes and validation-selected heuristics.
- Training: The baseline was optimized with cross-entropy loss for 100 epochs, using weight regularization and learning-rate decay.
- Baseline construction: The cropped-image classifier predicts the class of each window in a high-resolution image.
- Whole-slide inference: Whole-slide inference filters noisy predictions and aggregates cropped-image outputs using thresholds and heuristics selected on the validation set.Two pathologists separately developed the aggregation heuristics.
Attention Model
The proposed attention model analyzes large histological images with a convolutional feature extractor and grid-based attention, learning attention from tissue-level class labels without ROI annotations. It achieved stronger classification performance than the sliding-window baseline across classes, while attention maps highlighted class-specific and clinically on-target regions.
- Attention Model: 64 convolutional filters increase the robustness of the attention module because feature-space patterns may be too complex for a single filter.
- Attention Model: The attention module uses dropout with p=0.5 after concatenating feature vectors to reduce overfitting and encourage filters to capture different patterns.
- Attention Model: The attention model was trained using class-label cross-entropy without bounding-box locations as guidance for its attention maps.The model nevertheless identified regions of interest.
- Evaluation and Results: The model achieved better accuracy and F1 score than the sliding-window baseline for all classes.Its F1 score exceeded the baseline by at least 8% for each class.
- Evaluation and Results: The attention model performed well on normal, BE-no-dysplasia, and adenocarcinoma images but had poorer performance identifying BE-with-dysplasia images.Most misclassified BE-with-dysplasia samples were predicted as normal, likely reflecting that this class comprised only 11% of the dataset.
- Evaluation and Results: Attention weights were low across regions without target features and focused on specific regions associated with adenocarcinoma-related invasion.The attention map was described as clinically on-target in an example where BE-with-dysplasia progresses to adenocarcinoma.
DISCUSSION
The attention-based model detects Barrett esophagus and esophageal adenocarcinoma using tissue-level annotations rather than ROI bounding boxes. Its performance compares favorably with sliding-window methods, while remaining applicable to high-resolution images and potentially to other data-scarce diseases.
- The model detects Barrett esophagus and esophageal adenocarcinoma using an attention-based deep learning architecture.
- The model achieved higher classification performance than the state-of-the-art sliding-window model while using a relatively low number of parameters.
- The proposed model requires tissue-level reference labels per tissue, whereas sliding-window models require bounding-box annotations for each ROI.
- Its attention mechanism directs information flow toward local features useful for classification, while the architecture remains applicable to high-resolution images without resizing.
- Fewer annotation requirements could facilitate image-analysis research, and the classification scheme may extend to diseases with scarce training data or unavailable bounding-box annotations.
- The dataset was limited because experiments used slides from one medical center scanned with the same equipment, and the dataset was relatively small.
- Robustness and generalizability require verification on different classification tasks and larger datasets from multiple institutions.
- Current GPUs cannot process very large images directly, requiring tissue subdivision or feature-extractor optimization to reduce memory demands.
CONCLUSION
The study presented an attention-based model for high-resolution esophageal microscopy images that uses tissue-level labels rather than region-of-interest annotations. It marginally outperformed the current sliding-window method across four esophageal tissue classes.
- The attention-based model analyzes high-resolution microscopy images using weighted features from the entire slide.Its design parallels pathologists’ whole-slide examination rather than relying only on isolated crops.
- The model marginally outperformed the current sliding-window method on an esophagus-tissue dataset.
- The dataset included normal tissue, BE without dysplasia, BE with dysplasia, and adenocarcinoma.
- Previous microscopy-image methods were limited by bounding-box annotations and unscalable heuristics.
- The proposed model is trained end-to-end using only tissue-level labels, reducing annotation costs and creating opportunities for deep learning in digital pathology.
FUNDING
The research was supported in part by grants R01LM012837 and P20GM104416 from the National Institutes of Health.
- The study received partial support from NIH grants R01LM012837 and P20GM104416.
LEGENDS
The legends describe a grid-based attention architecture that extracts local features, assigns diagnostic importance to grid cells, and aggregates them for whole-slide classification. Attention visualizations show class-specific weighting across four tissue categories, with higher weights displayed in white.
- Attention visualizations: The figure displays whole-slide images and attention maps for normal, BE-no-dysplasia, BE-with-dysplasia, and adenocarcinoma inputs.Higher attention weight is white, lower weight is black, and maps are normalized to a maximum value of 1.
- Attention visualizations: Two expert pathologists qualitatively verified the accuracy of attended regions in adenocarcinoma images, while lower-risk class images received less attention.
- Supplementary materials: The supplementary materials include detailed annotation and architecture methods, additional attention-map examples, and class-distribution information.
- Grid-based Feature Extraction: The model divides a high-resolution image into grid cells and extracts features from each cell to form a structured feature map.The map preserves local geometric relationships while representing the whole high-resolution image.
- Implementation: The feature-extraction implementation uses a pre-activation ResNet-18 architecture to balance performance and GPU memory usage.
- Attention-based Classification: A learnable 3-dimensional convolutional filter operates on the feature map to generate attention values and an attention map.The filters have size k × d × d, where d denotes kernel height and width.
- Attention-based Classification: The attention map weights local features to produce a whole-slide global feature vector for classification.The resulting vector is passed through fully connected layers for whole-slide prediction.
- Attention-based Classification: Multiple attention modules generate separate feature vectors that are concatenated before fully connected classification.