Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Images

Muhammad Shaban; Ruqayya Awan; Muhammad Moazam Fraz; Ayesha Azam; David Snead; Nasir M. Rajpoot

Source-linked AI summary

Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Images

Muhammad Shaban, Ruqayya Awan, Muhammad Moazam Fraz, Ayesha Azam, David Snead, Nasir M. Rajpoot

arXiv:1907.09478v1Jul 22, 2019eess.IVcs.LGstat.ML

TL;DR

Small-patch CNNs limit high-resolution contextual information in digital histology images, motivating a larger-context approach. The paper stacks local representation learning with spatial feature aggregation on 1,792 × 1,792 inputs and reports superior performance across the evaluated comparisons.

Problem
CRC grading requires both high-resolution cellular information and the spatial organization of cells, while standard patch-based CNNs provide limited context.
Method
The framework uses an LR-CNN to encode local patch features and an RA-CNN to aggregate them according to their spatial organization.
Results
The context-aware networks achieve superior CRC grading performance compared with standard patch-based classifiers, with RA-CNN 3 obtaining the best rank-sum.
Takeaways & Limitations
The framework supports context-aware histology analysis across CRC grading and breast cancer classification without relying on a cancer-specific intermediate segmentation step.

Abstract

from arXiv · show

Digital histology images are amenable to the application of convolutional neural network (CNN) for analysis due to the sheer size of pixel data present in them. CNNs are generally used for representation learning from small image patches (e.g. 224x224) extracted from digital histology images due to computational and memory constraints. However, this approach does not incorporate high-resolution contextual information in histology images. We propose a novel way to incorporate larger context by a context-aware neural network based on images with a dimension of 1,792x1,792 pixels. The proposed framework first encodes the local representation of a histology image into high dimensional features then aggregates the features by considering their spatial organization to make a final prediction. The proposed method is evaluated for colorectal cancer grading and breast cancer classification. A comprehensive analysis of some variants of the proposed method is presented. Our method outperformed the traditional patch-based approaches, problem-specific methods, and existing context-based methods quantitatively by a margin of 3.61%. Code and dataset related information is available at this link: https://tia-lab.github.io/Context-Aware-CNN

I. INTRODUCTION

Histology grading requires both high-resolution cellular information and larger-scale spatial context, which standard small-patch CNNs do not capture well. The paper proposes a flexible context-aware framework that learns local features from large high-resolution inputs and aggregates them spatially.

Motivation: Standard CNN approaches commonly analyze small patches, limiting the contextual information available for histology grading.The motivation is especially strong for grading because distorted glandular structures vary in size and shape.
Motivation: CRC grading requires high-resolution cell information together with the spatial organization of cells in the tumor microenvironment.The paper uses visual fields selected by an expert pathologist as diagnostic regions containing enough context for reliable grading.
Framework: The proposed framework uses 1,792 × 1,792 images, 64 times larger than the standard patch classifier input.An LR-CNN converts patches into high-dimensional feature vectors while preserving their spatial arrangement for later aggregation.
Framework: The framework combines local representation learning with representation aggregation to make predictions from high-resolution features and spatial context.The architecture can incorporate any state-of-the-art image classifier for local representation learning.
Evaluation: The study explores multiple context-aware learning and training strategies and reports comparisons across more than 100 network models.The comparisons include traditional patch-based methods and existing context-aware learning methods.

II. RELATED WORK

Prior work incorporates histology context through down-sampling, patch selection, larger patches, or specialized sequence and graphical models, but these approaches have important task or design constraints. The proposed method instead combines flexible CNN-based representation learning with custom representation aggregation and avoids dependence on gland segmentation.

Context-aware learning: Down-sampling and sparse patch-selection methods incorporate context but are unsuitable or limited when cell information and context are both important.The related-work discussion identifies limited context or loss of high-resolution cellular information as constraints.
Context-aware learning: Prior high-resolution context methods used CRFs or 2D-LSTMs, including a 672 × 672 end-to-end trainable patch that is smaller than the proposed input.These methods were developed mainly for segmentation or related histology tasks.
Proposed distinction: The proposed design accommodates any state-of-the-art CNN for representation learning and adds a custom CNN architecture for representation aggregation.Representation learning and aggregation are stacked for context-aware learning from 1,792 × 1,792 images.
Problem-specific methods: Unlike a CRC method based on gland segmentation and handcrafted morphology features, the proposed framework does not depend on intermediate gland segmentation.The paper also presents the framework for breast cancer classification, indicating use beyond CRC grading.

III. THE PROPOSED METHOD

The proposed method stacks two CNNs: an LR-CNN extracts local patch representations into a spatially arranged feature-cube, and an RA-CNN aggregates that representation to learn contextual structure for prediction.

Network architecture: The LR-CNN converts high-resolution images into a high-dimensional feature-cube through patch-based feature extraction.The feature-cube preserves the spatial arrangement of the extracted patches.
Network architecture: The RA-CNN aggregates the learned representation to model spatial context from the feature-cube and produce a prediction.The framework explores different context-block architectures within the RA-CNN.
Network input: Each input image is represented with a class label and a coarse patch-level segmentation mask, then divided into M × N patches.The dataset definition supports classification and multi-task learning.
Network input: A patch dataset with corresponding labels is used to pretrain the LR-CNN.The supplied method description identifies this patch-level dataset as the pretraining source.

B. Local Representation Learning

The LR-CNN extracts local representations from image patches, pools them into consistent feature vectors, and rearranges them spatially into a feature-cube. An attention block can reweight this feature-cube to emphasize more useful image regions.

Local feature extraction: The LR-CNN processes input images patch by patch to produce local feature representations.The framework can use state-of-the-art classifiers such as ResNet50, MobileNet, Inception, or Xception as the LR-CNN.
Feature pooling: Global average or max pooling produces similarly dimensional feature vectors across patch and network variations.Pooling is applied before spatial rearrangement into the feature-cube.
Feature-cube construction: Pooled patch features are rearranged in their original M × N spatial order to construct the feature-cube for context-aware learning.
Feature attention: An attention block learns usefulness weights for feature-cube values and applies a Hadamard product to emphasize important image areas during label prediction.The attention mechanism reduces the influence of insignificant features and increases the impact of more important regions.

E. Context Blocks

The context block learns spatial relationships among patch features in the feature-cube. Three architectures provide different complexity and capabilities for capturing contextual information, including parallel receptive fields in the third design.

Context-block design: The context block learns spatial context by relating feature-cube features according to their image-patch locations.
Context-block design: Three context-block architectures are proposed with different complexity and contextual-capture capabilities.
B2 context block: B2 concatenates its final convolutional feature-maps with the input feature-maps to form the block output.
B3 context block: B3 processes feature-maps in parallel with different filter sizes to capture context from varying receptive fields.Its branches use 1 × 1 and 3 × 3 convolutions, batch normalization, ReLU activation, and average pooling.

F. Representation Aggregation for Context Learning

Representation aggregation applies cascaded context blocks to the feature-cube, then uses global average pooling, a fully connected layer, and softmax to produce the final prediction. The framework is trained end-to-end with categorical cross-entropy.

Context aggregation: A cascaded set of three same-type context blocks learns spatial context from the feature-cube.The cascade can use B1, B2, or B3 blocks.
Prediction: Global average pooling, a fully connected layer, and a softmax layer follow context aggregation to produce the final prediction.
Training objective: The proposed framework is trained end-to-end using a categorical cross-entropy classification loss.The classification loss is denoted Lcls(·).

G. Auxiliary Block

The auxiliary block adds coarse patch-level segmentation to image classification, enabling joint training with classification and segmentation losses. Training strategies additionally vary weighting, auxiliary supervision, and attention.

Auxiliary supervision: The auxiliary block produces a coarse patch-based segmentation mask alongside the image label.It uses primitive structures such as glands and their spatial organization to support classification.
Multi-task learning: Jointly minimizing segmentation and classification losses is reported to improve classification performance.
Joint loss: The joint loss combines classification and segmentation losses, with α controlling their relative contribution.The framework minimizes the joint loss using RMSprop.
Training strategies: Four training strategies are evaluated: standard, weighted, auxiliary, and attention.They differ in loss weighting, auxiliary segmentation supervision, and attention-based feature-cube weighting.

IV. DATASETS & PERFORMANCE MEASURES

The study evaluates its framework on colorectal cancer grading and reports the metrics used to compare models across three folds and different context settings.

Dataset: The evaluation uses colorectal histology images classified into normal, low-grade, and high-grade classes based on gland organization.The CRC dataset contains 139 images averaging 4,548 × 7,520 pixels at 20× magnification.
Metrics: Three metrics measure performance: average accuracy, F1 score, and Rank-sum measure.Average accuracy is the percentage of correctly classified images across the three folds.
Metrics: Rank-sum summarizes model accuracy under a specific setting to compare networks using different context blocks and local representation CNNs.The measure supports comparisons across models trained with different context configurations.
Metrics: Result colors rank methods relative to the best performer, with orange marking the best and other colors denoting progressively lower performance ranges.Green, blue, yellow, and red indicate results within 97.5%, 95%, 90%, and 85% of the best method, respectively.

V. EXPERIMENTS & RESULTS

Experiments compare patch classifiers with context-aware networks using large sliding-window inputs. Context-aware variants outperform standard patch-based classifiers, with RA-CNN 3 showing the strongest rank-sum performance.

Experimental setup: The evaluation compares traditional patch classifiers, existing context-aware approaches, and domain-oriented methods for colorectal cancer grading.The comparison is organized into three categories.
Experimental setup: The proposed method processes 1,792 × 1,792 inputs with a 224 × 224 stride, reusing local features before spatial context aggregation.The local representation CNN computes features once, after which the representation aggregation CNN applies a sliding window.
Patch baselines: Four 224 × 224 patch classifiers—ResNet50, Inception, MobileNet, and Xception—provide baseline CRC grading results.These networks serve as local representation CNNs for comparison.
Patch baselines: MobileNet achieves the highest mean accuracy, whereas Xception has the lowest standard deviation across the three folds.The classifiers have comparable overall performance, but they differ in average accuracy and consistency.
Context-aware results: RA-CNN 3 achieves the lowest Rank-sum and the context-aware networks outperform standard patch classifiers.The other two context-aware networks remain within 97.5% of the best performer.

D. Local Representation Robustness

The experiments analyze feature-extractor robustness, pooling and training strategies, and comparisons with domain-based and context-aware methods. Xception-based context-aware configurations are robust and produce smoother, more consistent predictions.

Local representation robustness: Xception is the most robust feature extractor across combinations of representation aggregation networks and feature-pooling strategies.Inception produces comparable results because its architecture substantially overlaps with Xception.
Training strategies: Auxiliary-loss training achieves the best overall accuracy across average- and max-pooled features, while every tested model exceeds the baseline local classifier.Attention performs best for max-pooled features, whereas standard training is comparable for average-pooled features.
Comparative evaluation: The proposed RA-CNN 3 with Xception and attention-based training outperforms BAM-based domain methods for colorectal cancer grading.The comparison addresses the multi-class CRC grading setting.
Comparative evaluation: Patch-classifier predictions are irregular, while the proposed method produces smooth and ground-truth-consistent predictions in visual CRC comparisons.Context-G predictions are smoother but misclassify the low-grade example.

VI. CONCLUSION

The paper presents a two-stage context-aware network that combines local feature learning with spatial representation aggregation over inputs 64 times larger than standard patches. Evaluations on colorectal grading and breast classification show improved performance over patch-based, domain-oriented, and existing context-based methods.

Contribution: The proposed network uses an LR-CNN for local histology representation and an RA-CNN for aggregating spatial patterns.The design targets cancer grading tasks requiring contextual information about structures such as distorted glands.
Contribution: The framework incorporates 64 times larger context than standard CNN patch classifiers.The large-context design addresses structures that standard patch classifiers cannot efficiently capture under computational and memory constraints.
Results: Qualitative and quantitative evaluations show that the method outperforms patch-based, domain-oriented, and existing context-based approaches.The framework is evaluated for colorectal cancer grading and breast cancer classification.
Implications: The authors identify potential use in other cancer-analysis tasks requiring large histology-image context and possible extension to whole-slide patient-survival analysis.The examples include Gleason grading and lung tumor growth-pattern classification.