PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Image Compression for Gigapixel Histopathology Image Analysis

David Tellez, Geert Litjens, Jeroen van der Laak, Francesco Ciompi

arXiv:1811.02840v2cs.CVeess.IV

TL;DR

Gigapixel image analysis with weak labels must recover sparse, unknown visual cues from images dominated by pixel-level noise. NIC compresses patches into learned spatially arranged representations, then trains a CNN for image-level prediction without fine-grained annotations. Across synthetic and histopathology tasks, it exploited global and local cues, and its visualized attention overlapped expert annotations.

  • Problem

    Gigapixel images contain subtle label-related patterns amid distracting noise, while the nature and spatial distribution of those cues are unknown.

  • Method

    NIC compresses gigapixel patches into compact neural representations while preserving spatial arrangement, then uses those representations to train a CNN for image-level prediction.

  • Results

    NIC exploited visual cues using global structure and local high-resolution information across synthetic and histopathological classification and regression tasks.

  • Takeaways & Limitations

    NIC provides a label-agnostic strategy for gigapixel analysis and supports classification, regression, and visualization in learned latent spaces.

  • Takeaways & Limitations

    NIC required substantial storage and I/O, and struggled with micro-metastases smaller than 2 mm in Camelyon16 data.

Abstract

from arXiv · show

We propose Neural Image Compression (NIC), a two-step method to build convolutional neural networks for gigapixel image analysis solely using weak image-level labels. First, gigapixel images are compressed using a neural network trained in an unsupervised fashion, retaining high-level information while suppressing pixel-level noise. Second, a convolutional neural network (CNN) is trained on these compressed image representations to predict image-level labels, avoiding the need for fine-grained manual annotations. We compared several encoding strategies, namely reconstruction error minimization, contrastive training and adversarial feature learning, and evaluated NIC on a synthetic task and two public histopathology datasets. We found that NIC can exploit visual cues associated with image-level labels successfully, integrating both global and local visual information. Furthermore, we visualized the regions of the input gigapixel images where the CNN attended to, and confirmed that they overlapped with annotations from human experts.

1 INTRODUCTION

Gigapixel image analysis with weak image-level labels must identify both the nature and location of sparse visual cues amid substantial pixel-level noise. NIC addresses this challenge by compressing gigapixel images into compact latent representations that preserve spatial structure for CNN-based prediction and visualization.

  • 1 INTRODUCTION: Gigapixel images contain over 1 billion pixels, with subtle label-related patterns mixed with distracting noise and unknown signal location.These uncertainties are framed as the what and where problems.
  • 1 INTRODUCTION: Patch-level annotation simplifies analysis but is tedious, error-prone, and unsuitable when image-level labels lack a known patch-level representation.This approach also limits learned representations to human annotator knowledge.
  • 1 INTRODUCTION: NIC maps gigapixel images from pixel space to compact latent representations, enabling CNN training for image-level labels using a single GPU.The method does not assume a particular nature or spatial distribution of visual cues.
  • 1 INTRODUCTION: The compressed representation preserves spatial arrangement while replacing patches with low-dimensional embedding vectors.This retains neighborhood relationships between patches for downstream CNN analysis.
  • 1 INTRODUCTION: NIC compares reconstruction-error, contrastive, and adversarial feature-learning encoders without manual annotations.The study evaluates these strategies on synthetic and histopathology datasets involving classification and regression labels.
  • 1 INTRODUCTION: Grad-CAM localizes CNN attention on gigapixel images and compares visualized regions with expert patch-level annotations.The analysis targets interpretability and knowledge discovery in medical applications.

2 NEURAL IMAGE COMPRESSION

NIC learns compact patch embeddings through unsupervised encoders and arranges them spatially into representations for gigapixel analysis. The paper compares variational autoencoding, contrastive learning, and adversarial feature learning as alternative encoding strategies.

  • 2 NEURAL IMAGE COMPRESSION: NIC divides a gigapixel image into patches, independently encodes each patch into a length-C vector, and stores vectors in their original spatial arrangement.Neighboring embeddings therefore correspond to neighboring patches in the source image.
  • 2 NEURAL IMAGE COMPRESSION: All encoding strategies learn the encoder through unsupervised auxiliary tasks and require no manual annotations.The compared families are reconstruction error minimization, contrastive training, and adversarial feature learning.
  • 2 NEURAL IMAGE COMPRESSION: A VAE encodes each patch as a probability distribution and decodes an embedding into a reconstruction, optimizing reconstruction error together with KL divergence.This produces a continuous latent space in which embedding changes correspond to input-data changes.
  • 2 NEURAL IMAGE COMPRESSION: Contrastive training uses a Siamese network to classify whether patch pairs came from the same or different spatial locations.Strong augmentation makes same-location patches visually different while preserving overall semantic structure.
  • 2 NEURAL IMAGE COMPRESSION: BiGAN trains a generator, discriminator, and encoder in a minimax game over real and generated image-embedding pairs.The encoder learns a low-dimensional representation intended for downstream tasks.

3 GIGAPIXEL IMAGE ANALYSIS

The method trains a CNN to predict image-level labels from compressed gigapixel images while exploiting both global spatial context and local high-resolution features. Cropping, augmentation, tissue-aware sampling, and parameter-efficient convolutions address overfitting and computational constraints, while Grad-CAM localizes the visual cues used for prediction.

  • CNN training: The CNN combines global context from embedding-vector arrangements with local high-resolution information within each embedding vector.
  • CNN training: Compressed images remain single training points, so the method uses spatial crops to increase the number and variability of CNN training samples.During training, crop centers are sampled randomly; during testing, predictions from uniformly distributed crops are averaged.
  • CNN training: Whole-slide images are sampled preferentially from tissue-dense regions because large empty areas contain no tissue.Training crops are sampled proportionally to distance from background, and test crops are drawn where tissue is present.
  • CNN training: Image-level rotations and mirroring encode each image eight times, with predictions averaged across these augmentations during testing.
  • CNN training: Depthwise separable convolutions reduce the CNN's parameter count while maintaining a similar level of performance.
  • Visualizing visual cues related to image-level labels: Grad-CAM uses gradients of intermediate feature maps with respect to the image-level label to weight feature maps and localize relevant visual cues.The visualization is applied to the first convolutional layer to maximize heatmap resolution.

4 EXPERIMENTS AND RESULTS

NIC was evaluated on synthetic data and public histopathology cohorts using image-level labels, with experiments spanning encoder comparisons, hyper-parameters, predictive performance, and saliency localization.

  • Experimental setup: NIC was tested first on synthetic data and then on public histopathology datasets, including Camelyon16, TUPAC16, and Rectum.The synthetic setup assessed method behavior and hyper-parameters; real cohorts supported downstream evaluation.
  • Synthetic experiments: The synthetic task required detecting a tilted rectangle and its class while combining local digit features with global spatial arrangement.The model performed this task without access to ground-truth masks.
  • Hyper-parameter analysis: Larger code sizes generally improved performance, especially for VAE and BiGAN, whereas smaller strides that increased spatial resolution hampered performance.NIC required training data on the order of thousands of images to perform well.
  • Encoder comparison: BiGAN achieved the best unsupervised patch-level performance, including strong results for challenging tissue classes such as blood cells and necrotic tissue.It achieved the best result for 5 of 9 classes and ranked first or second for 8 of 9 classes among unsupervised models in Task 2.
  • Image-level proliferation prediction: BiGAN obtained a 0.521 Spearman correlation for proliferation-speed prediction and 0.557 on the TUPAC16 Challenge test set, exceeding the reported 0.516 state-of-the-art comparison.The leaderboard’s first entry used additional manual annotations of mitotic figures and was not comparable with this setup.
  • Saliency analysis: Grad-CAM visualizations showed BiGAN focusing on tumor regions while discarding empty areas, fatty tissue, and healthy dense tissue, although failure cases highlighted non-tumorous regions.The heatmaps were described as difficult to interpret quantitatively.

5 DISCUSSION

NIC exploited weak image-level visual cues across synthetic and histopathological tasks, combining global structure with local high-resolution information. BiGAN-based compression was strongest across histopathological experiments, while performance remained constrained for very small lesions and limited training data.

  • Encoding strategies: BiGAN encoding was superior to other unsupervised strategies across all histopathological experiments.The authors attribute this advantage to BiGAN learning high-level features and semantics through adversarial feature learning.
  • Image-level prediction: NIC successfully predicted tumor proliferation speed from gene-expression-associated labels with unknown visual cues.The method surpassed the current state-of-the-art among image-level-based methods.
  • Visualization: CNN attention maps overlapped expert-annotated tumor regions only for BiGAN-compressed images and the supervised-tumor baseline.The BiGAN model simultaneously delimited metastatic lesions and identified tumor features within patch embeddings.
  • Limitations: BiGAN performed similarly to the supervised baseline for breast lymph-node metastasis, but remained below the Camelyon16 leaderboard result of 0.9935 AUC.The authors associate this gap with micro-metastases and possible overfitting from only a few hundred training images.
  • Limitations: NIC performance was proportional to lesion size and struggled with Camelyon16 micro-metastases smaller than 2 mm.The authors state that this limits applicability to tasks involving large lesions.
  • Future directions: Attention mechanisms may improve detection of small lesions, while more sophisticated encoders and gradient checkpointing are proposed extensions.These extensions target representation quality, relevant-region selection, and backpropagation toward encoder weights.

6 CONCLUSION

NIC distilled gigapixel images into compact learned representations that preserve a higher level of abstraction than low-level pixel arrays. These representations supported classification, regression, visualization, and potential additional latent-space applications.

  • Conclusion: NIC distilled relevant information into compact image representations and enabled CNN training on alternative learned representations.The approach shifts gigapixel analysis from low-level pixel arrays toward higher-level representations.
  • Conclusion: The learned latent space supported examples of classification, regression, and visualization.The authors identify data augmentation, generative modeling, content retrieval, anomaly detection, and image captioning as further applications.

Variational Autoencoder

The variational autoencoder maps image patches to compact stochastic embeddings and reconstructs the patches from them. Its objective balances reconstruction error with KL divergence regularization.

  • Architecture: The VAE encoder maps an input patch x ∈ R^P×P×3 to an embedding e ∈ R^C, while the decoder reconstructs x as x′.The implementation used P = 128 and C = 128.
  • Latent representation: The VAE encoder outputs µ and σ, representing the mean and standard deviation of a normal distribution for each input x.The embedding is sampled using n ∼ N(0, 1) and element-wise multiplication.
  • Architecture: The encoder used five strided-convolution layers followed by dense layers, ending with a linear layer containing C units.The convolutional layers used 32, 64, 128, 256, and 512 filters.
  • Architecture: The decoder began with a dense layer reshaped to 4 × 4 × 512 and used five upsampling-convolution stages before a three-channel tanh output.The upsampling stages used 256, 128, 64, 32, and 16 filters.
  • Training objective: The VAE objective minimizes reconstruction error plus γ-weighted KL divergence between the embedding distribution and N(0, 1).The authors used γ = 5 × 10^-5 and selected the encoder with the lowest validation loss.

Contrastive Training

Contrastive training learns compact patch embeddings by distinguishing pairs with the same or different semantic information. A Siamese network shares the encoder across both inputs and selects the encoder using validation accuracy.

  • Training data: The training data consisted of patch pairs x = {x^(1), x^(2)} with binary labels y and patch size P = 128.Each pair was used for a binary classification task.
  • Network: A two-branch Siamese network shared encoder weights across both inputs before concatenating their embeddings for an MLP classifier.The MLP ended with a single sigmoid unit.
  • Optimization: The encoder was selected from the Siamese model achieving the highest validation classification accuracy.Training minimized binary cross-entropy with Adam and reduced the learning rate from 1 × 10^-2 to 1 × 10^-5.

Bidirectional Generative Adversarial Network

The BiGAN compression model jointly trains a generator, encoder, and discriminator so images and embeddings become mutually consistent and realistic. The encoder is then selected from training to produce compressed representations.

  • Model structure: BiGAN uses a generator G, discriminator D, and encoder E to model images and their latent embeddings.G maps normally distributed latent variables to artificial images, while E maps real images to embeddings.
  • Adversarial training: The discriminator distinguishes real image-embedding pairs {x, e} from generated pairs {x′, z}, while G and E attempt to fool it.The adversarial objective encourages realistic generated images and embeddings close to the normal latent distribution.
  • Architectures: The encoder uses four strided-convolution layers followed by a dense layer with C units, and the generator uses a dense layer plus four upsampling layers.Both networks use batch normalization and leaky rectified activations in their convolutional or dense blocks.
  • Discriminator: The discriminator processes an image and a low-dimensional vector, concatenates the image embedding with the latent input, and classifies the pair with a sigmoid output.Its image-processing network matches the encoder architecture but has different weights.
  • Training protocol: BiGAN training uses Adam with 64-sample mini-batches, a fixed learning rate of 2 × 10^-4, and 200,000 epochs before selecting the final encoder.The input image is downsampled by a factor of 2 to improve training stability.

Mean-RGB Baseline

The mean-RGB baseline compresses each RGB patch by averaging intensity across spatial dimensions, while related encoders and a CNN support classification and visualization of compressed gigapixel images. Grad-CAM weights intermediate feature maps by output sensitivity to produce relevance heatmaps.

  • Mean-RGB Baseline: Mean-RGB compression represents each RGB patch with one value per color channel by averaging pixel intensity across its spatial dimensions.The channel index c covers the three RGB channels, while j and k index the two spatial dimensions.
  • Classification encoders: The supervised encoders use convolutional layers, a dense layer with 256 units, and either a sigmoid or nine-unit softmax output for classification.The binary and multiclass variants use the same encoder structure but different final outputs and cross-entropy losses.
  • Synthetic dataset: For the synthetic dataset, grayscale patches use P = 9 and C = 16, with encoder architectures reduced to two strided-convolution layers.The VAE, Contrastive, and Supervised encoders share this reduced architecture.
  • Downstream classification: An MLP trained on frozen encoder weights uses a 256-unit dense layer and sigmoid or nine-unit softmax outputs for the two classification tasks.Models are selected using the highest validation classification accuracy.
  • Gigapixel CNN: The gigapixel CNN uses eight strided depthwise-separable convolutional layers, regularization, dropout, and a final linear or sigmoid output for regression or classification.Training uses Adam with 16-sample mini-batches and reduces the learning rate when the validation metric plateaus.
  • Visualizing where the information is located: Grad-CAM computes feature-map gradients with respect to the class output, averages them spatially, and uses the resulting coefficients to weight feature maps.The weighted sum produces a two-dimensional heatmap highlighting regions relevant to prediction, with k = 1 selected for maximum resolution.
Loading 1811.02840v2…