Source-linked AI summary
Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning
Oren Z. Kraus, Lei Jimmy Ba, Brendan Frey
TL;DR
Microscopy analysis is constrained by the scarcity of single-cell labels and by pipelines that require assay-specific processing. The paper combines CNNs with multiple instance learning on full-resolution images, introducing adaptive Noisy-AND pooling to handle heterogeneous cell populations and outliers. Across mammalian and yeast datasets, the convolutional MIL models outperform comparison methods without segmentation steps.
Problem
Microscopy images are difficult to analyze with CNNs because datasets labeled at the single-cell level are scarce, while images contain many cells and outliers.
Method
The paper trains a CNN end-to-end with multiple instance learning on full-resolution microscopy images and uses adaptive Noisy-AND pooling with learned class-specific thresholds.
Results
Convolutional MIL models outperform several benchmarks across evaluations on mammalian and yeast microscopy datasets without requiring segmentation steps.
Takeaways & Limitations
The approach supports learning classifiers from whole-image labels while also providing predictions for cropped cells and localized cellular regions.
Takeaways & Limitations
The method assumes that bag labels reflect a thresholded proportion of positive instances, and some existing pooling functions were sensitive to outliers and failed on microscopy datasets.
Abstract
from arXiv · showhide
Convolutional neural networks (CNN) have achieved state of the art performance on both classification and segmentation tasks. Applying CNNs to microscopy images is challenging due to the lack of datasets labeled at the single cell level. We extend the application of CNNs to microscopy image classification and segmentation using multiple instance learning (MIL). We present the adaptive Noisy-AND MIL pooling function, a new MIL operator that is robust to outliers. Combining CNNs with MIL enables training CNNs using full resolution microscopy images with global labels. We base our approach on the similarity between the aggregation function used in MIL and pooling layers used in CNNs. We show that training MIL CNNs end-to-end outperforms several previous methods on both mammalian and yeast microscopy images without requiring any segmentation steps.
1 Introduction
Microscopy screening produces vast numbers of images, but analysis often depends on assay-specific pipelines and single-cell labels that are difficult to obtain. The paper addresses this challenge by training CNNs on full-resolution images with global labels through multiple instance learning.
- High-content screening can produce more than 10^5 images per day, making automated image analysis essential.
- Existing microscopy pipelines commonly require hand-tuned segmentation, feature extraction, and classification for each assay.
- Deep networks can learn representations end-to-end, but microscopy images contain many cells and outliers while single-cell-labeled datasets are scarce.
- The proposed CNN treats feature-map elements as instances and full-resolution images as labeled bags within a multiple instance learning framework.
- Because cellular heterogeneity and imaging artifacts violate the usual positive-if-any-instance MIL assumption, the paper introduces pooling that learns the instance proportion needed to activate each label.
2 Related Work
Prior microscopy-analysis methods rely on predefined features, while deep-learning studies commonly require pixel-level labels or target segmentation and localization. Multiple instance learning provides a framework for learning from bag-level labels when instance-level annotations are unavailable.
- Existing microscopy tools extract predefined features and then select task-relevant features, which can limit transfer to assays unlike their development datasets.
- A proteome-wide yeast localization study illustrates this limitation by resorting to manual image evaluation.
- Deep neural networks have been applied to microscopy segmentation using ground-truth pixel-level labels, unlike the paper's bag-level training setup.
- Fully convolutional networks convert fully connected layers to 1x1 convolutions to produce feature maps for each output category in segmentation tasks.
- Multiple instance learning represents sets of data points as bags and their members as instances when labels exist only for the sets.
3 Convolutional MIL Model for Learning Cellular Patterns
The model treats full-resolution microscopy images as MIL bags, using class-specific feature maps and global pooling to convert instance predictions into image-level classifications. Adaptive Noisy-AND pooling handles phenotype prevalence variability, while Jacobian maps enable cell localization and segmentation.
- 3 Convolutional MIL Model for Learning Cellular Patterns: The network classifies images of arbitrary size from global labels while also producing class-specific feature maps and cropped-cell instance predictions.A MIL pooling layer aggregates varying numbers of cells into image-level classifications.
- 3.1 Pooling layers as multiple instance learning: Each class-specific feature-map element is an instance, and the full-resolution image is the labeled MIL bag.Instance probabilities pij are pooled into a single bag-level probability for each class.
- 3.1 Pooling layers as multiple instance learning: Global pooling maps instance probabilities to one class prediction, allowing training and evaluation on images with arbitrary sizes and numbers of cells.An additional fully connected layer can model co-occurrence relationships between classes using softmax or sigmoid outputs.
- 3.2 Global pooling functions: Existing Noisy-OR and ISR pooling functions were sensitive to outliers, whereas lower-sharpness LSE and GM settings allowed more instances to contribute.The pooling analysis varied how closely functions approached max-like aggregation.
- 3.2.1 Adaptive Noisy-AND pooling function: Adaptive Noisy-AND activates a bag probability when the mean instance probability exceeds a learned class-specific threshold.The parameter bi provides an adaptable soft threshold, while a controls the activation slope.
- 3.3 Localizing cells with Jacobian maps: Jacobian maps are generated by back-propagating class-specific MIL activations, then thresholded and denoised to produce cellular segmentation masks.ReLU during back-propagation produces sharper, more localized maps.
4 Experiments
Experiments evaluate the MIL CNN on mammalian and yeast microscopy screens, including full images, cropped cells, classification, and localization. The model also generates segmentation maps from full-resolution images.
- Datasets: The breast cancer dataset contains 3-channel 1024x1280 images of 103 treatments grouped into 12 mechanism-of-action categories.
- Datasets: The yeast dataset contains 4,144 strains representing approximately 71% of the proteome, with about 2,200 whole images used for training.
- Model and training: The CNN uses class-specific feature maps and a global MIL pooling function to produce image-level predictions from arbitrary-sized inputs.
- Model and training: Training used Adam with a learning rate of 10^-3, channel-wise mean and standard-deviation normalization, and random 900x900 crops from full-resolution images.
- Evaluation: Performance was evaluated on full-resolution images, automatically cropped cells, and manually labeled yeast cells using classification metrics reported for each dataset.
- Localization: The localization method achieved a mean intersection over union of 81.2% against CellProfiler segmentation maps generated with global Otsu thresholding.
5 Discussion & Conclusions
Convolutional MIL combines CNN feature learning with pooling over full-resolution microscopy images, avoiding intermediate segmentation and single-cell labels. Adaptive Noisy-AND performs best in bag-level evaluations, while MIL models also identify phenotype-relevant cellular regions.
- Approach: Convolutional MIL models combine CNN classification with MIL pooling to learn from full-resolution microscopy images without intermediate segmentation.The approach supports both bag-level and instance-level classification.
- Benchmark results: The models outperform previously published results on both yeast and breast cancer datasets while requiring fewer preprocessing and postprocessing steps.Comparisons include cropped-cell CNNs, global-max-pooling MIL, yeast SVM ensembles, and breast-cancer factor-analysis results.
- Instance evaluation: MIL models achieve higher mean average precision on manually labeled cropped cells than on cells carrying noisy bag-level labels.This evaluation pattern supports learning phenotype-relevant cells from global image annotations.
- Localization: Class-specific Jacobian maps show that different image patterns activate distinct output feature maps, enabling localization of phenotype-relevant cellular regions.The maps provide a route to regional predictions in addition to global image predictions.
- Pooling: Adaptive Noisy-AND models perform best across bag-level evaluations by learning class-specific thresholds for varying cell densities.Lower scaling factors make pooling approach the mean, whereas higher values make it approach the maximum.
- Dataset-specific findings: On the breast cancer dataset, CNNs outperform previously reported treatment accuracy, while MIL CNNs and the recognition CNN perform similarly.The MIL CNNs retain an advantage because they do not require intermediate segmentation steps.
Supplementary Material
The supplementary material presents class examples for the yeast and breast cancer datasets, along with representative full-resolution images from each dataset.
- Dataset classes: Figure S1 presents the classes used for the yeast dataset.
- Dataset classes: Figure S2 presents the classes used for the breast cancer dataset.
- Representative images: Figure S3 shows a sample full-resolution image from the yeast dataset.
- Representative images: Figure S4 shows a sample full-resolution image from the breast cancer dataset.