Source-linked AI summary
Yottixel -- An Image Search Engine for Large Archives of Histopathology Whole Slide Images
S. Kalra, C. Choi, S. Shah, L. Pantanowitz, H. R. Tizhoosh
TL;DR
Pathology archives contain increasingly large collections of WSIs, but textual annotations do not adequately describe their visual content for retrieval. Yottixel indexes representative patch mosaics as compact barcodes, and its reported classification accuracies range from 69.84% to 93.20% across the specified diagnosis tasks.
Problem
Text-based searches provide limited access to anatomical and morphological similarities in pathology images, whose visual patterns are difficult to describe comprehensively.
Source: §1 · p. 2Method
Yottixel represents WSIs with mosaics of representative patches converted into a compact Bunch of Barcodes index for image retrieval.
Source: §3 · p. 6Results
Classification accuracy ranged from 69.84% for lung adenocarcinoma versus squamous cell carcinoma to 93.20% for the stated adrenal gland and uterine diagnosis comparisons.
Source: §5.2 · p. 13Takeaways & Limitations
The BoB index accelerates retrieval and alleviates computation and storage burdens for laboratory and clinic deployment.
Source: §3 · p. 6Treating retrieval as classification may be too restrictive because differently diagnosed samples can share morphology, and accurate diagnosis may require ancillary studies.
Source: §5.2 · p. 13
Abstract
from arXiv · showhide
With the emergence of digital pathology, searching for similar images in large archives has gained considerable attention. Image retrieval can provide pathologists with unprecedented access to the evidence embodied in already diagnosed and treated cases from the past. This paper proposes a search engine specialized for digital pathology, called Yottixel, a portmanteau for "one yotta pixel," alluding to the big-data nature of histopathology images. The most impressive characteristic of Yottixel is its ability to represent whole slide images (WSIs) in a compact manner. Yottixel can perform millions of searches in real-time with a high search accuracy and low storage profile. Yottixel uses an intelligent indexing algorithm capable of representing WSIs with a mosaic of patches by converting them into a small number of methodically extracted barcodes, called "Bunch of Barcodes" (BoB), the most prominent performance enabler of Yottixel. The performance of the prototype platform is qualitatively tested using 300 WSIs from the University of Pittsburgh Medical Center (UPMC) and 2,020 WSIs from The Cancer Genome Atlas Program (TCGA) provided by the National Cancer Institute. Both datasets amount to more than 4,000,000 patches of 1000x1000 pixels. We report three sets of experiments that show that Yottixel can accurately retrieve organs and malignancies, and its semantic ordering shows good agreement with the subjective evaluation of human observers.
1. Motivation
Text-based searches of pathology archives cannot adequately capture the visual patterns needed to retrieve similar images. Yottixel is motivated by the need to search whole-slide images despite their scale and complexity.
1. Motivation
- Digital pathology archives are growing, but convenient access to their stored diagnostic, research, and educational knowledge remains lacking.Source: §1 · p. 1
- Text annotations provide limited access to anatomical and morphological similarities because they cannot comprehensively describe the visual variability of pathology images.The passage notes that annotation effort is substantial and that visual patterns derived from basic tissue types are highly varied.Source: §1 · pp. 1–2
- Content-based image retrieval represents image content with feature vectors and retrieves similar images through nearest-neighbor matching rather than metadata.Useful features are intended to capture semantic structures despite changes in scale, rotation, translation, and some deformation.Source: §1 · pp. 2–3
- Whole-slide images are gigapixel-scale and visually complex, while many CBIR methods use low-level features or target much smaller natural images.Yottixel is presented as addressing these limitations with deep features and compact indexing.Source: §1 · p. 3
- The paper frames Yottixel as a prototype using deep architectures and an intelligent, compact “bunch of barcodes” indexing technique for pathology image search.Source: §1 · p. 3
2. Related Works
Prior histopathology retrieval work spans sub-image and whole-slide approaches, but methods face trade-offs in expert effort, scale, and storage. Yottixel is designed to index unlabelled WSIs with compact, transferable binary codes.
2. Related Works
- Sub-setting reduces processing time but requires expert intervention, whereas tiling supports full automation with more costly processing and careful design.Source: §2 · p. 4
- Earlier CBIR approaches used feature descriptors on query images or sub-images, while reported studies included 80% top-five accuracy on 50 IHC images and 93%/86% classification accuracy for FL/NB.The latter system evaluated 1,666 WSIs from follicular lymphoma and neuroblastoma slides.Source: §2 · p. 4
- Supervised kernel hashing compressed 10,000-dimensional features into ten bits, with reported 88.1% accuracy and 10 ms processing for 800 test images.Source: §2 · pp. 4–5
- Kernelized and supervised hashing uses semantic labels to improve search accuracy, but its optimization is time-consuming.The review contrasts it with unsupervised and semi-supervised methods and other supervised hashing approaches.Source: §2 · p. 5
- Yottixel’s motivation includes limited WSI-level search, reliance on labelled repositories, storage demands from real-valued indexing, and hash-code transfer concerns.The cited examples include patch-focused work and methods using labelled malignant-region delineations.Source: §2 · p. 5
- Yottixel combines deep features with color clustering, a representative patch mosaic, and a “bunch of barcodes” to index unlabelled images using explicit binary codes.The design targets fast search and indexes that can be transferred between locations.Source: §2 · p. 5
3. Yottixel – A Search Engine for WSI
Yottixel represents each whole-slide image (WSI) with a mosaic of selected patches and indexes that mosaic as a Bunch of Barcodes (BoB). Its indexing and search design targets fast, low-storage retrieval, with runtime searches available within one primary site or across all sites.
3. Yottixel – A Search Engine for WSI
- Yottixel combines a representative-patch mosaic with a BoB index in an end-to-end framework designed for high-speed, low-storage WSI retrieval.The mosaic represents the WSI with a small set of patches; its barcodes form the index.Source: §3 · p. 6
3.1. Offline Indexing Phase
- Color-based k-means segmentation divides each WSI into nine tissue regions, then spatially diverse patches are sampled to form a mosaic about 20 times smaller than the specimen area.The number of regions was chosen by manually inspecting WSIs; clustering may be done at 5× magnification.Source: Figure 3 · p. 6, §3.1 · p. 7
- For an average 700 MB WSI, the BoB index can be as small as 10 KB, using deep-network patch features converted into binary barcodes.The paper reports this index as 70,000 times smaller than the original file.Source: §3.1 · pp. 7–8
- MinMax binarization enables real-time searches by converting query features to barcodes and using accelerated CPU commands to calculate Hamming distances.The paper contrasts this with direct deep-feature distance searches, which pose computational-efficiency challenges in large databases.Source: §3.1 · p. 8
3.2. Runtime Search Phase
- Runtime search can be restricted to the query patch’s primary site or performed across all primary sites in the index.The framework runs runtime search alongside preemptive offline indexing.Source: §3 · p. 7, §3.2 · p. 8
4. Datasets
The evaluation uses a private UPMC collection and a public TCGA subset, with WSIs labeled by primary diagnosis and affected organ. The datasets have substantial class imbalance, and preprocessing included slide removal and tissue segmentation.
4. Datasets
- The study uses 300 UPMC WSIs and 2,020 TCGA WSIs, with compressed dataset sizes of 104 GB and 2 TB, respectively.The UPMC slides span more than 80 primary diagnoses across multiple organs; the TCGA slides come from a repository of more than 33,000 WSIs.Source: §4 · p. 8
- Each WSI was labeled with its primary diagnosis and affected organ, and slides with poor staining, low resolution, a single pyramid level, or large out-of-focus regions were removed.Five UPMC slides and seven NIH slides were removed.Source: §4 · p. 9
- UPMC tissue segmentation sometimes required manual correction, whereas the larger NIH dataset received no manual intervention.Both datasets had high class imbalance; in UPMC, 30% of WSIs belonged to the four largest classes.Source: §4 · p. 9
5. Experiments and Results
Across scan matching, classification, and user review, Yottixel retrieved WSIs with matching attributes and its rankings aligned with expert feedback. Results also show a trade-off between mosaic size and patch-level specificity, while classification is an imperfect proxy for diagnostic similarity.
5.1. Experiment Series 1: Scan-to-Scan Matching
- Except for rectum adenocarcinoma, accuracy reached 100% within ten retrievals, and Yottixel performed three to four times better than random selection.The comparisons cover scan-to-scan retrieval across primary-site and diagnosis tasks.Source: §5.1 · p. 11
- Smaller mosaics showed no significant performance improvement over larger ones for most experiments, except thymus, while patch-retrieval specificity can be lost with fewer indexed patches.The authors chose a 5% patch mosaic as a storage and retrieval compromise.Source: §5.1 · p. 12
5.2. Experiment Series 2: Classification
- Classification accuracy ranged from 69.84% for lung adenocarcinoma versus lung squamous cell carcinoma to 93.20% for uterine carcinosarcoma versus uterine corpus endometrial carcinoma.The task distinguished primary diagnoses within the same primary site using majority voting among the top five results.Source: §5.2 · pp. 12–13
- Treating retrieval as classification can be too restrictive because differently diagnosed samples may remain morphologically similar and diagnostically useful.The authors note that accurate diagnosis may also require immunohistochemistry and molecular testing.Source: §5.2 · p. 13
5.3. Experiment Series 3: Testing by Users
- Seven users evaluated randomized query images and top-three results across 48 queries, with the pathologist’s feedback analyzed separately from non-experts’ feedback.The seven participants were one pathologist and six non-clinical users with computer-vision experience.Source: §5.3 · p. 13
- The pathologist rated Q1 highest for Great responses, Q2 higher than Q3 for Great and Good, and Q3 highest for Very Bad responses.Both expert and non-expert users ranked Q1 more positively than Q3 overall.Source: §5.3 · p. 13
- For the pathologist, images marked Great had a much lower median Hamming distance than images marked Very Bad.Good and Great were treated as agreement, while Bad and Very Bad were treated as disagreement.Source: §5.3 · p. 14
6. Summary and Conclusions
Yottixel is an image-search platform for digital pathology that combines deep patch features with supervised and unsupervised algorithms. Preliminary tests on private and public WSI datasets found accurate classification-style retrieval and good agreement with expert evaluations, supporting further investigation.
6. Summary and Conclusions
- Yottixel combines supervised and unsupervised algorithms, using deep features to characterize patches and segmentation, clustering, networks, and distance metrics for retrieval.The search itself uses raw data without task-specific training.Source: §6 · p. 14
- Tests on 300 private and 2,025 public WSIs found accurate classification-style retrieval, striking visual similarity for most queries, and overall good conformance with expert evaluations.Both experts and non-experts evaluated the visual similarity of retrieved cases.Source: §6 · p. 14
- The preliminary results on a small, diverse private dataset and a medium-sized public dataset demonstrate feasibility and justify further investigation.The authors say they continue to improve the platform’s accuracy and speed for diagnostic, research, and educational use.Source: §6 · p. 14