Source-linked AI summary
Similar Image Search for Histopathology: SMILY
Narayan Hegde, Jason D. Hipp, Yun Liu, Michael E. Buck, Emily Reif, Daniel Smilkov, Michael Terry, Carrie J. Cai, Mahul B. Amin, Craig H. Mermel, Phil Q. Nelson, Lily H. Peng, Greg S. Corrado, Martin C. Stumpe
TL;DR
Large histopathology images and coarse metadata make regions of interest laborious to locate. SMILY uses deep-learning embeddings for reverse image search without labeled histopathology training data, and evaluations found similar histologic features, organ sites, and prostate cancer grades, supporting its use as a general-purpose search tool.
Problem
Large histopathology images lack precise region-of-interest annotations, so locating relevant features after metadata search can require laborious visual searching.
Method
SMILY creates neural-network image embeddings for patches and retrieves database patches with similar embeddings, without using labeled histopathology images.
Results
SMILY retrieved similar histologic features, organ sites, and prostate cancer grades in annotated-image and blinded pathologist evaluations, outperforming SIFT or random retrieval in reported comparisons.
Takeaways & Limitations
SMILY may serve as a general-purpose tool for searching large histopathology archives without developing separate tools for each application.
Takeaways & Limitations
The study evaluated a limited set of similarity axes and slides, so broader tissue-processing conditions and additional similarity dimensions require further validation.
Abstract
from arXiv · showhide
The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Though these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. Because pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep learning based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY's ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist's arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application.
2 Avoneaux Medical Institute, Baltimore, MD 21215, USA
The passage identifies an affiliation with the Avoneaux Medical Institute in Baltimore, Maryland.
- The passage lists the Department of Pathology and Laboratory Medicine.
- The listed institutional affiliation is associated with the University of Tennessee Health Science Center.
Introduction
Digital pathology enables large image archives to support diagnosis, research, and education, but locating small regions of interest remains difficult. SMILY addresses this gap with unlabeled-image reverse search evaluated across organs and pathology attributes.
- Histopathology slides can reach 100,000×100,000 pixels, while clinical annotations usually do not identify lesion locations.A lesion may occupy only 100 pixels across, requiring additional visual searching.
- Prior application-specific CBIR systems require labeled data and define similarity along task-specific axes such as organ, cancer, staining, or histologic features.
- SMILY is a histopathology reverse image search tool developed without labeled histopathology images.
- The evaluation covered breast, prostate, and colon images using pathologist annotations and blinded prospective comparisons against random retrieval.
Results
SMILY encoded image patches into embeddings and retrieved visually similar regions from large histopathology databases. Across quantitative and blinded pathologist evaluations, it retrieved related histologic features, organ sites, and prostate cancer grades, with scalable query performance.
- SMILY encoded image patches as numerical embeddings and retrieved database patches nearest to a query embedding.The evaluation database contained 127,000 patches from 45 slides, with 22,500 query patches from 15 additional slides.
- SMILY retrieved the correct prostate cancer Gleason pattern in 76.0% of cases versus 65.2% for SIFT, and matched both pattern and histologic feature in 25.3% versus 17.8%.Both comparisons were significant at p<0.001.
- In blinded pathologist studies, SMILY exceeded random retrieval for histologic-feature matching in prostate queries, 62.1% versus 26.8%, and across organs, 57.8% versus 18.3%.Both comparisons were significant at p<0.001.
- SMILY achieved 61.0% match quality for prostate cancer patches versus 30.0% for random results, while 68.3% of unambiguous organ-site results matched the query organ.Only 32.0% of individual search results had an organ site judged unambiguous.
- A TCGA database of about 10^9 patches yielded a median 1.3-second query time using 400 computers with 10 compute threads each.A naive single-machine implementation on 10^7 patches required 25 seconds per query.
Discussion
SMILY offers general-purpose, label-free reverse image search for histopathology, with demonstrated retrieval of similar tissue features, organs, and prostate cancer grades. Its broad applicability comes with accuracy, similarity-definition, and validation limits relative to specialized tools.
- SMILY retrieved similar histologic features, organ sites, and cancer grades in annotated-tissue analyses and blinded prospective pathologist studies.The authors describe this as the paper’s most comprehensive evaluation of reverse image search for histopathology.
- Because search intent may target different properties, an ideal system should provide breadth or enable interactive refinement rather than optimize one similarity axis.The authors identify ambiguity around matching stain, stain intensity, histologic feature, or lesion similarity.
- SMILY can support diagnosis, research, and education by searching similar lesions or features across large image archives.The discussion frames these as potential use cases, including rare-feature searches and large cohort research.
- A general-purpose tool may be useful despite lower accuracy than application-specific systems, whose separate development is impractical for every task.The paper identifies labeling costs, workflow integration, and legal or commercial issues as challenges for specialized tools.
- The core network was pretrained on natural images rather than histopathology, avoiding the need for large pixel-annotated pathology datasets.The authors suggest this may help generalize across laboratory and scanner characteristics, while noting histopathology-specific training could further improve embeddings.
- The study evaluated similarity along histologic features, organ site, and prostate cancer grade, leaving other similarity axes and targeted refinement for future validation.The authors also note that broader slide sampling is needed to capture more tissue-processing conditions and resulting images.
Methods
SMILY builds a searchable embedding database from histopathology patches and retrieves visually similar patches by comparing query embeddings. The methods combine pretrained deep ranking features, orientation handling, efficient lookup, annotated quantitative evaluation, and blinded pathologist assessment.
- Architecture and retrieval: SMILY compresses image patches into fixed-length embeddings with a deep ranking network and retrieves matches using L2 distance between 128-dimensional vectors.The system handles eight image orientations and presents one orientation for each distinct patch.
- Database and interface: Efficient lookup uses k-d trees with configurable leaf size and depth, parallelized comparisons, and a reported lookup time of seconds on a web interface.The implementation targets large numbers of embedding comparisons and sublinear lookup in the number of comparisons.
- Database and interface: SMILY creates a database of patch embeddings from slides, with users selecting 200–400-pixel query patches that are resized to 224×224 pixels before search.The web-based viewer displays a customizable number of retrieved patches.
- Evaluation design: Quantitative evaluation used pathologist-annotated histologic features and Gleason patterns, extracting balanced 300×300-pixel patches for database construction and testing.Annotations were used for evaluation, with hundreds to thousands of patches subsampled per category.
- Evaluation design: Prospective studies had pathologists rate four results per query, using binary axis scores and a 100-point overall match-quality score.Three pathologists participated, and 25% of queries were random negative controls with readers blinded to retrieval source.
Competing Interests
The authors disclose financial relationships involving Google LLC and Alphabet stock, along with compensation for pathology expertise.
- Several authors are Google LLC employees and Alphabet stockholders, while two pathologists were compensated for their expertise and time.
Figures
The figures depict SMILY’s patch-embedding search workflow, user interface, evaluation against baselines, pathologist assessment, and embedding-space structure.
- Overview: SMILY computes embeddings for database patches and a query, then returns a customizable number of the most similar patches.The workflow uses a convolutional neural network and computationally efficient embedding comparison.
- User interface: The interface displays a prostate query with retrieved patches, including magnified views and controls for prospective result scoring.
- Quantitative evaluation: Figure 3 compares histologic-feature and Gleason-grade retrieval with SIFT and random-search baselines using pathologist-provided annotations.
- Confusion matrices: Figure 4 maps top-5 retrieval hits across categories for prostate-only and multi-organ histologic-feature searches.Rows represent query categories and columns represent matched-result categories.
- Pathologist evaluation: Figure 5 summarizes blinded pathologist comparisons of SMILY and random retrieval for histology, organ site, and combined prostate similarity.
- Embedding structure: Embedding visualizations show stronger separation by histologic feature than by organ site.
Tables
The tables summarize balanced evaluation datasets for automated and manual studies, annotated prostate categories, and the multi-aspect match-quality scoring scheme.
- Large-scale quantitative study: Table 1 uses random subsampling to provide 5,000 patches per histologic feature per organ for large-scale quantitative evaluation.The sampling strategy was intended to avoid evaluation bias.
- Data categories: The prostate categories include non-tumor tissue and Gleason patterns 3, 4, and 5.
- Data limitations: The prostate patch counts exclude lymphocytes because none were found during a non-exhaustive specimen review.
- Pathologist studies: Table 2 uses the same database as Table 1 but subsamples queries to make manual pathologist evaluations tractable.
- Scoring: Similarity is scored separately for histology, organ, and Gleason pattern, with an overall match-quality score defined in 25-point increments.
Supplementary Material
The supplementary material examines magnification, metrics, database size, retrieval examples, and alternative embedding generators.
- Magnification: At 5X magnification, top-5 histology-match results are lower because fewer patches pass the 1,000-pixel filter.The reduced patch pool also affects search-result diversity.
- Additional performance metrics: Supplementary analyses compare histology metrics, top-k choices, database sizes, and searches using original prostate annotations.
- Retrieval examples: Example retrievals show related structures across prostate and breast queries and consistently similar gland architecture for colon queries.
- Embedding comparisons: Alternative embedding generators include SMILY’s deep ranking network, Inception V3, a Gleason-grading network, and SIFT.