PaperlayerView exact revision on arXiv

Source-linked AI summary

Quilt-1M: One Million Image-Text Pairs for Histopathology

Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, Linda Shapiro

arXiv:2306.11207v4cs.CVcs.CLcs.LG

TL;DR

Histopathology lacks comprehensive vision-language data despite the complexity of its visual information. This paper curates QUILT from expert educational YouTube videos, expands it into QUILT-1M with additional sources, and fine-tunes vision-language models that outperform competing models across diverse classification and retrieval tasks. The authors also report curation errors and insufficient performance from training CLIP from scratch as important limitations.

  • Problem

    Histopathology lacks comprehensive vision-language datasets, while categorical labels oversimplify its complex, information-dense image patterns.

  • Method

    The paper curates QUILT from expert-narrated YouTube videos using multiple models and algorithms, combines it with other sources into QUILT-1M, and fine-tunes vision-language models.

  • Results

    QUILT-1M models outperform state-of-the-art alternatives across zero-shot, linear-probing, and cross-modal retrieval tasks on diverse histopathology datasets.

  • Takeaways & Limitations

    QUILT-1M establishes a large open histopathology vision-language resource and supports improved representation learning across multiple sub-pathologies and tasks.

  • Takeaways & Limitations

    Handcrafted algorithms and LLMs introduce curation biases and errors, including approximately 5% classifier false positives and occasional unrectifiable ASR term substitutions.

Abstract

from arXiv · show

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has slowed comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering $1,087$ hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate QUILT: a large-scale vision-language dataset consisting of $802, 144$ image and text pairs. QUILT was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around $200$K samples. We combine QUILT with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: QUILT-1M, with $1$M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of QUILT-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across $13$ diverse patch-level datasets of $8$ different sub-pathologies and cross-modal retrieval tasks.

1 Introduction

Histopathology images contain complex, densely informative patterns that categorical labels cannot fully capture, while comprehensive vision-language datasets remain scarce. The paper introduces QUILT and QUILT-1M by curating expert-narrated YouTube content and combining it with other sources, then evaluates the resulting data for vision-language learning.

  • Motivation: Histopathology patches contain complex patterns, making single categorical labels an oversimplification of tissue characterization.Natural-language descriptions can instead link diverse features of histopathology sub-patch structures.
  • Motivation: Existing histopathology vision-language datasets remain limited, with ARCH at approximately 8K samples and OpenPath at about 200K.PMC-15M contains 15M biomedical image-text pairs, but histopathology specificity is ambiguous and the dataset is not openly available.
  • Contributions: The curation pipeline combines search, trained image-frame extraction and denoising, ASR, UMLS, LLM postprocessing, and domain-specific image-text pairing with duplicate removal.LLMs extract sub-pathology, medical, and region-of-interest text from the videos.
  • Contributions: QUILT contains 437,878 images aligned with 802,144 text pairs across microscopic magnifications from 10x to 40x.The dataset draws on 1,087 hours of educational histopathology videos from expert pathologists on YouTube.
  • Contributions: QUILT-1M combines QUILT with Twitter, research-paper, and Internet data to produce one million image-text pairs and the largest public histopathology vision-language dataset.QUILT does not overlap with current open-access histopathology data sources, enabling combination with other open datasets.
  • Contributions: Models trained with QUILT-1M outperform CLIP, PLIP, and BiomedCLIP across zero-shot classification, linear probing, and cross-modal retrieval evaluations.The evaluation spans 13 external histopathology datasets, and ablations identify the importance of QUILT.
  • Contributions: QUILT provides non-overlapping data, expert-derived textual descriptions, and multiple sentences per image for more expressive and diverse supervision.These properties are presented as advantages for computer scientists and histopathologists.

2 Related work

Prior histopathology representation learning largely relied on weak supervision or unimodal self-supervision, while vision-language datasets remained limited. Existing datasets and video-based supervision illustrate both the promise of richer paired data and the challenge of handling noisy video sources.

  • Machine learning for histopathology: Early computational pathology representation learning assigned a single weak label to each whole-slide image, producing sub-optimal patch-level models.More recent work applied self-supervised learning to unlabeled data, but most approaches remained unimodal.
  • Medical vision-language datasets: Histopathology vision-language representation learning is constrained by the scarcity of large datasets pairing images with descriptive text.Related medical datasets include chest radiographs with reports and limited de-identified PDF reports for whole-slide images.
  • Vision-language pairs on histopathology: ARCH contains 7,614 accessible histopathology image-text pairs, while OpenPath contains 200K pairs extracted from Twitter.OpenPath was the largest histopathology dataset until QUILT-1M.
  • Video data for self-supervision: Video data can provide dense, interconnected supervision for vision-language learning, as demonstrated by YouTube-based and localized-narrative studies.Video-based datasets are often noisier than datasets built from static sources.

3 Curating QUILT: Overview

QUILT is curated from narrative-style histopathology videos by extracting, denoising, and aligning visual and textual content. The resulting dataset combines medical and ROI descriptions across magnifications and diverse sub-pathology content.

  • Video selection: The pipeline filters videos for voiced English histopathology content and narrative presentation involving vocal descriptions alongside panning or zooming over whole-slide images.Videos are assessed for required medical content and narrative-style visual behavior before downloading suitable material at high resolution.
  • Image extraction: QUILT collects representative frames by using keyframe-based time intervals rather than extracting frames at fixed intervals.The approach accounts for instructors’ variable panning and zooming and extracts representative images from stable frames within temporal chunks.
  • Text processing: The curation pipeline combines ASR, denoising algorithms, UMLS-linked terminology checks, spell correction, and LLM-based extraction of medical and ROI text.LLMs filter medically relevant text and identify image-level medical descriptions and region-of-interest descriptions after ASR correction.
  • Dataset output: QUILT contains 437K unique images and 802K associated text pairs from 4,504 narrative videos spanning 1,087 hours, with content across multiple magnifications and histopathology sub-domains.The dataset includes 1.469M UMLS entity mentions, and its captions cover findings, organs, diseases, cells, neoplastic processes, and diagnostic procedures.

4 QUILTNET: Experiments training with QUILT-1M

QUILTNET is pretrained with a CLIP contrastive objective on QUILT-1M and evaluated across histopathology classification and cross-modal retrieval tasks. It outperforms competing models across broad zero-shot and linear-probing evaluations.

  • Training setup: QUILTNET uses CLIP contrastive pretraining to align image and text embeddings in a joint space.The InfoNCE objective increases cosine similarity for aligned image-text pairs and decreases it for unaligned pairs.
  • Evaluation: 13 downstream histopathology datasets support evaluation across zero-shot classification, linear probing, and cross-modal retrieval.The datasets span multiple sub-pathologies and include external evaluation sets plus a QUILT-1M holdout.
  • Linear probing: Table 1 reports linear-probing classification as accuracy % with standard deviation, alongside each dataset’s supervised state-of-the-art result.Camelyon refers to the PatchCamelyon dataset.
  • Zero-shot results: QUILTNET outperforms other models on all but two zero-shot datasets, where BiomedCLIP performs marginally better.Comparisons include CLIP, BiomedCLIP, and PLIP.
  • Linear probing: With 1% of labels, QUILTNET surpasses fully supervised performance on NCT-CRC and SICAPv2.The linear-probing study evaluates 1%, 10%, and 100% training-data settings across four datasets.

5 Discussion

The discussion presents QUILT-1M as a large open histopathology dataset whose pretraining supports state-of-the-art performance across several tasks. It also identifies curation, data, and training limitations that constrain interpretation and generalization.

  • Limitations: QUILT curation introduces bias and errors, including approximately 5% false positives from its histopathology classifier and occasional ASR medical-term substitutions.The paper states that current pipeline errors are not rectifiable.
  • Limitations: Training CLIP from scratch underperformed fine-tuning a pre-trained CLIP, suggesting one million image-text pairs may still be insufficient.The authors identify alternative self-supervised objectives as a possible direction for future work.
  • Data and societal bias: English-only videos and predominantly Western narrators may bias QUILTNET toward associated demographics and reduce performance for other cultural or linguistic groups.The paper links this concern to the composition of the collected educational videos.
  • Contributions: QUILT-1M is presented as the largest open-sourced histopathology dataset to date.The conclusion reports gains across zero-shot, few-shot, full-shot, and cross-modal retrieval tasks.
  • Results: QUILTNET establishes state-of-the-art results in zero-shot classification, linear probing, and cross-modal retrieval across varied sub-pathology tasks.The conclusion specifically notes performance against larger state-of-the-art models such as BiomedCLIP.

Supplementary material

The supplementary material documents the dataset pipeline, data analysis, model evaluation, representation analysis, and dataset documentation.

  • Data curation: Section A describes data-curation models, algorithms, and parsing pipelines.
  • Data analysis: Section B presents exploratory analysis of the collected data.
  • Evaluation: Section C provides pretraining and downstream-evaluation details.
  • Representations: Section D explores representations learned by the trained models.
  • Documentation: Section E supplies a datasheet for the QUILT dataset.

A Data curation models, algorithms and parsing pipelines

The curation pipeline combines video discovery, filtering, frame extraction, speech transcription, denoising, and image-text alignment, then supplements QUILT with PubMed, LAION, and Twitter data.

  • Collecting representative channels and videos: ≈65K potential YouTube matches were narrowed to ≈9K videos by processing frames to distinguish histopathology from general pathology content.Narrative-style filtering further selected ≈4K videos.
  • Frame extraction and video filtering: Keyframe thresholds vary from 0.008 for 5-minute videos to 0.25 for 200-minute videos before histopathology classification and narrative-style analysis.A pre-trained CLIP ViT-B-32 model compares sampled keyframes using cosine similarity.
  • Text extraction and denoising: Large-V2 Whisper converts speech to text, while the pipeline addresses medical transcription errors using keyword extraction, ontology checks, and specialized spell checking.The denoising pipeline cross-checks refined phrases against UMLS and histopathology terminology.
  • Image frame extraction and denoising: Static-frame detection and representative-frame selection reduce duplicate images within video chunks before image-text alignment.Stable frames are selected when available; otherwise, the most dissimilar unstable frames are used.
  • Aligning both modalities: Image-text alignment uses timing and matching keywords within video chunks, with text overlap retained to provide context for LLM correction.Images are not overlapped across chunks.
  • Adding external sources: PubMed processing yielded 59,371 histopathology image-caption pairs from 109,518 unique articles, while LAION and Twitter supplied additional image-text data.The PubMed search covered 2010–2022; LAION retrieval used image and text embeddings, and Twitter contributed 133,511 unique pairs.

A.4 Support Models, Ontology Databases and Algorithms

The pipeline uses ontology databases, classifiers, LLM prompting, and image-processing algorithms to support search, parsing, classification, and multi-plane figure preprocessing.

  • Ontology databases: Histopathology-specific ontologies condition the LLM to identify incorrect words, while all ontologies and UMLS validate its output.The listed resources include HPATH, MPATH, BCGO, CMPO, OCHV, FMA, NCIT, and UMLS.
  • PubMed parsing pipeline: PubMed curation searches the open-access database, downloads papers and media, extracts figure-caption pairs, separates multi-plane figures, and identifies histopathology images and magnification.Figure 11 summarizes this sequence.
  • Sub-pathology classification: The LLM classifies text into the top three of 18 sub-pathology types using few-shot prompting.The categories span fields including bone, cardiac, gastrointestinal, pulmonary, renal, and breast histopathology.
  • Multi-plane figure preprocessing: Sobel filters and binary thresholding locate boundaries and white gaps between sub-figures for multi-plane figure preprocessing.Successful crops retain histopathology regions, while unsuccessful crops preserve the original figure.
  • Privacy measures: Privacy measures include masking irrelevant or sensitive information, excluding videos that are not fully narrative, and releasing video IDs with reconstruction code instead of the complete dataset.Collection across diverse channels is also described as a privacy measure.

B Exploratory analysis of the collected data

The exploratory analysis describes QUILT’s distribution across 18 sub-pathology types and points readers to additional dataset statistics.

  • Dataset distribution: QUILT data are distributed across 18 sub-pathology types, with Figure 4 presenting the text distribution.The section describes this as a comprehensive analysis of the dataset’s text distribution.
  • Additional statistics: Table 6 provides supplementary statistics covering additional aspects of the QUILT dataset.The passage directs readers to the table for further statistical details.

C.1 External Evaluation Datasets

The external evaluation uses diverse patch-level histopathology datasets with varied image sizes, labels, tissue types, magnifications, and disease categories.

  • Patch-level evaluation datasets: PatchCamelyon contains 327,680 96×96-pixel color images with binary metastatic-tissue labels.The images come from histopathology scans of lymph node sections.
  • Whole-slide and disease-specific datasets: BACH comprises 400 breast-tissue whole-slide images labeled normal, benign, in-situ, or invasive carcinoma.Other listed datasets include Osteo and RenalCell, covering osteosarcoma and clear-cell renal cell carcinoma.
  • Evaluation configuration: Table 7 summarizes the downstream tasks and datasets used for evaluation, including SkinTumor as a subset of SkinCancer.The table records the evaluation dataset configuration.

C.2 QUILTNET Implementation

QUILTNET is implemented with OpenCLIP-based contrastive image-text training and specified image preprocessing. Its evaluation includes zero-shot classification, attention-map comparisons, and embedding visualization, with results organized in the referenced tables and figures.

  • Implementation: OpenCLIP provides the implementation foundation for QUILTNET training with contrastive image-text supervision.The experiments use PyTorch and up to four NVIDIA A40 GPUs.
  • Image preprocessing: 224-pixel images are used generally, while finetuned ViT-B-32 inputs are resized to 512 before random cropping.ViT-B-32 finetuning preserves one-to-one image-text alignment; other models use unstretched sampling.
  • Evaluation: Table 9 reports zero-shot image-classification accuracy across datasets and compares QUILTNET variants, including a PubmedBert text-encoder swap.The table marks models trained from scratch and identifies SkinTumor as the Neoplastic Subset of SkinCancer.
  • Evaluation: Table 10 defines the class prompts used for zero-shot image classification across datasets.The templates describe histopathology slides, images, pathology tissue, or tissue presence using the target class.
  • Interpretability: Figure 14 compares QUILTNET and CLIP attention maps, highlighting words by importance and generating masks with GradCAM.The comparison links textual word importance to visual attention-map outputs.
  • Embedding analysis: Table 11 visualizes QUILTNET image embeddings with UMAP across the datasets listed in Table 7.The visualization is intended to compare embedding structure across datasets.

E Datasheet for QUILT

The QUILT datasheet documents the dataset’s motivation, composition, collection, preprocessing, and distribution. It supports histopathology multimodal pretraining while retaining boundaries from sampling bias, noisy ASR, incomplete metadata, privacy constraints, and non-exhaustive video coverage.

  • Motivation: QUILT was created to train in-domain histopathology multimodal models for diagnostically relevant downstream tasks.The dataset can also support representation learning and supervised learning on metadata.
  • Data composition: The dataset contains more than one million image-text pairs from educational videos and less noisy sources such as PubMed articles.Instances include images, descriptive text, estimated magnification, and auxiliary medical metadata, though some fields are source-dependent.
  • Scope and use: The dataset has no recommended splits because it was curated mainly for pretraining rather than evaluation.The collected video list is not exhaustive, so additional instances may be collected in the future.
  • Collection: Data collection used the YouTube API and youtube-dl with probabilistic sampling guided by algorithms and heuristics.The data was collected from January 2023 to May 2023, although the videos themselves are often older.
  • Preprocessing: Preprocessing denoises ASR transcripts with a large language model, UMLS, and algorithms, then uses histopathology classifiers to select relevant segments and representative images.The raw data was saved but is not planned for direct release because of copyright and privacy concerns.
  • Limitations and distribution: Known limitations include YouTube-related biases, inaccuracies in curation models such as ASR, and missing auxiliary information for non-QUILT instances.The authors plan to release derived data and video links rather than the videos themselves, allowing users to delete videos.
Loading 2306.11207v4…