PaperlayerView exact revision on arXiv

Source-linked AI summary

Quilt-1M: One Million Image-Text Pairs for Histopathology

Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, Linda Shapiro

arXiv:2306.11207v4cs.CVcs.CLcs.LG

TL;DR

Histopathology lacks comprehensive vision-language datasets despite the complexity of tissue patterns. The paper builds QUILT-1M by combining YouTube-derived QUILT with data from other sources and fine-tunes vision-language models. QUILT-1M-trained models outperform prior baselines on classification and cross-modal retrieval tasks.

  • Problem

    Histopathology has a scarcity of comprehensive vision-language datasets, despite the complexity of image patterns and the potential of language descriptions to represent them.

    Source: §1 · p. 1

  • Method

    The authors curate QUILT from educational YouTube content, combine it with data from other sources into QUILT-1M, and fine-tune vision-language models using a contrastive objective.

    Source: §1 · p. 2

  • Results

    QUILT-1M-trained models outperform CLIP, PLIP, and BiomedCLIP across zero-shot classification, linear probing, and cross-modal retrieval, establishing state-of-the-art results in these tasks.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    QUILT-1M offers a million-pair public vision-language resource for histopathology research.

    Source: §1 · p. 2

    Handcrafted algorithms and LLM-based curation introduce biases and errors, including occasional classifier false positives and ASR medical-term substitutions that the current pipeline cannot rectify.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has slowed comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering $1,087$ hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate QUILT: a large-scale vision-language dataset consisting of $802, 144$ image and text pairs. QUILT was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around $200$K samples. We combine QUILT with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: QUILT-1M, with $1$M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of QUILT-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across $13$ diverse patch-level datasets of $8$ different sub-pathologies and cross-modal retrieval tasks.

1 Introduction

The paper introduces QUILT and QUILT-1M, histopathology image-text datasets built from educational videos and other sources, and reports that QUILT-1M-trained models outperform several baselines across classification and retrieval tasks.

1 Introduction

  • Categorical labels oversimplify histopathology’s complex, evolving disease subtypes, while natural-language descriptions can connect diverse features of tissue and sub-patch structures [20].Source: §1 · p. 1
  • Existing histopathology vision-language datasets remain limited in size: ARCH [20] has about 8K samples and OpenPath about 200K, while PMC-15M [71] spans biomedical domains with ambiguous histopathology specificity and is not openly available.Source: §1 · pp. 1–2
  • QUILT contains 437,878 images aligned with 802,144 text pairs across 10x–40x magnifications, curated from 1,087 hours of educational YouTube videos by expert pathologists.The pipeline combines GPT-3.5, handcrafted algorithms, human knowledge databases, and automatic speech recognition to extract and pair image-text content.Source: §1 · p. 2, Figure 1 · p. 2
  • Because QUILT does not overlap with existing open-access histopathology sources, it can be combined with Twitter, research papers, and internet data to form QUILT-1M, a one-million-pair public dataset.QUILT also provides expert-derived, expressive descriptions and multiple sentences per image for diverse perspectives.Source: §1 · p. 2
  • Models trained with QUILT-1M outperform CLIP [49], PLIP, and BiomedCLIP [71] on zero-shot classification, linear probing, and image-to-text and text-to-image retrieval across 13 external histopathology datasets.Ablations identify the importance of QUILT.Source: §1 · p. 2

2 Related work

Related work describes histopathology representation learning as mostly unimodal and identifies a shortage of aligned medical image-text data. The paper builds on work in video-based supervision and automated curation while exploring language descriptions for histopathology.

Machine learning for histopathology

  • Prior histopathology representation learning used weakly supervised single-label WSIs [27] and mostly unimodal self-supervision, whereas this work uses language descriptions to learn beyond a single label.Prior methods injected H&E stain composition [27] or hierarchical morphology [13], combined images with genomic features or descriptive text [20], and used natural-language augmentations [53].Source: §2 · p. 3

Medical vision-language datasets

  • Medical vision-language datasets are concentrated in radiology [30] [18], while PMC-15M [71] is not histopathology-specific or public, obscuring its histopathology-relevant share.The Cancer Genome Atlas provides reports for limited WSIs, whose large size—up to 120,000^2 pixels—has restricted processing to focused studies.Source: §2 · p. 3

Vision-language pairs on histopathology

  • ARCH contains 7,614 accessible histopathology image-text pairs [20] [23], while OpenPath's 200K Twitter pairs remained the largest dataset until QUILT-1M.Source: §2 · p. 3

Video data for self-supervision

  • Prior work used YouTube videos for vision-language training and gaming videos for causal modeling [6], while ASR and LLMs support dataset curation and annotation.Video can be noisier than static sources; localized narratives [47] provide dense supervision, and ASR [6] [71] and LLMs [35] [63] [15] [22] [67] [1] [60] have been used for curation, extraction, correction, and reasoning.Source: §2 · p. 3

3 Curating QUILT: Overview

QUILT’s curation pipeline uses models and handcrafted algorithms to address noisy speech, video selection, and image–text alignment challenges. The paper describes how it builds QUILT from YouTube videos, expands it into QUILT-1M with other sources, assesses curation quality, and reports dataset statistics.

3.1 QUILT: Collecting medical image and text pairs from YouTube

  • For video selection, QUILT excludes short, non-voiced, or non-English videos and retains narrative-style videos whose histopathology frames meet a similarity streak threshold.Narrative style requires three subsequent keyframes to each have cosine similarity ≥0.9, with streaks in at least 10% of selected frames; an ensemble of three classifiers identifies histopathology content.Source: §3.1 · pp. 4–5
  • The text pipeline uses Whisper [50], then combines keyword extraction, UMLS checks, specialized spell-checking, and LLM correction to denoise transcripts and extract medical and ROI text.Corrected terms are validated against UMLS and a curated histopathology vocabulary; LLMs are instructed to filter for relevant text without adding words, and to classify videos among 18 sub-pathology classes.Source: §3.1 · pp. 5–6
  • For image–text alignment, QUILT extracts representative frames from video chunks and maps images to medical or ROI text when nearby ASR keywords overlap with caption keywords.Stable frames are represented by their pixel-space median when available; otherwise, histopathology keyframes are deduplicated.Source: §3.1 · p. 6

3.2 QUILT-1M: Combining QUILT with other histopathology data sources

  • QUILT-1M expands QUILT with 59,371 PubMed image-text pairs, 22,682 LAION pairs, and 133,511 image-text pairs from OpenPath’s curated Twitter data.The Twitter data comprise 55,000 unique tweets with multiple captions per image; LAION-5B is described as containing over 5 billion image-text pairs [55].Source: §3.2 · pp. 6–7

3.3 Quality

  • Conditioned LLM correction precision is 57.9%, unconditioned precision is 13.8%, average ASR error rate is 0.79%, and sub-pathology classification accuracy is 94.9% top-3, 91.9% top-2, and 86.8% top-1.Classification accuracy was assessed on 100 randomly selected videos; extracting medically relevant text also removes identifiable information such as clinic addresses.Source: §3.3 · p. 7

3.4 Final dataset statistics

  • QUILT contains 802K associated text pairs and over 437K unique images, collected from 4,504 narrative videos spanning over 1,087 hours.Captions average 22.76 words and ROI text 8.68 words; images average 882 × 1,468 pixels. The dataset includes 1.469M UMLS entities mentioned in text (28.5K unique), images at three magnification scales, and covers 76.2% of medically related UMLS semantic types; it also covers 23.75% of non-medical semantic types.Source: §3.4 · pp. 7–8
  • QUILT’s captions span histopathology subtypes and UMLS entities, with 76.2% of represented semantic types medically related and 23.75% non-medical.Figure 4 shows distributions across sub-pathology types, UMLS semantic types, and image-text pair counts.Source: Figure 4 · p. 7, §3.4 · p. 8

4 QUILTNET: Experiments training with QUILT-1M

QUILTNET is trained by fine-tuning pretrained CLIP on QUILT-1M and evaluated on histopathology classification and cross-modal retrieval. It generally outperforms comparison models in zero-shot and linear-probing evaluations.

Results using zero-shot learning

  • QUILTNET outperforms CLIP, BiomedCLIP, and PLIP on all but two of 12 zero-shot tasks, where BiomedCLIP is marginally better.Figure 5 covers eight sub-pathologies; the two exceptions favor BiomedCLIP.Source: Figure 5 · p. 8, §4 · p. 9

Training setup

  • QUILTNET is fine-tuned from OpenAI-pretrained CLIP on QUILT-1M with a contrastive InfoNCE objective that brings aligned image-text embeddings together and separates unaligned pairs.The evaluated configurations use ViT-B/32 or ViT-B/16 image encoders and GPT2 with context length 77 or PubmedBert text encoders; the study also compares fine-tuning with training from scratch.Source: §4 · p. 8

Downstream histopathology datasets

  • The downstream evaluation spans 13 histopathology datasets covering varied cancer types, tissue classes, and patch-level classification tasks.The listed datasets include PatchCamelyon, NCT-CRC-HE-100K, SICAPv2, Databiox, BACH, Osteo, RenalCell, SkinCancer, MHIST, and lung and colon subsets of LC25000.Source: §4 · p. 8

Results using linear probing

  • With ViT-B/32 and GPT/77, QUILTNET outperforms BiomedCLIP, PLIP, and CLIP on most linear-probing datasets and exceeds fully supervised performance on NCT-CRC and SICAPv2 using 1% of labels.Evaluations use 1%, 10%, and 100% of training data sampled with three seeds; some 10% results exceed 100% because equal-per-class sampling can produce a more balanced subset on imbalanced datasets.Source: Table 1 · p. 9, §4 · p. 9

Results using cross-modal retrieval

  • Cross-modal retrieval evaluates zero-shot text-to-image and image-to-text nearest-neighbor matching on a QUILT-1M holdout set and ARCH, checking whether pairs rank within the top 1, 50, or 200.The supplied passage describes the evaluation but does not report retrieval results.Source: §4 · p. 9

5 Discussion

The discussion concludes that QUILT-1M supports strong histopathology vision-language performance while noting limitations in automated curation and the dataset’s demographic and linguistic coverage.

Limitations

  • Handcrafted algorithms and LLMs introduce curation biases and errors, including about 5% false positives from the histopathology classifier and occasional ASR medical-term substitutions.The authors say these errors are not rectifiable with the current pipeline; training CLIP from scratch underperformed fine-tuning a pre-trained CLIP, suggesting one million image-text pairs may be insufficient and motivating exploration of other self-supervised objectives.Source: §5 · pp. 9–10

Data Collection and Societal Biases

  • QUILT uses public videos and provides only video IDs so users can opt out, but its western-institution narrators and English-only focus may disadvantage other cultural or linguistic groups.The authors describe these steps as limiting privacy and consent harms and note that QUILTNET may perform better on data associated with the narrator demographics.Source: §5 · p. 10

Conclusion

  • QUILT-1M pre-training outperformed larger state-of-the-art models such as BiomedCLIP across sub-pathologies and tasks, establishing new state-of-the-art results in zero-shot, linear probing, and cross-modal retrieval.The conclusion also names few-shot and full-shot among the tasks where pre-training with QUILT is valuable.Source: §5 · p. 10

Supplementary material

The supplementary material covers QUILT data curation, collected-data analysis, pretraining and downstream evaluation, trained-model representations, and a dataset datasheet.

Section A

Section B

Section C

Section D

Section E

A Data curation models, algorithms and parsing pipelines … A.3 Histopathology and Magnification classifier

The appendix describes a multi-stage pipeline for curating histopathology video pairs and adding PubMed, LAION, and Twitter data, alongside histopathology and magnification classifiers.

A.1 Curating QUILT: an Overview

  • Keyword searches yielded ≈65K candidate videos, frame-based filtering narrowed them to ≈9K, and narrative-style screening selected ≈4K videos.The pipeline also filters non-voiced videos and checks video language through YouTube’s API or Whisper [50].Source: §A.1 · p. 16
  • Video-length-scaled keyframe thresholds and a histopathology classifier identify relevant frames; CLIP similarity ≥0.9 across three successive keyframes marks narrative streaks.For representative frames, the pipeline uses the median of stable frames or SSIM to select dissimilar frames when no stable interval is found; image-text alignment algorithms reduce duplicates and map pairs.Source: §A.1 · pp. 16–19
  • Whisper [50] transcripts are corrected with UMLS and GPT-3.5 because general-purpose ASR can misrecognize medical terms, while correction can introduce parsing errors.The passage notes the lack of open medical ASR training resources and limits of entity recognition, glossary search, and longest-common-substring correction for identifying errors.Source: §A.1 · p. 17

A.2.1 PubMed Open Access Articles

  • PubMed searches produced 109,518 unique articles and 162,307 images with captions, from which the pipeline extracted 59,371 histopathology image-caption pairs averaging 54.02 tokens.The search covered 2010–2022 and used pathology-related keywords, including 148 glossary terms.Source: §A.2.1 · pp. 19–20

A.2.2 Histopathology Image Retrieval from LAION

  • Retrieval from LAION-5B, followed by duplicate-image and non-English-pair removal, yielded 22,682 image-text pairs.The process retrieved the 3,000 most similar samples for each of 1,000 PubMed and QUILT pairs using image and text embeddings with cosine similarity.Source: §A.2.2 · p. 20

A.2.3 Twitter Data from OpenPath

  • The Twitter source comprised 55,000 unique tweets and 133,511 image-text pairs, with a one-to-many image-text relation; preprocessing followed OpenPath.The passage distinguishes this image-side one-to-many relation from OpenPath’s one-image-to-multiple-captions setup.Source: §A.2.3 · p. 20

A.3 Histopathology and Magnification classifier

  • An ensemble of two Conv-NeXt models and a DINO-feature linear classifier [11] identifies histopathology images, while a fine-tuned ConvNeXt-Tiny predicts three magnification groups.The histopathology ensemble covers IHC stains and tissue types; magnification labels group 1.25x–10x, 20x, and 40x patches, with training data spanning diverse morphologies and cancer types.Source: §A.3 · pp. 20–21

A.4 Support Models, Ontology Databases and Algorithms · A.5 Privacy preserving steps

A.4 describes ontology-guided LLM support and algorithms for classifying text and processing figures; A.5 outlines measures intended to protect privacy when handling and releasing the dataset.

A.4 Support Models, Ontology Databases and Algorithms

  • Histopathology-specific ontologies condition the LLM to identify incorrect words, while all ontologies with UMLS provide terms for validating its output.The listed ontologies include OCHV, FMA, BCGO 12, NCIT, MPATH, HPATH, and CMPO.Source: §A.4 · pp. 21–22
  • The pipeline searches PubMed open access using filters, downloads papers and media, pairs figures with captions, and processes multi-plane figures and histopathology images.Figure 11 depicts the workflow through its final result.Source: Figure 11 · p. 22
  • An LLM is prompted with 18 sub-pathology types to return the top three classifications for a given text.Figure 12 shows the few-shot prompting approach.Source: §A.4 · p. 22, Figure 12 · p. 23
  • Sobel filters, binary thresholding, size and ratio thresholds, and a histopathology classifier identify and retain individual histopathology sub-figures from multi-plane figures.Figures with more than five sub-figures are preserved whole to maintain reasonable resolution; when crops fail, the original image is stored.Source: §A.4 · pp. 22–23, Figure 13 · p. 24

A.5 Privacy preserving steps

  • Videos not fully in narrative style were excluded to reduce the risk of including private or sensitive content.Source: §A.5 · p. 23
  • Only YouTube video IDs were released, alongside code for researchers to recreate the dataset, rather than releasing the complete dataset.Source: §A.5 · p. 23
  • Data were collected from large and small channels to reduce overfitting to channel types and mitigate privacy concerns tied to channel-related biases.The paper also reports using an LLM to mask or remove irrelevant or sensitive information to reduce signal-to-noise; this measure is described as a privacy step.Source: §A.5 · p. 23

B Exploratory analysis of the collected data · C Pretraining and downstream evaluation details

QUILT’s exploratory analysis describes its data statistics, including text distribution across 18 sub-pathology types and supplementary dataset statistics.

B Exploratory analysis of the collected data

C.1 External Evaluation Datasets

The external evaluation collection spans histopathology datasets across multiple organs, with varied image and slide units, scales, and diagnostic or tissue-label schemes.

C.1 External Evaluation Datasets

  • PatchCamelyon provides 327,680 images for binary metastasis classification; NCT-CRC-HE-100K has 100,000 colorectal patches labeled cancer/normal; SICAPv2 has 10,340 prostate patches with four Gleason categories.PatchCamelyon images are 96×96px; NCT-CRC-HE-100K patches are 224×224px; SICAPv2 also includes global Gleason scores.Source: §C.1 · p. 24
  • Databiox covers 922 invasive ductal carcinoma cases; BACH [4] includes 400 breast-tissue WSIs; Osteo provides 1,144 osteosarcoma patches labeled viable, non-tumor, or necrotic tumor.Databiox samples span four magnifications, and BACH labels are normal, benign, in-situ, and invasive carcinoma.Source: §C.1 · pp. 24–25
  • RenalCell contains 27,849 clear-cell renal carcinoma images across five tissue textures; SkinCancer has 36,890 patches across 16 categories; MHIST has 3,152 colorectal-polyp patches with two labels.MHIST distinguishes hyperplastic polyps from sessile serrated adenomas.Source: §C.1 · p. 25
  • LC25000 [9] is split into 15,000 lung images labeled as adenocarcinoma, squamous cell carcinoma, or benign tissue and 10,000 colon images labeled as adenocarcinoma or benign tissue.Both subsets contain color images sized 768×768px.Source: §C.1 · p. 25

C.2 QUILTNET Implementation · D Exploration of trained model representations

QUILTNET implementation used OpenCLIP-based contrastive image-text training with specified hardware and training techniques; representation exploration presents attention-map comparisons and UMAP visualizations of image embeddings across datasets.

C.2 QUILTNET Implementation

  • Training used OpenCLIP, PyTorch, up to 4 NVIDIA A40 GPUs, gradient checkpointing, and bfloat16 automatic mixed precision; Table 8 lists training hyperparameters.The passage also states that the hyperparameters cover finetuning and training from scratch.Source: §C.2 · p. 26, Table 8 · p. 26
  • Models used 224-pixel images; finetuned ViT-B-32 inputs were resized to 512 then cropped to 224, with stretched image-text pairing, while other models used unstretched sampling.For unstretched models, medical-text sampling used probability p = sample prob or ROI-text sampling; sampling within either text type was uniform.Source: §C.2 · p. 26
  • Finetuned GPT/77 models used OpenAI CLIP [49] initialization, ViT-32 retained QuickGeLU13, and linear-probing performance was optimized by tuning learning rate, epochs, and weight decay.Source: §C.2 · p. 26
  • Table 9 reports zero-shot image-classification accuracy and compares text-encoder performance after replacing GPT2 with PubmedBert (PMB).The caption defines SkinTumor as the Neoplastic Subset of SkinCancer and PMB as PubmedBert, pretrained on PMC-15M with a 256-token context length.Source: Table 9 · p. 26
  • Zero-shot classification used the same four prompt templates for each dataset, including “a histopathology slide showing {c}” and “presence of {c} tissue on image.”The other templates were “histopathology image of {c}” and “pathology tissue showing {c}.”Source: Table 10 · p. 27

D Exploration of trained model representations

  • Figure 14 compares QUILTNET and CLIP attention maps generated with GradCAM [56], highlighting corresponding words according to their importance.Source: Figure 14 · p. 28
  • Table 11 presents a UMAP visualization of QUILTNET image embeddings from the datasets listed in Table 7.Source: Table 11 · p. 29

E Datasheet for QUILT

QUILT-1M was created to support histopathology multimodal learning, combining image-text pairs with descriptive captions and auxiliary metadata. Its datasheet also documents how the data was collected and processed, its limitations, and plans for distribution and maintenance.

Data composition

  • QUILT-1M contains more than 1 million image-text pairs intended for histopathology multimodal training, with captions, magnification estimates, UMLS entities, and subpathology metadata.Instances represent video chunks where histopathology content is stable; captions and auxiliary metadata also serve as labels or targets, and the data may support representation or supervised learning.Source: E Datasheet for QUILT · p. 29

Limitations and scope

  • The dataset is not exhaustive, has no recommended evaluation splits, and can contain noisy ASR; non-video records may lack auxiliary metadata, and videos themselves will not be released.The authors plan to distribute derived data and YouTube links so users can request video deletion.Source: E Datasheet for QUILT · p. 30

Collection and preprocessing

  • Collection used a probabilistic strategy to gather public YouTube data from January to May 2023, then applied LLM- and UMLS-based transcript denoising and classifiers to select video segments and images.The raw data was saved but is not planned for release due to copyright and privacy concerns; the authors plan to make preprocessing code public.Source: E Datasheet for QUILT · p. 30

Utility and limitations

  • The curation supports pretraining across histopathology subdomains, but YouTube biases and model inaccuracies, including ASR errors, remain limitations.Source: E Datasheet for QUILT · p. 31

Distribution and maintenance

  • The authors plan to distribute derived data and video links under the MIT license, release it as soon as possible for research use, and do not plan updates.The first authors will maintain the dataset; they will not release the videos themselves and report no fees or access restrictions.Source: E Datasheet for QUILT · p. 31

Legal and ethical considerations

  • No formal ethical review was conducted because the work did not involve human subjects, but the dataset can include deidentified patients and identifiable pathologists appearing in public videos.The authors report internal deliberation about scraping and note that videos were publicly available under YouTube’s Terms of Service.Source: E Datasheet for QUILT · p. 31

Related papers

Loading 2306.11207v4…