PaperlayerView exact revision on arXiv

Source-linked AI summary

In-context learning enables multimodal large language models to classify cancer pathology images

Dyke Ferber, Georg Wölflein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S. M. El Nahhas, Gustav Müller-Franzes, Dirk Jäger, Daniel Truhn, Jakob Nikolas Kather

arXiv:2403.07407v1cs.CV

TL;DR

Medical image classification generally requires computationally and technically demanding task-specific training, while in-context learning remains underexplored in medical image analysis. This study benchmarks GPT-4V with image-based in-context learning across three histopathology tasks and compares it with dedicated image classifiers. GPT-4V matched or exceeded specialized models using few examples, supporting generalist vision-language models for histopathology where annotated data are scarce.

  • Problem

    Medical image classification requires labeled, task-specific datasets and demanding model training, while in-context learning remains underexplored in medical image analysis.

  • Method

    The study benchmarks GPT-4V with zero-shot, random few-shot, and kNN-based image sampling across three histopathology datasets, comparing it with specialized classifiers.

  • Results

    GPT-4V achieved 83.3% accuracy on MHIST versus 82.7% for Virchow and 88.3% on PatchCamelyon versus 92.7% for Virchow.

  • Takeaways & Limitations

    Large vision-language models trained on non-domain-specific data can be applied out of the box to histopathology image-processing tasks, potentially broadening access where annotated data are scarce.

  • Takeaways & Limitations

    Experiments were restricted to a small sample size because the GPT-4V preview API permitted only a limited number of requests, and the study did not include ensembling methods.

Abstract

from arXiv · show

Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce.

Introduction

Medical histopathology models typically require substantial domain-specific data and task-specific training, while text-only in-context learning does not address multimodal medical information. This study evaluates whether GPT-4V can extend in-context learning to histopathology images and perform comparably to dedicated classifiers.

  • Motivation: Domain-specific foundation models require substantial image data and are restricted to vision applications.These requirements constrain their use in settings with limited annotated data and technical resources.
  • Motivation: Text-based in-context learning uses a few handcrafted examples supplied alongside a prompt to improve model performance.The approach is also known as few-shot prompting.
  • Motivation: Vision language models address the limitation that medical understanding depends on both textual and visual information.The study situates GPT-4V among recently developed multimodal models.
  • Study aim: The study benchmarks GPT-4V with in-context learning against dedicated image classifiers across three histopathology datasets.The evaluated tasks cover tissue classification, colon polyp subtyping, and breast tumor detection in lymph node sections.
  • Study aim: GPT-4V tissue-classification performance improved with in-context learning and was on par with specialist computer vision models.The comparison was made across three histopathology benchmarking datasets.

Methods

The study evaluates GPT-4V on three public histopathology datasets using zero-shot, random few-shot, and kNN-based few-shot prompting, alongside transfer-learned image classifiers. Similarity-based example selection uses Phikon image embeddings, while comparisons use balanced test samples and matched training data.

  • Datasets: The experiments use public CRC100K, PatchCamelyon, and MHIST histopathology datasets.The datasets cover colorectal tissue subtypes, metastatic breast cancer in lymph nodes, and colorectal polyp subtypes.
  • Datasets: PatchCamelyon contains 327,680 lymph-node image patches with binary labels for metastatic breast cancer tissue.The labels are close to a 50/50 class balance.
  • Datasets: MHIST contains 3,152 colorectal-polyp sections labeled as hyperplastic polyps or sessile serrated adenomas by seven-pathologist majority vote.The dataset has a 3:7 class split.
  • In-context learning: Evaluation uses zero-shot, random few-shot, and kNN-based few-shot sampling, with Phikon embeddings and cosine similarity selecting neighboring examples.The closest images for each label are shown first in the prompt, and same-patient embeddings are removed when available.
  • Comparative models: GPT-4V is compared with ResNet-18, ResNet-50, ViT-Tiny, and ViT-Small classifiers initialized from ImageNet weights.The classifiers are trained for ten epochs with Adam, a 0.001 learning rate, and cross-entropy loss.
  • Evaluation design: GPT-4V inference uses balanced test sets of 60 MHIST samples, 60 PatchCamelyon samples, and 120 CRC100K samples.MHIST test images were restricted to cases with unanimous expert consensus.
  • Prompting: Prompt development used ten random image tiles per label from each dataset and included hypothetical scenarios to address GPT-4V refusals of medical-data handling.The prompts also specified desired and undesired responses and structured outputs as JSON.

Results

Few-shot in-context learning substantially improves GPT-4V’s histopathology classification, with kNN-selected examples often outperforming random sampling and matching or exceeding specialist classifiers on some tasks. Gains also extend to multilabel CRC100K classification and produce more separated text-embedding clusters aligned with answers and ground truth.

  • 83.4% and 88.3% accuracy were achieved with ten-shot kNN sampling on MHIST and PatchCamelyon, respectively.These results concerned sessile-serrated adenoma versus hyperplastic polyps and breast-cancer lymph-node metastases versus tumor-free tissue.
  • Three- and five-shot prompting on PatchCamelyon outperformed all other models under the matched comparison setting.Ten-shot GPT-4V was also reported superior to two ResNet-style models and two vision transformers under equitable conditions.
  • Few-shot prompting improved difficult multilabel CRC100K classification across most labels, although GPT-4V remained below specialist-model performance.The best result used five-shot kNN sampling with 40 sample images, while zero-shot predictions showed systematic confusions among several tissue classes.
  • Few-shot sampling produced more distinct text-embedding clusters aligned with GPT-4V answers and ground-truth labels.The analysis used Ada-002 embeddings and t-SNE to examine changes from zero-shot through five-shot kNN sampling.
  • GPT-4V’s reasoning exposed multimodal evidence in challenging cases, including morphological signs and image-embedding neighborhoods that supported tumor interpretations.One illustrated case was labeled stroma by ground truth but showed tumor-like reasoning and a tumor-dominant embedding neighborhood.

Discussion

The study presents GPT-4V in-context learning as a proof of concept for data-efficient, multimodal medical image classification. It reports comparable performance to conventional classifiers, while noting sample-size, ensembling, class-specific, and embedding-retrieval limitations.

  • GPT-4V in-context learning classified microscopy images with performance comparable to conventional image classifiers trained on the same amount of data.The authors frame this as a proof of concept for adapting vision-language models without traditional retraining.
  • 82.7% versus 83.3% accuracy was reported for Virchow and GPT-4V on MHIST, while PatchCamelyon showed 92.7% versus 88.3%.The MHIST comparison excluded images without full inter-rater agreement, making that use case likely easier than the comparator study.
  • The experiments used a small sample size because the preview GPT-4V API permitted only limited requests.The same sample images were used for GPT-4V and the study’s image classifiers to maintain equitable comparisons.
  • The study did not include ensembling methods and observed suboptimal in-context-learning results for some CRC100K classes, including debris, mucus, and stroma.The authors leave investigation of these class-specific outcomes and possible solutions for future work.
  • The authors report no saturation point as the number of k-shot examples increased, suggesting that scaling the approach may yield further improvements.They propose extending the work to overcome current limitations and to evaluate other domains such as radiology imaging.

Funding

The paper lists institutional, governmental, charitable, and industry-related support, alongside author disclosures and a statement that some views are not necessarily those of the NHS, NIHR, or Department of Health and Social Care.

  • JNK received support from German government ministries, the German Cancer Aid, the German Academic Exchange Service, the German Federal Joint Committee, and ERA-NET Transcan projects.
  • The authors state that the expressed views are not necessarily those of the NHS, NIHR, or Department of Health and Social Care, and disclose no other funding.
  • The disclosures report shares, consulting services, lecture honoraria, and advisory-board honoraria involving multiple biomedical companies.

Supplementary Methods:

The supplementary methods define in-context learning and related prompting approaches, review medical-language-model developments, and describe conventional pathology classifiers and their data requirements.

  • In-Context Learning: In-context learning improves task performance without updating model parameters by using a small set of similar question-answer pairs in the prompt.Prompt engineering selects effective combinations of prompts and example solutions.
  • In-Context Learning: Chain-of-Thought prompting decomposes complex problems into smaller components and guides the model to build a solution strategy sequentially.
  • In-Context Learning: Self-consistency CoT aggregates multiple responses and selects a majority vote, while Tree-of-Thoughts prompting explores different reasoning paths.
  • In-Context Learning: Prompting methods can require expert knowledge and labor-intensive manual crafting, producing specialized prompts with limited cross-domain generalizability.The specialization may still be advantageous when prompts are tailored to individual patient cases.
  • Medical Language Models: Med-PaLM 2 and MedPrompt combine instruction finetuning or prompt engineering with multiple reasoning paths, ensemble refinement, and context retrieval.MedPrompt uses text embeddings to select context closely aligned with the target case and improved on Med-PaLM 2 in MultiMedQA.
  • Computational Pathology: Computational pathology traditionally uses ImageNet-pretrained CNNs such as ResNet and Inception, but histology’s cellular patterns, color distribution, and rotational invariance create domain-specific challenges.These approaches depend on extensively annotated datasets, while models such as CTransPath, Phikon, and Virchow are emerging as strong pathology foundation models.

Appendix A

Algorithm 1 uses k-nearest-neighbor image sampling to construct label-balanced in-context examples, formats them with prompts and target images, and records the language model’s JSON predictions.

  • Algorithm 1: The method precomputes an embedding for every image-label pair and initializes result storage for the task list.Each task contains a user prompt, target image, and ground-truth label.
  • Algorithm 1: For each target and possible label, it selects the k closest embeddings using cosine similarity while excluding the target image.Selected examples are constrained to share the candidate label.
  • Algorithm 1: The selected image-label pairs are stored as the context set used for in-context learning.The algorithm initializes the closest-image set and appends selected tuples to it.
  • Algorithm 1: The input combines a system prompt, task-specific user prompt, interleaved example images and labels, and the target image.The target is presented after the retrieved examples.
  • Algorithm 1: The language model is invoked for each task, and its response is stored with the ground-truth label in the final results.The final result is specified in JSON format containing thoughts and an answer.

System Prompt for CRC100K

The CRC100K system prompt frames GPT-4V as a professional assistant for classifying microscopic H&E images and constrains its response to structured tissue predictions.

  • System Prompt: The prompt presents a hypothetical pathology-assistance scenario and asks for a JSON-only response.It explicitly instructs the model not to return text outside the JSON format.
  • System Prompt: The model is instructed to classify microscopic hematoxylin-and-eosin images, with cancer and other tissue subtypes as the main focus.The prompt describes the assistant as supporting a pathologist.
  • System Prompt: The prompt requests detailed tissue descriptions and explanations for the assigned label while prohibiting refusal-style responses.It emphasizes localization and justification in the output.
  • System Prompt: The output template requires a reasoning field, one permitted tissue label, and a confidence score between 0 and 1.Allowed labels include Cancer, Normal, Adipose, Lymphocytes, Debris, Mucus, Muscle, and Stroma.

Zero-Shot Prompt for CRC100K

The CRC100K zero-shot prompt asks GPT-4V to classify an H&E tissue image among eight tissue categories, using explicit visual distinctions and a structured confidence-bearing response.

  • Zero-Shot Prompt: The task presents a microscopic H&E tissue slide in a simulated scenario and asks for a professional classification response.The prompt says the scenario has no real-world impact.
  • Zero-Shot Prompt: The available labels are Cancer, Normal, Adipose, Lymphocytes, Debris, Mucus, Muscle, and Stroma.The prompt maps these labels to colorectal adenocarcinoma, normal colon epithelium, fat tissue, and other tissue types.
  • Zero-Shot Prompt: The instructions emphasize distinguishing cancer from debris and stroma from muscle using tissue integrity, extracellular matrix, collagen fibers, cellularity, alignment, and color patterns.They also specify that disrupted tissue should favor Debris and matrix or collagen should favor Stroma.
  • Zero-Shot Prompt: The model is told to compare the patient image with learned tissue patterns and prioritize the dominant cell type when cancer and lymphocytes coexist.The prompt specifically highlights cancer-versus-normal and cancer-versus-debris distinctions.
  • Zero-Shot Prompt: The prompt reiterates the permitted labels, confidence interpretation, and JSON structure while instructing the model not to refuse assistance.The answer must use one listed tissue term without additional text.
  • Zero-Shot Prompt: The output template separates explanatory thoughts, the categorical answer, and the numeric score.The prompt repeats that the response should not be enclosed in markdown code blocks.

Few-Shot Prompt for CRC100K

The compact prompt materials define few-shot and zero-shot pathology classification tasks, specifying tissue labels, visual decision rules, example-based comparison, and structured outputs for CRC100K and PatchCamelyon.

  • Few-Shot Prompt for CRC100K: The CRC100K output is structured as explanatory thoughts, one exact tissue label, and a numeric score, within a simulated nonclinical setting.The prompt repeatedly requests JSON formatting and discourages refusal responses.
  • Few-Shot Prompt for CRC100K: CRC100K classification uses eight labels covering colorectal cancer, normal epithelium, adipose tissue, lymphocytes, debris, mucus, muscle, and cancer-associated stroma.The prompt presents these as the available tissue options.
  • Few-Shot Prompt for CRC100K: The CRC100K few-shot prompt supplies labeled example images so the model can learn patterns distinguishing tissue types before classifying a patient image.The workflow explicitly asks the model to compare the patient image with the examples.
  • Few-Shot Prompt for CRC100K: The CRC100K instructions resolve difficult cases using tissue integrity, collagen and extracellular matrix, dominant lymphocyte fraction, mucus cellularity, alignment, and color patterns.They particularly emphasize cancer-versus-debris and stroma-versus-muscle distinctions.
  • Zero-Shot Prompt for PatchCamelyon: PatchCamelyon zero-shot instructions require stepwise inspection of the image, cancer-related histopathology reasoning, an exact binary answer, and a confidence score.The output is requested in JSON format with explanatory thoughts and no additional label text.
  • System Prompt for PatchCamelyon: The PatchCamelyon task detects metastatic breast cancer in lymph node sections using the binary labels Cancer and No Cancer.The prompt identifies the input as a microscopic H&E-stained lymph-node slide.
  • Zero-Shot Prompt for PatchCamelyon: The PatchCamelyon prompt explicitly says to answer Cancer only when cancer cells are clearly visible and otherwise answer No Cancer.It also requests a best informed guess when uncertain.

Additional References

The additional references situate the study within prompting and reasoning methods for language models, medical applications of generalist models, and self-supervised learning for pathology and vision.

  • Prompting and reasoning: Prompting research includes few-shot learning, systematic prompt methods, chain-of-thought reasoning, self-consistency, tree search, and prompt evolution.These works span foundational in-context learning and increasingly structured strategies for eliciting or improving model reasoning.
  • Generalist models in medicine: Medical applications of generalist models include expert-level question answering and comparisons between generalist foundation models and special-purpose tuning.The cited works connect large-model capabilities to medical tasks and model-selection questions.
  • Representation learning: Contrastive pre-training is also represented for text and code embeddings, extending the reference landscape beyond image and pathology applications.This work provides a related example of representation learning through contrastive objectives.
  • Computational pathology: Computational pathology references address weakly supervised prediction, histopathology classification, and self-supervised representation learning.The cited studies include colorectal cancer pathway and mutation prediction, lung cancer analysis, and pathology-focused representation learning.
  • Self-supervised learning: The references cover self-supervised learning methods including bootstrap-based, contrastive, joint-embedding, masked-image, and transformer-based approaches.These methods are represented across general computer vision and histopathological image classification.
  • Pathology foundation models: Large pathology foundation-model efforts include Virchow and a general-purpose self-supervised model for computational pathology.These references represent scaling and domain-specific foundation-model development for digital pathology.
Loading 2403.07407v1…