PaperlayerView exact revision on arXiv

Source-linked AI summary

PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology

Yuxuan Sun, Hao Wu, Chenglu Zhu, Sunyi Zheng, Qizi Chen, Kai Zhang, Yunlong Zhang, Dan Wan, Xiaoxiao Lan, Mengyue Zheng, Jingxiong Li, Xinheng Lyu, Tao Lin, Lin Yang

arXiv:2401.16355v3cs.CV

TL;DR

Pathology LMM evaluation is hindered by a scarcity of high-quality specialized benchmarks. PathMMU combines diverse image-question data with expert review, and evaluations show that current models still lag behind pathologists.

  • Problem

    Pathology benchmarks are scarce, and PathVQA's limited textbook sources, caption-based question generation, and lack of expert validation constrain question diversity and reliability.

    Source: §1 · pp. 2–3

  • Method

    PathMMU combines diverse pathology image sources with GPT-4V-generated descriptions and questions, multi-LLM filtering, and review by seven pathologists.

    Source: §1 · p. 3, §3.2 · p. 7

  • Results

    GPT-4V achieved 49.8% accuracy, while advanced LMMs lagged behind pathologists; fine-tuned InstructBLIP variants improved 21.5% and 21.3% but remained below experts.

    Source: §1 · p. 3, §4.4 · p. 14, §5 · p. 14

  • Takeaways & Limitations

    The authors identify stronger visual-text integration and interpretable models as priorities for future pathology LMMs.

    Source: §5 · pp. 14–15

    PathMMU does not include multi-image processing because most current LMMs do not support multi-image inputs.

    Source: §5 · p. 15

Abstract

from arXiv · show

The emergence of large multimodal models has unlocked remarkable potential in AI, particularly in pathology. However, the lack of specialized, high-quality benchmark impeded their development and precise evaluation. To address this, we introduce PathMMU, the largest and highest-quality expert-validated pathology benchmark for Large Multimodal Models (LMMs). It comprises 33,428 multimodal multi-choice questions and 24,067 images from various sources, each accompanied by an explanation for the correct answer. The construction of PathMMU harnesses GPT-4V's advanced capabilities, utilizing over 30,000 image-caption pairs to enrich captions and generate corresponding Q&As in a cascading process. Significantly, to maximize PathMMU's authority, we invite seven pathologists to scrutinize each question under strict standards in PathMMU's validation and test sets, while simultaneously setting an expert-level performance benchmark for PathMMU. We conduct extensive evaluations, including zero-shot assessments of 14 open-sourced and 4 closed-sourced LMMs and their robustness to image corruption. We also fine-tune representative LMMs to assess their adaptability to PathMMU. The empirical findings indicate that advanced LMMs struggle with the challenging PathMMU benchmark, with the top-performing LMM, GPT-4V, achieving only a 49.8% zero-shot performance, significantly lower than the 71.8% demonstrated by human pathologists. After fine-tuning, significantly smaller open-sourced LMMs can outperform GPT-4V but still fall short of the expertise shown by pathologists. We hope that the PathMMU will offer valuable insights and foster the development of more specialized, next-generation LMMs for pathology.

1 Introduction

Pathology LMMs need rigorous evaluation, but existing pathology benchmarks have limitations in source diversity, question quality, and expert validation. PathMMU addresses this gap, and its evaluations reveal substantial distance between current models and pathologists.

1 Introduction

  • Pathology's diagnostic and treatment roles make precise image interpretation important, while general-purpose LMMs aim to extend beyond task-specific pathology models.The text describes applications including cervical cancer screening and biomarker assessment.Source: §1 · p. 2
  • PathVQA's textbook-only sources, caption-derived questions, and lack of expert review can yield limited, noisy questions that hinder evaluation of pathology LMMs.Captions may include details absent from images or cover only part of their content.Source: §1 · pp. 2–3
  • PathMMU combines 33,428 multimodal questions and 24,067 images from diverse sources with GPT-4V-generated descriptions and questions, multi-LLM filtering, and review by seven pathologists.Each question includes a detailed answer explanation.Source: §1 · p. 3
  • 49.8% accuracy is the best zero-shot result, achieved by GPT-4V, while the gap to professional pathologists is nearly 20%.Fifteen of the 18 evaluated LMMs achieved no more than 40% accuracy.Source: §1 · p. 3
  • LMMs show apparent robustness to image corruption, but limited performance and reliance on superficial features constrain what those tests establish about visual reasoning.The paper notes models may sometimes not use image information in reasoning.Source: §1 · p. 3
  • Fine-tuning general-purpose LMMs with substantial data significantly boosts pathological-image comprehension and reasoning, enabling them to easily surpass zero-shot GPT-4V, although a discernible performance gap from human experts remains.Source: §1 · p. 4
  • GPT-4 Turbo can use answer-pattern shortcuts, such as choosing common pathological options or distinctive choices, rather than analyzing the image deeply.The authors report that text-only GPT-4 Turbo surpasses the top-performing open-source LMMs on PathMMU.Source: §1 · p. 4

2 Related Works

Prior multimodal benchmarks assess general capabilities, while medical image-question datasets have chiefly focused on radiology. Pathology datasets exist, but their sources and question-generation approaches leave gaps that PathMMU is designed to address.

General multimodal benchmarks

  • General multimodal benchmarks evaluate broad perception, understanding, reasoning, and hallucination, including datasets spanning university-level subjects.Examples named include LAMM, MMBench, MMMU, HaELM, and HallusionBench.Source: §2 · p. 4

Medical benchmarks

  • Medical VQA datasets include radiology resources VQA-RAD and VQA-Med, while PMC-VQA generates questions from text-only image captions.The passage describes these as resources for evaluating AI capabilities in radiological diagnostics.Source: §2 · p. 4

Pathology benchmarks

  • PathVQA uses textbook captions to generate over 30,000 Q&As from 4,998 images, while Quilt-VQA has 1,283 Q&As and 985 images from teacher narrations.Caption-based generation can produce inaccurate, text-solvable, simplistic, or erroneous questions.Source: §2 · p. 5
  • PathMMU differentiates itself with diverse image-caption sources, rigorous expert review and filtering, and explanations for each answer.Its comparison with existing benchmarks emphasizes comprehensive sourcing and multitiered filtering.Source: §2 · p. 5, Figure 2 · p. 5

3 The Proposed PathMMU Benchmark

PathMMU is designed as a large, specialized pathology benchmark whose questions require image observation and include explanations. Its construction combines diverse data collection, description and question generation, automated filtering, and expert review.

3.1 Design Principle

  • PathMMU's design prioritizes specialized authoritative sources, image-dependent examination-style questions, scale, clear images, and explanations for answer choices.The dataset's average image resolution is approximately 900 × 700 pixels, and seven experts manually review it.Source: §3.1 · p. 6

3.2 The Construction of PathMMU

  • The collection stage integrates PubMed, educational videos, atlases, expert social-media posts, and classification datasets, yielding approximately 30,000 image-text pairs.Pathological regions are identified using trained ConvNeXt and YOLOv6-based models for PubMed and video sources, respectively.Source: §3.2 · pp. 6–7, Figure 3 · p. 7

Step 2: Detailed Description Generation and Question-answer Pairs

  • GPT-4V uses original captions as reference while generating detailed descriptions focused on cell and tissue morphology.The reference captions help reduce incoherent descriptions when image descriptions are generated.Source: Step 2: Detailed Description… · p. 7
  • GPT-4V generates three explained multiple-choice questions per image, and questions that at least three of four LLMs can guess correctly are excluded.The filtering is intended to prevent questions from being answerable through educated guesses without visual cues.Source: Step 2: Detailed Description… · p. 8
  • Seven pathologists review validation and test questions for image dependence, answerability, answer correctness, and exam-style plausibility before invalid items are removed.The final validation set has 710 Q&As and the test set has 9,677; two pathologist groups establish an expert-performance reference on 1,156 test-tiny questions.Source: Step 2: Detailed Description… · p. 8

4 Experiments

Experiments compare zero-shot LMMs and text-only LLMs, test robustness to image corruption and answer guessing, and evaluate fine-tuning on PathMMU. Across these tests, models show a substantial gap from expert performance, while fine-tuning improves results but explanation generation does not.

4.1 Zero-shot Evaluation of LMMs and LLMs

  • 71.8% human-expert accuracy exceeds GPT-4V’s 49.8% and LLaVA-1.5-13B’s 37.6%; 15 of 18 tested models score below 40%.The results indicate a substantial gap between current LMMs and professional pathologist standards.Source: Table 1 · p. 9, §4.1 · p. 10

4.2 Robustness of LMMs to Image Corruption

  • With Gaussian-noise images, LMM performance drops only 1.0%–4.5% and remains above random choice, suggesting predictions may rely on textual shortcuts.Performance drops across different sizes of the same model are remarkably similar.Source: Table 3 · p. 11, §4.2 · p. 12

4.1 Zero-shot Evaluation of LMMs and LLMs

  • 38.1% accuracy lets text-only GPT-4 Turbo outperform the best open-source LMM, LLaVA-1.5-13B, and exceed the second-best LLM by 6.5%.LLaVA-1.5-13B exceeds LLaVA-1.5-7B by 2.2%; InstructBLIP-FLAN-T5-XXL and BLIP-2 FLAN-T5 XXL exceed their smaller versions by 2.1% and 1.7%, respectively.Source: §4.1 · pp. 10–11

4.2 Robustness of LMMs to Image Corruption

  • LMMs appear robust across tested corruption types and levels, but the authors suggest subtle pathology details may be missed and shortcuts may be used.Qwen-VL-7B’s overall performance increases by 0.6% under corruption compared with its baseline.Source: Figure 4 · p. 10, Table 2 · p. 10, §4.2 · pp. 11–12

4.3 Analysis of LLMs’ Ability to Guess Answers

  • On 100 filtered, guessable samples, closed-source LLMs reached approximately 90%, outperforming image-enabled pathologists and slightly exceeding Vicuna-v1.5-13B.The authors interpret this as evidence that advanced LLMs can exploit question shortcuts.Source: Figure 5 · p. 12, §4.3 · p. 13
  • On 1,600 mismatched question-option samples, LLMs still answered about 50% correctly, while BERT and BiomedBERT also beat random guessing through question-option matching.BiomedBERT-large performs notably better than BERT, which the authors associate with biomedical pretraining.Source: Figure 5 · p. 12, §4.3 · p. 13

4.4 Fine-tuning Results

  • 21.5% and 21.3% gains let direct-answer fine-tuning of InstructBLIP-FLAN-T5-XL and XXL surpass GPT-4V, though both remain below expert performance.The gains are reported for the XL and XXL models, respectively.Source: Table 4 · p. 14, §4.4 · p. 14
  • Answer-plus-explanation fine-tuning reduced performance by 0.3% and 2.3% versus direct answers; the authors speculate joint generation may impede correct answering.The paper raises the question of how to leverage PathMMU’s interpretability information in training.Source: §4.4 · p. 14, Table 4 · p. 14

5 Conclusion

PathMMU is presented as an expert-reviewed pathology benchmark with human performance references, and the authors report that current LMMs still trail pathologists despite fine-tuning gains. The conclusion identifies visual grounding, shortcut behavior, and multi-image processing as areas needing further work.

5 Conclusion

  • Fine-tuned LMMs improve on PathMMU and can surpass GPT-4V, but remain below human expert performance.The authors also report that advanced LMMs lag in observing image details and sometimes neglect visual information.Source: §5 · p. 14
  • PathMMU combines data curation with strict manual review by seven pathologists and establishes a human expert performance benchmark.Source: §5 · p. 14
  • Current multimodal models may over-rely on text and neglect visual data, motivating work on better visual-text integration and interpretable models.Source: §5 · pp. 14–15
  • PathMMU does not benchmark multi-image processing, although pathologists often assess samples at varied magnifications and from different perspectives.The authors link this scope boundary to most current LMMs not supporting multi-image inputs.Source: §5 · p. 15

A.1 Statistical Distribution

The statistical overview emphasizes dataset scale and diversity, image clarity, and pathology-focused language in the refined descriptions, questions, and answer options.

A.1 Statistical Distribution

  • PathMMU pairs a substantial volume of Q&As with many images from diverse sources, using a low question-to-image ratio to broaden knowledge coverage.The authors argue that more images and sources broaden the knowledge spectrum when the question count is fixed.Source: §A.1 · p. 20
  • Images are primarily high-resolution single figures, and each undergoes manual clarity review.YouTube educational images typically have higher resolution because they are screenshots from screen-sharing sessions.Source: §A.1 · p. 20
  • GPT-4V caption refinement reduces irrelevant terms and shifts descriptions toward pathology terms such as “cell” and “tissue.”The questions emphasize tissue characteristics and diagnostic procedures, while answer options focus on morphology and diagnostic terms.Source: §A.1 · p. 20

A.2 Overview of the Composition of the PathCLS Subset

PathCLS draws on pathology classification datasets spanning tissue types, organs, and diagnostic tasks. The cited statistics tables cover the PathMMU train and test subsets, but the supplied captions do not state their values.

A.2 Overview of the Composition of the PathCLS Subset

  • PathCLS gathers data from ten distinct pathology classification datasets, including CRC100K, WSSS4LUAD [17], LC25000 [7], and PatchCamelyon.These sources cover colorectal tissue categories, tumor-normal distinctions, lung and colon adenocarcinomas, and lymph-node metastasis labels.Source: §A.2 · pp. 20–21
  • Table 5 and Table 6 present subset statistics for the PathMMU train and test sets, respectively.The supplied captions identify their scope but provide no statistical values.Source: Table 5 · p. 21, Table 6 · p. 21
  • Other sources include prostate images graded by Gleason categories in SICAPv2, breast histology categories in BACH, and tissue-region classes in Osteo.Source: §A.2 · p. 21
  • The listed sources also include skin-biopsy compartments and neoplasms in SkinCancer, and colorectal-polyp tissue patches in MHIST.Source: §A.2 · pp. 21–22

B.1 Comparison of GPT-4 and Vicuna in Performing Text-only Educated Guess

In text-only trials, GPT-4 acknowledges guessing while Vicuna falsely claims visual access. The authors hypothesize that common or frequent answer options influence both models’ guesses.

B.1 Comparison of GPT-4 and Vicuna in Performing Text-only Educated Guess

  • Without images, GPT-4 acknowledges that it guesses from typical occurrences in histology, whereas Vicuna fabricates visual observations and claims image access.Source: §B.1 · p. 22
  • The authors hypothesize that both models’ guesses are influenced by option commonality or frequency in their training data.Source: §B.1 · p. 22
  • The contrast is in how the models present their guesses: GPT-4 admits lacking visual interpretation, while Vicuna responds as if it can see the images.Source: §B.1 · p. 22

B.2 LMMs’ Performance Degradation towards Different Levels of Image Corruption

On test-tiny, increasing image corruption from levels 0–5 produced no significant downward trend in model performance; the case-study page provides examples of generated descriptions and GPT-4V outputs.

B.2 LMMs’ Performance Degradation towards Different Levels of Image Corruption

  • Across corruption levels 0–5, LMM performance shows no significant downward trend, suggesting minimal impact on PathMMU test-tiny scores.Fig. 7 visualizes performance across corruption levels; level 0 is the uncorrupted-image baseline.Source: §B.2 · pp. 22–23, Figure 7 · p. 23
  • Fig. 7 tracks model performance by corruption level on test-tiny, with level 0 denoting images without corruption.Source: Figure 7 · p. 23

C Case Study

  • The project page provides cases of image-description generation, invalid examples, and GPT-4V errors and correct answers.Source: C Case Study · p. 23

D Limitations

The authors note that expert judgments can vary and affect dataset curation, while the expert benchmark does not represent an upper limit across pathology expertise.

D Limitations

  • Pathologist judgments vary with diagnostic-threshold interpretation, emotional or physical state, and subspecialty experience.Source: D Limitations · p. 23
  • 90% agreement on 100 images still left validity disagreements, with each pathologist accepting some samples the other deemed invalid.Table 7 reports the cross-assessment of the same 100 samples.Source: Table 7 · p. 24, D Limitations · p. 24
  • The expert benchmark is not a dataset upper bound because even comprehensive doctors cannot cover all pathology expertise.Source: D Limitations · p. 24
Loading 2401.16355v3…