PaperlayerView exact revision on arXiv

Source-linked AI summary

PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology

Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Lin Sun, Zhongyi Shui, Yunlong Zhang, Honglin Li, Lin Yang

arXiv:2305.15072v2cs.CVcs.MM

TL;DR

Pathology MLLMs are constrained by scarce high-quality data and limited exploration of domain-specific model frameworks. The paper builds pathology datasets, PathCLIP, and an instruction-tuned assistant with specialized-model tools; PathAsst improves over a prior SOTA approach on closed-ended questions but slightly underperforms it on open-ended questions.

  • Problem

    Pathology lacks high-quality image-text data and comprehensive MLLM frameworks tailored to the domain.

    Source: Introduction · p. 1

  • Method

    The paper builds PathCap and PathInstruct, trains pathology-specific PathCLIP, and integrates it with Vicuna-13b in PathAsst, which can invoke eight pathology-specific sub-models.

    Source: Introduction · pp. 1–2, Pathology Dataset Construction · pp. 2–3

  • Results

    PathAsst improves on the previous SOTA model for closed-ended questions but slightly underperforms it for open-ended questions.

    Source: Experiments · p. 7

  • Takeaways & Limitations

    PathAsst showcases multimodal dialogue and pathology-image interpretation capabilities, with performance varying between closed-ended and open-ended questions.

    Source: Conclusion · p. 7, Experiments · p. 7

Abstract

from arXiv · show

As advances in large language models (LLMs) and multimodal techniques continue to mature, the development of general-purpose multimodal large language models (MLLMs) has surged, offering significant applications in interpreting natural images. However, the field of pathology has largely remained untapped, particularly in gathering high-quality data and designing comprehensive model frameworks. To bridge the gap in pathology MLLMs, we present PathAsst, a multimodal generative foundation AI assistant to revolutionize diagnostic and predictive analytics in pathology. The development of PathAsst involves three pivotal steps: data acquisition, CLIP model adaptation, and the training of PathAsst's multimodal generative capabilities. Firstly, we collect over 207K high-quality pathology image-text pairs from authoritative sources. Leveraging the advanced power of ChatGPT, we generate over 180K instruction-following samples. Furthermore, we devise additional instruction-following data specifically tailored for invoking eight pathology-specific sub-models we prepared, allowing the PathAsst to effectively collaborate with these models, enhancing its diagnostic ability. Secondly, by leveraging the collected data, we construct PathCLIP, a pathology-dedicated CLIP, to enhance PathAsst's capabilities in interpreting pathology images. Finally, we integrate PathCLIP with the Vicuna-13b and utilize pathology-specific instruction-tuning data to enhance the multimodal generation capacity of PathAsst and bolster its synergistic interactions with sub-models. The experimental results of PathAsst show the potential of harnessing AI-powered generative foundation model to improve pathology diagnosis and treatment processes.

Introduction

Pathology MLLMs remain limited by scarce high-quality data and relatively little exploration of model frameworks, even as digitized pathology datasets grow in size and complexity. The paper addresses this gap with pathology datasets, PathCLIP, and an instruction-tuned assistant that can invoke specialized models.

  • Growing pathology datasets and their complexity have increased the need for tools that improve pathologists’ analytical efficiency.Source: Introduction · p. 1
  • Pathology MLLMs face a gap in high-quality data and comprehensive model frameworks, while expanding digitized datasets increase the need for more sophisticated analytical tools.Source: Introduction · p. 1
  • 207K high-quality image-caption pairs form PathCap, and PathCLIP trained on PathCap achieves state-of-the-art pathology image retrieval and zero-shot classification results.Source: Introduction · pp. 1–2
  • PathAsst integrates PathCLIP with Vicuna-13b and uses 180K PathInstruct samples to train multimodal generation, alongside instruction data for invoking eight pathology-specific sub-models.Source: Introduction · p. 2

Related Work

Prior work spans instruction-tuned language and multimodal models, but pathology-specific multimodal systems and training datasets remain limited. The paper positions its datasets and models against gaps in domain adaptation, image-text quality, and dataset access.

  • General MLLMs use either specialized visual tools alongside an LLM or visual features integrated into LLM inputs, with instruction-following datasets supporting multimodal chat models.Source: Related Work · p. 2
  • Pathology multimodal methods have often combined vision encoders with LSTMs, while TraP-VQA applied a vision-language transformer and OpenPath used contrastive pretraining on Twitter-sourced pairs.Source: Related Work · p. 2
  • Pathology-adapted MLLMs remain underexplored; existing models often focus on caption generation and may underperform specialized professional pathology models.Source: Related Work · p. 2
  • Existing biomedical datasets are not pathology-tailored, while OpenPath’s Twitter pairs have relatively weak image-text correlation and require costly Twitter API access.Source: Related Work · p. 2

Pathology Dataset Construction

PathCap combines curated pathology image-caption pairs with expert annotations, while PathInstruct combines generated dialogue and description data with instructions for specialized-model invocation. The data pipeline filters and aligns source material, then refines captions and creates instruction-following samples.

Pathology Dataset Construction

  • PathCap contains 207K image-caption pairs: 197K from PubMed and internal pathology guideline books, plus 10K expert cytologist annotations; PathInstruct contains 180K instruction-following samples.Source: Pathology Dataset Construction · pp. 2–3
  • PathInstruct combines ChatGPT-generated detailed descriptions and conversational Q&As with model-invoking samples tailored to varied scenarios and user intent.Source: Pathology Dataset Construction · pp. 3–4
  • The collection pipeline parses PubMed papers and books into image-text pairs, then filters pathology images, separates subfigures and captions, and aligns sub-images with sub-captions.The authors report 15M samples from PubMed and 2K from books before filtering; the alignment and quality-filtering stage yields 195K high-quality pairs.Source: Pathology Dataset Construction · p. 3
  • ChatGPT refines captions to remove irrelevant information and make them more descriptive for training.Source: Pathology Dataset Construction · p. 3

PathAsst Framework Construction

PathAsst combines a pathology-trained visual encoder with Vicuna-13B for multimodal generation, trained in two phases using PathInstruct. At inference, it can retrieve paper abstracts and invoke specialized tools to formulate responses.

PathAsst Framework Construction

  • PathAsst maps PathCLIP image embeddings into the language embedding space with a fully connected layer, then concatenates visual and language embeddings as MLLM inputs.Source: Model Design and Training · p. 4
  • PathCLIP is fine-tuned from OpenAI CLIP on PathCap with contrastive learning that increases matched pathology image-text similarity and reduces similarity for non-pairs.Source: Model Design and Training · p. 4
  • PathAsst training first freezes the vision encoder and LLM while training the connecting layer, then trains with PathCLIP frozen on a 35K-sample set drawn from books and selected PubMed examples.Source: Model Design and Training · p. 4
  • Training uses a dialogue format with user instructions and assistant responses, and calculates loss only on the assistant responses.Source: Table 1 · p. 5
  • After invoking a specialized model, PathAsst combines the user query with that model’s output to produce a conclusive response.Source: Model Design and Training · p. 5
  • A retrieval system embeds 5.3M PubMed abstracts and supplies relevant information as context for user queries.Source: Model Design and Training · p. 5

Experiments

Experiments assess PathCLIP on image retrieval and zero-shot classification, and PathAsst on pathology question answering, image interpretation, and tool-assisted tasks.

Experiments

  • PathCLIP’s zero-shot classification evaluation covers CRC100K, WSSS4LUAD, LC-lung, and LC-colon; retrieval uses PubMed and books, while PathAsst is evaluated on PathVQA.PathVQA performance is measured with accuracy for closed-ended questions and F1-score for open-ended questions.Source: Experiments · pp. 5–6
  • 10.71-fold/11.07-fold: PathCLIP’s PubMed R@10 exceeds OpenAI CLIP/PLIP; on books, its R@10 is 5.78/2.38 times theirs.The book retrieval set is an unseen domain for PathCLIP training.Source: Experiments · p. 6, Figure 4 · p. 6
  • PathCLIP improves zero-shot F1 over CLIP and PLIP across CRC100K, WSSS4LUAD, LC-lung, and LC-colon.Against CLIP, gains are 32%, 19.5%, 57.2%, and 18.6%; against PLIP, they are 1.1%, 11.6%, 2.7%, and 7.3%, respectively.Source: Experiments · pp. 5–6
  • 8.7%/18.4%: PathAsst exceeds LLaVA on closed-ended/open-ended PathVQA, with PathCLIP adding 1.2%/0.8%; prior SOTA remains better on open-ended questions.Table 3 compares methods on PathVQA.Source: Experiments · pp. 6–7, Table 3 · p. 6
  • PathAsst invokes an LBC cell-generation model for a requested LSIL cell and a PD-L1 detector to mark positive cells and return counts for LLM analysis.The PD-L1 example presents results in a markdown table with accompanying analysis.Source: Figure 5 · p. 6, Experiments · p. 7, Figure 6 · p. 7
  • In image interpretation, PathAsst emphasizes cell morphology, including enlarged nuclei and irregular nuclear membranes; LLaVA misses the pathology context and MiniGPT-4 gives simpler descriptions.The example contrasts the models’ independent interpretations of pathology images.Source: Experiments · p. 7, Figure 7 · p. 7

Conclusion

The conclusion presents PathCap and PathInstruct as datasets for developing pathology foundation models, and PathCLIP and PathAsst as models built from them. PathAsst combines a pathology vision encoder, Vicuna-13b, and specialized tools for dialogue, image interpretation, and complex tasks.

Conclusion

  • PathCap and PathInstruct comprise 207K pathology image-text pairs and 180K instruction-following samples, respectively.The datasets were constructed by systematically collecting and processing pathology data from various sources.Source: Conclusion · p. 7
  • PathCLIP is reported to perform strongly in pathology cross-modal retrieval and zero-shot classification.PathAsst combines PathCLIP with Vicuna-13b as an instruction-tuned foundation model.Source: Conclusion · p. 7
  • PathAsst uses eight pathology-specific models and a 5.3-million-sized paper retrieval system to support multimodal dialogue, image interpretation, and complex tool-assisted tasks.The authors hope these datasets and model frameworks will aid the advancement of pathology foundation models.Source: Conclusion · p. 7

Related papers

Loading 2305.15072v2…