PaperlayerView exact revision on arXiv

Source-linked AI summary

MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, Yuyin Zhou

arXiv:2408.02900v3cs.CV

TL;DR

Existing medical datasets lack multigranular annotations connecting regional image information and often rely on paired expert-authored text, restricting scalability. MedTrinity-25M uses an automated pipeline to create multigranular image-ROI-description triplets from unpaired images; LLaVA-Tri achieves state-of-the-art results on three medical VQA benchmarks.

  • Problem

    Current medical datasets lack multigranular annotations revealing region-wise correlations and rely heavily on paired expert reports or captions, which restricts scalability.

    Source: §1 · p. 2

  • Method

    An automated pipeline uses domain-specific models to identify ROIs and retrieval-augmented generation to produce multigranular image-ROI-description triplets from images without paired text.

    Source: §1 · p. 2

  • Results

    LLaVA-Tri achieved state-of-the-art results on all three VQA benchmarks, including 75.0% accuracy on VQA-RAD.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MedTrinity-25M’s enriched annotations have the potential to support captioning, report generation, classification, segmentation, and large-scale pretraining of multimodal medical AI models.

    Source: §5 · p. 10

Abstract

from arXiv · show

This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with multigranular annotations for more than 65 diseases. These multigranular annotations encompass both global information, such as modality and organ detection, and local information like ROI analysis, lesion texture, and region-wise correlations. Unlike the existing multimodal datasets, which are limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and textual annotations in the form of image-ROI-description triplets without the need for any paired text descriptions. Specifically, data from over 30 different sources have been collected, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions. We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmented generation with the identified ROIs as guidance, resulting in multigranular textual descriptions. Compared to existing datasets, MedTrinity-25M provides the most enriched annotations, supporting a comprehensive range of multimodal tasks such as captioning and report generation, as well as vision-centric tasks like classification and segmentation. We propose LLaVA-Tri by pretraining LLaVA on MedTrinity-25M, achieving state-of-the-art performance on VQA-RAD, SLAKE, and PathVQA, surpassing representative SOTA multimodal large language models. Furthermore, MedTrinity-25M can also be utilized to support large-scale pre-training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain. We will make our dataset available.

1 INTRODUCTION

The introduction identifies missing region-wise annotations and reliance on paired text as limitations of current medical datasets. It presents an automated pipeline for multigranular triplets and reports state-of-the-art results across three VQA benchmarks.

1 INTRODUCTION

  • Current medical datasets lack annotations linking region-wise information, while reliance on paired expert reports or captions restricts their scalability.Local abnormal textures or structures may provide cues for inferring global disease or lesion type.Source: §1 · p. 2
  • The pipeline uses retrieval-augmented generation and domain-specific segmentation and grounding models to generate ROI-guided multigranular annotations without paired text descriptions.It transforms images into triplets containing global information and local ROI annotations, including bounding boxes, masks, and region-specific descriptions.Source: §1 · p. 2
  • MedTrinity-25M contains over 25 million image-ROI-description triplets, and the paper describes it as the largest multimodal dataset in medicine to date.The triplets are generated from large-scale images without paired ROIs or text.Source: §1 · p. 2
  • LLaVA-Tri achieved state-of-the-art accuracy on VQA-RAD (75.0%), SLAKE (87.8%), and PathVQA (65.3%) after pretraining on MedTrinity-25M.The paper reports these results across three external medical visual QA datasets.Source: §1 · p. 2

2 RELATED WORK

Prior medical multimodal models are trained on medical datasets, but training-data scale remains a challenge. Existing paired-image datasets trade off scale and clinical detail, while MedTrinity-25M generates multigranular annotations for independent images.

2 RELATED WORK

  • Med-Flamingo uses 0.8M interleaved and 1.6M paired medical image-text data; LLaVA-Med uses visual instruction tuning, and Med-Gemini adapts Gemini with long-form QA (Moor et al., 2023; Li et al., 2024a; Saab et al., 2024).The paper notes limited training-data scale as an ongoing challenge for medical multimodal foundation models.Source: §2 · p. 2
  • Report-paired datasets provide detailed descriptions but manual report creation is costly; PMC-OA offers up to 1.65 million image-caption pairs but lacks detailed clinical reports.RadGenome-Chest CT offers richer annotations but remains dependent on paired image-text data, limiting its scale.Source: §2 · p. 3
  • MedTrinity-25M’s automated pipeline generates multigranular annotations for independent images, producing a dataset with 25 million samples.This contrasts with prior datasets that rely on paired image-text data.Source: §2 · p. 3

3 MEDTRINITY-25M DATASET

MedTrinity-25M organizes medical images, visual ROIs, and structured multigranular descriptions into triplets, constructed through a pipeline combining metadata, ROI localization, medical knowledge retrieval, and MLLM generation. Its dataset analysis reports broad scale and diversity, richer annotations, and expert and LLM evaluation scores.

3.1 DATA TRIPLET

  • Each MedTrinity-25M sample is an image–ROI–description triplet, with descriptions covering image attributes, lesion details, and region-wise correlations.ROIs focus on abnormalities or, when none are present, the primary object or organ; multiple affected regions may be annotated.Source: §3.1 · p. 3

3.3 DATASET ANALYSIS

  • MedTrinity-25M contains 25,016,845 images across 10 modalities and over 65 diseases, and is described as the largest open-source multimodal multigranular medical dataset to date.The paper also reports richer annotations and longer descriptions than compared multimodal datasets, including MIMIC-CXR and SLAKE.Source: §3.1 · p. 3, §3.3 · p. 8

3.2 DATA CONSTRUCTION PIPELINE

  • The construction pipeline integrates metadata, locates ROIs, retrieves medical knowledge, then prompts MLLMs to generate descriptions that are combined with the image and ROIs.Metadata integration creates coarse captions; ROI localization and knowledge retrieval precede multigranular text generation.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.2.1 DATA PROCESSING

  • Coarse captions provide modality, organ, and disease context; ROI guidance supports local abnormality analysis and correlations with surrounding regions.Existing localization annotations are used when available; otherwise, pretrained expert models generate ROIs. The qualitative comparison reports that disease labels in coarse captions support disease identification and categorization.Source: §3.2.1 · pp. 5–6
  • A medical knowledge database supplies retrieved snippets relevant to coarse captions, which can help MLLMs standardize medical terminology and refine diagnoses using disease progression information.The database draws on PubMed, StatPearls, and medical textbooks; the pipeline retrieves the top eight snippets using caption embeddings.Source: §3.2.1 · pp. 6–7, Figure 7 · p. 7

3.2.2 GENERATION OF MULTIGRANULAR TEXT DESCRIPTION

  • The generation prompt combines captions, ROI information, and retrieved knowledge, directing MLLMs to produce global, ROI-focused, and region-correlation descriptions.ROI coordinates and area ratios are converted into text; LLaVA-Medcap generates descriptions for the dataset’s images.Source: §3.2.2 · p. 7

3.3 DATASET ANALYSIS

  • MedTrinity-25M achieved 0.85 in expert evaluation and 0.86 in LLM evaluation, while scores for modality, organ detection, and ROI analysis neared perfect.Source: §3.3 · p. 9

4 EXPERIMENT

LLaVA-Tri achieves state-of-the-art results on three medical VQA benchmarks after MedTrinity-25M alignment. Ablations report performance gains across tested multimodal models, while the training setup combines LLaVA with multiscale features and multigranular alignment.

4.1 LLAVA-TRI: ALIGNING MULTISCALE MLLM WITH MEDTRINITY-25M

  • LLaVA-Tri builds on LLaVA, adds LLaMA3 and multiscale feature extraction, and is trained first on PMC-15M image-text pairs, then on MedTrinity-25M for multigranular alignment.The initial PMC-15M pretraining uses 600K image-text pairs.Source: §4.1 · p. 9
  • 75.0% accuracy on VQA-RAD, 87.8% on SLAKE, and 65.3% on PathVQA establish LLaVA-Tri’s state-of-the-art results across all three benchmarks.Models were fine-tuned for 15 epochs on each dataset before evaluation.Source: §4.1 · p. 9, Table 3 · p. 10

4.2 ENHANCING MODEL PERFORMANCE THROUGH MULTIGRANULAR ALIGNMENT

  • 2.0%, 5.0%, and 1.6% improvements on VQA-RAD, SLAKE, and PathVQA, respectively, are reported for LLaVA-Tri with alignment versus without.The ablation reports significant gains across LLaVA-Tri, InternVL2-8B, MiniCPM-V-2.6-8B, and PubMedCLIP.Source: §4.2 · p. 10, Table 4 · p. 10

5 CONCLUSION

The conclusion presents MedTrinity-25M as a large-scale dataset of image-ROI-description triplets, built from unpaired images and intended to support varied medical multimodal tasks. The listed appendix materials include data sources, model comparisons, ROI examples, expert models, alignment evaluation, and generation prompts.

5 CONCLUSION

  • MedTrinity-25M contains over 25 million image-ROI-description triplets from more than 30 resources, spanning 10 modalities and over 65 diseases.Its automated pipeline generates multigranular visual and textual annotations from unpaired images.Source: §5 · p. 10
  • The dataset’s enriched annotations are described as supporting captioning, report generation, classification, segmentation, and large-scale pretraining of medical multimodal AI models.The paper presents these as potential uses.Source: §5 · p. 10

APPENDIX

  • The appendix lists data-source details, a quantitative LLaVA-Medcap–GPT-4V comparison, and examples of ROIs for normal and multiple regions.Source: APPENDIX · p. 17
  • Additional appendix materials cover expert ROI models, LLM alignment evaluation, and the MedTrinity-25M generation prompt.Source: APPENDIX · p. 17

A DATA SOURCE

The data-source table catalogs medical image datasets and their modalities, biological structures, quantities, and annotations. Its listed modalities range from CT and MR to histopathology, fundus, and PET.

A DATA SOURCE

  • Table 5 organizes medical image sources by dataset name, modality, biological structures, quantity, and text and disease annotations.The caption describes the table as covering these source attributes.Source: Table 5 · p. 17, A DATA SOURCE · p. 18
  • The source fragments include CT datasets such as FLARE23 and KIPA22, and microscopy data such as ihc4bc.Source: A DATA SOURCE · p. 18
  • Listed modalities include CT, MR, endoscopy, X-ray, ultrasound, histopathology, dermoscopy, microscopy, fundus, and PET.Source: A DATA SOURCE · p. 18

B QUANTITATIVE COMPARISON OF LLAVA-MEDCAP WITH GPT-4V

The paper compares LLaVA-Medcap’s generated descriptions with GPT-4V’s and reports greater average word count for LLaVA-Medcap. It then uses LLaVA-Medcap to generate descriptions for MedTrinity-25M.

B QUANTITATIVE COMPARISON OF LLAVA-MEDCAP WITH GPT-4V

  • LLaVA-Medcap is a fine-tuned LLaVA using LLaMA3 for linguistic capability and multiscale feature extraction for vision capability.The model was selected to generate descriptions for all 25 million MedTrinity-25M images.Source: B QUANTITATIVE COMPARISON OF… · p. 20
  • LLaVA-Medcap descriptions averaged 3.6% more words than GPT-4V descriptions for the same sample, and were characterized as more detailed.The comparison assessed detail using average word count.Source: B QUANTITATIVE COMPARISON OF… · p. 20

C EXAMPLES OF ROIS

ROIs primarily mark pathological findings, while normal cases focus on the primary object or organ. When diagnosis depends on broader context, ROIs can cover multiple lesion areas so local analysis contributes to a global view.

C EXAMPLES OF ROIS

  • ROIs predominantly capture pathological features; when abnormalities are absent, they typically mark the primary object or organ.Examples of normal ROIs are shown in Figure 10.Source: C EXAMPLES OF ROIS · p. 20
  • In chest radiography, ROIs can encompass lesions in both lungs, whose local texture analysis contributes to the overall diagnosis.The example requires analysis of the entire image, including both lungs.Source: Figure 11 · p. 21, C EXAMPLES OF ROIS · p. 21
  • When global context matters, ROIs may encompass multiple lesion areas, integrating local and global information for diagnosis.Source: C EXAMPLES OF ROIS · p. 21

D LIST OF EXPERT MODELS TO LOCATE ROIS

For datasets without localization annotations, pretrained expert models identify ROIs; Table 6 lists the models used for each dataset.

D LIST OF EXPERT MODELS TO LOCATE ROIS

E DETAILS OF LLM EVALUATION OF ALIGNMENT

GPT-4V evaluation examples show full alignment with human annotations across five criteria. The generation prompt breaks description into smaller tasks organized as a three-level hierarchy.

E DETAILS OF LLM EVALUATION OF ALIGNMENT

  • GPT-4V fully aligned with human annotations across all five criteria in the illustrated example, yielding perfect alignment scores.Figure 12 shows the example; Figure 13 presents the evaluation prompt.Source: E DETAILS OF LLM EVALUATION OF ALIGNMENT · p. 21

F PROMPT TEMPLATE FOR GENERATION OF MULTIGRANULAR TEXT DESCRIPTION

12 PMC-OA

The prompts guide multimodal models from image-wide understanding to ROI analysis and region-wise relationships. An illustrated evaluation example receives perfect scores across five criteria.

  • Global understanding prompts a comprehensive image description covering modality, anatomical structures, and approximate locations.Source: §12 · p. 22
  • Local analysis prompts examination of ROI locations, abnormalities, and textures.Source: §12 · p. 22
  • Region-wise correlation prompts analysis of relationships among regions and how surrounding areas may be affected by ROI lesions.The step aims to connect local abnormalities with organ-wide effects for disease diagnosis.Source: §12 · p. 22
  • The illustrated GPT-4V evaluation example aligns fully with human annotations across five criteria and receives perfect scores.Source: Figure 12 · p. 23
Loading 2408.02900v3…