PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Multimodal Medical Capabilities of Gemini

Lin Yang, Shawn Xu, Andrew Sellergren, Timo Kohlberger, Yuchen Zhou, Ira Ktena, Atilla Kiraly, Faruk Ahmed, Farhad Hormozdiari, Tiam Jaroensri, Eric Wang, Ellery Wulczyn, Fayaz Jamil, Theo Guidroz, Chuck Lau, Siyuan Qiao, Yun Liu, Akshay Goel, Kendall Park, Arnav Agharwal, Nick George, Yang Wang, Ryutaro Tanno, David G. T. Barrett, Wei-Hung Weng, S. Sara Mahdavi, Khaled Saab, Tao Tu, Sreenivasa Raju Kalidindi, Mozziyar Etemadi, Jorge Cuadros, Gregory Sorensen, Yossi Matias, Katherine Chou, Greg Corrado, Joelle Barral, Shravya Shetty, David Fleet, S. M. Ali Eslami, Daniel Tse, Shruthi Prabhakara, Cory McLean, Dave Steiner, Rory Pilgrim, Christopher Kelly, Shekoofeh Azizi, Daniel Golden

arXiv:2405.03162v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Medical multimodal modeling remains underexplored because clinical data are specialized and clinically grounded evaluations are limited. This paper fine-tunes Gemini-based Med-Gemini models across medical imaging and genomics, reporting promising performance across classification, VQA, and report generation.

  • Problem

    Medical multimodal modeling remains underexplored because clinical data are complex and evaluations grounded in clinical use cases are lacking, making rigorous evaluation across relevant applications necessary.

    Source: §1 · p. 2

  • Method

    The authors fine-tune three Gemini 1.5-based Med-Gemini models on medical data paired predominantly with descriptive free text for imaging and genomic tasks.

    Source: §1 · p. 4, §7 · p. 30

  • Results

    Med-Gemini shows promising performance across classification, VQA, and report generation, with best-in-class chest X-ray and CT reporting and more accurate disease and mortality risk prediction than standard linear PRS.

    Source: §1 · p. 4, §7 · p. 30

  • Takeaways & Limitations

    The findings demonstrate the potential of Gemini-based models across medical tasks.

    Source: §8 · p. 32

    CT results remain proof of concept and do not yet reach performance required for clinical use.

    Source: §7 · p. 31

Abstract

from arXiv · show

Many clinical tasks require an understanding of specialized data, such as medical images and genomics, which is not typically found in general-purpose large multimodal models. Building upon Gemini's multimodal models, we develop several models within the new Med-Gemini family that inherit core capabilities of Gemini and are optimized for medical use via fine-tuning with 2D and 3D radiology, histopathology, ophthalmology, dermatology and genomic data. Med-Gemini-2D sets a new standard for AI-based chest X-ray (CXR) report generation based on expert evaluation, exceeding previous best results across two separate datasets by an absolute margin of 1% and 12%, where 57% and 96% of AI reports on normal cases, and 43% and 65% on abnormal cases, are evaluated as "equivalent or better" than the original radiologists' reports. We demonstrate the first ever large multimodal model-based report generation for 3D computed tomography (CT) volumes using Med-Gemini-3D, with 53% of AI reports considered clinically acceptable, although additional research is needed to meet expert radiologist reporting quality. Beyond report generation, Med-Gemini-2D surpasses the previous best performance in CXR visual question answering (VQA) and performs well in CXR classification and radiology VQA, exceeding SoTA or baselines on 17 of 20 tasks. In histopathology, ophthalmology, and dermatology image classification, Med-Gemini-2D surpasses baselines across 18 out of 20 tasks and approaches task-specific model performance. Beyond imaging, Med-Gemini-Polygenic outperforms the standard linear polygenic risk score-based approach for disease risk prediction and generalizes to genetically correlated diseases for which it has never been trained. Although further development and evaluation are necessary in the safety-critical medical domain, our results highlight the potential of Med-Gemini across a wide range of medical tasks.

1. Introduction

The work develops Med-Gemini, a Gemini-tuned model family for medical imaging and genomics, and evaluates its capabilities across clinically relevant medical tasks.

Introduction

  • The authors improve selected open benchmarks by correcting erroneous labels, adding MIMIC-CXR question-answer pairs, and refining splits to reduce train-test contamination.Examples include MIMIC-CXR-JPG classification labels, VQA pairs for MIMIC-CXR, and splits for PAD-UFES-20 and VQA-Rad.Source: §1 · p. 2
  • Med-Gemini-2D improves chest X-ray report generation by 10% and 18% over the previous leading model across two datasets, surpasses baselines on 18 of 20 image-classification tasks, and Med-Gemini-Polygenic generalizes to untrained diseases.The models also demonstrate best-in-class performance on chest X-ray and CT report generation and chest X-ray classification, and outperform a standard linear polygenic risk score approach for disease and mortality risk prediction.Source: Figure 1 · p. 3, §1 · p. 4
  • The Med-Gemini family extends Gemini to medical image classification, VQA, report generation, and genomic risk prediction using 7 million samples from 3.7 million images and cases.The training data mostly pairs free text with medical data, avoiding the need for expensive expert labeling; the work explores both imaging and polygenic risk prediction.Source: §1 · p. 4
  • Evaluation spans 22 datasets, five tasks, six medical image modalities, and eight out-of-distribution datasets, using expert evaluation for report generation and open-question radiology VQA.The benchmarks combine open and curated datasets, compare Med-Gemini with prior work or non-medically tuned Gemini where possible, and emphasize clinically relevant metrics.Source: §1 · pp. 2, 4

2. Datasets

Med-Gemini was fine-tuned and evaluated using diverse de-identified medical datasets, with patient-level or image-disjoint splits used to prevent overlap. The datasets span multiple modalities and clinical tasks, with preprocessing tailored to imaging and genomic inputs.

2. Datasets

  • Training and evaluation splits were separated by patient identifier when available, or by case or image when identifiers were unavailable.All datasets were de-identified; open datasets followed their licenses, and private datasets were used with permission and appropriate licenses.Source: §2.1 · p. 4, §2 · p. 4

2.1.1. Public datasets

  • The cited medical-data resources span MIMIC-CXR chest X-rays, NLST CT-slice captions, medical VQA, and PMC-OA image-caption pairs; MIMIC-CXR test cases benchmarked classification, report generation, and VQA.MIMIC-CXR contributed 237,912 training images; NLST provided 1,759 training and 440 validation slice-caption pairs, and VQA-Med used 12,664 QA pairs in training. PMC-OA yielded 2,246,656 filtered image-caption pairs.Source: §2.1.1 · pp. 5–6

2.1.2. Private datasets

  • The training datasets comprised 207,603 unique pathology patches, 15,976 de-identified EyePACS fundus images (12,976 lesion-positive and 3,000 healthy), and 753,247 CT studies from 615,384 patients.Pathology patches with curated captions yielded 1,550,976 image-text pairs. CT-US1 used patient-level 70%/15%/15% training, validation, and test splits; a subset of 92 non-contrast head/neck test volumes was used for expert model assessment.Source: §2.1.2 · pp. 6–7

2.2.1. Public datasets

  • Evaluations included CheXpert, ChestX-ray14, VQA-RAD, and the TCGA test set, used as an out-of-distribution dataset to assess generalization to different histopathology-related tasks.CheXpert evaluation included only frontal views and treated unmentioned labels as negative; ChestX-ray14 evaluation used 1,962 cases for three radiologist-adjudicated conditions. The image-disjoint balanced VQA-RAD split comprised 2,248 QA pairs: 1,299 closed-ended and 949 open-ended.Source: §2.2.1 · pp. 7–9

2.1. Datasets for fine-tuning and instruction-tuning

  • More than 7 million samples from 3.7 million medical images and cases supported fine-tuning and instruction-tuning across radiology, pathology, ophthalmology, and genomics.Most training examples paired medical data with free text, avoiding the need for expensive expert labeling.Source: Table 1 · p. 8

2.2.2. Private datasets

  • Private held-out datasets comprised 263,021 Indian frontal chest X-rays and a 16-tissue-type histopathology set used to test generalization.For human evaluation, 300 IND1 cases were drawn from the same test set used by Tanno et al. (2024); the TTH test set assessed generalization across histopathology tasks.Source: §2.2.2 · p. 9

2.3. Data preprocessing

  • Preprocessing standardized radiology images and CT volumes, resized 2D inputs to 768 × 768, and represented genomic risk scores as images.CT volumes were clipped to [−1024 HU, 1024 HU], scaled to [0.0, 1.0], and resampled to 0.7 mm × 0.7 mm × 1.4 mm; genomic inputs encoded PRSs for 7,415 traits across three p-value thresholds. Eight in-distribution and six genetically correlated out-of-distribution health outcomes were selected for genomic prediction.Source: §2.3 · pp. 9, 11

2.2. Held-out datasets for evaluation and benchmarking

  • 22 evaluation datasets spanning five task types and seven medical image modalities included eight out-of-distribution datasets and more than 40,000 samples.The tasks covered classification, VQA, 2D and 3D report generation, and disease prediction from genetic risk embeddings.Source: Table 2 · p. 10

3. Modeling Methodology

Med-Gemini builds on Gemini 1.5 and uses modality-specific fine-tuning to create 2D, 3D, and genomics variants, followed by instruction tuning. Its training and serving use TPUv4 accelerator pods.

3.1. Model architecture

  • Gemini 1.5’s context window of up to 1 million tokens and video-understanding capabilities motivated its selection for Med-Gemini fine-tuning.Its video encoder can integrate frames, multi-slice images, text, or audio, and its variable input resolutions allow computational resources to be prioritized for high-resolution analysis.Source: §3.1 · p. 11

3.2. Multimodal fine-tuning

  • Three custom vision encoders, for 2D, 3D, and genomics, outperformed a single encoder across formats; fine-tuning both vision and language components improved visual understanding over Gemini 1.5 Pro’s native encoder.These encoders formed Med-Gemini-2D, Med-Gemini-3D, and Med-Gemini-Polygenic, with fine-tuning framed as captioning or VQA.Source: §3.2 · p. 11
  • Med-Gemini-2D jointly fine-tuned radiology, pathology, dermatology, and ophthalmology images; Med-Gemini-3D used video encoding to synthesize information across CT slices.The 3D approach replaces the video time axis with depth and could support other volumetric or time-series data, such as MRI or ultrasound. Med-Gemini-Polygenic trained on PRS projections to predict eight broad health outcomes in a captioning task.Source: §3.2 · p. 12
  • A subsequent instruction-tuning phase used curated multimodal instruction–response pairs to improve medical content understanding, instruction following, and tailored output generation.The phase fine-tuned Gemini 1.5 Pro.Source: §3.2 · p. 12

3.3. Model training and inference infrastructure

  • Med-Gemini training used large-scale TPUv4 accelerator pods across multiple data centers, with the Gemini architecture supporting efficient TPU serving at scale.The setup scaled up from the PaLM family.Source: §3.3 · p. 12

4. Evaluation and Results

The evaluation covered 22 datasets and more than 40,000 samples across four clinical task types and seven medical image modalities, including eight out-of-distribution datasets. Results included occasional correct clinical management but limited radiologist-level quality for head/neck CT reports, while Med-Gemini-Polygenic generally exceeded in-distribution PRS-based benchmarks and showed zero-shot performance on genetically correlated diseases.

4.1. Medical image classification

  • Med-Gemini outperformed Gemini Ultra on most in-distribution MIMIC-CXR labels, but out-of-distribution performance varied, including weaker fracture detection on ChestX-ray14.The table caption notes that both models fell short of chest X-ray-specialized models on out-of-distribution datasets.Source: §4.1 · p. 13, Table 3 · p. 14
  • Med-Gemini-2D outperformed Gemini Ultra on 7 of 9 in-distribution histopathology tasks and both held-out tasks, but did not surpass PathSSL on any task.Its embeddings also achieved comparable performance to PathSSL while showing strong results across other clinical domains.Source: §4.1 · p. 14, Figure 2 · p. 15
  • Med-Gemini-2D achieved skin-lesion classification performance close to Derm Foundation, with fine-tuning improving its F1 and accuracy.Both Gemini Ultra and Med-Gemini-2D produced robust linear-probing embeddings for this task.Source: §4.1 · p. 15, Table 4 · p. 15
  • Med-Gemini-2D exceeded Gemini Ultra on fundus lesion and anomaly classification, including hard-exudate specificity of 96.4% versus 39.0%.It underperformed the supervised anomaly-detection model, which used approximately 200× more labeled data; it was also miscalibrated on three other attempted lesion-detection tasks.Source: §4.1 · p. 16, Table 5 · p. 16

4.2. Visual question answering (VQA)

  • 71.9 expert-evaluated accuracy and 78.8 closed-ended accuracy on chest X-ray-only ELIXR-split VQA improved best-in-class results by 14 and 11.7, respectively.On Slake VQA, Med-Gemini-2D achieved 84.8 accuracy on closed-ended questions; its 75.8 mean tokenized F1 across English questions was below MedPaLM-M’s 89.3, with the latter using one-shot prompting.Source: Table 6 · p. 17, §4.2 · p. 18
  • On PathVQA, Med-Gemini-2D achieved 83.3 accuracy on Yes/No questions and 58.7 tokenized F1 overall, improving over Gemini; the dataset’s known quality issues warrant cautious interpretation.Its overall zero-shot result was slightly below MedPaLM-M’s 62.7 tokenized F1, which used a text-only one-shot prompt.Source: §4.2 · p. 18

4.3. Report generation for chest X-rays

  • Med-Gemini achieved a 24.36% RadGraph F1-score for MIMIC-CXR chest X-ray report generation, producing both FINDINGS and IMPRESSION sections for frontal anterior-posterior or posterior-anterior views.In radiologist comparisons, reports missed key findings including tube positions, nodules, and pneumothoraces.Source: Table 8 · p. 20, §4.3 · p. 20

4.4. Report generation for head/neck CT volumes

  • Med-Gemini-3D was presented as the first LLM-based generative AI model to interpret a 3D medical imaging modality end to end, from CT volume to text.The passage attributes feasibility to increased computational capacity and domain-specific pretraining amid data scarcity and the complexity of 3D image-report modeling.Source: §4.4 · p. 21
  • Missed findings and hallucinations each accounted for roughly half of error notes, while other reports incorrectly estimated or under-characterized white matter changes.Examples of hallucinations included subdural hematomas and cysts.Source: §4.4 · p. 21
  • 17% of head/neck CT reports were equivalent or superior to radiologists’ reports, while 53% were judged to result in equivalent patient care.The prose reports correct clinical management for 45% of normal and 57% of abnormal studies; the model used only the study’s single series with the most slices.Source: §4.4 · p. 21, Table 10 · p. 22, Figure 4 · p. 23

4.5. Disease prediction from genetic information

  • Med-Gemini-Polygenic achieved higher AUCs than PRS linear-model benchmarks for all in-distribution outcomes except glaucoma, and similar performance to benchmarks on six never-trained outcomes.The out-of-distribution outcomes were genetically correlated with training outcomes, and the model was not instructed about those associations.Source: §4.5 · p. 23
  • With PRSs and demographics, Med-Gemini-Polygenic often outperformed linear probes, largely by modeling nonlinear interactions between genomic information and demographics.When given demographic information alone, the model performed similarly to the linear probe.Source: §4.5 · p. 23
  • Reported AUCs are an upper bound because the PRS features were created using GWASs performed within the UK Biobank.The analyses focus on relative performance among models operating on the same in-sample data.Source: §4.5 · p. 23

5. Qualitative Results

Med-Gemini demonstrates plausible dialogue and image-interpretation capabilities across several medical imaging domains, while expert review and examples identify limitations in accuracy, completeness, detail, and follow-up interaction.

5. Qualitative Results

  • Examples span chest X-ray, CT, fundus, dermatology, and pathology dialogue, alongside 2D and 3D radiology report generation.The authors characterize the responses as accurate and reasonable overall, but expert review notes opportunities to improve phrasing, accuracy, appropriate detail, and completeness.Source: §5 · pp. 23–24
  • A proof-of-concept CXR autocomplete recovered emphysema and pulmonary edema concepts omitted from reports generated without a hint.The model was directed toward a specific region or organ, and the concepts were recovered using an autocomplete prefix hint.Source: §5 · p. 24, Figure 10 · p. 29
  • Despite no treatment- or symptom-related question-answer fine-tuning data, Med-Gemini gave simple, reasonable answers to such questions using Gemini pretraining knowledge.The authors caution that real-world diagnosis, prognosis, and treatment are more complicated and nuanced than these proof-of-concept examples.Source: §5 · p. 24
  • Histopathology examples show interpretation across tissues and findings with little visual context, but follow-up interaction and image exploration remain limited.The examples provide promising initial evidence of further interaction and reasoning, which the authors identify as an area for improvement.Source: Figure 7 · p. 26
  • Radiologist-reviewed CXR reports were considered reasonable across example conditions, with one noted hallucination involving increased AP diameter in the bottom-right case.The radiologist noted that increased AP diameter is usually detected from a lateral image.Source: Figure 8 · p. 27
  • 3D head CT examples show Med-Gemini identifying some abnormalities missed in radiologist reports, but also mischaracterizing present findings and hallucinating absent ones.The figure includes both a correct and an incorrect abnormal case.Source: Figure 9 · p. 28

6. Related Work

Related work traces the expansion of medical AI from domain-specific language models to multimodal systems spanning specialized and generalist approaches. It also identifies inconsistent evaluation as a barrier to comparing medical VLMs and motivates unified benchmarks.

The evolution of medical language models

  • Medical language models address clinical and biomedical domains, while other models demonstrate language-model applications to omic information.Examples include PubMedGPT and Med-PaLM for medical domains, and HyenaDNA and ProtLLM for omic information.Source: §6 · p. 29

Multimodal models in medicine

  • General-purpose multimodal models process text and images, with Gemini also exhibiting reasoning across video and audio modalities.These capabilities provide context for multimodal models developed for medical applications.Source: §6 · pp. 29–30
  • Medical VLMs span generalist-to-specialist approaches, covering multiple imaging modalities as well as focused radiology, histopathology, report-generation, and VQA tasks.Some systems target particular modalities or tasks, while others aim to cover broader medical applications.Source: §6 · p. 30

Multimodal evaluation benchmark and metrics

  • Generalist medical AI models such as Med-PaLM M aim to assist across medical tasks, but inconsistent evaluation motivates unified benchmarks.Existing studies vary in tasks, datasets, and metrics, hindering direct comparison even on the same dataset.Source: §6 · p. 30

7. Discussion

The paper presents three Gemini 1.5-based Med-Gemini models with early promise across medical tasks and modalities, trained predominantly on medical data and paired free-text reports. The discussion emphasizes that further evaluation and development are needed before clinical use.

Discussion

  • The three models address classification, VQA, and report generation across medical modalities, using paired descriptive reports to reduce reliance on expensive expert labeling.The modalities include 2D and 3D radiology, histopathology, ophthalmology, dermatology, and genetic risk scores.Source: §7 · p. 30
  • Future models may combine tasks and modalities, while longer context windows and improved reasoning could incorporate historical context into decision-making.The paper presents these capabilities as directions for future work.Source: §7 · pp. 30–31
  • Med-Gemini-3D CT results remain proof of concept and below clinical-use performance, amid challenges from data scale, architectural limits, and 3D task complexity.The authors expect architectures to improve and propose exploring other complex modalities.Source: §7 · p. 31

Closing the gap between benchmark and bedside

  • Benchmarks alone do not establish bedside performance; evaluations should test AI-human collaboration and patient outcomes, with diverse healthcare professionals guiding development.Specialist panels assessed report generation and question answering in this study, but further evaluation beyond academic benchmarks is needed for safety and reliability.Source: §7 · p. 31

Identifying and mitigating data bias and safety risks

  • Models may inherit data biases and errors that cause misdiagnoses or amplify systemic bias, so risks should be evaluated and mitigated and performance validated for each use case and population.Source: §7 · p. 31

Minimizing data contamination when evaluating zero-shot generalization in large models

  • Potential training-data contamination can inflate apparent zero-shot generalization, so prospective studies may help assess performance on genuinely novel concepts (Udandarao et al., 2024; Vogel et al., 2022).Prospective studies are typically more expensive and time-consuming than retrospective studies.Source: §7 · p. 31

8. Conclusion

The Med-Gemini family applies Gemini’s multimodal foundation to specialized medical data and shows promise across imaging and genomics. The findings also underscore the need for further research on safe clinical implementation and point toward integrated systems that work alongside humans.

8. Conclusion

  • The conclusion identifies specialized medical data as a reason general-purpose models may not naturally perform well in medicine.It frames multimodal generative AI, including Gemini, as having potential in healthcare.Source: §8 · p. 32
  • The Med-Gemini family builds on Gemini and is fine-tuned on radiology, histopathology, ophthalmology, dermatology, and genomics data.Its performance was assessed using established and custom clinical-relevance benchmarks, including medical-expert evaluations of CXR and CT reports and radiology VQA.Source: §8 · p. 32
  • Med-Gemini-2D sets a new standard for expert-evaluated chest X-ray report generation, while Med-Gemini-3D presents the first LMM-based report generation for 3D CT.Med-Gemini-2D also performs strongly in medical imaging VQA and classification; Med-Gemini-Polygenic outperforms conventional polygenic risk score methods in disease-risk prediction.Source: §8 · p. 32
  • The authors call for further rigorous research to ensure safe and effective real-world clinical implementation and envision integrated systems working alongside humans on multidisciplinary tasks.They describe the study as a step toward improving clinical efficacy and patient outcomes through such systems.Source: §8 · p. 32

Data Availability

Most datasets are publicly accessible with permissions, with several private-dataset exceptions and planned releases of labels, question-answer pairs, and dataset splits. The authors will not release model code or weights because of safety concerns, and report Alphabet funding and limited Gemini copy editing.

Data Availability

  • 53% of datasets are publicly accessible with permissions, except IND1, CXR-US2, CT-US1, Eyepacs, and TTH; planned releases include labels, question-answer pairs, and dataset splits.The planned releases cover updated MIMIC-CXR classification labels and custom VQA pairs, PAD-UFES-20 and VQA-Rad splits, and radiologist-recommended replacement VQA-Rad pairs; availability updates are pending.Source: Data Availability · p. 34

Code Availability

  • The authors will not open-source model code or weights due to safety concerns about unmonitored medical use, and plan partner and provider collaborations through Google Cloud APIs.The collaboration is intended to validate and explore safe applications of Gemini and Med-Gemini.Source: Code Availability · p. 34

Competing Interests

  • The study was funded by Alphabet, and authors affiliated with Google Research, Google DeepMind, and Verily are Alphabet employees who may own stock through standard compensation.Source: Competing Interests · p. 34

Use of AI in Manuscript Preparation

Appendix · A.1. Additional data details · A.1.1. Revised MIMIC-CXR classification labels

The appendix describes human-adjudicated revisions to MIMIC-CXR classification labels, addressing the dataset’s lack of ground-truth labels. Med-PaLM 2 labels matched the adjudicated ground truth more often than the original labels in the flagged cases.

A.1.1. Revised MIMIC-CXR classification labels

  • MIMIC-CXR JPG extracted structured labels from 277,827 radiology reports using CheXpert, but the dataset lacks ground-truth labels (Irvin et al., 2019).The work further adjudicated labels used in Xu et al. (2023).Source: §A.1.1 · p. 45
  • The refinement process used keyword searches to flag reports, then applied Med-PaLM 2 per label with two queries, totaling 23,824 queries.Labels that disagreed with the original labels were flagged for human verification.Source: §A.1.1 · p. 45
  • Three US-based board-certified radiologists reviewed 1,378 flagged labels, with a fourth US-based academic thoracic radiologist adjudicating reviewer disagreements.For each finding and report, reviewers selected positive, negative, uncertain, or not mentioned.Source: §A.1.1 · p. 45
  • Reviewers had strong inter-rater agreement (Fleiss’ κ=0.71) and were unanimous for 77% of labels; majority vote resolved 21% of disagreements, and the senior radiologist resolved 2%.Zero-round adjudication was performed on the reviewers’ annotations.Source: §A.1.1 · p. 45
  • 66% of Med-PaLM 2 labels matched the ground truth, while the original labels were correct in 19% of flagged cases.The revised labels are in preparation to be released.Source: §A.1.1 · p. 45
  • Table A.1 presents two Med-PaLM 2 prompt styles for extracting structured labels, including a template for identifying fractures.The prompts were applied to MIMIC-CXR radiology reports.Source: Table A.1 · p. 45, §A.1.1 · p. 45

A.1.2. Prompts for VQA and CXR classification evaluations

Med-Gemini used binary prompts for the top five MIMIC-CXR conditions because they yielded better macro F1 than multi-select prompts. Prompt templates were tailored to each model and evaluation dataset.

A.1.2. Prompts for VQA and CXR classification evaluations

  • Binary prompts yielded better macro F1 for Med-Gemini than multi-select prompts and were therefore used at evaluation.The explored options included binary questions for each of the top five conditions and multi-select prompts for all conditions together on the validation set.Source: §A.1.2 · p. 46
  • Gemini Ultra used binary prompts for the top five conditions and a multi-select prompt for normal/abnormal classification.Prompt templates were crafted for each model to optimize performance; Med-Gemini’s classification and VQA templates were very short because it was fine-tuned with clinical questions.Source: §A.1.2 · p. 46
  • VQA prompt templates were specified for each model and evaluation dataset, with the individual question substituted into the question placeholder.The table caption excludes formatting tokens.Source: Table A.2 · p. 46

A.1.3. New balanced splits for VQA-Rad dataset

To address image and patient overlap in VQA-Rad, the authors created roughly equal IMAGEID-based splits and balanced question difficulty and question types.

A.1.3. New balanced splits for VQA-Rad dataset

  • 202 of the 203 official test images also appear in training, so most test images are not held out from the train set.Some questions also share answers because VQA-Rad includes paraphrased questions.Source: §A.1.3 · p. 46
  • Building on Xu et al. (2023), the authors used IMAGEID-based splits to separate images and patients, then repartitioned the large test set into roughly equal train, validation, and test splits.A few remaining former test IMAGEIDs and their VQAs were assigned to the existing validation set.Source: §A.1.3 · p. 46
  • The splits approximately balanced open/closed question ratios within chest, head, and abdomen to align difficulty, and also equalized question-type distributions.Open-ended questions were considered more difficult for AI models.Source: §A.1.3 · pp. 46–47

A.1.4. Polygenic risk prediction … A.4. Supplementary Table for Performance Summary

The supplementary sections detail evaluation protocols and additional results across polygenic risk prediction, chest X-ray classification, and histopathology, then describe evaluation metrics and summarize aggregate Med-Gemini comparisons.

A.1.4. Polygenic risk prediction

  • Med-Gemini-Polygenic used PRS images and demographic information for outcome prompts and was directly prompted on OOD outcomes, while comparator linear probes transferred predictions from the most-correlated ID outcome.The OOD comparator outcome was selected by its highest Matthew’s correlation coefficient with the OOD outcome across training-set individuals.Source: §A.1.4 · p. 47

A.2.1. Data-efficient classification

  • About one order of magnitude below ELIXR at 64 samples, Med-Gemini visual-embedding linear probes otherwise showed robust data-efficient classification across five training sizes and two OOD datasets.Evaluation covered eight chest X-ray findings; the probes used frozen image embeddings.Source: Figure A.1 · p. 53, §A.2.1 · p. 53

A.2.2. Polygenic risk prediction

  • AUC metrics were relatively consistent across 400-person and larger balanced evaluation sets, with stable coronary artery disease and COPD results at 4,000 samples.The authors infer that the 400-person evaluation set represents overall model performance.Source: §A.2.2 · p. 53, Figure A.2 · p. 54, Figure A.3 · p. 55
  • Med-Gemini-Polygenic and an embeddings-and-demographics GBDT performed comparably and consistently beat the Best-PRS GBDT on ID outcomes; PRS-ensemble linear-model performance plateaued with at most 104 samples.The comparison attributes Med-Gemini-Polygenic’s performance to multiple PRSs and modeling nonlinear interactions between genomic representations and demographic information.Source: §A.2.2 · p. 54, Table A.12 · p. 57, Figure A.6 · p. 57

A.2.3. MIMIC-CXR classification

  • Revised MIMIC-CXR labels significantly improved measured chest X-ray classification F1-score compared with the original labels.The result demonstrates the impact of ground-truth label accuracy on model evaluation.Source: Table A.13 · p. 58, §A.2.3 · p. 58

A.2.4. Histopathology classification

  • Med-Gemini embeddings matched PathSSL on 6 of 9 in-distribution histopathology tasks, while Gemini and Med-Gemini-2D performed similarly overall.Med-Gemini trended toward higher mean AUC on 7 of 9 in-distribution tasks and both out-of-distribution tasks; 95% confidence intervals used blocked bootstrap resampling over test slides.Source: §A.2.4 · p. 58, Table A.14 · p. 59

A.3. Evaluation metrics

  • The evaluation used task-specific automated measures such as accuracy and AUC for classification and BLEU, Rouge-L, and RadGraph F1 for report generation; RadGraph F1 captures finding relationships but has limitations.RadGraph F1 may inherit entity- or relation-extraction errors, penalize partial matches, and fail to reflect differing clinical importance of errors.Source: §A.3 · pp. 60–61

A.4. Supplementary Table for Performance Summary

  • Table A.15 summarizes Med-Gemini against SoTA, Gemini, or a strong baseline where available, averaging pathology AUC across sub-datasets and report-generation results across normal and abnormal cases.The report-generation aggregate uses expert assessments of whether AI reports are superior or similar to original reports.Source: Table A.15 · p. 62, §A.4 · p. 62

Related papers

Loading 2405.03162v1…