PaperlayerView exact revision on arXiv

Source-linked AI summary

MedGemma Technical Report

Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Stefanie Anna Baby, Susanna Maria Baby, Jeremy Lai, Samuel Schmidgall, Lu Yang, Kejia Chen, Per Bjornsson, Shashir Reddy, Ryan Brush, Kenneth Philbrick, Mercy Asiedu, Ines Mezerreg, Howard Hu, Howard Yang, Richa Tiwari, Sunny Jansen, Preeti Singh, Yun Liu, Shekoofeh Azizi, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Riviere, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Elena Buchatskaya, Jean-Baptiste Alayrac, Dmitry Lepikhin, Vlad Feinberg, Sebastian Borgeaud, Alek Andreev, Cassidy Hardin, Robert Dadashi, Léonard Hussenot, Armand Joulin, Olivier Bachem, Yossi Matias, Katherine Chou, Avinatan Hassidim, Kavi Goel, Clement Farabet, Joelle Barral, Tris Warkentin, Jonathon Shlens, David Fleet, Victor Cotruta, Omar Sanseviero, Gus Martins, Phoebe Kirk, Anand Rao, Shravya Shetty, David F. Steiner, Can Kirmizibayrak, Rory Pilgrim, Daniel Golden, Lin Yang

arXiv:2507.05201v4cs.AIcs.CLcs.CV

TL;DR

Healthcare AI needs foundation models that can handle diverse medical data and tasks while reducing reliance on task-specific tuning. The paper introduces Gemma 3-based MedGemma models and the MedSigLIP encoder, which show strong performance across medical capabilities and improve further with fine-tuning. However, automated benchmarks remain only an initial step toward validating real-world utility.

  • Problem

    Healthcare AI requires foundation models that address diverse data, complex tasks, privacy constraints, and the need for less task-specific tuning data.

  • Method

    The paper develops medically tuned MedGemma vision-language models based on Gemma 3 and a MedSigLIP image encoder, then evaluates them across medical and general-purpose tasks.

  • Results

    MedGemma and MedSigLIP show strong capabilities across medical benchmarks, with further performance improvements after fine-tuning and performance comparable to or exceeding specialized image encoders.

  • Takeaways & Limitations

    The collection provides open medical image and text capabilities that can serve as a starting point for developing healthcare AI applications.

  • Takeaways & Limitations

    Automated benchmarks are only a first step toward real-world utility, and some may be near performance saturation; further evaluation on challenging benchmarks is needed.

Abstract

from arXiv · show

Artificial intelligence (AI) has significant potential in healthcare applications, but its training and deployment faces challenges due to healthcare's diverse data, complex tasks, and the need to preserve privacy. Foundation models that perform well on medical tasks and require less task-specific tuning data are critical to accelerate the development of healthcare AI applications. We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B. MedGemma demonstrates advanced medical understanding and reasoning on images and text, significantly exceeding the performance of similar-sized generative models and approaching the performance of task-specific models, while maintaining the general capabilities of the Gemma 3 base models. For out-of-distribution tasks, MedGemma achieves 2.6-10% improvement on medical multimodal question answering, 15.5-18.1% improvement on chest X-ray finding classification, and 10.8% improvement on agentic evaluations compared to the base models. Fine-tuning MedGemma further improves performance in subdomains, reducing errors in electronic health record information retrieval by 50% and reaching comparable performance to existing specialized state-of-the-art methods for pneumothorax classification and histopathology patch classification. We additionally introduce MedSigLIP, a medically-tuned vision encoder derived from SigLIP. MedSigLIP powers the visual understanding capabilities of MedGemma and as an encoder achieves comparable or better performance than specialized medical image encoders. Taken together, the MedGemma collection provides a strong foundation of medical image and text capabilities, with potential to significantly accelerate medical research and development of downstream applications. The MedGemma collection, including tutorials and model weights, can be found at https://goo.gle/medgemma.

1. Introduction

MedGemma addresses the limited medical understanding of general-purpose multimodal models with open, medically tuned vision-language models built on Gemma 3. The collection includes multimodal and text-only MedGemma variants plus the standalone MedSigLIP medical image encoder.

  • General-purpose multimodal models can lack nuanced medical understanding and robust interpretation and reasoning over medical data.
  • MedGemma is an open suite of medically tuned vision-language foundation models built on Gemma 3 while retaining its general-purpose capabilities.
  • The collection focuses on a 4B model accepting text, images, or both, and a 27B model optimized for text-only inputs; both output text.
  • MedGemma 4B performs strongly on VQA benchmarks, while both MedGemma variants are competitive on challenging text-only medical benchmarks and improve further after subdomain fine-tuning.The fine-tuned subdomains include chest X-ray reporting, histopathology classification, and electronic health record information retrieval.
  • A multimodal MedGemma 27B variant is released, but its evaluation remains preliminary; references to MedGemma 27B otherwise mean the text-only variant.
  • MedSigLIP is a standalone 400M-parameter medical image encoder that powers MedGemma’s image interpretation and supports data-efficient, zero-shot classification and retrieval.Its standalone performance is comparable to or exceeds that of specialized image encoders.

2. Methods

MedGemma training combines general-purpose replay with medical text and multimodal data, then adapts Gemma 3’s architecture through vision-encoder enhancement, pretraining, and post-training. The setup retains general visual-language capabilities while introducing medical specialization.

  • Medical training and evaluation datasets largely follow Med-Gemini, with general-purpose SigLIP and Gemma 3 data mixtures replayed during pretraining.
  • Text training uses teacher-generated responses and logits from multiple medical QA datasets plus approximately 200,000 synthetic medical questions.
  • Multimodal training focuses on 2D medical images, excludes 3D volumes and genomics, removes PathVQA and MedVQA over potential data-quality issues, and omits PAD-UFES-20.
  • The Gemma 3-compatible architecture uses an 896×896 image input, arbitrary image-text interleaving, and a 128k context length.
  • The 4B multimodal model uses all training stages, whereas the text-only 27B version uses only post-training.
  • The vision encoder is fine-tuned with over 33M medical image-text pairs while retaining original SigLIP data, and multimodal pretraining readapts the language model to the enhanced encoder.
  • Post-training adds medical text to distillation and medical image-text data to reinforcement learning; multimodal post-training uses reinforcement learning because it generalized better than supervised fine-tuning.

3. MedGemma Evaluations

The evaluation covers text and image classification, visual question answering, chest X-ray report generation, agentic behavior, and general-purpose benchmarks. It uses public medical datasets, defined prompting and sampling settings, and task-specific metrics and splits.

  • MedGemma is evaluated on text question-answering, image classification, visual question answering, chest X-ray report generation, agentic behavior, and non-medical benchmarks.
  • Unless otherwise reported, each evaluation uses one inference run per example, with temperature 0.0 for MedGemma on medical benchmarks and default temperatures elsewhere.
  • Medical QA evaluation uses official public test splits, while MedXpertQA is treated as out-of-distribution because it was not used in training.
  • Image classification spans chest X-ray, dermatology, histopathology, and retinal fundus datasets, using accuracy or macro F1 for common findings or diagnoses.
  • Chest X-ray classification covers specified findings across MIMIC-CXR, CheXpert, and ChestX-ray14, with alternative MIMIC-CXR label treatments and adjudicated labels for CXR14.
  • Radiology VQA uses tokenized F1 across open and closed questions plus yes/no accuracy, with a contamination-avoiding VQA-RAD split.
  • Chest X-ray report generation compares MedGemma reports with radiologist reports using RadGraph F1, and also includes a human expert comparison on 306 cases.
  • AgentClinic evaluates a physician-agent setting requiring history taking, medical-exam ordering and interpretation, and operation under incomplete information; general benchmarks assess specialization tradeoffs.

4. MedGemma Results

MedGemma performs strongly across text, image, visual-question-answering, report-generation, and agentic medical tasks while retaining general-purpose capabilities. Its medical specialization offers substantial size-to-performance and computational-cost advantages, though benchmark leakage remains a concern.

  • Text question-answering: MedGemma outperformed same-sized Gemma 3 variants across evaluated text-only biomedical question-answering tasks and was competitive with much larger models in many cases.Evaluations covered MedQA, MedMCQA, PubMedQA, MMLU subsets, AfriMed, and the out-of-distribution MedXpertQA benchmark.
  • Medical image classification: MedGemma substantially outperformed Gemma 3 baselines on medical image classification and also exceeded much larger API-based models.The evaluations covered radiology, histopathology, dermatology, and ophthalmology, using zero-shot generative classification.
  • Medical visual question answering: MedGemma 4B outperformed same-sized Gemma 3 on SLAKE and VQA-RAD while remaining competitive with much larger models on medical visual question answering.Comparisons exclude discriminative embedding-based approaches and include externally reported metrics with methodological qualifications.
  • Chest X-ray report generation: 81% of generated CXR reports resulted in the same or superior clinical decision as the original reports, compared with 73% for larger Med-Gemini reports in a similar study.Expert review found 68% of reports in normal studies and 49% in abnormal studies equal or superior to the originals.
  • Medical agentic behavior: MedGemma 27B outperformed Gemma 3 27B in AgentClinic, exceeded human physicians on AgentClinic-MedQA, and approached much larger models on both evaluated environments.The 4B variants were not well suited to this agentic task and had difficulty following the framework’s system instructions.
  • General-purpose capabilities and observed themes: MedGemma showed only minor decreases relative to same-sized Gemma 3 models on general benchmarks, while smaller specialized models offered strong size-to-performance efficiency and up to a 500fold computational-cost difference.Performance advantages were especially pronounced in medical vision tasks, but older benchmarks may permit test-data leakage.

5. MedGemma Fine-tuning Demonstration

Fine-tuning experiments demonstrate that MedGemma can be adapted to specialized multimodal and long-context EHR tasks. Adaptation improved task performance, including new state-of-the-art CXR report generation and gains concentrated in inter-dependent-record reasoning.

  • Fine-tuning demonstration: The study fine-tuned MedGemma on CXR report generation, pneumothorax classification, histopathology classification, and long-context EHR question answering.The multimodal tasks used supervised fine-tuning, while the EHR task used reinforcement learning; method choice depends on task and data conditions.
  • Fine-tuning demonstration: Inputs were adapted to each task: image plus indication for CXR reporting, and image plus multiple-choice prompt for pneumothorax and histopathology classification.For CXR reporting, the model generated the findings and impression sections.
  • Fine-tuning results: Fine-tuned MedGemma models more closely approached specialized state-of-the-art systems, with MedGemma 4B achieving a new SOTA RadGraph F1 score of 30.3 on CXR report generation.The final checkpoints were selected using validation-set performance after a single training epoch.
  • EHRQA benchmark: EHRQA provides programmatically generated QA pairs from longitudinal outpatient synthetic FHIR records, spanning Conditions, Medications, Observations, and other resource types.The dataset contains 81 records split into 42 training, 20 validation, and 19 test records; it lacks clinical notes.
  • EHR information retrieval and reasoning: Reinforcement learning substantially improved MedGemma 27B’s EHRQA accuracy, closing its gap with top-performing models.The greatest gains occurred on questions requiring reasoning across inter-dependent records.

6. MedSigLIP Evaluations

MedSigLIP was evaluated as a medical image encoder using zero-shot and data-efficient linear-probe classification across four imaging modalities. These evaluations measure baseline recognition and performance after lightweight classifier training.

  • Evaluation datasets: MedSigLIP evaluation covered chest X-ray, dermatology, ophthalmology, and histopathology datasets.The chest X-ray evaluation followed the ELIXR framework and included seven findings for linear probing.
  • Zero-shot evaluation: Zero-shot classification used text prompts for each class, averaged multiple prompt embeddings when needed, and compared image-text cosine similarities before softmax scoring and AUC evaluation.This approach measures baseline performance without task-specific classifier training.
  • Linear-probe evaluation: Linear-probe classification extracted MedSigLIP image embeddings and trained logistic regression on the training set, selecting hyperparameters on validation data before test evaluation.The text encoder was not used for linear probing.

7. MedSigLIP Results

MedSigLIP provides strong multi-domain medical image encoding, often matching or exceeding specialized models in zero-shot and linear-probe evaluations. Its CXR results are competitive despite lower image resolution, while task-specific training can further improve performance.

  • Overview: MedSigLIP supports zero-shot and linear-probe classification and retrieval across medical imaging domains.It is evaluated against corresponding HAI-DEF models, with zero-shot comparison unavailable for image-only Derm Foundation and Path Foundation models.
  • Chest X-ray: 2.0% higher average zero-shot CXR performance than CXR Foundation, despite MedSigLIP’s lower image resolution and multi-domain expertise.Fracture classification improved over ELIXR by 7.1%.
  • Chest X-ray: MedSigLIP performed strongly across seven CXR findings when linear-probe training used 512 or more examples.Figure 5 reports average results on CheXpert and CXR14 compared with HAI-DEF’s ELIXR-based CXR Foundation model.
  • Other domains: Dermatology zero-shot and linear-probe results outperformed Derm Foundation linear probes across 79 skin conditions.The comparison concerns distinguishing among 79 skin conditions.
  • Other domains: 9.8% higher linear-probe than zero-shot performance on five-class diabetic retinopathy classification, with no HAI-DEF comparator available.Histopathology linear probing modestly improved over zero-shot but remained 1.9% below Path Foundation linear-probe performance on average.
  • Overall findings: MedSigLIP offers strong single-model baseline performance across medical domains, while task-specific images and logistic regression can yield stronger results.Its zero-shot classification generally compares well with linear probes from dedicated domain-specific models.

8. Discussion

The discussion presents MedGemma and MedSigLIP as open medical foundation models with advantages over general models and potential across healthcare development settings. It also emphasizes that benchmark performance is only an initial snapshot requiring further real-world validation.

  • Findings: MedGemma and MedSigLIP showed strong performance across clinical reasoning, biomedical knowledge, report generation, and medical image classification benchmarks.Fine-tuning further improved performance and supports their use as starting points for healthcare AI applications.
  • Findings: Domain-specific pre-training and post-training improved MedGemma over base Gemma 3 models across all evaluated benchmarks and reached performance on par with much larger models.These advantages are presented relative to general models.
  • Developer considerations: MedGemma may be preferred over general API-based models when users need frozen behavior, lower costs, local or offline operation, medical capabilities, or adaptation control.Large models such as Gemini remain viable when optimal broad performance is required without these constraints.
  • Potential applications: The collection may support medical image retrieval, similar-case referencing, research cohorts, educational tools, and integration across radiology, histopathology, and ophthalmology data.Its multimodal capabilities include access to image and text embeddings.
  • Limitations: Automated benchmarks are only an initial step toward validating real-world utility, and some may be near saturation with limited headroom.The authors call for newer, higher-quality, more challenging benchmarks and further study of application development and agentic frameworks.
  • Open release: Open release is intended to facilitate community evaluation, improvement, and adaptation, with predictability and flexibility for healthcare developers.The authors identify openness as critical for many healthcare applications.

9. Conclusion

The conclusion reports robust capabilities for MedGemma across vision-language and text-only medical tasks and for MedSigLIP across medical domains. It pairs these possibilities with a requirement for careful downstream safety, performance, and reliability validation.

  • Conclusion: MedGemma demonstrates robust capabilities across a variety of vision-language and text-only medical tasks.
  • Conclusion: MedSigLIP demonstrates robust multi-domain capabilities and can serve as a strong medical foundation model.
  • Conclusion: The models’ breadth and efficiency offer possibilities across a range of medical use cases.
  • Conclusion: Downstream medical applications require thoughtful validation of safety, performance, and reliability.
  • Conclusion: Permissively licensed release gives the developer community access to models intended to enable useful and innovative medical applications.

11. Contributions and Acknowledgments

The contribution and acknowledgment section lists the paper’s authors and identifies co-first and co-last authorship roles.

  • Contributors: The paper credits Tiam Jaroensri through Yun Liu among its listed contributors.
  • Contributors: The listed contributors continue from Shekoofeh Azizi through Jean-Baptiste Alayrac.
  • Contributors: Additional contributors include Dmitry Lepikhин through David Fleet, followed by Victor Cotruta through Rory Pilgrim.
  • Authorship: The authorship note identifies co-first and co-last author designations.

Use of AI in Manuscript Preparation

The manuscript reports that its introduction was drafted manually and refined with Gemini 2.5 Pro, while radiology-report evaluation used a human scoring rubric.

  • The introduction was drafted manually, refined using Gemini 2.5 Pro, and manually checked for content accuracy.
  • The authors state that they take full responsibility for the manuscript’s content.
  • Table A1 defines the rubric used by radiologists to score generated chest X-ray reports.

B. Evaluation Prompts

The evaluation appendix specifies prompts, system instructions, class definitions, and task-specific prompt sets for ophthalmology, chest X-ray, and histopathology evaluations.

  • B. Evaluation Prompts: MedSigLIP evaluation used prompt-based zero-shot tasks, with class definitions listed in Table A2.
  • B. Evaluation Prompts: Evaluation settings used MedGemma temperature 0.0, model-default temperatures otherwise, and separate radiology and general medical system prompts.
  • B. Evaluation Prompts: Ophthalmology zero-shot prompts are provided in Table A3.
  • B. Evaluation Prompts: Chest X-ray zero-shot prompts are provided in Table A4.
  • B. Evaluation Prompts: Histopathology zero-shot prompts are provided in Table A5 for tissue and disease classification tasks.
  • B. Evaluation Prompts: Histopathology prompts include breast-carcinoma nucleopleomorphism scores 1 and 3 and tubule-formation scores 1 and 2.
  • B. Evaluation Prompts: Prostate prompts distinguish benign tissue from Gleason pattern 4 and pattern 5 prostate cancer.
  • B. Evaluation Prompts: The Cancer Genome Atlas prompts include bladder cancer and urothelial carcinoma examples.

E. Additional medical reasoning examples

The appendix presents additional clinical reasoning examples, expert commentary, and preliminary results for a multimodal MedGemma 27B variant.

  • Expert commentary: The appendix identifies the examples as open clinical reasoning cases in cardiology and infectious disease, including specialist commentary.
  • Cardiology examples: The cardiology examples assess MedGemma 27B reasoning about Entresto in a patient with heart failure and reduced ejection fraction.
  • Cardiology examples: The cardiology commentary notes an incorrect lowest Entresto dose and recommends considering estimated GFR, carvedilol titration, and empagliflozin.
  • Infectious-disease examples: The infectious-disease example prioritizes leptospirosis while considering dengue, hantavirus, rickettsial infections, and other causes of multisystem illness.
  • Infectious-disease examples: Specific serological and PCR testing was recommended for leptospirosis, dengue, and hantavirus, with urine PCR potentially useful after two weeks of symptoms.
  • Expert commentary: Expert commentary praised the infectious-disease answer’s structure and focus but noted geographic, testing-breadth, and turnaround-time concerns.
  • MedGemma 27B multimodal: The MedGemma 27B multimodal variant adds EHRQA and Chest ImaGenome training data for EHR understanding and chest X-ray anatomy localization.
  • MedGemma 27B multimodal: Preliminary results indicate improved language capabilities, EHR understanding, and anatomy localization relative to the 4B multimodal variant, while formal evaluation remains ongoing.
Loading 2507.05201v4…