Capabilities of Gemini Models in Medicine

Motivation and limitations

• Existing medical AI remains limited by uncertainty, confabulation, bias, tool use, clinician collaboration, and integration of complex multimodal longitudinal data.Although LLMs and LMMs show promise on medical question answering, reporting, and diagnostic dialogue, benchmark performance alone does not establish real-world utility.Source: §1 · p. 3

Model design

• Med-Gemini is a family of medicine-specialized models, with variants optimized for different capabilities and application scenarios rather than one generalist system.The models inherit Gemini’s language, conversational, multimodal, and long-context capabilities while accounting for training data, compute, and inference-latency trade-offs.Source: §1 · p. 4

Clinical reasoning

• 91.1% MedQA (USMLE) accuracy establishes Med-Gemini’s SoTA clinical reasoning, achieved with uncertainty-guided web search and exceeding prior Med-PaLM 2 models by 4.6%.Expert relabelling identified unanswerable and erroneous questions, supporting reliable characterization of the improvement; the strategy also reached SoTA on NEJM CPC and GeneTuring.Source: §1 · p. 4

Real-world utility

• Med-Gemini shows potential real-world utility in medical summarization, referral-letter generation, multimodal dialogue, education, clinician-facing tools, and biomedical research, but deployment requires rigorous further validation.The models are intended as assistive tools for expert clinicians, particularly in safety-critical applications such as diagnosis.Source: §1 · pp. 4–5

Multimodal capabilities

• Med-Gemini improves specialized medical multimodal performance through fine-tuning and customized encoders, reaching SoTA on Path-VQA, ECG-QA, and other benchmarks.The models also support clinically meaningful multimodal conversations across in-distribution and out-of-distribution modalities.Source: §1 · pp. 4–5

Long-context capabilities

• Med-Gemini demonstrates long-context reasoning on lengthy EHRs and medical videos, including needle-in-a-haystack retrieval, instructional video question answering, surgical action recognition, and CVS assessment.These capabilities target clinically meaningful decisions that require parsing large amounts of data and risk information overload.Source: §1 · pp. 4–5

2. Methods

• Med-Gemini adapts Gemini models through specialized reasoning, multimodal fine-tuning, custom encoders, and long-context chain-of-reasoning methods.The family includes Med-Gemini-M 1.0, Med-Gemini-L 1.0, Med-Gemini-M 1.5, and Med-Gemini-S 1.0, built on Gemini Pro, Ultra, and Nano variants.Source: §2 · pp. 5–6

2.1. Advanced reasoning via self-training and web search integration

• Med-Gemini-L 1.0 uses self-training with search and uncertainty-guided retrieval to generate, filter, and iteratively refine clinical reasoning with external evidence.Training creates MedQA-R and MedQA-RS datasets with synthetic reasoning and search context; inference invokes search when answer-choice entropy exceeds a threshold, then generates three targeted queries.Source: §2.1 · pp. 6–9

2.2. Multimodal understanding via fine-tuning and customized encoders

• Med-Gemini-M 1.5 is fine-tuned on eight multimodal tasks across six datasets, while Med-Gemini-S 1.0 adds a Flamingo-based encoder for raw 12-channel ECG question answering.The multimodal training covers radiology, pathology, dermatology, chest X-ray report generation, and image classification, with task mixtures weighted approximately by dataset size.Source: §2.2 · pp. 9–10

2.3. Long-context processing via instruction prompting and chain-of-reasoning

• For long-context medicine, Med-Gemini-M 1.5 uses in-context two-step reasoning to retrieve and interpret subtle EHR mentions, and whole-video prompting to analyze extended medical events.The EHR pipeline first retrieves evidence snippets and then determines entity presence; video prompting supports visual answer localization and Critical View of Safety assessment.Source: §2.3 · pp. 10–12

3. Evaluation

• Med-Gemini is evaluated across text reasoning, multimodal, and long-context medical tasks covering a wide range of capabilities.The evaluation includes clinical reasoning, visual question answering, long EHR understanding, medical video question answering, and surgical video assessment.Source: §3 · p. 12

3.1. Evaluation of advanced reasoning on text-based tasks

• Text-based evaluation measures Med-Gemini-L 1.0 on MedQA, NEJM CPC, and GeneTuring using accuracy-based metrics and uncertainty-guided search.MedQA uses four search iterations, while NEJM CPC and GeneTuring use one; physicians also relabel MedQA items to identify ambiguity and missing information.Source: §3.1 · p. 12
• Clinicians conduct blinded side-by-side comparisons of Med-Gemini-M 1.0 with human experts for medical summarization, referral-letter generation, and medical simplification.The tasks generate after-visit summaries, referral letters, and plain-language summaries from de-identified clinical or technical medical text.Source: §3.1 · p. 13

3.2. Evaluation of multimodal capabilities

• Multimodal evaluation covers seven VQA benchmarks spanning dermatology, radiology, pathology, cardiology, and cross-specialty medical datasets, using task-specific accuracy, exact-match, or token-level F1 metrics.Cross-specialty benchmarks are held out from training and fine-tuning, with evaluation focused on Med-Gemini-L 1.0 without multimodal fine-tuning; qualitative diagnostic dialogue is also assessed by expert clinicians.Source: §3.2 · p. 13

3.3. Evaluation of long-context capabilities on video and EHR tasks

• Long-context evaluation tests retrieval and reasoning over 200,000–700,000-word EHR examples, medical instructional videos, and laparoscopic surgical videos.The tasks include MIMIC-III-Needle-in-a-Haystack retrieval, video span localization, and Critical View of Safety assessment, with additional qualitative surgical action recognition.Source: §3.3 · pp. 13–14
• The long-context studies compare one-shot EHR performance with a heuristic baseline and quantify video localization and surgical safety prediction using precision, recall, IoU, mIoU, and accuracy.CVS assessment averages predictions across 572 annotated video clips, while EHR labels come from majority votes of three physicians.Source: §3.3 · p. 14

4.1. Med-Gemini demonstrates advanced reasoning on text-based tasks

• 91.1% accuracy on MedQA (USMLE) establishes a new SoTA, exceeding Med-PaLM 2 by 4.5% and MedPrompt by 0.9% through uncertainty-guided general web search.Med-Gemini-L 1.0 also exceeds AMIE by 13.2% on NEJM CPC top-10 accuracy and outperforms prior general-search models on seven GeneTuring modules.Source: Table 1 · p. 15, §4.1 · p. 15

4.1.1. Performance on long-form medical text generation

• Med-Gemini-M 1.0 was rated as good or better than experts more than half the time across after-visit summaries, referral letters, and medical simplification.For referral letters, generated responses were preferred or tied with experts across all evaluated samples.Source: §4.1.1 · p. 16, Figure 5 · p. 17

4.2. Med-Gemini demonstrates multimodal understanding across diverse tasks

• 44.5% average relative improvement over GPT-4V across seven multimodal benchmarks accompanied SoTA results on NEJM Image Challenge, USMLE-MM, and MMMU-HM.Med-Gemini-L 1.0 exceeded GPT-4V by 8.7%, 13.1%, and 2.6% on those three tasks, while other variants achieved SoTA on Path-VQA and ECG-QA.Source: §4.2 · pp. 17–18, Table 2 · p. 18

4.2.1. Preview of multimodal dialogue capabilities

• Qualitative out-of-distribution dialogues showed Med-Gemini-M 1.5 acquiring relevant images, diagnosing prurigo nodularis, analyzing chest X-rays, and explaining findings in non-technical language.Responses varied with prompting, and the authors state that real-world use requires considerable further development, evaluation, and validation.Source: §4.2.1 · p. 18, Figure 6 · p. 19, Figure 7 · p. 20

4.3. Med-Gemini shows long-context processing capability on long EHR and video tasks

• Med-Gemini-M 1.5 matched a task-dependent heuristic baseline for one-shot rare-condition retrieval from long EHRs, achieved SoTA on two MedVidQA MAVL tasks, and exceeded GPT-4V by 21% on CVS assessment.The supervised ResNet3D baseline remained better on CVS assessment, while seven inaccessible MedVidQA videos left 148 evaluable questions.Source: Table 3 · p. 21, §4.3 · p. 21

4.3.1. Applications of long-context capabilities in biomedicine

• Long-context demonstrations showed Med-Gemini-M 1.5 identifying surgical actions and conducting instructional video dialogue, with potential applications in surgical training and workflow analysis.The authors caution that more work is needed for nuanced capabilities such as anatomy grounding, educational assets, operative guidance, and patient-safety applications.Source: §4.3.1 · p. 23, Figure 11 · p. 25, Figure 12 · p. 26
• Med-Gemini-M 1.5 summarized extensive EHRs for conversational querying and synthesized 12 research papers into an accessible FTO–obesity mechanism with supporting experimental results.These examples suggest potential to reduce cognitive load for clinicians, patients, and genomic researchers, but require additional evaluation of clinically significant errors and bias.Source: §4.3.1 · pp. 23–24, Figure 13 · p. 27, Figure 14 · p. 28

Overall capabilities

• Med-Gemini demonstrates advances in clinical reasoning, multimodal understanding, and long-context processing across 25 tasks spanning 14 medical benchmarks.The benchmarks cover medical knowledge, clinical reasoning, genomics, waveforms, imaging, health records, and videos.Source: §5 · p. 29

MedQA performance

• Med-Gemini-L 1.0 achieves a new MedQA (USMLE) state of the art, but approximately 4% of questions lack information and 3% may contain labeling errors.The authors caution that benchmark gains may not directly reflect real-world clinical progress and that dataset size and quality require careful interpretation.Source: §5 · p. 29

Web search integration

• Web search integration may improve factual reliability by incorporating current medical knowledge, but authoritative-source restriction, multimodal retrieval, citation quality, and smaller-model use remain unstudied.These limitations motivate further research on search-result accuracy and relevance.Source: §5 · pp. 29, 31

Promising multimodal conversational capabilities

• Med-Gemini-M 1.5 shows promising untuned multimodal dialogue, including multi-turn interaction, information requests, explanations, and deference to human decisions, but clinical conversation was not rigorously benchmarked.Such capabilities could assist clinicians and patients while carrying significant risks.Source: §5 · p. 29

Opportunities with long-context processing

• Long-context processing enables few-shot retrieval and analysis of crucial information from lengthy health records and complex medical videos, potentially reducing clinicians’ cognitive load.The discussion also identifies future applications in genomic and multi-omic sequences, pathology, volumetric imaging, and integrated health-record analysis.Source: §5 · p. 30

Evaluation and responsible AI

• Responsible deployment of Gemini in healthcare requires attention to fairness, privacy, equity, transparency, accountability, bias, and sociotechnically grounded evaluation in specific clinical use-cases.The study considers 14 diverse, challenging benchmarks, while more than 350 medical benchmarks are available in the community.Source: §5 · pp. 31–32

• Gemini and Med-Gemini suggest expanded opportunities to accelerate biomedical discoveries and assist in healthcare delivery and experiences.Realizing this potential requires meticulous attention to system reliability and safety so AI can responsibly support scientific progress and medical care.Source: §6 · p. 32

9. Code Availability

• Model code and weights are not open-sourced because unmonitored use in medical settings poses safety risks.The authors instead plan validation with research partners, regulators, and providers before making the models available through Google Cloud APIs.Source: §9 · p. 32

A. Supplementary Table for Figure 1

• Table A1 aggregates Med-Gemini comparisons with prior SoTA and best GPT-4 methods across text-based, multimodal, and long-context tasks.Where published GPT-4 results were unavailable, the authors used public APIs with matched test sets and few-shot prompts; GPT-4 results were unavailable for three long-context tasks because of context-window limitations.Source: Table A1 · p. 43

B. Related Works

• Medical LLMs have progressed from general pretrained models and minimal fine-tuning toward specialized systems such as Med-PaLM, Med-PaLM 2, GPT-4 with Medprompt, and Med-PaLM-M.These systems target applications including diagnosis assistance, risk prediction, drug discovery, diagnostic dialogue, and psychiatric-function assessment, while this work uses Gemini through direct prompting or medical-task fine-tuning.Source: B. Related Works · p. 43
• Reasoning improvements combine prompting, process improvements, and external tools or knowledge, including chain-of-thought, decomposition, self-improvement, ensembles, retrieval, and web search.The authors integrate self-training with search to improve Med-Gemini’s reasoning capabilities.Source: B. Related Works · pp. 43–44
• Medical multimodal systems range from specialist models for individual disciplines to generalist systems such as Med-PaLM M and LLaVA-Med that address tasks across specialties.The report extends evidence for generalist multimodal medical capability with Med-Gemini while emphasizing application-specific trade-offs.Source: B. Related Works · p. 44
• Long-context medical modeling remains underexplored because most medical LLM evaluations use relatively short texts and single images.Prior work expanded clinical context from 512 to 4096 tokens and later toward 100K tokens, while Gemini advanced the boundary to one million tokens.Source: B. Related Works · pp. 44–45
• Med-Gemini addresses this gap by investigating long-context medical use cases involving video and long electronic health-record tasks.These settings target multimodal clinical information that conventional short-text and single-image evaluations do not capture.Source: B. Related Works · p. 45

• The instruction-fine-tuning dataset mixture and synthetic data were curated to improve Med-Gemini-L 1.0’s reasoning and ability to use web search.Source: Table C1 · p. 45
• The evaluation suite comprises benchmarks for text-based reasoning tasks.Source: Table C2 · p. 45

C.2. MedQA (USMLE) Relabeling

• The two-step protocol first assessed answerability, selected answers, missing information, and potential ambiguity, then revealed the MedQA ground truth to assess label errors and reconsidered answers.Ground truth was withheld initially to avoid bias when judging missing information, while multi-select responses allowed raters to identify questions with multiple acceptable answers.Source: §C.2 · pp. 45–46
• The study recruited 18 US primary care physicians, collected at least three independent ratings per question, and found that physicians took an average of 255 seconds per question.The original MedQA evaluation provided additional text material, whereas these raters were not instructed to use external material, though this was not explicitly controlled.Source: §C.2 · p. 46
• Agreement exceeded 87% for missing information, label errors, and ambiguity, but answer overlap was typically about 75% after ground-truth disclosure versus about 50% without it.Ambiguity was defined as selecting multiple options after revealing the ground truth; raters selected an average of 1.065 options before disclosure and 1.119 afterward.Source: §C.2 · pp. 46–47
• Filtering label errors had the biggest impact on Med-Gemini-L 1.0’s performance, while filtering missing-information or ambiguous questions reduced the remaining questions without significantly changing accuracy.Results were estimated with 1,000 bootstrap repetitions using majority or unanimous voting; accuracy and the remaining-question fraction are reported in Figure C1.Source: Figure C1 · p. 47, §C.2 · p. 47

Specialty breakdown

• Table C4 reports Top-1 and Top-10 performance on NEJM clinical pathology case studies across specialties with at least 10 cases.The results are organized by each case’s primary specialty as identified by NEJM.Source: §C.3 · p. 47, Table C4 · p. 48
• In most specialties, including Internal Medicine, Pediatrics, and Psychiatry, the best Top-1 and Top-10 performance is achieved by either Med-Gemini-L 1.0 without search or the model with search.The breakdown covers specialties with at least 10 cases.Source: §C.3 · p. 47, Table C4 · p. 48

C.4. Real-world use cases for advanced reasoning on text-based tasks

• Med-Gemini-M 1.0 is evaluated on three long-form text-generation tasks: after-visit summaries, referral letters, and simplified systematic-review summaries.Summary results are reported in Figure 5, with additional evaluation axes in Table C5.Source: §C.4 · p. 48, Table C5 · p. 48
• After-visit summaries are generated from 31 de-identified outpatient H&P notes and compared with clinician-written summaries by U.S.-based physician raters.Raters assess accuracy, coverage, coherence, succinctness, and overall quality; the model is prompted to produce structured, patient-facing summaries at a sixth-grade reading level.Source: §C.4 · pp. 48–50
• Clinician-authored reference documents are quality-controlled before comparison: AVSs are refined by a second clinician, referral letters are reviewed by a board-certified clinician, and plain-language summaries come from original review authors.This establishes expert-authored comparators for the three task evaluations.Source: §C.4 · pp. 48, 50, 51
• The after-visit-summary prompt extracts 12 patient-facing fields covering visit details, diagnoses, medications, measurements, follow-up actions, warning signs, and provider comments.The instructions require avoiding fabricated numbers, using patient-centered language, and writing at a sixth-grade reading level without abbreviations or jargon.Source: §C.4 · p. 49
• Referral letters are generated from de-identified notes recommending inter-specialty evaluation and compared blindly with clinician-generated letters across 25 examples.Three U.S. board-certified physicians evaluate every example, preferring the draft they would use as the starting point for a final letter while considering factual alignment with the note.Source: §C.4 · pp. 50–51
• Plain-language summaries are generated from 25 Cochrane systematic-review abstracts and compared with author-written summaries by blinded clinician raters.Evaluation covers grounding, coverage, succinctness, reading level, and overall quality, with ratings aggregated similarly to the referral-letter task.Source: §C.4 · pp. 51–52

Dataset composition

• The fine-tuning suite uses MultiMedBench’s four image-text datasets—Slake-VQA, Path-VQA, MIMIC-CXR, and PAD-UFES-20—plus ROCO, while an ECG-QA subset develops Med-Gemini-S 1.0’s health-signal encoder (Tanno et al., 2024; Tu et al., 2024a).The datasets cover radiology, pathology, dermatology, and multimodal medical-image captioning, with ECG-QA providing sensor-text data.Source: §D.1 · p. 52

MIMIC-CXR

• MIMIC-CXR supports binary abnormality classification, abnormality VQA, synthetic CXR VQA, and report generation using 377110 images from 65379 patients, with filtering to reduce hallucinated references to prior studies.The official split is used, and VQA tasks serve as auxiliary objectives for report generation; evaluation is deferred to a subsequent paper.Source: §D.1 · pp. 52–53

PAD-UFES-20

• PAD-UFES-20 contains 2298 smartphone skin-lesion images spanning six lesion types and supports image-plus-clinical-feature classification with augmentation, upsampling, and image-level and patient-level splits.The fine-tuning setup includes auxiliary tasks for class imbalance and reports close-ended multiple-choice prediction accuracy.Source: §D.1 · p. 53

Pathology and radiology VQA

• Path-VQA provides 998 pathology images and 32799 QA pairs, while Slake-VQA provides 642 bilingual radiology images and 14028 QA pairs across multiple modalities, organs, diseases, and question types.Path-VQA uses official train/validation/test splits, and Slake-VQA covers CT, MRI, and chest X-rays in English and Chinese.Source: §D.1 · pp. 53–54

Captioning and health signals

• ROCO contributes 29907 licensed image-caption pairs for captioning, and ECG-QA supplies single-ECG sensor-text questions with 159306 training, 31137 validation, and 41093 test samples.ROCO includes radiology and non-radiology images under specified open licenses, while ECG-QA includes yes/no, multiple-choice, and open-ended question formats.Source: §D.1 · p. 54

D.2. Multimodal evaluation datasets

• The evaluation adds three out-of-distribution datasets to assess Gemini’s multimodal capabilities.These complement the in-distribution datasets described elsewhere.Source: §D.2 · p. 54
• The NEJM Image Challenge dataset contains 942 clinical cases collected from 2005–2023, with images, questions, five answer options, and correct answers.The challenge spans radiographic, dermatologic, electrocardiographic, histopathologic, endoscopic, and ophthalmoscopic images (The New England Journal of Medicine, 2024).Source: §D.2 · pp. 54–55
• For fair comparison, 934 NEJM cases were ultimately evaluated after excluding two cases blocked by GPT-4V’s sexually explicit-image filters.The excluded cases were 20160519 and 20111103 (Buckley et al., 2023).Source: §D.2 · p. 55
• USMLE-MM comprises 46 image-containing multiple-choice questions from sample USMLE preparation exams, while MMMU-HM contains 150 medical questions across six health-related domains.MMMU-HM covers basic medical science, clinical medicine, diagnostics and laboratory medicine, pharmacy, and public health (Yue et al., 2023).Source: §D.2 · p. 55

D.3. Additional results for multimodal tasks

• On single ECG questions, Med-Gemini-S 1.0 achieved 57.7% accuracy with frozen Gemini and 58.4% with unfrozen Gemini, outperforming GPT-4’s 51.6% by 6.1% and M3AE’s 57.6% by 0.8%, respectively.Source: §D.3 · p. 55
• Med-Gemini-S 1.0 uses an ECG-specific encoder to process raw biomedical signals for ECG-QA.Gemini 1.0 Nano is augmented with the encoder before fine-tuning.Source: §D.3 · p. 55
• The evaluation compares frozen and unfrozen Gemini variants within Med-Gemini-S 1.0.The frozen configuration keeps Gemini unchanged, whereas the unfrozen configuration fine-tunes Gemini.Source: §D.3 · p. 55
• The frozen Med-Gemini-S 1.0 variant reaches 57.7% accuracy on single ECG questions.This is compared with GPT-4 using SE-WRN ECG features in input prompts.Source: §D.3 · p. 55
• The unfrozen Med-Gemini-S 1.0 variant reaches 58.4% accuracy on single ECG questions, exceeding M3AE’s 57.6%.M3AE is the ECG foundation-model baseline used for comparison.Source: §D.3 · p. 55

E. Additional details on long-context understanding tasks

• No long-context understanding result is supplied for subsection E.The provided passage reports only the ECG-QA results from subsection D.3.Source: §D.3 · p. 55

MIMIC-III Needle-in-a-Haystack

• MIMIC-III Needle-in-a-Haystack contains 200 human-labeled examples from 44 patients, testing binary condition retrieval across medical records ranging from 200K to 700K words.Examples comprise 121 positive and 79 negative majority-vote labels, with Krippendorff’s alpha of 0.77.Source: §E.1 · pp. 55–56

Medical Instructional Video QA

• MedVidQA is a video-language cross-modal dataset for Medical Visual Answer Localization, comprising 3,010 health-related instructional questions for 899 videos.Source: §E.1 · p. 56

Cholec80 and Cholec80-CVS

• Cholec80 contains 80 laparoscopic cholecystectomy videos, while Cholec80-CVS adds surgeon annotations for 572 segments scoring three Critical View of Safety criteria.For comparison, longer clips are filtered because GPT-4V accepts at most 300 low-resolution sampled frames; Resnet3D uses five consecutive validation folds.Source: §E.1 · p. 57

Rater protocol

• Three independent raters evaluated each of the benchmark’s 200 example questions to assess reliability.The agreement analysis was designed to establish the reliability of the EHR benchmark.Source: §E.2 · p. 58

Metrics

• Agreement was quantified using Jaccard similarity for rater-selection overlap and Krippendorff’s alpha for multiple-rater reliability.The Jaccard measures distinguish unanimous selections from selections made by at least two raters.Source: §E.2 · p. 58

Agreement results

• 0.83 unanimous-selection Jaccard similarity and 0.915 Jaccard similarity for agreement among at least two of three raters demonstrate strong consistency, while Krippendorff’s alpha of 0.77 indicates good agreement on medical-condition identification.These are the reported rater-agreement metrics for the long EHR understanding task.Source: §E.2 · p. 58, Table E3 · p. 58