Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis

1.2 Sample selection.

• Case studies draw radiology tasks from Radiopaedia, pathology images from PathologyOutlines, and localization samples from public segmentation and detection benchmarks.Source: §1.2 · p. 9

1.2.1 Case selection.

• Cases were selected from 2023 web appearances, favored board-reviewed Radiopaedia annotations with completion ratios above 90%, and aimed to cover modalities and multimodal cases across systems.Source: §1.2.1 · p. 10

1.2.2 Image processing.

• Because GPT-4V accepts at most four images, the evaluation uses selected key slices or prioritizes up to four images, with Radiopaedia’s default intensity window and specified rescaling for localization.For localization, CT images are clipped to [-300, 300], other images to the 0.5th and 99.5th intensity percentiles, and all images rescaled to [0, 1].Source: §1.2.2 · p. 10

1.2.3 Question prompts.

• Questions vary slightly across prompts to test response robustness, with report generation and diagnosis as the primary emphasis.Source: §1.2.3 · p. 10

1.2.4 Annotation or reference caption.

• Radiopaedia descriptions provide accuracy-checked but nonstandard references; only direct conflicts are marked definitely incorrect, leaving other claims for expert judgment.Source: §1.2.4 · pp. 10–11

1.3 Testing procedure.

• Testing uses fresh chats for new cases, two-round pathology queries, and repeated localization predictions scored by IOU, with both highest and averaged boxes assessed.Source: §1.3 · p. 11

1.4 Case demonstration.

• Case figures show prompts, GPT-4V responses, and references; red, green, and yellow distinguish incorrect, correct, and uncertain content, while localization figures contrast predicted and ground-truth boxes.Source: §1.4 · p. 11

1.5 Limitations of this report

• The authors call the evaluation non-exhaustive: webpage-only manual uploads limit it to qualitative analysis, while abnormal-case sampling, incomplete references, and 2D inputs constrain its scope.Source: §1.5 · pp. 11–12

2.1 GPT-4V can recognize the modality and anatomy of medical images.

• GPT-4V usually recognized modality and anatomy and showed promising plane recognition, but it sometimes confused modalities, anatomy, or the plane of an image.Examples include misclassifying pelvic MRI as knee MRI and uncertainty between breast MRI and CT.Source: §2.1 · p. 14

2.2 GPT-4V can hardly make accurate diagnoses.

• GPT-4V often failed to provide accurate image-based diagnoses, sometimes refusing, listing possible diseases, or failing completely; it could localize abnormalities when contrast was significant.The authors also describe safeguards that may lead it to refuse diagnosis or express uncertainty.Source: §2.2 · p. 14, §2.12 · p. 16

2.3 GPT-4V can generate structured reports, with incorrect content.

• Reports followed a structured template and generally covered relevant anatomy, but their content could be incorrect or fail to synthesize images from different modalities.For multimodal inputs, reports could devolve into separate image descriptions without a comprehensive conclusion.Source: §2.3 · p. 14

2.4 GPT-4V can recognize text and markers in images, but lacks understanding.

• GPT-4V could read text and visual markers and identify medical devices, but text could distract from image content, annotations could be misread, and OCR was not robust.Arrows and circles could direct attention to regions of interest; device identification was described as satisfying.Source: §2.4 · p. 15, §2.5 · p. 15

2.6 GPT-4V faces difficulties on analyzing multiple images.

• For multiple images, GPT-4V often described images separately; it could recognize views and sometimes improve over a single view, but struggled to combine modalities and could ignore context at four images.It could identify corresponding views and occasionally provide a comprehensive diagnosis when it understood their relationship.Source: §2.6 · p. 15, §2.6.1 · p. 15, §2.6.2 · p. 15

2.7 GPT-4V’s prediction heavily relies on patient’s medical history.

• Patient information and medical history affected outputs; context could focus attention on relevant areas, while missing context could prompt a normalcy prediction.With comprehensive history, the model could infer potential abnormalities using past medical conditions.Source: §2.7 · p. 15

2.8 GPT-4V cannot localize the anatomical structures or anomalies in medical images.

• Localization was poor: predictions could be far from ground truth, vary across repeated evaluations, or reflect positional biases; the average IoU across cases was 0.16.Some predictions received 0.0 IoU, and averaging repeated predictions could yield low IoU even when one prediction was acceptable.Source: §2.8 · p. 16

2.9 GPT-4V can change answers with guidance in multi-round interaction.

• User guidance could correct an initial image misclassification, but reports could remain inaccurate despite obvious abnormalities, and performance varied across modalities and prompts.In one example, correction of a pelvic-versus-knee MRI misclassification led to an accurate diagnosis; the same image also received inconsistent judgments under different prompts.Source: §2.9 · p. 16, §2.10 · p. 16, §2.11 · p. 16

3.1 Central Nervous System; 3.2 Head and Neck

• Central nervous system and head-and-neck cases span CT, MRI, X-ray, ultrasound, and angiography; CNS cases also include nuclear medicine imaging.The head-and-neck scope excludes the central nervous system.Source: §3.1 · p. 16, §3.2 · p. 17

3.3 Cardiac; 3.4 Chest; 3.5 Hematology

• Cardiac cases mainly use CT and X-rays, while chest cases span PET, CT, MRI, X-ray, and ultrasound, and hematology cases use CT, MRI, X-ray, and ultrasound.Source: §3.3 · p. 17, §3.4 · p. 17, §3.5 · p. 17

3.6 Hepatobiliary; 3.7 Gastrointestinal; 3.8 Urogenital; 3.9 Gynecology

• Hepatobiliary, gastrointestinal, urogenital, and gynecology cases cover liver and biliary, digestive, urinary, and female reproductive systems.The listed imaging includes X-ray, CT, MRI, and ultrasound for gastrointestinal and urogenital examinations.Source: §3.6 · p. 17, §3.7 · p. 17, §3.8 · p. 17

3.10 Obstetrics; 3.11 Breast; 3.12 Musculoskeletal

• Obstetrics examples are primarily ultrasound, breast cases center on mammography, and musculoskeletal cases cover the muscular and skeletal systems.Obstetrics may also use MRI, CT, and X-ray; breast examinations may also use ultrasound, MRI, and CT.Source: §3.10 · p. 18, §3.11 · p. 18, §3.12 · p. 18

3.13 Spine; 3.14 Vascular; 3.15 Oncology; 3.16 Trauma; 3.17 Pediatrics

• Spine, vascular, oncology, trauma, and pediatric examples span multiple modalities, including X-ray, CT, MRI, ultrasound, and nuclear medicine.Oncology examples also include PET; X-ray is the most commonly used trauma modality, while pediatric cases are framed as hard examples because most patients are adults.Source: §3.13 · p. 18, §3.14 · p. 18, §3.15 · p. 18

3.18 Localization

• The localization evaluation covers 12 targets, including chest and breast abnormalities, abdominal organs and tumors, and structures or tumors in spine and brain images.The tasks use chest, breast, and palm X-rays; abdomen CT; and spine or brain MRI.Source: §3.18 · p. 19

4.1 Procedure of pathology evaluation

• The evaluation used two conversational rounds: first an image-only report request, then a diagnostic question supplied with the tissue origin.The first round tested modality recognition and structured reporting without medical prompts; the second tested report modification and diagnosis with the tissue-origin prompt.Source: §4.1 · p. 19

4.2 Pros and cons of GPT-4V on pathology image analysis

• Without medical prompts, GPT-4V produced structured, detailed reports; it itemized observations in 7 of 20 cases and recognized glandular structures and epithelium features across tissues.It also identified all tested images as H&E-stained microscopic tissue samples.Source: §4.2 · p. 19
• With tissue origin provided, GPT-4V modified its reports, giving one diagnosis for predicted-normal cases and multiple potential options for predicted-abnormal cases.The authors describe this as prompt-guided modification of the report and diagnostic response.Source: §4.2 · p. 19
• Many cell and nuclei descriptions reflected generic H&E features rather than image-specific patterns, and some diagnoses could derive from general medical knowledge instead of image morphology.The authors illustrate the generic descriptions with descriptions of purple-stained nuclei and pink cytoplasm.Source: §4.2 · pp. 19–20
• GPT-4V misdiagnosed 4 cases as normal, correctly diagnosed 3 tumors, and gave vague diagnoses for the other 13 malignant tumors.For anus and uterus cases, its diagnoses varied between normal and malignant.Source: §4.2 · p. 20
• GPT-4V sometimes speculated beyond image evidence, including on tumor mass effect and after changing its diagnosis when prompted about a finding.Source: Figure 38 · p. 55, Figure 42 · p. 59
• GPT-4V overlooked abnormalities including gallbladder-neck calculus, splenic infarction, hepatic abscess, pancreatic inflammation, and acute cholecystitis.Source: Figure 39 · p. 56, Figure 41 · p. 58, Figure 43 · p. 60
• For an impacted gallbladder calculus, GPT-4V listed five possible effects on the hepatic duct, illustrating its tendency to give excessive information.Source: Figure 40 · p. 57
• When asked for exact measurements in a hepatic abscess case, GPT-4V gave responsible answers about fluid density or collection size.Source: Figure 44 · p. 61

Gastrointestinal: Case 1

• GPT-4V detected a left-kidney abnormality but failed to pinpoint the arrows or diagnose the duodenal diverticulum with lipoma.The case used CT images.Source: Figure 47 · p. 64

• GPT-4V’s requested structural report covering the colon, liver, and prostate hallucinated normality in a case of tumoral-infiltration-related colocolic intussusception.Source: Figure 103 · p. 120, Gastrointestinal: Case 1 · p. 120

Observations:

• In trauma cases, GPT-4V missed pancreatic transection, orbital and hand fractures, and bony avulsions despite recognizing hand X-rays; it also mistook a necklace for a possible surgical device.The pediatric X-ray case likewise was recognized by modality and anatomy but called normal.Source: Figure 105 · p. 122, Figure 106 · p. 123, Figure 107 · p. 124
• Across pediatric cases, GPT-4V called abnormal X-ray, CT, and MRI examples normal, including after patient context was added, while it understood an ultrasound color annotation.It also ignored text in one CT query and missed an enlarged heart in another.Source: Figure 109 · p. 126, Figure 110 · p. 127, Figure 111 · p. 128
• In urogenital report examples, GPT-4V recognized modalities or devices but failed to identify pathology, and one X-ray report hallucinated that each part was normal despite symptoms.The report-generation case for colovesical fistula remained unable to identify abnormalities despite images from all views.Source: Figure 115 · p. 132, Figure 119 · p. 136, Figure 123 · p. 140
• Urogenital VQA responses were better than report generation in some cases, and two viewing angles improved response quality over a single annotated perspective.For an adrenal mass, adding annotated CT images enabled a correct mass-presence answer while retaining the correct stone answer; extra patient information worsened responses.Source: Figure 116 · p. 133, Figure 117 · p. 134, Figure 121 · p. 138
• For the retrocaval ureter, adding a second angle improved VQA response quality, whereas additional patient information worsened responses in the adrenal myelolipoma case.The original adrenal CT alone supported a correct stone answer, and annotation also supported a correct mass-presence answer.Source: Figure 117 · p. 134, Figure 121 · p. 138, Figure 122 · p. 139
• Localization examples show poor or variable overlap: some predictions score 0.0 IOU, while repeated predictions can average much lower than a best single prediction.For the spleen, the highest IOU is 0.44; for the liver tumor, both the highest and averaged IOU are 0.0.Source: Figure 125 · p. 142, Figure 127 · p. 144
• Other localization examples included 0.0 IOU for breast mass and palm fracture, while the sacrum and cardiomegaly examples had highest IOUs of 0.47 and 0.42.The averaged IOUs were 0.36 for sacrum and 0.33 for cardiomegaly.Source: Figure 131 · p. 148, Figure 137 · p. 154, Figure 138 · p. 155

Anus

• GPT-4V produced a detailed report and identified epithelial structures, but offered several potential diagnoses without the correct one.The example is a patch of a whole-slide image for anal adenocarcinoma.Source: Figure 140 · p. 157

Bladder

• Given the bladder tissue prompt, GPT-4V identified epithelial structure and correctly diagnosed invasive urothelial carcinoma.The example is a patch of a whole-slide image from bladder tissue.Source: Figure 141 · p. 158

Bone

• The accompanying prompt asks GPT-4V to write a report for the image.Source: Bone · p. 159
• GPT-4V fails to provide the correct disease option for the plasma cell myeloma image, even with the tissue origin given in the prompt.Source: Figure 142 · p. 159
• The image is a patch of a whole-slide histopathology image of plasma cell myeloma from bone marrow.Source: Figure 142 · p. 159

Breast

• The accompanying prompt asks GPT-4V to write a report for the image.Source: Breast · p. 160
• In response to a report request, GPT-4V describes a microscopic tissue sample and says the pinkish-purple hue indicates common H&E staining.Source: Breast · p. 160
• GPT-4V includes the correct diagnosis among several alternatives for the breast invasive ductal carcinoma image, but appears to list options from the tissue prompt without image analysis.Source: Figure 143 · p. 160

Cervix

• In response to a report request, GPT-4V labels the sample as unknown tissue and the staining method as presumed H&E.Source: Cervix · p. 161
• The report describes a histological section but does not provide a specific tissue identification.Source: Cervix · p. 161
• GPT-4V identifies the cervix tissue origin without prior knowledge but fails to find abnormalities or detect the adenocarcinoma in situ.Source: Figure 144 · p. 161

Central nervous system

• In response to a report request, GPT-4V describes a histological slide with likely H&E staining and gives no specific tissue diagnosis.Source: Central nervous system · p. 162
• GPT-4V identifies abnormalities and correctly diagnoses glioma from the histopathology image and brain-tissue prompt, but cannot determine the tumor subtype.Source: Figure 145 · p. 162
• The image is a patch of a whole-slide histopathology image of glioblastoma from brain tissue.Source: Figure 145 · p. 162

Colon

• GPT-4V fails to detect abnormalities in the colon serrated adenocarcinoma image.Its report describes an H&E-stained tissue section but does not definitively identify the tissue type.Source: Figure 146 · p. 163, Colon · p. 163

Esophagus

• GPT-4V identifies tubular structures in the esophageal adenocarcinoma image but calls it normal tissue after receiving the tissue-origin prompt.One potential tissue type it supplied was correct.Source: Figure 147 · p. 164, Esophagus · p. 164

Hepatobiliary

• GPT-4V describes dense features in the hepatocellular carcinoma image, but none of its proposed diagnoses is correct.Its report identifies the image as a stained tissue section without a specific diagnosis.Source: Figure 148 · p. 165, Hepatobiliary · p. 165

Kidney

• GPT-4V gives detailed descriptions of cellular, nuclear, and stromal morphology but fails to identify clear cell renal cell carcinoma.Its report presents a general description of a stained tissue sample.Source: Figure 149 · p. 166, Kidney · p. 166

Lung

• For the lung squamous cell carcinoma image, GPT-4V notes tissue abnormalities but proposes lung diseases without a malignant tumor.Its report gives a general description of an H&E-stained sample.Source: Figure 150 · p. 167, Lung · p. 167

Lymph node

• GPT-4V describes irregular nuclei in the lymph-node image but misidentifies its tissue origin; after the lymph-node prompt, its potential diagnoses include the true diagnosis.Its report notes densely packed cells and a range of possible interpretations, including tumors and reactive conditions.Source: Figure 151 · p. 168, Lymph node · p. 168

Nasal cavity

• GPT-4V identifies abnormalities in the sinonasal carcinoma image and offers a correct diagnosis option after receiving the tissue-origin prompt.Its report describes a stained tissue sample and cellular features.Source: Figure 152 · p. 169, Nasal cavity · p. 169

Oral cavity

• For oral cavity squamous cell carcinoma, GPT-4V identifies epithelial tissue and abnormalities, then diagnoses correctly with the tissue-origin prompt.Its report describes varying nuclear size and shape and a dense cellular arrangement.Source: Figure 153 · p. 170, Oral cavity · p. 170

Pancreas

• GPT-4V identifies glandular tissue in the pancreatic ductal adenocarcinoma image but misdiagnoses it as normal tissue.Its report describes a stained microscopic tissue sample.Source: Figure 154 · p. 171, Pancreas · p. 171

Prostate

• GPT-4V identifies glandular structures in the prostate image but does not provide a reasonable diagnosis.Its report describes a microscopic tissue sample stained with H&E.Source: Figure 155 · p. 172, Prostate · p. 172

Skin

• For the cutaneous melanoma image, GPT-4V identifies the tissue origin and includes the correct answer among three potential diagnoses.Its report describes a histological section and observations.Source: Figure 156 · p. 173, Skin · p. 173

Stomach

• GPT-4V neither identifies the stomach tissue origin nor gives the accurate diagnosis for the diffuse gastric carcinoma image.Its report offers a general description of an H&E-stained slide.Source: Figure 157 · p. 174, Stomach · p. 174

Thyroid

• GPT-4V identifies glandular structures in the papillary thyroid carcinoma image and includes the correct diagnosis among three options.Its report describes a microscopic tissue sample and observations.Source: Figure 158 · p. 175, Thyroid · p. 175

Uterus

• For the endometrial clear cell carcinoma image, GPT-4V identifies glandular structures and includes the correct diagnosis among three options.Its report describes cellular structures with varying degrees of staining.Source: Figure 159 · p. 176, Uterus · p. 176