Source-linked AI summary
Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
Jong Hak Moon, Hyungyung Lee, Woncheol Shin, Young-Hak Kim, Edward Choi
TL;DR
Medical vision-language pre-training needed broader evidence beyond diagnosis classification and separate task-specific systems. MedViLL uses a unified BERT-based model with multimodal attention masking and is evaluated across four tasks and three radiographic datasets, with superior performance reported against varied baselines. The study supports unified medical vision-language representation learning while noting that medical settings should be expanded further.
Problem
Medical vision-language pre-training had limited evaluation beyond diagnosis classification, leaving whether one model can support diverse understanding and generation tasks insufficiently examined.
Method
MedViLL is a unified BERT-based medical vision-language model using a novel self-attention scheme for images and reports across understanding and generation tasks.
Results
MedViLL showed superior performance against various baselines, including task-specific architectures, across four downstream tasks and three radiographic image-report datasets.
Takeaways & Limitations
A unified medical vision-language pre-training model can be evaluated across diagnosis classification, retrieval, visual question answering, and report generation.
Abstract
from arXiv · showhide
Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL), which adopts a BERT-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performance for both vision-language understanding tasks (diagnosis classification, medical image-report retrieval, medical visual question answering) and vision-language generation task (radiology report generation). By statistically and rigorously evaluating the proposed model on four downstream tasks with three radiographic image-report datasets (MIMIC-CXR, Open-I, and VQA-RAD), we empirically demonstrate the superior downstream task performance of MedViLL against various baselines, including task-specific architectures. The source code is publicly available at: https://github.com/SuperSupermoon/MedViLL
I. INTRODUCTION
Medical vision-language pre-training addresses the need to learn shared representations from radiographic images and reports across diverse understanding and generation tasks. MedViLL is introduced as a unified model evaluated across four downstream tasks and multiple datasets.
- Radiographic images and reports provide mutually helpful semantic information for medical vision-language learning.
- Most prior vision-language pre-training studies focused on either understanding or generation, making joint learning across both task types challenging.
- Medical vision-language pre-training had mainly been studied for diagnosis classification, motivating evaluation on diverse understanding and generation tasks.
- MedViLL learns multipurpose joint vision-language representations for downstream diagnosis, treatment delivery, case-based reasoning, and report generation.
- The study evaluates diagnosis classification, image-report retrieval, medical visual question answering, and radiology report generation, including transfer across two Chest X-ray datasets.
- The authors position MedViLL as the first unified medical vision-language pre-training model covering both understanding and generation tasks.
B. VL Multimodal Researches in the Medical Domain
Medical vision-language research has largely used task-specific CNN-RNN systems or general-domain pre-training designs. MedViLL instead adopts a single-stream BERT-based architecture to support joint multimodal processing across understanding and generation tasks.
- CNN-RNN models remained dominant in medical vision-language learning and were mainly designed for either understanding or generation.
- General-domain vision-language pre-training commonly varies input embedding, visual feature embedding, and downstream-task designs.
- Single-stream and two-stream architectures show marginal downstream performance differences, while two-stream models use more parameters.
- MedViLL uses a single-stream architecture for architectural simplicity and time/space efficiency.
- CNN-based visual encoders with random sampling address information gaps associated with region-based visual embeddings limited by detector categories.
- Unified vision-language pre-training can use one BERT-based architecture for both understanding and generation by varying self-attention masks.
A. Dataset
The study uses paired Chest X-ray images and reports from MIMIC-CXR and Open-I, then embeds visual and textual features into a shared representation sequence for joint modeling.
- MIMIC-CXR contains 377,110 Chest X-ray images with corresponding reports, while Open-I contains 7,466 images and 3,851 reports.
- Frontal and lateral views must be distinguished to avoid mismatching image findings with report pairs.
- A CNN extracts spatially flattened visual features, while absolute location features inject body-position information from X-ray images.
- The visual feature sequence contains K vectors in R^c, and the location sequence contains K vectors in R^c.
- A shared semantic embedding differentiates visual features from language embeddings, and random visual-feature sampling is used during pre-training to reduce overfitting.
- Clinical reports are tokenized into N subwords with WordPiece and converted into d-dimensional vector representations.
2) Language Feature Embedding:
MedViLL combines visual and language inputs in a joint BERT-based embedding and is trained with masked language modeling and image-report matching objectives.
- Language features receive a shared semantic embedding that differentiates them from visual embeddings.
- Visual and language embeddings are concatenated with CLS and modality-specific separator tokens to form the joint embedding input.
- Masked language modeling replaces 15% of report tokens and trains the model to recover them from the remaining text and visual context.
- MedViLL uses a single-stream BERT architecture with CNN-derived visual inputs and tokenized reports for downstream understanding and generation tasks.
- Image-report matching classifies whether an image and report form a matching pair, using matching and non-matching pairs sampled at a 1:1 ratio.
C. Self-Attention Mask Schemes
The model compares self-attention masks that regulate interactions between visual and language features. BAR preserves causal language generation while allowing vision-language mixing during pre-training.
- Self-Attention Mask Schemes: Bidirectional masking permits unrestricted interaction between visual and language inputs for unconstrained contextual learning.
- Self-Attention Mask Schemes: Sequence-to-Sequence masking restricts language features to previous words and prevents visual features from attending to language features.
- Self-Attention Mask Schemes: BAR allows image features to mix with language features while preserving causal language generation.
- Self-Attention Mask Schemes: The attention mask matrix contains 0 for allowed attention and −∞ for disallowed attention.
- Self-Attention Mask Schemes: BAR permits all attention combinations except the language-query-to-language-key subpart represented by Eq. (13).
- Self-Attention Mask Schemes: The four evaluated pre-training variants are Bi, S2S, BAR, and Bi & S2S, with Bi & S2S alternating S2S and Bi masks across mini-batches.
A. Dataset Analysis
MIMIC-CXR and Open-I differ in both diagnosis-label distributions and data volume, motivating cross-dataset evaluation of model generalization.
- MIMIC-CXR shows mild label imbalance, with class ratios from 13.39% for support devices to 1.2% for pneumonia and pleural other.
- Open-I shows severe label imbalance, ranging from 28.8% for Others and cardiomegaly to 1.07% for support devices.
- The diagnosis-classification setup uses up to 14 CheXpert-derived labels with 14 linear heads and evaluates micro-average AUROC and F1.
2) Medical Image-Report Retrieval:
The study evaluates medical image-report retrieval, visual question answering, and report generation using multimodal representations and transfer across datasets.
- Medical Image-Report Retrieval: Retrieval evaluation ranks positive pairs among 100 image-report pairs using Hit@5, Recall@5, Precision@5, and mean reciprocal rank.
- Radiology Report Generation: Report generation sequentially recovers masked tokens until predicting SEP, and is evaluated with perplexity, clinical efficacy, and BLEU.
- MedViLL outperforms most models in out-of-domain evaluations, supporting its reported generalization ability.
1) Diagnosis Classification:
Across diagnosis classification, retrieval, VQA, and report generation, MedViLL is generally competitive and often statistically superior, especially under transfer to Open-I.
- Medical Image-Report Retrieval: In report-to-image retrieval, MedViLL outperforms all baselines on unseen Open-I except Bi, whose difference from MedViLL is not statistically significant.
- Medical Visual Question Answering: MedViLL significantly outperforms MEVF on VQA-RAD and has significantly higher open-ended-question accuracy for both ALL and CHEST settings.
- Radiology Report Generation: MedViLL achieves the best MIMIC-CXR report-generation performance with a statistically significant difference, while tying closely with TieNet on unseen Open-I.
- Attention maps show that MedViLL attends to disease-discriminatory regions without image or report annotations, according to a professional cardiologist.
2) Radiology Report Generation:
MedViLL’s radiology report-generation analysis compares generated and original reports alongside attention visualizations and retrieval-based clinical evidence. The authors caution that BLEU4 may not appropriately measure medically related report content.
- Report-generation evaluation: A generated report described extubation and nasogastric tube removal, clinically related to VATS described in the original report.The original report described completion of video-assisted thoracic surgery, while the generated report described related procedural events.
- Qualitative analysis: Attention visualizations and report comparisons provide qualitative analysis of MedViLL’s radiology report-generation behavior.Figure 6 visualizes attention regions and compares generated reports with original reports for the same chest X-ray.
- Report-generation evaluation: The authors conclude that BLEU4 is not an appropriate evaluation measure for radiology report generation, because clinically related descriptions may differ lexically.The VATS example motivates concern that lexical overlap does not fully capture medical report quality.
- Medical image-report retrieval: Top-three retrieval results shared the query’s diagnosis labels, whereas low-ranked samples contained irrelevant labels.This pattern occurred for both report-to-image and image-to-report retrieval cases.
- Overall evaluation: MedViLL was evaluated across four downstream tasks and three radiographic image-report datasets against baselines, including task-specific architectures.The paper reports statistically and rigorously evaluating the model across vision-language understanding and generation tasks.
VI. APPENDIX
The appendix examines attention-mask designs, visual-feature choices, and task-specific ablations across diagnosis classification, retrieval, VQA, and report generation. Using all CNN visual features produced poor in-domain and out-of-domain performance, while extractor choice affected downstream results.
- A. Ablation studies: Using all CNN visual features produced poor in-domain and out-of-domain performance, indicating over-fitting to the MIMIC-CXR training set.The experiments compared full sampling with random 80% sampling and evaluated several CNN extractors.
- A. Ablation studies: Visual feature extractor choice produced performance differences across downstream tasks.The tested extractors were ResNet-50, ResNet-101, and DenseNet-121.
- A. Ablation studies: The authors suggest architecture and hyper-parameter experiments could improve performance across visual feature configurations.Examples include alternative visual extractors and BERT transforms.
- Self-attention masks: Bidirectional attention permits all visual-language attention combinations to encourage unconstrained cross-modal context learning.The described mask is one of the self-attention schemes explored in the appendix.
- Ablation results: The appendix reports diagnosis AUROC and F1, retrieval performance, VQA-RAD accuracy, and report-generation metrics including perplexity, label metrics, and BLEU4.Tables VI–IX cover the four downstream evaluation areas.