Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge

1 INTRODUCTION

• Med-VLP learns generalizable multimodal representations from large-scale medical image-text data for tasks such as Med-VQA, Med-ITC, and Med-ITR.The introduction frames these applications as relevant to alleviating medical data scarcity.Source: §1 · p. 1
• VLP systems combine vision and language encoders, a multimodal fusion module, and pretext tasks; medical studies have applied this framework to improve medical vision-and-language tasks.Examples of pretext tasks include masked image modeling, masked language modeling, and image-text matching.Source: §1 · p. 1
• Existing Med-VLP methods capture image-text correspondences but often disregard complementary knowledge and lack explicit knowledge modeling.General-domain knowledge-enhanced VLP methods often rely on object detectors or scene-graph parsers, which are generally unavailable in the medical domain.Source: §1 · pp. 1–2
• The proposed UMLS-based approach aligns image and text representations through knowledge, injects knowledge into multimodal fusion, and uses knowledge-induced pretext tasks.The approach is evaluated on a benchmark spanning Med-VQA, Med-ITC, and Med-ITR.Source: §1 · p. 2
• State-of-the-art performance is reported on downstream tasks, and the benchmark covers Med-VQA, Med-ITC, and Med-ITR.The introduction reports results without providing metric values in these passages.Source: §1 · p. 2

2 RELATED WORK

• General-domain VLP methods vary in image and text features, use single- or dual-stream fusion, and train with tasks such as MLM, MIM, and ITM.This paper uses ViT-based grid features, BERT-based dynamic text features, dual-stream fusion, and MLM, MIM, and ITM.Source: §2 · p. 2
• Medical VLP extends VLP to medical image-text understanding; prior work includes general-domain models evaluated on disease classification and models pretrained on medical image-text data.The paper contrasts its scheme with prior work across datasets, model designs, pretraining tasks, and evaluation benchmarks.Source: §2 · p. 2
• Knowledge-enhanced VLP work includes ERNIE-ViL's scene-graph alignments and ROSITA's cross-modal knowledge mappings.The passage also identifies knowledge-related methods [28], but gives no further detail about them here.Source: §2 · p. 2
• This paper proposes three VLP-specific knowledge-injection perspectives without requiring object detectors or scene-graph parsers, which are unavailable in the medical domain.The related-work discussion also surveys four knowledge-injection categories for unimodal pretraining: embeddings combination [41] [59], data-structure compatibility [32] [45], knowledge supervision [46], and neural-symbolic methods.Source: §2 · p. 2

3 THE PROPOSED APPROACH

• The method follows pre-training on medical image-text pairs with transfer of the pre-trained model to medical vision-and-language downstream tasks.This follows the standard pre-train-and-fine-tune paradigm.Source: §3 · pp. 2–3

3.1 The General Med-VLP Framework

• The framework uses vision and language encoders, a co-attention fusion module, and pretext tasks such as MLM, MIM, and ITM.The fusion module uses separate vision and language Transformers with self-attention and cross-attention sub-layers.Source: §3.1 · pp. 3–4

3.2 Knowledge Extraction

• Extracted entities are linked to UMLS, then triples whose head and tail entities occur in the corpus entity set form the knowledge set.The extraction process uses named entity recognition and linking for entity disambiguation.Source: §3.2 · pp. 4–5

3.3 Knowledge Injection

• Knowledge aligns image and text representations, supplements multimodal fusion for reasoning, and guides pretext tasks to focus on critical medical information.The three designs are aligning through knowledge, reasoning using knowledge, and learning from knowledge.Source: §3.3 · p. 5
• For reasoning, entity representations attend to image representations, and an entity-text mapping matrix fuses entity information into text representations.The resulting text representations encode image and entity information.Source: §3.3 · p. 5
• For MLM, the model samples entities and masks their consecutive text-token spans rather than masking subwords randomly.The authors state that entities may be abstract or have a physical existence, directing attention to critical medical information in images and texts.Source: §3.3 · p. 5

4.1 Pre-training Setup

• Pre-training uses ROCO, MedICaT, and MIMIC-CXR, with samples shorter than three text tokens excluded and only frontal-view images retained for MIMIC-CXR.Nonradiology samples are filtered from ROCO and MedICaT; official splits are used for ROCO and MIMIC-CXR, while MedICaT reserves 1,000 pairs each for validation and test.Source: §4.1 · pp. 5–6

4.2 Vision-and-Language Transfer Tasks

• The reported comparison tables cover Med-VQA on three datasets, Med-ITC on MELINDA, and Med-ITR on ROCO, including zero-shot and fine-tuned retrieval results.Table 1 highlights the top and second-best results for each evaluation metric; Tables 2 and 3 compare methods on their respective tasks.Source: Table 1 · p. 6, Table 2 · p. 6, Table 3 · p. 6
• The benchmark evaluates Med-VQA on VQA-RAD, SLACK, and VQA-2019; Med-ITC on MELINDA; and Med-ITR on ROCO.Med-ITR reports zero-shot and fine-tuned performance, sampling 2,000 ROCO test pairs for evaluation because ranking is computationally costly.Source: §4.2 · pp. 6–7
• Med-VQA is treated as multi-label classification, Med-ITC as image-text pair classification, and Med-ITR as cross-modal retrieval in both image-to-text and text-to-image directions.Med-VQA and Med-ITC use accuracy; Med-ITR uses Recall@K for K=1, 5, and 10.Source: §4.2 · pp. 6–7

5.1 Main Results

• The approach achieves the best performance on all tasks, with substantial gains in Med-ITR and better Med-ITC performance without continued pre-training on MELINDA.For Med-VQA, it also achieves significant improvements over pre-training methods including MTPT and MMBERT.Source: §5.1 · p. 7

5.3 Qualitative Analysis

• In the alignment case, image and text representations both have high similarity to the linked UMLS entity, implicitly pulling the image and text representations closer.The linked entity is C4028269, “Nuclear magnetic resonance imaging brain.”Source: §5.3 · p. 7, Figure 3 · p. 8
• Entity-image attention mappings are better than subword-image mappings because entities are more complete semantic units.Words and subwords have smaller granularity, making image-text correspondences more challenging to learn.Source: §5.3 · p. 7
• The model correctly recovers masked medical entities, and the reconstructed images are reported as high quality.The knowledge-induced pretext task guides attention toward medical knowledge-related information.Source: §5.3 · p. 8, Figure 3 · p. 8

• The framework injects knowledge by aligning image and text representations before interaction, supplementing multimodal fusion for reasoning, and inducing pretext tasks focused on critical medical information.Source: §6 · p. 8
• The constructed benchmark covers Med-VQA, Med-ITC, and Med-ITR, and the approach achieves state-of-the-art performance on downstream datasets.Source: §6 · p. 8
• Further analyses indicate that the approach better learns vision-language correspondences and produces more generic and effective representations.Source: §6 · p. 8