PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images

Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, Weidi Xie

arXiv:2302.14042v3cs.CV

TL;DR

Medical vision-language models have limited generalization in fine-grained chest X-ray diagnosis, motivating methods that incorporate domain knowledge. KAD uses a knowledge encoder, extracted report entities, contrastive learning, and a Disease Query Network, and shows strong zero-shot and few-shot transfer across external X-ray datasets. Its zero-shot performance is comparable to supervised models and can exceed expert-radiologist averages on some pathologies.

  • Problem

    Medical vision-language models generalize poorly beyond diseases and findings seen during training because chest X-ray diagnosis depends on subtle signals and specialized terminology.

  • Method

    KAD learns representations from a medical knowledge graph, extracts entities and relations from reports, guides image-text contrastive learning, and uses a Disease Query Network for classification.

  • Results

    Across four external X-ray datasets, KAD achieves strong zero-shot performance comparable to fully supervised models, exceeds expert-radiologist averages on some CheXpert pathologies, and outperforms existing approaches under few-shot fine-tuning.

  • Takeaways & Limitations

    KAD supports arbitrary pathology queries and data-efficient transfer, indicating potential for AI-assisted radiography across varied clinical diagnosis categories.

  • Takeaways & Limitations

    KAD may perform poorly on pathologies absent from reports or disconnected from the knowledge base and cannot provide accurate segmentation.

Abstract

from arXiv · show

While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nature of medical tasks and the high demand for domain knowledge. To address this challenge, we propose a novel approach called Knowledge-enhanced Auto Diagnosis (KAD) which leverages existing medical domain knowledge to guide vision-language pre-training using paired chest X-rays and radiology reports. We evaluate KAD on {four} external X-ray datasets and demonstrate that its zero-shot performance is not only comparable to that of fully-supervised models, but also superior to the average of three expert radiologists for three (out of five) pathologies with statistical significance. Moreover, when few-shot annotation is available, KAD outperforms all existing approaches in fine-tuning settings, demonstrating its potential for application in different clinical scenarios.

1 Introduction

KAD addresses the limited generalization of medical vision-language models by injecting structured medical knowledge into chest X-ray pre-training. It combines report entity extraction, knowledge-encoder guidance, and disease-query classification, achieving strong zero-shot performance across external datasets.

  • Motivation: Medical vision-language models struggle with diseases and radiology findings beyond training categories because diagnosis depends on subtle regional signals and specialized terminology.The paper identifies domain knowledge as necessary for modeling these fine-grained medical concepts.
  • Approach: KAD trains a chest X-ray foundation model on paired images and reports while explicitly leveraging a medical knowledge graph.Its two-stage framework first learns knowledge-graph representations, then uses extracted report entities and relations to guide visual representation learning.
  • Approach: The workflow extracts clinical entities and relations, applies a pre-trained knowledge encoder for image-text contrastive learning, and optimizes a Disease Query Network for classification.At inference, a disease name serves as the query and the network outputs the pathology probability.
  • Approach: A query-based transformer enables zero-shot evaluation on arbitrary diseases or radiology findings and provides attention maps as visual evidence.The disease name is used as the query input for prediction.
  • Evaluation: Across four external X-ray datasets, KAD shows zero-shot performance significantly above existing medical visual-language models for 193 PadChest pathologies and comparable to fully supervised approaches.The paper also reports performance comparable to or better than expert radiologists on some CheXpert tasks and improved fine-tuning transfer with additional annotations.

2 Related Work

Prior medical vision-language work improves image-text alignment, while clinical entity and relation extraction preserves structured report information. Medical knowledge-enhanced models incorporate knowledge through model design or external inputs, but these lines of work motivate KAD’s combined approach.

  • Vision-Language Pre-training: Medical vision-language pre-training includes single-stream and two-stream architectures, with medical methods using contrastive learning and local image-text alignment.Examples include ConVIRT, LoVT, GLoRIA, and BioViL.
  • Medical Named-Entity Recognition: Medical named-entity-recognition methods extract clinical concepts, attributes, and relationships from radiology reports in structured formats.Relationship extraction helps retain information about connections between entities.
  • Medical Knowledge Enhanced Models: Medical knowledge-enhanced models either imitate radiological practice or modify architectures, or treat knowledge as external inputs that guide features or losses.The paper groups prior approaches into model-based and input-based categories.

3 Method

KAD combines medical knowledge encoding, report entity extraction, and query-based disease prediction to train a knowledge-guided visual-language model for chest X-rays. Its training aligns paired image-report representations while learning entity-presence predictions for flexible pathology queries.

  • Architecture: KAD trains a knowledge encoder on UMLS representations, extracts report entities, and uses image, knowledge, and disease-query encoders in a two-stage framework.The knowledge encoder is trained before visual-language learning, while entity extraction supplies presence labels for disease-query prediction.
  • Architecture: Disease Query Network uses disease names as queries to attend visual features and produce pathology likelihoods with visual evidence maps.The model can query arbitrary diseases or radiology findings at inference time.
  • Knowledge encoder: The knowledge encoder uses contrastive learning to bring textual descriptions associated with the same UMLS CUI closer in representation space.Positive descriptions include concepts, definitions, and relation-linked concepts; the loss uses a temperature parameter and indicator function.
  • Entity extraction: Radiology reports are converted into sequences of anatomy or observation entities with presence information, using heuristic rules, RadGraph, or ChatGPT.The pipeline labels extracted observations as present, uncertain, or absent, and defaults unmentioned entities to absent.
  • Representation learning: Visual features are extracted with a backbone, while report features come from the knowledge encoder and are used with contrastive image-text learning.The visual representation is a spatial feature map, whereas text features contain token-level representations; the contrastive objective uses cosine similarity and temperature scaling.
  • Training: The overall training objective sums image-text contrastive loss with binary cross-entropy for entity-existence prediction.The disease-query network uses extracted entity labels to supervise existence prediction.

4.1 Domain-specific Knowledge

KAD incorporates medical domain knowledge by combining a structured knowledge graph with report-to-entity extraction. UMLS provides the medical concepts and relations used to guide pre-training.

  • Domain-specific knowledge: KAD uses the Unified Medical Language System as a medical knowledge base and explores heuristic rules, RadGraph, and ChatGPT for report entity extraction.These components convert raw reports into structured medical information for knowledge-guided training.
  • UMLS knowledge graph: UMLS integrates medical concepts from multiple lexicons, assigning each entity a Concept Unique Identifier, definitions, synonymous names, and semantic types.The knowledge graph also supplies relations between medical concepts.

4.2 Datasets

KAD is pre-trained on paired chest radiographs and reports from MIMIC-CXR, then evaluated through zero-shot transfer and fine-tuning on four external X-ray datasets. The external datasets vary in labels, annotation sources, and evaluation targets.

  • Pre-training data: MIMIC-CXR contains 377,110 images from 227,835 radiographic studies involving 65,379 patients, with paired free-text reports and frontal or lateral X-ray views.Reports summarize radiologists’ findings for each radiographic study.
  • Evaluation datasets: The evaluation covers zero-shot transfer and fine-tuning across PadChest, NIH ChestXray14, CheXpert, and ChestX-Det10.These datasets support pathology classification, radiologist-annotated evaluation, and zero-shot grounding.
  • PadChest: PadChest contains 160,868 images with 174 radiographic findings and 19 differential diagnoses, but only 27% of labels are from board-certified radiologists.Evaluation uses only samples annotated by board-certified radiologists.
  • NIH ChestXray14: NIH ChestXray14 contains 112,120 images from 30,805 patients and provides 14 disease labels mined from radiology reports using natural language processing.The task predicts binary disease labels.
  • CheXpert: CheXpert includes 224,316 images from 65,240 patients, with validation annotations from three radiologists and test annotations from a five-radiologist consensus.The validation set is used to select prediction thresholds.
  • ChestX-Det10: ChestX-Det10 contains 3,543 images with box-level annotations from three radiologists for 10 diseases or abnormalities, supporting zero-shot grounding on its test set.The study follows the official data split.

4.3 Baselines

The baselines span medical image-text pre-training and medical image-only pre-training. For fair comparison, models are evaluated with a common ResNet-50 image-encoder backbone and, where needed, re-trained on MIMIC-CXR.

  • Baseline categories: Medical image-text baselines support zero-shot transfer and fine-tuning, whereas medical image-only baselines support fine-tuning evaluation.The comparison separates approaches by the modalities used during pre-training.
  • Medical image-text methods: The image-text comparison includes ConVIRT, GLoRIA, BioViL, CheXzero, and MedKLIP.ConVIRT uses bidirectional contrastive learning, GLoRIA combines global and regional contrastive learning, and BioViL uses a radiology-specific text encoder.
  • Comparison protocol: ConVIRT, GLoRIA, Model Genesis, and C2L are re-trained on MIMIC-CXR, while BioViL and CheXzero use officially released models.The re-training is performed for fair comparison.
  • Medical image-text methods: CheXzero fine-tunes CLIP on paired X-rays and reports using contrastive learning.Its pre-training uses image-text pairs.
  • Medical image-only methods: The image-only comparison includes Model Genesis, Comparing to Learn, and ImageNet pre-training.Model Genesis restores distorted images, C2L uses image- and feature-level mix-up, and ImageNet pre-training uses supervised natural-image learning.

4.4 Implementation details

KAD uses two-stage pre-training with image and report preprocessing, including entity extraction and knowledge-encoder training. Its zero-shot inference uses disease names as queries over visual features, while fine-tuning applies the same optimizer settings across datasets.

  • Data pre-processing: Radiology reports are preprocessed with ScispaCy named entity recognition and linking, while chest X-rays are resized to 224 × 224, normalized, and augmented.Augmentations include random resize crops, horizontal flips, rotations from –10 to 10 degrees, grayscale changes, and brightness, sharpness, and contrast adjustments.
  • Model pre-training: Stage 1 fine-tunes a PubMedBERT-initialized knowledge encoder for 100K steps using concept-definition pairs and concept-relation-concept triplets.Each mini-batch contains 64 concept-definition pairs and 64 concept-relation-concept triplets, with maximal sequence length 256.
  • Zero-shot transfer: For zero-shot transfer, the Disease Query Network takes disease names as queries and image features as keys and values to predict disease likelihoods.Average disease-to-visual cross-attention is used for grounding.
  • Model fine-tuning: During fine-tuning, all models use AdamW with lr = 1 × 10−4, the pre-training decay strategy, and batch size 64.The same learning rate is used across datasets.

4.5 Evaluation details

Evaluation reports classification and grounding metrics using defined procedures for thresholding, localization, and uncertainty estimation. MCC thresholds are optimized on validation data, Pointing Game measures grounding accuracy, and confidence intervals use bootstrap resampling.

  • Metric computation: AUC, MCC, F1, and ACC denote Area under the ROC Curve, Matthews Correlation Coefficient, F1 score, and Accuracy, respectively.AUC is obtained directly from model predictions, whereas MCC requires thresholded positive/negative predictions.
  • Metric computation: MCC predictions use thresholds optimized on the validation dataset before condition-based MCC scores are calculated.Inference first produces class probabilities, which are converted into positive/negative predictions using the optimized thresholds.
  • Grounding evaluation: Pointing Game evaluates zero-shot grounding by checking whether the maximum-response heatmap region overlaps the ground-truth mask.The final pointing game score is the resulting accuracy.
  • Confidence intervals: Confidence intervals are estimated with 1,000 non-parametric bootstrap samples, each resampled with replacement to the original dataset size.AUC, MCC, F1, and ACC are recomputed for every bootstrap sample.

5 Results

KAD incorporates medical knowledge and structured report information into chest X-ray vision-language pre-training, then evaluates zero-shot transfer, fine-tuning, grounding, and component contributions. Across external datasets, it improves unseen-disease diagnosis, data-efficient transfer, and several comparisons with supervised models and radiologists, while ablations support the roles of its main modules.

  • 5 Results: KAD uses domain knowledge from UMLS or RadGraph and evaluates zero-shot disease prediction across four external multi-center X-ray datasets.The model predicts whether a queried disease exists in each input image.
  • 5.1 PadChest: AUC ≥ 0.900 is achieved on 31 classes and AUC ≥ 0.700 on 111 of 177 unseen PadChest classes.PadChest evaluation uses 39,053 test images annotated by board-certified radiologists.
  • 5.2 NIH ChestXray14: 0.786 average AUC is achieved on NIH ChestXray14, with significantly higher AUC than the best baseline for 13 of 14 pathologies.KAD also improves all evaluation metrics in fine-tuning experiments with varying labeled-data percentages.
  • 5.3 CheXpert: KAD exceeds existing approaches on average CheXpert evaluation scores and achieves statistically higher MCC than three radiologists on most competition pathologies.Reported MCC advantages include atelectasis, edema, and pleural effusion; consolidation is slightly worse without statistical significance.
  • 5.4 ChestX-Det10: KAD improves both zero-shot classification and grounding on ChestX-Det10, including gains over GLoRIA exceeding 7% for calcification and 18% for fracture.Higher image resolution improves localization more than diagnosis, particularly for diseases with small bounding boxes.
  • 5.5 Ablation study: Entity extraction improves AUC from 0.772 to 0.786 on ChestX-ray14 and from 0.897 to 0.906 on CheXpert.These results support using standardized report information rather than aligning raw reports and images alone.
  • 5.5 Ablation study: Removing DQN causes the largest performance degradation, while removing random selection reduces alignment between image and text features and zero-shot performance.Knowledge-encoder pre-training also distinguishes disease features more clearly than an unfine-tuned PubMedBERT encoder.

6 Discussion

KAD extends chest X-ray vision-language pre-training with medical knowledge, achieving strong zero-shot and data-efficient transfer performance while providing visual evidence for predictions. Its scope remains bounded by validation needs, knowledge-base coverage, and coarse rather than precise localization.

  • Performance: KAD achieves statistically higher MCC than the mean performance of three radiologists across five CheXpert pathologies.The model can match or surpass experienced clinicians on some diagnostic tasks and serve as a reference when physicians disagree.
  • Performance: 6.8% performance gain on “consolidation” is reported versus fully supervised methods on PadChest, despite no PadChest training images.KAD also accepts arbitrary pathology queries, whereas fully supervised models are limited to closed category sets.
  • Generalization: 177 unseen classes were identified and diagnosed with high accuracy in zero-shot evaluation on PadChest.The approach extracts medical entities and uses UMLS knowledge to connect radiological concepts beyond direct raw-report matching.
  • Transfer: 1% labeled data yielded better results than other methods using 100% labels for cardiomegaly, effusion, pneumonia, pneumothorax, and hernia.The authors attribute this data-efficient transfer to benefits inherited from pre-training, including DQN and the pre-trained text encoder.
  • Explainability: KAD produces grounded heatmaps by averaging DQN cross-attention maps, and pointing-game results on ChestX-Det10 further confirm the explanations.Attention maps visualize model focus across different resolutions for clinical decision support.
  • Limitations: KAD requires a small validation set for disease-specific threshold tuning, may underperform on pathologies absent from the knowledge base and reports, and provides only coarse grounding.The method is not expected to predict some disconnected or unmentioned pathologies accurately in zero-shot evaluation and cannot provide accurate segmentation.

7 Conclusion

KAD is a knowledge-enhanced vision-language pre-training model for chest X-ray auto-diagnosis. It combines medical knowledge, extracted report entities and relations, image-text contrastive learning, and a Disease Query Network, with strong zero-shot results across four datasets.

  • Conclusion: KAD combines a knowledge encoder, extracted clinical entities and relations, image-text contrastive learning, and a Disease Query Network for chest X-ray classification.The knowledge encoder represents an existing medical graph, while the query network is optimized for diagnosis.
  • Conclusion: Across four datasets, KAD shows strong zero-shot performance against state-of-the-art self-supervised and supervised methods.It also exceeds the average performance of expert radiologists on multiple pathologies.
Loading 2302.14042v3…