PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation

Haibo Jin, Haoxuan Che, Yi Lin, Hao Chen

arXiv:2308.12604v2cs.CVcs.CL

TL;DR

Medical report generation must produce clinically accurate descriptions despite difficult disease identification and imbalanced disease distributions. PromptMRG uses a classification branch, diagnosis-derived token prompts, cross-modal retrieval, and adaptive disease-balanced learning; experiments report state-of-the-art clinical-efficacy performance on two benchmarks.

  • Problem

    Medical report generation remains difficult because accurate reports require clinical understanding and disease identification, while rare diseases are underrepresented and diagnosed unreliably.

  • Method

    PromptMRG combines a disease classification branch with diagnosis-driven token prompts, cross-modal feature enhancement, and self-adaptive disease-balanced learning.

  • Results

    PromptMRG achieves state-of-the-art clinical-efficacy performance on both evaluated MRG datasets.

  • Takeaways & Limitations

    The framework is reported to improve diagnostically correct report generation and narrow the gap between current MRG models and clinical demands.

  • Takeaways & Limitations

    The method requires disease labels for its classification branch, and its CLIP component requires sufficient image-report pairs for domain adaptation.

Abstract

from arXiv · show

Automatic medical report generation (MRG) is of great research value as it has the potential to relieve radiologists from the heavy burden of report writing. Despite recent advancements, accurate MRG remains challenging due to the need for precise clinical understanding and disease identification. Moreover, the imbalanced distribution of diseases makes the challenge even more pronounced, as rare diseases are underrepresented in training data, making their diagnostic performance unreliable. To address these challenges, we propose diagnosis-driven prompts for medical report generation (PromptMRG), a novel framework that aims to improve the diagnostic accuracy of MRG with the guidance of diagnosis-aware prompts. Specifically, PromptMRG is based on encoder-decoder architecture with an extra disease classification branch. When generating reports, the diagnostic results from the classification branch are converted into token prompts to explicitly guide the generation process. To further improve the diagnostic accuracy, we design cross-modal feature enhancement, which retrieves similar reports from the database to assist the diagnosis of a query image by leveraging the knowledge from a pre-trained CLIP. Moreover, the disease imbalanced issue is addressed by applying an adaptive logit-adjusted loss to the classification branch based on the individual learning status of each disease, which overcomes the barrier of text decoder's inability to manipulate disease distributions. Experiments on two MRG benchmarks show the effectiveness of the proposed method, where it obtains state-of-the-art clinical efficacy performance on both datasets. The code is available at https://github.com/jhb86253817/PromptMRG.

Introduction

PromptMRG addresses the difficulty of generating diagnostically accurate medical reports, especially under imbalanced disease distributions, by using diagnosis-driven prompts and complementary feature and learning strategies.

  • Medical report generation automatically produces free-text descriptions of medical images and could reduce radiologists’ reporting workload.
  • Accurate MRG requires both clinically correct abnormality identification and linguistically precise report generation.A prediction can closely match reference wording while misdiagnosing opacity and pneumonia.
  • Existing MRG methods often underuse diagnostic information, with a vanilla disease classifier outperforming most SOTA MRG methods on clinical-efficacy F1.Clinical efficacy evaluates the diagnostic accuracy of generated reports.
  • Biased disease distributions produce uneven clinical-efficacy performance, leaving rare-disease diagnoses unreliable.
  • PromptMRG adds a disease classification branch whose diagnostic outputs become token prompts guiding report generation.
  • Cross-modal feature enhancement retrieves similar reports using pretrained CLIP knowledge, while self-adaptive disease-balanced learning adjusts classification optimization by disease learning status.
  • PromptMRG achieves state-of-the-art clinical-efficacy performance on two MRG benchmarks.

Related Works

Prior MRG work uses encoder-decoder architectures, knowledge graphs, auxiliary tasks, and prompting to improve report generation and clinical efficacy. PromptMRG distinguishes itself by using diagnostic results as explicit prompts and by enabling end-to-end optimization.

  • Encoder-Decoder Architectures: MRG commonly adopts encoder-decoder architectures from image captioning, but longer reports and harder clinical-abnormality identification make it more challenging.Medical reports are typically longer than captions, and abnormalities in medical images are more difficult to identify than natural objects.
  • Knowledge Graphs: Knowledge graphs incorporate disease-organ relationships or symptom knowledge into report-generation models for dedicated abnormality feature learning.Later approaches can dynamically update graphs or distill symptom-graph knowledge during decoding.
  • Multi-Task Learning: Multi-task methods add auxiliary objectives such as disease classification, contrastive learning, and image-text matching to improve representations for MRG.Earlier disease-classification branches generally benefit report generation implicitly through discriminative feature learning.
  • PromptMRG’s Distinction: PromptMRG explicitly converts classification results into prompts for report generation, unlike prior methods that use classification mainly as a parallel auxiliary task.Its end-to-end optimization contrasts with RGRG’s separate module training and heuristic fusion of sentence candidates.
  • Prompting: Prompting adapts model inputs or trainable vectors to guide task performance across language, vision, and vision-language settings.Medical prompting has also been explored for transferring knowledge to medical object detection and for domain adaptation.

Method

PromptMRG combines an encoder-decoder report generator with diagnosis-driven token prompts, cross-modal feature enhancement, and self-adaptive disease-balanced learning. The framework uses diagnostic classification to guide generation and improve disease recognition despite imbalanced distributions.

  • Framework: PromptMRG uses an encoder to extract image features and a decoder that generates reports conditioned on visual features and diagnosis-driven prompts.The decoder predicts report tokens autoregressively, while the diagnosis-driven prompts represent disease classification results.
  • Diagnosis-Driven Prompts: The disease classification branch predicts four classes—Blank, Positive, Negative, and Uncertain—for each disease and converts the results into token prompts during inference.Four vocabulary tokens, [BLA], [POS], [NEG], and [UNC], represent the classification classes and explicitly guide report generation.
  • Diagnosis-Driven Prompts: PromptMRG uses auxiliary LLM labeling to obtain four additional abnormality labels—Aorta, Bone/Spine, Hemidiaphragm, and Lung Volume—from training reports.Vicuna-13B is used as the labeling assistant through disease-related prompts.
  • Cross-Modal Feature Enhancement: Cross-modal feature enhancement retrieves top-k report features with a MIMIC-pretrained CLIP and dynamically aggregates them with visual features for disease classification.The dynamic aggregation module applies self-attention followed by cross-attention, using the visual feature as the query; CLIP remains frozen while the aggregation module is trainable.
  • Self-Adaptive Disease-Balanced Learning: Self-adaptive disease-balanced learning adjusts classification objectives according to disease-specific learning dynamics rather than relying only on fixed class distributions.The method initializes class distributions from training statistics, updates them adaptively, and combines the resulting loss with the overall training objective using a balancing coefficient λ.

Experiments

Experiments evaluate PromptMRG with natural-language and clinical-efficacy metrics, comparisons, ablations, prompt analyses, and disease-balance analysis. The method achieves strong clinical efficacy, while diagnosis-driven prompting introduces a trade-off in some MIMIC NLG metrics.

  • Benchmark Comparison: PromptMRG achieves state-of-the-art performance on the three clinical-efficacy metrics across both MIMIC-CXR and IU X-Ray.On MIMIC, it obtains 0.476 F1, exceeding DCL by 10% absolute and KiUT by 15%; it exceeds RGRG by 2.9%, while the IU improvement over RGRG is 3.1%.
  • Benchmark Comparison: PromptMRG achieves the best results on all IU X-Ray NLG metrics and the best BLEU-1 on MIMIC.Its other MIMIC NLG metrics are weaker than IU results and do not consistently surpass the baseline.
  • Ablation Study: Adding DDP raises MIMIC F1 from 0.370 for the baseline to 0.444, while ADL, CFE, and SDL further increase it to 0.451, 0.464, and 0.468.The complete configuration yields the strongest ablation performance reported in the study.
  • Ablation Study: DDP reduces repeated frequent phrases and can lower NLG performance, whereas adding CFE and SDL improves NLG metrics but still generally trails the baseline except for BLEU-1.The authors relate this trade-off to DDP supplying diagnostic information and producing more diverse text.
  • Prompt Analysis: Explicitly representing diagnostic information as an embedding prompt improves clinical efficacy, including a 6.4% F1 increase, although token and text prompts perform better.The comparison concerns alternative prompt types for diagnosis-guided generation.
  • Disease Balance: SDL improves rare-disease performance, producing approximately 8% average absolute F1 improvement across seven rare diseases and over 12% for some diseases.Figure 6 sorts diseases by ascending training-set size when comparing models with and without SDL.

Conclusion and Future Work

PromptMRG addresses unsatisfactory clinical efficacy by converting classification results into generation prompts, enhancing diagnosis with retrieval, and balancing disease learning. Experiments on two datasets show superiority in generating diagnostically correct reports, while the method’s broader applicability remains bounded by data and label requirements.

  • Conclusion: PromptMRG converts disease-classification results into prompts that guide report generation toward diagnostically correct reports.Its classification branch supplies the diagnostic results used during generation.
  • Conclusion: CFE improves diagnostic accuracy through cross-modal retrieval and dynamic aggregation, while SDL adjusts each disease’s learning objective according to its learning status.These modules target feature enhancement and imbalanced disease learning, respectively.
  • Conclusion: PromptMRG demonstrates superiority on two datasets, especially in generating diagnostically correct reports and narrowing the gap between MRG models and clinical demands.The conclusion frames diagnostic correctness as the method’s principal advantage.
  • Future Work: The experiments use chest X-rays, disease labels are required for classification-branch training, and CFE needs sufficient image-report pairs for CLIP domain adaptation.The authors suggest unsupervised report clustering when labels are unavailable and leave other modalities with limited data for future work.

IU X-Ray Dataset

The authors reject the commonly used IU X-Ray split for disease-aware evaluation because some diseases have only two positive test samples. They therefore use the entire IU dataset for evaluation.

  • Dataset Evaluation: The Chen et al. IU X-Ray test split is unsuitable for disease-aware evaluation because consolidation and pneumonia each have only two positive samples.The authors propose evaluating on the entire IU dataset instead.
  • Dataset Evaluation: The proposed full-dataset IU evaluation is intended to ease disease-aware assessment compared with the commonly adopted 7:1:2 train-validation-test partition.The 7:1:2 partition was introduced by Chen et al. and later adopted by multiple works.

Hyperparameter Analysis

Hyperparameter analysis studies the loss-balance coefficient λ and the number k of retrieved reports using F1 and BLEU-4 on the MIMIC test set. Performance is stable across broad ranges, with the selected settings near the optimum.

  • Hyperparameters: λ balances LSDL, and k specifies the number of reports retrieved by CFE.The analysis evaluates both hyperparameters using F1 and BLEU-4.
  • λ Analysis: Across λ values from 0.2 to 12, F1 and BLEU-4 fluctuations remain within 1.2% and 0.5%, respectively.λ = 4 gives the best F1 and BLEU-4 scores used in the experiments.
  • λ Analysis: Too-small λ can weaken classification training, whereas too-large λ can impair language-model learning and diagnostic-information fusion.This reflects the balance between classification and report-generation objectives.

N-gram Statistics

PromptMRG with diagnosis-driven prompts generates fewer high-frequency phrases than the version without DDP on the MIMIC and IU tests, with comparable report lengths.

  • Table 2 reports comparisons with state-of-the-art methods using macro-averaged clinical efficacy metrics on MIMIC and IU.The table marks larger-than-224 image sizes, results evaluated by the authors, and best results in bold.
  • The method without DDP generates more high-frequency phrases than the method with DDP across all selected phrases on MIMIC.The phrases were selected from frequent 4-grams, 5-grams, 6-grams, and 7-grams in the MIMIC training set.
  • On average, the method without DDP generates twice as many high-frequency phrases as the method with DDP.Both methods produce reports averaging around 60 words, making report length comparable.

Macro-Averaged Results

Macro-averaged clinical efficacy metrics show larger gains for PromptMRG over existing methods than example-based metrics, while SDL improves disease-balanced diagnosis.

  • 6.3% is PromptMRG’s absolute improvement over RGRG on MIMIC macro-averaged F1.The corresponding improvement under example-based F1 is 2.9%.
  • 5.9% is PromptMRG’s improvement on IU macro-averaged F1, compared with 3.1% under example-based F1.The comparison indicates a larger gap after switching from example-based to macro-averaged F1.
  • 6.2% is the absolute improvement on MIMIC F1 when SDL is applied.The comparison supports SDL’s effectiveness in delivering disease-balanced diagnosis.

More Qualitative Results

Qualitative examples show PromptMRG making more accurate disease predictions than the baseline, including cardiomegaly and pleural effusion cases.

  • PromptMRG mostly correctly predicts diseases in the first example, whereas the baseline misses cardiomegaly and falsely predicts cardiac-silhouette enlargement and pneumonia.PromptMRG still incorrectly predicts opacity in this example.
  • PromptMRG identifies pleural effusion as positive in the second example, while the baseline fails to identify it.Figure 4 uses blue text for content consistent with ground truth and red text for incorrect content.
  • Figures 2 and 3 compare high-frequency phrase counts between methods with and without DDP on the MIMIC and IU tests.Figure 4 provides additional qualitative comparisons between the baseline and PromptMRG.
Loading 2308.12604v2…