PaperlayerView exact revision on arXiv

Source-linked AI summary

RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance

Chantal Pellegrini, Ege Özsoy, Benjamin Busam, Nassir Navab, Matthias Keicher

arXiv:2311.18681v3cs.CVcs.CL

TL;DR

Radiology report generators can be factually unreliable, while many medical vision-language systems offer limited interaction. RaDialog combines image features and structured findings with LLM instruction training, and reports gains in report generation and interactive tasks, with radiologists preferring it in 84.0% of cases.

  • Problem

    Existing radiology report generators can struggle with factual correctness, while many medical vision-language models are limited to visual question answering or single-step reporting rather than robust interaction.

    Source: §1 · p. 2

  • Method

    RaDialog integrates image features and structured findings with an LLM and trains it using a semi-automatically labeled, image-grounded instruct dataset for chest X-ray tasks.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    84.0% of cases: radiologists preferred RaDialog, which also outperformed XRayGPT in direct comparisons and showed gains in report generation and interactive tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    RaDialog supports report generation, follow-up dialogue, and feedback incorporation for expert-in-the-loop quality control in radiology.

    Source: §5 · p. 9

    In a 50-prediction MIMIC-CXR test-set analysis, frequent errors involved less common findings and severity, with occasional incorrect details about finding or device locations.

    Source: §4 · p. 9

Abstract

from arXiv · show

Conversational AI tools that can generate and discuss clinically correct radiology reports for a given medical image have the potential to transform radiology. Such a human-in-the-loop radiology assistant could facilitate a collaborative diagnostic process, thus saving time and improving the quality of reports. Towards this goal, we introduce RaDialog, the first thoroughly evaluated and publicly available large vision-language model for radiology report generation and interactive dialog. RaDialog effectively integrates visual image features and structured pathology findings with a large language model (LLM) while simultaneously adapting it to a specialized domain using parameter-efficient fine-tuning. To keep the conversational abilities of the underlying LLM, we propose a comprehensive, semi-automatically labeled, image-grounded instruct dataset for chest X-ray radiology tasks. By training with this dataset, our method achieves state-of-the-art clinical correctness in report generation and shows impressive abilities in interactive tasks such as correcting reports and answering questions, serving as a foundational step toward clinical dialog systems. Our code is available on github: https://github.com/ChantalMP/RaDialog.

1. Introduction

Existing radiology report generators can produce coherent text yet struggle with factual correctness, while many medical vision-language models lack robust interaction. RaDialog combines image features and structured findings with instruction training, reporting gains in report generation and interactive tasks.

1. Introduction

  • Prior report-generation methods can struggle with factual correctness, and many medical vision-language models focus on visual question answering or single-step reporting rather than robust interaction.The introduction frames these gaps against the need for automated chest X-ray reporting and collaborative diagnostic assistance.Source: §1 · pp. 1–2
  • RaDialog adds a secondary visual feature branch for structured clinical findings, which the paper reports improves clinical correctness by 13.3%.The pipeline also integrates image features and structured findings with conversation history and task-specific instructions before querying the LLM.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors propose variable instruct training and a semi-automatically labeled, image-grounded X-ray dataset to support dialog tasks and address catastrophic forgetting during LLM fine-tuning.They also use context dropping so the model must consider image information across tasks.Source: §1 · p. 2
  • Radiologists preferred RaDialog in 84.0% of cases, and it outperformed XRayGPT in direct comparisons across report-generation and interactive tasks.The paper presents RaDialog as a step toward clinical dialog systems for radiology.Source: §1 · p. 3

2. Methodology

RaDialog combines image and structured-finding inputs with conversation context and instructions for an LLM. Its training uses a diverse image-grounded instruct dataset and varies how much report text is available.

2.1. Model and Training

  • The visual feature extractor offers two adapters: RaDialog-align maps features through a BERT alignment module into 32 LLM tokens, while RaDialog-project directly projects image patches into 196 tokens.The align module is separately trained and frozen during later LLM training; the project variant trains the encoder and adapter jointly with the LLM without pretraining.Source: §2.1 · p. 3
  • A separate CLIP-based Structured Findings Extractor classifies core findings and converts positive labels into text for the model’s report prompt.It is trained for multi-label classification with log-weighted binary cross-entropy to address class imbalance.Source: §2.1 · pp. 3–4
  • The prompt concatenates image tokens, structured findings, conversation history, and task instructions, then an LLM fine-tuned with LoRA generates the response.Fine-tuning on radiology reports and instructions adapts medical knowledge and writing style while using parameter-efficient adaptation.Source: §2.1 · p. 4

2.2. Instruct Dataset

  • The instruct dataset contains about 580k samples spanning ten tasks, combining tasks built on existing datasets with replay tasks using pseudo-ground truth from a non-fine-tuned LLM.Tasks include report and impression generation, question answering, explanations, view classification, correction, summarization, easy-language rewriting, and region QA.Source: §2.2 · p. 4
  • Context dropping randomly supplies full, no, or partial report context for image-answerable tasks, with partial mode replacing the report in conversation history with half its sentences randomly removed.The augmentation is intended to reduce over-reliance on text and support image use when textual context is incomplete.Source: §2.2 · pp. 4–5

3. Experimental Setup

The evaluation uses MIMIC-CXR for in-distribution testing and IU-Xray for out-of-distribution testing, with clinical efficacy, text similarity, and language-generation metrics.

3. Experimental Setup

  • The study uses official MIMIC-CXR splits, comprising 377,110 chest X-rays and reports, and the IU-Xray test split for out-of-distribution evaluation.Samples with empty findings sections are excluded, and the evaluated models include report-only and instruct-trained variants.Source: §3 · p. 5
  • Clinical efficacy is measured as macro F1 over 14 CheXbert labels, alongside BertScore and BLEU, ROUGE, and METEOR metrics.The authors include conventional NLG metrics for completeness while noting they are not ideal measures of clinical correctness.Source: §3 · p. 5
  • Table 1 compares RaDialog with recent medical LVLMs; its notes identify fine-tuning status and flag multi-view-trained models evaluated with a single view.Source: Table 1 · p. 6

4. Results and Discussion

RaDialog outperforms other methods in clinical efficacy and BertScore for chest X-ray report generation, including out-of-distribution evaluation. Its instruct-trained variants also perform across interactive tasks, while component ablations and failure analysis identify important design choices and limits.

Radiology report generation

  • RaDialog significantly outperforms other methods in clinical efficacy and BertScore on MIMIC-CXR and IU-Xray, including out-of-distribution evaluation.RaDialogalign and RaDialogproject generate reports at 112 and 223 tokens per second, respectively.Source: §4 · p. 6

Ablation of Architectural Components

  • Fine-tuning the LLM improves performance, and the best ablation results combine visual and structured inputs.Table 2 compares a non-fine-tuned LLM with visual and structured inputs.Source: §4 · p. 6, Table 2 · p. 7

Interactive Downstream Tasks

  • Instruct training significantly improves impression generation over the report-only model and CheXagent, while RaDialog reaches an almost perfect view-classification score.Table 3 reports results on impression generation and view classification.Source: §4 · pp. 6, 8, Table 3 · p. 7
  • Around 33% improvement in report correction exceeds report-only models’ 10–25% and XRayGPT’s 10%; instruct models also outperform report-only models on findings prediction.The evaluation asks models to correct incorrect pathologies and tests binary or complete CheXpert findings prediction.Source: §4 · p. 7, Table 4 · p. 7
  • On Rad-ReStruct QA, RaDialogins attains F1 29.5, precision 61.8, and recall 41.6, exceeding hi-VQA’s recall of 33.3 while maintaining competitive precision and F1.The report-only model’s similar F1 of 28.7 accompanies recall of 28.8 and precision of 98.1, largely because it almost always predicts a negative answer.Source: §4 · p. 8

Limitations

  • In a 50-prediction failure analysis, common errors involved less frequent findings, severity judgments, and minor incorrect details; the authors frame the model’s utility around simpler cases and initial drafts.The authors describe radiologist-AI collaboration as part of that use.Source: §4 · p. 9

5. Conclusion

RaDialog combines report generation with interactive assistance, using structured findings and instruct training to support clinically accurate reports and dialogue. The conclusion highlights context dropping as a way to maintain image attention throughout conversations.

5. Conclusion

  • RaDialog is a large vision-language model for radiology report generation and auxiliary interactive assistance.It can engage in dialogue, answer follow-up questions, and incorporate feedback for expert quality control.Source: §5 · p. 9
  • A secondary image branch supplies structured radiology findings, while instruct training on a publicly available dataset supports domain-specific conversation and avoids catastrophic forgetting.Context dropping is used to enforce attention on the input image throughout the conversation.Source: §5 · p. 9
  • The authors present RaDialog as a step from static report generation toward a collaborative tool for clinical practice.Source: §5 · p. 9

Appendix A. Additional report generation results

RaDialog leads prior methods on clinical efficacy and BertScore in the reported MIMIC-CXR comparison. The appendix also describes which methods were comparable and what distinguishes RaDialog’s configuration.

Appendix A. Additional report generation results

Appendix B. Details on LVLM Configurations

  • RaDialog’s configuration differs through end-to-end fine-tuning, its dual-branch architecture, and task-specific RaDialog-Instruct training, not larger encoders, more image tokens, or a larger LLM.Source: Appendix B · p. 14, Table 6 · p. 15

C.1. Task Descriptions

The appendix describes ten X-ray tasks and their data or supervision sources, then gives example instruction prompts for several tasks. These tasks span report writing, question answering, and interactive report assistance.

C.1. Task Descriptions

  • Report-generation tasks produce findings from an X-ray and an impression from findings, using MIMIC-CXR reports as ground truth.Source: §C.1 · p. 15
  • Question-answering tasks cover CheXpert findings, detailed Rad-ReStruct existence, location and appearance, and binary or open-ended questions about specific regions.Source: §C.1 · p. 15
  • Other tasks classify the X-ray view, simplify or summarize reports, correct report errors, and explain which report text indicates a pathology.Source: §C.1 · pp. 15–16

C.2. Instruction Prompts

  • Example prompts vary how users request report generation, impressions, correction, explanations, view classification, and multiple forms of question answering.Source: §C.2 · p. 16
  • The prompt examples also include questions about heart, bone, and lung abnormalities, as well as requests for simpler language and report summaries.Source: §C.2 · p. 17

Appendix D. Additional Qualitative Results

Qualitative examples compare RaDialog’s report generation with other models on an out-of-domain IU-Xray image and contrast its instruct variants in zero-shot conversation.

Appendix D. Additional Qualitative Results

  • The report-generation comparison uses the same concise radiologist-style prompt across RaDialog, XrayGPT, GPT-4o, and Gemini-2.0.Source: Appendix D · p. 17
  • On an out-of-domain IU-Xray image, RaDialog identifies almost all findings correctly, while GPT-4o and Gemini-2.0 miss most and XrayGPT gets fewer right and hallucinates more.Source: Appendix D · p. 17, Figure 4 · p. 18
  • Figure 5 contrasts RaDialog-project-instruct and RaDialog-align-instruct on zero-shot conversational question answering.Source: Appendix D · pp. 17–18, Figure 5 · p. 18

E.1. Indication-based Methods

With indication text supplied, RaDialog surpasses several indication-based models on reported metrics. Scaling the LLM alone brings little performance gain while slowing inference.

E.1. Indication-based Methods

  • Adding the indication section to RaDialog leads to a significant jump in NLG metrics; the comparison addresses competitors that also use indication input.Source: §E.1 · p. 18, Table 7 · p. 19
  • With indication input, RaDialog outperforms MedPaLM-12b and MAIRA-1 7b on all metrics, and MAIRA-1 84b on text-based metrics, with comparable clinical efficacy.Source: §E.1 · p. 18, Table 7 · p. 19

E.2. Impact of Model Size

  • Scaling up the LLM does not yield a relevant performance increase and slows inference, so the authors use the 7-billion-parameter version.Source: Table 8 · p. 19, §E.2 · p. 19

Appendix F. Implementation Details

The appendix specifies the models, optimization settings, and training schedules used for RaDialog and its visual-feature and alignment components.

Appendix F. Implementation Details

  • RaDialog initializes its LLM with vicuna-7b and fine-tunes it using LoRA, with learning rates and epoch counts varying by model variant.RaDialog-align-instruct/report use a 3 × 10−4 learning rate for one or four epochs; RaDialog-project uses 2×10−5 for up to five epochs with early stopping.Source: Appendix F · p. 19
  • BioVil-T is fine-tuned to classify CheXbert findings with log-weighted cross-entropy and then used for visual feature extraction.Its classifier is trained for six epochs at a learning rate of 5 × 10−5.Source: Appendix F · p. 19
  • RaDialog-align uses BERT to align text and image features, training for four epochs with cosine annealing and linear warmup.The learning rate follows a cosine annealing schedule from 1 × 10−5 to 1 × 10−4.Source: Appendix F · p. 19
Loading 2311.18681v3…