PaperlayerView exact revision on arXiv

Source-linked AI summary

Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation

Ryutaro Tanno, David G. T. Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Karan Singhal, Shekoofeh Azizi, Tao Tu, Mike Schaekermann, Rhys May, Roy Lee, SiWai Man, Zahra Ahmed, Sara Mahdavi, Yossi Matias, Joelle Barral, Ali Eslami, Danielle Belgrave, Vivek Natarajan, Shravya Shetty, Pushmeet Kohli, Po-Sen Huang, Alan Karthikesalingam, Ira Ktena

arXiv:2311.18260v3eess.IVcs.CLcs.CVcs.LG

TL;DR

Radiology report generation faces a shortage of clinicians and unresolved questions about the clinical quality of AI-generated reports. This study evaluates Flamingo-CXR across two care settings and countries, including an assistive workflow; revised AI reports were preferred to expert-only reports in most evaluated cases.

  • Problem

    Existing report-generation models leave headroom for clinical-accuracy improvement, while common language-generation metrics inadequately assess factual correctness and consistency.

    Source: §1 · p. 2, Automated measures of report quality · p. 4

  • Method

    The authors specialize a vision-language foundation model as Flamingo-CXR and evaluate its reports with certified radiologists in autonomous and clinician-assisted settings across US intensive care and Indian inpatient data.

    Source: §1 · p. 3, §3 · p. 11

  • Results

    At least one radiologist preferred or rated the AI report equivalent to the ground truth in over 60% of cases in both datasets; revised AI reports were preferred by at least one radiologist in 80% of IND1 and 66% of MIMIC-CXR cases.

    Source: §3 · pp. 12–13

  • Takeaways & Limitations

    The proof-of-concept evaluation indicates initial promise for AI assistance that augments physicians’ report-writing process.

    Source: §3 · p. 13

    AI reports can under-report findings and make errors in counting and spatial reasoning, including incorrect finding locations.

    Source: §3 · p. 13

Abstract

from arXiv · show

Radiology reports are an instrumental part of modern medicine, informing key clinical decisions such as diagnosis and treatment. The worldwide shortage of radiologists, however, restricts access to expert care and imposes heavy workloads, contributing to avoidable errors and delays in report delivery. While recent progress in automated report generation with vision-language models offer clear potential in ameliorating the situation, the path to real-world adoption has been stymied by the challenge of evaluating the clinical quality of AI-generated reports. In this study, we build a state-of-the-art report generation system for chest radiographs, $\textit{Flamingo-CXR}$, by fine-tuning a well-known vision-language foundation model on radiology data. To evaluate the quality of the AI-generated reports, a group of 16 certified radiologists provide detailed evaluations of AI-generated and human written reports for chest X-rays from an intensive care setting in the United States and an inpatient setting in India. At least one radiologist (out of two per case) preferred the AI report to the ground truth report in over 60$\%$ of cases for both datasets. Amongst the subset of AI-generated reports that contain errors, the most frequently cited reasons were related to the location and finding, whereas for human written reports, most mistakes were related to severity and finding. This disparity suggested potential complementarity between our AI system and human experts, prompting us to develop an assistive scenario in which Flamingo-CXR generates a first-draft report, which is subsequently revised by a clinician. This is the first demonstration of clinician-AI collaboration for report writing, and the resultant reports are assessed to be equivalent or preferred by at least one radiologist to reports written by experts alone in 80$\%$ of in-patient cases and 60$\%$ of intensive care cases.

1. Introduction

Radiology report generation could address clinical needs, but evaluating the clinical quality of generated reports and their usefulness as assistive tools remains difficult. The paper presents Flamingo-CXR and evaluates it in autonomous and assistive settings across US intensive care and Indian inpatient care.

1. Introduction

  • A global radiologist shortage limits access to expert care and contributes to heavy workloads, delays, and errors in clinical decisions.Radiology reports inform diagnosis, treatment, and patient management.Source: §1 · p. 1
  • Radiology reports synthesize imaging findings, clinical context, an overall impression, and recommendations, beyond narrow pathology classification or quantification.Vision-language progress and digitized multimodal radiology data have enabled systems to generate complete free-text image descriptions.Source: §1 · p. 2
  • Prior work left gaps in comprehensive expert evaluation, analysis of why reports were preferred, and assessment of AI report generation as an assistive tool.The passage notes that only one study directly compared preferences for AI- and human-generated reports, while only one evaluated utility in a narrow clinical setting.Source: §1 · p. 2
  • Most existing report-generation models were built from scratch, while pretrained vision-language foundation models offer an opportunity to transfer knowledge to specialized medical tasks.The passage frames this as an opportunity to improve clinical accuracy in report generation.Source: §1 · p. 2
  • The study presents Flamingo-CXR and evaluates generated reports in autonomous and clinician-assistive settings across US intensive care and Indian inpatient care.The evaluation moves beyond automated metrics to compare clinician preferences for AI-generated and human reports.Source: §1 · p. 3

2. Results

Flamingo-CXR was evaluated on chest X-rays from US and Indian settings using automated metrics, disease-label comparisons, and blinded radiologist assessments. Expert ratings found complementary errors across AI and human reports, while clinician revision improved preference ratings on the edited-case subset.

2.1. Adapting a vision-language foundation model to report generation

  • Flamingo-CXR fine-tuned a vision-language foundation model on MIMIC-CXR and IND1 data to generate findings and impression sections from frontal chest radiographs.The data came from US emergency-department care and Indian regional centers.Source: §2.1 · pp. 3–4

Automated measures of report quality

Disease classification in comparison to human radiologists

  • For frequent IND1 conditions, Flamingo-CXR matched or exceeded held-out radiologists’ Kendall’s tau agreement, but agreement was lower for rare edema and enlarged cardiomediastinum.The latter conditions had training-set prevalence rates of 0.19% and 0.15%, respectively.Source: Disease classification in comparison… · p. 5, Figure 2 · p. 6

2.2. Expert evaluation of AI-generated and human-written reports

  • Sixteen certified radiologists evaluated AI-generated and original reports in pairwise preference and error-correction tasks, with each report assessed by two radiologists.The evaluation covered 554 cases across MIMIC-CXR and IND1.Source: §2.2 · p. 6

Pairwise preference test

  • At least one rater judged AI reports equivalent or preferable in more than 60% of cases across both datasets; for normal cases, this held in 90%.MIMIC-CXR abnormal cases showed the largest human advantage, with raters agreeing the original was more accurate or descriptive in 38% of cases.Source: Pairwise preference test · p. 7, Figure 3 · p. 7

Error correction

  • Error rates favored AI reports on IND1 but favored original reports on MIMIC-CXR, while abnormal cases had more detected errors than normal cases in both datasets.Clinically significant errors appeared in 19.9% of Flamingo-CXR reports overall; the human-report comparison also identified non-negligible errors.Source: Error correction · p. 8, Figure 4 · p. 9
  • Reported errors were non-overlapping (73% for MIMIC-CXR; 79% for IND1), suggesting frequent inconsistency in detected issues between AI-generated and original reports and potential complementarity between AI and human experts.In 17% and 40% of such cases, clinically significant errors appeared in human reports but not corresponding AI-generated reports.Source: Error correction · p. 10

2.3. Clinician-AI collaboration

  • 80% of IND1 cases and 66% of MIMIC-CXR cases were rated equivalent or preferred by at least one radiologist after clinician revision, versus 63% and 54% for AI alone on edited cases.The comparison covers only cases with at least one edit in the initial error-correction task.Source: §2.3 · pp. 10–11

3. Discussion

The study evaluates Flamingo-CXR across US intensive-care and Indian inpatient settings, including autonomous reports and clinician-revised drafts. Expert assessments show promise alongside substantial variation in report quality, reader judgments, and the errors made by AI and humans.

3. Discussion

  • Earlier evaluations were less diverse and granular; this study includes autonomous and assistive scenarios in US intensive care and Indian inpatient settings, with categorized error types.A prior study on the same US dataset assumed ground-truth reports were correct, neglecting interpretation variability.Source: §3 · p. 12
  • 77% of IND1 normal cases received unanimous ratings that AI reports were at least equivalent to human reports, while MIMIC-CXR showed higher AI error rates.The authors note that MIMIC-CXR reports were generally more detailed and less templated than IND1 reports, and argue for evaluation across clinical contexts and regions.Source: §3 · p. 12
  • High inter-rater variability in error correction and preference judgments underscores the value of multiple readings per case.The authors contrast this with prior work that evaluated each case only once.Source: §3 · p. 12
  • Human and AI reports showed different errors: examples include human misclassification of cardiomegaly or pleural thickening, and AI under-reporting, counting, and location errors.Flamingo-CXR had only the current radiograph at 1 megapixel, whereas the original radiologists also had patient history and previous scans.Source: §3 · pp. 12–13
  • 80% of IND1 and 66% of MIMIC-CXR cases had AI-generated reports with expert revisions preferred by at least one radiologist over reports written by experts alone.The authors describe this as the first evaluation of chest-radiograph report generation in an assistive setting and a proof of concept.Source: §3 · p. 13
  • The assistive results were not perfect, with inter-rater variability and questionable edits among possible explanations; the study did not assess whether revision saves time.The authors also note that collaboration can produce a less accurate report than a clinician working alone and that more interactive forms of assistance remain to be developed.Source: §3 · p. 13

A.1. Model

Flamingo-CXR adapts a version of the Flamingo vision-language model to radiology report generation.

A.1. Model

  • Flamingo-CXR adapts a version of Flamingo, a vision-language model, to the radiology report generation task.The section says it covers the task, architecture, optimisation, and inference.Source: §A.1 · p. 19

A.2. Task

Flamingo-CXR generates findings and impression sections from frontal chest X-rays by fine-tuning a pretrained vision-language model. Its training combines data from US and Indian datasets, reweights healthy and abnormal cases, and adds an abnormality-classification loss.

A.2. Task

  • Flamingo-CXR generates the findings and impression sections of radiology reports from frontal chest X-rays.The task formulation follows recent report-generation studies.Source: §A.2 · p. 19
  • The model uses a 400-million-parameter Flamingo with a vision encoder, language model, and connective module to incorporate images into token prediction.The connective module combines a Perceiver Resampler and cross-attention layers.Source: §A.2 · p. 19
  • Fine-tuning minimizes weighted report negative log-likelihood across MIMIC-CXR and IND1, with data-specific coefficients and inverse-frequency weighting for healthy versus abnormal cases.The weighting is intended to penalize inaccurate reports equally across healthy and abnormal cases, especially given IND1’s high proportion of healthy cases.Source: §A.2 · p. 19
  • An auxiliary CheXpert-derived abnormality-classification loss was added, which the authors found helpful for sensitivity across conditions.Binary labels indicate whether any extracted thoracic condition is present.Source: §A.2 · p. 20
  • The language component is frozen during fine-tuning, while the vision encoder and connective component are updated; default report generation uses beam search with width 3.Nucleus sampling with p=0.9 is used to generate multiple reports for disease-classification ROC curves.Source: §A.2 · p. 20

A.3. Datasets and pre-processing

The study evaluates Flamingo-CXR on frontal chest radiographs from the Indian IND1 and US MIMIC-CXR datasets. Preprocessing removes MIMIC-CXR training examples referencing prior studies and standardizes image size and intensity.

A.3. Datasets and pre-processing

  • IND1 contains 263,021 frontal radiographs from five regional Indian centers, with 250,066 training, 4,960 validation, and 7,995 test samples.A 2,306-case subset has radiologist consensus labels for six thoracic conditions for classification evaluation.Source: §A.3 · p. 20
  • MIMIC-CXR’s official split provides 222,758 training, 1,808 validation, and 3,269 test examples before the study’s filtering.The study keeps frontal AP and PA scans, findings and impression sections, and reports containing an impression section.Source: §A.3 · p. 20
  • Because over 50% of MIMIC-CXR examples contain prior scans, the authors remove training examples referencing prior studies to reduce hallucinated references; the resulting split has 90,968 training, 688 validation, and 1,931 test examples.Evaluation metrics are still reported on all test examples for comparison with prior studies.Source: §A.3 · p. 21
  • Images from both datasets are resized to 320 × 320 with aspect ratio preserved, padded if needed, and normalized to zero mean and unit standard deviation.Training also uses color jitter and resize/crop augmentation.Source: §A.3 · p. 21

A.4. Expert evaluation

Sixteen certified radiologists in India evaluated reports using preference and error-correction tasks. The tasks used the same sampled cases, and raters checked image quality before annotation.

A.4. Expert evaluation

  • Sixteen certified radiologists in India performed the evaluations, with raters from the initial preference or error-correction phase excluded from the human-AI collaboration evaluation.The raters completed CITI training for MIMIC-CXR evaluation, and the interface was validated with an expert before large-scale evaluation.Source: §A.4 · p. 21
  • The study randomly sampled fixed numbers of normal and abnormal cases from IND1 and MIMIC-CXR, using more abnormal cases to improve abnormality coverage.The same cases were annotated in the error-correction and pairwise-preference tasks.Source: §A.4 · p. 21
  • In pairwise preference testing, raters saw the image and two reports without knowing which was AI-generated, then chose report A, report B, or equivalence and explained their choice.Raters could zoom in on the image as needed.Source: Figure 8 · p. 22
  • In error correction, raters assessed report accuracy, selected disagreements and their reason, judged clinical significance, and supplied replacement text.They first confirmed whether the image quality was sufficient to complete the task.Source: Figure 9 · p. 23

B.1. Automated evaluation using NLG and clinical metrics

The appendix reports automated evaluation of Flamingo-CXR on IND1 and MIMIC-CXR using NLG and clinical metrics. It identifies tables and ROC curves that expand the automated results presented in the main text.

B.1. Automated evaluation using NLG and clinical metrics

  • Tables 3 and 4 compare Flamingo-CXR with prior methods using NLG and clinical metrics on MIMIC-CXR and IND1, respectively.The supplied passage gives the tables’ scope but no metric values or comparative outcomes.Source: §B.1 · p. 24
  • Figure 10 expands Figure 2 with separate ROC curves for six clinical conditions: Cardiomegaly, Pleural Effusion, Lung Opacity, Edema, Enlarged Cardiomediastinum, and Fracture.The passage describes the figure’s coverage but does not state curve outcomes.Source: §B.1 · p. 24
  • The appendix’s automated evaluation covers both datasets and both language-generation and clinical metrics.The passage refers to detailed results precluded from the main text.Source: §B.1 · p. 24

B.2. Additional qualitative results

The examples show clinically significant errors in both directions: some occur in human reports but not Flamingo-CXR, while others occur in AI reports but not the ground truth.

B.2. Additional qualitative results

  • Table 3 compares MIMIC-CXR report-generation metrics across models, indicating which report sections each model generates; higher metric values are better.Source: Table 3 · p. 24
  • Table 4 reports IND1 automated report-generation metrics and disease-classification F1 scores for two radiologists; no published report-generation metrics exist for this private dataset.Source: Table 4 · p. 24
  • Figure 10 plots Flamingo-CXR's ROC curve and operating point against two radiologists' TPR and FPR pairs for six IND1 conditions.The operating point uses the default Beam 3 inference scheme.Source: Figure 10 · p. 25
  • Tables 5 and 6 illustrate clinically significant errors found only in the human report and only in the Flamingo-CXR report, respectively.These examples show differences in both directions between AI-generated and ground-truth reports.Source: §B.2 · p. 24, Table 5 · p. 26, Table 6 · p. 27

Related papers

Loading 2311.18260v3…