Source-linked AI summary
Inspecting state of the art performance and NLP metrics in image-based medical report generation
Pablo Pino, Denis Parra, Pablo Messina, Cecilia Besa, Sergio Uribe
TL;DR
Image-based medical report generation is commonly assessed with NLP metrics, although their relationship to clinical correctness is uncertain. The paper benchmarks weak baselines against state-of-the-art models and finds that some weak baselines nearly match some state-of-the-art systems on BLEU and ROUGE-L, motivating closer study of clinical evaluation.
Problem
NLP metrics such as BLEU and ROUGE-L are commonly used for medical report generation, but their ability to measure clinical correctness is questioned.
Source: §1 · p. 1Method
The authors compare weak baselines and CNN-LSTM models against state-of-the-art systems using NLP metrics and the CheXpert Labeler.
Source: §2 · p. 1, §2.1 · p. 2Results
Some weak baselines perform almost as well as some state-of-the-art models on BLEU and ROUGE-L, leaving their clinical comparison unresolved.
Source: §3 · p. 2Takeaways & Limitations
The authors call for evaluating clinical accuracy, including radiologist assessment and testing MIRQI for report correctness.
Source: §3 · p. 2, Broader Impact · p. 3Scalable resources for expert physicians to judge generated-report quality are lacking.
Source: Broader Impact · p. 3
Abstract
from arXiv · showhide
Several deep learning architectures have been proposed over the last years to deal with the problem of generating a written report given an imaging exam as input. Most works evaluate the generated reports using standard Natural Language Processing (NLP) metrics (e.g. BLEU, ROUGE), reporting significant progress. In this article, we contrast this progress by comparing state of the art (SOTA) models against weak baselines. We show that simple and even naive approaches yield near SOTA performance on most traditional NLP metrics. We conclude that evaluation methods in this task should be further studied towards correctly measuring clinical accuracy, ideally involving physicians to contribute to this end.
1 Introduction
Medical report generation from images has been approached with deep learning, but commonly used NLP metrics may not capture clinical correctness. The paper examines whether weak baselines can match state-of-the-art models on these metrics.
- Prior systems commonly use CNN encoders and RNN decoders to generate reports, and are evaluated with BLEU, ROUGE-L, or CIDEr-D [e.g. 7, 9, 12, 14, 15].Source: §1 · p. 1
- NLP metrics have been questioned as measures of medical correctness, and small score improvements may not indicate better report quality [9] [17].Source: §1 · p. 1
- The study benchmarks weak baselines against state-of-the-art models on the IU X-ray dataset and finds comparable scores on NLP metrics.Source: §1 · p. 1
- A constant report can score BLEU 0.56 and ROUGE-L 0.676 while stating incorrect clinical facts in two sentences [2].Source: Figure 1 · p. 1
2 Methods and Results
The study compares weak baselines and CNN-LSTM models with state-of-the-art systems using NLP and CheXpert metrics. Weak baselines approach or exceed some state-of-the-art scores on BLEU and ROUGE-L, while the clinical metrics raise concerns about disease detection.
2 Methods and Results
- Five weak baselines include random, constant, nearest-neighbor, top-sentences-N, and top-words-N report generation.Source: §2 · p. 2
- CNN-LSTM variants generate reports word by word; the attention variant adds attended spatial image features, and DenseNet-121 performed best among tested CNNs.The tested CNNs were MobileNet-v2, ResNet-50, and DenseNet-121.Source: §2 · p. 2
2.1 Results
- CheXpert metrics indicate that models do not detect diseases correctly; the baseline AUC for binary classification is 0.5.Simple baselines can achieve high accuracy because about 76% of dataset sentences describe no abnormalities.Source: §2.1 · p. 2
- CIDEr-D shows a strong difference between two state-of-the-art models and naive baselines despite similar accuracy.Source: §2.1 · p. 2
- Weak baselines achieve scores near to or better than state-of-the-art models on BLEU and ROUGE-L in Table 1.Source: Table 1 · p. 2, §2.1 · p. 2
3 Conclusions and future work
Some weak baselines perform almost as well as some state-of-the-art models on BLEU and ROUGE-L, leaving uncertain whether the models lack clinical advantage or the metrics fail to distinguish them. The authors identify clinical correctness and medical judgment as open evaluation questions.
- Some weak baselines perform almost as well as some state-of-the-art models on BLEU and ROUGE-L, but clinical significance remains unresolved.Source: §3 · p. 2
- The authors plan to ask radiologists to assess generated reports and test MIRQI, a metric designed to measure clinical correctness.Source: §3 · p. 2
- The paper raises whether NLP metrics correlate with medical judgment and how to measure correctness directly in generated reports.Source: §3 · p. 2
Broader Impact
The paper calls for evaluating medical report generation by clinical accuracy rather than relying on traditional NLG metrics. It identifies expert human judgment as the gold standard despite limited scalable physician resources.
- Expert physician judgment is presented as the gold standard for assessing generated-report quality, despite a lack of scalable resources.Source: Broader Impact · p. 3
- The paper urges further study of BLEU and ROUGE appropriateness and development of metrics that measure medical accuracy.Source: Broader Impact · p. 3