Source-linked AI summary
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation
Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, Yuexian Zou
TL;DR
Radiology report generation could reduce reporting burden and support abnormality recognition, but data-driven systems struggle with visual and textual bias. PPKED imitates radiologists by exploring abnormal regions and prior knowledge, then distilling these sources into reports; it outperforms previous state-of-the-art models on MIMIC-CXR and IU-Xray.
Problem
Radiology report generation is difficult because visual and textual data biases can cause models to miss rare abnormalities and produce general reports.
Method
PPKED uses PoKE to extract abnormal regions, PrKE to encode retrieved reports and a medical knowledge graph, and MKD to distill the explored knowledge into reports.
Results
PPKED outperforms previous state-of-the-art models on both MIMIC-CXR and IU-Xray.
Takeaways & Limitations
The approach generates meaningful and robust reports supported by accurate abnormal descriptions and regions.
Abstract
from arXiv · showhide
Automatically generating radiology reports can improve current clinical practice in diagnostic radiology. On one hand, it can relieve radiologists from the heavy burden of report writing; On the other hand, it can remind radiologists of abnormalities and avoid the misdiagnosis and missed diagnosis. Yet, this task remains a challenging job for data-driven neural networks, due to the serious visual and textual data biases. To this end, we propose a Posterior-and-Prior Knowledge Exploring-and-Distilling approach (PPKED) to imitate the working patterns of radiologists, who will first examine the abnormal regions and assign the disease topic tags to the abnormal regions, and then rely on the years of prior medical knowledge and prior working experience accumulations to write reports. Thus, the PPKED includes three modules: Posterior Knowledge Explorer (PoKE), Prior Knowledge Explorer (PrKE) and Multi-domain Knowledge Distiller (MKD). In detail, PoKE explores the posterior knowledge, which provides explicit abnormal visual regions to alleviate visual data bias; PrKE explores the prior knowledge from the prior medical knowledge graph (medical knowledge) and prior radiology reports (working experience) to alleviate textual data bias. The explored knowledge is distilled by the MKD to generate the final reports. Evaluated on MIMIC-CXR and IU-Xray datasets, our method is able to outperform previous state-of-the-art models on these two datasets.
1. Introduction
Radiology report generation is motivated by the workload and diagnostic risks of manual reporting, but visual and textual data biases make rare abnormalities difficult to capture. PPKED addresses these biases by exploring posterior abnormal-region knowledge and prior medical and reporting knowledge, then distilling them for report generation.
- Automated reporting can reduce radiologists’ writing burden and remind less experienced radiologists about abnormalities that might otherwise be omitted.
- Image-captioning-style systems face visual bias because normal images dominate datasets, obscuring rare and diverse abnormal regions.
- Reports also exhibit textual bias because descriptions of normal regions and repeated normality sentences dominate the training text.
- PPKED models radiologists’ workflow with PoKE for abnormal regions, PrKE for prior knowledge, and MKD for knowledge distillation.PrKE uses retrieved prior radiology reports and a medical knowledge graph as working experience and medical knowledge.
- PPKED outperforms previous state-of-the-art models on both MIMIC-CXR and IU-Xray.The paper reports experiments and analyses on both public datasets as evidence of effectiveness.
2. Related Works
Radiology report generation extends image captioning and paragraph generation by requiring structured, region-specific medical observations. Existing systems are challenged by data deviation, while PPKED adds explicit abnormal-region extraction and broader prior-knowledge guidance.
- Image Captioning and Paragraph Generation: Radiology report generation produces a multi-sentence paragraph in which each sentence focuses on a medical observation for a specific image region.
- Image Captioning and Paragraph Generation: Hierarchical recurrent networks use paragraph-level and sentence-level recurrent modules to generate long descriptions from image features.
- Radiology Report Generation: Writing reports is time-consuming for experienced radiologists and error-prone for inexperienced radiologists.
- Radiology Report Generation: Existing HRNN-based systems can generate fluent but general reports because data deviation weakens visual grounding and abnormal narratives.
- Radiology Report Generation: Prior approaches separate normal and abnormal writing or combine template retrieval with generation to mitigate data deviation.
- Radiology Report Generation: PPKED differs by explicitly extracting abnormal regions and using many retrieved reports plus a medical knowledge graph as latent guidance.The approach treats retrieved reports as latent guidance rather than fixed templates, partly because templates may change over time.
3. Posterior-and-Prior Knowledge Exploring-and-Distilling (PPKED)
PPKED models radiologists’ workflow by extracting abnormal regions, retrieving relevant medical and reporting knowledge, and distilling both sources into radiology reports.
- Backgrounds: PPKED takes image features, a fixed topic bag, retrieved reports, and a medical knowledge graph as inputs to generate reports.The image features use ResNet-152; the topic bag contains 20 common topics or findings, while prior knowledge comes from retrieved reports and graph nodes.
- Basic Module: Multi-head attention computes associations among features, enabling the model to correlate posterior and prior knowledge before report generation.The basic module combines multi-head attention with a feed-forward network, followed by dropout, residual connection, and layer normalization.
- Posterior Knowledge Explorer (PoKE): PoKE identifies abnormal regions by first selecting relevant disease topics and then mining topic-related image features.The resulting regions are aligned with relevant topics, imitating radiologists’ assignment of disease topic tags during image examination.
- Prior Knowledge Explorer (PrKE): PrKE uses abnormal-region features to retrieve prior knowledge from radiology reports and a medical knowledge graph.The attended outputs represent prior knowledge related to the input image’s abnormalities and are intended to alleviate textual data bias.
- Multi-domain Knowledge Distiller (MKD): MKD decodes the final report from posterior and prior knowledge, using Adaptive Distilling Attention to retain useful correlated knowledge.At each decoding step, it receives the current word embedding and fixed position embedding, then predicts the next word.
4. Experiments
Experiments on IU-Xray and MIMIC-CXR evaluate PPKED against established report-generation systems and analyze its components. PPKED outperforms prior state-of-the-art methods, while ablations and qualitative analyses associate its gains with exploring and distilling posterior and prior knowledge.
- Datasets, Metrics and Settings: The experiments use IU-Xray and MIMIC-CXR, with IU-Xray reduced to 6,471 images and 3,336 reports after filtering.IU-Xray uses 70%-10%-20% train-validation-test splits, while MIMIC-CXR uses official splits.
- Main Results: PPKED outperforms previous state-of-the-art models across all metrics on both MIMIC-CXR and IU-Xray.The comparison includes radiology report-generation systems and image-captioning models.
- Effect of Posterior Knowledge Explorer: PoKE raises CIDEr from 0.261 to 0.285 over the Base model and improves recognition of selected abnormalities.The evaluated categories are Emphysema, Pneumonia, Cardiomegaly, Pneumothorax and Lesion.
- Effect of Prior Knowledge Explorer: Both PrMK and PrWE improve performance, with PrWE producing larger gains than PrMK.The analysis links PrWE to incorporating similar reports and improving descriptions of normal regions.
- Effect of Prior Knowledge Explorer: Combining PrMK with PrWE yields overall improvement, while combining PoKE and PrKE unites their respective mitigation of visual and textual data biases.The paper evaluates these combinations in settings (d) and (e).
- Effect of Multi-domain Knowledge Distiller: The Transformer-based decoder outperforms the LSTM-based decoder, and ADA further boosts performance by adaptively distilling explored knowledge.Figure 3 visualizes attended regions, retrieved reports, graph nodes, distilling weights and text alignment.
- Effect of Multi-domain Knowledge Distiller: ADA assigns larger distilling weights to abnormality sentences, whereas retrieved reports contribute more strongly when generating normality sentences.The reported pattern is attributed to abnormality knowledge in G′ Pr and normality knowledge in retrieved reports W′ Pr.
- Qualitative Analysis: Qualitative analysis reports alignment with ground-truth reports, accurate abnormal descriptions, and correspondence between explored prior knowledge and visualized abnormalities.The analysis characterizes PPKED reports as structured and robust, with balanced normal and abnormal sentences.
5. Conclusion
PPKED explores and distills posterior and prior knowledge for radiology report generation, imitating radiologists’ working patterns to address data bias. Experiments on MIMIC-CXR and IU-Xray demonstrate effectiveness and performance beyond previous state-of-the-art models.
- PPKED explores and distills posterior and prior knowledge to alleviate data bias in radiology report generation.The approach imitates radiologists’ working patterns.
- Experiments on MIMIC-CXR and IU-Xray verify the authors’ arguments and demonstrate the method’s effectiveness.
- The approach generates meaningful and robust reports supported by accurate abnormal descriptions and regions.
- PPKED outperforms previous state-of-the-art models on the two public datasets.