Source-linked AI summary
AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation
Di You, Fenglin Liu, Shen Ge, Xiaoxia Xie, Jing Zhang, Xian Wu
TL;DR
Medical report generation must address normal-region and normal-image dominance alongside very long output sequences. AlignTransformer combines disease-tag-guided hierarchical visual alignment with a multi-grained Transformer decoder, achieving competitive results on IU-Xray and MIMIC-CXR and favorable radiologist evaluation.
Problem
Medical report generation is challenged by data bias, in which normal visual content dominates abnormal content, and by very long report sequences.
Method
AlignTransformer hierarchically aligns predicted disease tags with visual regions to obtain multi-grained disease-grounded features, then uses a gated Multi-Grained Transformer to generate reports.
Results
AlignTransformer achieves competitive results with state-of-the-art methods on IU-Xray and MIMIC-CXR, with human evaluation by radiologists further supporting its effectiveness.
Takeaways & Limitations
The framework generates meaningful reports with accurate abnormal descriptions and regions and is reported as useful for assisting radiologists and reducing their workload.
Abstract
from arXiv · showhide
Recently, medical report generation, which aims to automatically generate a long and coherent descriptive paragraph of a given medical image, has received growing research interests. Different from the general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to 1) the serious data bias: the normal visual regions dominate the dataset over the abnormal visual regions, and 2) the very long sequence. To alleviate above two problems, we propose an AlignTransformer framework, which includes the Align Hierarchical Attention (AHA) and the Multi-Grained Transformer (MGT) modules: 1) AHA module first predicts the disease tags from the input image and then learns the multi-grained visual features by hierarchically aligning the visual regions and disease tags. The acquired disease-grounded visual features can better represent the abnormal regions of the input image, which could alleviate data bias problem; 2) MGT module effectively uses the multi-grained features and Transformer framework to generate the long medical report. The experiments on the public IU-Xray and MIMIC-CXR datasets show that the AlignTransformer can achieve results competitive with state-of-the-art methods on the two datasets. Moreover, the human evaluation conducted by professional radiologists further proves the effectiveness of our approach.
1 Introduction
Medical report generation aims to reduce radiologists’ workload, but existing image-captioning approaches struggle with rare abnormalities, data bias, and long reports. AlignTransformer addresses these challenges by aligning disease tags with multi-grained visual regions and using a Transformer decoder, achieving competitive results on two datasets.
- Medical report generation can assist radiologists with diagnosis and reduce the workload of writing reports.
- Existing models are biased toward normal images and regions, so they may omit rare but important abnormalities.Abnormal regions occupy only a small part of many medical images.
- Align Hierarchical Attention predicts disease tags and hierarchically aligns them with visual regions to extract disease-grounded features focused on abnormalities.The alignment operates across coarse and fine visual granularities.
- Multi-Grained Transformer adaptively exploits coarse- and fine-grained disease-grounded features through gating and generates reports with a Transformer.The design targets the generation of long medical reports.
- AlignTransformer targets data bias and long sequences, achieving competitive results with state-of-the-art methods on IU-Xray and MIMIC-CXR and receiving radiologist evaluation.
2 Related Works
Image captioning typically produces short descriptions of prominent visual content, whereas visual paragraph generation targets longer, coherent reports or stories. The related-work discussion situates medical report generation within these broader generation tasks.
- Image captioning generates descriptive sentences but usually produces short outputs focused on the most prominent visual contents.Such outputs may not fully convey an image’s rich feature information.
- Visual paragraph generation aims to generate long and coherent reports or stories.
3 Approach
AlignTransformer combines hierarchical alignment of disease tags with visual regions and a multi-grained Transformer decoder. The framework produces disease-grounded features at multiple granularities and adaptively uses them to generate long reports.
- Framework: The framework consists of an encoder that extracts visual features and a Transformer-based decoder that generates reports.The encoder is Align Hierarchical Attention, while the decoder is Multi-Grained Transformer.
- Align Hierarchical Attention: Multi-head attention aligns disease tags with visual regions by using tags to retrieve relevant regions and aligned regions to filter irrelevant tags.The resulting representations combine aligned visual and tag features.
- Align Hierarchical Attention: Align Hierarchical Attention repeats the alignment process hierarchically to obtain multi-grained disease-grounded visual features.The first round uses the original visual features and disease tags, and subsequent rounds produce additional aligned representations.
- Multi-Grained Transformer: The Multi-Grained Transformer uses adaptive exploiting attention to weight visual features of different granularities for each target word.The decoder contains L = 3 layers and uses learned weights to adaptively exploit the multi-grained features.
- Multi-Grained Transformer: The decoder predicts each next word from the final decoder-layer output using a softmax distribution and is trained with cross-entropy loss.The report targets are supplied by radiologists during training.
4 Experiments
Experiments evaluate AlignTransformer on IU-Xray and MIMIC-CXR using automatic metrics, comparisons with recent state-of-the-art models, and radiologist judgments. The framework achieves competitive results and is preferred over R2Gen in human evaluation.
- Datasets, Metrics and Settings: The experiments use the public IU-Xray and MIMIC-CXR datasets to evaluate medical report generation.IU-Xray contains 7,470 images and 3,955 reports; MIMIC-CXR contains 377,110 images and 227,835 reports.
- Automatic Evaluation: Table 1 compares AlignTransformer with representative models including PPKED, R2Gen, and SentSAT + KG.The evaluation reports BLEU, METEOR, and ROUGE-L, while Hit measures radiologist pick-up percentages.
- Automatic Evaluation: AlignTransformer achieves competitive results with previous state-of-the-art methods on both datasets under all reported metrics.This comparison is presented as evidence of the framework’s effectiveness.
- Human Evaluation: Radiologists preferred AlignTransformer over R2Gen when judging 200 randomly selected generated-report pairs.Two professional radiologists selected the more accurate report for each pair without knowing which model produced it.
- Ablation Study: The ablation study includes AHA and MGT configurations alongside a ResNet-50 encoder with the Transformer decoder baseline.The table defines N as the number of alignments in AHA and identifies the full model as w/ AHA+MGT.
5 Analysis
Analysis shows that AHA and MGT each improve the framework’s performance, while AHA provides disease-grounded abnormal information and MGT exploits multi-grained features.
- Ablation Study: AHA improves performance across all metrics, with up to 18.8% gain in BLEU-4 when N = 3.The result is attributed to disease-grounded visual features containing accurate abnormal information.
- Ablation Study: MGT further boosts performance under all metrics by adaptively exploiting multi-grained disease-grounded visual features.The analysis uses this result to support the effectiveness of the MGT module.
- Visualization and Example: Compared with R2Gen, AlignTransformer avoids some inexact descriptions and repeated normality sentences in the analyzed example.The analysis links those R2Gen outputs to the dominance of normal images in the dataset.
6 Conclusions
AlignTransformer targets data bias and very long sequences in medical report generation. Experiments report accurate abnormal descriptions and regions, competitive results on two public datasets, and the best human preference, supporting assistance for radiologists.
- Conclusion: AlignTransformer is designed to alleviate data bias and model very long sequences in medical report generation.Its reported purpose addresses both the dominance of normal content and the length of medical reports.
- Conclusion: The framework generates meaningful reports with accurate abnormal descriptions and regions.This conclusion is stated as an overall outcome of the proposed method.
- Conclusion: AlignTransformer achieves competitive results with state-of-the-art models on two public datasets and receives the best human preference.The conclusion reports both automatic-comparison and human-evaluation outcomes.
- Conclusion: The reported results support using the framework to assist radiologists in clinical decision-making and reduce their workload.This is the paper’s stated practical implication.