PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive and Explainable Region-guided Radiology Report Generation

Tim Tanida, Philip Müller, Georgios Kaissis, Daniel Rueckert

arXiv:2304.08295v1cs.CVcs.CLcs.LG

TL;DR

Radiology report generators often use image-level features rather than explicitly focusing on salient anatomical regions, with limited explainability and interactivity. RGRG detects regions, generates region-specific descriptions, and visually grounds sentences; on MIMIC-CXR, it outperforms competitive baselines on language-generation and clinically relevant metrics.

  • Problem

    Existing report generators rely on image-level features rather than salient anatomical regions, while offering limited explainability and interactivity in a safety-critical domain.

    Source: §1 · p. 1

  • Method

    RGRG detects localized anatomical regions, generates anatomy-specific sentences to compose reports, visually grounds sentences in bounding boxes, and supports radiologist selection or drawing of regions.

    Source: §1 · pp. 1–2

  • Results

    On MIMIC-CXR, RGRG outperforms competitive baselines on both language-generation and clinically relevant metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors describe radiologist involvement through interaction as crucial for integrating report-generation systems into real clinical settings.

    Source: §6 · p. 8

    The authors note that erroneous automated diagnostic evaluations can directly harm patients and that detailed explanations may exacerbate automation bias.

    Source: §6 · p. 8

Abstract

from arXiv · show

The automatic generation of radiology reports has the potential to assist radiologists in the time-consuming task of report writing. Existing methods generate the full report from image-level features, failing to explicitly focus on anatomical regions in the image. We propose a simple yet effective region-guided report generation model that detects anatomical regions and then describes individual, salient regions to form the final report. While previous methods generate reports without the possibility of human intervention and with limited explainability, our method opens up novel clinical use cases through additional interactive capabilities and introduces a high degree of transparency and explainability. Comprehensive experiments demonstrate our method's effectiveness in report generation, outperforming previous state-of-the-art models, and highlight its interactive capabilities. The code and checkpoints are available at https://github.com/ttanida/rgrg .

1. Introduction

Chest X-ray reporting is time-consuming, and existing methods can produce incomplete or inconsistent reports [32] while lacking regional focus, explainability, and interactivity. RGRG generates anatomy-specific descriptions that can be composed into a report and visually grounded in detected regions.

1. Introduction

  • RGRG uses object detection to extract localized anatomical features and generate individual, anatomy-specific sentences that compose the final report.The authors describe this regional divide-and-conquer approach as a way to support completeness and consistency.Source: §1 · pp. 1–2
  • Each generated sentence is visually grounded in a predicted bounding box, enabling radiologists to verify its correctness; existing models instead may show attention heatmaps [7] [18] [29].The authors suggest that visual verification may increase trust.Source: §1 · p. 2
  • A pre-trained language model generates descriptions for individually selected detected regions, and radiologists can also draw bounding boxes to request descriptions of regions of interest.The authors assess manually drawn boxes to demonstrate robustness of selection-based sentence generation.Source: §1 · p. 2
  • On MIMIC-CXR, RGRG selects salient anatomical regions and concatenates their descriptions into complete reports, outperforming competitive baselines on language-generation and clinically relevant metrics.The introduction reports no numerical results for these comparisons.Source: §1 · p. 2

2. Related work

The related work situates the method among Transformer-based and region-wise report generation, dense captioning, and controllable captioning, emphasizing detected-region features and independently generated region sentences as key distinctions.

2.1. Radiology report generation

  • Unlike ’s dedicated region heads over CNN patch features, this method detects region features and uses a shared Transformer decoder, with more supervision but less complexity and greater interactivity and explainability.Earlier systems used CNN-RNNs [17] [18] [25], followed by Transformers [50] and variants incorporating relation memory [7], memory matrices [6], abnormal-region mechanisms [28] [27], and pretrained components [33]; this work uses a pretrained language-model decoder.Source: §2.1 · p. 2

2.2. Image captioning

  • Radiology reports exceed typical captions in length and diversity and describe multiple regions, while normal-heavy data complicate descriptions of specific abnormalities; dense captioning localizes and describes salient regions.This region-by-region approach aligns with radiologists’ practice of localizing and describing anatomical regions individually.Source: §2.2 · pp. 2–3
  • Anatomy-based mode lets radiologists select automatically boxed regions for independent sentence generation, unlike controllable captioning that typically produces one image-level caption [5] [8].Controllable captioning can steer descriptions by specifying regions [8] or an abstract scene graph [5].Source: §2.2 · p. 3

3. Method

RGRG generates full chest X-ray reports, supports sentence generation for radiologist-selected detected anatomies, and generates sentences for regions marked with manually drawn boxes.

3.2. Modules

  • GPT-2 Medium generates sentences conditioned on region features through pseudo self-attention, while Faster R-CNN with a ResNet-50 backbone detects anatomical regions.The language model is fine-tuned on PubMed abstracts; the region features are injected into self-attention using newly initialized key and value projections.Source: §3.2 · pp. 3–4

3.1. Overview

  • The model detects 29 chest anatomical regions, selects salient regions for description, and uses abnormality classification to inform region selection and sentence generation.Its four modules comprise an object detector, region selection and abnormality classifiers, and a language model that generates sentences conditioned on selected-region visual features.Source: §3.1 · p. 3, Figure 2 · p. 4

3.3. Training

  • Training proceeds from the object detector, to the detector with binary classifiers, and then to end-to-end training; the language model’s region-injection projections are the only trainable language-model parameters.The weighted loss combines detector, region-selection, abnormality-classification, and language-model losses, with weights 1.0, 5.0, 5.0, and 2.0, respectively; language-model training uses regions with reference sentences.Source: §3.3 · p. 4

3.4. Inference

  • For full-report inference, RGRG concatenates sentences from selected regions and uses BERTScore to remove similar duplicates, retaining the longer sentence.The longer sentence is kept because it tends to contain more clinically relevant information.Source: §3.4 · p. 4
  • For interactive sentence generation, radiologists can select detected anatomical regions or draw bounding boxes around regions whose visual features are passed to the language model.Anatomy-based generation first detects and outlines all 29 regions, then uses the radiologist’s selected regions for sentence generation.Source: §3.4 · p. 5

4. Experimental setup

The experiments evaluate radiology report generation, anatomy-based sentence generation, and selection-based sentence generation using Chest ImaGenome, language-generation metrics, clinical-efficacy metrics, and task-specific analyses.

4.1. Dataset and pre-processing

  • Chest ImaGenome v1.0.0, derived from MIMIC-CXR, provides scene graphs for 29 anatomical regions and an official split of 166,512 training, 23,952 validation, and 47,389 test images.Images are resized to 512×512 with aspect ratio preserved, normalized, and augmented during training; report generation uses findings sections, with empty-findings reports excluded, leaving 32,711 test images with references.Source: §4.1 · p. 5

4.2. Evaluation metrics

  • Report-level evaluation uses BLEU, METEOR, ROUGE-L, and CIDEr-D, while sentence-level evaluation uses METEOR; clinical-efficacy metrics assess the presence of prominent observations.The clinical-efficacy metrics compare generated and reference reports on observation presence status to capture diagnostic accuracy.Source: §4.2 · p. 5

4.3. Evaluation strategy and baselines

  • Report generation is compared with prior models; anatomy-based generation has no direct baselines and is assessed qualitatively and with per-anatomy METEOR and an Anatomy-Sensitivity-Ratio.The ratio compares micro-averaged METEOR for anatomy-matched sentences against an anatomy-agnostic score; 1.0 indicates no apparent relation, and higher values indicate closer alignment.Source: §4.3 · p. 5
  • On full-report NLG metrics, the model is competitive with or outperforms prior state-of-the-art models; on CE metrics, it beats non-RL models by large margins and remains competitive with two CE-optimized RL models.The supplied captions do not include metric values.Source: Table 1 · p. 6, Table 2 · p. 6
  • Selection-based sentence generation is evaluated by varying manually drawn boxes in position, aspect ratio, or scale and measuring the effect on METEOR.The analysis tests whether small box deviations have negligible impact, larger deviations reduce sentence quality, and outputs remain sensitive to the selected region’s rough position.Source: §4.3 · p. 6

5. Results and discussion

The model is competitive with or outperforms prior systems on report-generation metrics, and its anatomy-guided sentences and selection-based generation show useful behavior under evaluation. Results also examine component effectiveness through ablation.

5.1. Radiology report generation

  • The model is competitive with or outperforms prior models on conventional NLG and clinical CE metrics, setting a METEOR state-of-the-art and large CE gains over non-RL-optimized models.It is nevertheless outperformed by, while remaining competitive with, the baseline explicitly optimized on CE metrics via reinforcement learning.Source: §5.1 · pp. 6–7
  • BLEU-4 improves by ∆+10.5% over baseline [32], while METEOR improves by ∆+9.8% over [33] and ∆+6.3% over.The authors note that [33] and used processed references that can raise BLEU scores; ROUGE-L is lower, which they attribute to low region-selection precision.Source: §5.1 · pp. 6–7
  • Micro-averaged CE F1 increases by +10.0% (∆+22.4%) over non-RL-optimized baseline [32], and example-based CE F1 increases by +6.3% (∆+16.4%) over its best baseline.The model remains competitive with, but is outperformed by, baseline [32], which was explicitly optimized on CE metrics via RL.Source: §5.1 · p. 7

5.2. Anatomy-based sentence generation

  • The Anatomy-Sensitivity-Ratio is 1.938, indicating anatomy-specific sentences score almost twice as high as anatomy-agnostic sentences.Predicted regions generally align closely with ground truth, and the model generates pertinent anatomy-related sentences; abnormal-region descriptions may score lower because references are more diverse, despite clinical accuracy.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Selection-based sentence generation

  • Sentence-generation performance declines slowly with aspect-ratio variation, somewhat more with scale variation, and rapidly with position variation, making close manual placement sufficient but position important.With sufficient position deviation, generated sentences reach the anatomy-agnostic score threshold.Source: §5.3 · pp. 7–8

5.4. Component analysis and ablation study

  • Ablations of region selection and abnormality classification verify their effectiveness and relevance to overall model performance.The study also reports module-specific quantitative results and further qualitative and fine-grained quantitative analyses in the appendices.Source: §5.4 · p. 8

6. Conclusion · A. Detailed analysis

The paper presents a region-guided report generation method that describes salient anatomical regions, visually grounds generated sentences, and supports radiologist interaction. Its clinical use is qualified by supervision and scope limitations, while visual grounding and interaction are intended to encourage verification.

6. Conclusion

  • The method detects salient anatomical regions and generates region-specific descriptions, visually grounding report sentences to support explainability.Radiologists can select anatomies or draw bounding boxes to guide sentence generation.Source: §6 · p. 8
  • Experiments verify the method’s effectiveness in generating clinically accurate reports and offering interactive capabilities.The authors present the approach as a direction for further research into region-guided radiology report generation.Source: §6 · p. 8
  • Bounding-box sentence generation is more sensitive to position but robust to aspect ratio and scale, supporting the viability of interactive box selection.Figure 5 compares METEOR scores as boxes deviate from ground truth and includes an anatomy-agnostic reference threshold.Source: Figure 5 · p. 8
  • Overreliance and automation bias pose risks, and detailed explanations may exacerbate them; the authors believe visual grounding and interaction can encourage radiologist verification and intervention.The discussion cites automation bias, overreliance, and the possibility that explanations exacerbate bias.Source: §6 · p. 8
  • The method requires strong supervision currently available only in Chest ImaGenome, limiting generalization to other report-generation tasks.Adapting the approach to limited supervision is identified as a future research direction.Source: §6 · p. 8
  • The method considers chest X-rays in isolation, although diagnosis usually uses prior examinations; localized comparison relations across sequential exams could improve clinical accuracy.Chest ImaGenome contains such comparison relations for anatomical regions.Source: §6 · p. 8
  • Region-guided generation does not cover reference-report sentences about objects that cannot be attributed to an anatomical region, such as surgical clips.The authors suggest a hybrid system combining region- and image-level visual features to generate both region-level and image-level sentences.Source: §6 · p. 8

A.1. Ablation study · A.2. Object detector results

The ablations show that region selection improves language-generation metrics and clinical-efficacy recall, while abnormality classification further increases recall. The detector identifies nearly all anatomical regions with adequate IoU, despite imperfect reference boxes.

A.1. Ablation study

  • Region selection trains the language model only on visual features paired with non-empty references, removing its implicit region-selection task; generated reports averaged 52 tokens versus 39 for the base model.The base model was trained with 2.2 times more empty than non-empty references, which may contribute to shorter reports and low CE recall.Source: §A.1 · p. 12

A.2. Object detector results

  • The detector is evaluated with IoU rather than mAP because each anatomical region typically appears exactly once per image, and results include micro-averaged IoU and detected-region counts.IoU is calculated as summed intersection areas divided by summed union areas.Source: §A.2 · p. 12

A.1. Ablation study

  • Region selection raised BLEU-4 by ∆+9.6% and METEOR by ∆+19.3%, and improved CE recall by +19.2% (∆+53.5%) versus the base model.CE precision decreased, possibly because low precision for normal-region selection leads to more false-positive abnormal descriptions.Source: §A.1 · p. 12, Table 4 · p. 13
  • Adding abnormality classification improved CE recall by +10.2% (∆+28.4%), with a slight CE precision cost and negligible BLEU-4 and METEOR effects.The authors argue that conventional NLG metrics are ill-suited to evaluating clinical accuracy [29] [38].Source: §A.1 · p. 12, Table 4 · p. 13

A.2. Object detector results

  • The detector averaged 0.887 IoU while detecting almost all 29 regions per image; imperfect ground-truth box overlap likely lowered IoU, but was considered acceptable for report generation.The goal is consistent anatomy-related sentences rather than perfect object detection.Source: §A.2 · p. 12, Table 5 · p. 13

A.1. Ablation study

  • Combining region selection and abnormality classification in RGRG again increased CE recall; module results indicate high recall alongside precision limitations.Region selection has lower precision for normal regions, while abnormality classification has decreased precision.Source: §A.1 · p. 12, Table 6 · p. 13, Table 4 · p. 13

A.3. Region selection and abnormality classification results · B. Detailed results · B.1. Qualitative anatomy-based sentence generation results

Region selection usually captures regions described in reference reports, while normal regions are often selected despite not being described. The abnormality classifier usually detects abnormal regions but sometimes misclassifies normal ones; Figure 6 illustrates anatomy-based sentence generation on five test images.

A.3. Region selection and abnormality classification results

  • Region-selection recall is high for normal and abnormal regions, but precision is low for normal regions, so extra normal descriptions can lower ROUGE-L precision.The evaluation counts regions with a corresponding reference sentence as positive for selection and ground-truth abnormal regions as positive for classification.Source: §A.3 · p. 13
  • Low precision for normal-region selection is expected because whether to mention pathology-free regions is a subjective choice left to the radiologist.Source: §A.3 · p. 13
  • Abnormal-region selection precision is 1.0 because abnormal regions are always included in reference reports, leaving no false positives in that subset.Source: §A.3 · p. 13
  • Recall scores for normal and abnormal region subsets cannot be directly compared.Source: §A.3 · p. 13
  • For the abnormality classifier, recall is high and precision is low: abnormal regions are usually detected, while normal regions are sometimes misclassified.The classifiers are evaluated using precision, recall, and F1 score.Source: §A.3 · p. 13

B.1. Qualitative anatomy-based sentence generation results

  • Figure 6 shows anatomy-based sentence generation for five test images, with the upper three abnormal and lower two normal.Predicted regions use dashed boxes, ground-truth regions use solid boxes, and sentence colors correspond to regions.Source: Figure 6 · p. 14

B.2. Qualitative full report generation results

Across the three examples, generated reports largely captured reference information, but the cases expose limitations involving normal-region selection and findings dependent on prior radiographs.

B.2. Qualitative full report generation results

  • In the healthy case, the generated report included all reference information and correctly mentioned both tubes, but gave a slightly wrong endotracheal-tube value and added four clinically accurate negative observations.The region selection module has low precision for normal regions because whether to mention them is arbitrary; describing more regions than the reference can lower ROUGE-L, although these additional observations were clinically accurate.Source: §B.2 · p. 14
  • In the middle case, the report correctly described right pleural effusion but reversed its change over time, duplicated accurate observations, and missed a possible small left effusion.Because the method considers each X-ray in isolation, it cannot reliably generate sentences depending on prior images; localized comparison relations across sequential exams may help, while interaction and transparency can help address missed findings.Source: §B.2 · p. 14
  • In the pathological right-hand case, the generated report captured most findings, whereas every reference sentence depended on prior radiographs, underscoring the need for sequential information.Source: §B.2 · p. 14

B.3. Qualitative selection-based sentence generation results … C.1. Module details

The section shows how bounding-box selection affects generated sentences, provides detailed clinical and anatomy-level metric breakdowns, and describes key detector and language-model implementation choices.

B.3. Qualitative selection-based sentence generation results

  • Boxes placed above the pathology can miss it, producing an upper-lobe opacity or clear-lungs sentence; radiologists must position boxes accurately.The authors consider both descriptions accurate for the selected locations.Source: §B.3 · p. 16
  • Aspect-ratio and scale variations still identify pneumonia, but limited surrounding context can lead to an incorrect upper-versus-lower-lobe location.For aspect ratio 2 and scale 2, the generated sentence incorrectly places pneumonia in the upper lobe.Source: §B.3 · p. 16
  • Selection-based generation lets radiologists draw boxes anywhere in the image, but accurate positioning and sufficient surrounding context support accurate sentences.Source: §B.3 · p. 16

B.4. Detailed clinical efficacy metrics results

  • Table 7 reports per-observation and micro-averaged clinical efficacy metrics; fracture scores 0.0 because the dataset has no fracture-describing sentences.The caption notes that combining image-level features with observation sentences from MIMIC-CXR may further improve clinical accuracy.Source: Table 7 · p. 17

B.5. Detailed anatomy-level results

  • Table 8 reports anatomy-based sentence generation using METEOR and object detection using IoU for each of 29 anatomical regions.Source: Table 8 · p. 17

C.1. Module details

  • Using Faster R-CNN RoI feature vectors directly as region visual features hurt detector performance, possibly because detector and report-generation modules couple their features.Source: §C.1 · p. 18
  • The language model uses the Hugging Face GPT-2 implementation (transformers 4.19.2) with the healx/gpt-2pubmed-medium checkpoint [35].Source: §C.1 · p. 18

C.2. Training … D.3. Clinical efficacy metrics

The supplementary sections specify the staged training, inference, dataset, and reference-report procedures, and explain how reference casing affects NLG scores. They also define the clinical efficacy metrics used to compare generated and reference reports.

C.2. Training

  • Positive-example weights for the region-selection and abnormality-classification binary cross-entropy losses are 2.2 and 6.0, respectively, to account for class imbalance.These are the weights for Lselect and Labnormal in the overall training loss.Source: §C.2 · p. 18
  • The model uses three training stages, with batch size, initial learning rate, and epoch count set to 16, 1e-3, and 6; 16, 5e-4, and 9; and 2, 5e-5, and 2, respectively.Training used AdamW [30], weight decay 1e-2, validation-loss-based learning-rate reduction and early stopping; batches were gradient-accumulated to 64, and training took about 45 hours on one NVIDIA A40.Source: §C.2 · p. 18

C.3. Inference

  • Sentence generation uses beam search of width 4 and removes generated sentences with BERTScore above a 0.9 threshold.The threshold was based on best validation performance, and BERTScore used uncased base DistilBERT [46].Source: §C.3 · p. 18

D.1. Dataset and pre-processing

  • Training uses Chest ImaGenome v1.0.0 scene graphs and MIMIC-CXR-JPG v2.0.0 [22] images, with image augmentations applied at 50% probability each.Augmentations include 20% brightness and contrast jitter, zero-mean Gaussian noise with variance [10, 50], and affine translation up to ±2% and rotation up to ±2°; redundant whitespace and superfluous introductory phrases are removed from dataset sentences.Source: §D.1 · p. 18

D.2. Reference reports and processing

  • The reference reports are MIMIC-CXR findings sections extracted with the dataset authors’ tool, with no further processing applied.The authors note that papers [33] most likely processed reports further, including lowercasing, but does not specify its processing or provide code for verification.Source: §D.2 · pp. 18–19
  • When references were lowercased, our method scored BLEU-1 0.400, BLEU-2 0.266, BLEU-3 0.187, and BLEU-4 0.135, a ∆+8.9% against the best baseline; METEOR and CE scores remained unchanged.Source: §D.2 · p. 19
  • The authors recommend emphasizing CE metrics because NLG scores depend on reference processing, while CE metrics compare disease presence independently of sentence structure or casing.They characterize CE metrics as allowing fairer comparisons while capturing diagnostic accuracy.Source: §D.2 · p. 19

D.3. Clinical efficacy metrics

  • Clinical efficacy scores classify 14 observations with CheXbert, then follow [33] for example-based scores over all observations and [32] for micro-averaged scores over five observations.The five observations are atelectasis, cardiomegaly, consolidation, edema, and pleural effusion; [33] treats only positive as positive, whereas [32] also treats uncertain as positive.Source: §D.3 · p. 19

D.4. Variation sampling for evaluation of selection-based sentence generation

The variation-sampling evaluation tested how changes to ground-truth anatomical bounding boxes affect selection-based sentence generation. It compared micro-averaged per-anatomy METEOR scores against inference with unmodified boxes.

D.4. Variation sampling for evaluation of selection-based sentence generation

  • On the first 1,000 test samples, trained RGRG inference compared position, aspect-ratio, and scale perturbations with the unmodified-box baseline using micro-averaged per-anatomy METEOR.Each variation type was tested at several degrees, with each experiment consisting of one inference pass over all 1,000 samples.Source: §D.4 · p. 19
  • Position changes were sampled independently for each anatomical region and sample from zero-mean normal distributions, then added to box coordinates in proportion to box width and height.The x and y offsets each use standard deviation σ.Source: §D.4 · p. 19
  • Aspect-ratio perturbations were multiplicative and lognormally distributed, with the 1-σ interval [e^-σ, e^σ]; width and height were adjusted while preserving box area.The modified aspect ratio is used to derive the new width and height from the original area.Source: §D.4 · pp. 19–20
  • Scale perturbations were sampled from a lognormal distribution and applied multiplicatively to both the box width and height.The resulting dimensions are used to compute the modified box.Source: §D.4 · p. 20

Related papers

Loading 2304.08295v1…