PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive and Explainable Region-guided Radiology Report Generation

Tim Tanida, Philip Müller, Georgios Kaissis, Daniel Rueckert

arXiv:2304.08295v1cs.CVcs.CLcs.LG

TL;DR

Radiology report generation is difficult because existing image-level methods may omit or contradict findings while offering limited explainability and interactivity. RGRG detects anatomical regions, generates region-specific descriptions, and composes them into reports; experiments on MIMIC-CXR show stronger performance than competitive baselines alongside visual grounding and interactive control. The paper also discusses automation-bias risks despite these explanations and interventions.

  • Problem

    Existing radiology report-generation methods rely on image-level features, can be factually incomplete or inconsistent, and provide limited explainability and interactivity.

  • Method

    RGRG detects salient anatomical regions, conditions a language model on each region, generates individual descriptions, and concatenates the selected sentences into a report.

  • Results

    On MIMIC-CXR, RGRG outperforms competitive baselines in language-generation and clinically relevant metrics while supporting visually grounded and interactive report generation.

  • Takeaways & Limitations

    Visual grounding and interactive region selection allow radiologists to verify, target, and intervene in sentence generation within the clinical workflow.

  • Takeaways & Limitations

    Detailed explanations may exacerbate automation bias, so visual grounding and interactivity do not eliminate the risk of overreliance.

Abstract

from arXiv · show

The automatic generation of radiology reports has the potential to assist radiologists in the time-consuming task of report writing. Existing methods generate the full report from image-level features, failing to explicitly focus on anatomical regions in the image. We propose a simple yet effective region-guided report generation model that detects anatomical regions and then describes individual, salient regions to form the final report. While previous methods generate reports without the possibility of human intervention and with limited explainability, our method opens up novel clinical use cases through additional interactive capabilities and introduces a high degree of transparency and explainability. Comprehensive experiments demonstrate our method's effectiveness in report generation, outperforming previous state-of-the-art models, and highlight its interactive capabilities. The code and checkpoints are available at https://github.com/ttanida/rgrg .

1. Introduction

RGRG addresses the difficulty and opacity of radiology report generation by detecting salient anatomical regions and generating region-specific sentences. The region-grounded design supports explainability and interactive radiologist control while improving reported performance on MIMIC-CXR.

  • Motivation: Chest X-rays are common and clinically important, making their time-consuming interpretation a substantial workload for radiologists.The workload is further exacerbated by shortages of trained radiologists in many healthcare systems.
  • Motivation: Existing image-level report-generation methods can produce factually incomplete or inconsistent reports and lack transparent, interactive control.Their limited focus on salient anatomical regions also restricts explainability and adaptability to radiologists’ preferences.
  • Method: RGRG uses object detection to extract localized anatomical-region features and generates individual anatomy-specific sentences for the final report.The approach divides whole-report generation into simpler region-level sentence-generation tasks.
  • Explainability and interaction: The model visually grounds each generated sentence in a predicted anatomical bounding box, enabling radiologists to verify sentence correctness.This provides a more direct grounding mechanism than attention heatmaps described for existing models.
  • Explainability and interaction: Radiologists can select detected anatomical structures or draw bounding boxes to generate targeted descriptions interactively.The manually drawn-box evaluation demonstrates robustness of the selection-based sentence-generation task.
  • Results: On MIMIC-CXR, concatenated salient-region descriptions produce factually complete and consistent reports while outperforming competitive baselines on language-generation and clinically relevant metrics.The full-report module selects salient anatomical regions before composing their descriptions.

2. Related work

Prior radiology report-generation work largely adapts image-captioning and Transformer architectures to whole-image reporting. Related dense-captioning and controllable-captioning ideas instead localize salient regions and support region-specific descriptions, aligning more closely with radiologists’ workflow.

  • Transformer-based radiology report generation: Recent radiology report-generation studies increasingly use Transformer architectures, sometimes adding relation memory, memory matrices, or disease-region alignment.These adaptations target interactions between visual and textual features or attention to abnormal regions.
  • Image captioning: Radiology reports differ from typical captions because they are longer, more diverse, and describe multiple observations across anatomical regions.This motivates methods beyond generating a single caption for the whole image.
  • Image captioning: Dense image captioning localizes and describes individual salient regions by conditioning a language model on region-specific features.This formulation more closely matches radiologists’ practice of localizing and describing anatomical regions one by one.
  • Controllable image captioning: Controllable image captioning motivates RGRG’s anatomy-based mode, where radiologists manually select regions for description generation.The related-work passage introduces this interaction as part of the proposed anatomy-based sentence-generation setting.

3. Method

RGRG detects chest anatomies, selects salient regions, and generates region-specific sentences that are concatenated into a report. Its design also supports anatomy selection and manually boxed regions for interactive sentence generation.

  • Interactive use cases: The model supports full-report generation, radiologist selection of detected anatomies, and sentence generation from manually drawn boxes.
  • Overview: RGRG detects 29 anatomical chest regions and extracts localized visual features for each.
  • Modules: Binary classifiers select regions requiring descriptions and encode whether each region is normal or abnormal.The abnormality signal helps select pathological regions and generate descriptions that capture potential pathologies.
  • Modules: A medical-abstract-pretrained transformer language model generates sentences independently for each selected region.The decoder conditions on the associated region visual features, with pseudo self-attention injecting those features into the language model.
  • Report formation: The final report concatenates selected-region sentences after removing overly similar duplicates with BERTScore.
  • Training: Training combines object-detection, two weighted binary-classification, and language-model cross-entropy losses.The loss weights are λobj = 1.0, λselect = 5.0, λabnormal = 5.0, and λlanguage = 2.0.

4. Experimental setup

The evaluation uses Chest ImaGenome, derived from MIMIC-CXR, and assesses report generation, anatomy-based sentence generation, and selection-based sentence generation. It combines language-overlap metrics with clinical-efficacy measures and robustness testing under bounding-box perturbations.

  • Tasks: RGRG is evaluated on full report generation, anatomy-based sentence generation, and selection-based sentence generation.
  • Datasets: Chest ImaGenome provides scene graphs for frontal chest X-rays with bounding boxes for 29 anatomical regions and associated sentences.
  • Metrics: Full reports are assessed with BLEU, METEOR, ROUGE-L, and CIDEr-D, while sentence generation uses METEOR.
  • Metrics: Clinical efficacy metrics supplement conventional NLG metrics because word-overlap measures are ill-suited to clinical correctness.
  • Anatomy-based evaluation: Anatomy-based generation reports per-anatomy and subset METEOR scores plus an Anatomy-Sensitivity-Ratio.
  • Selection-based evaluation: Selection-based generation varies bounding-box position, aspect ratio, and scale relative to ground-truth boxes to simulate manual annotation.

5. Results and discussion

RGRG performs strongly on report-level and clinical-efficacy evaluation, generates anatomy-related sentences, and supports robust selection-based generation under some box deviations. Its main reported weakness is lower ROUGE-L associated with imprecise region selection.

  • Radiology report generation: RGRG is competitive with or outperforms prior models on NLG metrics and sets a new state of the art on METEOR.
  • Anatomy-based sentence generation: The generated examples include region-specific descriptions covering pneumonia, cardiomegaly, tube position, and pleural or pulmonary findings.
  • Limitation: ROUGE-L is lower than the baselines, which the authors attribute to low precision in region selection and over-selection of regions.
  • Radiology report generation: +10.0% micro-averaged CE F1 and +6.3% example-based CE F1 improve over the best non-RL baselines.These correspond to relative increases of ∆+22.4% and ∆+16.4%, respectively.
  • Anatomy-based sentence generation: Predicted bounding boxes generally align closely with ground truth, and the model produces pertinent anatomy-related sentences.
  • Anatomy-based sentence generation: An Anatomy-Sensitivity-Ratio of 1.938 indicates nearly twice the METEOR score for anatomy-conditioned versus anatomy-agnostic sentences.Abnormal-region descriptions often reference earlier examinations and have more diverse references, which may lower scores.
  • Selection-based sentence generation: Selection-based generation is robust to aspect-ratio and scale deviations but sensitive to position deviations.Performance declines slowly with aspect-ratio changes, somewhat more with scale changes, and quickly reaches the anatomy-agnostic threshold under position variation.

6. Conclusion

The method generates radiology reports by detecting salient anatomical regions and producing region-specific descriptions, with visual grounding and interactive controls for radiologist involvement. Experiments support its report-generation effectiveness and interactive capabilities, while its dependence on strong supervision limits generalization.

  • RGRG detects salient anatomical regions and generates region-specific descriptions to compose radiology reports.
  • Generated sentences are visually grounded on anatomical regions, enabling radiologists to inspect the basis of each description.
  • Radiologists can select anatomies or draw bounding boxes to control sentence generation interactively.
  • Experiments verify clinically accurate report generation and the method’s interactive capabilities.
  • Strong supervision from the Chest ImaGenome dataset is currently required, making generalization to other report-generation tasks difficult.

A.1. Ablation study

The ablation study shows that abnormality classification primarily improves clinical recall, while region selection improves language and clinical metrics by training sentence generation on selected non-empty regions. Anatomical detection is adequate for the report-generation objective, despite imperfect dataset boxes.

  • Ablation study: Adding abnormality classification improves CE recall by +10.2% (∆+28.4%) while negligibly affecting BLEU-4 and METEOR and slightly reducing CE precision.
  • Ablation study: Region selection boosts the base model across BLEU-4, METEOR, and clinical efficacy metrics.
  • Ablation study: Training on non-empty reference sentences increases BLEU-4 by ∆+9.6%, METEOR by ∆+19.3%, and CE recall by +19.2% (∆+53.5%).
  • Ablation study: The combined RGRG model further increases CE recall, supporting contributions from both abnormality classification and region selection.
  • Object detector results: The object detector identifies almost all 29 anatomical regions per image with an average IoU score of 0.887.

A.3. Region selection and abnormality classification results

Region selection usually retrieves regions described in reference reports, while abnormality classification usually detects abnormal regions. Both modules have reduced precision for normal regions, reflecting the optional nature of describing pathology-free anatomy.

  • Region selection has high recall for both normal and abnormal regions, but low precision for normal regions.
  • Low normal-region precision causes extra normal regions to appear in generated reports and can lower ROUGE-L precision.
  • Describing pathology-free regions is optional in reports, so the expected normal-region precision is limited by a subjective reporting decision.
  • Abnormal-region precision is 1.0 because abnormal regions are always included in reference reports, preventing false positives within that subset.
  • The abnormality classifier has high recall and low precision, usually detecting abnormal regions while sometimes misclassifying normal regions.

B.1. Qualitative anatomy-based sentence generation results

Anatomy-based generation produces region-specific sentences that generally capture reference findings across normal and abnormal chest X-rays, while exposing localized evidence for inspection. Qualitative cases also reveal errors involving precise measurements, temporal comparisons, and missed findings.

  • Qualitative findings: Generated sentences can contain clinically acceptable wording differences but may give slightly inaccurate numerical values or severity and location descriptions.Examples include endotracheal-tube distances, borderline versus moderate cardiomegaly, mild versus moderate edema, and tortuous versus ectatic aorta.
  • Abnormal cases: The model correctly describes several localized findings, including a borderline enlarged heart, pulmonary edema, absent pleural disease, a tortuous aorta, and upper-lung opacity.These examples demonstrate coverage of both normal and abnormal anatomical regions, although wording and severity sometimes differ from the references.
  • Qualitative findings: Generated region-specific sentences generally match the clinical content of reference sentences across normal and abnormal test images.Examples include matching descriptions of a normal right lung and cardiac silhouette, as well as clinically correct tube-placement and pathology descriptions.
  • Normal cases: The generated report may include more normal-region observations than the reference because selecting normal regions is difficult to learn and reference inclusion is arbitrary.In one healthy-image example, four additional negative observations were clinically accurate despite not appearing in the reference report.
  • Limitations: The method misses some findings and cannot reliably generate sentences requiring comparisons with previous radiographs because each X-ray is processed in isolation.A potential small left pleural effusion was missed, and an effusion trend was described in the wrong direction.

B.3. Qualitative selection-based sentence generation results

Selection-based sentence generation lets radiologists draw regions of interest and obtain descriptions, but output quality depends strongly on box placement and available context. The method is more robust to some scale and aspect-ratio changes than to positional errors.

  • Interactive capability: Radiologists can manually draw bounding boxes around image areas of interest to generate corresponding descriptions.The experiment evaluates how manually drawn boxes affect selection-based sentence generation.
  • Position variation: Bounding-box position is crucial: boxes placed above the pneumonia pathology can miss it and produce accurate descriptions of unaffected upper-lung regions.The authors caution that radiologists must position boxes accurately for correct pathology detection.
  • Aspect-ratio variation: Both tested aspect ratios identify pneumonia, but one generates an incorrect upper-lobe location because the box lacks sufficient surrounding information.The pathology is detected, yet its relative upper- versus lower-lobe position is misidentified.
  • Scale variation: Both tested scale variations identify pneumonia, but the smaller-context condition also mislabels it as upper-lobe pneumonia.The authors attribute this location error to insufficient surrounding information in the small-scaled box.
  • Practical caveat: Selection-based generation adds clinical workflow flexibility, provided manually drawn boxes are correctly positioned and include enough surrounding information.The paper presents correct positioning and contextual coverage as the primary practical caveat.

C.2. Training

Training uses staged optimization of the detector, region modules, and sentence generator on Chest ImaGenome-derived data, with class-imbalance weighting and standard augmentation and stopping procedures. Evaluation choices also show that report preprocessing materially affects NLG scores, whereas CE metrics are less sensitive to casing and structure.

  • Training procedure: Positive-example weights are set to 2.2 for Lselect and 6.0 for Labnormal to address region-selection and abnormality-classification class imbalance.The weights compensate for imbalances between regions with or without sentences and abnormal versus normal regions.
  • Training procedure: The model is trained in three stages, including an object detector with region selection and abnormality classification modules.The supplied training description explicitly identifies the staged procedure and the combined detector-region-module stage.
  • Data and optimization: Training uses Chest ImaGenome v1.0.0, derived from MIMIC-CXR, with data augmentation, AdamW optimization, validation-based learning-rate reduction, and early stopping.Augmentations include color jitter, Gaussian noise, and small affine transformations, each applied with 50% probability.
  • Sentence generation: The sentence generator uses beam search with width 4 and a BERTScore threshold of 0.9 to remove highly similar generated sentences.The threshold is intended to remove duplicates while minimizing elimination of relevant information.
  • Evaluation considerations: Lowercasing reference reports changes BLEU scores, including BLEU-1: 0.400 and BLEU-4: 0.135, while METEOR and CE scores remain unchanged.The authors use this result to argue that NLG metrics depend strongly on reference-report processing, whereas CE metrics are processing-invariant.

D.4. Variation sampling for evaluation of selection-based sentence generation

Variation sampling evaluates selection-based generation by independently perturbing anatomical-region boxes in position, aspect ratio, and scale. Position changes are additive relative to box size, whereas aspect-ratio and scale changes are sampled multiplicatively in log-space.

  • Experimental setup: The experiment independently samples box variations for each anatomical region in the first 1000 test-set samples.The subset reduces computational requirements while applying the trained RGRG model for selection-based inference.
  • Position variation: Position offsets ∆x and ∆y are sampled independently from zero-mean normal distributions and applied relative to the original box width and height.The modified coordinates shift both horizontal edges by ∆x·w and both vertical edges by ∆y·h.
  • Aspect-ratio and scale variation: Aspect-ratio and scale variations are sampled from Lognormal(0, σ^2), reflecting multiplicative perturbations in log-space.The 1-σ interval for both variations is [e^-σ, e^σ].
  • Aspect-ratio variation: For aspect-ratio variation, the modified width and height preserve the original area after applying the sampled aspect ratio.The updated dimensions are computed from the modified aspect ratio and unchanged area A.
  • Scale variation: For scale variation, width and height are multiplied by the sampled scale factor before the modified box is computed.The procedure then reuses the coordinate-construction step applied after aspect-ratio changes.
Loading 2304.08295v1…