PaperlayerView exact revision on arXiv

Source-linked AI summary

MAIRA-2: Grounded Radiology Report Generation

Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Anton Schwaighofer, Anja Thieme, Sam Bond-Taylor, Maximilian Ilse, Fernando Pérez-García, Valentina Salvatelli, Harshita Sharma, Felix Meissen, Mercy Ranjit, Shaury Srivastav, Julia Gong, Noel C. F. Codella, Fabian Falck, Ozan Oktay, Matthew P. Lungren, Maria Teodora Wetscherek, Javier Alvarez-Valle, Stephanie L. Hyland

arXiv:2406.04449v2cs.CLcs.CV

TL;DR

Radiology report-generation models need verifiable, clinically useful outputs without hallucinations or omissions. This paper adds contextual inputs and spatial grounding, introduces LLM-based RadFact evaluation, and develops MAIRA-2 for grounded and non-grounded chest X-ray reports. MAIRA-2 achieves state-of-the-art performance on existing report-generation benchmarks and establishes grounded reporting as a new task.

  • Problem

    Automated radiology drafts must replicate or exceed radiologist reports without hallucinations or omissions, while existing evaluation limitations leave correctness and grounding insufficiently assessed.

  • Method

    The paper develops MAIRA-2 with realistic reporting context and spatial finding annotations, alongside RadFact, an LLM-based sentence-level evaluation framework.

  • Results

    MAIRA-2 achieves state-of-the-art performance on existing report-generation benchmarks and improves prior lexical scores by 17% to 30%.

  • Takeaways & Limitations

    Grounded reporting and RadFact extend chest X-ray report generation toward outputs that combine textual correctness with verifiable finding localization.

  • Takeaways & Limitations

    Existing evaluation approaches are limited by reliance on predefined finding classes, specialized models, or error types and by not supporting grounded-report evaluation.

Abstract

from arXiv · show

Radiology reporting is a complex task requiring detailed medical image understanding and precise language generation, for which generative multimodal models offer a promising solution. However, to impact clinical practice, models must achieve a high level of both verifiable performance and utility. We augment the utility of automated report generation by incorporating localisation of individual findings on the image - a task we call grounded report generation - and enhance performance by incorporating realistic reporting context as inputs. We design a novel evaluation framework (RadFact) leveraging the logical inference capabilities of large language models (LLMs) to quantify report correctness and completeness at the level of individual sentences, while supporting the new task of grounded reporting. We develop MAIRA-2, a large radiology-specific multimodal model designed to generate chest X-ray reports with and without grounding. MAIRA-2 achieves state of the art on existing report generation benchmarks and establishes the novel task of grounded report generation.

Introduction

The paper targets clinically useful radiology report generation by adding realistic reporting context and spatial grounding to improve verifiability and utility. It introduces MAIRA-2 for grounded and non-grounded chest X-ray reporting and RadFact for sentence-level evaluation.

  • Introduction: Automated drafts must replicate or exceed radiologist reports without hallucinations or omissions and remain easy to verify.These requirements motivate evaluation beyond conventional report-generation outputs.
  • Introduction: Grounded reporting links individual described findings to image locations through annotations such as bounding boxes.The authors propose requiring spatial grounding for each described finding while allowing non-localisable observations to remain unannotated.
  • Introduction: MAIRA-2 generates grounded and non-grounded chest X-ray reports while incorporating lateral views, prior studies, and report-context sections.Inputs include the current lateral image, prior frontal image and report, Indication, Technique, and Comparison when available.
  • Introduction: RadFact uses LLM logical inference to evaluate generated-sentence factuality against reference reports and to assess grounding annotations.This provides an interpretable sentence-level view of errors while supporting both correctness and completeness evaluation.
  • Introduction: The work releases MAIRA-2, an open-source RadFact implementation, and an annotation protocol for grounded-reporting datasets.These resources are intended to support further research on grounded radiology report generation.

Methods

The methods define grounded reports, assemble multimodal contextual inputs, and train MAIRA-2 to generate findings with optional spatial annotations. The model uses a radiology-specific vision encoder, language model, adapter, and coordinate-token representation.

  • Grounded report definition: A grounded report is a Findings section whose sentences describe individual observations and may carry spatial annotations locating those observations.Non-findings, normality statements, and findings without a specific location do not require annotations; this work uses bounding boxes.
  • Data: MAIRA-2 is trained on public and private chest X-ray datasets spanning findings generation, grounded reporting, and phrase grounding.The training data include 510,848 report-generation examples from 226,077 adult patients, including 72,226 grounded examples.
  • MAIRA-2 architecture: MAIRA-2 follows a LLaVA-like architecture with a frozen Rad-DINO-MAIRA-2 image encoder, a trainable adapter, and a language model trained on grounded and non-grounded examples.The language model is initialized from Vicuna 7B v1.5, while the adapter is a four-layer MLP.
  • Context inputs: The model uses current frontal and lateral images, optional prior images and reports, and Indication, Technique, and Comparison context.Missing optional inputs are represented as N/A, and available images and sections are interleaved with image tokens in the prompt.
  • Grounding representation: Coordinate tokens encode bounding-box top-left and bottom-right corners on a discretized image grid, allowing grounded sentences to be generated alongside boxes.Horizontal and vertical coordinates are encoded as disjoint token sets.

RadFact: An evaluation suite for (grounded) reports

RadFact evaluates report correctness, completeness, and spatial grounding through LLM-based logical entailment rather than fixed finding classes or specialized radiology models. Its metrics separate text-only and text-and-grounding precision and recall.

  • RadFact framework: RadFact uses LLM logical inference to evaluate generated reports against ground truth without relying on predefined error categories, finding classes, or radiology-specialized models.The framework provides fine-grained sentence-level metrics for correctness and completeness, with grounding evaluation when annotations are present.
  • Text metrics: Logical precision measures the fraction of generated sentences entailed by the reference, while logical recall measures the fraction of reference sentences entailed by the generation.These metrics respectively penalize hallucinations and omissions.
  • Grounding metrics: Grounding precision and recall measure whether logically entailed findings also have spatially entailed annotations.Spatial precision and recall additionally penalize incorrect grounding among all grounded sentences.
  • Grounding metrics: Spatial entailment requires at least 50% of sentence-associated pixels to fall within the union of matched evidence boxes.RadFact therefore evaluates both the textual hypothesis and its associated spatial evidence.
  • Evaluation protocol: The evaluation compares conventional lexical and radiology-specific metrics and reports median scores with 95% confidence intervals from 500 bootstrap replicates.The reported comparisons include BLEU-4, RadCliQ, RadGraph-F1, and macro-averaged CheXbert F1.

Results

MAIRA-2 matches or exceeds prior findings-generation approaches across reported metrics and improves lexical and clinical evaluation scores. On grounded reporting, it achieves strong logical and grounding performance, while spatial metrics reveal remaining localization errors.

  • Grounded report generation: On GR-Bench, RadFact logical scores are consistently above 70%, indicating relatively low rates of omissions and hallucinations.The reported grounded-report results cover both logical report quality and spatial grounding.
  • Grounded report generation: GR-Bench grounding precision is 69%, meaning 69% of logically correct generated sentences are also correctly grounded.Grounding recall is above 90%, indicating reliable coverage of ground-truth boxes for correctly predicted findings.
  • Grounded report generation: On PadChest-GR, RadFact grounding precision and recall are 80% and 77%, respectively.The authors describe these scores as more balanced than the corresponding GR-Bench grounding results.
  • Grounded report generation: Only 52.9% of MAIRA-2’s generated sentences were confirmed true on the MIMIC-CXR grounded-report test set.The spatial metrics also show that the model sometimes generates boxes associated with incorrect sentences.

A Grounded Reporting

The grounded-reporting evaluation distinguishes logical correctness from spatial grounding, using separate RadFact metric groups. The supplied figure fragments show distinct scales for these metric displays.

  • RadFact logical precision / recall are evaluated as one metric group.
  • RadFact spatial precision / recall are evaluated as a separate metric group.
  • The displayed metric scales include ranges marked from 0 to 80 and from 0 to 40.

B Findings Generation

MAIRA-2 generates findings reports that generalize across datasets and achieve strong RadFact performance, while qualitative review identifies mostly minor errors and highlights the value of reporting context and multi-view inputs.

  • Findings Generation: 57% logical precision and 49% logical recall on PadChest establish a benchmark where prior English findings-generation results were unavailable.
  • Findings Generation: 71% logical precision and 68% logical recall on IU-Xray show generalization to an unseen dataset.
  • Qualitative review: A thoracic radiologist judged outputs acceptable as drafts comparable to a junior-to-mid-level resident, but requiring expert review before sign-off.
  • Qualitative review: On twenty GR-Bench cases, 14/20 reports required fewer than two corrections and 123/135 sentences were acceptable as-is.
  • Qualitative review: Omissions were the most common error category, accounting for 15 of 25 corrections, including lower sensitivity to minor findings and occasional inconsistency.
  • Reporting context: Removing prior-study context significantly reduced metrics and produced comparison mentions near the dataset background rate, while training with that context reduced mentions when unavailable at inference.

Discussion

MAIRA-2 demonstrates grounded radiology report generation and improves chest X-ray reporting by combining radiology-specific multimodal modeling with richer clinical context. RadFact provides fine-grained evaluation, while remaining limitations include missed findings, dataset constraints, and gaps between benchmark performance and clinical readiness.

  • MAIRA-2 improves findings generation through a radiology-specialised image encoder, Vicuna 7B, and comprehensive chest X-ray reporting inputs.Inputs include current frontal and lateral views, prior imaging and report information, indication, technique, and comparison sections.
  • Ablations associate the additional inputs with fewer hallucinations and better clinical accuracy, although the model may exploit report-section shortcuts rather than fully use imaging information.The paper notes that alternative methods to token concatenation may incorporate additional imaging information more effectively.
  • Radiologist review found many generated sentences acceptable as-is and reports potentially suitable as first drafts, but missed findings were the most common error.The authors therefore identify improving recall as necessary.
  • RadFact uses flexible LLM-based logical inference to assess sentence-level report errors and support both grounded and non-grounded reporting.It targets factuality and completeness at a finer granularity than fixed finding-category evaluations.
  • The evaluation and grounding evidence remain limited because RadFact penalises partial errors, cannot detect within-report inconsistencies, and grounded datasets lack desirable combinations of inputs.Qualitative analyses were predominantly conducted with a single radiologist, limiting generalisability.
  • MAIRA-2 establishes a new state-of-the-art on MIMIC-CXR and demonstrates that grounded radiology reporting is feasible.The paper positions grounding as a step toward clinical impact in automated radiology report generation.

A.1 Why is grounded reporting a useful task?

Grounded reporting links each reported finding to its image location, extending radiology reports beyond text and supporting verification, comprehension, and clinical communication. The task differs from phrase grounding because it generates all image findings with localisations without requiring findings as input.

  • Clinical utility: Image grounding can help verify AI-generated text and support image understanding, diagnosis, clinician comprehension, and patient communication.These benefits are especially relevant when findings are small, overlapping, or outside the reporter’s core expertise.
  • Clinical utility: Grounded reporting may also improve engagement with images for non-radiology clinicians and communication with patients during image review.The passages describe grounding as supporting comprehension and deeper engagement beyond the text report.
  • Task definition: Grounded reports pair descriptions of all image findings with their localisations, rather than grounding only a specified phrase.The task does not require the phrases or findings to be provided in advance.
  • Task design: Each sentence is designed to describe at most one finding, enabling more precise localisation and avoiding the many-to-many mapping between sentences and regions.This dataset design addresses the challenge that a single sentence can otherwise describe multiple findings and regions.
  • Reporting context: Realistic reporting context includes indication, prior studies, lateral views, and report sections, with dataset-specific handling of missing studies and views.The supplied passages describe context inputs and dataset construction across MIMIC-CXR, PadChest, USMix, and IU-Xray.

B.2 Additional MAIRA-2 model and training details

MAIRA-2 is trained as a multitask autoregressive model using radiology images and reporting context, while representing bounding-box coordinates with separate horizontal and vertical token sets. Its grounded-reporting data are constructed by extracting single-finding sentences and adding expert spatial annotations.

  • Training: MAIRA-2 is trained with conventional autoregressive cross-entropy loss in a multitask setting, using a frozen image encoder and trainable adapter and language-model parameters.Training uses a single stage and the final checkpoint is evaluated.
  • Inputs and prompting: The model prompt can include images, report sections, lateral views, prior studies, and task-specific instructions, while omitting unavailable views or studies.GroundRep, FindGen, and phrase-grounding tasks use different instructions and input configurations.
  • Spatial representation: Bounding boxes are encoded with separate horizontal and vertical coordinate tokens, using N = 100 grid positions instead of a shared flat coordinate vocabulary.The design is intended to provide an inductive bias for learning two-dimensional representations.
  • Datasets: The training and evaluation setup uses multiple radiology datasets with patient-level separation and task-specific grounded-reporting subsets.The supplied passages identify PadChest-GR, GR-Bench, and held-out USMix grounded-reporting data.
  • Data preparation: Grounded-reporting data are derived by extracting single-finding sentences and acquiring spatial annotations for each finding.The extraction targets visible or explicitly absent chest-X-ray findings, while experts annotate bounding boxes for positive findings.

B.5.1 Extraction of sentences from reports

The paper converts narrative Findings reports into single-finding sentence lists so RadFact can evaluate report correctness, completeness, and grounding. LLM processing removes speculation and splits multi-finding sentences, with limited loss for most findings but substantial loss for pneumonia mentions.

  • Conversion effects: The conversion changes word and pathology distributions, so the authors compare processed and original reports using standard report-generation metrics.This comparison is performed on the MIMIC validation set.
  • Conversion effects: For most findings, conversion preserves high recall, but pneumonia recall is approximately 3.4%, removing over 96% of original pneumonia mentions.The paper attributes this exception to pneumonia often being expressed as a speculative interpretation of other findings.
  • Sentence extraction: LLM preprocessing splits sentences containing multiple findings and removes speculative clinical interpretations before concatenating the derived sentences.The prompt uses GPT-4 to produce single-finding sentences for RadFact processing.
  • Sentence extraction: Example conversions split combined negatives into separate sentences and remove the speculative pneumonia interpretation while retaining the underlying opacity finding.The examples separate lobar consolidation, effusion, pneumothorax, silhouette, and bone findings into individual sentences.
  • RadFact evaluation: RadFact evaluates sentence lists in both directions, measuring logical precision for generated sentences and logical recall for ground-truth sentences.Grounding metrics additionally assess whether logically entailed findings are spatially entailed by their evidence regions.
  • RadFact evaluation: Spatial entailment requires a region’s pixel mask to be sufficiently contained within its evidence mask, using a 0.5 pixel-precision threshold.The framework supports multiple boxes and pixel-mask annotations.

C.2 Implementation details

RadFact uses LLM-based entailment verification to evaluate report correctness and completeness, with evidence-producing outputs that support grounding assessment. The implementation compares GPT-4 and Llama3-70B and supplements RadFact with text-only and box-level evaluations.

  • RadFact implementation: The entailment verifier outputs evidence before classification, and this prompting strategy improved performance.Outputs are constrained to parseable YAML using Pydantic.
  • RadFact implementation: RadFact processes each report sentence in both directions, requiring approximately six to seven entailment queries per report.The authors describe the resulting performance drop as negligible relative to the throughput gain.
  • RadFact implementation: RadFact-Llama3’s logical F1-score correlates 0.59 [0.51, 0.66] with ReXVal error counts and 0.62 [0.55, 0.68] for clinically significant errors.The paper reports lower correlation than CheXprompt but broader sentence-level entailment coverage.
  • Evaluation metrics: Text-only evaluation combines lexical metrics with radiology-specific clinical metrics, including ROUGE-L, BLEU, METEOR, RadGraph-F1, RadCliQ, and CheXbert measures.Additional evaluations report CheXpert macro- and micro-averaged F1 scores.
  • Evaluation metrics: Grounding-only evaluation completes boxes for grounded phrases and computes pixel-mask overlap, complementing RadFact’s binary sentence-level grounding assessment.The two evaluations respectively measure box quality in isolation and text-grounding entailment.

D.2 Findings generation – additional results

Additional experiments evaluate MAIRA-2 on held-out findings generation, grounded reporting, and phrase grounding tasks. The model outperforms or competes with relevant baselines, while IU-Xray results require caution because that dataset may be easier than MIMIC-CXR.

  • Findings generation: MAIRA-2 produces higher ROUGE-L and statistically equivalent CheXbert Micro F1-14 scores on held-out IU-Xray findings generation.IU-Xray is used for external validation and comparison with LLaVA-Rad.
  • Findings generation: High RadFact scores on IU-Xray suggest higher-quality reports than on MIMIC-CXR, although the difference may reflect IU-Xray being an easier dataset.The authors state that additional inputs do not appear to induce significant spurious dataset-level associations.
  • Grounded reporting: Grounded reporting is evaluated on GR-Bench and PadChest-GR, where RadFact, CheXpert F1, and box or grounding metrics are reported with confidence intervals.No previously published grounded-reporting results are available for direct comparison.
  • Phrase grounding: On phrase grounding, MAIRA-2 achieves competitive performance against MedRPG and TransVG and appears to strongly outperform the multitask ChEX model.Comparisons use mean intersection over union on MS-CXR subsets.

D.5 Synergy between findings generation and grounded reporting training

Ablations show that findings generation training transfers positively to grounded reporting, while realistic prior-study, comparison, lateral-view, and technique inputs materially affect report quality and factuality. Removing these inputs can increase hallucinations or reduce diagnostic and lexical metrics.

  • FindGen–GroundRep synergy: FindGen training improves GroundRep text metrics, quality, and clinical factuality, whereas GroundRep training has limited impact on Findings generation.Most reverse-transfer metrics have overlapping confidence intervals.
  • Prior and comparison inputs: Removing prior studies during training or inference lowers Macro F1-14 by 8.5% and 10.3%, respectively, with inference removal producing the larger drop.The authors interpret this as evidence that prior studies support more factually correct reports.
  • Multi-view inputs: Omitting the technique section at inference reduces ROUGE-L by 15.59% and drives lateral mentions above the 35.57% ground-truth rate.The results identify technique information as a strong indicator for generating lateral-view mentions.
  • Multi-view inputs: Removing both lateral-view and technique inputs lowers lateral mentions to 5.1% and reduces Macro F1-14 by 5.15%.The authors conclude that both inputs help reduce lateral-mention hallucinations and support accurate diagnosis.

E.3 Findings generation examples from MIMIC-CXR

Qualitative MIMIC-CXR examples compare MAIRA-2 with reference reports and, in selected cases, Med-Gemini using RadFact and grounding assessments. The examples show strong overlap in some cases but also missed subtle findings, localization errors, and limited evidence for clinically important differences.

  • Scope: Quantitative comparison with models generating other report sections is not possible because these examples concern findings generation rather than Impression generation.The passage explicitly limits comparison to models trained for other sections such as Impression.
  • Comparison with Med-Gemini: The four comparisons with Med-Gemini found little difference between candidate phrases and reference text, but the examples were mostly normal and clinically limited.Two radiologists noted only minor differences in missed or hallucinated findings, conciseness, and ordering.
  • Qualitative findings: The qualitative examples illustrate that MAIRA-2 can reproduce report content while still missing subtle observations or adding borderline findings not present in the reference.Examples include aortic calcification, situs inversus, pleural findings, and other borderline or difficult-to-see observations.
  • RadFact examples: RadFact examples ranged from high precision and recall to zero precision and recall when MAIRA-2 missed subtle findings such as scarring, atelectasis, or hyperinflation.The examples include moderate scores of 0.78 precision and 0.64 recall, alongside a low-scoring normal-appearing report that omitted mild abnormalities.
  • Grounded reporting: Grounding precision was high when generated findings and boxes matched reference findings, but fell when findings were incomplete, overspecific, or localized to the wrong side.The examples report grounding precision of 1.0, 0.5, and 0.0 under these differing conditions.

F.2 Findings

Radiologist review of 20 reports identified 25 corrections, chiefly omissions, while also revealing contradictions, overspecificity, incorrect progression, and location errors. Most edits had minor or no treatment implications, but two missed findings were judged potentially significant, and the outputs were considered acceptable drafts requiring senior review.

  • Correction burden: Thirty percent of reports required no edits, while edited reports received a median of one correction and an average of 1.78 corrections.The 20 reports contained 135 generated phrases, with a median of six phrases per report.
  • Error types: Omissions were the most prominent error type, accounting for 15 of 25 corrections through newly added or augmented finding phrases.Twelve omissions required entirely new phrases and three augmented existing phrases.
  • Error types: Additional errors included contradictions within reports, overspecific descriptions, unsupported progression claims, and incorrect anatomical locations.Examples include calling bones intact alongside degenerative changes, asserting cholecystectomy from surgical clips, and assigning a pacemaker two locations.
  • Clinical implications: Two missed findings had potentially significant clinical implications: a lingular infiltrate requiring clinical action and acute rib fractures explaining chest pain.Most other missed, misclassified, or insufficiently described cases were rated minor or had no treatment implications.
  • Clinical utility: Radiologists judged the outputs acceptable as drafts comparable to junior-to-mid-level resident performance, but requiring senior radiologist or consultant review before sign-off.The reviewed cases were relatively uncomplicated, averaging 1.8 pathologies per report and lacking the overlap or interaction common in ICU or postoperative settings.
  • Evaluation implications: The review supports evaluating completeness, disambiguation, language clarity, internal consistency, minor-finding sensitivity, and device knowledge alongside aggregate metrics.The authors describe fine-grained phrase-level assessment as a source of concrete corrections and model-improvement requirements.
Loading 2406.04449v2…