Source-linked AI summary
MIRROR: Multimodal Intelligent Radiology Reasoning and Observation Reporter
Vignesh Nagarajan, Sriram Venkatapathy
TL;DR
MIRROR addresses the gap between opaque radiology predictions and fluent reports that may add unsupported findings by constraining report generation to structured classifier evidence. The system makes report findings auditable, but its ChestMNIST classifier still emits no positive prediction for 11 of 14 labels despite macro AUROC 0.729 and 1.6–6.8× ranking lift over a random ranker.
Problem
Radiology models often provide probabilities without location or rationale, while report generators can produce fluent prose that is inconsistent with the underlying prediction.
Method
MIRROR chains classification, Grad-CAM localization, and evidence-only report generation through a shared modality registry.
Results
Macro AUROC 0.729 and 1.6–6.8× random-ranker precision coexist with no positive prediction for 11 of 14 labels on ChestMNIST.
Takeaways & Limitations
Constraining the language layer makes report findings auditable against classifier evidence but does not ground the surrounding prose.
Takeaways & Limitations
The only quantitative benchmark is ChestMNIST, where the default-threshold classifier is not a working detector; other modalities lack trained checkpoints and explanation quality is unmeasured.
Abstract
from arXiv · showhide
A radiologist reading a model's output faces two problems. The model returns a number and no reason, and any system that turns that number into readable prose can quietly add claims the model never made. MIRROR is a research prototype built to separate those failures. It chains a multi-label classifier, a Grad-CAM localizer that turns each positive finding into a named anatomical region, and a report writer that receives the labels, probabilities, and regions but never the image. Because the language layer cannot see pixels, it cannot assert a finding the classifier did not make. We are precise about what that buys: a MIRROR report's findings are auditable against the probability vector, while the sentences framing them are ordinary generated text, and we show one stating a cardiothoracic ratio the system never measured. One registry holds the taxonomy, anatomy, and phrasing for chest X-ray, brain MRI, and head CT, so adding a modality is a data change; all three are routed and tested, one is trained. On ChestMNIST that classifier reaches macro AUROC 0.729 and ranks better than chance on all 14 labels, at 1.6 to 6.8 times the precision a random ranker would get. Yet at the default 0.5 threshold it emits no positive prediction at all for 11 of them, and its excellent-looking Brier score of 0.045 sits beside the 0.047 earned by a predictor that ignores the image. The discrimination is real; the decisions are not. Under the class imbalance normal in radiology, aggregate metrics flatter models that do nothing, and should be reported against that floor.
1 Introduction
MIRROR combines classification, localization, and report generation while preventing the language layer from accessing the image, making finding-level claims auditable against structured evidence. Its introduction also frames multimodality as registry data and emphasizes that strong discrimination can coexist with absent thresholded decisions and misleading aggregate metrics.
- Motivation and architecture: MIRROR routes image evidence through prediction and localization before report generation, while the language layer receives structured upstream evidence rather than the raw image.This constraint targets fluent reports that assert findings the classifier never produced.
- Grounding and limitations: Finding-level grounding is checkable under MIRROR’s constraint, but detail-level grounding is not, as demonstrated by a generated report.The paper explicitly distinguishes what structured evidence guarantees from unsupported reporting detail.
- Multimodality: Three modality taxonomies are registered, routed, and tested through a registry whose taxonomy, anatomical vocabulary, and report phrasing are data rather than code; one is benchmarked.The registered modalities are chest X-ray, brain MRI, and head CT.
- Implementation: The open-source implementation includes three backbones, two explainability methods, an LLM backend with a deterministic offline fallback, DICOM ingest, and two inference engines under one response contract.These components are presented as implementation contributions rather than evaluation findings.
- Empirical result: The chest-radiograph result shows real discrimination on every label but absent decisions on almost all, while Brier and AUPRC require comparison with prevalence-implied no-skill floors.The introduction argues that aggregate metrics become interpretable only against those baselines.
2 Literature Review
Prior work established large-scale chest-radiograph classification, visual attribution, interpretability evaluation, and radiology report generation, but also exposed limitations in post-hoc explanations and hallucination-prone prose. MIRROR is positioned at the intersection of these strands.
- Datasets and classification: 112,120 frontal radiographs across 14 pathologies made NIH ChestX-ray8/14 a standard benchmark for supervised chest-radiograph learning.CheXpert and MIMIC-CXR extended scale, uncertainty labeling, and paired free-text reports; CheXNet reported radiologist-level pneumonia detection with a 121-layer DenseNet.
- Visual explanation: Grad-CAM generalized class activation mapping to arbitrary architectures, while Score-CAM removed gradient dependence through perturbation-based weighting.LIME and SHAP provide model-agnostic attribution, and pointing-game and intersection-over-union protocols evaluate localization against annotated lesions.
- Critiques of post-hoc explanation: Saliency methods can be insensitive to the model and data they purport to explain, motivating sanity checks and broader measurement of interpretability.Rudin argues that high-stakes decisions should favor inherently interpretable models over post-hoc explanations of black boxes.
- Report generation: Radiology report generators span joint image-text models, memory-driven transformers, and decoders optimized for factual correctness, yet hallucination remains a recurring failure mode.Generated prose can assert findings the model never detected, while explainable AI and report generation are usually studied separately.
3 System Architecture
MIRROR is a three-layer pipeline that passes classifier outputs and named anatomical regions into a report writer that never receives the image. A shared modality registry supports tested routing across chest X-ray, brain MRI, and head CT, while only the local chest-X-ray stack implements the benchmarked grounded architecture.
- Pipeline: MIRROR chains three layers, with each layer’s output serving as the grounded input to the next.The classifier produces evidence, localization names the activated anatomy, and reporting consumes that structured evidence.
- Pipeline: The classifier uses ImageNet-pretrained DenseNet-121, EfficientNet-B0, or ViT-B/16 with a multi-label head sized to the active modality taxonomy.The taxonomy sizes are 14 labels for chest X-ray and 11 each for brain MRI and head CT.
- Localization: For each positive label, Grad-CAM or Score-CAM maps activation to a 3 × 3 anatomical region, whose name—not the heatmap—travels downstream.The default top-k setting is k=3; vocabulary varies by modality and imaging plane.
- Reporting: The report layer receives labels, glosses, probabilities, statuses, and region names but no image, while below-threshold predictions remain visible as pertinent negatives.A Claude backend and deterministic template fallback share the same evidence, so backend swaps affect fluency rather than asserted findings.
- Modality scope: All three taxonomies have tested routing, head sizing, anatomical vocabulary, and phrasing, but only chest X-ray is trained and benchmarked.No brain-MRI or head-CT checkpoint exists, so adding another modality is a data change rather than a predictive claim.
- Inference engines: The local engine implements the grounded architecture, whereas the hosted engine replaces it with a single vision LLM that directly reads the image.The two engines share a response contract, but only the local stack is benchmarked for the paper’s architectural claims.
4 Experimental Setup
The experiments evaluate MIRROR’s 14-label chest-X-ray classifier on a fixed ChestMNIST subset using CPU DenseNet-121 training, clinically relevant metrics, and explicit no-skill floors. They also verify that localization and report-writing layers leave predictions unchanged while measuring their runtime cost.
- Data: 7,200 training, 800 validation, and 12,000 test images form the ChestMNIST evaluation, with the test set sampled using seed 42 from the official split.ChestMNIST is the downsampled derivative of NIH ChestX-ray14 and retains its 14-label taxonomy.
- Data and training: 64 × 64 source images are upsampled to 224 × 224, so few-millimetre radiographic signatures are largely destroyed before DenseNet-121 training.Training uses CPU DenseNet-121 with AdamW at learning rate 3 × 10−4, weight decay 10−5, batch 32, and the best validation macro AUROC from four epochs.
- Metrics: The evaluation reports per-label and macro AUROC, macro AUPRC, 0.5-threshold sensitivity, specificity, PPV, NPV, Brier score, and Expected Calibration Error.Expected Calibration Error uses 10 equal-width bins.
- Metrics: A random ranker’s expected average precision equals prevalence p, while a constant predictor returning p scores a Brier of exactly p(1−p).These no-skill floors are reported because average precision and Brier score are uninterpretable in isolation on an imbalanced multi-label task.
- Post-hoc regression test: Predictions must be identical across classification-only, localization, and full-pipeline conditions because added layers modify neither weights nor logits.The harness verifies this implementation property and profiles each added layer’s per-stage wall-clock cost.
5 Results
MIRROR’s classifier shows genuine ranking signal across ChestMNIST labels, but default-threshold decisions and aggregate calibration metrics are misleading under severe class imbalance. Synthetic controls confirm the evaluation harness responds to visual signal, while localization adds roughly 40% CPU wall-clock cost without changing predictions.
- Overall performance: 0.729 macro AUROC, 0.135 macro AUPRC, and 0.031 macro F1 were achieved on the 12,000-image ChestMNIST test split.The published ResNet-18/50 baseline is approximately 0.77 AUROC, but used more data, epochs, resolution, and GPU training.
- Thresholded decisions: 11 of 14 labels produce no positive predictions at the default 0.5 threshold, yielding exactly 0.0 sensitivity and 1.0 specificity for those labels.Only Effusion, Infiltration, and Pneumothorax fire; macro sensitivity is 0.019 and macro F1 is 0.031.
- Ranking versus decisions: 1.6× to 6.8× lift over prevalence shows every one of the 14 labels ranks above chance despite the mostly silent thresholded outputs.The passage identifies Fibrosis at 1.6× and Cardiomegaly at 6.8×, locating the failure at thresholding and score calibration rather than representation.
- Calibration: 0.0453 macro Brier for the trained classifier is only 0.0019 below the 0.0472 macro Brier of a prevalence-only constant predictor.The measured advantage is about 4%, illustrating why aggregate calibration metrics can flatter models under class imbalance.
- Harness control: 0.917 mean AUROC for seven synthetic signal-bearing labels versus 0.533 for seven no-signal labels confirms that the harness responds to injected visual signal.The groups do not overlap, and the worst signal-bearing label exceeds the best no-signal label by 0.25 AUROC.
- Runtime and ablation: 41.4 and 36.2 ms are the measured Grad-CAM additions per study, while the offline template report adds 0.03 ms.Predictions remain identical across conditions, with maximum probability change 0.000 over n=24 studies; full-pipeline latency is 136.2 ms versus 100 ms for prediction alone.
6 Discussion
MIRROR guarantees that reportable findings are auditable against the classifier’s outputs, but this is narrower than faithful explanation or inherent interpretability. Its metrics show genuine ranking ability alongside ineffective default-threshold decisions, while the same predict–evidence–explain interface can span different inference backends.
- Finding-level grounding: MIRROR’s language layer can assert exactly the findings produced by the classifier, because it receives labels, probabilities, and region names rather than the image.A reader can check the report against the probability vector without trusting the model’s other properties.
- Limits of the guarantee: MIRROR does not claim faithful saliency-based explanation; inherent interpretability would be the stronger answer.The authors frame their contribution as auditability rather than explanation and have not run the cited saliency sanity checks.
- Discrimination versus decisions: Macro AUROC 0.729 and AUPRC lift of 1.6 to 6.8 times prevalence coexist with no positive prediction for 11 of 14 labels.The authors use the full metric panel to show that discrimination and decision behavior can diverge under the same model.
- Backend independence: The predict, show evidence, explain interface works across a PyTorch stack and a hosted vision LLM, despite backend-specific asymmetry.The shared JSON response separates the proposed interface from the inference engine.
7 Ethics, Safety, and Limitations
MIRROR is a research prototype whose outputs require licensed-radiologist verification and must not guide diagnosis or treatment. Its evaluation is limited, and its report writer can generate details that the system never measured.
- Safety: MIRROR is not a medical device, and every output is a draft requiring verification by a licensed radiologist.The system is neither reviewed nor cleared by a regulatory body, and outputs must not be used for diagnosis or treatment.
- Ungrounded detail: A generated report described a normal cardiac silhouette, a cardiothoracic ratio below 0.5, and clear costophrenic angles, although none was measured or localized.The pipeline produced label probabilities and saliency regions, but computed no cardiothoracic ratio or anatomical boundaries.
- Data governance: The work required no ethics-board review because it used existing public, de-identified benchmark datasets and collected no new human-subject data.No raw images, weights, or protected health information enter version control, and DICOM ingestion extracts only non-PHI technical tags.
- Limitations: 64-pixel source resolution and a 7,200-image budget define the sole quantitative benchmark, while the default threshold does not yield a working detector.The benchmark is ChestMNIST, and no clinical claim follows from it.
- Limitations: Brain-MRI and head-CT paths are implemented and tested but lack trained checkpoints, so MIRROR makes no predictive claim for those modalities.Explanation quality is also unmeasured.
8 Conclusion
MIRROR’s central contribution is a traceable, registry-driven pipeline whose structured evidence makes report findings auditable without guaranteeing the surrounding prose. On ChestMNIST, it shows meaningful ranking performance through macro AUROC and random-ranker comparisons.
- MIRROR composes classification, evidence localization, and report generation into one traceable, registry-driven pipeline.Its forward-looking value is the boundary between auditable structured findings and unconstrained generated prose.
- Structured evidence makes a report’s finding set auditable, but does nothing for the prose around it.The language layer is constrained to structured evidence, while the surrounding sentences remain ordinary generated text.
- 0.729 macro AUROC was achieved by the ChestMNIST classifier.
- 1.6 to 6.8 times better than a random ranker, the classifier ranked every ChestMNIST label within that range.
Reproducibility
MIRROR provides public code, a hosted demo, and a zero-setup local CLI that runs without a checkpoint or API key. Reproducibility is supported by metadata-stamped result files and torch-free unit tests.
- Access: Public code, a hosted demo, and a zero-setup CLI expose the MIRROR pipeline for local or hosted use.The CLI command is `python -m demo.run_demo <image>`.
- Access: The local CLI uses ImageNet weights with an offline template backend and requires no checkpoint or API key.
- Metadata: Every results JSON records the seed, git commit, and library versions in a reproducibility block.
- Testing: Unit tests are torch-free, while the ChestMNIST result is specified in `configs/chestmnist.yaml`.