Source-linked AI summary
The Diagnosis a Reporter Leaves Unspoken: Surfacing Frozen Tumor Features for Brain-Tumor MRI Reporting
Khawaja Murad ul Hassan, Ruqiyya Adil, Adil Qayyum, Rida Hassan, Asad Mansoor Khan, Muhammad Usman Akram, Mehran Ebrahimi
TL;DR
Brain-MRI report generators can retain diagnostic information without verbalizing it. NeuroFusion surfaces that signal through structured field conditioning and a single-pass decoder, improving diagnosis and prose comparisons while lowering latency; its scope remains assistive, with small and pilot evaluation components.
Problem
A same-backbone CoT reporter misreads meningiomas and metastases as glioma despite diagnostic information being recoverable from frozen segmentation features.
Method
NeuroFusion conditions a fast single-pass draft-then-review decoder on discriminative field-classifier outputs over per-lesion features while retaining the identical Mistral backbone.
Results
NeuroFusion won 8 of 9 prose-content comparisons, restored diagnosis recall to 0.92 for meningioma and 0.75 for metastasis, and reduced latency to approximately 73-89 versus 457 seconds per case.
Takeaways & Limitations
The results support head-surfacing as an assistive reporting approach that exposes latent diagnostic information while preserving machine-checkable reporting and lower-latency decoding.
Takeaways & Limitations
The reader study is a single-institution pilot with two neurologists and nine cases, and the underlying segmentation bounds geometric completeness.
Abstract
from arXiv · showhide
A capable brain-MRI report generator can still be, in effect, diagnostically silent. When a multi-chain chain-of-thought (CoT) reporter built on a medical Mistral-7B backbone is evaluated on held-out cohorts, it names most meningiomas and almost all metastases "glioma" (diagnosis recall 0.44/0.07). Yet the answer is not absent from the model: a supervised linear probe applied to its frozen segmentation features recovers the three tumour cohorts at 0.82 macro-F$_1$ (5-fold cross-validation; chance $\approx$0.33). We introduce NeuroFusion, an assistive reporter that surfaces this latent signal rather than overriding it: discriminative field-classifier heads over per-lesion features condition a fast, single-pass draft-then-review decoder on their committed outputs. Built on the identical Mistral backbone, this restores the diagnosis (meningioma 0.92, metastasis 0.75) and wins 8 of 9 prose-content comparisons across three held-out cohorts (RaTEScore, RadGraph-F$_1$, GREEN; Holm-corrected paired BCa), with no significant loss on the ninth, at 5-6x lower latency ($\approx$80 vs. 457 s/case). A controlled negative result sharpens the mechanism: a learned diagnosis pin that overrides the decoder instead of merely informing it collapses out-of-distribution metastasis recall to 0.03. Grammar-constrained decoding keeps 92.3% of records schema-valid, making every sentence entailment-checkable (7.5% contradicted vs. 36.8% for the direct baseline). In a blinded nine-case pilot, two board-certified neurologists independently rated NeuroFusion highest in every tumour type, the only system with zero critical errors, and gave it the top-rated sign-off in eight of nine cases (six outright, two ties).
1 Introduction
NeuroFusion addresses diagnostic suppression in brain-MRI reporting by surfacing information already present in frozen segmentation features. It restores diagnosis and improves prose quality while reducing latency and adding auditability.
- 0.82 macro-F1: a supervised probe separated three tumour cohorts from frozen segmentation features, despite same-backbone diagnosis recall of 0.44/0.07.The CoT reporter misread more than half of meningiomas and nearly every metastasis as glioma.
- NeuroFusion uses discriminative field-classifier heads to condition a single-pass decoder on committed structured outputs, without a separate diagnosis model.The approach is termed head-surfacing and leaves diagnosis generation to the language model.
- 8 of 9 prose-content comparisons improved against the identical-backbone CoT variant, with no significant regression on the ninth.The gain was individually significant on the held-out metastasis cohort.
- The system adds machine-checkable records and per-sentence faithfulness checking, while neurologists rated it highest in every tumour type in a blinded study.The reader study is identified as the first blinded neurologist reader study for this task.
2 Related Work
Prior brain-reporting systems ground generation in image regions but generally leave diagnosis to the language model and expose no checkable schema. NeuroFusion instead conditions reporting on structured field predictions while requiring the decoder to commit to those fields.
- Brain MRI reporting: Existing brain MRI generators ground reports in image regions but leave diagnosis to the language model and expose no checkable schema.The paper benchmarks directly against AutoRG-Brain and BrainGemma3D.
- Constrained decoding and faithfulness: XGrammar guarantees syntactic validity, while RadGraph, RaTEScore, and GREEN assess entity-level grounding beyond n-gram overlap.
- Label-conditioned reporting: Unlike region-grounded reporting, label-conditioned reporting supplies what a finding is rather than only where it is.
- Positioning: The paper’s distinction is that gains require a decoder trained to commit to fields; merely providing identical head argmax outputs does not suffice.The same-base CoT model still misreads diagnosis when given those outputs as text.
3 Method
NeuroFusion converts segmented lesions into bounded per-lesion visual tokens and structured field predictions, then generates a schema-constrained report in one draft-then-review pass.
- Pipeline overview: A frozen MedNeXt backbone segments the tumour, while connected components retain up to four largest lesions for per-lesion processing.Lesions are volume-ranked and represented with centroid-based positional information.
- Pipeline overview: A per-lesion Q-Former compresses pooled, position-tagged features into 32 tokens, producing a bounded visual input for the language model.
- Head-surfacing and single-pass decoding: Linear field classifiers predict structured fields from pooled per-lesion Q-Former features, while diagnosis remains for the language model to recover.
- Head-surfacing and single-pass decoding: The classifier outputs and segmentation-derived location are injected as prompt text to guide field and diagnosis generation.The prompt includes cues such as temporal, heterogeneous, and ring.
- Head-surfacing and single-pass decoding: XGrammar masks each decoding step to schema-permitted tokens, preventing syntactically invalid records.
- Head-surfacing and single-pass decoding: The single K=1 pass reconciles draft enum fields against head argmax outputs without a second language-model call.Only the router, Q-Former, field classifiers, and LoRA adapter are trained.
4 Experimental Setup
The evaluation combines patient-disjoint held-out cohorts, same-backbone and external baselines, structured and prose metrics, pre-specified statistical testing, and a blinded neurologist pilot.
- Data: 121 BraTS-2020 cases with radiologist-reviewed structured reports support training, with 39 held out for in-distribution testing and 50 for calibration.Two RadGenome cohorts extend training in a patient-disjoint manner.
- Data: BraTS-MET contains 60 held-out cases unseen by the reporter during training, making metastasis out-of-distribution for the reporter.The segmenter was metastasis-fine-tuned, but the reporter saw no metastasis reports.
- Systems and baselines: All systems in the primary comparison share the identical Mistral language model, including NeuroFusion, the CoT reporter, and the diagnosis-pin variant.External baselines include M3D-LaMed and LLaVA-Med on native free-text tasks.
- Metrics and protocol: The evaluation reports schema validity, structured field macro-F1, three prose metrics, and a blinded five-dimension clinical rubric.Diagnosis recall is treated as a descriptive lexical proxy rather than a primary endpoint.
- Metrics and protocol: The primary endpoint is RaTEScore on meningioma, with nine same-base metric-cohort cells tested using Holm correction and TOST equivalence bounds of ±0.03.Intervals use paired per-case BCa bootstraps with 20,000 resamples.
- Reader study: Two board-certified neurologists independently rated nine randomly selected held-out cases across five axes, critical errors, and forced sign-off.The blinded pilot reports descriptive means and critical-error counts rather than p-values.
5 Results: in distribution and the cross-cohort win
NeuroFusion improves prose reporting across held-out tumour cohorts while restoring diagnosis and reducing latency, but directly pinning a learned diagnosis classifier fails out of distribution.
- In distribution: 92.3% of in-distribution cases produced schema-valid records (36/39; Wilson 95% CI [0.80, 0.97]) with grammar-constrained decoding.Without grammar-constrained decoding, schema validity collapses toward zero.
- In distribution: 0.449 structured-field accuracy was achieved by the record-verbalizer, versus ≤0.379 for both VLM baselines.Its Opus clinical score was 3.11 and RadGraph score was 0.275.
- Cross-cohort win: 8 of 9 prose-content comparisons were wins with 0 losses and 1 non-significant cell against the same-base CoT reporter.NeuroFusion won every glioma and meningioma cell and two metastasis metrics; the remaining metastasis RadGraph comparison was non-significant.
- Cross-cohort win: +0.117 was the largest lift on RaTEScore-meningioma, the primary endpoint, at 5-6× lower latency than CoT.Latency was approximately 73-89 versus 457 seconds per case on an A100.
- Negative result: 0.03 was the out-of-distribution metastasis recall after the diagnosis pin was applied, declining from 0.75 in distribution.The diagnosis pin produced 0 of 9 gains, whereas the language model’s own recall transferred without collapsing.
- External baselines: 0.596 RaTEScore and 3.17 blinded-rubric score made NeuroFusion significantly better than every learned baseline on metastasis.The narrowest margin was GREEN: 0.267 versus AutoRG’s 0.212, p=0.02.
6 Clinician Reader Study
In a blinded nine-case pilot, NeuroFusion was rated highest across tumour cohorts, with no critical errors and the most sign-off selections.
- Reader-study results: NeuroFusion was rated highest in every tumour cohort and surpassed its CoT predecessor in each cohort.The mean ratings were 4.00 versus 1.00 on meningioma and 3.47 versus 3.27 on metastasis.
- Reader-study results: Zero critical errors occurred for NeuroFusion across all nine cases, whereas every other system had four to eight.The reported Wilson 95% upper bound for NeuroFusion was 0.34.
- Study design: Table 3 summarizes independent two-neurologist ratings across nine cases, with means over five 0-5 axes, critical-error counts, and sign-off credits.The table includes three cases per cohort and excludes BrainGemma3D from the glioma cohort.
- Reader-study results: NeuroFusion received the top-rated sign-off in 8 of 9 cases, uniquely in six and tied in two.The ninth sign-off went to LLaVA-Med; ties were credited under prespecified rules.
7 Analysis
NeuroFusion combines machine-checkable reporting with image-grounded field surfacing, improving content while preserving a documented dependence on segmentation quality and assistive use.
- Auditability: 7.5% of NeuroFusion sentences were contradicted versus 36.8% for the direct baseline at the record-verbalizer operating point.The entailment judge evaluated every sentence in the machine-checkable record; both arms used Claude Opus 4.8.
- Mechanism: Head-surfacing adds chiefly diagnosis and intensity-related content that the free-running language model otherwise discards, while geometric accuracy remains segmentation-bounded.A mask-reading rule edges out NeuroFusion on the specific geometric score in all three cohorts.
- Content analysis: NeuroFusion leads RaTEScore in all three cohorts and RadGraph on GLI and MEN, while GREEN is LLM-scored.The clinical rubric corroborates rather than drives these judge-free metric results.
- Limitations: Geometric completeness is bounded by segmentation quality, including metastasis-fine-tuned Dice 0.67, and empty predicted masks are routed to manual review.The reader study is also limited to a single-institution pilot with two neurologists and nine cases.
- Limitations: NeuroFusion is best understood as an assistive drafting tool rather than an autonomous one.Future work includes larger multi-site reader studies, human validation of the entailment judge, and component-level ablations.
- Overall analysis: NeuroFusion beats the identical backbone’s chain-of-thought variant on 8 of 9 content comparisons at 5-6× lower latency while producing a verifiable record.The conclusion also reports a diagnosis-pin negative result and the highest neurologist ratings with zero critical errors.