PaperlayerView exact revision on arXiv

Source-linked AI summary

BrainDiff: Longitudinal Report Generation for Multimodal Brain MRI

Krish Patel, Peirong Liu

arXiv:2609.00593v2cs.CV

TL;DR

BrainDiff addresses the lack of longitudinal brain-MRI report generation, where subtle interval changes require comparing paired studies rather than reading one scan alone. It combines paired-study visual conditioning with grounding interventions and staged training, outperforming single-study and general-purpose baselines while retaining most internal performance externally.

  • Problem

    Longitudinal brain-MRI reporting remains unaddressed despite the clinical need to describe subtle, spatially distributed interval changes across paired studies.

  • Method

    BrainDiff independently resamples paired study modalities, conditions decoding on both images and the prior report, and uses prior-report dropout, counterfactual grounding, and staged curriculum training.

  • Results

    BrainDiff exceeds NeuroVFM by +0.0222 rg_er on report content and reaches 41.7% balanced accuracy for interval-change direction versus 32.6%.

  • Takeaways & Limitations

    The system retains 91% of its internal performance on an independent multi-site cohort, while grounding interventions raise image reliance 2.5-fold.

  • Takeaways & Limitations

    Main results are scored against LLM-synthesized comparison reports, and the change-module negative result is bounded to the frozen NeuroVFM grid at 16 mm with affine-only normalization.

Abstract

from arXiv · show

Neuroradiologists rarely read a brain MRI in isolation, yet automated brain-MRI report generation has been built almost entirely for single studies. Temporal analysis has been explored on chest radiography and chest CT, but to our knowledge, longitudinal reporting for brain MRI, where interval change is often subtle and spatially distributed, remains unaddressed. We present BrainDiff, the first longitudinal vision-language system for brain MRI. BrainDiff outperforms both frontier general-purpose and single-study neuroimaging models on the same patient pairs. Moreover, BrainDiff retains 91% of internal RadGraph-XL entity+relation F1 (rg_er) on an external, cross-hospital cohort. Beyond the system, we contribute three analyses. First, we identify two independent grounding levers: a counterfactual objective with prior-report dropout, which increases measured image reliance by ~47%, and a staged curriculum. Together, these interventions raise image reliance 2.5-fold from the baseline. Second, we provide a factorial over prior-report availability and image identity, isolating a visual contribution of +0.0387 rg_er, which grows when the prior report is withheld. Third, a cheap change-decodability test for candidate backbones shows that interval change is decodable far more weakly than single-study pathology (0.60 vs. 0.77 AUROC). Code is publicly available at https://github.com/jhuldr/BrainDiff.

1. Introduction

Brain MRI follow-up reporting requires comparing paired studies because interval changes are subtle, spatially distributed, and complicated by missing sequences and scarce longitudinal pairs. BrainDiff addresses this gap with a longitudinal brain-MRI vision-language system and grounding analyses designed to measure and increase image reliance.

  • Follow-up brain MRI is commonly interpreted against a prior, with report value centered on asserted interval change.
  • Brain MRI longitudinal reporting is challenging because anatomy is complex, sequences may be missing, interval-change signals are subtle, and true patient pairs are scarce.
  • Providing two studies and a prior report can encourage textual-prior reliance or hallucinated comparisons, obscuring how much the model uses images.
  • BrainDiff is presented as the first longitudinal vision-language system for brain MRI and is validated through intervention ablations, external BIND validation, and model-agnostic diagnostics.
  • 2.5-fold higher image grounding is obtained without additional parameters by combining prior-report dropout with a counterfactual objective and staged curriculum.

2. Related Work

Medical report generation has progressed toward grounded multimodal systems and foundation-model backbones, while longitudinal conditioning has been studied primarily in chest imaging. BrainDiff builds on these directions for volumetric brain MRI, where token volume, missing sequences, and limited paired data create additional challenges.

  • Report generation: Report generation has evolved from CNN-RNN captioning and transformers toward grounded generation, graph-based clinical metrics, and medical vision-language models.
  • 3D medical foundation models: Volumetric medical systems use pooling or learned resamplers because 3D studies produce far more patch tokens than X-rays.
  • Longitudinal and temporal reporting: Longitudinal conditioning research has focused mainly on chest imaging, including temporal representations, study alignment, prior-report conditioning, and longitudinal chest CT fusion.

3. Method

BrainDiff compares paired brain-MRI studies by independently handling available modalities, resampling visual features, and decoding them with the prior report as temporal context. Its staged training adds grounding, report-writing, and temporal-comparison objectives, while an explicit difference pathway is separately introduced and ablated.

  • Input and task: BrainDiff generates a free-text interval-change report from two studies containing up to four structural sequences after affine registration to MNI-152.
  • Model setup: The ViT-B encoder and Qwen3-14B decoder remain frozen while LoRA adapters provide the trainable parameters; the ChangeMapEncoder is absent from production.
  • Paired visual prefix: Each block-modality series is resampled independently, allowing paired studies with different sequence sets to enter a long-context decoder with temporal embeddings.
  • Difference pathway: The optional ChangeMapEncoder attends over local prior-study neighborhoods, forms a misregistration-corrected residual, and pools dense difference features into coordinate-tagged change groups.
  • Grounding objectives: Counterfactual grounding penalizes higher loss for the correct prior than a swapped prior, while 30% prior-report dropout regularizes against textual shortcuts.
  • Stage 1 grounding corpus: Stage 1 programmatically creates 3D grounding supervision from lesion-segmentation datasets, expanding box-caption pairs into reference, captioning, and region-grounding tasks.
  • Curriculum: Four training stages install spatial grounding, single-study reporting, difference-path pretraining, and temporal-comparison language in sequence.

4. Experiments

Experiments compare BrainDiff with frontier and single-study baselines, assess external generalization, and quantify image grounding under controlled interventions. BrainDiff improves interval-change classification and report content, while grounding interventions increase image reliance with a small report-quality trade-off.

  • Model comparison: BrainDiff reaches 41.7% balanced accuracy for interval-change direction versus NeuroVFM’s 32.6%.NeuroVFM assigns 48–54% of each true class to Mixed/unclear, whereas BrainDiff predictions concentrate on the correct class.
  • Model comparison: BrainDiff exceeds Opus 5 and GPT-5.6 Sol on the 64-study frontier subset, scoring 0.3870 versus 0.2083 and 0.2920 rg er.The corresponding margins are +0.1787 and +0.0950 rg er.
  • External validation: On BIND, BrainDiff scores 0.3506 rg er versus 0.3837 internally, a ∼9% relative drop without site-specific adaptation.BIND contains 800 usable pairs processed through the identical pipeline.
  • External validation: The larger BLEU-4 drop than rg er suggests that surface phrasing is more site-specific than clinical entities and relations.The passage attributes better generalization to the entity- and relation-level content scored by rg er.
  • Image reliance: Grounding interventions raise image reliance ∼47%, from +0.0263 to +0.0387 rg er, with a +0.0124 paired difference.This trades a −0.0059 rg er change on correct scans for a 1.5% report-quality cost; swapped valid scans isolate grounding.
  • Image reliance: Adding the counterfactual objective, prior-report dropout, and curriculum increases image reliance by +0.0231 rg er with 95% CI [+0.0160, +0.0301].Reliance rises monotonically as interventions are added, indicating near-independent contributions to grounding.

5. Further Analysis and Discussion

The analyses separate report-prior, image-identity, and backbone contributions to longitudinal reporting. They show stronger image reliance from grounding interventions, limited change decodability in frozen features, and a resolution-linked worsening error.

  • 5.1. What the model sees: Image reliance is defined as rg er with correct scans minus rg er with another patient’s scans.The factorial varies prior-report availability and image identity while keeping rows identical.
  • 5.2. What the backbone preserves: The ChangeMapEncoder adds +0.0009 rg er, with CI [−0.0048, +0.0065] including zero.BLEU-4 and METEOR also move slightly against the delta-on model (−0.0050 and −0.0069).
  • 5.2. What the backbone preserves: Interval change is less decodable than single-study pathology: 0.601 versus 0.766 AUROC from frozen NeuroVFM features.Pooling between-patient magnitude inflates decodability to 0.702, while augmented duplicate pairs yield 0.997 AUROC.
  • 5.2. What the backbone preserves: BrainDiff assigns 46% of truly worsened studies to Stable, compared with 27% to Worsened.The paper links this safety-critical error to coarse spatial resolution and sub-millimeter lesion-growth differences.
  • 5.3. Limitations: The comparison targets are LLM-synthesized from two radiologist reports, which favors text overlap and partially confounds the prior-report result.The negative difference-pathway result is specific to the frozen NeuroVFM grid at 16 mm under affine-only normalization.

6. Conclusion

BrainDiff introduces longitudinal brain-MRI report generation and combines system evaluation with grounding and backbone diagnostics. It improves interval-change direction over a single-study baseline despite weak preservation of interval change in the frozen backbone.

  • 6. Conclusion: BrainDiff is presented as the first longitudinal report generator for brain MRI.It is evaluated against frontier general-purpose and single-study neuroimaging models on the same patients.
  • 6. Conclusion: BrainDiff improves interval-change direction over the single-study baseline despite the frozen backbone preserving interval change more weakly than single-study pathology.The conclusion characterizes this as a bounded negative result for the explicit difference encoder.

Supplementary Material

The supplementary material documents grounding, validation, difference-pathway pretraining, interval-change labels, and classifier validation.

  • Supplementary Material: S1 analyzes staged-curriculum effects on grounding, while S2 validates against real radiologist Impression text.
  • Supplementary Material: S3 details difference-pathway pretraining, S4 defines and characterizes interval-change labels, and S5 details and validates the interval-change classifier.

S1. What the curriculum buys

The curriculum primarily changes model behavior rather than final report quality: it reduces reliance on the prior report and increases reliance on images.

  • S1. What the curriculum buys: The three curriculum arms are indistinguishable on headline rg_er when the prior report is available.Their report-available intervals overlap heavily.
  • S1. What the curriculum buys: 0.0288 lower prior-report reliance distinguishes the full curriculum from no stage 2, while the gap versus no curriculum is 0.0176.Both intervals clear zero, and stage 2 is identified as responsible for reducing prior-report reliance.
  • S1. What the curriculum buys: The correct-scan image-reliance contribution reaches +0.0387 rg_er under the full curriculum, versus +0.0299 without stage 2 and +0.0266 without curriculum.This measurement is independent of prior-report availability.
  • S1. What the curriculum buys: The curriculum’s value is a shift toward image grounding, making it an additional grounding lever rather than a data-efficiency tool.Its behavioral effect complements the counterfactual objective.

S2. Validation against real radiologist text

BrainDiff’s advantage persists when generated reports are evaluated against real radiologist Impression text rather than synthesized comparison reports. This validates the direction of the reported temporal-conditioning benefit, while the evaluation remains a ranking test rather than an absolute-quality claim.

  • S2. Validation against real radiologist text: The evaluation re-scores unchanged model outputs against real radiologist Impression text that was never shown to the models.BrainDiff receives only the prior report and images during generation.
  • S2. Validation against real radiologist text: The real-text evaluation is a ranking-preservation test, not an absolute-quality claim, because radiologist Impressions are short and stylistically unlike generated comparison reports.The rg_er metric penalizes this stylistic mismatch.
  • S2. Validation against real radiologist text: Some Impression comparisons may derive from visits whose scans are unavailable.This constrains how completely the real-reference evaluation can align text with observed imaging.
  • S2. Validation against real radiologist text: +0.0075 rg_er is BrainDiff’s advantage over NeuroVFM against real radiologist text, with an interval clearing zero.The result has the same direction and significance as the synthetic-reference margin.

S3. Difference-pathway pretraining

The difference pathway is pretrained to encode spatially specific interval changes, but it does not measurably improve report quality. The reported limitation is attributed largely to the backbone rather than the difference module.

  • S3. Difference-pathway pretraining: The ChangeMapEncoder is pretrained separately with the vision encoder frozen during S3, then trained further during S4.Auxiliary reconstruction, compression, and dense heads are discarded, leaving only the change map for S4.
  • S3. Difference-pathway pretraining: 44,882 longitudinal pairs from MR-RATE, OASIS-3, BraTS-METS, and BraTS-GLI support pretraining, including augmented duplicates with zero true interval change as nuisance controls.These controls separate acquisition-driven feature differences from actual change without requiring change labels.
  • S3. Difference-pathway pretraining: The three equally weighted losses encourage spatially specific reconstruction, pair discrimination, and token-level saliency.InfoNCE operates on the coarse map, while saliency uses per-token cosine distance between warped prior and current features.
  • S3. Difference-pathway pretraining: Token-grid supervision and gated pooling make changed tokens dominate the coordinate cell into which they are pooled.Supervision is not applied to the coarsened map because its per-cell target is near-uniform.
  • S3. Difference-pathway pretraining: +0.0009 rg_er is the difference pathway’s report-quality contribution, with 95% CI [−0.0048, +0.0065].The pathway does not measurably improve report quality, and the limitation is attributed largely to the backbone.

S4. Interval-change labels

BrainDiff assigns seven-way interval-change labels and groups them into four clinically directional classes. The held-out generation analysis uses 1,444 complete pairs after excluding 20 incomplete cases.

  • S4. Interval-change labels: Each S4 pair receives one of seven labels: Stable, Improved, Resolved, Progressed, New lesion, Mixed interval change, or Indeterminate.
  • S4. Interval-change labels: The paper groups the seven source labels into four classes by clinical direction, merging Resolved with Improved and New lesion with Progressed.
  • S4. Interval-change labels: Mixed/unclear combines Mixed interval change and Indeterminate, which lack a single established direction and is more difficult by construction.This heterogeneity plausibly contributes to low recall for the class.
  • S4. Interval-change labels: 1,444 of 1,464 held-out labeled pairs are usable for generation because 20 lack complete imaging at both timepoints.Classifier fidelity uses all 1,464 labeled pairs instead.

S5. Interval-change classification

The interval-change classifier parses the Impression into entities and directional polarities, then deterministically assigns a four-way change class. Its agreement with reference labels establishes an empirical ceiling for downstream evaluation rather than radiologist agreement.

  • Classifier design: The procedure reads only the Impression because it states the interval verdict, whereas the findings body also lists stable and changed items that can create spurious directions.This design avoids promoting an incidental cue into the report’s overall change class.
  • Classifier design: The classifier uses a curated 1,850-term entity vocabulary and a 41-term change lexicon to parse Impression sections into entity–polarity pairs.It applies longest-match entity extraction and excludes size adjectives such as “small” or “large” from polarity cues.
  • Classifier design: Reports are assigned Stable, Progressed, Improved, Mixed interval change, or Indeterminate according to asserted directional cues and explicit indeterminacy.Mixed applies when multiple directional polarities appear; absent polarity also yields Indeterminate.
  • Evaluation procedure: Reference-report fidelity bounds model evaluation because even perfect generated reports cannot exceed the classifier’s agreement with LLM-assigned labels.This ceiling measures internal fidelity to assigned labels, not agreement with a radiologist.
  • Evaluation procedure: The same classifier supports interval-change confusion analysis and a timepoint-reversal probe that tests whether asserted direction changes when study order is reversed.The reversal analysis excludes nondirectional cases and uses signed directions derived from the classifier.
Loading 2609.00593v2…