PaperlayerView exact revision on arXiv

Source-linked AI summary

BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

Qizhen Lan, Mengchen Fan, Hang Zhang, Jingwei Duan, Moule Lin, Jialin Chen, Baocheng Geng, Xiaoqian Jiang

arXiv:2610.06571v1cs.CVcs.AI

TL;DR

Brain MRI VLM evaluation needs to assess evidence tracing across serial studies, MRI sequences, volumetric anatomy, and report-grounded references, beyond isolated-image tasks. BrainTRACE provides a benchmark and decomposed evaluation tracks; its results show that local successes rarely compose into grounded longitudinal interpretation.

  • Problem

    Existing medical VQA and 3D imaging benchmarks cover parts of brain MRI interpretation, but do not fully evaluate longitudinal, multi-sequence, volumetric, report-grounded reasoning.

    Source: §1 · pp. 1–2

  • Method

    BrainTRACE is a report-grounded benchmark that separates permitted visual inputs from hidden clinical references and organizes evaluation by reasoning depth and evidence demands.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across 20 model configurations, best broad compatible closed-form accuracy is 41.9%, strongest open-ended clinical pass is 19.8%, and best case success rate is 1.4%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Current VLM successes do not reliably compose into longitudinal, report-grounded clinical synthesis, and volumetric access alone is insufficient.

    Source: §5 · p. 10

    The benchmark covers MR-RATE-derived longitudinal brain MRI cases with sufficient follow-up, reports, and segmentation evidence, and its scores are not estimates of absolute clinical performance.

    Source: G Limitations and Responsible Use · p. 35

Abstract

from arXiv · show

Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences, localize findings within volumetric anatomy, and translate this evidence into report-grounded assessments. Existing medical VQA and 3D imaging benchmarks capture important parts of this workflow, but often evaluate brain MRI through isolated images, static volumes, or ungrounded report-style answers, thereby obscuring failures in the evidence chain that support clinical validity. We introduce BrainTRACE, a report-grounded benchmark for evaluating whether vision-language models can trace the evidence structure required for longitudinal brain MRI interpretation. BrainTRACE contains 7,273 scored VQA instances derived from 1,778 longitudinal patients, 7,299 MRI studies, and approximately 29k co-registered 3D MRI sequence volumes. The benchmark is organized by five levels of clinical reasoning, from acquisition recognition to case-level synthesis, and by evidence demands covering longitudinal comparison, report-grounded references, multi-sequence integration, and volumetric spatial evidence. BrainTRACE supports rendered inputs compatible with standard VLM interfaces, a 3D-evidence condition, and a decomposed case-reasoning track that audits six steps in a longitudinal evidence chain. Evaluation of 20 VLM configurations shows that current systems can identify isolated visual cues but rarely compose them into grounded longitudinal interpretations. We release the benchmark specification, evaluation lists, scoring implementation, scoring rubrics, and audit-record format to support reproducible progress in brain MRI VLM evaluation.

1 Introduction

BrainTRACE frames brain MRI VLM evaluation as tracing report-grounded evidence across serial studies, sequences, and volumetric anatomy. It introduces a diagnostic benchmark and finds that local visual successes rarely compose into grounded longitudinal interpretation.

1 Introduction

  • Brain MRI follow-up requires jointly assessing lesion location, sequence-specific appearance, and longitudinal change rather than interpreting an isolated image or sequence.A trustworthy interpretation should follow an auditable evidence chain.Source: §1 · pp. 1–2
  • BrainTRACE separates model-facing visual evidence from hidden clinical references, using reports and segmentation-derived measurements to define supported answers and material contradictions.Models receive only permitted visual inputs, questions, and answer formats.Source: §1 · pp. 2–3
  • Its two diagnostic axes separate clinical reasoning depth from evidence demands, helping distinguish wrong operations, missing evidence use, and unsupported clinical synthesis.Reasoning depth runs from acquisition recognition to case-level synthesis; evidence demands include longitudinal, multi-sequence, volumetric, and clinical reasoning.Source: §1 · p. 3
  • BrainTRACE formalizes evaluation as longitudinal evidence tracing, where clinical validity depends on comparing serial studies and grounding conclusions in relevant imaging and clinical references.The benchmark contains 7,273 scored instances derived from 1,778 patients, 7,299 studies, and approximately 29k co-registered 3D MRI sequence volumes.Source: §1 · pp. 2–3
  • Current VLMs show a longitudinal evidence composition gap: best broad compatible closed-form accuracy is 41.9%, strongest open-ended clinical pass is 19.8%, and best case success rate is 1.4%.The evaluation covers 20 model configurations.Source: §1 · p. 3

2 Related Work

Prior medical VQA and brain-imaging resources cover important components of evaluation, but BrainTRACE makes report-grounded longitudinal brain MRI follow-up its central unit. It jointly tests temporal comparison, sequence integration, volumetric localization, burden, and clinical reasoning.

2.1 Medical VQA Benchmarks

  • Medical VQA benchmarks span selected 2D radiology images, broad multimodal tasks, paired-image difference reasoning, structured diagnosis, 3D imaging, CT text, spatial grounding, and longitudinal CT diagnosis.These resources do not define brain MRI evaluation as report-grounded longitudinal evidence tracing.Source: §2.1 · p. 3

2.2 Brain Imaging Benchmarks and Resources

  • Brain imaging benchmarks address multimodal comprehension, anomaly localization, 2D-to-3D bridging, and cross-sectional report generation, while complementary resources support segmentation and longitudinal cohorts.These resources still do not make longitudinal brain MRI follow-up the central unit of report-grounded VLM evaluation.Source: §2.2 · p. 4
  • BrainTRACE jointly evaluates report-grounded comparison across longitudinal studies, multi-sequence integration, volumetric localization and burden assessment, and clinical reasoning beyond surface description.Table 1 summarizes benchmark-level coverage of these evidence attributes.Source: §2.2 · p. 4

3 BrainTRACE Design and Construction

BrainTRACE builds report-grounded VQA items by separating permitted model inputs from hidden clinical references and diagnostic labels. Its task taxonomy, evidence tags, evaluation tracks, and human review make the evidence chain and its potential failure points explicit.

3.1 From clinical records to evaluation instances

  • Each evaluation item separates permitted model-facing evidence, a hidden report- or measurement-derived reference, and diagnostic labels for reasoning depth, evidence demand, and track.Closed-form items use normalized targets; open-ended and case-level items specify required facts and clinically material contradictions.Source: §3.1 · p. 4

3.2 Tasks, evidence demands, and evaluation tracks

  • The five reasoning levels progress from acquisition and basic detection through single-study characterization, interval comparison, trajectory and response, to case-level synthesis.Evidence tags capture longitudinal comparison, complementary MRI sequences, volumetric evidence, and interpretation beyond surface recognition; the six-step case track makes the evidence chain explicit.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 Report-grounded benchmark construction

  • The benchmark uses longitudinal MR-RATE cases with at least three MRI studies, paired reports, and segmentation-derived measurements, and builds tasks from report-supported findings linked to imaging and measurements.Human reviewers inspect task cards for report support, timepoint alignment, visual evidence availability, answer validity, redaction, and scoring consistency.Source: §3.3 · pp. 5–6

3.4 Quality Control

  • Hidden references store target answers, required facts, contradictions, and scoring criteria; case-track steps receive item-level review by an ABR-certified radiation oncologist.Evaluation inputs exclude reports, parsed findings, segmentation masks, measurement tables, and hidden references, with quality checks for redaction, support, visual evidence, and scoring consistency.Source: §3.4 · p. 6

4 Experiments

The experiments evaluate zero-shot VLMs using permitted evidence inputs and score both answer accuracy and factual clinical acceptability. Results show that local successes rarely compose into case-level longitudinal interpretation.

4.1 Experimental Setup

  • Twenty model configurations were evaluated zero-shot using permitted visual evidence, questions, answer formats, and allowed metadata; reports and hidden references were withheld.Malformed outputs, refusals, and abstentions remained in the denominator.Source: §4.1 · p. 7

4.2 Evaluation Metrics

  • Closed-form items used exact-match scoring, while open-ended and case-level answers received factual clinical audits of required facts, critical slots, errors, and unsupported claims.Step Pass averages item-level pass indicators; Case Success requires all six steps in a case to pass.Source: §4.2 · pp. 7–8

4.3 Experimental Results

  • 41.9% was GPT-5’s best broad-compatible closed-form accuracy, while GPT-5.4 achieved the best 3D evidence accuracy at 42.2%.Aggregate accuracy varied across clinical skills and did not indicate reliable use of burden, temporal-change, or treatment-response evidence.Source: §4.3 · p. 8
  • 19.8% of broad open-ended questions and 15.6% of 3D open-ended questions passed for GPT-5.4, the strongest model on this axis.Failures included omitted disease status, reversed interval direction, wrong-timepoint evidence, and unsupported anatomy.Source: §4.3 · pp. 8–9, Table 3 · p. 8
  • 24.5% was the best aggregate Step Pass, yet only two model rows reached 1.4% Case Success, showing attrition across all six steps.Case Success requires all six steps in the same longitudinal case to pass.Source: §4.3 · p. 9, Figure 4 · p. 9
  • Latest-timepoint-only inputs reduced accuracy by 37.5 points for Qwen3.5-27B and 23.3 points for GPT-5.4; single-T2 and image-free inputs also reduced performance.The matched ablations used n = 120 and a constructed 50.0% full-evidence baseline, conditional on the balanced sample.Source: §4.3 · p. 9, Table 4 · p. 9

5 Conclusion

BrainTRACE evaluates whether VLMs can trace evidence across serial brain MRI studies, sequences, volumetric anatomy, and report-derived references. The results show that local recognition rarely composes into longitudinal clinical synthesis.

5 Conclusion

  • BrainTRACE is a report-grounded benchmark for longitudinal brain MRI VLM evaluation, and its main finding is that local recognition rarely survives through clinical synthesis.The paper reports that volumetric access alone is insufficient without stronger multi-sequence, temporal, and evidence-grounded reasoning.Source: §5 · p. 10

A Benchmark positioning and scope

The appendix positions benchmark resources by whether they jointly cover report-grounded longitudinal brain-MRI reasoning across four evidence attributes. It treats these attributes as scope descriptors rather than quality rankings.

A Benchmark positioning and scope

  • The comparison asks whether resources jointly evaluate report-grounded longitudinal brain-MRI reasoning across longitudinal, multi-sequence, volumetric, and clinically interpretive evidence.The appendix frames this as a narrow benchmark-design question, not a ranking of prior resources.Source: A Benchmark positioning and scope · p. 15
  • Table 5 uses L, M, V, and C for longitudinal, multi-sequence, volumetric, and clinical-reasoning evidence; its scale values use each source’s native unit.The table’s evidence attributes are not quality judgments, and its scale should not be read as a normalized dataset-size comparison.Source: A Benchmark positioning and scope · p. 15, Table 5 · p. 15

B Dataset construction and task coverage

BrainTRACE turns longitudinal MR-RATE follow-up records into auditable brain MRI VQA instances, with hidden report-grounded references and task labels for clinical reasoning depth and evidence demands. Its factual-audit scoring distinguishes local step performance from success across a complete case.

B.1 Construction and audit records

  • Each item separates model-facing questions and permitted visual evidence from hidden references grounded in reports or measurements and retained for scoring audits.The model-facing record includes the rendered question, answer format, permitted metadata, and visual evidence; the hidden record preserves the target, rubric, evidence, and checks.Source: §B.1 · p. 16

B.2 Upstream MR-RATE source and retained follow-up cohort

  • BrainTRACE retains 1,778 patients with 7,299 MRI studies and 65,974 series-level metadata rows, requiring at least three studies per patient.The source is MR-RATE; the retained cohort supports follow-up evaluation but is not claimed to represent all neuroradiology indications.Source: §B.2 · pp. 16–17
  • The cohort spans broad age groups, follow-up durations, common sequences, multiple scanner vendors, and both 1.5T and 3.0T studies, without implying population representativeness.Report-keyword groups characterize the cohort rather than serve as official diagnostic prevalence labels.Source: §B.2 · pp. 17–18, Figure 6 · p. 17, Figure 7 · p. 18

B.3 Benchmark task coverage

  • The benchmark separates clinical reasoning depth from evidence demand, labeling operations and whether answers require longitudinal, multi-sequence, volumetric, or clinical evidence.The task design matrix reports denominators for broad-compatible, 3D-evidence, and case-reasoning tracks, with volumetric evidence split into pseudo-3D and native-volume forms.Source: §B.4 · p. 18, §B.5 · p. 19, Table 6 · p. 20

B.4 Clinical-level taxonomy

  • Its five reasoning levels progress from acquisition recognition and single-study findings through interval change and trajectory assessment to integrated case-level judgment.These levels specify the evidence operation tested, not merely increasing difficulty.Source: §B.4 · pp. 18–19

C.1 Factual-audit scoring for open-ended and case-level answers

  • Open-ended answers pass only when required facts and critical slots are satisfied, with no critical contradiction or clinically material unsupported claim.The audit prioritizes factual clinical acceptability over prose similarity; omissions and contradictions are tracked separately.Source: §C.1 · p. 20, §C.1.1 · pp. 21–22

C.1.1 Scoring fields and pass rule

  • Case Success requires all six steps in one case to pass, making it stricter than Step Pass; two neuroradiologists independently reviewed a 60-step subset.The expert-review exercise evaluates judge agreement, not clinical certification.Source: §C.1.1 · p. 21, §C.5.1 · p. 23

C.4 Per-step attrition in case-reasoning

  • Claude Opus 4.6 reached 7.1% final-synthesis pass, while GPT-5-mini had the highest aggregate Step Pass at 24.5%; both had 0/70 full-case survival.Earlier steps passed at roughly 30–43% for the strongest configurations, illustrating attrition during case-level composition.Source: §C.4 · p. 23

C.5.2 Headline metrics

The scoring audit examines agreement between the factual-audit judge and reviewers, sensitivity to judge family, and reproducibility of the released evaluation records. The appendix also reports uncertainty summaries and clarifies the interface scope of the 3D-evidence comparison.

C.5.2 Headline metrics

  • κw was 0.737 and 0.645 for reviewer–judge pairs, versus 0.821 between reviewers on the 1–5 quality scale.Judge specificity was 1.000 against both reviewer references; sensitivity was 0.484 and 0.536.Source: §C.5.2 · p. 23, Figure 9 · p. 24

C.5.3 Per-step decomposition

  • Final-synthesis κw was lower than earlier-step agreement for all three reviewer and judge pairings.The reported inter-human κw falls to 0.31 at S6, while reviewer–judge values are 0.42 and 0.50.Source: §C.5.3 · p. 23

C.5.4 Per-model decomposition

  • Reviewer–judge agreement was similar in rank and magnitude across GPT-5.4 and Qwen3-VL-30B steps, rather than being driven by one model family.The reported reviewer–judge κw values were 0.699 and 0.772 for Reviewer 1, and 0.628 and 0.664 for Reviewer 2.Source: §C.5.4 · p. 23

C.5.5 Scope of the validation

  • Validation covers two reviewers and 60 decomposed case-reasoning steps, not de novo adjudication of the full benchmark.The released grades and judge fields can support extension to more reviewers, items, or scoring revisions.Source: §C.5.5 · p. 24

C.6 Cross-family judge sensitivity audit

  • On 52 sampled items per cell, the release judge passed 32.7% of GPT-5.4 outputs versus 15.4% for each cross-family judge; clinically acceptable answers remained rare across families.For Claude Opus 4.6 and Gemini 2.5 Pro outputs, rates were closer across judges; Gemini-output pass agreement was κ = 0.898.Source: §C.6 · p. 24, Figure 10 · p. 25

C.7 Evaluation records and reproducibility

  • Released records preserve item sets, answers, scoring decisions, rubric fields, judge identity, and the six ordered case-step decisions needed to recompute reported denominators.The release also includes rendering specifications, evaluation interfaces, canonical item lists, exclusion registries, and audit records.Source: Appendix C.1 · p. 25, §C.7 · p. 25

D Additional empirical diagnostics

  • Figure 11 consolidates uncertainty estimates across headline metrics, including paired-bootstrap intervals for matched ablation samples that remain below zero.The intervals use nonparametric percentile bootstrap over raw scored records or judge-step records.Source: Appendix C.1 · p. 25, Figure 11 · p. 26

D.1 Paper-reported 3D-evidence results

  • The 3D table compares available model interfaces on shared clinical targets, not equivalent input pathways or the isolated value of one 3D representation.Nonnative-volume VLMs use 16-slice axial renderings, whereas M3D-LaMed and RadFM use supported native-volume pathways.Source: §D.1 · p. 26

D.2 Ranking sensitivity across evaluation axes

Evaluation-axis ranks vary, so the appendix cautions against reducing BrainTRACE to one aggregate score. It also calibrates level accuracy against chance floors, showing that recognition strength does not necessarily carry over to longitudinal reasoning.

D.2 Ranking sensitivity across evaluation axes

  • Model order changes across evaluation axes, which reward different aspects of evidence use and should not be collapsed into one aggregate score.The axes include closed-form recognition, open-ended acceptability, 3D evidence, Step Pass, and Case Success.Source: §D.2 · p. 26, Figure 12 · p. 27

D.3 Calibrated level difficulty

  • Calibrated difficulty score sets each level’s random baseline to 0 and perfect accuracy to 100, accounting for level-specific answer-option floors.Negative values indicate below-chance behavior.Source: §D.3 · pp. 27–28, Figure 13 · p. 28
  • GPT-5 raw L3 accuracy is 45.6% versus 32.9% on L2, while its CDS values are +14.0 and +12.2 because L3 has a higher chance floor.The calibrated comparison supports the reported interpretation that recognition-level strength does not imply reliable longitudinal reasoning.Source: §D.3 · p. 28
  • Qwen3.5-27B’s L3 CDS is -6.0 despite 33.0% raw accuracy, while its L1 CDS is +35.4, illustrating that raw accuracy can obscure below-chance performance.The calibration prevents levels with higher answer-option floors from appearing easier than they are.Source: §D.3 · p. 28

E Evidence-channel ablation details

Matched evidence-channel comparisons test whether selected closed-form items depend on serial context, non-T2 sequences, and visual inputs. The reported paired drops are negative for both audited models, but the diagnostic set is not a population-wide estimate.

E Evidence-channel ablation details

  • The comparisons hold question text, answer options, permitted non-image fields, and scoring fixed, and apply only to selected closed-form items.They are diagnostic comparisons, not population-wide performance estimates over all BrainTRACE items.Source: E Evidence-channel ablation details · p. 28

E.1 Evidence retained in each comparison

  • Removing prior studies tests dependence on serial context; removing non-T2 sequences tests multi-sequence dependence; withholding images tests residual dependence on text, priors, or permitted metadata.The T2-only condition retains the longitudinal T2 series, while latest-timepoint retains the current study.Source: §E.1 · p. 28

E.2 Finding and interpretation

  • Paired evidence-channel drops remain negative for both audited models, supporting dependence of the selected items on retained evidence rather than only question text or answer priors.Figure 11 presents paired bootstrap intervals computed within each matched comparison.Source: §E.2 · p. 29
  • Matched item sets share a 50.0% full-evidence reference accuracy, making paired drops comparable within this diagnostic design rather than population-level estimates.No-image abstentions count as incorrect under the benchmark’s closed-form exact-match rule.Source: §E.2 · p. 29

F Qualitative failure cases

The case atlas pairs success and failure examples across five clinical levels, using the same task template within each pair. It illustrates clinically material errors that can occur despite fluent responses.

L1: Image and acquisition recognition

  • At L1, the failure reverses left/right laterality despite otherwise fluent description and matched required facts.The audit treats the laterality contradiction as clinically material.Source: Figure 14 · p. 30

L2: Anatomical localization and single-study finding recognition

  • At L2, the failure omits a 52 mm left peritrigonal lesion with ill-defined borders, a false negative rather than a wording difference.Both examples use a single axial T2W input and receive the same model response.Source: Figure 15 · p. 31

L3: Disease-specific interpretation and interval change

  • At L3, the failure reports increasing size when the metastasis decreased from 13 mm to 8 mm across longitudinal multi-sequence inputs.The successful example identifies stable lesions.Source: Figure 16 · p. 32

L4: Longitudinal trajectory or quantitative change assessment

  • At L4, the failure mislocalizes a finding to the cerebellum instead of the right occipital region and reads an increasing trajectory as stable.The examples use three serial T2W studies.Source: Figure 17 · p. 33

L5: Case-level and integrated clinical judgment

  • At L5, the failure labels progressive bone metastases as resolving hemorrhage, assigning the wrong diagnostic category.The examples present three timepoints and T2W plus SWI sequences.Source: Figure 18 · p. 34

G Limitations and Responsible Use

BrainTRACE is limited to longitudinal brain MRI follow-up cases from an MR-RATE-derived cohort with sufficient reports, follow-up, and segmentation evidence. Its scores measure agreement with report-grounded references under the benchmark specification, not absolute clinical performance or a new adjudicated diagnosis.

G Limitations and Responsible Use

  • BrainTRACE covers an MR-RATE-derived cohort with English reports and sufficient follow-up, report, and segmentation evidence, not other anatomies, languages, sparse referrals, or all neuroradiology indications.Source: G Limitations and Responsible Use · p. 35
  • Scores support reproducible model comparison, but should be reported by reasoning level, attribute, and acquisition subgroup rather than treated as absolute clinical capability.Source: G Limitations and Responsible Use · p. 35
  • References come from clinical reports and segmentation-derived measurements, which can omit secondary findings, compress timepoints, express uncertainty, or carry measurement errors.The case-reasoning track receives item-level review by an ABR-certified radiation oncologist, but reports remain clinical documents rather than pixel-level annotations.Source: G Limitations and Responsible Use · p. 35
  • Open-ended scores use a GPT-family factual-audit judge, and a small cross-family audit found judge-family sensitivity for GPT-5.4 outputs.Source: G Limitations and Responsible Use · p. 35
  • The 3D evidence track compares available model interfaces, not 3D representation alone, because rendered 16-slice and native-volume pathways confound the input interface.Source: G Limitations and Responsible Use · p. 35
  • Raw imaging is not redistributed, so users must obtain upstream MR-RATE access and follow its data-use terms.Source: G Limitations and Responsible Use · p. 35
Loading 2610.06571v1…