PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Where to Look: Anatomical Grounding and Guided Attention for Cardiac MRI Vision-Language Models

Bangwei Guo, Xiao Chen, Boris Mailhe, Jia Yao, Yiqing Wang, Ankush Mukherjee, Yikang Liu, Zheyuan Zhang, Hang Yu, Terrence Chen, Shanhui Sun

arXiv:2609.39899v1cs.CV

TL;DR

CMR VLMs have limited cardiac-specific supervision and explicit anatomical grounding, despite clinicians’ need to focus on question-relevant structures. This paper builds grounding and clinical QA datasets and introduces CARA-VL, which routes predicted anatomy into decoder attention; it leads evaluated models across internal tasks, with promising external generalization.

  • Problem

    Medical VLMs have comparatively limited CMR supervision, and cardiac models still face challenges in accessible training data and explicit localisation of question-relevant anatomy.

    Source: §1 · pp. 1–2

  • Method

    The authors construct anatomical-grounding and clinical QA datasets and train CARA-VL with grounding pretraining plus question-routed anatomical priors that guide decoder attention.

    Source: §1 · p. 2

  • Results

    Across all 14 internal tasks, CARA-VL leads evaluated models; on SAX it reaches 90.7% mean BA, 22.6 percentage points above CMR-CLIP.

    Source: §4.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    The data and method provide a practical framework for studying cardiac visual understanding, with anatomical learning presented as a promising direction for specialised medical VLMs.

    Source: §5 · p. 9

    Training labels inherit assumptions and errors from mask- and measurement-based rules, require independent clinical validation, and external evaluation currently covers SAX cine only.

    Source: §5 · p. 9

Abstract

from arXiv · show

Cardiac magnetic resonance imaging (CMR) enables assessment of cardiac anatomy, ventricular function, and myocardial tissue characteristics. Clinicians interpret these images by identifying cardiac structures and focusing on the regions relevant to each clinical question, motivating anatomically guided vision-language models (VLMs). Yet CMR-specific supervision for anatomical localisation and clinical question answering remains limited. To address this gap, we investigate fine-grained CMR visual question answering through anatomical grounding and guided attention. We construct 128,915 anatomical-grounding and 42,799 clinical QA pairs across short-axis cine, late gadolinium enhancement, and long-axis cine. These datasets support anatomical recognition, localisation, and clinical assessment without requiring paired reports for individual training images. To help the model learn where to look, we introduce Cardiac Anatomy-Routed Attention (CARA), which selects predicted anatomical priors according to the question and guides decoder attention with learned task-specific strengths. Combining anatomical grounding pretraining with CARA yields our model, CARA-VL. Experiments demonstrate CARA-VL's strengths in clinical assessment and regional localisation across CMR imaging settings, with promising generalization to an external clinical cohort. Together, our data and method provide a practical framework for studying and advancing cardiac visual understanding in VLMs. We will release the QA data derived from public datasets upon publication.

1 INTRODUCTION

CMR supports assessment of cardiac anatomy, function, and tissue, but CMR-specific supervision and explicit anatomical grounding for VLMs remain limited. The paper introduces grounding data and question-routed attention for fine-grained CMR VQA, with reported gains across internal tasks.

1 INTRODUCTION

  • CMR interpretation requires identifying relevant structures before assessing morphology, motion, or tissue characteristics, and often involves specialist expertise and time-consuming quantitative analysis.Examples include wall thickness for hypertrophy, changes across phases for ventricular function, and scar presence, extent, and transmurality for myocardial injury.Source: §1 · p. 1
  • Medical VLMs have comparatively limited cardiac MRI supervision, while cardiac-specific models face challenges in accessible training data and explicit localisation guidance.The introduction links limited cardiac MRI supervision to potential poor generalization on fine-grained CMR tasks.Source: §1 · pp. 1–2
  • The work combines anatomical recognition and localisation with clinical assessment through CARA-VL, which routes predicted anatomical priors by question to guide decoder attention.The learned attention strengths are task-specific.Source: §1 · p. 2
  • The datasets contain 128,915 anatomical-grounding and 42,799 clinical QA pairs across SAX cine, LGE, and LAX cine, without requiring paired reports for each training image.Clinical tasks are informed by report analysis; grounding uses expert segmentation annotations and clinical QA uses quality-controlled image-derived measurements.Source: §1 · p. 2
  • CARA-VL outperforms evaluated baselines on all internal tasks and shows promising external generalization.Ablations support complementary benefits from anatomical grounding pretraining and CARA.Source: §1 · p. 2

2 CLINICALLY GROUNDED QA DATASET CONSTRUCTION

The dataset pipeline converts report-informed clinical needs, multi-source images, masks, and measurements into complementary grounding and clinical QA supervision across three CMR settings.

2.1 REPORT-GUIDED CLINICAL QA TASK SPECIFICATION

  • Clinical QA tasks are selected through standardised report review, finding-frequency analysis, and clinical criteria, yielding 14 tasks across SAX cine, LGE, and LAX cine.Selection considers clinical relevance, explicit measurement or grading criteria, and whether available data and tools support reliable reference answers.Source: §2.1 · pp. 2–3

2.2 IMAGE DATA COLLECTION AND STANDARDIZATION

  • Multi-source public and in-house data cover SAX cine, LGE, and LAX cine; images are resampled, cropped, and upsampled, with automatic masks manually reviewed when reference masks are unavailable.The in-house image–report cohort is reserved for external evaluation.Source: §2.2 · p. 3

2.3 STAGE-1: ANATOMICAL GROUNDING DATASET

  • Stage 1 uses expert-annotated cases and ground-truth masks to create nine grounding task types covering context, identification, localisation, and spatial reasoning.The dataset contains 128,915 QA pairs over 45,773 images.Source: §2.3 · pp. 3–4

2.4 STAGE-2: CLINICAL QA DATASET

  • Stage 2 derives clinical answers from anatomical measurements and task-specific rules, with quality-controlled masks and no mask overlays or reference measurements supplied to the model.The resulting dataset contains 42,799 QA pairs; inputs may be a single image or an ordered end-diastolic and end-systolic pair.Source: §2.4 · p. 4

3 CARA: CARDIAC ANATOMY-ROUTED ATTENTION

CARA predicts coarse cardiac-anatomy occupancy maps and routes question-relevant priors into decoder attention. It is trained alongside a two-stage grounding-to-clinical-VQA pipeline.

3 CARA: CARDIAC ANATOMY-ROUTED ATTENTION

  • CARA adds a learned task-specific strength to question-selected anatomical priors, connecting occupancy prediction with decoder attention.The two-stage training figure depicts grounding first and joint clinical answer generation and occupancy prediction second.Source: §3 · p. 4, Figure 2 · p. 5

3.1 PREDICTING ANATOMICAL OCCUPANCY

  • CARA predicts image-specific soft occupancy maps for the LV cavity, myocardium, and RV cavity on the visual-token grid, using masks only for training supervision.At inference, the maps are predicted directly from CMR images without masks as model inputs.Source: §3.1 · p. 4

3.2 ANATOMY-ROUTED ATTENTION INJECTION

  • For each predefined clinical task, CARA selects relevant anatomical evidence and injects its routed occupancy prior as a sequence-aligned bias in decoder attention.Hypertrophy tasks route to myocardium, while LV-size and EF tasks route to the LV cavity; positive, zero, and negative strengths respectively promote, preserve, or suppress attention to the routed anatomy.Source: §3.2 · p. 5

3.3 TWO-STAGE TRAINING WITH ANATOMICAL SUPERVISION

  • Training first learns grounding with the language model frozen, then adapts the VLM on clinical QA while jointly optimising occupancy prediction and task-specific attention strengths.The region loss combines weighted binary cross-entropy and soft Dice; images without valid masks contribute only to QA loss, and region loss does not update the vision encoder.Source: §3.3 · p. 6

4 EXPERIMENTS

CARA-VL leads the internal clinical-assessment and localisation tasks, and its external results vary by reference standard; ablations show gains from anatomical grounding and CARA. The experiments evaluate general medical and cardiac-specific baselines across CMR settings.

4.1 IMPLEMENTATION DETAILS

  • The final Stage-2 checkpoint is evaluated on ACDC SAX, M&Ms-2 LAX, held-out LGE, and external In-house B; general medical baselines are tested zero-shot.Training starts from Qwen2.5-VL-3B-Instruct, with three epochs per stage; Stage 2 uses LoRA.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • 90.7% mean BA and 90.0% mean macro-F1 on SAX; CARA-VL leads all 14 internal tasks and improves regional hypertrophy micro-F1 to 61.8% versus 20.3% for the strongest baseline.Its gains also extend to structural, functional, and scar assessment in LAX and LGE.Source: §4.2 · p. 7, Table 1 · p. 7
  • 79.5% / 72.4% mean BA / macro-F1 under Rule-based labels on In-house B, versus 67.5% / 49.1% for CMR-CLIP; results are more mixed under Report-based labels.CMR-CLIP leads on hypertrophy and EF under Report-based labels, and its regional hypertrophy and wall-motion micro-F1 scores exceed CARA-VL's.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 ABLATION

  • Removing Stage-1 reduces mean BA and macro-F1 by 2.3 and 2.1 points, while removing CARA reduces them by 3.9 and 3.5 points.The full model exceeds the variant without either component by 4.5 BA points and 4.0 macro-F1 points; Stage-1 provides a larger gain when CARA is enabled.Source: §4.3 · p. 8, Table 3 · p. 8

5 DISCUSSION

The study presents anatomical supervision as valuable for cardiac VLMs, linking regional grounding with clinical assessment across CMR tasks. CARA-VL performs strongly across internal tasks and shows external-cohort performance, but several limits constrain interpretation and clinical use.

5 DISCUSSION

  • Figure 3 compares attention maps with and without CARA for hypertrophy and LV-size questions on the same SAX image.Maps average attention across heads in the final eight decoder layers at the last prompt token; ROI attention is the fraction allocated to target anatomy.Source: Figure 3 · p. 9
  • CARA-VL leads across 14 internal tasks, while external evaluation shows generalization alongside remaining gaps in agreement with clinical reporting.The tasks cover three CMR modalities and assess cardiac structure, function, tissue characteristics, and regional localisation.Source: §5 · p. 9
  • Training answers derived from masks and image-based measurements inherit their procedures’ errors and assumptions and require independent clinical validation.External evaluation currently covers SAX cine only, and performance remains insufficient for clinical use.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents CARA-VL and a CMR QA pipeline built from anatomical annotations and quantitative measurements, while the surrounding sections explain the clinical background, related work, and task construction. The paper reports leading performance across internal tasks and describes the scope and boundaries of its data and evaluation.

6 CONCLUSION

  • CARA-VL combines anatomical grounding with anatomy-routed attention for fine-grained CMR question answering, using supervision across SAX cine, LGE, and LAX cine without paired reports for individual training images.It leads the evaluated models across 14 internal tasks; external evaluation shows generalization to an unseen clinical cohort alongside remaining gaps in agreement with clinical reporting.Source: §6 · pp. 9–10

A RELATED WORK

  • General medical VLMs have broad medical VQA capabilities, but their training offers limited systematic cardiac MRI supervision; CMR-specific models still lack standardized fine-grained clinical QA formulations and often rely on non-public cohorts.CMR assessment depends on cardiac anatomy, cine dynamics, physical-scale measurements, and sequence-specific pathology.Source: A RELATED WORK · p. 14

B CLINICAL BACKGROUND ON CARDIAC MRI

  • CMR interpretation uses ventricular cavities and myocardium to assess structure, function, and tissue; SAX, LAX, and LGE provide complementary imaging views.SAX resolves ventricular cross-sections and cardiac-cycle changes; LAX shows chamber geometry; LGE delineates hyperenhanced tissue, including scar.Source: B CLINICAL BACKGROUND ON CARDIAC MRI · p. 14, §B.1 · pp. 14–15

B.2 CARDIAC STRUCTURE, FUNCTION, AND TISSUE CHARACTERISTICS

  • The background distinguishes wall thickening from cavity enlargement and describes cardiac function through ventricular volumes, ejection fraction, and regional wall motion.EF expresses stroke volume as a percentage of EDV; strain describes deformation, while scar transmurality and circumferential extent describe different dimensions of involvement.Source: §B.2 · pp. 15–16

C DATA CONSTRUCTION DETAILS

  • The data pipeline builds Stage-1 grounding answers from expert masks and Stage-2 clinical answers from physical measurements on reference or automatically generated masks.The sources, mask provenance, and stage roles are listed in Table 5.Source: C DATA CONSTRUCTION DETAILS · p. 16

C.1 FROM FREE-TEXT REPORTS TO A CLOSED QUESTION SET

  • The clinical task set is derived from a structured audit of 3,215 reports, with fields classified as unmentioned, explicitly normal, or abnormal and candidate questions selected for clinical relevance and measurable criteria.The 14 tasks exclude findings unsupported by available measurements or data variation; task-specific grading criteria use image-based measurements, with borderline cases excluded for selected tasks.Source: §C.1 · pp. 16–18, Table 4 · p. 18

C.2 DATA SOURCES

The data collection combines public and in-house CMR sources across SAX cine, LGE, and LAX cine, with separate grounding and clinical QA construction. Expert masks and image-derived measurements support these tasks without requiring paired reports for each training image.

C.2 DATA SOURCES

  • In-house B is reserved for external evaluation, while the corpus draws on public datasets and distinct in-house image-only and image–report collections.The sources cover SAX cine, LGE, and LAX cine.Source: §C.2 · pp. 18–19

C.3 PHYSICAL-SCALE RENDERING

  • Images are resampled by sequence, intensity-clipped, cropped to 224 × 224, and upsampled to 448 × 448; all compared models receive the same preprocessing.Cases with unresolved spacing inconsistencies or truncated anatomy are excluded.Source: §C.3 · p. 19

C.4 ANATOMICAL GROUNDING SUPERVISION

  • 128,915 grounding QA pairs span SAX, LAX, and LGE, using expert masks for nine tasks covering image context, anatomical recognition, localisation, and spatial relations.The dataset contains 77,482 SAX, 35,040 LAX, and 16,393 LGE pairs.Source: §C.4 · pp. 19–20

C.5.1 SEGMENTATION AND QUALITY CONTROL

  • Mask harmonisation maps labels to LV cavity, myocardium, and RV cavity, while SAX screening checks gross failures but not fine-grained boundary accuracy.Missing annotations are treated as unavailable, not as evidence that a structure is absent.Source: §C.5.1 · pp. 21–22

C.5 CLINICAL QA CONSTRUCTION

  • Clinical QA pairs use task-specific criteria to convert reviewed-mask measurements, metadata, reference values, or tracking outputs into answers paired with eligible images.Stack-level questions use stack measurements, while slice-specific questions use displayed-slice measurements.Source: §C.5 · p. 21, §C.5.1 · p. 22

C.5.2 SHORT-AXIS CINE REFERENCE CONSTRUCTION

  • SAX references derive hypertrophy, chamber size, ejection fraction, RV dilation, strain, and regional wall motion from masks, tracking, or stack-level measurements.Hypertrophy is assigned at a segment mean thickness of at least 13 mm; regional motion is reduced below 20% mean systolic thickening.Source: §C.5.2 · pp. 22–23

C.5.3 LGE REFERENCE CONSTRUCTION

  • LGE references use expert scar masks for presence, derive regional transmurality from scar-to-wall ratios, and classify circumferential extent from sampled rays.Automatically labelled extent cases with c in [0.40, 0.60) are excluded, while manual-review decisions can retain them.Source: §C.5.3 · pp. 23–24

C.5.4 LONG-AXIS CINE REFERENCE CONSTRUCTION

  • LAX EF and EDV labels are drawn from same-patient SAX grades or volumes, while GLS is computed from full-sequence tracking and paired with ED/ES images.Intermediate, missing, and sign-inconsistent positive GLS values are excluded.Source: §C.5.4 · p. 24

C.5.5 QA ASSEMBLY AND DATASET SUMMARY

The assembled clinical training set contains 42,799 QA pairs across 14 tasks, with task-specific image inputs and reference answers. The figures provide examples of the tasks and answer formats.

C.5.5 QA ASSEMBLY AND DATASET SUMMARY

  • 42,799 clinical QA pairs cover 14 tasks: 29,114 SAX, 5,645 LGE, and 8,040 LAX pairs.Task counts vary with annotation availability, measurement validity, and quality filtering.Source: §C.5.5 · p. 24
  • Task-specific answers are paired with one ED image, ED/ES pairs, or one LGE image, and masks support label construction rather than appearing as model overlays.Hypertrophy uses one ED image; other SAX and all LAX tasks use ED/ES pairs; LGE uses one image.Source: §C.5.5 · p. 24
  • Figures 8 and 9 show unmarked inputs and verbatim training QA for all 14 tasks, with normal and severe examples for four-grade tasks.The examples span ACDC SAX, MyoPS LGE, and M&Ms-2 LAX data.Source: §C.5.5 · p. 26, Figure 8 · p. 26, Figure 9 · p. 27

C.6 SPLITS AND EVALUATION SETS

Evaluation uses patient-level partitions and an external cohort with rule-based and report-based reference labels where available. Those reference standards differ in thresholds, coverage, and measurement sources.

C.6 SPLITS AND EVALUATION SETS

  • Internal evaluation comprises 3,220 QA pairs across ACDC SAX, held-out LGE patients, and M&Ms-2 LAX, with patient identities shared across views and reconstructions for splitting.In-house B is reserved exclusively for external evaluation.Source: §C.6 · p. 27
  • In-house B external evaluation includes 612 examinations from 461 patients and 14,336 SAX QA pairs, with rule-based and report-based labels where available.Repeated examinations are retained, and evaluation is performed per QA instance.Source: §C.6 · p. 27
  • External report and rule-based references differ in thresholds and anatomical coverage, and report-based GRS/GCS use wall-motion-score proxies rather than direct strain references.Table 9 compares reference labels, not model scores; disagreement reflects measurement sources, coverage, and thresholds.Source: §C.6 · pp. 28–29, Table 9 · p. 28
  • Report-derived evaluation requires documented, unambiguous task-level answers, while paired-reference analysis additionally requires comparable entries from both sources.Regional paired comparisons count matched regions or segments rather than QA instances.Source: §C.6 · p. 29

D.1 MODEL ARCHITECTURE

The architecture predicts anatomical occupancy maps and routes question-specific priors into decoder attention using learned task strengths.

D.1 MODEL ARCHITECTURE

  • A shared occupancy head maps post-merger image features to LV-cavity, myocardial, and RV-cavity occupancy maps.The head uses 1 × 1 and 3 × 3 convolutions while preserving the spatial grid.Source: §D.1 · p. 29
  • For each clinical task, a peak-normalised anatomical prior is scaled by an independent learned strength and added at visual-key positions across decoder attention.The attention injection preserves the original causal and padding masks.Source: §D.1 · p. 29, Table 10 · p. 29

D.2 TRAINING LOSSES

Stage 2 combines answer generation with anatomical occupancy supervision, while the QA loss also trains the attention-routing parameters. The regional loss is restricted to the occupancy head and does not update the visual encoder.

D.2 TRAINING LOSSES

  • Stage 2 optimises QA loss plus 0.5 times regional loss, with the regional loss comprising weighted binary cross-entropy and soft Dice.The QA loss is next-token cross-entropy averaged over answer tokens, excluding prompt and padding tokens.Source: §D.2 · p. 29
  • Reference masks are average-pooled onto the visual-token grid to provide soft occupancy targets for LV, MYO, and RV.LGE supervision uses LV and MYO targets only; images without valid anatomical targets contribute only to QA loss.Source: §D.2 · pp. 29–30
  • Weighted binary cross-entropy uses positive-class weights of 15, 40, and 15 for LV, MYO, and RV, respectively.The loss is evaluated from logits for numerical stability.Source: §D.2 · p. 30
  • Soft Dice is computed across the local batch for each structure and averaged over valid structures, with smoothing constant one.If no valid annotation is available, the regional loss is zero.Source: §D.2 · p. 30
  • Detached features make the regional loss update the occupancy head without affecting the visual encoder, while QA loss updates the head and attention strengths.The QA loss reaches the attention strengths through the differentiable attention bias.Source: §D.2 · p. 30

D.3 TRAINING PROTOCOL

The training protocol oversamples a small task subset within a shuffled pool. The evaluation metrics define classification, regional-localisation, and attention measures, with task-specific averaging conventions.

D.3 TRAINING PROTOCOL

  • Stage 2 oversamples 203 non-transmural examples to 1,249 records, expanding the pool from 42,799 QA pairs to 43,845 training records.No other task is oversampled; records receive unit loss weight, and task frequencies follow the resulting pool composition.Source: §D.3 · p. 30

D.4 EVALUATION METRICS

  • Scores are computed per QA instance and reported as percentages; balanced accuracy and macro-F1 average over classes.For binary tasks, sensitivity and specificity are positive- and negative-class recall, respectively.Source: §D.4 · p. 30
  • SAX averages equally weight six global classification tasks and exclude regional tasks; external averages are computed separately for each reference.LV size, EF, and EDV are folded into normal versus abnormal for these averages, while GRS and GCS remain separate tasks despite shared report-based wall-motion proxies.Source: §D.4 · p. 30
  • Regional localisation uses pooled counts for micro-F1, alongside exact match and mean Jaccard comparisons of affected-region sets.Exact match requires equal sets; mean Jaccard assigns one when both sets are empty.Source: §D.4 · pp. 30–31
  • ROI-attention evaluation pairs each visual token’s attention weight with its reference occupancy for the target anatomy.The visual-token index set is denoted Ivisual.Source: §D.4 · p. 31

E ADDITIONAL EXPERIMENTAL RESULTS

The additional-results section identifies supplementary four-grade, binary, and regional evaluations, and clarifies scoring conventions and reference coverage. It does not provide numerical outcomes in the supplied passages.

E ADDITIONAL EXPERIMENTAL RESULTS

  • Supplementary tables cover four-grade LV size, EF, and EDV classification, additional binary classification, and regional hypertrophy and wall-motion localisation.The tables include internal test sets and In-house B results under two reference standards.Source: E ADDITIONAL EXPERIMENTAL RESULTS · p. 31, Table 11 · p. 31, Table 12 · p. 32
  • Four-grade BA is macro recall and macro-F1 weights all four classes equally; internal SAX scores retain archived reporting precision.The remaining scores are recomputed from corresponding saved predictions.Source: E ADDITIONAL EXPERIMENTAL RESULTS · p. 31
  • The external paired-reference analysis scores the same predictions against task-specific coverage and excludes missing or indeterminate reference fields.Report-based GRS/GCS labels are wall-motion-score proxies, not measured strain.Source: E ADDITIONAL EXPERIMENTAL RESULTS · p. 32
  • Regional measures compare affected-region sets within scored coverage, so they are localisation metrics rather than pixel-level segmentation scores.No regional metric is included in classification averages.Source: E ADDITIONAL EXPERIMENTAL RESULTS · p. 33

Related papers

Loading 2609.39899v1…