PaperlayerView exact revision on arXiv

Source-linked AI summary

CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment

Kunyang Li, Hai Nguyen, Joshua Lowe, Chenguang Zhao, Peace C. Madueme, Mehdi Hedjazi Moghari, Mubarak Shah, Pegah Khosravi, Yuzhang Zhang

arXiv:2610.01166v1cs.CVcs.AI

TL;DR

Current medical VLMs cannot reliably derive quantitative measurements from multidimensional cine CMR images alone, limiting their use for assessments that combine visual and measured evidence. CineMR integrates cardiac analysis tools into interleaved VLM reasoning and achieves higher benchmark performance than the unfine-tuned backbone. Its findings support tool-augmented VLMs as a promising approach for assistive cardiac image assessment.

  • Problem

    Current medical VLMs cannot be assumed to reproduce the multistep quantitative CMR analysis pipeline reliably from image inputs alone.

    Source: §1 · p. 2

  • Method

    CineMR combines an initial imaging hypothesis, executable cardiac analysis tools, and criterion-based interpretation of returned measurements in an interleaved VLM process.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    35.9% pass@1 and 58.9% pass@4 overall, versus 1.5% pass@1 for Qwen3-VL-8B, on the multi-cohort cine CMR benchmark.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The findings support tool-augmented VLMs as a promising approach for assistive cardiac image assessment.

    Source: §1 · p. 4, §6 · p. 11

    This retrospective public-dataset study primarily evaluates short-axis cine CMR, and some benchmark categories use study-specific rules rather than complete clinical diagnoses.

    Source: C Limitations and Broader Impact · p. 18

Abstract

from arXiv · show

Cardiovascular magnetic resonance (CMR), including cine imaging, is a reference standard for the noninvasive assessment of cardiac morphology and ventricular function. Cine CMR interpretation integrates qualitative visual assessment with quantitative measurements of ventricular volumes, ejection fraction, myocardial mass, wall thickness, and regional wall motion. Current medical vision-language models (VLMs) cannot reliably derive quantitative measurements from multidimensional cine images without analysis tools. We present CineMR, a tool-augmented VLM that invokes cardiac image-analysis tools and integrates their outputs into interleaved reasoning for quantitative CMR assessment. We also construct a multi-cohort visual question answering benchmark covering quantitative metric extraction, multiclass diagnosis, and differential diagnosis, together with tools for segmentation, phase selection, volumetry, morphometry, and regional wall motion analysis. CineMR is trained with supervised fine-tuning (SFT) on tool-interaction traces followed by Group Relative Policy Optimization (GRPO) with conditional tool-use rewards. On the multi-cohort cine CMR benchmark, CineMR achieves 35.9% pass@1 and 58.9% pass@4, compared with 1.5% pass@1 for the Qwen3-VL-8B backbone and 0.0% and 7.0% pass@1 for LLaVA-Med v1.5 and MedGemma-4B, respectively. Correct tool invocation reaches 99.8% after GRPO, up from 78.9% after SFT. Live tool outputs improve ventricular measurement accuracy by 20.4--23.7% over direct model predictions, and removing all tools reduces pass@1 from 35.9% to 27.9%. These results highlight the importance of reliable tool use for quantitative cine CMR reasoning and support CineMR as a promising approach for assistive cardiac image assessment. Code, benchmark resources, and model weights are available at https://github.com/AI-MIND-Lab/CineMR.

1 Introduction

Cine CMR requires visual interpretation alongside quantitative measurements, but current VLMs cannot be assumed to derive those measurements reliably from images alone. CineMR integrates cardiac analysis tools with criterion-based reasoning and outperforms the unfine-tuned backbone on a multi-cohort benchmark.

1 Introduction

  • Current medical VLMs cannot be assumed to derive quantitative CMR measurements reliably from images alone, despite the multistep image analysis and criterion-based interpretation required.CMR interpretation combines visual assessment with measurements, while definitive diagnosis may also require clinical and complementary imaging information.Source: §1 · pp. 1–2
  • CineMR equips a VLM with cardiac image-analysis tools and trains it to integrate explicit measurements into interleaved reasoning.The tools cover segmentation, volumetry, morphometry, and regional wall motion, with measurements used at quantitative branches of the phenotype classifier.Source: §1 · pp. 2–3
  • The benchmark organizes questions around a study-specific Rule-Based Cardiac Phenotype Classifier that links quantitative measurements to reference answers and expected tool use.It covers ten harmonized outcome categories and assigns each question a complexity level and expected tools.Source: §1 · p. 2
  • Training uses supervised fine-tuning on tool-interaction traces followed by GRPO with conditional rewards for tool invocation only when the final answer is correct.The reward is intended to encourage beneficial tool use while reducing unnecessary calls.Source: §1 · pp. 3–4
  • 35.9% pass@1 and 58.9% pass@4 overall, versus 1.5% pass@1 for Qwen3-VL-8B, on the multi-cohort cine CMR benchmark.The benchmark spans ten harmonized outcome categories and six task-complexity levels.Source: §1 · p. 4

2 Related Work

Prior work includes medical agents with task-specific tools and deep-learning methods for cardiac MRI analysis. CineMR combines executable quantitative CMR tools with interleaved VLM reasoning for criterion-based phenotype assessment.

Tool-Augmented Medical Agents

  • Medical tool-using agents cover operations such as segmentation, classification, and annotation, while some organize tools into diagnostic or clinical risk-assessment workflows.Examples include MMedAgent, VILA-M3, AURA, MedAgent-Pro, and AgentMD.Source: §2 · p. 4
  • Many existing agents emphasize perceptual operations rather than tools that return explicit quantitative imaging measurements, and several rely primarily on prompting or supervised fine-tuning for tool selection.CineMR adds quantitative tools and a study-specific classifier-structured training dataset.Source: §2 · p. 5

AI for Cardiac MRI Analysis

  • Prior cardiac MRI work covers segmentation, quantitative characterization, and radiomics, but had not combined executable quantitative cine CMR tools with interleaved VLM generation for criterion-based phenotype assessment.CineMR generates an initial imaging hypothesis, invokes quantitative tools, and integrates their measurements using criteria encoded in a study-specific classifier.Source: §2 · p. 5

3 Method

CineMR frames cine CMR assessment as interleaved reasoning and quantitative tool use, with benchmark decisions tied to explicit cardiac phenotype criteria. Its method combines a structured classifier dataset with tools for ventricular measurement and regional wall-motion analysis.

3.1 Problem Formulation

  • CineMR treats each question as an agentic sequence of reasoning or tool calls, runtime tool outputs, and a final numerical, binary, or categorical answer.Source: §3.1 · p. 5

3.2 Rule-Based Cardiac Phenotype Classifier-Structured Dataset

  • The benchmark encodes published and study-specific quantitative criteria in deterministic decision rules, then reconciles each candidate diagnosis against the independently confirmed cohort label.Source: §3.2 · pp. 5–6
  • The classifier covers ten harmonized categories using published criteria or documented project-specific thresholds, including cohort-specific MVR adjustment for ACDC but not M&Ms or M&Ms-2.Source: §3.2 · p. 6, Figure 4 · p. 6
  • The 19 question templates span six levels, from numeric extraction and criterion evaluation to multicategory prediction and targeted differential diagnosis, with increasing selective and compositional tool use at L3–L6.Source: §3.2 · pp. 6–7
  • 43 of 66 classifier thresholds were grounded in guidelines or peer-reviewed studies; the remaining 23 were project-specific thresholds with documented derivations and published precedents.Source: §3.2 · p. 7

3.3 Cardiology-specific Quantitative Tool Suite

  • The tool chain segments LV, RV, and myocardium, selects ED and ES phases, and computes ventricular volumes and ejection fractions; indexed volumes are reported only when body surface area is available.Source: §3.3 · pp. 7–8
  • Morphometric analysis derives myocardial mass, MVR, maximal EDWT, septal-to-lateral wall-thickness ratio, and sphericity index for classifier branches.Source: §3.3 · p. 8
  • Regional wall-motion analysis classifies AHA segments as normal, hypokinetic, or akinetic; cine wall motion alone does not establish etiology or myocardial scar.Source: §3.3 · p. 8

4 Training Pipeline

CineMR uses supervised fine-tuning followed by agentic GRPO to train tool-interaction behavior. Its reward combines answer quality and format with conditional tool-use and live-execution rewards, while penalizing repetition.

Stage 1: Cold-start SFT

  • SFT trains Qwen3-VL-8B-Instruct on reference tool-use traces for tool-call syntax, interleaved reasoning, and question-to-tool associations.Source: §4 · p. 8
  • SFT rollouts are added as training traces only when they produce a correct boxed answer, meet a tool-protocol threshold, and, for classification, remain self-consistent across repeated samples.Source: §4 · p. 8

Stage 2: Agentic GRPO

  • GRPO samples response groups and optimizes the policy using group-normalized advantages, with a KL-divergence penalty toward the frozen Stage-1 SFT reference policy.Source: §4 · p. 8
  • The composite reward scores answer accuracy, format, successful live tool execution, and repetition; tool bonuses depend on answer correctness, and reward weights were not sensitivity-tested.Source: §4 · p. 9

5 Experiments

CineMR is evaluated on a multi-cohort cine CMR benchmark using held-out test performance and live tool execution. It outperforms the compared baselines overall, with gains concentrated in criterion-based quantitative assessment; ablations examine tool selection, measurement integration, and tool access.

5.1 Setup

  • The benchmark contains 11,531 question-answer pairs across three cohorts and ten outcome categories, with patient-level partitions preventing patient data from crossing splits.It includes 7,020 training, 1,191 validation, and 3,320 test items; the test split is reserved for final evaluation.Source: §5.1 · p. 9
  • Responses are scored against deterministic reference answers, using item-level tolerances for numerical answers and normalized exact matching for categorical and binary answers.With four rollouts, pass@1 averages the fraction correct per item, while pass@4 measures whether at least one rollout is correct.Source: §5.1 · p. 9
  • SFT and GRPO are compared with four rollouts per item at temperature 0.7, and training uses tool-integrated traces followed by tool-integrated GRPO.The SFT setup uses LoRA; the reported implementation specifies separate LoRA settings and fixed heuristic reward weights for GRPO.Source: §5.1 · p. 9

5.2 Results

  • 35.9% overall pass@1 for GRPO exceeds 23.6% for cold-start SFT and 1.5% for Qwen3-VL-8B; GRPO also leads across all three cohorts.The matched SFT and GRPO models use the same live-tool protocol; Table 2 reports held-out cohort results.Source: §5.2 · p. 10, Table 2 · p. 10
  • 58.7% pass@1 on L3–L4 criterion-based tasks for GRPO exceeds 54.5% for SFT and at most 3.0% for either off-the-shelf baseline.On L5–L6 full-diagnosis tasks, GRPO reaches 18.6% versus 18.3% for SFT; the paper notes that upstream errors can propagate through multi-step decisions.Source: §5.3 · p. 10, Table 3 · p. 10

5.3 Ablations

  • GRPO raises the exact-match correct tool-call rate from 78.9% after SFT to 99.8%; live measure_volume outputs are more accurate than final boxed answers across the four volume and EF question types.The live tool outputs are correct on 28–34% of questions versus 4–11% for final boxed answers; Figure 5 summarizes tool-use diagnostics.Source: §5.3 · p. 10, Figure 5 · p. 11
  • Removing all tools lowers GRPO pass@1 from 35.9% to 27.9%, while withholding only the three metric-producing tools lowers it to 34.8%.The confidence intervals for full-tool and no-tool settings do not overlap, and the paper reports that externally computed image-derived evidence adds information beyond visual reasoning alone.Source: §5.3 · p. 11

6 Conclusion

CineMR combines vision-language interpretation with cardiac image-analysis tools and a rule-based classifier for quantitative cine CMR assessment. Its benchmark results support the potential of this approach, but the study-specific categories are not complete clinical diagnoses and the framework is not validated for autonomous clinical use.

6 Conclusion

  • CineMR combines segmentation, phase selection, volumetry, morphometry, and regional wall-motion tools with a Rule-Based Cardiac Phenotype Classifier.The classifier maps the resulting evidence to benchmark categories.Source: §6 · p. 11
  • On the constructed multi-cohort benchmark, supervised fine-tuning and reinforcement learning improve performance over the unfine-tuned backbone, with a further, more modest improvement from quantitative tools.Source: §6 · p. 11
  • The benchmark categories and study-specific rules are not complete clinical diagnoses, and CineMR has not been validated for autonomous clinical use.The authors call for external evaluation of generalizability, measurement robustness, calibration, and clinical utility.Source: §6 · p. 11

A Acronym Glossary

The glossary table is presented as a single reference for abbreviations used in the main text and appendix.

A Acronym Glossary

  • Table 4 collects abbreviations used in the main text and appendix as a single point of reference.The passage says each abbreviation is also spelled out at first use in the running text.Source: A Acronym Glossary · p. 16, Table 4 · p. 16

B Evaluation Question Templates

The evaluation templates cover 19 canonical questions mapped to cognitive layers and tool requirements. Released data also includes paraphrase variants of the canonical questions.

B Evaluation Question Templates

  • Each displayed question is the canonical form, while released data adds two LLM-generated paraphrase variants per base question.The variants provide lexical diversity through rephrasing, context enrichment, and specificity variation.Source: B Evaluation Question Templates · p. 16
  • The 19 evaluation templates are mapped to cognitive layers, tool requirements, and the tools invoked when tools are used.The table marks tool use as required or optional, and a template can override its layer’s default.Source: B Evaluation Question Templates · p. 16, Table 5 · p. 17

C Limitations and Broader Impact

CineMR’s quantitative classifications depend on segmentation-derived measurements, while broader use is bounded by retrospective short-axis evaluation and the need for validation and clinician oversight.

C Limitations and Broader Impact

  • Segmentation errors can propagate into final categories; left ventricular under- or over-segmentation frequently misdirected the classifier in L5 full-assessment tasks.This could occur even when subsequent reasoning was otherwise consistent with the extracted measurements.Source: C Limitations and Broader Impact · p. 18
  • Future segmentation-free approaches may reduce segmentation-related error propagation, but their accuracy, robustness, and generalizability across acquisition protocols and scanner vendors require dedicated validation.The authors frame this as a possible direction as multimodal foundation models and end-to-end learning advance.Source: C Limitations and Broader Impact · p. 18
  • CineMR is not intended for autonomous clinical use; outputs should be reviewed by qualified clinicians, and overreliance without validation, quality control, and safeguards could contribute to assessment errors.The stated risk is especially relevant when upstream segmentation is inaccurate.Source: C Limitations and Broader Impact · p. 18
  • The retrospective study uses public datasets and primarily short-axis cine CMR, and some study-specific benchmark categories are not complete clinical diagnoses.The authors call for prospective external validation across institutions, scanner vendors, and patient populations, including assessment of calibration, subgroup performance, and robustness.Source: C Limitations and Broader Impact · p. 18

Related papers

Loading 2610.01166v1…