PaperlayerView exact revision on arXiv

Source-linked AI summary

From Image Interpretation to Clinical Reasoning: Upstream Physician-Context-Aware Multimodal Learning with Causal Reinforcement Learning

Jialu Pi, Yanan Ma, Weijie Chen, Owen Crystal, Shubham Trivedi, Stephen Xie, Anna Silverman, Matthew Stib, Chadi Ayoub, Reza Arsanjani, Imon Banerjee

arXiv:2609.38924v1cs.CV

TL;DR

Opportunistic MACE prediction could use routine CXRs and physician-authored histories, but medical vision-language models are primarily optimized for radiology interpretation rather than prognostic reasoning. This paper combines these inputs with role-decoupled multimodal reasoning and causal reinforcement learning. Its reported results include stronger multimodal performance than image-only and text-only baselines, improved reasoning quality, and gains from causal RL on selected measures.

  • Problem

    Medical models rely on structured variables or dedicated cardiac imaging biomarkers, while medical vision-language models are primarily optimized for radiology interpretation rather than prognostic reasoning.

    Source: §I · p. 1

  • Method

    The framework combines CXRs and physician-authored histories with separate reasoning and prediction language models, a causal reinforcement learning policy, and causal token pruning.

    Source: §I · p. 2

  • Results

    Multimodal models outperformed image-only and text-only baselines, and causal RL raised Mayo ED AUC from 0.804 to 0.843 while raising MIMIC Green Score from 0.715 to 0.776.

    Source: B. Ablation Study · p. 8, §VI · p. 9

  • Takeaways & Limitations

    The framework achieved higher GREEN scores and stronger expert preference across internal and external cohorts while maintaining competitive discrimination.

    Source: §VI · p. 9

    The study is retrospective; prospective evaluation is needed to determine effects on clinician trust, workflow efficiency, or patient outcomes, and broader healthcare-system and imaging-protocol validation is needed.

    Source: §VI · p. 10

Abstract

from arXiv · show

Major adverse cardiovascular events (MACE) remain the leading cause of mortality worldwide. Opportunistic screening using routinely acquired clinical data offers a scalable approach for identifying high-risk individuals before acute events occur. Although chest X-rays (CXRs) capture latent cardiovascular biomarkers and clinical histories provide complementary patient context, existing medical vision-language models are primarily optimized for radiology interpretation rather than prognostic reasoning. We propose a causal reinforcement learning framework for multimodal clinical reasoning that integrates CXRs and physician-authored clinical histories for opportunistic MACE prediction. The framework introduces (1) a role-decoupled dual-LLM architecture that separates reasoning from risk prediction, (2) a dual-action causal reinforcement learning policy for evidence selection and reasoning optimization, and (3) causal token pruning to learn compact multimodal representations. Evaluated on an internal cohort, an emergency department cohort, and the external MIMIC dataset, the proposed framework consistently outperformed unimodal baselines and state-of-the-art medical vision-language models, achieving AUROCs of 0.720, 0.760, and 0.845, respectively. It also substantially improved reasoning quality, achieving higher GREEN scores and higher expert preference while maintaining robust predictive performance across diverse patient populations.

I. INTRODUCTION

The introduction motivates opportunistic MACE prediction from routine CXRs and physician-authored histories, while identifying a gap between image-interpretation models and clinically grounded prognostic reasoning. It proposes causal reinforcement learning to align multimodal reasoning with clinical prediction objectives.

I. INTRODUCTION

  • Existing MACE models rely mainly on structured variables or dedicated cardiac imaging biomarkers, limiting their use of multimodal information already available in routine care.Medical vision-language models are largely trained for image interpretation and report generation, and may not capture relationships between presentation, imaging, and later cardiovascular outcomes.Source: §I · p. 1
  • CXRs capture structural cardiopulmonary abnormalities linked to future cardiovascular events, while ordering-physician notes add symptoms, history, risk factors, medications, and diagnostic intent.Together, these modalities more closely reflect the information clinicians synthesize during diagnostic reasoning.Source: §I · p. 1
  • The paper proposes integrating CXRs and physician-authored clinical histories with causal reinforcement learning for multimodal clinical reasoning.Its innovations include separate reasoning and prediction language models, a dual-action policy for token selection and reasoning optimization, and causal token pruning for compact representations.Source: §I · pp. 1–2

II. BACKGROUND

  • Clinical reasoning must connect heterogeneous, incomplete evidence to outcomes, while notes vary in style and may omit information or leave reasoning implicit.Existing preference-based foundation-model reinforcement learning has largely targeted other domains rather than clinical reasoning over multimodal, temporal, and causal evidence.Source: §II · p. 2
  • Image-report alignment and plausible text generation do not by themselves optimize whether generated reasoning supports accurate clinical predictions or meaningful decisions.Foundation-model-generated rewards can be noisy and high-variance, despite preference-based methods using models as trajectory evaluators.Source: §II · p. 2

III. METHODS

The framework combines CXR images and physician-note context to generate clinical reasoning and predict MACE. It separates reasoning generation from downstream risk classification and outcome-phrase generation.

A. Overall Framework

  • The framework aligns multimodal representations with clinical decision objectives and uses one language model for reasoning and another for risk classification and outcome-phrase generation.The reasoning model integrates CXR images and clinical information to generate risk summaries; the second model performs downstream prediction and generation.Source: A. Overall Framework · p. 3
  • Each study uses a CXR and associated clinical history and demographic information summarized from the ordering physician note.The objectives include estimating MACE probability, generating multimodal reasoning, and producing an outcome description consistent with predicted risk.Source: A. Overall Framework · p. 3

B. Role-Decoupled Dual-LM Framework (Stage-I Supervised Training)

  • A role-decoupled architecture assigns multimodal risk reasoning to LLM1 and risk classification plus outcome-phrase generation to BioGPT, which shares its backbone across those downstream tasks.The separation is designed to reduce competing optimization objectives between reasoning and prediction.Source: B. Role-Decoupled Dual-LM Framework… · p. 3
  • Because discrete generated reasoning cannot pass classifier gradients back to the input, a trainable adapter also projects the VLM's last-valid-token representation into the downstream model's hidden space.The downstream model receives both the generated reasoning and this direct multimodal representation.Source: 1) Multimodal Reasoning Model · p. 3

1) Multimodal Reasoning Model:

The supervised stage combines classification and outcome generation, while the next stage intervenes on model inputs to train reasoning for relevant evidence and prediction objectives. The intervention policy acts on the model's information pathway rather than clinical causal relationships.

1) Multimodal Reasoning Model:

  • The downstream classifier fuses the reasoning representation with a projected multimodal feature and produces class probabilities, with the positive-class probability interpreted as predicted MACE risk.Classification uses class-weighted cross-entropy to account for class imbalance.Source: 2) Risk Classification and Event… · p. 3

2) Risk Classification and Event Generation:

  • BioGPT appends an outcome phrase after reasoning, with its target determined by the classifier's predicted class and the ground-truth label.A false-positive prediction uses an event combination sampled from the training-set empirical distribution.Source: 2) Risk Classification and Event… · pp. 3–4
  • Stage-I training combines classification cross-entropy with a generation loss computed on target outcome tokens, while masking the prefix tokens.The relative contributions are controlled by separate classification and generation weights.Source: 2) Risk Classification and Event… · p. 4

C. Intervention-Guided Causal Reinforcement Learning (Stage-II Causal RL Training)

  • Stage-II learns an intervention policy that removes selected input tokens and controls reasoning length, using downstream classification and generation performance as reinforcement signals.The policy state includes fused predecoder multimodal representations and token metadata; “causal” refers to interventions on the model's information pathway, not causal inference about clinical exposures or outcomes.Source: C. Intervention-Guided Causal… · p. 4, 1) Quantized Causal Policy State · p. 4

1) Quantized Causal Policy State:

The causal policy represents each multimodal token in its study context, quantizes these states into reusable discrete contexts, and uses the resulting policy features to choose evidence retention and reasoning length.

1) Quantized Causal Policy State:

  • Each token's policy state concatenates its embedding with the mean embedding of the full multimodal input, so token evaluation reflects study-wide context.The mean embedding supplies a global summary rather than evaluating each token in isolation.Source: 1) Quantized Causal Policy State · p. 4
  • An encoder maps token-context states to latents assigned to codebook prototypes, grouping continuous states into a finite set of reusable policy contexts.The codebook uses exponential moving averages, and a commitment term encourages encoder outputs to stay close to selected codes.Source: 1) Quantized Causal Policy State · p. 4
  • Decoded policy features condition a joint action that samples a binary token-retention mask and a reasoning-length bin.Token retention is sampled from a Bernoulli distribution and reasoning length from a categorical distribution.Source: 2) Policy Actions and Causal… · p. 4, 1) Quantized Causal Policy State · p. 4

2) Policy Actions and Causal Intervention:

The policy chooses which multimodal tokens to retain and how much reasoning to generate. Rollout rewards combine prediction error, token retention, and outcome-generation loss, with the ground-truth outcome determining the RL target.

2) Policy Actions and Causal Intervention:

  • The policy uses token-retention decisions and a selected reasoning-length bin to control evidence and the maximum generation budget.Constraint rules retain system and task tokens, preserve minimum fractions of image and clinical tokens, and keep clinical subwords as complete words.Source: 2) Policy Actions and Causal… · pp. 4–5

3) Rollout Reward:

  • Each rollout’s reasoning and unchanged study-level multimodal feature feed BioGPT, while the intervention changes the VLM reasoning pathway rather than the direct classifier pathway.The classifier outputs the MACE-positive probability for the rollout.Source: 3) Rollout Reward · p. 5
  • The reward uses prediction error, retained-token proportion, and ground-truth outcome-generation loss, favoring interventions that retain useful evidence while removing unnecessary tokens.The outcome target is based on the ground-truth MACE label and observed event combination, not the classifier-derived supervised-training gate.Source: 3) Rollout Reward · p. 5

4) Group-Relative Policy Optimization:

Group-relative optimization compares intervention rewards within each study and applies a clipped loss to the joint policy actions. The policy objective also includes compactness and exploration terms, while drift is controlled without a reference-language-model KL penalty.

4) Group-Relative Policy Optimization:

  • The patient-specific baseline uses each study’s rollout-reward mean and standard deviation, avoiding a separately trained value model.Standard GRPO compares each rollout with the other interventions sampled for that study.Source: 4) Group-Relative Policy Optimization · p. 5
  • The joint action combines the constrained token-retention mask and selected length bin, and its clipped GRPO loss optimizes the causal policy.The joint action is sampled from the policy conditioned on decoded policy features.Source: 4) Group-Relative Policy Optimization · p. 5
  • Policy drift is controlled through clipped importance ratios, entropy and sparsity regularization, and hard evidence-retention constraints rather than an explicit reference-model KL penalty.The objective also includes commitment loss and a keep-probability term; sparsity discourages excessive token retention and entropy supports exploration.Source: 4) Group-Relative Policy Optimization · p. 5

A. Datasets

The study evaluates paired chest radiographs and clinical notes across internal inpatient and cardiology cohorts, an emergency-department cohort, and an external MIMIC cohort. Cohort characteristics and note-processing procedures are described alongside implementation details.

A. Datasets

  • Three internal cohorts from four sites comprised retrospective inpatient, prospective inpatient, and retrospective cardiology patients, with one-year MACE rates of 25.37%, 35.57%, and 51.54%, respectively.Together, these high-risk cohorts were used for model development and internal evaluation; patients without sufficient outcome follow-up were excluded.Source: 1) Internal Cohort · p. 5

1) Internal Cohort:

  • The emergency-department cohort was assembled to assess distribution shift and had a one-year MACE rate of 10.82%, with low reported CKD, diabetes, and hypertension prevalence.The reported prevalences were 0.3% for CKD, 0.9% for diabetes, and 1.5% for hypertension.Source: 2) Shift cohort (ED) · p. 5

2) Shift cohort (ED):

  • The external MIMIC-IV eICU evaluation sampled 5,000 inpatient CXR studies from 4,616 patients and reported a one-year MACE rate of 21.82%.Outcomes were curated from clinical notes, ICD codes, and inpatient mortality data.Source: 3) External-Shift cohort (MIMIC) · p. 5

3) External-Shift cohort (MIMIC):

  • For MIMIC, ordering-physician notes created on or before the CXR date were matched by closest document time, summarized into 3–4 sentences, and missing or empty notes were excluded.The summaries covered available demographics, social and clinical history, medications, reason for visit, and symptoms.Source: 4) Data processing and Implementation · pp. 5–6

4) Data processing and Implementation:

  • The framework was implemented in PyTorch and trained on four NVIDIA RTX A6000 GPUs using bfloat16 precision.Images used each VLM’s native dynamic-resolution processor, and clinical histories entered through multimodal instruction prompts.Source: 4) Data processing and Implementation · p. 6

B. Evaluation Protocol

The evaluation compares unimodal and multimodal baselines, including shared-decoder and role-decoupled systems, and initializes causal RL from a supervised VLM–BioGPT model. Prediction, reasoning quality, clinical utility, and expert preference are assessed using separate metrics and procedures.

B. Evaluation Protocol

  • Baselines included CXR-only and clinical-text-only models, alongside multimodal VLMs evaluated under classification, shared-decoder multitask, and role-decoupled training paradigms.The role-decoupled framework separates multimodal reasoning from downstream risk prediction and outcome generation.Source: B. Evaluation Protocol · p. 6
  • Causal RL started from supervised VLM–BioGPT training, using 80% of internal training data for Stage I and 20% for Stage II policy optimization.The policy selected token-retention masks and reasoning-length budgets of 96 or 128 tokens; its reward weighted prediction error, retained-token proportion, and outcome-generation loss at 1.0, 0.1, and 0.5.Source: B. Evaluation Protocol · pp. 6–7

C. Evaluation Metrics

  • Prediction was measured with AUROC, PPV, and NPV, while LLM-judged reasoning quality used GREEN, completeness, and hallucination metrics; clinical utility included actionability and calibration.CXR-derived findings were excluded from the reasoning-quality metrics.Source: C. Evaluation Metrics · p. 7
  • Experts from radiology, medicine, medical training, and AI/data science compared reasoning preference and evidence support on 20 randomly selected cases.The comparison was blinded and randomized, with paired bootstrap resampling over 2,000 matched-case resamples for significance assessment.Source: C. Evaluation Metrics · p. 7

A. Quantitative Performance

The proposed two-LLM model with causal reinforcement learning showed gains over comparison models in predictive performance and clinical reasoning, with results varying across cohorts. Ablations indicate complementary contributions from text, multimodal components, and causal RL.

A. Quantitative Performance

  • The proposed model raised Mayo internal GREEN from 0.405 ± 0.453 to 0.547 ± 0.425 and Mayo ED from 0.269±0.406 to 0.660±0.392; overall GREEN on MIMIC was similar.It also improved several clinical utility measures, including prediction justification, calibration error, and expert preference; many comparisons reached p < 0.001.Source: A. Quantitative Performance · p. 7, Table III · p. 8
  • Across MIMIC demographic and comorbidity strata, the models had comparable AUC, while the proposed model achieved higher and more stable GREEN scores.The passage interprets this pattern as more consistent reasoning across subgroups while maintaining predictive performance.Source: A. Quantitative Performance · p. 7

B. Ablation Study

  • On Mayo-Internal, removing the text adapter reduced AUC, NPV, and PPV from 0.718, 0.801, and 0.492 to 0.639, 0.746, and 0.437, respectively.The ablation study also tested modality contributions, representation learning, reasoning supervision, and model architecture.Source: B. Ablation Study · pp. 7–8
  • On MIMIC-External, removing text or the text adapter reduced AUC from the Baseline value of 0.762 to 0.638 and 0.719, respectively; removing images lowered Mayo ED AUC from 0.843 to 0.752.Removing the image modality yielded a higher MIMIC Green Score than the Proposed model, 0.832 vs. 0.776.Source: B. Ablation Study · p. 8
  • On Mayo ED, causal RL increased AUC from 0.804 to 0.843 and PPV from 0.233 to 0.293, and improved MIMIC Green Score from 0.715 to 0.776.The passage attributes complementary roles to text-adapter learning, multimodal inputs, and causal RL; using one LLM generally weakened PPV and precluded generation-quality evaluation.Source: B. Ablation Study · p. 8

VI. DISCUSSION

The discussion argues that separating reasoning from risk prediction and optimizing it with causal reinforcement learning addresses limitations of medical vision-language models. It reports complementary prognostic information from CXRs and clinical histories, stronger reasoning quality, and boundaries on validation and interpretation.

VI. DISCUSSION

  • The role-decoupled dual-LLM architecture separates multimodal reasoning from risk prediction to address competing objectives and explanations weakly aligned with clinical decisions.The discussion describes intervention-guided causal reinforcement learning as optimizing reasoning while retaining a differentiable connection between multimodal evidence and downstream classification.Source: §VI · p. 9
  • In a true-negative example, the baseline predicted high risk while the proposed RL-based model correctly predicted low risk with clinically grounded reasoning.Fig. 3 also compares expert preferences across user groups, but the supplied discussion passages do not state the figure's preference results.Source: Figure 3 · p. 9
  • CXRs and physician-authored clinical histories provide complementary prognostic information for MACE, and multimodal models outperformed image-only and text-only baselines.The discussion says this combination yields more informative patient representations than either modality alone.Source: §VI · p. 9
  • The framework achieved higher GREEN scores and stronger expert preference than conventional single-LLM architectures across internal and external cohorts while maintaining competitive discrimination.The discussion attributes improved faithfulness to selecting clinically relevant evidence and suppressing spurious multimodal associations.Source: §VI · p. 9
  • Across demographic and comorbidity subgroups, GREEN-score improvements were consistent, suggesting reduced subgroup-dependent variability and potential robustness and fairness.Fig. 2 compares three models on external MIMIC subgroup AUC and average GREEN score.Source: Figure 2 · p. 9, §VI · p. 10
  • Retrospective evaluation, limited health-system and imaging-protocol coverage, documentation variation, and intervention modeling constrain claims; explanations are evidence-supported reasoning, not causal clinical inference.Prospective evaluation is needed to determine effects on clinician trust, workflow efficiency, or patient outcomes.Source: §VI · p. 10
Loading 2609.38924v1…