Source-linked AI summary
Whole-Body MRI Classification via Prompt-Based Clinical Conditioning
Laura Daza, Marta Hasny, Cristina González, Julia A. Schnabel
TL;DR
Conventional multimodal fusion struggles when structured clinical variables are incomplete, despite their complementarity with WB-MRI for disease classification. TACTIC treats clinical attributes as prompts that condition visual feature learning, and across five tasks it improves over image-only models with available clinical data while remaining effective with limited inputs.
Problem
Clinical variables complement WB-MRI for diagnosis but are often missing or heterogeneous, limiting fusion methods that assume complete fixed-dimensional inputs.
Method
TACTIC encodes native structured clinical attributes as prompts that condition WB-MRI representations through a prompt-conditioned transformer.
Results
Across five disease-classification tasks, TACTIC consistently improves over image-only models when clinical data is available and remains effective with limited clinical variables.
Takeaways & Limitations
Prompting provides a flexible multimodal mechanism for exploiting complementary clinical cues when tabular-data availability is heterogeneous.
Abstract
from arXiv · showhide
Combining whole-body magnetic resonance imaging (WB-MRI) with clinical variables has the potential to improve systemic disease diagnosis by leveraging complementary sources of patient information. However, structured clinical variables are often incomplete or missing, limiting the applicability of conventional multimodal fusion methods that assume fixed inputs. In this work, we propose TACTIC (Tabular-Attribute Conditioned Transformer for Image Classification), a prompt-based multimodal framework that integrates WB-MRI and structured clinical data through conditional visual feature learning. By encoding clinical attributes as prompts, TACTIC supports an arbitrary number of tabular inputs and naturally handles missing data without requiring imputation or fixed input structures. We evaluate TACTIC on five WB-MRI classification tasks spanning systemic and oncologic applications, including diabetes, chronic obstructive pulmonary disease (COPD), breast cancer, prostate cancer, and metastasis diagnosis. Across all tasks, TACTIC consistently improves performance over image-only baselines when clinical information is available while maintaining strong predictive capability under incomplete tabular inputs. Our results demonstrate the effectiveness of prompt-based models as a flexible approach for improving WB-MRI analysis using clinical context. The model weights and code are available at https://github.com/lauradaza/TACTIC
1 Introduction
WB-MRI provides broad systemic imaging, but clinically useful diagnosis also requires structured variables that are frequently incomplete. TACTIC addresses this gap by using clinical attributes as prompts for flexible image-feature conditioning across five disease tasks.
- WB-MRI captures multiple organs without ionizing radiation, supporting systemic disease assessment and population-scale screening.
- Multimodal diagnosis combines imaging with complementary clinical variables such as laboratory results, genetic predispositions, and medical history.
- Missing, sparse, or heterogeneous clinical variables challenge conventional fusion methods that depend on fixed-dimensional tabular inputs.
- TACTIC encodes native clinical variables as prompts that condition WB-MRI representations and support an arbitrary number of clinical inputs.
- TACTIC is evaluated on diabetes, COPD, breast cancer, prostate cancer, and metastasis classification, improving over image-only models when clinical data is available.
- The overview combines random clinical-feature subsets with an output token to guide image-feature aggregation through cross-attention under variable data availability.
2 Method
TACTIC uses clinical attributes as prompts that interact with WB-MRI features in a transformer, while training explicitly simulates different levels of clinical-data availability.
- TARTE jointly encodes each clinical attribute’s name and value into an embedding, while Primus-M serves as the WB-MRI image encoder.
- TACTIC first applies self-attention to clinical prompts, then uses cross-attention to guide patient-specific aggregation of image features.
- Body masking and mask-aware pooling reduce WB-MRI background content while aiming to preserve information during image encoding.
- TARTE training samples subsets C_more and C_fewer of clinical attributes and applies the TabMoFe loss to encourage better predictions with more information.
- The final tabular objective combines the TabMoFe loss with cross-entropy losses from the more-attribute and fewer-attribute subsets.
- TACTIC samples random clinical-attribute subsets, concatenates each with an output token, and conditions predictions on these prompts alongside image features.
- The image-only loss conditions the model on the output token with an empty clinical prompt set, providing an image-only training condition.
3 Experimental Setting
Experiments use UK Biobank WB-MRI and clinical data across five balanced disease-classification tasks, with fixed train/validation/test splits and specified pretraining and fine-tuning procedures.
- The dataset includes over 500,000 UK Biobank participants, including 80,000 with WB-MRI and corresponding clinical information.
- WB-MRI consists of neck-to-knee T1-weighted dual-echo Dixon acquisitions, and experiments use fat-only and water-only reconstructions.
- The study evaluates diabetes, COPD, breast cancer, prostate cancer, and metastasis using ICD-10-derived labels.
- Participants are split into training, validation, and test sets using a 70–10–20 ratio, with balanced training subsets for under-represented conditions.
- Primus pretraining runs for 50 epochs with 90% masking, followed by task fine-tuning using the same architecture and training configuration.
- Imaging and multimodal tuning use 40% image masking, batch size 48, and gradient accumulation over six steps, yielding an effective batch size of 288.
4 Results
Across five UK Biobank classification tasks, TACTIC generally improves WB-MRI classification by conditioning image representations on clinical data, including under incomplete availability. Its advantage varies by task, with strongest robustness demonstrated for COPD attribute sparsity.
- Performance Comparison: TACTIC improves upon the image-only baseline across all five classification tasks and achieves the best overall performance for four diseases.For diabetes, it matches the best fusion approach despite clinical variables alone achieving an AUROC of 87.1.
- Performance Comparison: TACTIC achieves the highest AUROC for COPD and breast cancer, where neither imaging nor tabular data alone is sufficient for optimal prediction.Conventional fusion methods may degrade performance relative to the tabular-only baseline in these settings.
- Performance Comparison: For prostate cancer, Concat Fuse performs best because WB-MRI provides limited discriminative information compared with the tabular baseline.TACTIC benefits from clinical conditioning but gains less when imaging contributes minimal task-relevant information.
- Performance Comparison: For metastasis detection, TACTIC obtains the largest improvement over the image-only model while some fusion methods fail to improve it.The result indicates that prompting can incorporate complementary patient information without compromising visual representations when clinical variables may introduce noise.
- Incomplete Clinical Data: TACTIC consistently achieves the highest COPD performance across random, ascending-importance, and descending-importance attribute availability.Attribute importance is estimated from TARTE attention weights, and performance is evaluated across varying fractions of visible clinical attributes.
- Incomplete Clinical Data: Under random attribute selection, TACTIC outperforms all multimodal approaches across the full availability range, including when only a small clinical subset is provided.Its performance steadily improves as more attributes become available and remains above the image-only starting point when low-importance attributes are provided.
5 Conclusion
TACTIC is a prompt-based framework that integrates WB-MRI with structured clinical variables without requiring fixed or complete tabular inputs. Across systemic and oncological tasks, it improves over image-only models when clinical information is available and remains effective with limited clinical data.
- Conclusion: TACTIC complements WB-MRI with structured clinical variables by conditioning image representations on clinical attribute prompts.The framework is designed for disease classification and patient-specific context integration.
- Conclusion: TACTIC remains effective with limited clinical variables and shows greater robustness than previous fusion approaches as tabular availability varies.Its performance is reported as more stable under incomplete clinical data settings.
- Conclusion: Prompting enables more effective exploitation of complementary clinical cues than direct feature aggregation within the reported WB-MRI tasks.The findings highlight flexible multimodal frameworks for clinically realistic settings with heterogeneous data availability.