Source-linked AI summary
Knee3DVLM: Dual-Sequence Full-Volume Vision-Language Modeling for Comprehensive Knee MRI Assessment
Maryam Baizhigitova, Andrew Seohwan Yu, Po-Hao Chen, Naveen Subhas, Sixu Chen, Xinxin Wang, Kunio Nakamura, Richard Lartey, Xiaojuan Li, Mingrui Yang
TL;DR
Three-dimensional knee MRI VLM research remains limited in comprehensive, multi-sequence whole-joint assessment. Knee3DVLM pairs full-volume DESS and TSE inputs; its fused configuration led the tested configurations on all four aggregate endpoints in the primary task.
Problem
Existing knee MRI VLMs generally process a single series and address isolated diagnostic questions rather than comprehensive, multi-sequence whole-joint assessment.
Source: §1 · p. 2Method
Knee3DVLM is a sequence-aware framework for full-volume knee MRI interpretation using paired DESS and TSE inputs.
Source: §1 · p. 2Results
Knee3DVLM led all three configurations on four primary-task aggregate endpoints, with 72.98% average accuracy; it was also numerically higher across five shared 3DReasonKnee categories.
Source: §4.1 · p. 8, §5 · p. 10Takeaways & Limitations
These findings support sequence-aware fusion for comprehensive knee MRI assessment and underscore the need for class-aware evaluation.
Source: §6 · p. 10Published 3DReasonKnee results lack per-grade confusion matrices, so balanced accuracy cannot be recovered and the comparison is not complete grading-performance evidence.
Source: §4.2 · p. 10
Abstract
from arXiv · showhide
Vision-language models (VLMs) are increasingly being applied to three-dimensional medical imaging, but their application to knee MRI remains limited, particularly for interpreting the complementary sequences used in clinical practice. We introduce Knee3DVLM, a sequence-aware VLM that uses full-volume DESS and fluid-sensitive TSE MRI to predict 57 anatomically resolved binary diagnostic targets derived from the MRI Osteoarthritis Knee Score (MOAKS) for structured reporting. We evaluated DESS-only, TSE-only, and paired DESS-TSE configurations using subject-disjoint Osteoarthritis Initiative partitions. In a held-out cohort of 1,074 examinations, the fused model achieved 72.98% average accuracy, 71.17% balanced accuracy, 78.96% mean ROC-AUC, and 78.74% macro ROC-AUC, the highest values among the three configurations. In a secondary multiclass analysis aligned with the released 3DReasonKnee cohort, Knee3DVLM was numerically higher than the strongest reported 3DReasonKnee configuration across five pathology categories. These findings support dual-sequence full-volume modeling for comprehensive knee MRI assessment.
1 Introduction
Knee MRI assessment requires reconciling complementary fluid-sensitive TSE and high-resolution DESS findings across the whole joint, but existing knee VLMs generally use single-series inputs. Knee3DVLM addresses this gap with sequence-preserving full-volume fusion and evaluations of single- and dual-sequence configurations.
Introduction
- Knee MRI VLMs generally process a single series and isolated diagnostic questions, limiting comprehensive multi-sequence whole-joint assessment.3DReasonKnee adds regional localization, reasoning traces, and structured MOAKS severity assessments, but uses DESS alone.Source: §1 · p. 2
- Knee3DVLM uses independent full-volume DESS and TSE encoders, stream-specific tokenizers, modality embeddings, and token-level fusion before language-space projection.The design preserves acquisition identity while integrating complementary contrasts without collapsing them at the image or encoder-input level.Source: §1 · p. 2
- The study compares DESS-only, TSE-only, and fused DESS-TSE configurations on a primary 57-target binary task to assess sequence-specific performance and the effect of combining sequences.The analysis examines whether performance follows clinically coherent tissue contrasts.Source: §1 · p. 2
- A secondary benchmark aligns OAI data with 3DReasonKnee identifiers and compares category-wise exact-grade average accuracy using that study’s reported metric.The comparison is reported in Table 5.Source: §1 · p. 2
2 Related Work
MOAKS and OAI provide the assessment framework and data context, while DESS and fluid-sensitive TSE offer complementary views of knee pathology. Prior 3D models motivate full-volume reasoning, but Knee3DVLM extends 3DReasonKnee’s single-DESS, question-based approach to paired sequences and comprehensive whole-joint output [1] [2].
Semiquantitative Knee MRI Assessment
- MOAKS assesses whole-joint cartilage, bone marrow, osteophyte, meniscal, synovial, and other structural findings, and OAI provides longitudinal MRI and clinical data at scale.Fluid-sensitive TSE offers greater validity and statistical power for bone marrow lesion assessment, whereas DESS supports high-resolution cartilage and osseous-surface morphology.Source: §2 · p. 2
Three-Dimensional Medical Vision-Language Models
Knee MRI Reasoning
- 3DReasonKnee links DESS volumes to subregion-specific diagnostic questions, bounding boxes, clinician-authored reasoning, and MOAKS grades.Knee3DVLM retains its anatomically explicit emphasis but extends the input to paired DESS and fluid-sensitive TSE and the output to comprehensive structured whole-joint assessment.Source: §2 · pp. 2–3
3 Methods
Knee3DVLM combines full-volume DESS and TSE MRI to predict 57 anatomy-specific binary MOAKS targets, with a secondary exact-grade analysis over 109 attributes. The study uses subject-disjoint OAI partitions and validation-selected thresholds for test predictions.
3.1 Dataset and Cohort.
- The OAI cohort contains 7,970 studies from 2,652 participants assigned to fixed subject-disjoint partitions; the three configurations share a 1,074-examination primary test cohort.The common cohort was formed after preprocessing and matching prediction records across DESS-only, TSE-only, and paired-model configurations.Source: §3.1 · p. 3
3.2 Target Construction.
- The primary task maps valid nonzero MOAKS grades to abnormal across 57 anatomy-specific targets, while secondary exact-grade analysis retains 109 original MOAKS attributes aligned to 3DReasonKnee.Missing or non-evaluable binary readings are masked rather than treated as negative; the exact-grade analysis uses original OAI fields, not generated reports, and is secondary because grades are imbalanced.Source: §3.2 · p. 4
3.3 Model Architecture.
- Independent trainable DCFormer encoders process matched full-volume DESS and TSE, compress each stream into 16 modality-tagged tokens, then fuse 32 tokens for language-conditioned 57-target prediction.The model jointly trains the encoders, sequence-specific tokenizers, fusion module, projector, and LoRA-adapted language backbone.Source: §3.3 · pp. 5–6
3.4 Training and Inference.
- Each target’s decision threshold maximizes validation balanced accuracy and is locked before test evaluation; the same deterministic reporting procedure is applied to all three configurations.Missing test labels are excluded separately for each target.Source: §3.4 · p. 7
3.5 Evaluation.
- Balanced accuracy is the primary endpoint; evaluation also reports accuracy, sensitivity, specificity, mean ROC-AUC, and macro ROC-AUC, with the two AUC summaries differing in target weighting.Mean ROC-AUC weights targets by evaluable examinations, while macro ROC-AUC weights targets equally; exact-grade category accuracy is the unweighted mean of attribute-level accuracies within each category.Source: §3.5 · pp. 7–8
4 Results
Dual-sequence fusion achieved the highest performance on every prespecified aggregate binary endpoint and was numerically higher than the published 3DReasonKnee configuration across all five multiclass categories. Results also show sequence-specific strengths and important limits on interpreting exact-grade accuracy.
4.1 Primary 57-Target Binary Results.
- The PCL estimate is substantially uncertain because only six positive PCL examinations were available in the held-out cohort.Source: §4.1 · p. 8
- DESS had higher balanced accuracy than TSE for cartilage (72.77% vs. 70.62%) and osteophytes (71.29% vs. 69.61%), while TSE led in five other pathology groups.TSE values were BMLs 71.34% vs. 69.55%, meniscal abnormalities 68.40% vs. 67.18%, ACL abnormalities 80.42% vs. 75.95%, effusion-synovitis 72.86% vs. 71.64%, and Hoffa-synovitis 65.71% vs. 64.22%.Source: §4.1 · p. 8, Table 3 · p. 8
- ACL abnormalities had the strongest group-level discrimination result at 87.99% ROC-AUC; cartilage had the highest mean ROC-AUC (80.45%) and macro ROC-AUC (80.38%) among multi-target groups.These group-level metrics were reported for Knee3DVLM.Source: §4.1 · p. 9, Table 4 · p. 9
- 72.98% average accuracy, 71.17% macro balanced accuracy, 78.96% mean ROC-AUC, and 78.74% macro ROC-AUC made Knee3DVLM the top configuration on every prespecified aggregate endpoint.Versus Knee3DVLM-TSE, fusion improved these endpoints by 2.62, 2.07, 2.13, and 2.29 percentage points, respectively; versus Knee3DVLM-DESS, by 0.39, 0.75, 2.03, and 1.85 points, respectively.Source: §4.1 · pp. 8–9, Table 2 · p. 8
- Fusion achieved the highest balanced accuracy in all four multi-target pathology groups: BML, cartilage, osteophytes, and meniscal abnormalities.Among single-sequence models, DESS performed better for cartilage and osteophytes, while TSE performed better for BMLs, meniscal abnormalities, ACL abnormalities, effusion-synovitis, and Hoffa-synovitis.Source: §4.1 · pp. 8–9, Table 3 · p. 8
4.2 Multiclass Grading and Comparison with 3DReasonKnee.
- Knee3DVLM was numerically higher than the published Qwen2.5VL-3B configuration without chain-of-thought supervision [3] in all five multiclass categories.Category-wise average accuracy: BML 72.11% vs. 70.60%; cartilage 62.32% vs. 60.20%; osteophyte 35.22% vs. 34.50%; meniscus 40.32% vs. 38.70%; Others 65.20% vs. 61.60%. The comparison used category-wise average accuracy on a cohort aligned by subject, visit, and knee side.Source: §4.2 · p. 9, Table 5 · p. 9
- Grade 0 represented 85.41% of ground-truth component labels, and published 3DReasonKnee results lack per-grade confusion matrices, preventing recovery of their balanced accuracy.The table therefore provides a benchmark-aligned comparison using the reported metric, not a complete comparison of grading performance; exact-grade accuracy can remain high despite weak minority-grade performance.Source: §4.2 · pp. 9–10
5 Discussion
The discussion interprets sequence-specific strengths as anatomically and clinically plausible, while noting they do not prove that sequence contrast caused performance differences. In the secondary exact-grade experiment, Knee3DVLM was numerically higher than the strongest 3DReasonKnee configuration across all shared categories, but the study did not evaluate reader performance or patient outcomes.
5 Discussion
- DESS's advantage for cartilage and osteophytes is anatomically plausible given its high-resolution depiction of articular morphology and tissue interfaces, but does not establish causality.DESS was selected in the OAI protocol for quantitative cartilage morphometry, and prior whole-joint studies found chondral abnormalities well visualized with high-resolution fat-suppressed DESS.Source: §5 · p. 10
- TSE's higher performance for BML, meniscus, ACL, effusion, and synovitis aligns with fluid-sensitive imaging's clinical role in visualizing these abnormalities.Fat-suppressed TSE increases conspicuity of marrow edema-like signal, joint fluid, and soft-tissue edema; ESSR recommendations emphasize multiplanar fluid-sensitive imaging for systematic knee assessment.Source: §5 · p. 10
- The findings support sequence-specific encoders that preserve morphology-dominant DESS and fluid-sensitive TSE features before token-level fusion.A prior multireader study found that adding a short fluid-sensitive PD-FSE acquisition to DESS improved positive and negative agreement for comprehensive knee assessment.Source: §5 · p. 10
- 5/5 shared categories: Knee3DVLM was numerically higher than the strongest published 3DReasonKnee configuration.Accuracy was 72.11% vs. 70.60% for BML, 62.32% vs. 60.20% for cartilage, 35.22% vs. 34.50% for osteophyte, 40.32% vs. 38.70% for meniscus, and 65.20% vs. 61.60% for Others. Among the study's models, DESS led in cartilage and osteophyte accuracy, TSE in meniscus and Others, and Knee3DVLM in BML.Source: §5 · p. 10
- The study evaluates diagnostic targets and deterministic structured reports, not prospective reader performance, workflow impact, or patient outcomes.Source: §5 · p. 10
6 Conclusion
Knee3DVLM combines full-volume, whole-joint assessment with paired DESS and fluid-sensitive TSE inputs for three-dimensional knee MRI. Its fused model led the tested configurations on the primary binary task and was numerically higher than the strongest published 3DReasonKnee baseline in all five secondary exact-grade categories, while the results underscore the need for class-aware evaluation.
6 Conclusion
- The fused model achieved the highest average accuracy, balanced accuracy, mean ROC-AUC, and macro ROC-AUC among the three tested configurations on the primary binary task.The tested configurations used DESS-only, TSE-only, and paired DESS-TSE inputs.Source: §6 · p. 10
- On the secondary multiclass exact-grade benchmark, Knee3DVLM was numerically higher than the strongest published 3DReasonKnee baseline in all five reported categories.The conclusion underscores the need for class-aware evaluation.Source: §6 · p. 10
- Knee3DVLM addresses a gap in three-dimensional knee MRI by combining full-volume, whole-joint assessment with paired DESS and fluid-sensitive TSE inputs.Source: §6 · p. 10