Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening

Yuhao Huang; Yuanji Zhang; Yuhuan Lu; Dong Ni; P. Ellen Grant; Davood Karimi

Source-linked AI summary

Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening

Yuhao Huang, Yuanji Zhang, Yuhuan Lu, Dong Ni, P. Ellen Grant, Davood Karimi

arXiv:2608.14763v1Aug 14, 2026eess.IVcs.AIcs.CVcs.LG

TL;DR

Routine ultrasound may not capture fetal ventricular enlargement as comprehensively as MRI, so VIFBA learns MRI-informed volumetric and abnormality assessment from ultrasound videos alone. On held-out data, it achieved strong performance across ventricular volume estimation, VM severity classification, and broader abnormality alerting.

Problem
Ultrasound ventricular width measurements are operator-dependent and limited in reflecting overall enlargement, while MRI volumetry is more reliable but costly and inaccessible for routine screening.
Method
VIFBA combines video representation learning, MRI-guided cross-modal alignment, task-specific volume and severity prediction, and retrieval-augmented VLM verification using ultrasound alone at inference.
Results
VIFBA achieved the best overall performance, including an MAE of 0.5909, Pearson correlation of 0.9907, and Acc of 0.9400.
Takeaways & Limitations
VIFBA provides a more affordable and widely deployable approach for quantitative fetal ventricular assessment from routinely acquired ultrasound videos.

Abstract

from arXiv · show

Assessment of ventriculomegaly (VM) on fetal brain ultrasound relies primarily on measuring lateral ventricular atrial width on standard planes, which is operator-dependent and may not fully reflect the overall ventricular enlargement. Fetal brain MRI provides more reliable volumetric information but is costly and less accessible for routine use. To address these limitations, we propose VIFBA, an ultrasound video-based framework for fetal brain assessment that predicts MRI-derived lateral ventricular volume, classifies VM severity, and identifies potential non-VM fetal brain abnormalities. Our contribution is three-fold. First, we introduce a joint-embedding predictive architecture (JEPA)-inspired tube latent prediction objective that leverages spatio-temporal coherence in ultrasound videos to enhance representation learning. Second, we develop a contrastive cross-modal alignment strategy that transfers structural information from MRI to ultrasound during training, while requiring ultrasound alone at inference. Third, we augment VIFBA with a training-free vision-language model and retrieval augmentation to verify uncertain predictions and identify potential non-VM fetal brain abnormalities. We validated VIFBA on a large dataset comprising 857 cases (3,196 videos) with paired fetal brain ultrasound and MRI examinations. On held-out test data, VIFBA achieved an MAE of 0.5909 mL and Pearson correlation coefficient of 0.9907 for ventricular volume regression, 0.9400 accuracy for VM severity classification, and an F1 score of 0.7764 for multi-abnormality classification, substantially outperforming single-task baselines, video-based strong competitors, and state-of-the-art foundation models. By enabling MRI-informed volumetric assessment from routine ultrasound alone, VIFBA offers a practical and potentially broadly deployable pathway toward accurate and affordable prenatal brain screening.

I. INTRODUCTION · II. RELATED WORK · A. Deep Learning in Fetal Brain Analysis

VIFBA addresses limitations of operator-dependent 2D ultrasound measurements by learning MRI-informed ventricular volumetry and VM assessment from ultrasound videos alone. The framework also targets multimodal fetal brain analysis and uncertainty-aware screening for non-VM abnormalities.

I. INTRODUCTION: VM is a common fetal central nervous system abnormality whose variable outcomes make accurate ventricular quantification and severity assessment clinically important.Severity and etiology influence outcomes, risk stratification, and clinical decision-making.
I. INTRODUCTION: Ultrasound defines VM using lateral-ventricle atrial width, but selecting standard planes and landmarks makes this measurement operator-dependent.VM is defined as ≥10 mm and categorized as mild, moderate, or severe according to atrial width.
I. INTRODUCTION: Ventricular volume provides a more global measure than 2D width, while MRI offers clearer ventricular boundaries and more reliable volumetric information.Volumetric assessment commonly uses 3D ultrasound or MRI.
I. INTRODUCTION: VIFBA jointly estimates ventricular volume and VM severity from ultrasound videos using shared backbones and task-specific regression and classification branches.The framework is designed for variable multi-view ultrasound videos and scarce paired ultrasound–MRI data.
I. INTRODUCTION: VIFBA introduces tube latent prediction and contrastive cross-modal fusion to transfer MRI structural information into ultrasound representations during training.MRI-informed feature learning is retained while inference requires ultrasound alone.
I. INTRODUCTION: A retrieval-augmented, training-free visual-language model further optimizes uncertain predictions and alerts potential non-VM fetal brain abnormalities.The framework was validated using 857 paired ultrasound–MRI cases.
A. Deep Learning in Fetal Brain Analysis: Deep learning research has advanced prenatal imaging, with ultrasound supporting routine screening and MRI complementing it for clearer intracranial visualization and suspected-abnormality evaluation.Ultrasound research includes standard-plane localization, view classification, quality assessment, and anatomical segmentation or measurement.
A. Deep Learning in Fetal Brain Analysis: Existing ultrasound and MRI studies are usually separated, while VM assessment remains largely 2D and abnormality analysis often remains coarse or limited to selected diseases.These gaps motivate combining complementary modalities with volumetric and broader abnormality assessment.

B. Medical Ultrasound Foundation Model … A. Foundation Model-driven Ultrasound Video Modeling

The paper situates VIFBA within ultrasound foundation-model and LLM/VLM-assisted imaging research, addressing limitations in abnormality verification while using pretrained ultrasound representations and MRI-informed supervision for video-based ventricular assessment.

B. Medical Ultrasound Foundation Model: Foundation models enable transferable medical-imaging representations that can be adapted to diverse downstream tasks with limited annotations.Examples include BiomedCLIP for biomedical vision-language representation learning and medical SAMs for promptable image segmentation.
B. Medical Ultrasound Foundation Model: Ultrasound foundation models have expanded through large-scale, multi-organ, multi-center, and multi-device pretraining, demonstrating label-efficient downstream performance.The passage also describes EchoCare as increasing ultrasound pretraining data from 2 million to 4.5 million images.
B. Medical Ultrasound Foundation Model: Cardiac and fetal ultrasound studies have developed vision-language, segmentation, keyframe-detection, and generalizable representation-learning models.The cited examples include EchoCLIP, EchoPrime, EchoONE, FrameONE, and FetalCLIP.
C. LLM/VLM-assisted Medical Image Analysis: Existing LLM/VLM-assisted medical-imaging systems mainly support diagnosis, classification, report generation, or interpretation through integrated model outputs and medical knowledge.Examples include ChatCAD, ChatCAD+, and FAA-Net.
C. LLM/VLM-assisted Medical Image Analysis: Their applicability remains limited for verifying existing predictions and identifying abnormalities absent from training, which is important for heterogeneous fetal-brain evidence.The passage specifically highlights current ultrasound videos, model predictions, and heterogeneous clinical evidence as relevant to this limitation.
III. METHOD: During training, VIFBA encodes each ultrasound video with a pretrained vision foundation model, uses ventricular-volume regression and VM-classification branches, and incorporates paired MRI for complementary supervision.During inference, the framework produces predictions from ultrasound videos, although the supplied passage truncates the remaining inference description.
A. Foundation Model-driven Ultrasound Video Modeling: Mean-pooled temporal features pass through a shared projection block to regression and classification heads optimized with MSE and class-balanced weighted CE losses.The regression head predicts a scalar ventricular volume, while the classification head produces a C-class disease logit vector.

B. Tube Latent Prediction for Representation Enhancement

The tube latent prediction objective strengthens ultrasound-video representations by modeling local spatio-temporal structure when supervision and fine-grained annotations are limited. It uses tube-wise masking with teacher-student prediction and temporal token encoding to enforce coherent representations over masked tokens.

Motivation: TLP addresses limited supervision by constraining intrinsic local spatio-temporal structure that supervised video-level objectives capture only weakly.This is intended to help exploit anatomical and dynamic information in ultrasound videos when training data and fine-grained annotations are limited.
Spatio-temporal masking: Tube-wise masking hides selected spatial patch locations across contiguous temporal spans, producing masked videos for prediction.The mask is applied over the patch grid and mapped to pixel-space patterns in each frame.
Teacher-student prediction: A teacher-student branch processes the original and masked videos, with teacher parameters updated as an exponential moving average of the student.The masked video is encoded by the student, while the unmasked video is encoded by the teacher.
Temporal prediction objective: A temporal token encoder contextualizes tokens across frames for each spatial location, while the predictor maps student tokens toward teacher representations.The teacher temporal encoder is maintained as an EMA copy of the student encoder, and the TLP loss is computed only at masked spatiotemporal positions.

C. MRI-Guided Cross-Modal Training Alignment

VIFBA aligns ultrasound and MRI representations during training using paired examinations, while requiring only ultrasound videos at inference. MRI features extracted with BOUNTI guide ultrasound toward structurally informative embeddings that are fused for prediction.

Training and inference: Paired ultrasound-MRI data enable feature-level alignment during training, while inference uses ultrasound videos alone.MRI offers clearer and more stable anatomical structure than ultrasound but is more expensive and less accessible in routine practice.
MRI representation: BOUNTI’s pretrained U-Net and Attention U-Net segmentation models extract MRI bottleneck features that are pooled and concatenated.These features provide the MRI representation used for cross-modal alignment.
Shared embedding space: LayerNorm-linear-ReLU projectors map MRI and ultrasound representations into a shared embedding space for alignment.The MRI projector ψ(·) and ultrasound projector φ(·) use the same architecture.
Contrastive alignment: Contrastive alignment normalizes both modalities, computes within-batch cosine similarities, and uses same-case ultrasound-MRI pairs as positives.The positive-pair mask P identifies samples belonging to the same case, while τ is a learnable scaling factor.
Feature fusion: The alignment loss encourages MRI-consistent ultrasound structure, which is concatenated with the original ultrasound representation and processed by a fusion module.The fusion module consists of LayerNorm, Linear, and ReLU layers.

D. VLM Verification and Non-VM Brain Abnormality Alerting

VIFBA performs case-level fusion across ultrasound videos, then uses retrieval-augmented, training-free VLM verification to recalibrate uncertain predictions and screen for potential non-VM fetal brain abnormalities. Retrieved reference cases provide visual, diagnostic, and volumetric evidence for reliability assessment and abnormality alerts.

Case-level inference: Case-level inference averages features from all videos of a subject before producing regression and classification outputs, improving robustness to multiple views.The final classification label is obtained with argmax from the case-level classification output.
VLM verification: Retrieval augmentation addresses uncertain predictions by comparing the query representation with clinically similar historical cases and incorporating their reference patterns.Uncertainty can arise from heterogeneous video quality, incomplete anatomical observations, and inconsistent predictions across views.
VLM verification: When calibration is required, retrieved class distributions and ventricular volumes are aggregated with normalized relevance weights and combined using the global adjustment score β=0.3.The final diagnosis category is selected by applying argmax to the calibrated class distribution.
Non-VM abnormality alerting: The abnormality-aware prompt enables training-free screening for non-VM findings, reporting every candidate whose alert score satisfies s_i>τ.The VLM outputs ranked suspected abnormalities and confidence scores s_i∈[0,1], supported by retrieved abnormal references and diagnostic descriptions.

IV. EXPERIMENTS · A. Datasets

The study assembled an expert-verified paired fetal brain ultrasound–MRI dataset for validating volumetric and diagnostic assessment. It included 857 cases and 3,196 ultrasound videos, with MRI-derived ventricular-volume ground truth and labels for VM and other abnormalities.

A. Datasets: 857 cases and 3,196 ultrasound videos were collected between January 2022 and February 2026, averaging 3.73 videos per case.The dataset included several MRI slice stacks per case, and gestational age ranged from 16.57 to 38.00 weeks.
A. Datasets: MRI volumes were reconstructed with NesVoR, while BOUNTI and FeTA produced bilateral lateral-ventricle segmentations for subsequent expert review.Ventricular-volume ground truth was computed by summing segmented voxels and multiplying by corresponding voxel spacing.
A. Datasets: Diagnostic labels covered six non-VM abnormality categories, including neural tube, midline-structure, ventricular-system, posterior-fossa, and neurodevelopmental abnormalities.The categories were derived from radiology reports and LLM analysis; the supplied passage also indicates a sixth category, A6.
A. Datasets: Among 281 VM cases, 128 (45.6%) were isolated, while the remainder had one to four additional brain-abnormality categories.The corresponding VM-case counts associated with A1–A6 were 40, 63, 22, 54, 61, and 10.
A. Datasets: Among 277 non-VM abnormal cases, 185, 55, and 12 were associated with one, two, and three brain-abnormality categories, respectively.The passage additionally notes that 25 of the 277 non-VM abnormal cases belonged to an uncommon group outside categories A1–A6.
A. Datasets: Twenty-five cases involved uncommon abnormalities such as dural sinus malformation, intracranial venous thrombosis, venous sinus malformation, and Vein of Galen malformation, which were excluded from the study categories.All generated labels were reviewed and verified by a fetal ultrasound expert with over 10 years of clinical experience.

B. Implementation Details

VIFBA was implemented and trained under specified software, hardware, optimization, and video-processing settings. Evaluation used task-specific metrics, fixed random seeds, and repeated statistical comparisons.

Implementation: VIFBA used Python 3.9.25, PyTorch 2.0.1, a single 48 GB NVIDIA RTX 6000 Ada GPU, 100 epochs, batch size 8, and AdamW at 1e−4.The MRI projector used a learning-rate scaling factor of 0.1 relative to the main optimizer; 64 video frames were uniformly sampled.
Evaluation metrics: Regression used MAE, RMSE, MSE, Pearson correlation, and Spearman correlation, while VM evaluation used accuracy, macro-precision, macrorecall, macro-F1, and macro-AUC.The proposed model was evaluated on both regression and classification tasks.
Statistical analysis: Experiments used fixed random seeds unless otherwise specified, with selected experiments repeated across 10 random seeds to assess robustness and statistical significance.Paired differences across repeated runs were tested for normality before selecting the statistical comparison procedure.
Statistical analysis: One-sided paired t-tests were used when normality held; otherwise, Wilcoxon signed-rank tests assessed whether the target method outperformed comparisons.Tests used lower-value directions for MSE, RMSE, and MAE, and higher-value directions for Pearson, Spearman, and classification metrics.

C. Comparison with Video-Based Methods

VIFBA substantially outperformed representative video-based models and single-task baselines on lateral ventricular volume regression and VM severity classification. Multi-seed analyses and video- versus case-level evaluations further supported its performance across fetal brain imaging conditions.

Compared methods: The comparison included I3D, R(2+1)D, TimeSformer, InternVideo2, VideoMamba, and V-JEPA2 with task-specific regression or classification heads.Tables II and III compared these representative video-analysis methods with VIFBA for ventricular volume estimation and VM severity classification.
Volume regression: VIFBA reduced volume-regression MSE, RMSE, and MAE to 0.7507, 0.8664, and 0.5909, respectively, outperforming compared video models and Baseline (Reg).It also achieved Pearson and Spearman correlation coefficients of 0.9907 and 0.9730, respectively.
VM severity classification: VIFBA achieved the best VM severity classification results, with Acc 0.9400, Pre 0.9357, F1 0.9130, and AUC 0.9793.V-JEPA2 reached Acc 0.7700 and F1 0.7208, while Baseline (Cls) reached Acc 0.7900 and F1 0.7568.
Statistical analysis: Multi-seed statistical analyses further confirmed significant performance improvements and attributed VIFBA’s stronger representations to foundation model-based encoding, latent predictive learning, and MRI-informed supervision.The results indicated that general video-analysis model designs were insufficient for fine-grained fetal brain ultrasound predictions.
Video- and case-level evaluation: Video-level predictions varied with imaging views, video quality, and incomplete anatomical observations, while aggregated case-level predictions showed good agreement with ground truth across brain and VM-severity groups.Evaluations covered normal brain, non-VM abnormal brain, and different VM severity groups.

D. Ablation Study · E. Impact of Different Numbers of Videos

The ablation study shows that TLP, CMTA, and VLM contribute complementary gains, with full VIFBA achieving the strongest reported regression and classification results. Increasing the number of aggregated ultrasound videos improves performance, especially from one to three, while reducing prediction variability.

D. Ablation Study: TLP alone achieved an MSE of 2.1778, an MAE of 1.0739, and an Acc of 0.8800, indicating benefits for video representation learning.The ablation evaluated tube latent prediction, cross-modal training alignment, and abnormality verification and alerting.
D. Ablation Study: 1.3936 MSE and 0.9140 MAE were achieved by combining TLP and CMTA, while TLP with VLM reached an Acc of 0.9100.The combined modules further improved performance over individual components.
D. Ablation Study: 0.7507 MSE, 0.5909 MAE, 0.9907 Pearson correlation, 0.9400 Acc, and 0.9130 F1 were achieved by full VIFBA.These were the best overall results reported in the ablation study.
D. Ablation Study: Statistically significant improvements appeared in most comparisons, although individual-module gains were limited in some settings and inconsistent across metrics.Experiments used 10 different random seeds, and the complementarity of the three components was highlighted across ablation variants.
E. Impact of Different Numbers of Videos: One to five videos were randomly sampled per case, with all available videos used when fewer were present.Sampling was repeated five times per setting to reduce the influence of random video selection.
E. Impact of Different Numbers of Videos: Mean performance curves used minimum-to-maximum shaded ranges and standard-deviation error bars to summarize repeated video-selection experiments.The analysis examined case-level feature aggregation with different numbers of selected videos.
E. Impact of Different Numbers of Videos: Increasing the number of selected videos consistently improved regression and classification, with most gains occurring from 1 to 3 before performance gradually converged.Narrower shaded regions with more videos indicated reduced prediction variability and improved stability of case-level feature aggregation.

F. Comparison with Different Foundation Model Encoders · G. VM Verification and Multi-Abnormality Alerting

Foundation-model encoders consistently strengthened VIFBA, with ultrasound-specific models generally outperforming general biomedical encoders and FetalCLIP achieving the strongest regression errors. Retrieval-augmented VLM verification substantially improved multi-abnormality alerting, led by GPT-5.6 Sol, while qualitative results supported close volume agreement and effective abnormality identification.

F. Comparison with Different Foundation Model Encoders: All evaluated foundation-model encoders achieved strong performance versus conventional deep models, supporting foundation representations for fetal brain ultrasound video analysis.
F. Comparison with Different Foundation Model Encoders: Ultrasound-specific encoders generally outperformed general biomedical encoders, including PMC-CLIP and BiomedCLIP.
F. Comparison with Different Foundation Model Encoders: FetalCLIP achieved the lowest MSE, RMSE and MAE while maintaining strong correlation and classification performance.Figure 9 compared performance variation across 10 independent runs, with statistical significance marked where applicable.
G. VM Verification and Multi-Abnormality Alerting: The one-stage VIFBA-CLS-S1 achieved an EMR of 0.3200 for multi-abnormality prediction, while the two-stage VIFBA-CLS-S2 improved performance.The two-stage design first identifies abnormal brains and then predicts specific abnormality categories.
G. VM Verification and Multi-Abnormality Alerting: GPT-5.6 Sol achieved a Pre of 0.7207, a Rec of 0.8933, an F1 score of 0.7764, an HL of 0.0440, and an EMR of 0.7500.GPT-5.6 Terra achieved an F1 of 0.7687 and an EMR of 0.7400.
G. VM Verification and Multi-Abnormality Alerting: Retrieval-augmented VLM verification improved abnormality-aware screening over task-specific classifiers by leveraging retrieved visual references and textual descriptions.
G. VM Verification and Multi-Abnormality Alerting: Qualitative results showed close agreement between predicted and ground-truth ventricular volumes and successful identification of potential fetal brain abnormalities in most cases.The VLM workflow retrieves reference cases and provides verification, alerting suggestions, and potential abnormality types with alert scores.
G. VM Verification and Multi-Abnormality Alerting: Retrieval-augmented VLM verification offers a training-free extension for multi-abnormality alerting when diagnostic evidence is sparse, heterogeneous, and distributed across anatomical regions.It incorporates similar historical cases and clinically relevant textual descriptions to detect potential non-VM abnormalities beyond VM severity classification.

V. DISCUSSION AND CONCLUSION

VIFBA unifies ultrasound-video ventricular volumetry and fetal brain abnormality assessment by combining temporal representation learning with MRI-guided cross-modal alignment. The framework aims to provide a more affordable, widely deployable alternative to MRI while extending screening beyond isolated ventriculomegaly, with future work focused on broader validation and multimodal expansion.

Framework: VIFBA unifies fetal lateral ventricular volume estimation and brain abnormality assessment directly from ultrasound videos.The framework integrates an ultrasound foundation model, temporal modeling, and JEPA-inspired latent predictive learning.
Framework: MRI-guided cross-modal alignment transfers structural information to ultrasound representations during training while requiring only ultrasound at inference.This design addresses MRI’s high cost and limited availability for routine clinical use.
Abnormality screening: Retrieval-augmented vision-language models extend fetal brain ultrasound assessment toward multi-disease alerting beyond isolated ventriculomegaly.The approach targets diverse, coexisting, rare, or underrepresented abnormalities that conventional disease-specific models may not cover well.
Future work: Future work will validate VIFBA on larger multi-center cohorts, add finer-grained abnormality categories, and incorporate advanced multimodal foundation models with larger paired ultrasound–MRI datasets.These directions target generalizability across ultrasound systems and clinical settings, as well as robustness and clinical applicability.