Towards Whole-Study Screening for Congenital Heart Disease in Fetal Ultrasound Using Multiple Instance Learning

• Prenatal CHD detection remains below fifty percent and varies widely across institutions, despite CHD being the most common birth defect.Detection is worse in rural and underserved areas, where access to expertise and resources is limited.Source: §I · p. 1
• Current pipelines rely on manually selected or view-classified frames, and standard-plane classifiers may discard abnormal cardiac images before diagnosis.These image-level designs also require per-frame predictions to be combined into a subject-level diagnosis.Source: §I · p. 2
• The proposed design processes the whole study, gates cardiac frames, and uses MIL to produce a subject-level prediction with ranked supporting frames.MIL learns from study-level labels without frame-level diagnostic supervision.Source: §I · p. 2
• The disease-robust cardiac-frame identifier remains reliable on abnormal hearts, while a hierarchical severity head improves sensitivity to under-represented non-critical CHD over a flat classifier.The framework pairs self-supervised representation learning with a transformer-based MIL aggregator.Source: §I · pp. 2–3
• On external CARDIUM, label-free CORAL adaptation raises the proposed model's transfer AUC from 0.513 to 0.944.CARDIUM was collected at another institution and contributes curated cardiac frames rather than full sweeps.Source: §I · pp. 2–3

A. Prenatal CHD diagnosis

• Prenatal CHD systems span view classifiers, segmentation, lesion-specific models, and foundation models, but high performance on curated views is rarely established on whole studies or across sites.This work targets that gap directly.Source: A. Prenatal CHD diagnosis · p. 3

B. Multiple-instance learning

• MIL learns bag-level labels without instance-level supervision, and prior medical applications include pathology, imaging, and video.Existing distribution-shift-robust MIL methods operate on curated bags, whereas this work builds bags from raw studies by identifying cardiac frames first.Source: B. Multiple-instance learning · p. 3

C. Domain adaptation

• Domain-adaptation methods align feature statistics, learn domain-invariant representations, or refine target decision boundaries; this work adopts label-free CORAL for whole-study feature representations.CORAL aligns second-order statistics without target annotations.Source: C. Domain adaptation · p. 3

A. Problem formulation and overall framework

• Studies receive a single label for healthy control, non-critical CHD, or critical CHD, with binary detection nested within severity grading.Critical CHD requires neonatal catheter-based or surgical intervention; non-critical CHD does not, and no frame-level diagnostic labels are required.Source: A. Problem formulation and overall… · p. 3
• The MIL formulation treats the subject as a bag and each frame as an instance, requiring permutation-invariant aggregation of frozen frame embeddings into a bag-level prediction.The factorization uses an instance embedding, learned aggregator, and bag classifier.Source: A. Problem formulation and overall… · p. 3
• The pipeline pre-trains a frame encoder, gates retained frames using a cardiac-frame identifier, and aggregates the gated bag with a transformer and learnable classification token.Attention over frames ranks supporting frames for review; shared features feed binary detection and detect-then-grade severity heads.Source: A. Problem formulation and overall… · pp. 3–4
• Cross-cohort transfer involves appearance, prevalence, and bag-composition shifts, addressed respectively by feature alignment, label-free inference rules, and cardiac gating with a fixed frame cap.FUSE training studies have a median of 298 frames versus 4 curated frames in CARDIUM.Source: A. Problem formulation and overall… · p. 4

B. Representation learning (Stage 1)

• The ViT-L/16 frame encoder is trained from random initialization with a masked-autoencoder objective on 369,846 unlabeled fetal ultrasound frames from 1,650 fetuses.In-domain test subjects and the external cohort were withheld from pre-training, so no evaluation subject influenced the encoder.Source: B. Representation learning (Stage 1) · p. 4
• The masked-autoencoder objective masks 70% of input patches and reconstructs standardized pixels for masked patches, with visible patches excluded from the loss.A lightweight decoder reconstructs the masked patches from encoder output.Source: B. Representation learning (Stage 1) · p. 4
• Ultrasound-specific augmentation and corpus foreground-pixel normalization are used instead of natural-image statistics.Augmentations include crops, flips, affine transforms, and Gaussian blur.Source: B. Representation learning (Stage 1) · p. 4

C. Frozen frame embedding

• The pre-trained encoder is used strictly as a frozen feature extractor and is never fine-tuned for any downstream task.Every frame is embedded using the same representation, so downstream experiments compare gating, aggregation, and adaptation strategies over a fixed representation.Source: C. Frozen frame embedding · p. 4

D. Cardiac-frame identification

• The cardiac-frame identifier classifies frames as non-target, abdominal, or cardiac, with cardiac covering five target planes.It is a probe on the same frozen embeddings, so gating adds a head rather than a second backbone.Source: D. Cardiac-frame identification · p. 5
• Training the identifier on both normal and CHD studies improves cardiac-view recognition on CHD studies without reducing performance on normals.The passage motivates disease-inclusive training because malformed hearts are where a normal-anatomy-trained view classifier fails.Source: D. Cardiac-frame identification · p. 5
• The calibrated gate passes forward only frames whose cardiac probability clears a confidence threshold.The threshold is fitted on a held-out annotated split, not on test or external data.Source: D. Cardiac-frame identification · p. 5

E. Transformer-based MIL aggregator (Stage 2)

• Self-attention lets each frame’s contribution depend on the full bag, unlike instance-independent attention pooling.The paper motivates this choice with views interpreted relative to other views in the same study and apparent abnormalities checked against better frames of the same plane.Source: E. Transformer-based MIL aggregator… · p. 5
• A shallow, two-layer transformer processes gated frame embeddings and a learnable classification token to produce the bag representation.The aggregator uses no positional encoding, making its read-out permutation-invariant to frame order, while remaining dependent on bag composition and size.Source: E. Transformer-based MIL aggregator… · pp. 5–6
• The classification token’s last-layer attention, averaged over heads and renormalized over valid frames, ranks supporting frames for clinician review.The paper cautions that the aggregator’s default per-frame weights are off the loss path and have no interpretive meaning; rankings use Eq. (7).Source: E. Transformer-based MIL aggregator… · p. 6

F. Training objectives

• Focal loss with inverse-frequency class weights addresses imbalance, while the severity term uses a hard CHD-masked grading loss.The class weights are based on training-partition subject frequencies; the reported implementation uses no label smoothing or resampling in the main training loop.Source: F. Training objectives · p. 6
• Two binary heads share the pooled representation: one detects CHD, and the other distinguishes non-critical from critical CHD.The heads are combined by the chain rule, yielding a hierarchical formulation rather than two independent models.Source: F. Training objectives · p. 6
• The grading term is masked to CHD-positive bags, normalized by their count, and defined as zero when a batch contains no CHD bag.Healthy-control bags contribute exactly zero gradient to the grading head; weighting factors balance subclass recall.Source: F. Training objectives · p. 6
• The binary CHD detection score sums the predicted probabilities for non-critical and critical CHD, while severity is decoded with two thresholds.The thresholds allow detection and grading to be operated independently.Source: F. Training objectives · p. 6

G. Cross-site adaptation

• Deep CORAL adds a differentiable penalty matching second-order statistics of source and target representations to the supervised objective.The source batch is labeled and the target batch is unlabeled.Source: G. Cross-site adaptation · p. 7
• Alignment is applied at the 32-dimensional penultimate bag embedding, the last representation that is both trainable and task-relevant.The frame encoder is frozen, while aligning two-dimensional logits would directly conflict with classification; the penalty is zero if either batch has fewer than two bags.Source: G. Cross-site adaptation · p. 7
• CARDIUM labels are never used in any loss, metric, or threshold during training.Unlabeled CARDIUM bags are drawn through a separate cyclic loader and constructed using the same gate and threshold as the source bags.Source: G. Cross-site adaptation · p. 7

A. Datasets and cohort

• FUSE combines an in-house cohort and three public fetal-ultrasound datasets, with an independent external cohort reserved for out-of-domain evaluation.FUSE contains 869 subjects split into 692 training and 177 in-domain test subjects.Source: A. Datasets and cohort · p. 7
• The in-house data span seven institutions, at least 13 Philips and GE scanners, gestational ages of 18–28 weeks, and three levels of care.Postnatal echocardiography provides the reference standard.Source: A. Datasets and cohort · p. 7
• Public sources are FETAL PLANES DB, MFUSPAC, and RFCHD; only diagnostically evaluable subjects enter diagnosis splits.Subjects without an acceptable-quality cardiac-target-plane frame are used only for self-supervised pre-training.Source: A. Datasets and cohort · p. 7
• Frame-level annotations cover 479 subjects and five cardiac target planes, with 406 HC, 36 CCHD, and 37 NCCHD subjects.Annotations also include abdominal or non-target labels and image quality on a 0–2 scale.Source: A. Datasets and cohort · p. 7
• CARDIUM contains 790 cases, including 767 HC and 23 CHD, and its labels are excluded from training, model selection, and threshold selection.Its unlabeled bags are used only as the alignment target; the pre-training corpus also includes FASSD abdominal images and excludes CARDIUM and all in-domain test subjects.Source: A. Datasets and cohort · p. 7

B. Preprocessing

• Scanner overlays, including text, measurements, and annotations, are removed because they may encode site-specific shortcuts rather than cardiac anatomy.Each study is de-identified before overlay removal.Source: B. Preprocessing · pp. 7–8
• Frames are cropped to the active ultrasound region, and frames predominantly dark within that region are excluded.The retained region is converted to 224×224 model input.Source: B. Preprocessing · p. 8
• For video clips, one frame is randomly sampled from every ten consecutive frames to reduce redundancy while preserving sweep coverage.The same preprocessing pipeline is applied across cohorts to avoid introducing cross-site differences through data preparation.Source: B. Preprocessing · p. 9

C. Training configuration

• The frame encoder is a ViT-L/16 trained from random initialization with masked-autoencoder learning on 369,846 masked B-mode frames from 1,650 fetuses.Pre-training masks 70% of input frames and reconstructs normalized pixels.Source: C. Training configuration · p. 9
• A cardiac-frame identifier is trained on frozen frame embeddings, and its selected frames form the bags used to train the MIL aggregator and task heads.The identifier uses focal loss; the aggregator and heads use AdamW, with diagnosis focal loss and a two-stage severity objective.Source: C. Training configuration · p. 9

D. Evaluation protocol

• AUC is the primary threshold-free metric; sensitivity and specificity are reported at the Youden-optimal threshold fitted on validation data.The minimum-validation-loss checkpoint is evaluated once on the in-domain test set and CARDIUM.Source: D. Evaluation protocol · p. 9

E. Internal CHD detection

• AUC 0.985 and specificity 0.990 for the proposed model exceed NATMED (0.861 / 0.600) and FetalCLIP (0.867 / 0.710) on 177 internal test studies.Fig. 3(A) plots AUC, sensitivity, and specificity for the three models.Source: E. Internal CHD detection · p. 9

F. Cross-site domain adaptation

• External AUC rises from 0.513 to 0.944 with label-free CORAL adaptation for the proposed model, while NATMED improves from 0.506 to 0.557 and FetalCLIP from 0.571 to 0.616.Without the cardiac-frame gate, the same model has external AUC 0.18 before adaptation; Table IV reports transfer AUC by model configuration.Source: F. Cross-site domain adaptation · p. 9, Table IV · p. 10

G. Representation quality

• The proposed encoder’s unsupervised UMAP representation shows more distinct HC and CHD regions than ResNet or FetalCLIP, with remaining overlap near the class boundary.The projection averages apical four-chamber frame features and uses no diagnosis labels.Source: G. Representation quality · pp. 9–10, Figure 5 · p. 10

H. Cardiac-view identification performance

• The cardiac-frame identifier achieves cardiac-view AUC 0.925 on HC studies and 0.838 on CHD studies, outperforming NATMED and SonoNet.Recall is lower on CHD than HC frames: 0.584 versus 0.757.Source: H. Cardiac-view identification… · p. 10

I. Hierarchical severity classification

• The hierarchical classifier raises non-critical CHD sensitivity from 0.188 to 0.688 and critical CHD sensitivity from 0.846 to 0.923, with specificity 0.968.Across the CHD subclasses, balanced accuracy rises from 0.517 to 0.805 and AUC from 0.303 to 0.841.Source: §I · p. 10, Figure 6 · p. 10

J. Attention interpretability

• Top-ranked frames are diagnostic-quality cardiac views, predominantly apical four-chamber; cardiac frames receive more attention than their share of each study, while non-cardiac frames receive less.Figure 7 shows the five highest-attention frames for two HC and two CHD subjects; Figure 8 compares attention fractions by cardiac versus pooled non-cardiac views and marks mean frame fractions.Source: J. Attention interpretability · p. 10, Figure 7 · p. 11, Figure 8 · p. 11

K. Ablation study

• AUC 0.958 with whole-study MIL rises to 0.985 with cardiac gating on the internal test set; on CARDIUM, gating raises AUC from 0.180 to 0.513 and CORAL raises it to 0.944.The table isolates whole-study MIL, cardiac-frame gating, and label-free CORAL; gating selects frames while CORAL aligns feature distributions across domains.Source: K. Ablation study · pp. 10–11, Table V · p. 11

V. CONCLUSION

• The whole-study framework combines self-supervised representations, disease-robust cardiac-frame identification, and transformer-based MIL to produce case-level diagnoses and severity from study-level labels, while returning supporting frames.The conclusion presents this approach as removing the assumption that cardiac frames are preselected by a clinician.Source: §V · p. 11
• The test set yields AUC 0.985 at specificity 0.990; on an independent external cohort, label-free CORAL adaptation yields AUC 0.944 when limited to identified cardiac frames.Cross-site experiments show that models trained on one dataset require adaptation before deployment at a new site because of differences in scanners, populations, disease prevalence, and acquisition protocols.Source: §V · p. 11