PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond In-Distribution Metrics: A Systematic Out-of-Distribution Evaluation of Congenital Heart Disease Segmentation

Aniketh Vijesh, Shrisharanyan Vasu, Abhijit Ramesh, Clare Pomeroy-Ward, Harikrishnan Anil Maya, Sarin Xavier, Mahesh Kappanayil, Gilad Gressel

arXiv:2609.17068v2cs.CVcs.AI

TL;DR

CHD segmentation methods have rarely been tested across distribution shifts, and whether joint CT/CMR training improves OOD generalization remains open. This study systematically evaluates architectures, training settings, and pretraining strategies using ImageCHD as a held-out target. In-distribution ranking does not align with target performance: nnU-Net's 0.77 validation Dice falls to 0.51 on ImageCHD, while SwinUNETR reaches 0.67 Dice.

  • Problem

    CHD segmentation studies largely lack explicit OOD benchmarking, and whether joint CT/CMR training improves generalization remains untested.

    Source: §1 · p. 2

  • Method

    The study compares representative CHD segmentation architectures under CT+CMR training, CT-only training, and progressively expanded labeled ImageCHD adaptation, with self-supervised pretraining variants.

    Source: §1 · p. 2

  • Results

    nnU-Net's highest overall mean ID Dice of 0.77 falls to 0.51 on ImageCHD, while SwinUNETR achieves the strongest ImageCHD performance at 0.67 Dice.

    Source: §3.2 · p. 6

  • Takeaways & Limitations

    The contribution of joint CT/CMR training to ImageCHD transfer varies substantially by method, and these comparisons do not isolate architecture from other pipeline differences.

    Source: Single-Modality Training: Private CT… · p. 8

    The findings reflect the specific source-target shift from a complex-CHD-heavy private cohort to ImageCHD, which is dominated by septal defects; SwinUNETR's advantage is bounded to the evaluated datasets and settings.

    Source: §3.5 · p. 10

Abstract

from arXiv · show

Congenital heart disease (CHD) diagnosis and surgical planning often require patient-specific 3D anatomical models, but manual segmentation is labor-intensive, particularly in complex anatomies. Although deep-learning methods can automate this process, they are typically evaluated in-distribution, despite clinically relevant shifts in scanner, protocol, institution, population, and imaging modality. We present, to our knowledge, the first systematic evaluation of out-of-distribution (OOD) generalization in CHD segmentation, using ImageCHD as a held-out target cohort. We compare representative segmentation architectures under combined CT and CMR training, CT-only training, self-supervised pretraining, and limited target-domain adaptation. In-distribution performance proves to be a poor indicator of cross-cohort robustness: nnU-Net achieves the highest validation Dice (0.77) but falls to 0.51 on ImageCHD, while SwinUNETR generalizes substantially better, reaching 0.67 Dice. MAE and JEPA pretraining provide only modest additional benefit, suggesting that architecture contributes more to robustness than the tested pretraining strategies in this setting. When limited target-domain supervision is introduced, all SwinUNETR variants exceed 0.76 Dice with only 11 labeled ImageCHD cases. These findings demonstrate that conventional in-distribution evaluation can obscure clinically important generalization failures and support explicit cross-dataset testing as a key component of CHD segmentation evaluation.

1 Introduction

Complex congenital heart disease requires labor-intensive anatomical segmentation, while existing CHD methods have largely been evaluated only in-distribution. This study examines OOD generalization across model architectures, training modalities, and pretraining strategies, including whether joint CT/CMR training helps.

1 Introduction

  • CHD anatomical models support diagnosis, surgical planning, and catheter-based treatment, but accurate segmentation remains labor-intensive in complex anatomies.Cardiac CT and cardiovascular MR provide volumetric imaging for this work.Source: §1 · p. 2
  • Existing CHD segmentation studies do not explicitly benchmark OOD generalization, despite clinical shifts in scanners, protocols, institutions, populations, and modalities.Heterogeneous, long-tailed CHD anatomy may be poorly represented in any single training cohort.Source: §1 · p. 2
  • The study evaluates multi-modal CT/CMR training alongside architecture and pretraining strategy because whether joint training improves OOD generalization remains untested.Prior CHD work largely treats CT and CMR as separate settings.Source: §1 · p. 2
  • The authors present a systematic CHD OOD study using ImageCHD as a held-out target and compare architectures, pretraining strategies, training cohorts, and progressively added target labels.The evaluated models include nnU-Net, CardiacSeg, Zhu-Net, and SwinUNETR, with SwinUNETR trained from scratch or with MAE- or JEPA-style pretraining.Source: §1 · p. 2

2 Methodology

The evaluation compares segmentation models trained with combined CT+CMR, CT alone, or limited ImageCHD supervision, then assesses in-distribution and held-out OOD performance. It uses three patient-level cohorts and evaluates representative supervised and self-supervised segmentation methods.

2 Methodology

  • Models are evaluated on in-distribution validation and held-out ImageCHD data after combined CT+CMR training, CT-only training, or progressively expanded labeled ImageCHD subsets.Performance is measured with Dice, HD95, and ASSD.Source: §2 · pp. 2–3, Figure 1 · p. 3

2.1 Datasets for OOD Evaluation

  • Three patient-level cohorts simulate shifts in modality, institution, scanner, and CHD case mix, with public ImageCHD reserved as the OOD target.Training uses private 3D-Labs CT and public HVSMR-2.0 CMR; all cohorts share six evaluated cardiac structures.Source: §2.1 · pp. 3–4, Table 1 · p. 4

2.2 Segmentation Methods

  • The benchmark spans supervised and transformer-based methods, plus CardiacSeg and SwinUNETR with MAE or JEPA self-supervised pretraining.Dice measures volumetric overlap, while HD95 and ASSD measure boundary accuracy in millimetres; lower boundary-distance values are better.Source: §2.2 · pp. 4–5

3 Experiments and Results

Across combined CT+CMR training, CT-only training, and ImageCHD adaptation, SwinUNETR variants perform strongest on the held-out ImageCHD cohort, despite nnU-Net leading in-distribution validation. Performance varies by diagnosis and architecture, and the observed advantage is bounded by the evaluated cohorts and settings.

3.1 Implementation Details

  • Experiments use patient-level cohorts and repeated five-seed evaluation; the common pipeline resizes volumes to 128^3, while nnU-Net configures preprocessing separately.Self-supervised pretraining uses only the corresponding training pool, with no external data.Source: §3.1 · pp. 5–6

3.2 Results

  • SwinUNETR variants lead on ImageCHD: scratch reaches 0.67 Dice, while MAE-pretrained reaches 0.68 and JEPA-pretrained reaches 0.67; nnU-Net falls from 0.77 validation Dice to 0.51.The tested MAE and JEPA initializations provide only marginal gains over scratch SwinUNETR.Source: Figure 2 · p. 6, §3.2 · p. 6
  • Across most represented diagnoses, SwinUNETR variants retain their relative ImageCHD advantage, while qualitative examples show more coherent ImageCHD masks than other methods.Some least-represented and anatomically complex groups have lower performance, with comparisons cautioned by small case counts.Source: §3.2 · pp. 6–7, Figure 3 · p. 7, Figure 4 · p. 7
  • Removing HVSMR-2.0 lowers ImageCHD Dice across methods: scratch SwinUNETR drops from 0.67 to 0.64, while nnU-Net drops from 0.51 to 0.36.The degradation varies considerably by method; the study notes that these comparisons do not isolate architecture from other pipeline differences.Source: Single-Modality Training: Private CT… · pp. 7–8, Figure 5 · p. 8

3.3 Target-Domain Sample Efficiency

  • All SwinUNETR variants exceed 0.76 Dice after adding only 11 labeled ImageCHD cases, and pretraining's benefit is concentrated primarily in the zero-shot regime.Evaluation uses a fixed 22-case ImageCHD holdout as labeled cases are added in increments of 11.Source: §3.3 · pp. 8–9, Figure 6 · p. 9

3.4 Interpreting the SwinUNETR Effect

  • SwinUNETR's stronger ImageCHD performance may reflect its hierarchical shifted-window Transformer encoder and convolutional decoder, while pretraining gains may be limited by source-only data.The authors describe the architectural explanation as a possibility and note that the pretraining advantage diminishes when target labels are introduced.Source: §3.4 · p. 10

3.5 Limitations

  • Findings reflect a specific source-target case-mix shift, and the SwinUNETR advantage is limited to the evaluated datasets and experimental settings.The authors also note that fixed 128^3 resizing may reduce fine anatomical detail and broader hyperparameter tuning could improve performance.Source: §3.5 · p. 10, §3.3 · p. 10

4 Conclusion

The evaluation finds that in-distribution validation rankings do not predict ImageCHD performance, with SwinUNETR variants strongest on the held-out cohort. The authors support explicit cross-dataset testing while limiting conclusions to the evaluated settings pending further cohorts and controlled comparisons.

  • In-distribution validation rankings do not align with ImageCHD performance: nnU-Net leads validation, while SwinUNETR variants achieve the strongest held-out performance and target-domain sample efficiency.Source: §4 · p. 10
  • Joint CT+CMR training improves ImageCHD transfer for most evaluated methods, though the magnitude varies considerably across models.Source: §4 · p. 10
  • MAE and JEPA pretraining provide only modest gains under combined CT+CMR training and no improvement over scratch SwinUNETR in the CT-only experiment.Source: §4 · p. 10
  • Further target cohorts and controlled comparisons are needed before attributing findings conclusively to architecture or extending them broadly to CHD distribution shifts.Source: §4 · p. 10

Related papers

Loading 2609.17068v2…