CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentation

1. Introduction

• Domain shifts can reduce machine-learning performance and clinical applicability, while medical privacy and expert annotation costs make large annotated datasets unlikely for many problems.The paper motivates domain adaptation as a response to the scarcity of labelled medical data.Source: §1 · p. 2
• Existing unsupervised domain-adaptation techniques had mainly been tested on private, small, or single-class datasets, motivating a large, multiclass benchmark.The challenge was organized with MICCAI 2021 and is described as the first large, multiclass benchmark for unsupervised cross-modality domain adaptation.Source: §1 · p. 2
• CrossMoDA targets vestibular schwannoma and cochlea segmentation on hrT2 scans, a non-contrast alternative to ceT1 used for diagnosis and surveillance.The paper notes growing interest in hrT2 imaging because it mitigates risks associated with gadolinium-containing contrast agents and is more cost-efficient than ceT1 imaging.Source: §1 · p. 2

2. Related work

• WMH and SCGM evaluations use small test sets and single-class tasks, with domain shifts limited to the same modality or scanner setting.WMH test sets contain at most 10 scans; SCGM test sets contain at most 20 scans.Source: §2 · p. 3
• IVDM3Seg, MM-WHS, and CHAOS also have small test sets; MM-WHS supports multiclass cross-modality evaluation but has only four test cases.IVDM3Seg is a single-class task with four test scans, and CHAOS uses test sets of four MR and six CT scans.Source: §2 · pp. 3–4
• BraTS cross-modality adaptation is created by removing modalities and is described as having limited clinical relevance.The review distinguishes it from unsupervised pathology adaptation between tumour grades.Source: §2 · p. 4
• Segmentation evaluations for unsupervised domain adaptation used datasets that were private, small, or single-class.Private datasets prevent comparison with new methods, while test sets in several public benchmarks are also small.Source: §2 · pp. 3–4

3.1. Overview

• Participants adapted from annotated ceT1 scans to unannotated hrT2 scans to segment vestibular schwannoma and cochleas on hrT2.The challenge focused on two structures involved in vestibular schwannoma follow-up and treatment planning.Source: §3.1 · p. 4

3.2.1. Data overview

• The dataset extends the publicly available Vestibular-Schwannoma-SEG collection, with its open scans used for training and validation and a private extension reserved for testing.The test set was withheld from participants to prevent access to its scans.Source: §3.2.1 · p. 4

3.2.2. Image acquisition

• The target training, validation, and test sets had similar distributions of patient age, gender, operative status, and hrT2 acquisition features.All scans were acquired on a 1.5T Siemens Avanto scanner; hrT2 imaging used either CISS or TSE sequences.Source: §3.2.1 · p. 4, §3.2.2 · p. 4, Table 1 · p. 5

3.2.3. Annotation protocol

• Vestibular schwannomas were segmented by a neurosurgeon and physicist, while radiology fellows segmented both cochleas using hrT2 as reference.The annotations include the cochlear basal turn and modiolus; both cochleas were included because treating the remaining cochlea as background led to poor segmentation performance in preliminary fully supervised results.Source: §3.2.3 · p. 5

3.2.4. Data curation

• The data were de-identified, and no further image preprocessing was used because acquisition was consistent; training and validation data were released under CC-BY-4.0.VS contour lines were converted to label maps, and images and masks were distributed as compressed NIfTI files.Source: §3.2.4 · p. 5

3.3. Challenge setup

• Validation used automated submissions with a one-submission-per-day limit, while the private test set was evaluated through Docker containers with one submission allowed.The validation limit was intended to reduce supervised hyperparameter selection; all participant containers passed quality control.Source: §3.3 · p. 5

4.1. Choice of the metrics

• The evaluation selected Dice similarity coefficient (DSC) and average symmetric surface distance (ASSD) rather than relying on a single segmentation metric.The paper cites their simplicity, rank stability, and ability to assess segmentation accuracy as reasons for selection.Source: §4.1 · p. 6
• If a prediction contains only background, its ASSD is set to the maximum distance between test-set voxels, 350 mm.This specifies how the metric handles an empty predicted segmentation.Source: §4.1 · p. 6

4.2. Ranking scheme

• Team rank scores average ranks across cases, regions, and measures, then average cumulative ranks across patients; tied values receive the lowest tied rank.Teams are ranked by the resulting rank score.Source: §4.2 · p. 6
• Ranking stability was assessed by comparing the original ranking with rankings from 1,000 bootstrap samples of 137 test cases using Kendall’s τ.Each sample was drawn with replacement; on average, 63% of distinct cases were retained.Source: §4.2 · p. 6

5. Participating methods

• Teams bridged the domain gap with image-to-image translation, MIND cross-modal features, or discrepancy-based feature alignment.MIND features supported registration-based label propagation or segmentation ensembles; discrepancy methods included discriminative losses and minimal-entropy correlation.Source: §5 · p. 6
• Samoyed combined target-aware CycleGAN translation that preserves structure with self-training on real hrT2, ensembling 2D and 3D models.The team also applied all-but-largest-component suppression to vestibular schwannoma predictions.Source: §5 · pp. 6, 8
• PKU BIALAB paired pixel alignment with self-training, while jwc-rad used CUT translation and nnU-Net with reduced tumour-signal augmentation.PKU BIALAB used NiceGAN for alignment; jwc-rad reduced labelled tumour signal intensity by 50% for augmentation.Source: §5 · p. 8
• MIP combined three translators, Mean Teacher fine-tuning and CLEAN-LAB fusion; PremiLab used disentangled style transfer and attention, while Epione-Liryc regularized CycleGAN with Pair-Loss.Epione-Liryc also refined segmentations with majority voting, k-Means, Mean-Shift, and a dense CRF.Source: §5 · pp. 8–9
• MedICL ensembled 2.5D and 3D models with scanner augmentation; DBMI extended CUT to 3D and added detection, Hi-Lib used GAN-based translation augmentation, and smriti161096 self-trained on hrT2 pseudo-labels.Hi-Lib used CycleGAN and CUT to generate paired source-like augmentations; smriti161096 also pre-trained segmentation on ceT1 and generated pseudo-hrT2 images with 2D CycleGAN.Source: §5 · p. 9
• IMI propagated labels from registered source scans and trained nnU-Net on fused noisy labels; GapMIND trained Deeplab on MIND features with target pseudo-labels.IMI used STAPLE for label fusion, while GapMIND obtained target pseudo-labels through image registration and STAPLE fusion.Source: §5 · pp. 9–10
• gabybaldeon added adversarial feature-level adaptation after CycleGAN translation, whereas SEU Chen aligned domains by minimizing weighted source cross-entropy and target entropy.SEU Chen used the adapted nnU-Net for VS and an ensemble of its two models for cochlea segmentation.Source: §5 · p. 10
• skjp and IRA used adversarial domain classifiers with Gradient Reversal Layers to adapt segmentation networks trained on ceT1 and hrT2 scans.skjp trained separate networks for VS and cochleas; IRA attached its reversal layer to earlier network blocks.Source: §5 · p. 10

6.1. Overall segmentation performance

• Samoyed won with a rank score of 2.7 and was the only team with median DSC above 85% for both structures.Other top-five teams had median DSC above 80% for each structure.Source: §6.1 · p. 10
• All top ten teams used image-to-image translation; their medians were higher and IQRs smaller than those of other approaches.MIND-feature teams ranked eleventh and twelfth, while feature-alignment approaches ranked last.Source: §6.1 · p. 10

6.2. Evaluation per structure and impact on the rank

• VS performance was less robust: top-ten teams’ average DSC and ASSD IQRs were respectively 2.6 and 16 times larger for VS than for cochleas.The winner’s DSC was below 60% on 8% of VS cases (N=11), versus 1% of cochlea cases (N=1).Source: §6.2 · p. 10, Figure 2 · p. 11, Figure 3 · p. 11
• More uniform cochlea location, volume, and intensity may explain the greater robustness there and the collapse of feature-alignment methods on VS.The authors present this as a possible explanation.Source: §6.2 · p. 12
• Cochlea performance was more discriminative for ranking: the top seven teams’ median DSC ranged from 83.3% to 88.4% for VS and 75.2% to 85.7% for cochleas.The winner outperformed the others on cochleas, while the second team’s best VS performance did not win the challenge.Source: §6.2 · p. 12, Table 3 · p. 12, Table 4 · p. 12

6.3. Remarks about the ranking stability

• The proposed ranking had median [IQR] Kendall’s τ of 1 [1-1] across 1,000 bootstrap samples, and the winner ranked first in every sample.Each sample resampled 137 test cases with replacement.Source: §6.3 · p. 12, Figure 5 · p. 14
• Multi-metric ranking was more stable than single-metric ranking: median Kendall’s τ was 0.98 for DSC, 0.98 for ASSD, and 1 for their combination.The comparison used bootstrap samples.Source: §6.3 · p. 13, Figure 6 · p. 14
• The first seven ranks stayed the same across ranking-method variations, although aggregate-then-rank with the mean was less robust because missing segmentations produced ASSD outliers.The compared variants changed the aggregation technique or whether metrics were aggregated before or after ranking.Source: §6.3 · p. 13

7.1. Performance of automated segmentation methods

• Full supervision outperformed all teams: VS gap 4.1% median DSC and 0.11mm median ASSD; cochlea gap 2% median DSC and 0.03mm median ASSD, with tighter IQRs.The full-supervision model was trained on paired hrT2 scans and ceT1 scans with manual annotations.Source: §7.1 · p. 14

7.2. Analysis of the top-ranked methods

• Image-to-image translation was the most successful approach; all but one team used 2D translation on axial slices, using CycleGAN or its NiceGAN and CUT extensions.Source: §7.2 · p. 15
• Results did not establish which translation approach was best; identical 2D CUT implementations scored 83.1% and 71.7% median cochlea DSC, and the study could not identify optimal parameter combinations.Preprocessing and segmentation-network choices were also noted as performance factors.Source: §7.2 · p. 15
• The top three teams used nnU-Net, which automatically configures preprocessing, architecture, training, and postprocessing; nearly all teams used a U-Net backbone.Source: §7.2 · p. 15
• The two top-performing teams’ self-training significantly improved performance for both teams (Wilcoxon p < 0.01) and followed translation-based pseudo-target training.Self-training fine-tuned the trained networks to manage the small remaining gap between pseudo-target and real target images.Source: §7.2 · p. 15

7.3. Limitations and future directions for the challenge

• The challenge did not address robustness to acquisition-setting changes: each domain came from one centre and scanner, and 96% of test scans shared sequence parameters.Source: §7.3 · p. 15
• Image-to-image translation methods assume a bijective relationship between source and target domains.Source: §7.3 · p. 15

8. Conclusion

• Top-performing teams achieved high performance close to full supervision using image-to-image translation to generate pseudo-target images, then training segmentation networks with source-image annotations; self-training improved performance.Source: §8 · p. 15

• Reuben Dorent is credited with conceptualization, methodology, software, formal analysis, resources, data curation, writing, and visualization.Source: CRediT authorship contribution statement · p. 15
• Aaron Kujawa and Jonathan Shapey share credit for conceptualization, methodology, data curation, and writing; Shapey is also credited with funding acquisition.Source: CRediT authorship contribution statement · p. 15
• Sébastien Ourselin is credited with supervision and funding acquisition, while numerous contributors are credited with methodology and software.Source: CRediT authorship contribution statement · p. 15