PaperlayerView exact revision on arXiv

Source-linked AI summary

SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining

Benjamin Billot, Douglas N. Greve, Oula Puonti, Axel Thielscher, Koen Van Leemput, Bruce Fischl, Adrian V. Dalca, Juan Eugenio Iglesias

arXiv:2107.09559v4eess.IVcs.CV

TL;DR

Brain MRI segmentation is challenged by variation in contrast and resolution, while tools for broad variability remain needed. SynthSeg trains on synthetic scans with randomized contrast and resolution; across 5,000 scans spanning eight datasets, six modalities, and 10 resolutions, it maintained uniform accuracy and considerably outperformed domain adaptation on larger shifts.

  • Problem

    Brain scans vary substantially in contrast and resolution, while tools that handle this variability are lacking, limiting quantitative morphometry across clinical datasets.

    Source: §1.1 · p. 2

  • Method

    SynthSeg trains a segmentation network on synthetic scans generated on the fly with randomized contrast, resolution, morphology, artefacts, and noise.

    Source: §3.1 · p. 3

  • Results

    Across 5,000 scans spanning eight datasets, six modalities, and 10 resolutions, SynthSeg maintained uniform accuracy, nearly matching supervised CNNs on their training domains and considerably outperforming domain adaptation for larger shifts.

    Source: §7 · p. 12

  • Takeaways & Limitations

    SynthSeg can reduce training-labeling burden by requiring only one set of anatomical segmentations rather than paired images and labels for every new domain.

    Source: §6 · p. 12

    Evaluation relied heavily on automated label maps, although some datasets used manual segmentations and others underwent visual quality control.

    Source: §6 · p. 12

Abstract

from arXiv · show

Despite advances in data augmentation and transfer learning, convolutional neural networks (CNNs) difficultly generalise to unseen domains. When segmenting brain scans, CNNs are highly sensitive to changes in resolution and contrast: even within the same MRI modality, performance can decrease across datasets. Here we introduce SynthSeg, the first segmentation CNN robust against changes in contrast and resolution. SynthSeg is trained with synthetic data sampled from a generative model conditioned on segmentations. Crucially, we adopt a domain randomisation strategy where we fully randomise the contrast and resolution of the synthetic training data. Consequently, SynthSeg can segment real scans from a wide range of target domains without retraining or fine-tuning, which enables straightforward analysis of huge amounts of heterogeneous clinical data. Because SynthSeg only requires segmentations to be trained (no images), it can learn from labels obtained by automated methods on diverse populations (e.g., ageing and diseased), thus achieving robustness to a wide range of morphological variability. We demonstrate SynthSeg on 5,000 scans of six modalities (including CT) and ten resolutions, where it exhibits unparalleled generalisation compared with supervised CNNs, state-of-the-art domain adaptation, and Bayesian segmentation. Finally, we demonstrate the generalisability of SynthSeg by applying it to cardiac MRI and CT scans.

1. Introduction · 2. Related works

SynthSeg uses fully randomized synthetic training data to segment brain scans across contrasts and resolutions without retraining or fine-tuning. The paper reports strong generalization against existing methods and presents the approach as a way to analyze heterogeneous clinical scans.

1.1. Motivation

  • Manual brain delineation is costly and variable, while clinical scans vary in orientation, spacing, resolution, partial voluming, and contrast, limiting automated morphometry at scale.The paper argues that tools spanning contrasts, resolutions, and populations could unlock analysis of clinical scans currently left unexplored in hospital PACS.Source: §1.1 · pp. 1–2

1.2. Contributions

  • SynthSeg trains on scans generated from anatomical labels, fully randomizing orientation, contrast, resolution, and artifacts, with randomized preprocessing so it handles scans with or without preprocessing.The scans are generated on the fly from a Bayesian-inspired model, and the network is not exposed to real scans during training.Source: §1.2 · p. 2
  • Because training requires label maps rather than real images, SynthSeg can use automatically generated maps from diverse populations and needs training only once for new contrasts or resolutions.The approach combines contrast and resolution robustness without retraining, unlike supervised CNNs and domain adaptation methods.Source: §1.2 · p. 2
  • SynthSeg approaches supervised CNN accuracy on their training domains, outperforms domain adaptation and Bayesian segmentation on all tested datasets, and achieves state-of-the-art cross-modality cardiac segmentation.The paper frames its contribution as robustness to both contrast and resolution without retraining, extending earlier work on contrast adaptiveness and partial-volume simulation.Source: §1.2 · p. 2

2. Related works

  • Bayesian segmentation estimates an unsupervised intensity likelihood per test scan, but takes about 15 minutes per scan and struggles with low-resolution partial voluming.Inferring high-resolution labels from low-resolution scans can become intractable because it requires marginalizing over possible high-resolution label configurations.Source: §2 · p. 3
  • Supervised CNNs, including UNet, are fast and accurate on training domains but generalize poorly to unseen contrasts and resolutions; modality dropout does not cover unseen domains.Data augmentation, adversarial augmentation [12], and contrastive learning improve intra-modality generalization but remain insufficient cross-modality.Source: §2 · p. 3
  • Domain adaptation bridges a source domain to a specific target through feature-space mapping, image translation [34], or both [11], but still requires retraining for each new domain.Test-time adaptation methods [46] achieve state-of-the-art intra-modality adaptation results through light fine-tuning at test time.Source: §2 · p. 3
  • Earlier synthetic-data methods generate scans with surrogate domain variations and constructed ground truth, but face a reality gap and still require domain-specific retraining.Domain randomization instead expands synthetic-data variability by randomizing generation parameters; this paper combines it with a Bayesian-inspired model to target brain MRI segmentation across contrasts and resolutions without retraining.Source: §2 · p. 3

3. Methods

SynthSeg trains a segmentation network on synthetic image–target pairs generated on the fly, randomizing contrast, resolution, morphology, artefacts, and noise. Its central contribution is domain randomization, enabling exposure to varied image conditions without relying on real training images.

3.1. Generative model

  • Training images are generated on the fly with randomized contrast, resolution, morphology, artefacts, and noise at each mini-batch.The generative model samples synthetic scans conditioned on segmentations.Source: §3.1 · p. 3

3.1.1. Label map selection and spatial augmentation

  • Training label maps can be manual, automatic, or combined if they share a labeling convention, and are spatially augmented with affine and diffeomorphic transforms.The nonlinear deformation is built from a Gaussian-sampled 10 × 10 × 10 × 3 velocity field and integrated to avoid holes or foldings.Source: §3.1.1 · pp. 3–4

3.1.2. Initial HR synthetic image

  • An initial high-resolution synthetic scan is sampled from a Gaussian mixture model conditioned on the augmented labels, with means and standard deviations randomized each mini-batch.The standard deviations jointly model tissue heterogeneity and scanner thermal noise.Source: §3.1.2 · p. 4

3.1.3. Bias field and intensity augmentation

  • Synthetic intensities are augmented with a smooth multiplicative bias field, rescaling to [0, 1], and a random Gamma transform.The Gamma exponent is sampled in the logarithmic domain from a zero-mean Gaussian distribution.Source: §3.1.3 · p. 4

3.1.4. Simulation of resolution variability

  • Acquisition direction, slice spacing, and thickness are randomized; images are blurred to simulate thickness, downsampled to low resolution, then upsampled to high resolution for training.The thickness blur approximates the slice excitation profile, and the resulting training target remains a crisp high-resolution segmentation.Source: §3.1.4 · pp. 4–5

3.1.5. Model output and segmentation target

  • The central contribution is domain randomization: hyperparameters controlling the uniform priors are tuned on a validation set and analyzed for sensitivity.The generated inputs pair a scan with a target segmentation formed from the deformed label map, with unwanted labels reset to background.Source: §3.1.5 · p. 5

3.2. Segmentation network and learning

  • A 3D U-Net is trained on sampled image–target pairs with soft Dice loss and Adam for 300,000 steps at a 10^-4 learning rate.The five-level network uses batch normalization, skip connections, and 3 × 3 × 3 convolutions; training takes around seven days on an Nvidia Quadro RTX 6000 GPU.Source: §3.2 · p. 5

3.3. Inference

  • At inference, scans are resampled and intensity-normalized, predictions from original and left-right-flipped inputs are averaged, and the largest connected component is retained per label.Inference averages about ten seconds per scan on an Nvidia TitanXP GPU, including preprocessing and postprocessing.Source: §3.3 · p. 6

4. General Experimental Setup

The experiments assess SynthSeg across diverse brain datasets, populations, modalities, and resolutions, using label-only training and comparisons with supervised, domain-adaptation, and Bayesian methods.

4.1. Brain scans and ground truths

  • Experiments span eight datasets (5,000 scans, six modalities, ten resolutions), including heterogeneous T1, ageing/AD, clinical FLAIR, CT, and simulated low-resolution scans.The ten test resolutions comprise 1 mm^3 and 3, 5, or 7 mm spacing along each of the axial, coronal, and sagittal directions; simulated downsampling enables assessment by resolution, while FLAIR represents real-life slice selection profiles.Source: §4.1 · p. 6, Table 1 · p. 6
  • Manual labels are available for T1-39 and MSp, while FreeSurfer segmentations provide visually quality-controlled ground truths for the other datasets.FreeSurfer is described as robust across independent T1 datasets, with Dice scores of 0.85–0.88; the authors treat scores above 0.85 as equally good and use its silver standard to assess Dice below that level.Source: §4.1 · pp. 6–7

4.2. Training segmentations and population robustness

  • SynthSeg trains on 20 T1-39, 500 HCP, and 500 ADNI label maps, with no real scans, to broaden morphological variety across populations.Synthetic images are designed to align perfectly with their segmentations, permitting automated labels; extra-cerebral labels from a Bayesian approach are dropped with 50% probability, lesion labels are pasted with 50% probability, and training maps are flipped and cropped to 160^3 volumes.Source: §4.2 · p. 7

4.3. Competing methods

  • Comparators are the supervised T1 baseline, nnUNet, test-time adaptation (TTA) [46], SIFA [11], and Bayesian SAMSEG.Learning-based methods are trained twice and selected using validation data; methods are assessed with hard Dice and 95th-percentile surface distance (SD95, mm).Source: §4.3 · p. 7

5. Experiments and Results

The section outlines four experiments: comparisons with competing methods, an ablation study, a neuroimaging group study, and an extension to cardiac MRI and CT.

5.1. Robustness against contrast and resolution

  • On T1-39, SynthSeg reaches 0.88 versus 0.91 Dice for supervised networks; on other T1 datasets, it improves mean Dice by 2.5 points over the T1 baseline and has higher lower-quartiles.Source: §5.1 · p. 7
  • For the same subject, SynthSeg produces almost identical features for 1 mm T1 and 5 mm axial T2 scans, effectively super-resolving the latter to 1 mm.The T1 baseline produces noise outside its training domain, whereas SynthSeg maintains a consistent representation across contrasts and resolutions.Source: §5.1 · p. 7, Figure 5 · p. 8
  • SynthSeg achieves the best scores on all nine target domains, with statistical significance for six Dice results and all nine SD95 results.The comparison covers competing methods across datasets at native resolution.Source: §5.1 · p. 7, Table 2 · p. 8
  • TTA and SIFA improve supervised-network generalisation, but SynthSeg outperforms them across contrasts and resolutions without retraining.TTA results fluctuate for larger domain gaps, while SIFA partly corrects this; on CT, SIFA improves SD95 by 14.92 mm.Source: §5.1 · p. 8
  • SAMSEG averages 0.83 Dice across 1 mm MR contrasts, but falls to 0.71 on 3 mm CT and 0.64 on 5 mm FLAIR, respectively 5 and 14 points below SynthSeg.Source: §5.1 · p. 8
  • SynthSeg loses only 3.8 Dice points on average from 1 mm to 7 mm slice spacing, compared with SAMSEG’s 7.6-point loss, mainly due to thin structures such as the cortex.SynthSeg maintains good accuracy across tested resolutions despite information loss and partial-volume effects at low resolution.Source: §5.1 · p. 8

T1mix FSM-DBS FLAIR CT

Across experiments, SynthSeg maintained segmentation accuracy across contrast and resolution shifts and extended to cardiac MRI and CT without retraining. The results also show how domain randomisation and diverse training label maps contribute to robustness.

T1mix FSM-DBS FLAIR CT

  • SynthSeg produced very accurate segmentations across contrasts and resolutions, while the T1 baseline erred outside its training domain, TTA degraded across larger domain gaps, and SAMSEG suffered low-resolution PV effects.TTA performed very well on T1mix, and SAMSEG yielded coherent results at 1 mm resolution.Source: Figure 6 · p. 9
  • On T1 and MSp-T1, SynthSeg was clearly superior, and its gap to training-data performance narrowed as resolution fell, nearly vanishing at 7 mm.This narrowing made SynthSeg particularly useful for low-resolution scans.Source: T1mix FSM-DBS FLAIR CT · p. 9

5.2. Ablations on DR and training label maps

  • SynthSeg-R and SynthSeg-RC lost 1.4 and 2.6 average Dice points; overly tight GMM priors lost 2.1, while bias, deformation, and lesion ablations lost 3.7, 4.3, and 3.9 points.Slightly tighter GMM priors performed almost identically to SynthSeg; lesion ablation mainly affected ADNI and FLAIR.Source: §5.2 · p. 9, Figure 9 · p. 10
  • One training map yielded Dice scores of 0.68–0.80; scores plateaued at five maps (10 for MSp-PD), and SynthSeg converged with fewer examples than the T1 baseline.Adding automated maps raised Dice by 1.9 and 2.0 points on ADNI and FLAIR, respectively; among 30 ADNI subjects with the largest ventricles, training on all maps improved scores by 4.7 points over manual maps alone.Source: §5.2 · p. 10, Figure 10 · p. 10, Figure 11 · p. 11

5.3. Alzheimer’s Disease volumetric study

  • For hippocampal atrophy, both methods were close to the ground truth on HR T1, but SAMSEG achieved less than half the expected effect size on FLAIR while SynthSeg remained closer to the HR reference.The FLAIR scans were 5 mm axial, with the hippocampus visible in only two to three slices and affected by heavy partial-volume effects.Source: §5.3 · p. 11, Table 3 · p. 11

5.4. Extension to cardiac segmentation

  • All seven cardiac regions scored above 0.8 Dice; mean Dice was 0.84 for MMWHS MRI and 0.88 for MMWHS CT.Scores were similar to Chen et al. 11’s 0.82 cross-modality result, though the resolutions differed (1 mm for SynthSeg versus 2 mm). SynthSeg was not retrained on the cardiac datasets.Source: Table 4 · p. 11, §5.4 · pp. 11–12, Figure 12 · p. 11

6. Discussion

SynthSeg uses synthetic scans with fully randomized contrast and resolution to segment brain scans, including CT, without retraining or fine-tuning. It requires anatomical segmentations rather than real training images, and automated labels can improve robustness to morphological variability.

6. Discussion

  • SynthSeg’s fully randomized synthetic contrast and resolution train a network to learn domain-independent features, enabling segmentation without retraining or fine-tuning.The synthetic samples deliberately make no attempt at realism.Source: §6 · p. 12
  • Contrast- and resolution-specific SynthSeg variants perform worse; randomization may mitigate model assumptions and reflects evidence that augmentation beyond realism can improve generalization.The authors suggest both effects may explain the domain-constrained variants’ poorer performance.Source: §6 · p. 12
  • SynthSeg reduces labeling effort by requiring one set of anatomical segmentations, often fewer training examples than supervised CNNs, and no paired real images for each domain.The authors recommend adding visually checked automated segmentations, which can improve robustness to morphological variation such as ageing; synthetic scans are aligned with their ground truths by design.Source: §6 · p. 12
  • Training labels must cover test-scan tissues, but intensity clustering can supply missing structures; SynthSeg handles mild within-structure or spatially distant deviations from its Gaussian model.Intensity clustering enabled state-of-the-art cardiac segmentation despite original labels omitting finer structures and surrounding tissues.Source: §6 · p. 12
  • Although evaluation used many automated labels, manual-label datasets and visual quality control were included, and generalization remained strong with either manual or automated ground truths.Indirect evaluation also supports SynthSeg’s accuracy and clinical utility.Source: §6 · p. 12
  • SynthSeg can be deployed on heterogeneous clinical scans without retraining, avoiding reliance on rarely available 1 mm T1 scans or clinic-based neural-network training expertise.It achieves results comparable to supervised CNNs on their training domain, especially at low resolution.Source: §6 · p. 12

7. Conclusion

SynthSeg segments brain MRI and CT scans across contrasts and resolutions without retraining or fine-tuning. Its broad generalisation supports clinical neuroimaging and suggests potential for other medical imaging problems.

7. Conclusion

  • Across 5,000 scans from eight datasets, six modalities, and 10 resolutions, SynthSeg maintained uniform accuracy, nearly matched supervised CNNs on their training domain, and considerably outperformed domain adaptation for larger shifts.It scored slightly better than domain-adaptation methods for small domain gaps.Source: §7 · p. 12
  • SynthSeg was more accurate than Bayesian segmentation, robust to PV effects, and much faster; it detected AD atrophy on HR and LR scans and achieved state-of-the-art cardiac cross-modality segmentation.Source: §7 · p. 12
  • Future work will explore adversarial networks for synthetic-scan quality, CNN denoising, alternative architectures, and performance across lesions, tumours, and pathologies.The lesion-simulation ablation provides initial evidence of robustness to lesions.Source: §7 · pp. 12–13
  • Distributed with FreeSurfer, the single trained model removes retraining requirements and can enable reproducible segmentations and quantitative analyses of large existing clinical datasets.This could improve the characterisation and diagnosis of neurological disorders.Source: §7 · p. 13

Supplement 1: Examples of generated synthetic scans

Representative generated synthetic scans show broad diversity in image appearance and anatomy.

Supplement 1: Examples of generated synthetic scans

  • Generated scans vary in contrast, resolution, size, shape, bias fields, skull stripping, lesions, and anatomical morphology.Fig. S1 shows representative samples from the presented generative model.Source: Figure S1 · p. 16

Supplement 2: Values of the generative model hyperparameters

Table S1 documents the generative model hyperparameters and the units and input range used for interpreting them.

Supplement 2: Values of the generative model hyperparameters

  • Table S1 lists generative-model hyperparameters, with intensity parameters assuming inputs in [0, 255], rotations in degrees, and spatial measures in millimeters.Source: Table S1 · p. 17

Supplement 3: List of label values used during training

Table S2 lists the labels used for image synthesis, prediction, and evaluation, and describes how skull stripping and contralateral labels are modeled during generation.

Supplement 3: List of label values used during training

  • During generation, skull stripping is modeled in 50% of cases, with imperfect stripping in a further 25% of total cases; contralateral structures use distinct R/L labels.Imperfect stripping removes extra-cerebral labels except cerebro-spinal fluid.Source: Table S2 · p. 17

Supplement 4: Versions of the training label maps

Training label maps vary lesion-label inclusion and skull-stripping quality to build robustness to white matter lesions and imperfect skull stripping.

Supplement 4: Versions of the training label maps

  • Training maps include versions with or without lesion labels and with perfect, imperfect, or no skull stripping to build robustness to lesions and skull stripping variation.Lesion labels are mainly in cerebral white matter, with additional occurrences in the cerebellum, thalamus, pallidum, and putamen.Source: Figure S2 · p. 18
  • Lesion labels are obtained with FreeSurfer [23] and pasted onto the existing label maps.Source: Figure S2 · p. 18
  • Extra-cerebral labels are obtained with a Bayesian segmentation approach [56].Source: Figure S2 · p. 18

Supplement 5: Modifications to the nnUNet, and TTA, and SIFA methods · Supplement 6: Number of retraining for each value of N · Supplement 8: Values of the generative model hyperparameters used in the heart experiments

The supplements document validation-set-improving modifications to nnUNet, TTA, and SIFA, alongside the retraining protocol and generative-model settings used in the heart experiments. They also describe how enhanced label maps support cardiac segmentation training.

Supplement 5: Modifications to the nnUNet, and TTA, and SIFA methods

  • The reported method modifications improved validation scores over the competing methods’ default implementations.The paper states that all competing methods use their default implementations except for the listed minor changes.Source: Supplement 5: Modifications to the… · p. 18
  • nnUNet 6 gained 0.09 validation Dice by flipping only along the right/left axis, matching SynthSeg’s augmentation strategy.Source: Supplement 5: Modifications to the… · p. 18
  • TTA [46]7 gained 0.16 Dice from a five-layer image normaliser, 0.06 from increasing the atlas-use beta threshold from 0.25 to 0.4, and 0.03 from SynthSeg-matched augmentations.The atlas was rigidly registered to each test scan with NiftyReg; the source notes this was not done in the original implementation because its scans were pre-aligned.Source: Supplement 5: Modifications to the… · p. 18
  • SIFA [11]8 gained 0.18 Dice on validation after adding online spatial, intensity, and bias augmentations matching SynthSeg.Source: Supplement 5: Modifications to the… · p. 18

Supplement 6: Number of retraining for each value of N

  • For training-subject comparisons, label maps were drawn from manual T1-39 segmentations, and low N values received more retrainings to offset variability in random subject selection.Scores were averaged across retrainings.Source: Table S3 · p. 19
  • Cardiac training label maps combine manual foreground delineations, clustered intensity-based subregion labels, and clustered background labels, with N ranging from 3 to 10 classes.One enhanced map is randomly selected to synthesise each training image, while its target segmentation is reset to the initial manual labels.Source: Figure S3 · p. 19

Supplement 8: Values of the generative model hyperparameters used in the heart experiments

  • The heart-experiment generative-model hyperparameter table specifies intensity inputs in [0, 255], rotations in degrees, and spatial measures in millimeters.The supplied passage does not provide the table’s individual hyperparameter values.Source: Table S4 · p. 19

Related papers

Loading 2107.09559v4…