Source-linked AI summary
Combining General and Domain-Specific Pretext Tasks for Brain MR Image Segmentation
Tasneem Nasser, Susanne Schmid, Roberto Souza, Naser El-Sheimy
TL;DR
Limited annotations create a challenge for medical image segmentation, while general self-supervised tasks may not capture anatomically meaningful brain features. The study combines image inpainting with voxel-level brain age prediction and evaluates the pretrained models across three segmentation tasks. The multitask approach performs strongly across tasks, with particular advantages in several low-data settings.
Problem
Medical image segmentation faces limited annotated data, and general self-supervised objectives may not exploit domain-specific anatomical knowledge in brain MR images.
Source: §1 · p. 1Method
The study compares single-task image inpainting and voxel-level brain age prediction with multitask pretraining that jointly optimizes both objectives before downstream segmentation fine-tuning.
Source: §1 · p. 2, §2.1 · p. 2, §2.5 · p. 4Results
The multitask framework achieved the strongest or among the strongest performance across evaluated tasks, with particular advantages in several low-data settings; no single pretext task was uniformly optimal.
Source: §4 · pp. 12, 14Takeaways & Limitations
The findings support combining complementary domain-specific and generic supervisory signals, particularly when annotated data are limited.
Source: §5 · p. 14For MS experiments, paired T1w and FLAIR requirements restricted pretraining to the single-source CNS cohort, potentially constraining transferability.
Source: §4 · p. 12
Abstract
from arXiv · showhide
A key challenge in medical image analysis is the scarcity of large annotated datasets for specific populations and diseases. As deep learning models rely heavily on labeled data, effective transfer learning strategies are needed to reduce the dependence on manual annotations. Self-supervised learning has emerged as a promising approach for developing foundation models by enabling the learning of transferable feature representations from large-scale unlabeled medical imaging datasets. In this study, we investigate voxel-level brain age prediction as a domain-specific self-supervised pretext task and compare it with image inpainting, a widely used non-domain-specific alternative. We further propose a multitask self-supervised pretraining framework that jointly optimizes both objectives to learn complementary neuroimaging representations. The pretrained models are evaluated on three downstream magnetic resonance image segmentation tasks: multiple sclerosis lesion segmentation, ischemic stroke lesion segmentation, and cortical brain structure segmentation. Overall, the proposed multitask pretraining framework consistently outperformed the single-task pretrained models and training from scratch across most experimental settings, demonstrating the benefit of combining domain-specific and general self-supervised learning pretext tasks for the development of generalizable neuroimaging foundation models.\ Code Availability: The source code used in this study is publicly available at https://github.com/TasneemN/Combining-General-and-Domain-Specific-Pretext-Tasks-for-Brain-MR-Image-Segmentation/
1. Introduction
Limited annotations motivate self-supervised pretraining, but general visual objectives may not capture anatomically meaningful neuroimaging features. This work combines a domain-specific brain-age objective with image inpainting and evaluates transfer across age-related and non-age-related segmentation tasks.
1. Introduction
- Limited expert annotations make supervised medical image segmentation challenging because deep learning models require large labeled datasets to learn useful representations.Image annotation is time-consuming and requires expert knowledge.Source: §1 · p. 1
- General self-supervised objectives may not exploit domain-specific anatomical knowledge in 3D brain MR images, motivating domain-specific pretext tasks.Voxel-level brain age prediction uses chronological age as supervision for healthy subjects to encourage anatomically and biologically meaningful representations.Source: §1 · p. 1
- The study compares image inpainting and voxel-level brain age prediction individually and jointly, evaluating transfer to MS, ischemic stroke, and cortical structure segmentation.The lesion tasks are not directly associated with brain aging, whereas cortical structure segmentation is age-related.Source: §1 · p. 2
- The main contribution is demonstrating voxel-level brain age prediction, particularly with multitask pretraining, as an effective pretext task across age-related and non-age-related brain MR segmentation.Source: §1 · p. 2
2.1. Proposed Framework
The framework compares single-task image inpainting and voxel-level brain age prediction with a multitask approach that combines both, then fine-tunes the pretrained models on three segmentation tasks.
2.1. Proposed Framework
- SwinUNETR is the backbone, selected after preliminary architecture experiments using voxel-level brain age prediction.Source: §2.1 · p. 2
- The framework compares image inpainting, voxel-level brain age prediction, and their combined multitask approach against single-task alternatives.The combined approach is intended to learn more transferable brain MR imaging representations.Source: §2.1 · p. 2
- The pretrained models are evaluated on MS and ischemic stroke lesion segmentation as non-age-related tasks and cortical brain structure segmentation as an age-related task.Pretraining input sequences are matched to the requirements of each downstream task.Source: §2.1 · p. 2
2.2. Datasets
Pretraining datasets and downstream segmentation datasets are assembled for four task groupings, with imaging modalities matched to each task and dataset-specific preparation described.
2.2.1. Datasets for Pretext Tasks
- The pretext datasets comprise paired T1w–FLAIR CNS scans for MS and a balanced multi-source T1w cohort for stroke and cortical segmentation.The multi-source cohort includes 1,710 scans; the CNS pretraining split contains 423 training, 122 validation, and 63 test scans.Source: §2.2.1 · p. 2, Figure 2 · p. 3
2.2.2. Dataset for MS Lesion Segmentation
- The framework transfers pretrained models to MS lesion segmentation, using paired T1w and FLAIR inputs for the corresponding pretraining configuration.The supplied dataset passage reports that the CNS cohort provides paired T1w and FLAIR scans for pretraining.Source: §2.2.1 · p. 2, Figure 1 · p. 3
2.2.3. Dataset for Ischemic Stroke Lesion
- After excluding acute- and subacute-stage cases, the ischemic stroke experiments use 649 chronic-stage scans from the 1,453-case ISLES 2026 training dataset.The exclusions focus the analysis on a more stable stage of stroke pathology.Source: §2.2.3 · p. 3
2.2.4. Dataset for Cortical Brain Structure Segmentation
- Cortical segmentation uses 101 healthy-participant T1w images from Mindboggle-101, with cortical labels manually edited under the DKT protocol.The DKT labels provide reference cortical parcellations for training and evaluating automated segmentation.Source: §2.2.4 · p. 4
2.3. Preprocessing
Preprocessing registers paired FLAIR to T1w for CNS scans, resamples all scans to isotropic resolution, normalizes intensities, and uses cropped patches for training.
2.3. Preprocessing
- For CNS, FLAIR is rigidly registered to the subject's T1w reference, and the transformation resamples FLAIR into T1w space.This step ensures spatial correspondence between the modalities.Source: §2.3 · p. 4
- All scans are resampled to 1 mm³ isotropic resolution and their intensities are normalized to [0,1].Source: §2.3 · p. 4
- Because of GPU memory limitations, models are trained on randomly cropped 128×128×128 image patches.Source: §2.3 · p. 4
2.4. Model Architecture
The study compares U-Net, UNETR, and SwinUNETR as candidate backbones, selecting SwinUNETR for subsequent pretraining and segmentation experiments.
2.4. Model Architecture
- SwinUNETR was selected as the backbone for all subsequent pretraining and downstream segmentation experiments.A preliminary comparison using voxel-level brain age prediction found it performed best across both pretraining dataset configurations.Source: §2.4 · p. 4
- U-Net relies on convolutional operations, while UNETR uses a Transformer-based encoder to capture global context.SwinUNETR combines hierarchical feature extraction with window-based self-attention to model local details and long-range spatial dependencies.Source: §2.4 · p. 4
- The backbone comparison evaluated U-Net, UNETR, and SwinUNETR using voxel-level brain age prediction.The comparison covered two pretraining dataset configurations.Source: §2.4 · p. 4
2.5. Self-Supervised Pretraining Strategies
The study independently pretrained SwinUNETR using image inpainting, voxel-level brain age prediction, or a multitask objective combining both, then fine-tuned the models for downstream segmentation.
2.5.1. Image Inpainting
- Image inpainting trains the model to reconstruct missing regions of an MR volume using surrounding anatomical context.Random coarse-dropout masks remove 12 cubes of 32 × 32 × 32 voxels from each 128 × 128 × 128 input patch, masking approximately 19% of the volume.Source: §2.5.1 · p. 4
2.5.2. Voxel-Level Brain Age Prediction
- Voxel-level brain age prediction assigns an age value to each brain-mask voxel, yielding a spatial age map rather than one age estimate for the whole brain.Chronological age supplies the labels; voxel-wise MAE trains the model to predict the age map.Source: §2.5.2 · pp. 4–5
2.5.3. Multitask Framework
- The multitask framework jointly optimizes image inpainting and voxel-level brain age prediction to learn complementary contextual and anatomical representations.Both tasks share a SwinUNETR encoder and decoder but use task-specific prediction heads; the age head predicts only for visible brain regions.Source: §2.5.3 · p. 5
2.6. Fine-tuning for the Downstream Segmentation Tasks
The study evaluates pretrained models and training from scratch across three segmentation tasks and progressively larger labeled-data subsets, selecting fine-tuning configurations using validation performance.
2.6. Fine-tuning for the Downstream Segmentation Tasks
- Across tasks, validation-selected fine-tuning configurations were evaluated at multiple training-set sizes to assess pretraining under differing levels of labeled-data availability.The experiments were designed to examine whether pretraining’s relative benefit changes as more labeled data become available.Source: §2.6 · p. 5
2.6.1. MS Lesion Segmentation
- MS lesion segmentation used paired T1w and FLAIR images; four initialization strategies were compared across training sets of 11, 22, 30, 42, 50, and 58 subjects, with validation/test sets fixed at 20/15.Source: §2.6.1 · pp. 5–6
2.6.2. Ischemic Stroke Lesion Segmentation
- Ischemic stroke segmentation used T1w scans and compared pretrained strategies with training from scratch across nested subsets of 10 to 400 cases.Pretraining used a larger, multi-source normative dataset; experiments also tested multitask decoder reinitialization followed by end-to-end fine-tuning.Source: §2.6.2 · p. 6
2.6.3. Cortical Brain Structure Segmentation
- Cortical structure segmentation used T1w scans and DKT parcellations, comparing the pretrained strategies with training from scratch across nested subsets of 10, 25, 45, and 65 cases.The multitask model was tested with and without decoder reinitialization; the other pretrained models were fine-tuned by updating all parameters.Source: §2.6.3 · pp. 6–7
3. Results
Across the segmentation evaluations, multitask pretraining was strongest in several low-data settings, but its advantage varied by task and training-set size. In cortical segmentation, image inpainting led at the larger evaluated training sizes, while brain-age pretraining alone did not outperform training from scratch.
3.1. Self-Supervised Pretraining Strategies
- SwinUNETR had the lowest brain-age-prediction MAE in both tested settings, and multitask pretraining improved reconstruction performance and voxel-level age-prediction accuracy relative to the corresponding single-task frameworks.These pretraining comparisons motivated downstream segmentation evaluation of the combined representations.Source: §3.1 · p. 7, Table 2 · p. 8
3.2.1. Multiple Sclerosis Lesion Segmentation
- For MS segmentation, multitask pretraining generally achieved the highest Dice scores across most training-set sizes, with its largest improvements over scratch at 11 subjects.Training from scratch scored highest at 30 subjects, and the authors note that the available dataset cannot establish whether the observed trend persists at substantially larger sizes.Source: §3.2.1 · pp. 7–8, Table 3 · p. 8
3.2.2. Ischemic Stroke Lesion Segmentation
- Approximately 6–7 Dice percentage-point gains over training from scratch were reported for multitask pretraining on the smallest ischemic-stroke training subsets.Multitask pretraining achieved the highest segmentation performance up to 150 labeled cases; the gap between pretrained and scratch models became minimal beyond approximately 200 cases.Source: §3.2.2 · p. 8, Table 4 · p. 9
- For ischemic stroke segmentation, both multitask fine-tuning variants outperformed the individual pretraining strategies and scratch baseline through 150 cases, though their relative ranking varied by training size.Fine-tuning all pretrained layers led at 10 and 20 cases, while decoder reinitialization led at 50 and 100 cases; the variants were comparable at 150.Source: §3.2.2 · p. 8, Table 4 · p. 9
3.2.3. Cortical Brain Structure segmentation
- At 10 cortical-segmentation cases, Multitask–Decoder Reinitialized scored 0.251 ± 0.049 Dice versus 0.186 ± 0.023 for training from scratch.At 25 cases, Multitask–FT All scored 0.633 ± 0.044 versus 0.493 ± 0.063 for training from scratch; at 45 and 65 cases, image inpainting achieved the highest mean Dice scores, 0.762 ± 0.005 and 0.772 ± 0.015.Source: Table 5 · p. 10, §3.2.3 · pp. 10–11
- In cortical segmentation, voxel-level brain-age pretraining alone scored below training from scratch at all four training-set sizes.The difference was especially pronounced at 10 cases: 0.063 ± 0.010 Dice for brain-age pretraining versus 0.186 ± 0.023 for scratch.Source: §3.2.3 · pp. 10–11
4. Discussion and Future Perspectives
The study combines voxel-level brain age prediction with image inpainting to learn complementary representations for brain MR segmentation. The multitask framework performed strongest or among the strongest overall, with particular advantages in several low-data settings, though performance depended on task and training-set size.
4. Discussion and Future Perspectives
- In MS segmentation, combining the pretext tasks generally yielded the strongest performance, but scratch achieved the highest Dice score with 30 training subjects.Because the dataset was relatively small, the authors could not determine whether this exception reflected sampling variability or a genuine learning characteristic.Source: §4 · p. 12
- For stroke segmentation, multitask pretraining performed best with up to 150 labeled cases, with gains over scratch reaching approximately 6–7 absolute Dice percentage points in the smallest subsets.The pretrained-versus-scratch performance gap diminished as labeled cases increased; fine-tuning all layers led with 10 and 20 cases, while decoder reinitialization followed by end-to-end fine-tuning led with 50 and 100.Source: §4 · p. 13
- Multitask pretraining combined relatively stable behavior across training-set sizes with improvements observed across subjects and acquisition centers, not just a small subset of test cases.Higher Dice scores across centers and subjects suggest the improvements were not restricted to a particular center or imaging condition.Source: §4 · p. 13
- For cortical segmentation, multitask pretraining led with 10 and 25 cases, whereas image inpainting led with 45 and 65; brain-age pretraining alone never beat scratch.The authors suggest that cortical boundary delineation may align more closely with structural representations learned through inpainting than with features learned for age prediction.Source: §4 · pp. 13–14
- Although brain-age pretraining alone performed poorly on cortical segmentation, both multitask variants beat scratch and either single-task strategy with 10 and 25 cases.The authors suggest age-related supervision may complement reconstruction-derived structural and contextual information when downstream supervision is particularly limited.Source: §4 · p. 14
- Across the three tasks, no single pretext task was uniformly optimal, while multitask pretraining showed particular advantages in several low-data settings and was strongest or among the strongest overall.The findings support combining age-related supervision with a generic reconstruction objective to capture complementary aspects of brain MR images.Source: §4 · pp. 12, 14
- The age-related transfer assessment was indirect because cortical parcellation evaluates anatomical boundary delineation rather than a task directly associated with aging.The authors call for evaluation on downstream tasks more directly tied to aging and age-related structural changes.Source: §4 · p. 14
- The evaluation covered segmentation rather than other neuroimaging applications, and MS pretraining was limited to a single-source cohort because paired T1w and FLAIR data were required.The authors recommend expanding evaluation to classification and prediction tasks and investigating larger multi-center normative datasets with paired modalities and age information.Source: §4 · p. 14
5. Conclusion
The study concludes that combining voxel-level brain age prediction with image inpainting can improve representation transfer in brain MR imaging, particularly when annotated data are scarce. Transfer effectiveness also depends on pretraining-data scale and diversity.
5. Conclusion
- The multitask approach showed its greatest benefit in low-data settings across three downstream segmentation tasks.The conclusion highlights combining complementary domain-specific and generic supervisory signals when annotations are limited.Source: §5 · p. 14
- The framework combines voxel-level brain age prediction with image inpainting to learn more transferable brain MR representations.Brain age prediction is the domain-specific pretext task; image inpainting supplies the generic supervisory signal.Source: §5 · p. 14
- Pretraining effectiveness was influenced by cohort scale and diversity, with stronger transfer observed for a larger multi-source cohort.Source: §5 · p. 14