PaperlayerView exact revision on arXiv

Source-linked AI summary

ReSeg: Retraining-free segmentation of abdominal organs via generalizable semantic representation learning

Derong Yu, Guoyan Zheng

arXiv:2610.29090v1cs.CVcs.AI

TL;DR

Abdominal organ segmentation needs to generalize across clinical domains, while existing approaches rely on real medical images or require finetuning, and domain-randomized methods have limited abdominal performance. ReSeg trains two stages on synthetic images to learn generalizable representations and organ-specific segmentation. It reports superior performance across seven datasets spanning CT, MRI, and PET without real training images.

  • Problem

    Existing approaches depend heavily on real medical image datasets, while domain-randomized segmentation has limited abdominal performance or requires dataset-specific finetuning.

    Source: §5 · pp. 42, 44

  • Method

    ReSeg uses synthetic-data engines for two-stage training: organ-agnostic semantic representation learning followed by organ-specific segmentation with pretrained weights and semantic guidance.

    Source: §1 · pp. 5–6

  • Results

    ReSeg reports superior performance over state-of-the-art methods on seven abdominal segmentation datasets across CT, MRI, and PET.

    Source: §1 · p. 7

  • Takeaways & Limitations

    ReSeg demonstrates a retraining-free route to abdominal organ segmentation across multiple modalities without acquiring real training images.

    Source: §1 · pp. 6–7

    Training uses publicly available segmentation masks from the TotalSegmentator CT dataset, which encompasses 1,204 patients.

    Source: §3.1 · p. 9

Abstract

from arXiv · show

Abdominal organ segmentation is essential in modern medical applications. While existing deep learning approaches demonstrate promising performance, their heavy reliance on large-scale image data raises concerns over generalization capability and privacy risks. This motivates the development of generalizable segmentation training paradigms based on domain-randomized image synthesis, facilitating generalization across unseen real image domains. However, existing paradigms either suffer from limited segmentation performance when applied to abdominal organs, or require dataset-specific finetuning. To address these limitations, we propose ReSeg, a novel two-stage retraining-free framework for domain-agnostic zero-shot abdominal organ segmentation. At the first stage, our organ-agnostic synthetic data-driven pretraining approach exploits semantically and visually diverse synthetic images generated by a dedicated data engine to optimize a Generalizable Semantic Representation Network (GSRNet). The network is thereby capable of learning semantic features robust against variations in anatomical structure, imaging contrast, resolution and spatial context, which provide universal representation capabilities that serve as a foundation for segmentation learning. Subsequently, in the second stage, we continue to train an Organ-specific Encoder-decoder Segmentation Network (OESNet), whose encoder is initialized with the pretrained weights. This training process is facilitated by another dedicated data engine, which generates diversified visual images while strictly retaining the inherent spatial layout of abdominal organs. Comprehensive evaluation on seven abdominal organ segmentation datasets across three different modalities including CT, MRI, and PET demonstrates that ReSeg consistently outperforms existing state-of-the-art (SOTA) methods while eliminating the need for real training images.

1. Introduction

ReSeg addresses the need for abdominal segmentation that generalizes across domains without real training images. It combines synthetic-data representation learning with organ-specific segmentation and reports superior performance across seven datasets and three modalities.

1. Introduction

  • The work targets abdominal segmentation challenges including weak soft-tissue boundaries, patient-varying organ morphology, and inconsistent field of view across clinical settings.These factors make anatomical and contextual variation central to the problem.Source: §1 · p. 3
  • Existing approaches rely heavily on large medical image datasets, while domain-randomized methods have limited abdominal performance or require downstream finetuning.Abdominal organs have substantial morphological variation, and realistic synthetic data are difficult to generate without standardized atlases and refined templates.Source: §1 · pp. 4–5
  • ReSeg uses a Generalizable Appearance model to synthesize images from organ shape templates and introduces GenRep and GenSeg for representation and organ-specific segmentation learning.GenSeg retains inherent anatomical spatial configurations while varying appearance; background simulation uses a Label Processing model.Source: §1 · pp. 5–6
  • In stage one, GenRep trains GSRNet on semantically and visually diverse synthetic images to learn features robust to anatomical, appearance, resolution, and context variation.The framework uses a Multi-patch Context-aware Contrastive loss to encourage domain-invariant feature alignment across contextual variations.Source: §1 · pp. 5–6
  • In stage two, OESNet uses the pretrained encoder and a Semantics-guided Segmentation decoder to learn abdominal-organ-specific segmentation from images that retain organ layout.The decoder incorporates semantic priors from GSRNet.Source: §1 · pp. 5–6
  • ReSeg outperforms state-of-the-art approaches across seven abdominal organ segmentation datasets spanning CT, MRI, and PET.The evaluation is described as comprehensive.Source: §1 · p. 7

2. Related work

Related work spans dataset-specific segmentation, self-supervised representation learning, and domain-randomized synthesis. The discussion identifies limited abdominal performance and downstream finetuning as gaps ReSeg seeks to address.

2.1. Dataset-specific models for medical image segmentation

  • U-shaped encoder-decoder networks, attention, transformers, and Mamba variants advance segmentation, but these methods are typically trained on a single dataset and remain dataset-specific.Attention can emphasize target regions, while transformers and Mamba-based models capture long-range spatial dependencies.Source: §2.1 · p. 7

2.2. Self-supervised learning for medical image segmentation

  • Self-supervised methods use reconstruction or contrastive losses, but often rely on target-dataset heuristics and single-modality similarity; UAE extends CT-MRI learning but remains insufficient under major shifts such as CT or MRI to PET.UAE's iterative training is also described as computationally intensive.Source: §2.2 · pp. 7–8

2.3. Domain randomization for medical image segmentation

  • Domain-randomized synthesis has enabled retraining-free segmentation in brain, vascular, and cellular-nuclei settings, but direct abdominal application is limited by anatomical variability and synthetic-data challenges.Abdominal organs occupy complex backgrounds and vary morphologically; the lack of standardized atlases and refined templates complicates realistic synthesis.Source: §2.3 · pp. 8–9
  • Anatomix learns appearance-agnostic representations but overlooks contextual consistency and requires downstream finetuning for segmentation.ReSeg is presented as learning features robust to both appearance and contextual changes before organ-specific learning.Source: §2.3 · p. 9

3. Method

ReSeg uses organ shape templates to create synthetic training data without real images, then trains a generalizable semantic representation network and an organ-specific segmentation network. Its data engines separately support representation learning with randomized layouts and segmentation learning with preserved abdominal anatomy.

3.1. Overview

  • ReSeg uses TotalSegmentator segmentation masks as shape templates, excludes real images, and pretrains GSRNet before training OESNet with pretrained encoder weights.OESNet is trained on AMI and guided by semantic priors from GSRNet.Source: §3.1 · pp. 9, 11

3.2.1. Generalizable Appearance (GA) model

  • GA randomizes label-conditioned intensities with GMMs and dynamic mean lower bounds, while resolution degradations and augmentations further diversify synthetic images.The dynamic lower-bound strategy enhances contrast variation; resolution variation includes conditions such as one low-resolution axis in MR scans.Source: §3.2 · p. 13, §3.2.1 · pp. 14–15

3.2. Data engines

  • GenRep creates paired synthetic images from randomized organ layouts, whereas GenSeg varies appearances while preserving the spatial layout of organs from one patient.The resulting databases are RPI for representation learning and AMI for segmentation learning.Source: §3.2 · p. 13

3.2.2. Data engine for representation learning

  • GenRep assembles randomized label maps from organ templates sampled across patients and pairs each map with images of distinct appearances but identical semantic layout.The randomized maps and paired images form the RPI database.Source: §3.2.2 · pp. 15, 17

3.2.3. Data engine for segmentation learning

  • GenSeg synthesizes appearance-diverse images from each patient’s organ map while retaining anatomy, then selects labels for the liver, spleen, and both kidneys as segmentation targets.Its label processing also merges specified anatomical structures and constructs a coherent foreground mask.Source: §3.2.3 · pp. 17–18

3.3. Organ-agnostic semantic representation learning

  • GSRNet uses a shared CNN encoder with Context-aware Semantic and Appearance-aware Semantic decoders to learn voxel-wise embeddings robust to contextual and appearance changes.The decoders use Multi-patch Context-aware Contrastive and Dual-view Prototypical Contrastive losses, respectively.Source: §3.1 · p. 11, §3.3 · p. 18

3.3.1. Training

  • Training samples paired RPI images: MCC trains the context decoder using cropped patches, while DPC optimizes the appearance decoder.The framework also resizes the original image pair to provide global spatial context.Source: §3.3.1 · p. 18, Figure 5 · p. 19

17 end

The framework trains organ-specific segmentation on labeled synthetic patches, using pretrained semantic features to guide segmentation, then applies the network to real images without finetuning. Its SFE modules align semantic embeddings with segmentation features and use them to refine predictions.

17 end

  • The MCC loss uses multi-patch positive samples from different spatial contexts while separating negative samples, promoting robust domain-invariant, context-aware representations.The CS decoder samples anchor points from overlapping patches; points with the same labels are positives and those with different labels are negatives.Source: §17 · pp. 20–21

3.3.2. Inference

  • After pretraining, GSRNet concatenates its appearance-aware and context-aware embeddings to form the generalizable semantic embedding used for segmentation guidance.The SS decoder incorporates this embedding through hierarchical SFE modules.Source: §3.3.2 · p. 22, Figure 7 · p. 23

3.4.1. Segmentation architecture overview

  • OESNet is trained on randomly cropped synthetic image patches paired with spatially aligned ground-truth label patches, with its encoder initialized from pretrained GSRNet weights.The decoder uses semantic priors from frozen GSRNet in parallel with the segmentation network.Source: §3.4.1 · p. 24

3.4.2. Semantics-guided Feature Extraction (SFE) module

  • At each of four resolution levels, an SFE module uses GSRNet's generalizable semantic embedding to guide segmentation feature extraction.The embedding is aligned to the segmentation feature, converted to attention, and applied to refine that feature; parallel depthwise convolutions with kernel sizes 1, 3, 5, and 7 capture complementary information.Source: §3.4.1 · p. 24, §3.4.2 · pp. 25–26

3.4.3. Segmentation loss function

  • The segmentation objective applies Dice and cross-entropy losses at all four decoder layers with deep supervision.Ground-truth labels are downsampled for levels d > 0.Source: §3.4.3 · p. 26

3.4.4. Inference

  • At inference, OESNet segments real clinical images from unseen domains without retraining or finetuning, using overlapping sliding-window patches and the highest-resolution prediction.The stated overlap ratio is 70%.Source: §3.4.4 · p. 26

4. Experiments and results

The experiments evaluate abdominal organ segmentation across seven datasets spanning CT, MRI, and PET. Four core datasets support broad comparisons and ablations, while three additional datasets assess zero-shot generalization.

4.1. Datasets

  • Seven public abdominal datasets across CT, MRI, and PET were used to evaluate segmentation generalization.Four core datasets support zero-shot and few-shot comparisons, reference baselines, and ablations; three generalization datasets assess zero-shot performance only.Source: §4.1 · p. 27
  • The core evaluation comprises BTCV and AMOS-CT in CT, plus L2RAb-MR and AMOS-MR in MRI.These datasets provide the core comparisons, including dataset-specific training/testing splits where applicable.Source: §4.1 · pp. 27–28
  • The generalization evaluation uses WORD in CT, MRISegmentator in MRI, and AutoPET-Organ in PET.All 100 annotated AutoPET-Organ cases were used for testing; MRISegmentator evaluation used its official 240 test cases.Source: §4.1 · p. 28

4.2. Comparison methods

ReSeg is compared with zero-shot and few-shot segmentation approaches, alongside supervised reference baselines. The implementation uses separate pretraining and segmentation-learning stages with specified optimization settings.

4.2. Comparison methods

  • The comparison includes SAM-based and synthetic-data zero-shot methods, meta-learning and finetuning few-shot methods, and supervised nnUNet reference baselines.SynthSeg was trained using TotalSegmentator CT masks for the abdominal-organ comparison; the nnUNet upper bound uses same-dataset training and testing, while the lower bound crosses modalities.Source: §4.2 · pp. 29–30

4.3. Implementation details

  • ReSeg pretraining uses paired synthetic volumes for the AS decoder and six overlapping patches for the CS decoder, with contrastive-learning settings including τc = 0.33 and τa = 0.1.Pretraining ran for 45,000 iterations with SGD, momentum 0.9, and weight decay 1.0 × 10^-4.Source: §4.3 · p. 30
  • Segmentation learning uses 128 × 128 × 128 patches and trains OESNet for 80,000 iterations with SGD at learning rate 0.01.Few-shot comparison methods used K = 1 support images, and ZeroShotSAM received M = 1 foreground point per 2D slice.Source: §4.3 · pp. 30–31

4.4. Evaluation protocol

The evaluation measures segmentation overlap with DSC, averaging scores first across cases for each organ and then across target organs. Volumetric predictions are evaluated in the original test-volume space.

4.4. Evaluation protocol

  • DSC quantifies spatial overlap between predicted segmentations and ground-truth labels.The paper uses DSC as its 3D medical-image segmentation metric.Source: §4.4 · p. 31
  • Evaluation averages per-case DSC scores across cases to obtain each target organ's score, then averages across target organs.The latter value is the reported average DSC.Source: §4.4 · p. 31
  • DSC is computed in the original space of test volumes; 2D benchmarks use axial slices, while 3D models process whole 3D images.Source: §4.4 · p. 31

4.5. Comparison results

ReSeg outperformed zero-shot and few-shot competitors on the core CT and MR datasets, and surpassed zero-shot methods across the evaluated CT, MRI, and PET generalization datasets. Statistical analysis also found stronger median DSC scores and tighter score distributions for ReSeg on the core datasets.

4.5.1. Results on the core evaluation datasets

  • ReSeg achieved the best average DSC on BTCV (88.14%) and AMOS-CT (88.64%) among the compared zero-shot and few-shot methods.ZeroShotSAM was the strongest zero-shot competitor, while VoCo led the few-shot methods; Fig. 8 compares segmentation boundaries visually.Source: §4.5.1 · pp. 32–33, Figure 8 · p. 32
  • ReSeg achieved the best average DSC on L2RAb-MR (86.64%) and AMOS-MR (90.38%), ahead of the best zero-shot and few-shot competitors.Fig. 9 provides visual comparisons on both MR datasets.Source: §4.5.1 · pp. 33–34, Figure 9 · p. 34
  • ReSeg had superior median DSC scores and tighter score distributions; improvements over competing methods were significant on AMOS-CT, L2RAb-MR, and AMOS-MR.Case-wise DSC scores were organ-wise averaged, and paired Wilcoxon signed-rank tests used a 0.05 significance level.Source: §4.5.1 · pp. 34–35, Figure 10 · p. 35

4.5.2. Results on the generalization evaluation datasets

  • On WORD, MRISegmentator, and AutoPET-Organ, ReSeg improved average DSC over the second-best zero-shot method by 5.37%, 7.67%, and 55.72%, respectively.ReSeg requires neither finetuning nor prompts; ZeroShotSAM requires test-time finetuning and foreground prompt points. On PET, the other zero-shot methods failed to produce meaningful results, while ReSeg produced reasonable segmentations.Source: §4.5.2 · pp. 35, 37, Table 3 · p. 36, Figure 11 · p. 36

4.6. Ablation study

Ablations show that the dynamic lower-bound strategy, MCC loss, pretrained encoder initialization, and SFE modules each contribute to ReSeg's segmentation performance. The MCC and OESNet component studies report statistically significant differences.

4.6.1. Results on the impact of the dynamic lower bound strategy of the GA model

  • Using a fixed rather than dynamic lower bound reduced average DSC by 1.54% on L2RAb-MR, 5.35% on AMOS-MR, 0.95% on BTCV, and 3.42% on AMOS-CT.The dynamic lower bound varies the mean parameters in the GA model's GMMs to increase contrast variation.Source: §4.6.1 · pp. 37–38

4.6.2. Results of comparing the MCC loss against its variants

  • MCC achieved 88.45% average DSC across four abdominal datasets, exceeding the AS decoder-only and dual-decoder variants by 1.96% and 1.21%, respectively (both P < 0.001).The variants omit contextual consistency; the dual-decoder variant uses voxel-wise contrastive loss on original image pairs.Source: §4.6.2 · p. 38

4.6.3. Results of sub-component ablations on the semantics-guided segmentation architecture of OESNet

  • Removing SFE modules reduced average DSC to 86.60% (a 1.85% drop), while also disabling pretrained encoder initialization reduced it to 84.35% (a 4.10% total drop; both P < 0.001).The study compares pretrained encoder transfer with random initialization and SFE with standard convolutional blocks.Source: §4.6.3 · p. 39

4.7. Extension to cardiac segmentation

ReSeg extends to cardiac segmentation by keeping its pretrained GSRNet fixed and retraining OESNet with pseudo cardiac data. It outperformed SynthSeg on both MMWHS-CT and MMWHS-MR.

4.7. Extension to cardiac segmentation

  • Cardiac adaptation kept GSRNet fixed, modified the GenSeg label-processing model, and retrained OESNet on pseudo cardiac data; inference required no retraining or finetuning.Training used 13 MMWHS-MR label maps; the evaluation used remaining MR and CT scans after two scans were reserved for validation.Source: §4.7 · p. 41
  • ReSeg outperformed SynthSeg on MMWHS-CT with 87.76% average DSC, a 4.13% improvement, and on MMWHS-MR with 84.42%, a 3.59% improvement.Fig. 12 visually compares the methods on both datasets.Source: Figure 12 · p. 40, §4.7 · pp. 41–42

5. Discussion

The discussion attributes ReSeg's performance to synthetic-data engines and a two-stage framework that learns general semantic representations before organ-specific segmentation. It presents ReSeg as a retraining-free approach that generalizes across abdominal modalities and can extend beyond abdominal organs.

5. Discussion

  • The discussion links the GA model's dynamic GMM lower bound to improved contrast variation and generalization, addressing the poor abdominal performance of directly applied domain-randomized methods.The discussion identifies the performance evidence in Tables 1–4.Source: §5 · pp. 42–43
  • ReSeg's two stages learn organ-agnostic semantic representations and then abdominal-organ-specific features, using GenRep and GenSeg synthetic-data engines.GenRep supplies randomized-label-map image pairs for GSRNet training with MCC loss; GenSeg preserves organ spatial arrangements for OESNet segmentation learning.Source: §5 · pp. 42–43
  • MCC promotes domain-invariant, context-aware features across overlapping pseudo-image patches, while OESNet uses pretrained semantic priors to guide feature extraction.The discussion contrasts MCC with voxel-wise contrastive learning that overlooks contextual consistency, and points to the Table 5 and Table 6 results.Source: §5 · p. 43
  • The authors report that ReSeg can extend to new anatomical targets by modifying the segmentation data engine's LP model and retraining OESNet.Cardiac results are cited as evidence of generalization beyond abdominal organs.Source: §5 · pp. 43–44
  • ReSeg uses domain-randomized synthetic data without real training images and is presented as outperforming few-shot methods that require annotated real data.The discussion contrasts this with biomedical foundation models' reliance on large real-image datasets and with domain-randomization approaches that underperform on abdominal organs.Source: §5 · p. 44

6. Conclusion

ReSeg combines synthetic-data-driven semantic pretraining with semantics-guided segmentation training to support generalizable abdominal-organ segmentation without retraining. Validation across seven abdominal datasets spanning CT, MRI, and PET showed superior generalization capability versus SOTA methods.

6. Conclusion

  • ReSeg uses two stages to achieve generalizable abdominal-organ segmentation: organ-agnostic semantic representation learning followed by semantics-guided segmentation.The first stage learns robust semantic features from diverse synthetic images; the second uses synthesized images retaining anatomical spatial layouts.Source: §6 · p. 45
  • The pretrained network supplies universal semantic representations, while hierarchically integrated semantic priors support learning abdominal-organ-specific features in the decoder.The segmentation network is optimized with diversified visual images synthesized with anatomical spatial layouts.Source: §6 · p. 45
  • Validation on seven abdominal datasets across CT, MRI, and PET demonstrated superior generalization capability compared with SOTA methods.Source: §6 · p. 45

CRediT authorship contribution statement

The contribution statement assigns distinct writing, validation, software, research, and conceptual roles to Derong Yu and Guoyan Zheng.

CRediT authorship contribution statement

Related papers

Loading 2610.29090v1…