PaperlayerView exact revision on arXiv

Source-linked AI summary

What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentation

Amal Saqib, Tausifa Jan Saleem, Numan Saeed, Mohammad Yaqub

arXiv:2608.13660v1cs.CVcs.LG

TL;DR

Continual gynecological image segmentation must adapt to sequentially arriving data across shifting modalities, anatomy, and annotations while preserving prior knowledge. This paper analyzes forgetting across encoder–decoder depth using block-wise ablations and depth-constrained adaptation, finding that bottleneck-adjacent updates largely preserve prior tasks whereas shallower updates cause abrupt forgetting.

  • Problem

    Sequential gynecological segmentation involves shifting modalities, anatomical structures, and annotation completeness, but existing methods have limited evidence of preserving previously learned tasks.

  • Method

    The study combines block-wise encoder–decoder ablations with depth-constrained adaptation experiments during sequential continual segmentation.

  • Results

    Early encoder and late decoder regions are most performance-critical, while restricting updates near the bottleneck largely preserves prior tasks and shallower updates cause abrupt forgetting.

  • Takeaways & Limitations

    Forgetting is strongly influenced by where adaptation occurs within the encoder–decoder hierarchy, supporting depth-aware continual segmentation analysis.

  • Takeaways & Limitations

    The study evaluates datasets as a fixed task sequence that introduces progressively more heterogeneous clinical shifts in modality, annotations, and anatomy.

Abstract

from arXiv · show

Medical image segmentation models are typically trained under the assumption that all data are available simultaneously. However, in clinical practice, datasets often arrive sequentially, requiring models to adapt continuously to evolving data distributions. We study this problem in gynecological image segmentation, where substantial heterogeneity across imaging modalities, anatomical structures, and annotation protocols creates a particularly challenging continual learning setting. Under these large distribution shifts, existing continual learning methods struggle to preserve previously learned knowledge, leading to catastrophic forgetting. To better understand forgetting in this setting, we investigate how different encoder--decoder regions influence segmentation performance and forgetting during continual gynecological segmentation. Through block-wise ablation analysis, we observe that ablating early encoder and late decoder regions results in the largest performance degradation, indicating that segmentation performance depends unevenly across the network hierarchy. Using controlled adaptation experiments, we further show that forgetting remains limited when updates are restricted to bottleneck-adjacent regions, but increases sharply once shallower encoders and decoders become trainable, even when only a small subset of parameters is updated. These findings suggest that forgetting in the encoder-decoder architecture is strongly influenced by where updates occur across network depth during continual learning. Full code and analysis pipelines will be made publicly available upon acceptance.

1. Introduction

Continual gynecological image segmentation is challenging because sequential datasets span diverse modalities, targets, anatomy, and annotations, while existing methods struggle to preserve prior knowledge under large distribution shifts. This work analyzes how encoder–decoder depth affects segmentation performance and forgetting through block-wise ablations and depth-constrained adaptation.

  • Motivation: Continual gynecological segmentation is difficult because sequential datasets differ across imaging modalities, segmentation targets, anatomical variability, and annotation completeness.The modalities include ultrasound, MRI, and PET/CT.
  • Problem: Existing continual learning methods struggle to preserve prior knowledge under the large distribution shifts present in gynecological image segmentation.The paper notes that prior approaches do not explicitly examine how forgetting varies across encoder–decoder depth.
  • Findings: Block-wise ablation identifies early encoder and late decoder regions as the most performance-critical stages of the encoder–decoder network.The analysis examines how different network regions contribute to segmentation performance.
  • Findings: Forgetting remains limited when adaptation is confined to bottleneck-adjacent regions but increases sharply as updates extend toward shallower encoder and decoder stages.Controlled depth-constrained adaptation experiments relate update location to knowledge retention during continual segmentation.

2. Related Work

Prior work has established encoder–decoder segmentation methods and continual-learning strategies, but most gynecological segmentation studies assume static training. Recent analyses indicate that forgetting varies across network depth, motivating systematic control of which regions are adapted.

  • Gynecological Image Segmentation: Gynecological segmentation studies use encoder–decoder architectures such as U-Net and nnU-Net under static training assumptions.These methods and public datasets span diverse clinical settings, but typically assume all datasets are available simultaneously.
  • Continual Learning: Continual-learning methods are commonly categorized as regularization-based, distillation-based, and replay-based approaches.EWC constrains updates using estimated parameter importance, while LwF preserves prior knowledge through output- or feature-level consistency.
  • Depth-Wise Forgetting: Forgetting is not uniformly distributed across network depth, with deeper layers undergoing larger representational changes while earlier layers remain relatively stable.This depth-wise asymmetry has been identified during sequential training.
  • Depth-Wise Forgetting: Layer-wise studies show that network layers differ in performance importance and robustness to perturbations, but primarily examine representational drift rather than systematically controlling adapted regions.This leaves the effect of selectively updating network regions insufficiently characterized.

3. Method

The method analyzes depth-dependent forgetting in continual gynecological segmentation through block-wise ablations and depth-constrained adaptation. It uses a task-incremental, multi-head U-Net with depth-wise encoder–decoder blocks and progressively restricted or expanded trainable regions.

  • Method overview: The study uses two complementary analyses: block-wise ablation measures regional contributions to segmentation performance, while depth-constrained adaptation tests how update location affects forgetting.These analyses are applied after evaluating existing continual learning methods in the gynecological segmentation setting.
  • Continual segmentation setup: The task-incremental setup assigns each segmentation dataset its own label space and uses task-specific heads with a shared encoder–decoder backbone.The backbone is partitioned into U-Net blocks at different spatial-resolution levels.
  • Depth-wise architecture: The U-Net separates the bottleneck from depth-wise encoder and decoder blocks, with corresponding pairs operating at the same spatial-resolution level and linked by skip connections.Encoder blocks run from shallow stages toward the bottleneck, whereas decoder blocks run from the bottleneck toward the output.
  • Block-wise ablation: After each sequential task, individual encoder or decoder blocks are ablated by zeroing their output activations, and the resulting segmentation-performance change is measured.The unmodified network provides Dfull, while the ablated network provides Db; larger relative block contributions indicate greater reliance on that block.
  • Depth-constrained adaptation: During adaptation, only the task-specific head is initially trainable; the bottleneck and then paired encoder–decoder blocks are progressively unfrozen from deepest toward shallower stages.Encoder–decoder pairs at the same depth are adapted together because skip connections link them and they jointly process features at a shared spatial resolution.

4. Experimental Setup

The experiments evaluate continual gynecological segmentation across three public datasets formulated as sequential tasks with challenging shifts in modality, anatomy, and annotation structure. Models use standardized preprocessing and unified two-channel inputs, while performance and forgetting are assessed with DSC and degradation on previously learned tasks.

  • Datasets: Three public datasets—UMD, UT-EndoMRI, and ECPC-IDS—represent differing imaging modalities, anatomical targets, and annotation structures.UT-EndoMRI contributes 103 T1FS volumes selected for strongest preliminary single-task performance.
  • Preprocessing and input representation: All datasets are preprocessed with nnU-Net to standardize voxel spacing, orientation, and label formatting across tasks.Inputs use two channels: duplicated single-modality inputs or separate PET and CT channels for ECPC-IDS.
  • Continual learning formulation: Each dataset is formulated as a separate task in a sequential learning setting using a fixed task order.The ordering progresses from a relatively consistent, fully annotated task toward heterogeneous clinical settings with greater modality variation, partial annotations, and anatomical variability.
  • Training frameworks: Experiments use nnU-Net, Lifelong nnU-Net, and MultiTalent nnU-Net for sequential and joint multi-dataset training.Unless otherwise stated, models use the default 3D full-resolution configuration; quantitative results use standard nnU-Net 5-fold splits and fold 0 unless specified.
  • Evaluation: Segmentation is evaluated with DSC, while forgetting is the average degradation across the sequence on previously learned tasks.Lower forgetting indicates better retention, and negative values indicate positive backward transfer; results are reported as mean ± standard deviation.

5. Results and Discussion

Continual gynecological segmentation exhibits severe, uneven forgetting under heterogeneous task transitions, while retention and transfer depend strongly on which encoder–decoder depths are adapted. Shallow encoder and late decoder regions are critical for performance and can trigger forgetting even under highly restricted updates.

  • Continual-learning performance: Sequential fine-tuning collapses UMD and ECPC-IDS performance to 0.00 DSC after adaptation, whereas rehearsal reduces average forgetting from 0.7531 to 0.0815.Rehearsal preserves substantially higher performance across both tasks but requires storing samples from previously seen datasets.
  • Continual-learning performance: Forgetting remains uneven across tasks: rehearsal achieves 0.0830 forgetting on UMD but 0.6430 on ECPC-IDS.The disparity persists despite mitigation strategies, motivating analysis across encoder–decoder depth.
  • Depth-wise contribution: Ablating shallow encoder blocks E0–E2 nearly collapses UMD segmentation with δb ≥0.99, while ablating final decoder stages D0–D2 causes maximal degradation at δb = 1.00.These results indicate strong reliance on early encoder and late decoder regions for accurate prediction.
  • Selective freezing: Training only E0 and D0 causes UMD performance to collapse to zero during UMD → ECPC-IDS, despite ECPC-IDS tumor DSC reaching only 0.1128.The same near-total UMD collapse occurs for the UMD → UT-EndoMRI transition, showing severe forgetting under highly constrained adaptation.

6. Conclusion and Future Work

The study finds that forgetting in continual gynecological segmentation depends strongly on where adaptation occurs within the encoder–decoder hierarchy. Bottleneck-adjacent updates largely preserve prior tasks, whereas extending updates toward shallower stages causes abrupt forgetting, motivating depth-aware continual learning strategies.

  • Depth-constrained adaptation: Restricting updates near the bottleneck largely preserves previously learned tasks during continual adaptation.This finding comes from depth-constrained adaptation experiments.
  • Depth-constrained adaptation: Extending adaptation toward shallower encoder and decoder stages leads to abrupt forgetting, even when only a small parameter subset is trainable.The result indicates that forgetting is not determined solely by the number of updated parameters.
  • Post-adaptation analysis: Different adaptation constraints produce distinct depth-wise importance profiles after adaptation.These analyses show that adaptation can change how segmentation performance depends on different encoder–decoder regions.
  • Future work: Future continual learning methods may benefit from strategies that explicitly account for the uneven functional importance of network regions.The conclusion frames selective preservation of retention-critical representations as a direction for future work.

Appendix A. Additional Experiments and Analyses · A.1. Dataset Statistics

Figure 9 summarizes dataset-level annotation and preprocessing statistics used in the experiments, providing context for variation in segmentation performance and forgetting across tasks.

  • A.1. Dataset Statistics: Figure 9 summarizes the dataset-level statistics used in the experiments.The figure covers annotation and preprocessing statistics.
  • A.1. Dataset Statistics: The reported statistics concern dataset annotation.
  • A.1. Dataset Statistics: The reported statistics also concern dataset preprocessing.
  • A.1. Dataset Statistics: These statistics provide additional context for observed variation in segmentation performance.
  • A.1. Dataset Statistics: These statistics also provide additional context for observed variation in forgetting across tasks.
  • Appendix A. Additional Experiments and Analyses: The appendix presents these statistics as supplementary analysis accompanying the experiments.

A.2. UT-EndoMRI Sequence Selection

The UT-EndoMRI dataset includes four MRI sequences, but T1FS and T2 are most consistently available after preprocessing and quality control. Consequently, these sequences provide the largest usable case subset for experimentation.

  • Sequence availability: The UT-EndoMRI dataset contains four MRI sequences: T1, T1FS, T2, and T2FS.
  • Sequence availability: Sequence availability varies substantially across patients after preprocessing and quality control.
  • Sequence availability: T1FS and T2 are the most consistently available sequences and provide the largest usable subset of cases for experimentation.

A.3. Continual Learning Hyperparameter Variants

The section evaluates hyperparameter variants for rehearsal, Learning without Forgetting, and Elastic Weight Consolidation using task-wise DSC heatmaps and average forgetting. Larger rehearsal memories improve retention, while EWC exposes a stability–plasticity tradeoff between adaptation and preservation.

  • Evaluation setup: The study compares additional hyperparameter variants for rehearsal, Learning without Forgetting, and Elastic Weight Consolidation.Figure 10 reports task-wise DSC heatmaps, and Table 8 summarizes average forgetting across configurations.
  • Rehearsal: Increasing rehearsal memory size consistently improves retention by preserving greater access to previously observed samples.The passage notes that larger memories may be impractical clinically because of storage, privacy, and data-sharing constraints.
  • Elastic Weight Consolidation: EWC exhibits a strong stability–plasticity tradeoff: weak regularization supports later-task adaptation but fails to preserve earlier knowledge, whereas stronger regularization substantially reduces forgetting.The passage states that stronger regularization severely limits adaptation.

A.4. Additional Qualitative Depth-Constrained Adaptation Example

During the UMD → ECPC-IDS transition, qualitative segmentation structure remains largely preserved when adaptation is limited to bottleneck-adjacent regions but rapidly collapses when shallower encoder and decoder stages become trainable. This additional example supports the sharp forgetting transition observed quantitatively.

  • Qualitative depth-constrained adaptation: Segmentation structure remains largely preserved when adaptation is restricted to bottleneck-adjacent regions during the UMD → ECPC-IDS transition.This behavior is consistent with the main paper’s findings.
  • Qualitative depth-constrained adaptation: Forgetting rapidly increases once shallower encoder and decoder stages become trainable.The qualitative example shows predictions collapsing as trainable regions expand toward shallower stages.
  • Qualitative depth-constrained adaptation: The observed collapse further supports the sharp transition in forgetting identified by the quantitative analyses.Figure 11 presents this transition using green ground-truth contours and red predicted regions.
Loading 2608.13660v1…