PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Data Imbalance in Class Incremental Surgical Instrument Segmentation

Shifang Zhao, Long Bai, Kun Yuan, Feng Li, Jieming Yu, Wenzhen Dong, Guankun Wang, Mobarakol Islam, Nicolas Padoy, Nassir Navab, Hongliang Ren

arXiv:2610.34925v1cs.CV

TL;DR

Continual surgical instrument segmentation must learn new instruments while retaining previously learned classes, but data imbalance and surgical data constraints complicate the task. SurgCSS combines class-aware inpainting-based blending, contrastive Class Desensitization Loss, and dynamic weight fusion; experiments report improved balance across old and new classes and an average 11.82% improvement over state-of-the-art continual-learning methods in robotic surgical scenarios.

  • Problem

    Continual surgical segmentation must learn new classes without catastrophic forgetting, while imbalanced data, privacy constraints, and visually similar instruments complicate learning.

    Source: §1 · pp. 1–2

  • Method

    SurgCSS combines class-aware inpainting and foreground-background blending, contrastive Class Desensitization Loss, and dynamic fusion of model weights.

    Source: §1 · p. 2

  • Results

    SurgCSS balances performance across old and new classes and outperforms state-of-the-art continual-learning methods, with average 11.82% improvement in robotic surgical scenarios.

    Source: §5 · pp. 12–13

  • Takeaways & Limitations

    The framework addresses both old-versus-new and within-time-point class imbalance in continual surgical segmentation.

    Source: §1 · p. 2

    The class-aware foreground-background overlay may introduce data inconsistencies and biases because it does not effectively model their interaction.

    Source: §4.3.1 · p. 12

Abstract

from arXiv · show

In surgical instrument segmentation, the increasing variety of instruments over time poses a significant challenge for existing neural networks, as they are unable to effectively learn such incremental tasks and suffer from catastrophic forgetting. When learning new data, the model experiences a sharp performance drop on previously learned data. Although several continual learning methods have been proposed for incremental understanding tasks in surgical scenarios, the issue of data imbalance often leads to a strong bias in the segmentation head, resulting in poor performance. Data imbalance can occur in two forms: (i) class imbalance between new and old data, and (ii) class imbalance within the same time point of data. Such imbalances often cause the dominant classes to take over the training process of continual semantic segmentation (CSS). To address this issue, we propose SurgCSS, a novel plug-and-play CSS framework for surgical instrument segmentation under data imbalance. Specifically, we generate realistic surgical backgrounds through inpainting and blend instrument foregrounds with the generated backgrounds in a class-aware manner to balance the data distribution in various scenarios. We further propose the Class Desensitization Loss by employing contrastive learning to correct edge biases caused by data imbalance. Moreover, we dynamically fuse the weight parameters of the old and new models to achieve a better trade-off between the biased and unbiased model weights. To investigate the data imbalance problem in surgical scenarios, we construct a new benchmark for surgical instrument CSS by integrating datasets. Extensive experiments demonstrate the effectiveness of the proposed framework, achieving significant performance improvement against existing baselines.

1. Introduction

Continual surgical instrument segmentation is challenged by catastrophic forgetting and data imbalance, compounded by privacy restrictions, workflow variation, and visually similar instruments. SurgCSS combines class-aware data synthesis, Class Desensitization Loss, and dynamic weight fusion, and the paper introduces a benchmark integrating four public datasets.

1. Introduction

  • Class imbalance, catastrophic forgetting, privacy restrictions, workflow variation, and similar instrument appearances complicate continual segmentation in surgical settings.The data distribution varies with surgical phase durations and erratic instrument occurrences; old classes are more dispersed while new classes can be concentrated.Source: §1 · pp. 1–2
  • SurgCSS combines inpainting-based class-aware blending, Class Desensitization Loss, and dynamic weight fusion to address class imbalance in continual surgical segmentation.The framework targets imbalance between old and new classes and within a time point.Source: §1 · p. 2
  • Class-aware blending with inpainting separates instrument foregrounds from high-fidelity surgical backgrounds to address data scarcity.The approach is presented as a way to construct a small class-balanced set.Source: §1 · p. 2
  • Class Desensitization Loss aims to reduce confusion between visually similar classes and correct segmentation boundary deviations.Source: §1 · p. 2
  • Dynamic weight fusion combines incremental and rebalance models to preserve detailed class representations while improving segmentation accuracy.Source: §1 · p. 2
  • The paper integrates four public datasets into a benchmark for class-incremental surgical instrument segmentation.The benchmark supports comparisons and ablation studies.Source: §1 · p. 2

2. Related Work

Related work covers general and medical continual segmentation, class-imbalance methods, and surgical image synthesis. The paper identifies imbalance, visual similarity, and data scarcity as challenges not fully addressed by prior approaches in surgical settings.

2.1. Class-Incremental Continual Learning

  • General continual segmentation methods address background shift, distillation, feature stability, or task-specific representations, while medical approaches include synthetic data and architectural adaptations.The paper notes that distillation may struggle with similar instruments near class boundaries, and parameter fusion or regularization may not address severe class imbalance.Source: §2.1 · pp. 2–3

2.2. Learning with Class Imbalance

  • Class-imbalance methods modify data distributions, learning algorithms, or both, and prior continual-learning work includes output-layer bias correction and learnable weight scaling.The paper states that medical continual segmentation has often emphasized architecture over data scarcity and imbalance.Source: §2.2 · pp. 3–4, §2.1 · p. 3

2.3. Image Synthesis

  • Surgical image synthesis ranges from synthetic datasets and GANs to diffusion and video generation, while inpainting is presented as a way to create realistic, class-balanced data.The paper contrasts synthesis with copy-paste approaches, noting the potential for new visual contexts and object appearances.Source: §2.3 · p. 4

3. Methodology

SurgCSS adds a rebalance stage after incremental training to address segmentation-head bias. It combines synthetic class-balanced data, Class Desensitization Loss, and Dynamic Weight Fusion to balance updates while retaining knowledge from the incremental model.

3.1. Problem Formulation: Incremental Stage

  • The incremental stage updates a model trained on prior classes to recognize a new class while seeking to preserve performance on earlier classes.The problem formulation uses feature- and logits-based distillation to address forgetting across disjoint task class sets.Source: §3 · p. 4, §3.1 · p. 4
  • Abundant new-class samples and limited old-class exemplars produce uneven segmentation-head updates and bias predictions toward new classes.The incremental model starts from the previous model’s weights and adds an output channel for the new class.Source: §3.1 · p. 4

3. Methodology

  • The method adds a post-training rebalance stage that freezes the feature extractor and fine-tunes the segmentation head on synthetic, class-balanced data before fusing the weights.This is intended to address imbalance between new and old classes while retaining prior knowledge.Source: §3 · p. 4, §3.2 · p. 5, Figure 2 · p. 5

3.2.1. Inpaint and Blending

  • Inpainting generates reusable surgical backgrounds, which are combined with augmented instrument foregrounds to form class-balanced synthetic samples.The inpainting model is fine-tuned on surgical data to remove instruments; foregrounds and backgrounds are augmented before blending.Source: §3.2.1 · pp. 5–6

3.2.2. Class Desensitization Loss

  • Class Desensitization Loss uses contrastive learning with class-aware blending to improve intra-class compactness and inter-class separation.It targets confused classes and complements pixel-wise cross-entropy and binary cross-entropy losses.Source: §3.2.2 · p. 6

3.2.3. Dynamic Weight Fusion

  • Dynamic Weight Fusion combines incremental and rebalanced model parameters to retain detailed knowledge while reducing bias.It selectively fuses parameters whose changes meet a dynamic threshold based on the most variable parameters after rebalancing.Source: §3.2.3 · p. 7

4.1. Implementation

The benchmark integrates four public surgical datasets into a larger, imbalanced instrument-segmentation dataset and evaluates incremental learning without access to earlier training data. Experiments use a common segmentation setup and evaluate four task configurations.

4.1.1. Datasets

  • The integrated benchmark contains 18 instrument classes across 66 videos and 26,638 frames, with instance-level masks and visually similar classes.The training and test sets contain 19,933 and 6,705 frames, respectively.Source: §4.1.1 · pp. 7–8

4.1.2. Experiments Setting

  • Four overlapped-protocol tasks—13-1, 13-5, 9-3, and 17-1—evaluate incremental learning when prior-stage data are unavailable.At each stage, future classes may appear labeled as background; x-y denotes x old classes and y classes introduced per step.Source: §4.1.2 · p. 8

4.1.3. Implementation Details

  • Experiments use DeepLabv3+ with a ResNet-101 backbone, resizing all images to 224×224 and training each incremental step for 20 epochs.The rebalance stage uses one step of three epochs and around 9 GB of VRAM.Source: §4.1.3 · p. 8

4.2. Experimental Results

Across continual-learning tasks, SurgCSS addresses class bias while preserving and improving base-class performance. On task 13-1, it reports gains over PLOP and EWF, while visual comparisons show predictions comparable to offline training.

4.2. Experimental Results

  • 27.5% and 6.15% mIoU gains over PLOP and EWF, respectively, accompany base-class improvements of 40.41% and 7.55% on task 13-1.The paper reports comparisons across tasks 13-1, 13-5, 9-3, and 17-1 on its surgical dataset and task 13-5 on a four-component dataset.Source: §4.2 · pp. 8–9, Table 1 · p. 9, Table 2 · p. 10
  • For Vessel Sealer, Other, Ultrasound Probe, Grasping Retractor Sealer, Large Needle Driver, Catheter, and Grasper, predictions are reported as better than competitors and comparable to offline training.The visual comparison notes fewer over- and under-predictions and less confusion among instrument classes.Source: Figure 6 · p. 9, §4.2 · p. 10
  • As continual learning progresses, model updates increasingly favor new classes, and the authors report more uniform summation-average and t-SNE distributions with reduced class bias.The paper attributes differing improvement magnitudes between PLOP and EWF to their distinct approaches to representation capacity and weight fusion.Source: §4.2 · p. 9

4.3. Ablation Study

The ablations examine class rebalancing, class-aware blending, background synthesis, and weight fusion. Their reported findings favor class-aware blending and weighted fusion while identifying limitations of simple overlays and low-quality synthetic data.

4.3. Ablation Study

  • The rebalance stage improves performance by 1.90%, 2.07%, and 17.61% in 13-1, 13-5, and 9-3, while CDL improves 17-1 by 4.55%.The reported DB Index, CH Index, and WCSS improvements indicate a more uniform feature distribution.Source: §4.3 · p. 10, Table 4 · p. 10

4.3.1. Class-aware Blending

  • Class-aware blending outperforms random blending as sample counts increase, while smaller sets such as 100 samples can particularly harm new-class performance.The authors attribute the overall improvement to better balance between base and new classes.Source: §4.3.1 · pp. 10–11, Table 5 · p. 12
  • LDM is selected for inpainting; directly cropping backgrounds weakens CDL's enhancement, and low-quality generation degrades performance.The authors report LDM performs strongly relative to the other evaluated generation methods.Source: §4.3.1 · p. 11, Table 6 · p. 12, Table 7 · p. 12
  • The foreground-background overlay is simple and may introduce data inconsistencies and biases because it does not effectively model their interaction.Despite this caveat, the authors report improved downstream segmentation performance.Source: §4.3.1 · p. 12
  • With α = 1.0 and β = 1.0, the model performs best on base and overall classes, with a reported 3.96% improvement.This is the reported loss-weight comparison for task 13-1 on the surgical dataset.Source: §4.3.1 · p. 11, Table 8 · p. 12

4.3.2. Fusion Strategy

  • 50.13% overall performance is achieved by Weighted Fusion with top-k = 10% and λ = 0.01, a 0.3% improvement over the 49.83% baseline.The Replace strategy falls to 45.88% overall at top-k = 1% and 43.80% at top-k = 50%.Source: Table 9 · p. 12, §4.3.2 · p. 12

5. Discussion and Conclusion

SurgCSS combines class-aware blending with inpainting, Class Desensitization Loss, and Dynamic Weight Fusion to address imbalance in continual surgical segmentation. The authors report an average 11.82% improvement over state-of-the-art continual learning methods, while noting training-cost and synthetic-data limitations.

5. Discussion and Conclusion

  • SurgCSS separates instrument foregrounds from surgical backgrounds for class-aware blending with inpainting, and uses CDL and DWF to improve class representation.The authors describe the framework as a two-stage pipeline for continual segmentation in robotic surgical scenes.Source: §5 · p. 12
  • The two-stage pipeline outperforms state-of-the-art continual learning methods by an average 11.82% in robotic surgical scenarios.The authors also report that conventional segmentation techniques degrade under class imbalance and existing continual methods do not address that imbalance effectively in surgical contexts.Source: §5 · pp. 12–13
  • The framework requires offline data generation and extra generative-model training, increasing training cost; synthetic artifacts can impair separation of visually similar instruments.A generator trained on one dataset's background characteristics may need retraining or substantial updates for significantly different surgical domains.Source: §5 · p. 13
  • The authors identify real-world clinical validation, lightweight deployment, and real-time inference in hospitals as future challenges.They call for validation of the approach's feasibility, robustness, and applicability in clinical environments.Source: §5 · p. 13

Declaration of Using Generative AI and AI-assisted Technologies

The authors report using ChatGPT to improve the paper's readability and language, with human oversight and control.

Declaration of Using Generative AI and AI-assisted Technologies

Related papers

Loading 2610.34925v1…