SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

1 Introduction

• In CISS, background pixels can represent future classes, past classes, or true background, creating semantic drift as classes arrive incrementally.Prior methods addressed the background-label issue only partially: modified losses, while pseudo-labeled past classes but did not handle future classes.Source: §1 · pp. 1–2
• Earlier CISS methods had significantly lower mIoU than joint training with all labels, and their background-label strategies limited fine-grained per-class, per-pixel probabilities.The paper argues that made probability learning difficult by adding class probabilities to the losses, while lacked a way to handle future classes within background pixels.Source: §1 · p. 2
• SSUL-M adds an Unknown label for salient objects in the background, uses separate sigmoid classifiers and freezing for past classes, and stores a tiny exemplar memory.The memory also improves mIoU for current classes, contrary to the common belief that it only prevents forgetting of past classes.Source: §1 · p. 2
• The paper identifies multi-label prediction as another critical CISS challenge and proposes SSUL-M to address it.Unlike classification, segmentation must avoid incorrect pixel labels as well as recover correct ones.Source: §1 · p. 2
• SSUL-M achieved state-of-the-art performance on popular benchmarks, with a significantly larger margin over recent baselines as incremental tasks increase.The paper also reports extensive ablation studies and quantitative and qualitative analyses.Source: §1 · p. 2

2 Related Work

• In classification CIL, exemplar-memory methods combined with knowledge distillation achieved state-of-the-art performance by rebalancing biased classifier predictions.Neural CIL can suffer catastrophic forgetting from score bias toward new classes under imbalanced training data.Source: §2 · p. 2
• Earlier CISS methods did not use exemplar memory: addressed background-label drift, while augmented background labels with pseudo-labels.The paper presents its use of exemplar memory and potential future classes within background labels as a first for CISS to its knowledge.Source: §2 · p. 2
• Salient-object detection identifies visually distinctive objects, and SSUL uses an off-the-shelf saliency detector to define the Unknown class within the background label.Saliency maps have also been used to filter background regions for weakly supervised segmentation [29] [31] [14] and to guide mask mixing in data augmentation [17] [16].Source: §2 · pp. 2–3

3.1 Notations and Problem Setting

• Each task provides image-label pairs, and its dummy BG label can denote past objects, future objects, or true background; SSUL adds a separate unknown label.The model predicts classes learned so far or true background, while the added cu label is intended to improve learning capability for future classes.Source: §3.1 · p. 3

3.2 Two Additional Challenges of CISS

• BG labels can drift across tasks, risking past-class forgetting or interference with future-class learning; current-class bias can lower both past- and current-class mIoU.Because segmentation is multi-label, avoiding wrong pixel predictions matters alongside predicting the correct class, unlike single-label classification.Source: §3.2 · pp. 3–4

3.3 Contribution 1: Label Augmentation for BG Class

• BG-labeled salient pixels become cu, while confident previous-model predictions restore past-class labels, separating potential future objects from true background.The paper uses saliency detection for unknown labels and assigns past-class pseudo-labels only when predictions are sufficiently confident.Source: §3.3 · p. 4

3.4 Contribution 2: Stable Score Learning

• SSUL-M freezes the feature extractor and past classifiers and trains new-class filters with sigmoid-BCE, limiting noisy BG/cu targets from overwhelming frozen past-class scores.With BCE, BG or unknown scores rise only to a limited absolute level regardless of past-class scores.Source: §3.4 · pp. 5–6

3.5 Contribution 3: Tiny Exemplar Memory

• Class-balanced memory sampling preserves at least one correctly labeled example per past class, and memory samples serve as anchors for prediction precision.The authors report this strategy is more helpful than random sampling.Source: §3.5 · p. 6, §3.6 · p. 7

3.6 Summary

• The algorithm applies label augmentation and stable-score learning to current-task and memory batches, then updates memory with class-balanced sampling.Equal amounts of data are sampled from the current task and memory; memory samples act as anchor points for prediction precision.Source: §3.6 · p. 7

4.1 Experimental Setting

• Experiments use Pascal VOC 2012 and ADE20K in the overlapped setup, report mIoU, and compare regularization-based and CISS-specific baselines alongside Joint Training.Pascal VOC scenarios include 10-1, 15-1, 15-5, 19-1, and the more challenging 5-3; ADE20K has 150 stuff and object classes.Source: §4.1 · p. 7, §4.2 · pp. 7–8

4.2 Experimental results on benchmark dataset

• SSUL consistently outperforms baselines across Pascal VOC scenarios, even without exemplar memory, with larger gaps in more challenging settings.The paper notes that MiB and PLOP are robust in simpler two-task settings but drop rapidly in more challenging scenarios.Source: §4.2 · p. 7, Table 1 · p. 8
• On VOC 15-1, SSUL improves mIoU for base (0-15) and newly learned (16-20) classes across incremental steps; across 20 class orderings, SSUL and SULL-M achieve higher mIoUs and smaller variations than baselines.Source: §4.2 · p. 8
• About 12%: in VOC 15-1, storing 5–6 images per class improved mIoU for classes 16–20 by about 12%.SSUL-M further strengthens SSUL particularly for newly learned classes at t ≥2.Source: §4.2 · p. 8
• In VOC 15-1 qualitative examples, PLOP forgets intermediate-step classes and produces false positives, while SSUL and SSUL-M retain past classes and learn new ones.SSUL-M further reduces false-positive predictions relative to SSUL.Source: §4.2 · p. 8, Figure 4 · p. 9

4.3 Ablation Study

• On VOC 15-1, removing the Unknown class lowers mIoU for both base and subsequent classes; freezing helps old and new classes, while Softmax+CE increases forgetting of base classes.The authors attribute more stable score learning to Sigmoid+BCE.Source: §4.3 · p. 9, Table 3 · p. 10
• Saliency-map quality has little effect, while initializing new-class weights from the previous unknown classifier is most effective, helping limit forgetting under noisy pseudo-labels.The authors connect quicker convergence to reduced exposure to noisy pseudo-label fitting.Source: §4.3 · p. 9, Table 4 · p. 10
• Class-balanced memory sampling outperforms random sampling, especially for new classes; increasing memory helps new-class mIoU more than base-class mIoU, then performance becomes robust at sufficiently large sizes.Class-balanced sampling ensures at least one sample per class, avoiding omissions that can occur with random sampling.Source: §4.3 · p. 10, Table 4 · p. 10

5 Concluding Remarks and Limitation

• SSUL-M combines unknown-class labels, stable score learning, and tiny exemplar memory for class-incremental semantic segmentation.The authors report that the contributions were effective across various CISS scenarios and that SSUL-M significantly outperformed baselines.Source: §5 · p. 10
• Saliency-based unknown-class modeling may be unsatisfactory for stuff segmentation because saliency maps mainly target things or objects.Source: §5 · p. 10
• Freezing the model may impair learning new classes when those classes were not captured by the unknown label during the base task.Source: §5 · p. 10

Supplementary Materials for SSUL: Semantic Segmentation with Unknown Label

• The listed authors are Sungmin Cha, Beomyoung Kim, Youngjoon Yoo, and Taesup Moon.Source: Supplementary Materials for SSUL:… · p. 13
• The supplied version is arXiv:2106.11562v3, dated 19 November 2021.Source: Supplementary Materials for SSUL:… · p. 13

1.1 On model freezing and plasticity

• Model freezing improves mIoU for old and new classes; accurate old-class predictions limit false positives for new classes.The authors describe this effect as counterintuitive because freezing can seem to restrict plasticity.Source: §1.1 · p. 13
• The unknown class further improves new-class performance by learning potential-future-class representations; on VOC 15-1, SSUL also exceeds PLOP for old and new classes.Source: §1.1 · p. 13
• For VOC 15-1, exemplar memory raises new-class mIoU from 36.59% (SSUL) to 49.01% (SSUL-M).The authors attribute the gain to old-class ground-truth labels in memory reducing false positives for new classes.Source: §1.1 · p. 13

1.2 The case for stuff (area-like) and thing (object-like) classes in CISS

• Saliency-based unknown modeling is useful when true-background pixels are numerous in thing-only datasets, but may not help stuff-and-thing datasets with many true-background pixels.When such datasets contain few true-background pixels, the discussion says unknown modeling can be applied without saliency maps with reasonable performance; the authors describe the many-background case as uncommon.Source: §1.2 · pp. 13–14

1.3 The details on sampling strategy for tiny examplar memory

• Class-balanced memory sampling allocates M/|C1:t−1| samples per previous class, then updates the memory to M/|C1:t| per class as new classes arrive.Unlike random sampling, this strategy is intended to retain at least one sample for every class and avoid omitting minority classes.Source: §1.3 · p. 14

2 Additional Details of Datasets

• Pascal VOC 2012 contains 13,487 images, divided into 10,582 training, 1,449 validation, and 1,456 test images.Source: §2 · p. 14
• ADE20K contains 25,210 images, divided into 20,210 training, 2,000 validation, and 3,000 test images.Source: §2 · p. 14
• The experiments follow the same settings as PLOP.Source: §2 · p. 14

3.1 The experimental result on reducing the initial number of classes

• SSUL significantly outperforms baselines even with very small initial class sets, including VOC 5-1, 2-1, and 2-2 tasks.The passage also reports this pattern for VOC 5-3; Table 1 presents the small-initial-class results.Source: Table 1 · p. 14, §3.1 · p. 14

3.2 The additional ablation study on weight transfer

• Random initialization never matched weight transfer, even with more iterations; longer runs risked fitting noisy pseudo-labels, while shorter runs left new-class predictions inaccurate.The authors attribute the trade-off to faster convergence with weight transfer and overfitting or inaccuracy with random initialization, depending on iteration count.Source: Table 2 · p. 15, §3.2 · p. 15

3.3 The effect of using the saliency-detector

• 67.61% mIoU (+6.49%) is reached only with proper weight transfer; without it SSUL is 61.89%, near 61.12% without saliency and Unknown.Saliency raises PLOP from 54.64% to 57.01% (2.37%), still below SSUL at 67.61%.Source: Table 3 · p. 15, §3.3 · p. 15

3.4 Additional hyperparameter search for τ

• At τ = 0, performance is 67.44% all mIoU; τ = 0.7 slightly boosts performance, and the method is reported robust to τ.The authors say omitting thresholding includes some noisy pseudo-labels; thresholding is used to increase confidence and reduce noise.Source: Table 4 · p. 16, §3.4 · p. 16

3.5 The experimental result on disjoint setup

• SSUL and SSUL-M again outperform other baselines in the disjoint setup.The authors note that this setup assumes only classes observed so far appear in each input image, which may be impractical.Source: Table 5 · p. 16, §3.5 · p. 16

3.6 The details of experimental results of Pascal VOC 2012

• Table 6 reports Pascal VOC 2012 mIoU performance by class.Source: Table 6 · p. 16, §3.6 · p. 16

3.7 The details of experimental results of class orderings

• Table 7 details SSUL and SSUL-M performance across VOC 15-1 class orderings, following the Pascal VOC 2012 ordering used in PLOP.Source: §3.7 · p. 16, Table 7 · p. 17

• The authors report fewer false positives than on Pascal VOC 2012 and argue that ADE20K seldom exhibits background semantic shift because its labels for things and stuff are clear and dense.Source: §4 · p. 17
• In ADE20K 100-10 over six tasks, black Unknown pixels are correctly relabeled as future classes, including a fan at step 5 and a plate at step 6, while prior knowledge is retained.Source: Figure 1 · p. 17, §4 · p. 17