PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

Umberto Michieli, Pietro Zanuttigh

arXiv:2103.06342v3cs.CVcs.AIcs.LGcs.MM

TL;DR

Class-incremental semantic segmentation must accommodate new categories without retaining previous training data, while avoiding catastrophic forgetting. The paper shapes latent representations with prototype matching, feature sparsification, and contrastive learning, complemented by output-level distillation. Evaluations on Pascal VOC2012 and ADE20K report effectiveness and state-of-the-art performance.

  • Problem

    Class-incremental semantic segmentation requires learning new categories while preserving prior categories, but conventional models trained incrementally suffer catastrophic forgetting.

  • Method

    The framework regularizes latent representations with prototype matching, feature sparsification, and contrastive learning, then complements them with output-level knowledge distillation.

  • Results

    The approach outperforms state-of-the-art continual-learning methods on standard semantic segmentation benchmarks.

  • Takeaways & Limitations

    Latent-space organization provides a common framework for different continual-segmentation protocols and can be combined with existing techniques.

  • Takeaways & Limitations

    The evaluation commonly initializes the ResNet-101 backbone with ImageNet pre-training, and pre-training can affect accuracy.

Abstract

from arXiv · show

Deep neural networks suffer from the major limitation of catastrophic forgetting old tasks when learning new ones. In this paper we focus on class incremental continual learning in semantic segmentation, where new categories are made available over time while previous training data is not retained. The proposed continual learning scheme shapes the latent space to reduce forgetting whilst improving the recognition of novel classes. Our framework is driven by three novel components which we also combine on top of existing techniques effortlessly. First, prototypes matching enforces latent space consistency on old classes, constraining the encoder to produce similar latent representation for previously seen classes in the subsequent steps. Second, features sparsification allows to make room in the latent space to accommodate novel classes. Finally, contrastive learning is employed to cluster features according to their semantics while tearing apart those of different classes. Extensive evaluation on the Pascal VOC2012 and ADE20K datasets demonstrates the effectiveness of our approach, significantly outperforming state-of-the-art methods.

1. Introduction

The paper addresses catastrophic forgetting in class-incremental semantic segmentation by organizing latent representations while retaining decoder knowledge. Its framework combines prototype matching, feature sparsification, contrastive learning, and output-level distillation across multiple continual-learning protocols.

  • Motivation: Class-incremental semantic segmentation must learn new categories while preserving performance on previously learned classes, but standard models are not designed for incremental updates.
  • Method: Output-level distillation preserves the decoder’s previous discriminative knowledge and complements latent-space constraints.
  • Method: Prototype matching preserves consistent latent representations for old classes across learning steps.
  • Method: Feature sparsification makes room for novel classes, while attraction-repulsion contrastive learning clusters same-class features and separates different-class features.
  • Scope: The framework covers sequential, disjoint, and overlapped incremental protocols and is designed to combine with existing techniques.
  • Contribution: The approach combines contrastive learning, sparsity, and prototype matching and outperforms state-of-the-art continual-learning methods on standard segmentation benchmarks.

3. Problem Definition and Setups

The paper formulates class-incremental semantic segmentation as learning expanded label sets over successive steps while organizing latent representations to preserve old classes and accommodate new ones. Its objective combines cross-entropy, prototypes matching, contrastive attraction-repulsion, sparsity, and optional output-level distillation.

  • Problem Definition and Setups: The segmentation model maps images to pixel-wise class probabilities through an encoder-decoder architecture, with encoder features aligned to downsampled segmentation maps.The predicted mask assigns each pixel the class with maximum predicted probability.
  • Problem Definition and Setups: At each continual-learning step, the previous label set is expanded with novel classes and a new subset of training data.The model therefore learns over multiple iterations rather than from the complete training set at once.
  • Problem Definition and Setups: The training objective combines cross-entropy with prototypes matching, contrastive learning, and sparsity losses, whose weights balance the multiple components.The background probability is replaced by the probability of either an old class or background in the current model.
  • Problem Definition and Setups: Prototypes matching keeps current class centroids close to representations learned in previous steps, while prototype updates use running averages and exclude unreliable background statistics.Pseudo-labels are used to compute old-class prototypes when current images lack old-class annotations.
  • Problem Definition and Setups: Contrastive learning clusters same-class features near their prototypes and separates different-class prototypes, while sparsity reduces active feature channels to free latent-space capacity for new classes.Output-level distillation additionally transfers old classifier knowledge to preserve discriminability for previous classes.
  • Problem Definition and Setups: Output-level distillation constrains the decoder to preserve previous-class behavior alongside latent-space regularization.The overview depicts this decoder constraint together with prototypes matching, clustering, sparsity, and repulsive contrastive forces.

5. Training Procedure

The experiments use Pascal VOC 2012 and ADE20K with a common DeepLab-v3+ implementation and specified optimization settings. Training uses different initial and subsequent learning rates with polynomial decay, batch size 8, and 30 epochs for Pascal VOC 2012.

  • Training Procedure: Pascal VOC 2012 provides 10,582 training images and 1,449 validation images, with 21 pixel-level classes including background.The validation split is used for testing because the test set is not publicly available.
  • Training Procedure: ADE20K is used as a second large-scale benchmark dataset alongside Pascal VOC 2012.The supplied passage identifies ADE20K as containing 22,210 images, including 2,000 referenced in the dataset description.
  • Training Procedure: All compared methods use DeepLab-v3+ with a ResNet-101 backbone and output stride 16, initialized from an ImageNet-pretrained model.The proposed strategy is described as agnostic to backbone architecture.
  • Training Procedure: Optimization uses SGD with an initial learning rate of 10^-2, 10^-3 in later learning steps, polynomial decay with power 0.9, and batch size 8.Pascal VOC 2012 models are trained for 30 epochs per learning step.

6. Experimental Results

The experiments compare SDR with continual-learning baselines across multiple incremental protocols and datasets. SDR achieves strong performance, including higher final mIoU than MiB in disjoint and overlapped VOC scenarios, while sparsity and contrastive components also benefit standard segmentation.

  • Evaluation setup: SDR is evaluated against FT, offline training, ILT, CIL, MiB, LwF, and LwF-MC using a common Deeplab-v3+ with ResNet-101 backbone.FT provides a lower limit, while offline single-shot training provides an upper reference.
  • VOC2012 results: Adding five classes at once causes a larger performance drop across methods, especially in disjoint and overlapped protocols, because of increased domain shift.The added classes belong to different macroscopic semantic groups.
  • VOC2012 results: Sequentially adding the final five classes produces the largest accuracy drop, with LwF and MiB showing poor final accuracy while ILT and CIL are comparable to SDR.The sequential setting repeatedly adds single classes from different semantic contexts.
  • VOC2012 results: SDR raises final mIoU from MiB’s 33% and 36.7% to 48.1% and 39.2% in disjoint and overlapped scenarios, respectively.The reported improvement is especially large in the disjoint scenario.
  • ADE20K results: On ADE20K disjoint scenarios, SDR outperforms competitors in every evaluated incremental scenario.The scenarios add 50 classes at once, 50 classes in groups of 10, or 100 classes in two groups of 50.
  • Ablation study: Sparsity and contrastive learning improve standard non-incremental segmentation by 0.9% on Pascal VOC2012 and 1% on ADE20K.The gains occur despite using Deeplab-v3+, already described as state of the art.

7. Conclusion

The paper shapes latent representations to reduce forgetting in continual semantic segmentation while improving recognition of novel classes. Its prototypes, sparsity, and contrastive constraints complement existing methods and are effective across the reported evaluations.

  • Contributions: Prototypes matching preserves latent consistency for old classes, sparsification limits cross-talk between class features, and contrastive learning separates features by semantic class.The techniques regularize latent-space learning while supporting novel-class recognition.
  • Implications: The proposed techniques can be combined with previous methods such as knowledge distillation and are intended for future study in standard segmentation and open-set domain adaptation.The conclusion also identifies combining the approach with output-level techniques as future work.

Supplementary Material

The supplementary material examines design choices behind latent-space constraints and reports additional ablations and qualitative or quantitative results. It identifies class-wise normalization and the chosen sparsity formulation as important empirical choices.

  • Design choices: Prototype matching uses Euclidean distance to enforce latent-space consistency for previously seen classes.The encoder is encouraged to produce similar representations for old classes at subsequent learning steps.
  • Design choices: The sparsity objective uses a ratio of a stretching function, the sum of exponentials, to a linear sum after class-wise maximum normalization.The supplementary analysis contrasts this formulation with common L0 and L1 sparsity losses.
  • Design choices: Class-wise maximum normalization gives each class equal importance, while alternatives normalize per feature, globally, or by feature-wise L2 norm.The alternatives listed are norm max, norm max overall, and norm L2.
  • Design choices: Alternative stretching functions such as squared or cubic sums can replace the sum of exponentials, but the supplementary material evaluates the selected formulation empirically.The text presents these as possible alternatives rather than reported improvements.
  • Ablations: Different normalization rules consistently reduce results, while alternative stretching functions lower final mIoU by about 2% to 4%.Entropy minimization is competitive in 15-1 but drops about 2–3% in mIoU when one class is added.
  • Feature analysis: Adding the sparsity constraint reduces spurious activations in the 0.3–0.85 range associated with uncertain and unreliable predictions.The comparison is between fine-tuning and fine-tuning augmented with the sparsity regularizer.

9. Additional Ablation Studies

Additional ablations show that limited data, rather than only novel-class learning, affects continual segmentation, while ImageNet pre-training materially influences absolute performance. The proposed method's relative improvements and method ranking remain supported under these conditions.

  • Random Split: 69.9% mIoU was achieved on a five-part random split, versus 75.4% for joint training, 5.6% for FT (disjoint), and 48.1% for SDR (disjoint).The small gap to joint training contrasts with the much larger gap to FT, supporting the importance of handling unseen classes.
  • Considerations on Pre-Training: ImageNet pre-training may limit measured forgetting and raise accuracy for concepts already represented by the backbone.The backbone was initialized on ImageNet, the standard setup in continual semantic segmentation.
  • Considerations on Pre-Training: Without ImageNet pre-training, VOC2012 reached 24.4% mIoU after 30 epochs and 40.9% after 120 epochs.The results indicate that training complex segmentation networks from scratch is difficult with the available in-the-wild data.
  • Considerations on Pre-Training: In VOC2012 15-1 disjoint, SDR achieved 13.5%, compared with 4.1% for FT and 10.9% for MiB without pre-training.Although final continual-learning mIoUs were lower, the improvement of SDR over the compared methods was preserved.

10. Additional Qualitative Results

Qualitative results across Pascal VOC2012 and ADE20K show that the proposed method better preserves old classes while accommodating new ones than the compared approaches. On ADE20K, it also maintains object-shape details and avoids background overestimation in challenging scenarios.

  • Pascal VOC2012: On Pascal VOC2012, the proposed method alleviates forgetting while accommodating new classes across sequential, disjoint, and overlapped protocols.Compared methods often become biased toward novel classes or the special background class.
  • ADE20K: On ADE20K, the method largely outperforms the compared approaches across the 100-50, 100-10, and 50-50 scenarios.Its outputs better capture object-shape details and avoid degenerating into background overestimation.

11. Qualitative Results Across Incremental Steps

Across incremental steps, qualitative results indicate that the proposed method preserves prior semantic content and accommodates new classes more robustly than MiB. The advantage is visible in both Pascal VOC2012 and ADE20K, including repeated and large class additions.

  • Pascal VOC2012: In Pascal VOC2012 disjoint 15-1, SDR preserves content semantics across five incremental steps, while MiB degenerates after introducing the sheep class.SDR's latent representations are less damaged across subsequent steps, with superior spatial localization and latent disentanglement.
  • ADE20K: The ADE20K 100-10 protocol is the most challenging because the model adapts five times to new, possibly unrelated classes.This setting is shown in Figure S7.
  • ADE20K: Across ADE20K 100-10 steps, the proposed method shows only gradual degradation, while MiB successively loses wall, curtain, and hand-basin regions.In another scene, MiB mislabels the sky as background at the third step, whereas the proposed method maintains good results.
  • ADE20K: After two ADE20K increments of 50 classes, both methods are initially precise, but MiB produces less precise maps after the second step.The proposed model identifies additional structures more reliably, consistent with preserving previous representations while accommodating new classes.

12. Quantitative Results: per-Class Accuracy

Per-class results show that fine-tuning and some competitors become biased toward newly added classes, whereas the proposed method more consistently balances retention of old classes with learning of new ones. This pattern persists across VOC2012 disjoint scenarios and multiple incremental steps.

  • VOC2012 disjoint protocols: When one class is added in VOC2012 19-1, FT and LwF-MC learn the new class but catastrophically forget previous classes, producing poor final mIoU.Their high pixel accuracy for the new class can coexist with low IoU because predictions are biased toward that class.
  • VOC2012 disjoint protocols: In VOC2012 19-1, the proposed method and other competitors distribute performance gains more evenly across classes, resulting in higher mIoUs.These methods are more balanced while learning the new class and alleviating forgetting.
  • VOC2012 disjoint protocols: In the 15-5 scenario, the proposed approach outperforms competitors in both retaining old-class knowledge and learning new classes.Fine-tuning is unable to preserve previous knowledge, while other competitors retain it better but remain below the proposed method.
  • VOC2012 disjoint protocols: In the 15-1 scenario, the proposed method outperforms all competitors and remains scalable across multiple incremental steps.Most competing approaches become biased toward the few latest classes, reducing IoU for other classes.
Loading 2103.06342v3…