Source-linked AI summary
Modeling the Background for Incremental Learning in Semantic Segmentation
Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulò, Elisa Ricci, Barbara Caputo
TL;DR
The paper addresses catastrophic forgetting in incremental semantic segmentation, where background labels change meaning because each step annotates only some classes. It proposes background-aware distillation and classifier initialization, reporting improved performance over regularization-based methods on small- and large-scale datasets.
Problem
Incremental semantic segmentation must learn new classes without old training data, but background annotations can represent old or future classes and intensify catastrophic forgetting.
Method
The paper introduces a distillation-based objective with two background-shift loss terms and a classifier initialization strategy tailored to evolving background semantics.
Results
The approach outperforms regularization-based incremental-learning methods by a large margin on both small- and large-scale datasets.
Takeaways & Limitations
Explicitly modeling background semantics allows the network to learn new classes without deteriorating its ability to recognize old ones.
Takeaways & Limitations
The setting assumes label sets introduced at different learning steps are disjoint except for the void/background class, and considers training sets annotated only for new classes.
Abstract
from arXiv · showhide
Despite their effectiveness in a wide range of tasks, deep architectures suffer from some important limitations. In particular, they are vulnerable to catastrophic forgetting, i.e. they perform poorly when they are required to update their model as new classes are available but the original training set is not retained. This paper addresses this problem in the context of semantic segmentation. Current strategies fail on this task because they do not consider a peculiar aspect of semantic segmentation: since each training step provides annotation only for a subset of all possible classes, pixels of the background class (i.e. pixels that do not belong to any other classes) exhibit a semantic distribution shift. In this work we revisit classical incremental learning methods, proposing a new distillation-based framework which explicitly accounts for this shift. Furthermore, we introduce a novel strategy to initialize classifier's parameters, thus preventing biased predictions toward the background class. We demonstrate the effectiveness of our approach with an extensive evaluation on the Pascal-VOC 2012 and ADE20K datasets, significantly outperforming state of the art incremental learning methods.
1. Introduction
The paper studies incremental class learning for semantic segmentation, where catastrophic forgetting is intensified because the background class changes meaning across learning steps. It proposes background-aware distillation and classifier initialization, evaluated across established datasets and settings.
- Semantic segmentation models struggle to learn new categories while preserving performance on old ones, producing catastrophic forgetting.
- Incremental segmentation is difficult because old and unseen object classes may be labeled as background when only a subset of classes is annotated.
- The paper introduces two loss terms that explicitly model the semantic distribution shift within the background class.
- The approach combines a new objective function with a classifier initialization strategy designed to address the evolving semantics of background.
- The method is benchmarked against previous incremental class learning methods on Pascal-VOC 2012 and ADE20K under different experimental settings.
2. Related Works
Prior incremental learning research has focused mainly on image classification and has imposed restrictions in some segmentation settings. This paper formulates a broader semantic-segmentation problem and explicitly addresses background shift with comprehensive benchmark comparisons.
- Semantic Segmentation: Most semantic segmentation methods assume an offline setting in which training data for all classes is available beforehand.
- Semantic Segmentation: Earlier segmentation incremental-learning work covered specialized domains or constrained label changes, limiting its applicability.
- Semantic Segmentation: It introduces a formulation without restricting the domains or how label spaces change, and evaluates state-of-the-art methods on common segmentation benchmarks.
- Incremental Learning: Incremental learning methods are commonly categorized as replay-based, regularization-based, or parameter-isolation-based.
- Incremental Learning: The paper also uses distillation but focuses specifically on modeling background shift, which is peculiar to semantic segmentation.
3. Method
The method addresses incremental semantic segmentation by modeling the evolving semantics of the shared background class in both supervised learning and distillation. It also initializes novel-class classifiers from the old background classifier to reduce biased predictions and stabilize learning.
- Overall framework: The framework combines cross-entropy for learning new classes with distillation for preserving old knowledge.The overview describes these as the two standard incremental-learning objectives, modified to account for background semantics.
- Problem setup: Incremental segmentation expands the label set over multiple learning steps while training each updated model with data annotated for newly introduced classes.At step t, the previous label set is expanded with new classes, and the updated model must retain knowledge of earlier classes.
- Background modeling: Background annotations may include previously seen or future unseen classes, creating a semantic distribution shift across incremental steps.The shared background class denotes classes absent from the current step, which can belong to seen or not-yet-seen categories.
- Revisiting Cross-Entropy Loss: The modified cross-entropy compares background ground truth with the combined probability of background and old classes rather than background probability alone.This preserves uncertainty about whether background pixels actually belong to previously learned classes and avoids simply ignoring them.
- Revisiting Distillation Loss: The novel distillation loss keeps current probabilities normalized over all classes and matches the old background probability to the combined probability of background and novel classes.This accounts for the possibility that the old model labeled pixels from currently learned classes as background.
- Classifier initialization: Novel classifier parameters are initialized using the old background classifier so the old background probability is uniformly distributed among the new classes.The initialization follows the same background-modeling principle as the revised distillation loss and improves learning stability and final results.
4. Experiments
The experiments compare incremental-learning baselines with MiB across Pascal-VOC 2012 and ADE20K settings, using mean IoU to evaluate old and new classes. MiB consistently improves the balance between retaining old knowledge and learning new classes, with strong gains in multi-step and overlapped scenarios.
- Baselines and evaluation: The study compares six regularization-based ICL baselines, fine-tuning, joint offline training, and MiB using mean IoU averaged across classes and learning steps.The baselines include prior-focused EWC, PI, and RW, plus data-focused LwF, LwF-MC, and ILT.
- Pascal-VOC 2012 results: Nearly 10%: MiB surpasses baselines in all overlapped 15-5 cases, demonstrating its ability to use information contained in the background class.In the disjoint scenario, MiB improves over the best baseline by 4.6% on old classes, 2% on novel classes, and 4% on all classes.
- Pascal-VOC 2012 results: 15-1 multi-step learning is especially difficult: existing methods fall below 7% on both old and new classes, while MiB reaches 46.2% on old disjoint classes and nearly 13% on new classes.In the overlapped setup, MiB reaches 35.1% on old classes and nearly 13% on new classes; overall performance drops 11% because background classes are unconstrained.
- Ablation study: MiB doubles novel-class performance from 11.9% to 22.1% in 19-1 and triples it from 4.5% to 13.5% in 15-1 through classifier initialization.The ablation also finds that the modified cross-entropy and distillation loss improve preservation and learning of old and new classes.
- ADE20K results: 0.3%: MiB's gap from joint training on new ADE20K classes in the 100-50 setting, while also improving performance on old classes.LwF, ILT, and LwF-MC obtain 21.1%, 22.9%, and 34.2% on past classes, respectively, whereas MiB is reported as superior overall.
- ADE20K results: 11.9%: MiB outperforms the best baseline in the three-step 50-50 ADE20K setting and achieves the highest mIoU at every step.The largest gap occurs at the intermediate step, which combines incremental learning with subsequent preservation against forgetting.
5. Conclusions
The paper addresses incremental class learning in semantic segmentation under background-class semantic shift caused by missing annotations for old classes. Its objective function and classifier initialization strategy model this shift and substantially improve retention of old and learning of new classes.
- The study targets incremental semantic segmentation when new training data omit annotations for old classes, causing background semantic shift and worsening catastrophic forgetting.
- The proposed objective function explicitly models the evolving semantics of the background class during incremental learning.
- A classifier initialization strategy complements the objective by helping the network learn new classes without deteriorating recognition of old classes.
- The approach outperforms regularization-based incremental class-learning methods by a large margin on both small- and large-scale datasets.