Source-linked AI summary
Incremental Learning Techniques for Semantic Segmentation
Umberto Michieli, Pietro Zanuttigh
TL;DR
Incremental semantic segmentation must learn new pixel classes while retaining performance on old ones, despite catastrophic forgetting and no access to previous images. The paper formalizes this setting and uses distillation from the previous model, including logits and intermediate features, with encoder freezing. On Pascal VOC2012, the proposed methods outperform standard fine-tuning and alleviate forgetting, although they remain below one-step training with all data available.
Problem
Incremental semantic segmentation lacks established methods for dense labeling, where images contain both new and previously learned classes and old images cannot be stored.
Method
The framework distills knowledge from the previous model through output-logit and intermediate-feature losses, combined with cross-entropy and encoder freezing.
Results
The proposed methods largely outperform standard fine-tuning on Pascal VOC2012 and alleviate catastrophic forgetting.
Takeaways & Limitations
High accuracy on previously learned classes can be preserved without storing previous images, using only the previous model during updates.
Takeaways & Limitations
The incremental results remain lower than one-step training when all training examples are available simultaneously.
Abstract
from arXiv · showhide
Deep learning architectures exhibit a critical drop of performance due to catastrophic forgetting when they are required to incrementally learn new tasks. Contemporary incremental learning frameworks focus on image classification and object detection while in this work we formally introduce the incremental learning problem for semantic segmentation in which a pixel-wise labeling is considered. To tackle this task we propose to distill the knowledge of the previous model to retain the information about previously learned classes, whilst updating the current model to learn the new ones. We propose various approaches working both on the output logits and on intermediate features. In opposition to some recent frameworks, we do not store any image from previously learned classes and only the last model is needed to preserve high accuracy on these classes. The experimental evaluation on the Pascal VOC2012 dataset shows the effectiveness of the proposed approaches.
1. Introduction and Related Work
The paper frames incremental semantic segmentation as learning new pixel classes while preserving prior knowledge without retaining old images. It addresses this gap with model-based distillation and feature-preserving strategies evaluated on Pascal VOC2012.
- Motivation: Incremental learning must add new task knowledge while preserving performance on previously learned tasks, but catastrophic forgetting undermines this goal.Earlier work primarily studied image classification and object detection, where the incremental setting differs from dense pixel labeling.
- Related Work: Existing exemplar-based methods retain selected data from previous tasks, creating privacy, storage, and scalability limitations.A prior semantic-segmentation approach stores old image patches and assumes new images contain only new classes.
- Problem Gap: Semantic segmentation introduces a distinct incremental setting because one image can contain pixels from both newly added and previously learned classes.This dense labeling structure differs from classification settings that typically produce a single label for an image.
- Contribution: The proposed framework distills knowledge from the previous model through output and intermediate-feature strategies while optionally freezing the encoder.The approach updates the current model for new classes without storing previous training images.
- Contribution: Experiments on Pascal VOC2012 report high accuracy without storing previous examples, using only the last model to preserve old-class knowledge.The paper presents this as the first evaluated standard-dataset framework for incremental semantic segmentation without retaining previously seen images.
2. Problem Formulation
The formulation evaluates incremental segmentation by balancing learning unseen classes against preserving old-class accuracy under a no-old-data setting. New training images may contain both old and new classes, with class co-occurrence affecting the task.
- Problem Formulation: Incremental semantic segmentation learns new-class pixel labels while avoiding excessive deterioration of performance on previously learned classes.Evaluation therefore considers accuracy on both new and old classes.
- Problem Formulation: The general framework does not use previously seen images, and new images contain unseen classes alongside pixels from old classes.The approach is intended to scale as the number of classes grows.
- Data Model: The dataset assigns each pixel a unique class from C, with background represented as a special class when present.Training and testing use the corresponding splits of the available image set D.
- Incremental Steps: At step k, the seen classes become S_k = S_{k−1} ∪ U_k, where U_k is disjoint from previously learned classes.The new training set contains images with pixels from S_{k−1} ∪ U_k and is disjoint from earlier training samples.
- Class Composition: New-step images are restricted to include at least one unseen class, while old-class occurrence depends strongly on which classes are added.For example, dining-table images may commonly contain chairs but rarely airplanes.
- Experimental Settings: Different class-ordering and class-batch choices produce different experiments, including single-class, batch, and sequential additions.The paper follows settings from prior work to reproduce these experiment types.
3. Methodology
The paper reframes incremental learning for semantic segmentation and develops distillation-based strategies that preserve previous knowledge while learning new classes. The methodology combines supervised learning with output- or feature-level distillation, optionally freezing the encoder.
- 3. Methodology: The proposed framework reframes incremental learning techniques for semantic segmentation and introduces strategies explicitly targeted to dense labeling.The approaches are designed to learn new classes while retaining previously learned segmentation knowledge.
- 3. Methodology: At each incremental step, the training set contains images with at least one new class and may also include pixels from previously seen classes and background.The current model is updated repeatedly as new class sets are introduced.
- 3. Methodology: The model is trained with a cross-entropy loss for class labeling and a distillation loss that retains knowledge of previously seen classes.The previous model is loaded at each step, and the two losses are combined; λD controls their balance.
- 3.1. Distillation on the Output Layer: Output-level distillation uses a masked cross-entropy between the previous and current models’ softmax outputs for already seen classes.The loss depends only on the previous model, avoiding large storage requirements.
- 3.2. Distillation on Intermediate Feature Space: Freezing the encoder constrains learning of new classes to the decoder while preserving the feature-extraction capabilities reached in the previous training stage.The encoder-freezing variant was evaluated both with and without distillation loss.
- 3.2. Distillation on Intermediate Feature Space: Feature-level distillation applies an L2 loss to keep intermediate representations close between the previous and current encoders.The L2 choice reflects that the distilled layer is an internal feature stage rather than a classification layer; combining the described losses did not provide relevant enhancements.
4. Experimental Results
Experiments on Pascal VOC2012 evaluate incremental semantic segmentation when one or five classes are added, either simultaneously or sequentially. Knowledge distillation and encoder freezing reduce forgetting while preserving learning of new classes.
- Experimental settings: Pascal VOC2012 experiments use Deeplab v2 and evaluate adding one class, five classes simultaneously, or five classes sequentially.Results are computed on the validation split because the test set is unavailable.
- Addition of one class: Fine-tuning after adding only tv/monitor reduces mIoU to 65.1%, versus 73.6% when all 21 classes are learned together.Fine-tuning also produces only 20.1% IoU for tv/monitor despite high pixel accuracy, because of false positives.
- Addition of one class: Output distillation alone improves average mIoU by 3.3%, while freezing the encoder alone improves it by 5.4% over the standard approach.Both procedures act as regularization constraints and can improve performance on old and newly added classes.
- Addition of one class: Combining encoder freezing with output distillation reaches 71.5% mIoU, a 6.4% improvement over standard training.The combined approach also improves performance on the new class relative to either method alone.
- Addition of five classes: Adding five classes simultaneously causes lower results and stronger forgetting, while output distillation generally outperforms the other approaches.Visually similar new classes can mislead old classes; sequential addition yields a 20% mIoU gain for the best proposed method over standard training.
- Addition of five classes: Performance drops depend on the added class, with the largest drops occurring when sheep or train are introduced because they are sparsely correlated with other classes.These classes mainly appear alone or with person.
5. Conclusion and Future Work
The paper introduces incremental semantic segmentation methods that combine distillation losses with encoder freezing, using only the previous model rather than stored images. On Pascal VOC2012, the proposed methods outperform standard fine-tuning, although incremental segmentation remains challenging compared with one-step training.
- 5. Conclusion and Future Work: The proposed framework combines task-specific distillation losses with cross-entropy and encoder freezing to learn new classes while preserving old ones.The distillation losses operate in the output and intermediate-feature spaces.
- 5. Conclusion and Future Work: The method requires no stored images from previous datasets and uses only the previous model, reducing memory consumption.
- 5. Conclusion and Future Work: On Pascal VOC2012, the proposed methods largely outperform standard fine-tuning and alleviate catastrophic forgetting.
- 5. Conclusion and Future Work: Incremental semantic segmentation remains challenging because its results are lower than those of the same architecture trained once with all examples available.