PaperlayerView exact revision on arXiv

Source-linked AI summary

PLOP: Learning without Forgetting for Continual Semantic Segmentation

Arthur Douillard, Yifu Chen, Arnaud Dapogny, Matthieu Cord

arXiv:2011.11390v3cs.CV

TL;DR

Continual semantic segmentation must add new classes while limiting catastrophic forgetting, which is aggravated when old classes are collapsed into the background. PLOP combines Local POD feature distillation with entropy-based pseudo-labeling to preserve spatial relationships and identify old-class background pixels. It significantly outperforms state-of-the-art methods across existing and newly proposed challenging CSS benchmarks.

  • Problem

    Continual semantic segmentation faces catastrophic forgetting and background shift because old-class pixels are collapsed into the background at later learning steps.

  • Method

    PLOP combines Local POD multi-scale feature distillation with confidence-based, entropy-thresholded pseudo-labeling of old classes within current background pixels.

  • Results

    PLOP significantly outperforms state-of-the-art approaches in existing CSS scenarios and several newly proposed challenging benchmarks.

  • Takeaways & Limitations

    Preserving short- and long-range spatial relationships while retrieving old classes from background provides PLOP’s supported strategy for addressing both CSS challenges.

Abstract

from arXiv · show

Deep learning approaches are nowadays ubiquitously used to tackle computer vision tasks such as semantic segmentation, requiring large datasets and substantial computational power. Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. However, continual learning methods are usually prone to catastrophic forgetting. This issue is further aggravated in CSS where, at each step, old classes from previous iterations are collapsed into the background. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level. Furthermore, we design an entropy-based pseudo-labelling of the background w.r.t. classes predicted by the old model to deal with background shift and avoid catastrophic forgetting of the old classes. Our approach, called PLOP, significantly outperforms state-of-the-art methods in existing CSS scenarios, as well as in newly proposed challenging benchmarks.

1. Introduction

Continual Semantic Segmentation updates segmentation models with new classes, but faces catastrophic forgetting and ambiguous background labels. PLOP addresses these challenges through feature distillation and pseudo-labeling, and outperforms state-of-the-art approaches across evaluated scenarios.

  • Motivation: Continual Semantic Segmentation (CSS) enables models to learn new classes sequentially without retraining from scratch.Unlike traditional segmentation, CSS does not require all classes to be known and learned simultaneously.
  • Challenges: Catastrophic forgetting causes neural networks to abruptly lose previously learned knowledge when learning new information.This challenge is especially important when previously seen data cannot be stored for privacy reasons.
  • Challenges: Background shift makes current background pixels semantically ambiguous because they may belong to real background, old classes, or future classes.Without distinguishing old-class pixels from true background, background shift can worsen forgetting.
  • Approach: PLOP combines multi-scale feature distillation with confidence-based pseudo-labeling to preserve spatial knowledge and recover old classes within current background pixels.Local POD preserves short- and long-range spatial relationships, while pseudo-labeling uses old-model predictions to identify old classes.
  • Evaluation: PLOP significantly outperforms state-of-the-art methods in existing CSS scenarios and newly proposed challenging benchmarks.The paper reports improvements across several datasets and evaluation settings.

2. Related Work

Related work spans offline semantic segmentation, continual learning, and emerging continual semantic segmentation. Existing CSS methods face an unrealistic annotation assumption and insufficient preservation of spatial information, motivating PLOP’s treatment of background shift and forgetting.

  • Semantic Segmentation: Semantic segmentation methods improve accuracy by incorporating spatial or contextual information through architectures and mechanisms such as atrous convolution, encoder-decoder designs, and attention.Strip Pooling and POD-related approaches capture global and local statistics through spatial aggregation.
  • Continual Learning: Continual learning methods address catastrophic forgetting through rehearsal, adaptive architectures, and dynamically specialized subnetworks.Rehearsal may retain raw images, compressed features, or generated training data.
  • Continual Semantic Segmentation: Continual semantic segmentation remains relatively new, with prior work extending continual learning methods from medical applications to general segmentation.Segmentation is mostly studied offline, while continual learning has generally focused on image classification.
  • Continual Semantic Segmentation: Prior general CSS work assumes previously learned categories are properly annotated in new images, overlooking semantically ambiguous background pixels.These background pixels can represent old classes, future classes, or real background.
  • Continual Semantic Segmentation: Existing CSS methods address background shift and forgetting with output-level losses, but output distillation alone is insufficient to preserve spatial information.This limitation motivates feature-level preservation of spatial relationships.

3. PLOP Segmentation Learning Framework

PLOP addresses catastrophic forgetting and background shift in continual semantic segmentation by combining Local POD feature distillation with uncertainty-based pseudo-labeling. Local POD preserves spatial relationships at multiple scales, while pseudo-labels recover old-class pixels hidden in the background.

  • Continual semantic segmentation framework: CSS training collapses old and future classes into background, making background pixels ambiguous and aggravating catastrophic forgetting.At step t, the model must predict all previously seen classes even though ground-truth masks label only current classes and background.
  • Multi-scale local distillation with Local POD: PLOP uses feature distillation between old and current models to retain knowledge while balancing rigidity against plasticity.The distillation loss compares representations from intermediate feature levels rather than directly constraining raw predictions.
  • Multi-scale local distillation with Local POD: Local POD pools feature embeddings over multiple spatial regions and scales, preserving both long-range and short-range spatial relationships.Its first scale is equivalent to global POD, while subsequent scales constrain statistics over increasingly local regions.
  • Solving background shift with pseudo-labeling: PLOP pseudo-labels background pixels using old-model predictions and filters uncertain labels with class-specific entropy thresholds.Non-background pixels retain ground-truth labels, while accepted old-class pseudo-labels are incorporated into the classification loss.
  • Solving background shift with pseudo-labeling: The total PLOP objective combines pseudo-label classification with Local POD distillation, weighted by the hyperparameter λ.The pseudo-label term is adaptively weighted using ν, the ratio of accepted old-class pixels to all such pixels.

4. Experiments

PLOP is evaluated across established and newly proposed continual semantic segmentation protocols, datasets, class orders, and domain-shift settings. It generally improves performance, stability, and resistance to forgetting relative to prior methods.

  • Datasets and protocols: PLOP is evaluated on Pascal-VOC 2012, ADE20k, and Cityscapes using multiple continual semantic segmentation protocols.The evaluation includes class-incremental and continual-domain settings, with testing over all classes seen over time.
  • Quantitative evaluation: +86% on all classes, +90% on old classes, and +56% on new classes versus MiB in the most challenging VOC 15-1 setting.PLOP also reports a joint-model mIoU of 77.40% and a +24% average-mIoU improvement over MiB.
  • Quantitative evaluation: 2.35 p.p is PLOP’s improvement over MiB on ADE20k 100-10, the longest setting reported in the main ADE20k experiments.PLOP is comparable to MiB on 100-50 and improves by 1.09 p.p on 50-50.
  • Quantitative evaluation: PLOP’s mIoU remains high over VOC 15-1 steps while MiB’s quickly deteriorates.The evolution is attributed to improved resilience to catastrophic forgetting.
  • Stability: Across 20 random VOC 15-1 class orderings, PLOP surpasses MiB in average mIoU and has lower variability.The reported standard deviation on all classes is 5% for PLOP versus 10% for MiB.
  • Ablation and introspection: The ablation study reports a +20 p.p advantage for the full Local POD configuration across all metrics.The authors attribute this to integrating long- and short-range dependencies; pseudo-labeling also improves old- and new-class performance.
  • Model introspection: Visualization shows MiB forgetting initial classes and favoring new ones, whereas PLOP predicts old and new classes more consistently.This behavior is illustrated across time on VOC 15-1 test images, including after train causes dramatic background shift.

5. Conclusion

The conclusion identifies catastrophic forgetting and background shift as central CSS challenges and presents PLOP as addressing both. Across existing and newly proposed benchmarks, PLOP performs significantly better than existing baselines.

  • Conclusion: PLOP addresses catastrophic forgetting with Local POD, which preserves long- and short-range spatial relationships during continual learning.The method is described as offering a trade-off between rigidity and plasticity for CSS.
  • Conclusion: PLOP addresses background shift with an efficient pseudo-labeling strategy for old-class pixels in the background.
  • Conclusion: PLOP performs significantly better than all existing baselines across the paper’s CSS benchmarks.The benchmarks include existing scenarios and novel evaluations of long-term learning capacity and stability.

A.1. Further Work

The paper handles old and current classes in its CSS setting, while identifying the use of future-class information as a direction for further work.

  • Further work: The proposed treatment covers old and current classes, but does not exploit future-class information already present in task pixels.The authors suggest investigating zero-shot approaches for this future information.

A.2. Algorithm view of Local POD

The Local POD algorithm distills representations by comparing multi-scale pooled embeddings from current and old-model feature maps across selected layers.

  • Algorithm: Distillation applies Local POD across all selected feature layers and accumulates the resulting loss.
  • Algorithm: LocalPOD computes an L2 distance between current- and old-model POD embeddings.
  • Algorithm: The embeddings are computed over S spatial scales and local subsets of the feature maps.The algorithm iterates over scales and spatial windows before concatenating the pooled embeddings.
  • Algorithm: The algorithm returns the accumulated distillation loss after processing the selected layers.

A.3. Reproducibility

The reproducibility setup evaluates PLOP across three segmentation datasets with fixed architectures, training schedules, class orders, and implementation choices. Experiments use shared initial-step weights and no task ID at inference.

  • Datasets: Experiments cover Pascal-VOC, ADE20k, and Cityscapes, spanning 20, 150, and 19 classes respectively.The datasets contain 10,582/1,449, 20,210/2,000, and 2,975/500 training/testing images, respectively.
  • Implementation details: All experiments use DeepLab-V3 with a ResNet-101 backbone pretrained on ImageNet.The uncertainty threshold is τ = 1e−3; models train for 30 Pascal-VOC epochs and 60 ADE20k epochs per CSS step.
  • Implementation details: Local POD uses three spatial scales—1, 1/2, and 1/4—because adding more scales produced diminishing returns experimentally.It is applied before ReLU with squared pixel values, using batch size 24 across two GPUs.
  • Class ordering details: Pascal-VOC and ADE20k use fixed class orderings [1, 2, ..., 20] and [1, 2, ..., 150] across evaluated models.Continual-domain Cityscapes follows a specified ordering of domains/cities.
  • Reproducibility setup: The background class is always learned during the first task, and weights trained at the initial step are reused across models.Pascal-VOC 15-1 and ADE20k 100-10 training took less than 2 and 16 hours, respectively.

A.4. Additional Experiments

Additional experiments examine component ablations and alternative Pascal-VOC continual-learning scenarios. They report strong gains from pseudo-labeling and broad superiority of PLOP over prior methods in the evaluated comparisons.

  • Model ablation: Pseudo-labeling alone raises validation mIoU from 3.99 to 19.74 in the reported Pascal-VOC ablation.Table 7 evaluates PLOP components in 15-5 and 15-1 using 20% of the training set for validation.
  • Pascal-VOC 2012 Disjoint: In Disjoint Pascal-VOC scenarios, PLOP matches MiB in 15-5 but significantly outperforms previous state-of-the-art methods in 19-1 and 15-1.The 15-5 differences between PLOP and MiB are reported as not significant.
  • Pascal-VOC 2012 Overlap: In Overlap Pascal-VOC scenarios, PLOP surpasses naive Fine Tuning, PI, RW, LwF, and other evaluated methods across all CSS scenarios.The comparison includes weight-constraint and knowledge-distillation baselines.
  • Pascal-VOC 2012 Disjoint: Overlap is described as more realistic than Disjoint for Pascal-VOC comparisons.The paper reports Disjoint results primarily for comparison.
Loading 2011.11390v3…